At a high level, this release contains:

• Stable version of new HA system (was previously experimental)
• Support for multi-tenancy
• Improved PostgreSQL permissions
• Add text format ingest
• Improvements to locking behavior

Stable version of new HA system

This release stabilizes our new system for supporting Prometheus running in high-availability mode (the previous release had an experimental version). This new HA system allows for easier deployments by allowing Prometheus and Promscale to scale independently and communicate through a load-balancer.

The new system works as follows: Prometheus is configured to attach external labels named cluster and __replica__ to the samples it sends. Each Prometheus replica of an HA Pair would send the same cluster label but a unique __replica__ label. Promscale would then elect a single Prometheus replica per cluster to write data at any point in time and would automatically failover if that Prometheus replica dies.

When deploying this new HA system, any Prometheus instance can send data to any Promscale instance via a load-balancer or similar mechanism. The correctness of the "replica election" does not depend on any mapping between Prometheus and Promscale nodes and is correct no matter the number of Prometheus or Promscale instances. This new system is enabled on the Promscale side with the CLI flag --high-availability and relies on the Prometheus instance being configured with cluster and __replica__ external labels.

The stabilization process included additional support for late and out-of-order data.

Support for multi-tenancy

Promscale supports multi-tenancy by having different Prometheus servers corresponding to different tenants all write data to the same set of Promscale servers. Each Prometheus server will indicate the tenant it is writing using either custom headers or external labels. Tenant= information will be saved in a special __tenant__ label.

When executing PromQL queries, a Promscale server can be configured to either return data for all tenants or to restrict its answers to a particular tenant or set or tenants. In this way, a tenant can be given access to a particular Promscale PromQL endpoint for
querying its own data while being sure that it could not access other tenant's data.

Improved PostgreSQL permissions

Promscale adopts a Role Based Access Control (RBAC) philosophy to managing permissions at the SQL layer. Many users will choose to simply use a single PostgreSQL user for all Promscale operations. But we also allow users more granular control over permissions management. This release includes significant improvements to this system to allow better control of which permissions to grant different
Promscale instances.

Please see the documentation for more info.

Add text format ingest

Add Prometheus and open-metrics text format support to ingest pipeline. This allows users to push data in this format directly to Promscale.

Improvements to locking behavior

This is a performance enhancement to improve the speed with which new metrics, series, and labels are created.

CLI flag changes

• Renamed -enable-ha experimental flag to -high-availiability to keep consistent with other modes (e.g. -read-only).
• Added promql-max-points-per-ts to allow configuration of max points per timeseries when executing PromQL queries.
• Added -promql-lookback-delta and -promql-max-samples (int64) to allow PromQL evaluation as per the query. For more info, see PromQL evaluation flags.

Prom-migrator

• Add support for TLS certificates
• Add support for reading auth info from file
• Ensure time-ordering of samples when using concurrency (necessary for writing to long-term stores that require time-ordered writes)
• First non-beta release, reflecting the increased stability of prom-migrator.

Thanks

@niconosenzo for PR #551
@Pizzahd88 for issue #576
@Jontte for issue #582
@andrejpodzimek for issue #572
@mprochowski for PR #602
@stappersg for PR #611

Changelog

1df71c9 -version flag returns exit code 1
a83c617 Add Bors Integration
4f6eb08 Add DB connection handling end-to-end test
c2bca96 Add E2E tests for plain multi-tenancy type.
0bcd7a9 Add Prometheus and open-metrics text format support to ingest pipeline
3f95f75 Add SQL Update script
3769a7f Add backfilling support for HA mode
c5a55f6 Add header in multi-tenant files.
178b935 Add more privilege testing in upgrade snapshot
c1c3173 Add new HA method to docs
319a669 Add pprof mutex profile
218f52e Add unit tests for multi-tenancy.
943e794 Adds support for multi-tenancy in Promscale.
3db9961 Adjust read-only role tests
cb2e867 Change the add-key lock to the series instead of the data table
6eb2a39 Cleanup of SQL permissions
891f40c Compose multi-tenancy types to interfaces.
d1efcb8 Documentation for roles and permissions
47f9dd6 Documents ways to delete data in Promscale.
9433428 E2E tests for token-based multi-tenancy type.
c56267f Extend E2E tests to ensure time-orderedness.
9c6e713 Fix HA bug with last write interval
c763342 Fix cross platform docker
26a5303 Fix delete end-to-end tests
ff4af80 Fix duplicate constant
68f40f2 Fix gaps in graphs when migrating on time-based storages.
4f0d9e9 Fix golden file end-to-end test for latest TSDB version
da39c1d Fix permissions system
c8bd679 Fix prom-migrator version flag
9c2b397 Fix regex anchor
c640aa1 Fix upgrade tests on Mac
0cf42bb Fixes for multinode support
b04dbc5 Further simplify HA Lease and LeaseClient implementation and interface
40cc63a Give prom_admin all permissions from prom_modifier and prom_maintenance.
5e8966f HA service/lease refactor
4bfcbab Implement flag to optionally accept non-mt writes.
7e19b4d Improve locking on metric creation.
bc3ab1b Improve password masking utility
4fa4edb Improve password masking utility
03ea762 In new HA setup read path fails when the read query comes from prometheus as promethues appends replica labelset from external labelset for readRead requests. So dropping the replica labelSet in labelmatchers for read requests, when Promscale is in HA mode.
9a6336d Integrate multi-tenancy with ingestor and querier.
5f2908b Lockdown execute_everywhere
bd4d97f Maintenance also inside docker container
82c5809 Make PromQL engine max-sample and look-back delta customizable.
2425c04 Make per timeseries points limit configurable.
9ca4998 Prepare for the 0.4.0 release
8992b93 Prepare for the next development cycle
ffb68d0 Prom-migrator: Support for TLS certificates & auth from file.
4302d6f Refactor auth in prom-migrator to be passed as config.
338d2e2 Refactor data parsing to use an HA data filter
1843a5a Refactor ingest path
8135579 Refactor priviliges to avoid default privileges
ef10ff2 Refactor snapshot to give better diff output
9b0e6c8 Refactor tenancy to implement preprocessing stage.
a99b028 Refactor write pipeline to support multiple formats and preprocessors
3af97e9 Replace timescale extension schema with a variable
0fb7353 Small fixup for release
769fe6e Split label fetching into txns of < 1000 labels
577b62a Store test logs as GA artifacts
af6453b Switch to session locks for maintenance tasks
8a0b5b1 Tighten function permissions.
9b49a86 Turn off ingestor in read-only mode
ba0b61b Update PromQL upto 27th april.
77815a3 Use small datasets for non-extended unit-tests.
c876f5f add docker-compose file without dependency to build promscale image on setting up Promscale.
80a53a8 change the default log level to info from debug
a96f306 fix ha write logging
4956753 provide option for image pull policy in helm chart

Docker images

• docker pull timescale/promscale:0.4.0
• docker pull timescale/promscale:0.4
• docker pull timescale/promscale:latest