LogQL and PromQL range queries now match Loki and Prometheus exactly
What changed
- Range queries evaluate on the same timestamps as Loki and Prometheus (#1014, #1027). Dashboards built against Loki or Prometheus now show the same points and values in Gigapipe. Graphs may shift slightly after upgrading: some points move to the correct timestamp, and some values change by the few lines that sit exactly on a window boundary.
- Exact (T−R, T] windows on the fast path. rate and count_over_time with no parser, keep, drop, unwrap or non-empty line filter are served from the metrics_15s rollup when every evaluation time, the range and the offset are multiples of 15 seconds. These queries now count exactly what Loki counts, including lines stamped exactly on a window edge.
- Faster dashboards over long ranges. A 7-day sum by (…) (rate(…[5m])) at a 60s step over 151M log lines reads about 40M rows from the rollup and returns in under 2 seconds on a laptop. Earlier builds of this fix took 14–28 seconds and 3.3 GiB of memory.
- Label lookups stay bounded to the streams that have data in the queried window (#702), on single nodes and clusters.
- Timestamps in Loki and Prometheus query parameters are read in any unit (#1051, thanks @Guess-Who-sec). The query, label and series endpoints used to assume a fixed unit. A client sending a different one got a 1970 time range back, meaning empty results or a full table scan; Grafana Logs Drilldown sending seconds to a Loki endpoint that expects nanoseconds was one case. Timestamps are now recognised as seconds, milliseconds, microseconds or nanoseconds from their size, and RFC3339 strings are accepted too. Requests that already worked return the same results. Affected endpoints: Loki query_range, query, tail, index/stats, labels, label values and series, plus Prometheus label values.
Schema change: applied automatically
This release adds a start_count column to metrics_15s and updates its materialized view. No manual migration script is needed. On startup, Gigapipe in MODE=all (the default), MODE=writer or MODE=init_only:
- adds the column and updates the view in place: a metadata change, with no data copied or backfilled;
- records a cut-off time in the settings table, at least one minute in the future.
On clusters, both run ON CLUSTER.
What to be careful with
- Upgrade the writer before the readers. Readers never change the schema. A new reader running against a schema that hasn't been upgraded still returns correct results, but takes a slower path on every fast-path query until the writer, or a MODE=init_only run, has upgraded the schema.
- If you set OMIT_CREATE_TABLES=true, nothing upgrades on its own. Run the new image once with MODE=init_only against your database, or apply the schema change yourself. Until then, readers stay on the slower path.
- Data from before the upgrade is queried more slowly until it ages out. Rollup rows written before the cut-off don't have start_count. Windows before the cut-off stay exact, but also read the raw lines on their window edges from samples_v3. At short steps that can mean scanning most of the raw table for that period. The cost goes away as TTL removes pre-upgrade data. Dashboards over recent data aren't affected.
- Keep the samples_v3 and metrics_15s TTLs equal while pre-upgrade data is still queried (METRICS_15S_TTL_DAYS). Windows before the cut-off need both tables to hold the same lines. If one has dropped data the other still holds, those counts are wrong.
- Per-stream queries hold one value per stream and evaluation time while they run. That covers max, min, count, avg, stddev or stdvar by, topk, and no aggregation; a sum doesn't. On very high stream counts over long ranges, size the ClickHouse server profile accordingly: max_memory_usage caps a query, and max_bytes_before_external_group_by and max_bytes_before_external_sort let it spill to disk instead of failing.
What to look for after upgrading
- The cut-off was recorded: the log line metrics_15s: start_count from appears at startup. On a fresh install it reads 1970-01-01T00:00:00Z.
- The view has the new column: SHOW CREATE TABLE metrics_15s_mv includes start_count.
- If you run with METRICS_15S_ENABLED=false: that view still contains WHERE samples.type != 2.
- No leftover metrics_15s_mv_bak table. If one exists, startup recovers it automatically.
- Small value changes on existing dashboards are expected, typically one or two lines per point at window edges.
Rollback
Rolling back to the previous version is safe: older binaries ignore the new column. Upgrading again afterwards picks up where it left off.
Compatibility
Tested on ClickHouse 25.3 and 26.10, single node and clusters.
What's Changed (by PRs)
- fix(reader): use magnitude-aware time parsing in Loki query handlers by @Guess-Who-sec in #1051
- fix(query): evaluate PromQL and LogQL range queries on the request grid by @44YHC in #1028
Full Changelog: v5.5.5...v5.6.0