grafana/loki v3.3.0 on GitHub

3.3.0 (2024-11-19)

⚠ BREAKING CHANGES

blooms: Introduce a new block schema (V3) (#14038)
blooms: Index structured metadata into blooms (#14061)

Features

ability to log stream selectors before service name detection (#14154) (d7ff426)
add app_name as a service label (#13660) (f2a16f4)
add backoff mechanism to the retention process (#14182) (3136880)
add functions to common.libsonnet for warpstream (#14123) (2bde071)
add gauge loki_ingest_storage_reader_phase (#14679) (f5b0fb6)
add gauge to track the partition_id (#14713) (a142b3d)
add missing cluster label to mixins (#12870) (547ca70)
add query user and query source to "executing query" log lines (#14320) (4d69929)
add retries for s3 ObjectExists calls (#14062) (73cbbb0)
add structured metadata to the promtail push API (#14153) (66cffcb)
Add support for partition ingester in dashboards (#14498) (70deebf)
Allows to configure client_max_body_size (#12924) (809a024)
Apply patterns line length limit to json message key (#14296) (41fafd8)
blooms: Add bloom planner and bloom builder to backend target (#13997) (bf60455)
blooms: disk-backed queue for the bloom-planner (backport k227) (#14927) (1f6828b)
blooms: Index structured metadata into blooms (#14061) (a2fbaa8)
blooms: Only write key and key=value to blooms (#14686) (3af0004)
Configurable list of json fields to mine patterns (#14528) (7050897)
detected field values (#14350) (7983f94)
distributors: Use a pool of worker to push to ingesters. (#14245) (f80d68a)
Do not add empty blooms to offsets (#14577) (51c42e8)
Extract task computing into a strategy interface (#13690) (ab5e6ea)
fluentd-plugin-datadog-loki: support custom http headers (#14299) (e59035e)
helm: ✨ add additional service annotations for components in distributed mode (#14131) (5978f13)
helm: add configurable extraEnvFrom to admin-api and enterprisegw (#14533) (5d78a3a)
helm: Add kubeVersionOverride for Helm chart (#14434) (0935d77)
helm: Add persistence option to memcached on Helm chart (#13619) (ef1df0e)
helm: add tolerations to pattern-ingester statefulset (#13605) (09530c0)
helm: Allow setting node attributes to tokengen and provisioner (#14311) (c708ae6)
helm: Replace bloom compactor with bloom planner and builder (#14003) (08e61ca)
helm: update chart with loki version 3.2.0 (#14281) (11b92ee)
Helm: Update Loki Helm chart for restricted environments (#14440) (adc7538)
implement IsRetryableErr for S3ObjectClient (#14174) (fc90a63)
Implement owned streams calculation using Partition Ring (#14282) (3c36ba9)
Implement WAL segment ingestion via Kafka with partition ring (#14043) (d178f4c)
Improve pattern ingester tracing (#14707) (80aec25)
ingester: implement partition shuffle sharding for ingester (#14304) (1a4436c)
Introduce new ObjectExistsWithSize API to (#14268) (ac422b3)
Introduce shardable probabilistic topk for instant queries. (backport k227) (#14765) (02eb024)
jsonnet: Allow to name prefix zoned ingesters (#14260) (fac3177)
kafka: Add Ingestion from Kafka in Ingesters (#14192) (b6e9945)
kafka: Add support for SASL auth to Kafka (#14487) (e2a209c)
kafka: Enable querier to optionally query partition ingesters (#14418) (633bb5e)
kafka: enqueue commit offset only once per batch process (#14278) (beca6f3)
kafka: Implement limiter using partition ring for Kafka (#14359) (5cbb239)
kafka: Remove rate limits for kafka ingestion (#14460) (83a8893)
kafka: Replay kafka from last commit before allowing ingesters to become ready (#14330) (39b57ec)
kafka: Start ingester flush loop before trying to catch up from Kafka (#14505) (524ed81)
logcli: add gzip compression option (#14598) (4d3f9f5)
loki: include structured_metadata size while asserting rate limit (#14571) (a962edb)
max-allowed-line-length: add config to set max-allowed-line-length in pattern ingester (#14070) (0780456)
mixin / add loki compaction not successfull alert (#14239) (da04f50)
mixin, allow overriding of some labels by parameterizing mixin recording/alert rules (#11495) (f1425b6)
mixins / allow bloom dashboards disabling (#14177) (ce2e6d5)
mixins: Allow hiding useless rows in loki-operational (#13646) (3aa4f22)
mixins: merge resources dashboards for ssd into one (#13471) (45b8719)
move detected field logic to query frontend (#14212) (36ace66)
move metric aggregation to a per-tenant config (#14709) (c1fde26)
New bloom planning using chunk size TSDB stats (#14547) (673ede1)
Optionally require writes to kafka on Push requests (#14186) (7c78232)
revert "feat: add functions to common.libsonnet for warpstream" (#14129) (18c27f9)
ruler: enables ruler store that uses clients from thanos-io/objstore pkg (#11713) (8bca2e7)
storage: AWS backend using thanos.io/objstore (#11221) (b872246)
storage: Azure backend using thanos.io/objstore (#11315) (5824e3d)
storage: GCS backend using thanos.io/objstore (#11132) (c059ace)
support ruler sidecar in singleBinary mode (#13572) (684baf7)
track discarded data by usageTracker (#14081) (c65721e)

Bug Fixes

detected_fields: return parsed labels when parsers are passed (#14047) (aa1ac99)
Add additional validation for timeout while retrieving headers (#14217) (8322e51)
Add s3 principal to iam policy attached to sqs in lambda-promtail terraform code (#14619) (db0889e)
Add tenant limits as dependency to pattern ingester (#14665) (31eea90)
aggregated-metrics: correctly create logfmt string (#14124) (63e84b4)
allow any level for aggregated metrics (#14255) (c001a1d)
allow rename of structuremetadata labels (#13955) (2d4792a)
always write detected_level when enabled, even if unknown (#14464) (41c6b6c)
blooms: Check length of tasks before accessing first element in slice (#14634) (601f549)
blooms: Copy chunks from ForSeries (backport k227) (#14864) (d10f79c)
blooms: Do not restart builders when planner disconnects (backport k227) (#14922) (213e8ee)
blooms: Exclude label filters where label name is part of the series labels. (#14661) (d1668f6)
blooms: Fix panic in initialisation of the bloom planner and builder (#14110) (8307c42)
blooms: Fix strategy logger and add task test (backport k227) (#14921) (dc36a1e)
blooms: Fix tenants slice on loadTenantTables (backport k227) (#14901) (540dd5a)
blooms: Skip multi-tenant TSDBs during bloom planning (backport k227) (#14888) (631cff3)
build: Use Debian Bullseye base image for build image (#14368) (3beb8ff)
canary: Reconnect immediately upon tail max duration (#14287) (9267ee3)
ci: fixed Publish Rendered Helm Chart Diff workflow (#14365) (6de6420)
ci: updated helm diff rendering workflow (#14424) (916e511)
config: Copy Alibaba and IBM object storage configuration from common (#14297) (59ff1ec)
config: migrate renovate config (#14646) (a67d8ef)
correct _extracted logic in detected fields (#14064) (1b3ba53)
correct OTLP documentation typo (#14602) (063c590)
deps: update aws-sdk-go-v2 monorepo (#13986) (6f49123)
deps: update aws-sdk-go-v2 monorepo (#14742) (53a1ab7)
deps: update github.com/grafana/dskit digest to 687ec48 (#14395) (c2f38e1)
deps: update github.com/grafana/dskit digest to 7c41a40 (#14277) (f39cdbd)
deps: update github.com/grafana/dskit digest to 931a021 (#14032) (7c18642)
deps: update github.com/grafana/dskit digest to b69ac1b (#14355) (9d7a6ea)
deps: update github.com/grafana/dskit digest to f52de24 (#14319) (a4f3edf)
deps: update github.com/twmb/franz-go/pkg/kfake digest to cea7aa5 (#14590) (688c42a)
deps: update k8s.io/utils digest to 702e33f (#14033) (b7eecc7)
deps: update module cloud.google.com/go/bigtable to v1.33.0 (#14580) (a0920ed)
deps: update module cloud.google.com/go/pubsub to v1.45.0 (#14361) (4351238)
deps: update module cloud.google.com/go/pubsub to v1.45.1 (#14650) (f173708)
deps: update module cloud.google.com/go/storage to v1.46.0 (#14744) (8e45116)
deps: update module github.com/alicebob/miniredis/v2 to v2.33.0 (#14721) (7bfda25)
deps: update module github.com/aws/aws-sdk-go to v1.55.5 (#14715) (03f0f5a)
deps: update module github.com/axiomhq/hyperloglog to v0.2.0 (#14722) (0167b22)
deps: update module github.com/baidubce/bce-sdk-go to v0.9.189 (#14044) (7fb34b4)
deps: update module github.com/baidubce/bce-sdk-go to v0.9.192 (#14337) (6f7cae2)
deps: update module github.com/baidubce/bce-sdk-go to v0.9.196 (#14651) (478085a)
deps: update module github.com/baidubce/bce-sdk-go to v0.9.197 (#14682) (b898294)
deps: update module github.com/coder/quartz to v0.1.2 (#14652) (7459e07)
deps: update module github.com/felixge/fgprof to v0.9.5 (#14338) (a2ad3aa)
deps: update module github.com/fsouza/fake-gcs-server to v1.50.2 (#14313) (275c97c)
deps: update module github.com/hashicorp/raft to v1.7.1 (#14005) (e9cec1d)
deps: update module github.com/ibm/go-sdk-core/v5 to v5.17.5 (#14045) (677d217)
deps: update module github.com/ibm/go-sdk-core/v5 to v5.18.1 (#14716) (8395acd)
deps: update module github.com/ibm/ibm-cos-sdk-go to v1.11.1 (#14342) (aa82a7c)
deps: update module github.com/klauspost/compress to v1.17.10 (#14352) (e23c5ed)
deps: update module github.com/minio/minio-go/v7 to v7.0.76 (#14006) (51f9376)
deps: update module github.com/minio/minio-go/v7 to v7.0.77 (#14353) (d0e3ef7)
deps: update module github.com/minio/minio-go/v7 to v7.0.80 (#14654) (eec2513)
deps: update module github.com/ncw/swift/v2 to v2.0.3 (#14356) (c843288)
deps: update module github.com/prometheus/client_golang to v1.20.5 (#14655) (e12f843)
deps: update module github.com/schollz/progressbar/v3 to v3.17.0 (#14720) (4419d0f)
deps: update module github.com/shirou/gopsutil/v4 to v4.24.10 (#14719) (3280376)
deps: update module github.com/shirou/gopsutil/v4 to v4.24.9 (#14357) (c8e6a9d)
deps: update module github.com/shopify/sarama to v1.43.3 (#14059) (1cf4813)
deps: update module github.com/spf13/afero to v1.11.0 (#14060) (bbbd82b)
deps: update module go.etcd.io/bbolt to v1.3.11 (#14358) (b7bccfc)
deps: update module golang.org/x/net to v0.29.0 (#14341) (1b6b9da)
detected_fields: always return empty array as null (#14112) (93009d4)
distributor: validate partition ring is kafka is enabled (#14303) (8438d41)
do not retain span logger created with index set initialized at query time (#14027) (4e41744)
downgrade grpc to fix regression (#14065) (8c38d46)
enable service detection for otlp endoint (#14036) (4f962ef)
Expand matching for additional variations (#14221) (71d7291)
fix bug in query result marshaling for invalid utf8 characters (#14585) (f411a07)
helm: add missing loki.storage.azure.chunkDelimiter parameter to Helm chart (#14011) (08c70cc)
helm: Check for rbac.namespaced condition before creating roles (#14201) (3f47f09)
helm: Fix persistence configuration for Memcached (#14049) (ee6e1cf)
helm: Fix wrong port name referenced for ingress NetworkPolicy (#12907) (963a25b)
helm: Various fixes and enhancements for bloom components (#14128) (dc0cbd4)
Improve docs for min and max table offsets (backport k227) (#14929) (3161fdc)
kafka: Fixes partition selection in distributors (#14242) (3f47233)
kafka: Fixes writer initialization for arm32 (#14115) (4da035b)
kafka: Set namespace for Loki kafka metrics (#14426) (8aa8a2b)
label_format: renamed label should use ParsedLabel category (#14515) (82fb2f0)
level detection for warning level (#14444) (242a852)
lint errors (#14574) (99ef900)
log-to-span: timestamp.Time should be called with milliseconds (#14196) (f8d9143)
logcli: Check for errors before checking for exists when fetching data (backport k227) (#14906) (31b2a63)
logcli: create new tail response for every line (#14525) (bcfd0d1)
logql: Fix panic in json parsing when using empty array index (#14393) (833bf0d)
logql: updated JSONExpressionParser not to unescape extracted values if it is JSON object. (#14499) (08b1a90)
missing dep PartitionRing for Ingester (#14292) (6354ded)
mixin: Remove pod label from disk usage aggregation (#14180) (5d45c96)
mixins / loki-resources-overview panel layout (#14178) (8f54ec6)
mixins: add backend path section in loki-operational for single scalable deployment (#13023) (16881ab)
mixins: disk space utilization panels with latest KSM versions (#13486) (0ea7431)
mixins: retention dashboards fix metric name (#14617) (c762b9b)
More correctly report starting phase during kafka-reader startup (#14632) (ea798e0)
move partition_id into label to make PromQL easier (#14714) (e6cf423)
nix build, downgrade toolchain to go1.23.1 (#14442) (26dfd62)
pattern: Fixes latency metric namespace for tee to pattern (#14241) (ae955ed)
promtail config unmarshalling (#14408) (a05431f)
promtail parser for azureeventhubs message without time field (#14218) (2e62abb)
promtail: validate scrape_config job name, do not allow duplicate job names (#13719) (f2d3499)
Propagate query stats from quantile & topk queries (#13831) (78b275b)
remove usage of unsafe string in label adapter unmarshal (#14216) (758364c)
Rename mispelled filename (#14237) (cf1d4a3)
report correct status code for metric and log queries in metrics.go (#12102) (900751c)
Report PSRL error message correctly (#14187) (a475153)
Revert "fix(deps): update module github.com/shirou/gopsutil/v4 to v4.24.9 (#14357)" (#14437) (d53955b)
s3: disable client retries when congestion control is enabled (#14588) (cff9f43)
sharding: apply offset to both from and through in shard request (#14256) (17c472d)
skipping label if it contains special symbol (#14068) (55e374e)
storage/chunk/client/aws: have GetObject check for canceled context (#14420) (5f325aa)
Transform ObjectExistsWithSize into GetAttributes (#14329) (2f56f50)
Update AWS storage timeout error for Go 1.23 behavior (#14226) (a4642b5)
Update renovate ignore for operator API with new module path (#14581) (c9b2907)
Wait for OwnedStreams service in Ingester startup (#14208) (a4aee4f)

Performance Improvements

blooms: Remove compression of .tar archived bloom blocks (#14159) (cdf084f)
logql: Micro-optimizations for IP filter (#14072) (c5083c7)

Miscellaneous Chores

blooms: Introduce a new block schema (V3) (#14038) (5395daf)