signalfx/splunk-otel-collector v0.164.0 on GitHub

This Splunk OpenTelemetry Collector release includes changes from the opentelemetry-collector v0.164.0
and the opentelemetry-collector-contrib v0.164.0 releases where appropriate.

🛑 Breaking changes 🛑

(Splunk) fluentforwardreceiver: Remove the fluent_forward receiver from the default configurations (#8352)
The receiver is still part of the distribution and can be added back to a custom configuration.
(Splunk) splunk_inputs: Remove the splunk_inputs receiver. Reading a Splunk .conf tree is now the splunkconf config provider's job. (#8330)
(Splunk) splunk_outputs: Remove the splunk_outputs exporter. The forwarding destination now comes from the splunkconf config provider reading outputs.conf. (#8331)
(Contrib) exporter/prometheus_remote_write: Always preserve leading and consecutive underscores in label names. (#50429)
Remove -pkg.translator.prometheus.PermissiveLabelSanitization from --feature-gates; disabling this gate now causes a startup error.
(Contrib) pkg/azurelogs: Promote the pkg.translator.azurelogs.EmitV1LogConventions and pkg.translator.azurelogs.DontEmitV0LogConventions feature gates to beta. (#52096)
code.function.name and code.file.path are now emitted instead of code.function and code.filepath by default.
The gates no longer control the event.name resource attribute, which is always emitted.
(Contrib) pkg/ottl: Remove the stable ottl.set.allowNil feature gate. (#48714)
(Contrib) pkg/translator/pprof: Return an error instead of panicking when profiling data contains invalid profile references. (#51908)
(Contrib) pkg/zipkin: Promote "pkg.translator.zipkin.EmitV1HttpConventions", "pkg.translator.zipkin.DontEmitV0HttpConventions", "pkg.translator.zipkin.EmitV1ScopeConventions", and "pkg.translator.zipkin.DontEmitV0ScopeConventions" feature gates to Beta. (#50909)
This changes the default behavior to emit the new semantic convention attributes instead of the old deprecated ones.
The Zipkin translator will now use http.response.status_code (replacing http.status_code), otel.scope.name
(replacing otel.library.name), and otel.scope.version (replacing otel.library.version) by default when emitting spans.
(Contrib) processor/filter: remove stable processor.filter.defaultErrorModeIgnore gate (#52238)
(Contrib) receiver/file_log: Promote the filelog.protobufCheckpointEncoding feature gate to stable. Checkpoints are always written as protobuf. (#43266, #52037)
The feature gate can no longer be disabled and will be removed in v0.165.0.
Existing JSON checkpoints are still read and are rewritten as protobuf on the next save.
(Contrib) receiver/kubelet_stats: Remove the deprecated aws.volume.id, fs.type, gce.pd.name, glusterfs.endpoints.name, glusterfs.path and partition resource attributes. (#40477)
(Contrib) receiver/mongodb: Change mongodb.operation.latency.time from a gauge to a monotonic cumulative sum (#51909)
opLatencies.*.latency is a cumulative total, so the metric is now a monotonic sum like every
other cumulative duration in the receiver. It is enabled: false and stability: development.
(Contrib) receiver/prometheus: Remove the stable receiver.prometheusreceiver.IgnoreScopeInfoMetric feature gate. (#47312)
(Contrib) processor/probabilistic_sampler: Add the otelcol_processor_probabilistic_sampler_count_spans_processed metric and deprecate otelcol_processor_probabilistic_sampler_count_traces_sampled in favor of it. (#50954)
The deprecated metric remains available for compatibility.
NOTE: Marked as breaking only because new metrics are enabled by default, potentially impacting resource usage and billing.
(Contrib) receiver/prometheus_remote_write: Add the otelcol_receiver_prometheusremotewrite_resource_cache_evictions internal metric counting evictions from the resource metrics cache. (#52195)
NOTE: Marked as breaking only because new metrics are enabled by default, potentially impacting resource usage and billing.
(Contrib) receiver/sql_server: Add sqlserver.tempdb.file.size and sqlserver.tempdb.allocation.waiters.count metrics, and extend sqlserver.database.tempdb.space. (#52057)
The two new metrics are disabled by default.
For configurations enabling sqlserver.database.tempdb.space, tempdb.state=used is replaced by
user_objects, internal_objects, version_store, and mixed_extents. Update queries and dashboards
that use the used series; sum the replacement states to obtain total used space.

🚩 Deprecations 🚩

(Splunk) receiver/smartagent: Deprecate ten Smart Agent monitors and provide replacement receivers (#22803)
The cpu, memory, vmem, load, filesystems, and process monitors are deprecated in favor of the host_metrics receiver. Use docker_stats instead of docker-container-stats, iis instead of windows-iis, k8s_events instead of kubernetes-events, and elasticsearch instead of the Smart Agent elasticsearch monitor. These monitors will be removed on or after December 2026.
(Splunk) receiver/smartagent: Require the receiver.smartagentreceiver.TelegrafExec feature gate to use the Smart Agent telegraf/exec monitor. (#8274)
Configurations using this monitor fail validation unless the gate is enabled with --feature-gates=+receiver.smartagentreceiver.TelegrafExec. The monitor is planned for removal in December 2026.
(Contrib) processor/transform: Deprecate the flatten_data option in favor of the new flatten.log option. (#51766)
Replace flatten_data: true with flatten: {log: true}. flatten_data will be removed in a future release.
(Contrib) receiver/kafka: Deprecate the built-in jaeger_proto and jaeger_json encodings in favour of jaegerencodingextension. (#39113)
(Contrib) receiver/kafka: Deprecate partition_processing.max_buffered_batches in favour of partition_processing.max_prefetch.batches. It will be removed in v0.166.0. (#52253)
(Contrib) receiver/kafka: Deprecate the built-in json logs encoding in favour of jsonlogencodingextension. (#39113)

💡 Enhancements 💡

(Splunk) otelcollauncher: Introduce process health check CLI command to otelcollauncher (#8358)
Adds a health subcommand to otelcollauncher, which queries the collector's healthcheckv2 endpoint
and prints the results. The command accepts --host and --port flags to specify the target address,
and a --component flag to filter the results by component name.
Note: This requires the healthcheckv2 extension to be enabled in the collector configuration.
(Splunk) msi: Mark a few MSI properties as hidden so their values are not written to the installer log. (#8350)
(Splunk) msi: Validate Splunk Platform properties the same way install.sh does. (#8351)
The installation now fails if SPLUNK_PLATFORM_LOGS_INDEX or SPLUNK_PLATFORM_METRICS_INDEX is set without
SPLUNK_PLATFORM_URL, or if SPLUNK_PLATFORM_URL is set with SPLUNK_SETUP_COLLECTOR_MODE=gateway.
(Splunk) otelcollauncher: Add an internal command-family dispatch framework to otelcollauncher (#8125)
Introduces cmd/otelcollauncher/cli, an extensibility point for future otelcollauncher
subcommands, along with a help command listing registered subcommands and per-command
help via help <command> or <command> --help/-h. --help/-h with no command still
pass through to the collector or supervisor, matching existing launcher behavior when no
subcommand is passed.
(Splunk) otelcollauncher: Introduce process lifecycle CLI commands to otelcollauncher (#8121)
Adds start, stop, status, and restart subcommands to otelcollauncher, proxying to
systemd on Linux (systemctl) or the Service Control Manager on Windows. These commands
require the collector to be installed as a system service; existing behavior when no
subcommand is passed is unchanged.
(Splunk) packaging: Update Splunk OpenTelemetry Java agent to v2.33.1 (#8362)
(Contrib) cmd/opampsupervisor: Add an agent::stop_grace_period config option to control how long the Supervisor waits for the Collector to exit after a graceful shutdown signal before forcibly killing it. (#50000)
Defaults to 10s when unset.
(Contrib) exporter/awss3: Add access_key and secret_key options to configure static AWS credentials (#52140)
(Contrib) extension/opamp: Report component status attributes in OpAMP health messages. (#50330)
(Contrib) extension/opamp: Add reports_heartbeat capability, enabled by default, to allow the OpAMP server to configure the agent's heartbeat interval (#51427)
(Contrib) pkg/kafka/configkafka: Add dial_timeout configuration option to control the timeout for establishing connections to Kafka brokers. (#51910)
(Contrib) pkg/ottl: Promote traces, metrics, and logs signals from beta to stable. (#51068, #47305)
(Contrib) processor/k8s_attributes: Add experimental kubelet /pods polling as an opt-in pod metadata source. (#14475)
(Contrib) processor/resource_detection: Add attributes.included and attributes.excluded options to the env detector to filter which attribute keys from OTEL_RESOURCE_ATTRIBUTES are emitted. Both lists support * and ? as wildcards, and excluded is applied after included. (#49970)
(Contrib) receiver/k8s_objects: Add an opt-in informer-based observer behind the receiver.k8sobjects.useInformerObserver feature gate. (#43602)
(Contrib) receiver/kafka: Make max_decompress_batch_bytes configurable in the Kafka receiver consumer configuration. (#51845)
(Contrib) receiver/kafka: Add partition_processing.max_in_flight.window. Set it to cross_fetch so exporters that wait to fill a batch, for example with sending_queue.wait_for_result and batch.min_size, get records from more than one fetch instead of sending small batches on the flush timeout. (#51804)
(Contrib) receiver/kafka: Add partition_processing.max_in_flight.bytes to limit the record bytes each partition holds while it processes records concurrently. Use it to bound memory when record sizes vary. (#51804)
(Contrib) receiver/kafka: Add partition_processing.max_prefetch.bytes to cap the bytes of fetched batches waiting for each partition worker. Use it to bound memory when batch sizes vary. (#52253)
(Contrib) receiver/kubelet_stats: Promote container.cpu.time to beta stability and align it with Semantic Conventions v1.43.0. (#52197)
(Contrib) receiver/mongodb: Add operation latency count and non-tailable collection scan metrics (#51909)
Adds mongodb.operation.latency.count and
mongodb.query_executor.collection_scan.non_tailable.count, both disabled by default.
(Contrib) receiver/mysql: Add mysql.client.name to db.server.query_sample, the client driver's self-reported identity. (#51317)
(Contrib) receiver/prometheus_remote_write: Add resource_cache_size to configure the number of job/instance pairs kept in the resource metrics cache. (#52195)
The default remains 1000.
(Contrib) receiver/prometheus_simple: headers and proxy_url are now applied to the scrape. Reserved header names and proxy_url credentials are rejected at validation. (#52081)
(Contrib) receiver/prometheus_simple: timeout now sets the scrape timeout, capped at collection_interval. Previously it had no effect. (#52081)
(Contrib) receiver/sql_server: Add db.server.procedure_definition event to capture the DDL definition of stored procedures observed in db.server.top_procedure. (#51946)

🧰 Bug fixes 🧰

(Splunk) receiver/gnmi: Fix Shutdown to wait for in-flight client goroutines when no gNMIReverse server is configured, preventing a goroutine leak on receiver shutdown.
(#8357)
(Splunk) msi: Allow Splunk Platform logs and metrics configurations to be merged when SPLUNK_CONFIG is set. (#8351)
(Core) exporter/otlp_http: Report the URL the request was actually sent to in HTTP status error messages when a middleware rewrites the request URL. (#14673)
Middleware RoundTrippers that clone the request before rewriting its URL left
the configured placeholder URL in 4xx/5xx error messages. The exporter now
reads the destination from the response instead.
(Core) pkg/otelcol: Redact non-empty literal defaults in unexpanded env provider references such as ${env:NAME:-literal} in the raw config reported by the OpAMP extension when reports_raw_config is enabled (#16141)
Empty defaults such as ${env:NAME:-} stay visible. Other provider schemes are unchanged.
(Contrib) exporter/prometheus_remote_write: Drop exponential histogram data points without a sum instead of exporting a zero sum. (#52232)
(Contrib) exporter/prometheus_remote_write: Apply the timeout to each attempt to send a request instead of to the whole retry loop, so that failed requests are retried until retry_on_failure.max_elapsed_time instead of being dropped after timeout. (#51983)
Previously a backend outage longer than timeout (5s by default) dropped all the data, as the exporter retried a request only for the duration of the timeout.
The retry delay now also grows up to retry_on_failure.max_interval. It was not passed to the backoff and stayed at retry_on_failure.initial_interval, which would send a request every 50ms by default for the whole max_elapsed_time. The retries are also interrupted when the exporter shuts down.
retry_on_failure.max_elapsed_time bounds the export of a batch of metrics as a whole, also when the batch is split into several requests (max_batch_size_bytes), and the deadline of the shutdown bounds the flush of the queue.
When only one of timeout and http.timeout is set, the other one now takes the same value instead of staying at its 5s default, which would otherwise cap every attempt at 5s.
(Contrib) exporter/signalfx: Treat non-retryable 4xx responses as permanent errors instead of retrying them indefinitely (#51389)
The SignalFx exporter shares the HandleHTTPCode helper with the Splunk HEC
exporter. Previously only 400, 401, and 403 were classified as permanent; every
other 4xx (404, 405, 410, 413, 414, 415, 422, ...) fell through and was
retried forever even though retrying an identical request cannot succeed. Those
4xx responses are now permanent, except 408 Request Timeout and 409 Conflict
(transient) and 429 Too Many Requests (throttle). 5xx responses remain
retryable, and 503 continues to honor Retry-After.
(Contrib) exporter/splunk_hec: Fix inverted condition in multi-metric batching that prematurely terminated batching and dropped valid metrics when encountering an oversized event. (#52142)
(Contrib) exporter/splunk_hec: Treat non-retryable 4xx responses as permanent errors instead of retrying them indefinitely (#51389)
Previously only 400, 401, and 403 were classified as permanent; every other
4xx (404, 405, 410, 413, 414, 415, 422, ...) fell through and was retried
forever even though retrying an identical request cannot succeed. The shared
HandleHTTPCode helper now treats all 4xx as permanent except 408 Request
Timeout and 409 Conflict (transient) and 429 Too Many Requests (throttle).
5xx responses remain retryable, and 503 continues to honor Retry-After. This
helper is shared with the SignalFx exporter, whose retry classification
changes in the same way.
(Contrib) extension/docker_observer: Fix endpoint flapping by selecting a stable IP address for containers attached to multiple networks. (#51726)
(Contrib) extension/google_cloud_logentry_encoding: Do not drop log entries with an unparsable httpRequest.requestUrl. (#52044)
The raw value is kept in url.full; url.path, url.query and url.domain are omitted.
(Contrib) internal/kafka: Reject extensions that do not implement Token(context.Context) during OAUTHBEARER SASL configuration instead of panicking at runtime. (#51958)
(Contrib) pkg/ottl: Fix context inference for statements that pass an all-uppercase converter name, such as SHA256, as a function argument. (#52192)
(Contrib) pkg/winperfcounters: Close the PDH query when adding the counter fails, preventing a handle leak on counter initialization errors. (#51048)
(Contrib) processor/cumulative_to_delta: Handle NaN sums on exponential histograms like on histograms, so the next sum is no longer emitted as a NaN delta (#51965)
(Contrib) processor/cumulative_to_delta: Include the scope attributes in the identity of a stream, so scopes that only differ in their attributes are no longer converted as one stream (#52162)
(Contrib) processor/resource_detection: Fix a panic in the ECS detector when the task ARN has only 5 colon-separated parts (#52074)
parseTaskARN checked for at least 5 parts but read the 6th one, so a task ARN without a resource part
caused an index out of range panic during detection.
(Contrib) processor/resource_detection: Bypass environment proxies for EKS detector requests to the in-cluster Kubernetes API. (#52301)
Besides this, the EKS detector configuration now accepts API server configuration to control
how it connections to the Kubernetes API server.
(Contrib) receiver/aws_cloudwatch: Persist acknowledged log checkpoints so failed polls can retry unread logs after a restart when storage is enabled. (#51944)
(Contrib) receiver/mongodb: Skip top command when mongodb.operation.time is disabled and log warning on BSONObjectTooLarge error (#50712)
(Contrib) receiver/postgresql: Fix a scrape error that discarded every metric for a table or index dropped while the receiver was collecting (#52041)
(Contrib) receiver/prometheus: Fix config reload failing for multi-line secrets such as an inline tls_config.key (#52080)
(Contrib) receiver/prometheus_remote_write: Refuse an out of range symbol reference on a target_info series, as on every other metric name. (#52070)
A request carrying one now gets a 400 where it used to get a 204 with the
sample counted as written.
(Contrib) receiver/prometheus_simple: Forward inline TLS PEM (ca_pem, cert_pem, key_pem) and server_name_override to the underlying Prometheus receiver instead of silently dropping them. (#52029)
(Contrib) receiver/prometheus_simple: Forward the tls min_version and max_version settings to the scrape config; previously they had no effect. (#52081)
(Contrib) receiver/snmp: Create the datapoint with an empty attribute when an indexed attribute column returns an empty value, instead of failing the scrape. (#51875)
An empty OCTET STRING, such as ifPhysAddress on a loopback or null interface, is a valid value.
Previously every scrape of such a device reported "metric OID attribute value is blank" and dropped those datapoints.
A row the attribute column does not return at all still fails as before.
(Contrib) receiver/snmp: Add max_repetitions configuration option (defaulting to 25) for GETBULK requests to prevent request timeouts on MTU boundaries. (#51925)
(Contrib) receiver/splunk_enterprise: Send the search job TTL to Splunk in seconds instead of nanoseconds (#50940)
The receiver sets a TTL on every search job it dispatches, derived from the configured
timeout. The value was formatted from a Go duration with %d, which emits its raw
nanosecond count, and Splunk's setttl action reads the argument as seconds. A default
timeout: 60s therefore asked Splunk to retain each dispatched job for roughly 1900 years,
so search artifacts accumulated until the search dispatch directory filled and Splunk
reported extreme search lag. The duration is now converted to whole seconds.
(Contrib) receiver/yang_grpc: Log a warning when duplicate YANG module definitions are loaded, identifying conflicting source paths. (#50917)

Docker Image Manifests:

Linux (amd64, arm64, ppc64le) and Windows (2019 amd64, 2022 amd64):

quay.io/signalfx/splunk-otel-collector:0.164.0
digest: sha256:b47bb5e100eee4b68c450b29e711a150f6ade7d441a6e4593ca9c74e72252c7c