github apache/druid druid-39.0.0
Druid 39.0.0

2 hours ago

Apache Druid 39.0.0 contains over 680 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 42 contributors.

See the complete set of changes for additional details, including bug fixes.

Review the upgrade notes and incompatible changes before you upgrade to Druid 39.0.0.
If you are upgrading across multiple versions, see the Upgrade notes page, which lists upgrade notes for the most recent Druid versions.

Important features, changes, and deprecations

This section contains important information about new and existing features.

Java

Support for Java 21 has been dropped. You must upgrade to Java 25.

DDL statements for the catalog (experimental)

Druid now supports CREATE and ALTER statements to manage metadata in the catalog. Use them to declare a table's columns, time partitioning, clustering, aggregate projections, and physical layout, instead of setting these in each ingestion spec.

This feature is off by default. To enable it, set druid.sql.planner.enableCatalogDdl to true.

For more information, see Catalog.

#19830 #20377 #20452 #19770 #19711

Kafka header-based filtering

Kafka supervisors can now filter records by their headers. This reduces ingestion CPU and memory use when you only need some of a topic's records.

To use the filter, add headerBasedFilterConfig to the supervisor's ioConfig. Use an in filter where dimension is the header name and values lists the header values to ingest. Records whose header value isn't in the list are dropped, and offsets still advance.

Filtering is permissive: records are ingested if they don't have the header, if the header value is null, empty, or can't be decoded, or if evaluation fails.

The new ingest/events/filtered metric reports how many records header filtering dropped.

#18525

Typed query context parameters

Common query context parameters listed in Query context reference and Scan queries are now typed QueryContextParameter descriptors. The following changes apply to the query context handling:

  • Integer and long parameters reject values that aren't exact integers instead of truncating or wrapping them. For example 1.5, and values out of range. Integral values like 5.0 or "5.0" are still accepted. Invalid values now fail with an error message: "Query context parameter [x] should be ...."
  • A null in the query context, or SET x = NULL, no longer clears a default configured with druid.query.default.context or the Broker dynamic configuration. The default applies as if the key were not set. Previously the null reached readers, which used the built-in default. To override a configured default, set an explicit value.
  • maxSubqueryBytes is a typed string parameter that accepts auto, disabled or a non-negative number of bytes, as the subquery guardrail does. Invalid values such as SET maxSubqueryBytes = TRUE now fail during validation, and numbers such as 1e9 are now accepted.

#20555

Druid JDBC driver

Druid now provides its own JDBC driver that uses the /druid/v2/sql/ endpoint. We recommend that you switch from the Avatica driver to this driver. The new driver provides the following benefits: driver-side connection management that doesn't require sticky load balancing, streaming results, support for SET statements, and support for non-native SQL engines.

#19951

Netty 4.2

Druid now uses Netty 4.2 for its HTTP client. Previously, Druid used Netty 3. This is a substantive change in memory characteristics:

  • Direct memory: The HTTP client now consumes some direct memory. Operators running close to their -XX:MaxDirectMemorySize budget may need to increase it.
  • Pooled buffers: Buffers are now pooled, so resident memory may appear higher than under Netty 3, even when idle. This is normal for a pooled allocator.

There are new configs that control the behavior of the HTTP client: druid.global.http.connectTimeout, druid.global.http.allocator, druid.broker.http.connectTimeout, and druid.broker.http.allocator. You can also set some Netty configs directly.

#19567

Quality of service filtering for Coordinator

You can now enable QoS filtering for the Coordinator to prevent a burst of slow API requests from exhausting the thread pool and blocking health checks.

Use druid.coordinator.server.maxConcurrentRequests to configure this feature (default: max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8))). Set this property to 0 to turn it off.

Leader-status endpoints are excluded from the cap so load balancers/monitors stay unaffected.

#19271

Functional area and related changes

This section contains detailed release notes separated by areas.

Ingestion

  • Added support for RE2/J regex for the regex input format #19514
  • Changed the S3 client to use the AWS-recommended standard retry policy. You can customize the retry strategy using druid.s3.retryMode and druid.s3.maxAttempts #19892
  • Protobuf ingestion now keeps the default value of proto3 fields that are not marked optional. Previously an int32 field holding 0 (or a string holding "", a bool holding false, an enum holding its first value) was ingested as null. Such fields now ingest as their actual value. #20485

SQL-based ingestion

  • Workers now only pass the necessary worker context instead of the entire worker context to the task context. #20650

Streaming ingestion

Ingesting from selected partitions

Kafka supervisors now accept ioConfig.partitionIds, which lets you ingest only selected partitions of a topic. For example, you can use this for independent diagnostic ingestion.

#20474

Cost-based autoscaler improvements
  • You can now simulate the cost-based autoscaler for Kafka to help determine the optimal task count for different amounts of lag. #19687
  • You can now set a critical lag threshold for the cost-based autoscaler. When lag reaches 75% of the threshold, the autoscaler bypasses the scale-up boundary. When lag reaches 100%, the autoscaler scales up to the maximum task count. #19655
  • You can now choose to configure cooldown for the cost-based autoscaler for a supervisor in one of 2 ways:
    • High sensitivity: Set minScaleUpDelay / minScaleDownDelay < taskDuration. For example if minScaleUpDelay = taskDuration / 2, there would be at most 1 scaling event between 2 task rollovers
    • Low sensitivity : Set minScaleUpDelay / minScaleDownDelay > taskDuration. In this case, cooldown is effectively ignored and the scaling happens on task rollover only.
      #20706
  • The cost-based autoscaler now uses running averages instead of instantaneous values of task metrics to make decisions for the cost based auto-scaler. This leads to smoother scaling but may sometimes come at the cost of slightly delayed responses to change in metric trends that occur towards the end of the task lifecycle. For higher sensitivity to changing trends, set minScaleUpDelay and minScaleDownDelay to be smaller than taskDuration.
Other streaming ingestion improvements
  • Fixed a bug where a Kafka indexing task would report a poll-idle-ratio of 1.0 (i.e. fully idle) before any sample was actually available, which could skew the supervisor autoscaler's averaged idle ratio across tasks. #20783

Querying

Query planning time

You can now limit how long the Broker spends planning a SQL query with the new druid.sql.planner.maxPlanningTimeMs runtime property. You can also set it per query using the maxPlanningTimeMs query context parameter. When planning exceeds the configured budget, the query fails with an HTTP 504 Query timeout error instead of tying up a Broker thread. The default value is 0, which disables the timeout.

#20314

Fixed JDK HTTP client failures caused by query response trailers

Brokers and Historicals now send the X-Druid-Response-Complete and X-Error-Message HTTP trailers only when the request includes the TE: trailers header. Clients that read these trailers must send that header. This fixes query failures (java.net.ProtocolException: CR expected) in java.net.http.HttpClient-based clients, including the Druid JDBC driver, on JDK 17.0.21, 21.0.13 and 25.0.5.

#20717

Other querying improvements

  • You can now use Dart to query metadata tables such as those in the sys schema #19952
  • CAST(x AS LONG) is now accepted as an alias for CAST(x AS BIGINT) in Druid SQL #20402
  • Added datasource filter pushdown to sys.segments table #19718
  • Added framing support to the FIRST_VALUE and LAST_VALUE window functions #19301
  • Improved the performance of timeseries, timeBoundary, and scan queries that run in descending order #19947
  • Improved the performance of COALESCE #18400
  • Improved S3 API usage efficiency for MSQ tasks using durable storage and MSQ export #19893
  • Improved validation for catalog types and parsing for native type strings #19769
  • Upgraded to Calcite 1.42.0 #19976 #19370

Cluster management

Kubernetes node labels

The druid-kubernetes-extensions extension now supports a k8sNodeLabel dynamic config provider, which resolves configuration values from the labels of the Kubernetes node a Druid process runs on.

#20404

Peon pod template

You can now retrieve the Peon pod template names currently configured on the Overlord using the new GET /druid/indexer/v1/k8s/taskrunner/podTemplates endpoint. It returns the name for each configured template. The endpoint requires CONFIG read permission and returns 404 when the Overlord is not using the custom template pod adapter.

#19712

Environment variables

The environment dynamic config provider now omits variables that are not set in the environment instead of producing a null value that fails consumers such as the Kafka supervisor with a NullPointerException. A skipped variable is logged at info level.

#20381

Connection pools

You can now configure how the Druid connection pool tracks connection demand using druid.global.http.poolImplementation.

The default is retaining. With retaining, the pool keeps every connection it opens and replaces each stale or broken connection with a new one, so the pool stays at its high-water mark.

#20542 #20273

HTTP response headers

Druid can now optionally return X-Druid-Server, X-Druid-Service, and X-Druid-Version on HTTP responses, allowing operators to identify the process and version that served a direct or Router-proxied request. These headers expose cluster topology and version information, so only enable them if all clients are permitted to receive it. To enable them, set druid.server.http.enableResponseIdentityHeaders to true.

#20268

Improved LDAP support

Added support for LDAP group-based authorization on servers that do not return the memberOf attribute (such as OpenLDAP). Configure groupBaseDn and groupSearch on the LDAP credentials validator to enable a reverse group lookup that resolves group memberships automatically.

#19432

Table visibility

Users who query tables they don't have READ permission for now see a "not found" error instead of a "forbidden" error.

To restore the previous behavior, set druid.sql.planner.authorizeTableVisibility to false.

Additionally, when set to true, INSERT and REPLACE require both READ and WRITE access to the target table. If this property is set to false, they only require WRITE permission. Regardless of the value of this property, tables only appear in INFORMATION_SCHEMA if the user has READ access to them.

This is a breaking change that affects the permissions you need. For more information, see permissions changes for SQL-based ingestion.

#20075

Java home variable

Middle Managers now honor JAVA_HOME when using the bin/java script, which should be present in the working directory.

Druid uses the following logic

  • If you specified a command (such as if config.getJavaCommand is not null), use that.
  • If bin/run-java exists, use that
  • Use java

#19709

Add QoS filtering for Coordinator endpoints

Added configurable QoS filtering to the Coordinator to prevent a burst of slow API requests from exhausting the thread pool and blocking health checks .
Added a new configuration: druid.coordinator.server.maxConcurrentRequests (default: max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8)), set <= 0 to disable).
Leader-status endpoints are excluded from the cap so load balancers and monitors remain unaffected.

#19271

Other cluster management improvements

  • Added a KILL task lock type that can be used by embedded kill tasks so that the UnusedSegmentsKiller does not skip intervals with active ingestions #19921
  • Added support for PostgreSQL to the export-metadata tool #19698
  • Peons running tasks that do not serve queries (for example, batch, compaction, and MSQ tasks) now ignore druid.server.http.unannouncePropagationDelay and shut down without waiting. Real-time streaming tasks still honor the delay #20444
  • Improved port binding. Druid nodes can now advertise a different plaintext port for peer discovery while binding Jetty to the actual service port #19598
  • Reduced the heap footprint of loaded segments #20467
  • Fixed the shutdownAllTasks Overlord API (POST /druid/indexer/v1/datasources/{datasource}/shutdownAllTasks), which returned a 404 error since Druid 32.0.0. #20824

Data management

Skip intervals during compaction

Auto-compaction now supports a skipIntervals setting in the datasource compaction config, allowing specific intervals to be excluded from compaction in addition to the existing skipOffsetFromLatest behavior. This is useful, for example, to exclude an interval that a backfill job is actively writing to, or one that consistently fails compaction.

#19792

Better visibility for large segment files

Druid now reports when your range or hash partitioning settings produce overly large segment files (hot partitions) in a time chunk. Native batch ingestion with hash or range partitioning now logs a warning and emits the ingest/segments/oversized metric. The metric counts published segments with more than double maxRowsPerSegment rows, or more than triple targetRowsPerSegment for range partitioning. The task completion report also includes this count as oversizedSegments.

#20173

Faster segment searches

You can now speed up segment searches in the in-memory timeline by enabling fast interval search. This feature uses an interval tree index to organize the timeline in memory for faster loading and searching of segments.

To use this feature, set druid.segment.timeline.fastIntervalSearch to true.

#19138 #19861

Partial segment loading

Druid now supports partial segment loading for virtual storage (on by default). To turn it off, set either of the following to false:

  • the runtime property druid.indexer.task.virtualStoragePartialDownloadsEnabled
  • the query context parameter virtualStoragePartialDownloadsEnabled

#19739

Other data management improvements

  • Added support for MD5 request checksums for S3-compatible storage. To enable this feature, set druid.s3.enableLegacyMd5 to true #20055
  • Added support for pushing, pulling, and killing segments in Azure and Google Cloud deep storage without zip compression #20454 #20327 #20382 #20380
  • Improved the loading algorithm to better disperse requests across Historical tiers when using virtual storage #20155
  • Improved the efficiency of embedded kill tasks #19772

Metrics and monitoring

Tier storage metrics

The tier storage metrics now provide better observability for Historicals, with or without virtual storage. Two metrics, tier/required/capacity and tier/total/capacity, are deprecated and will be removed in a future release. Use tier/storage/required and tier/storage/capacity instead.

#20199

task/run/time metric

The statsd and prometheus emitters now emit the taskStatus dimension on task/run/time. This is a breaking change. For more information, see alerts on task/run/time.

#20122

Other metrics and monitoring improvements

  • Peon task metrics for clusters that use the Kubernetes Overlord extension now include a podTemplate dimension. This makes it easier to monitor clusters that map different tasks to different pod templates #20001
  • Added mergeBuffer/maxSpillProximity which measures how close any single groupBy query came to spilling during the emission period, as a fraction in [0.0, 1.0] #19627
  • Added taskType to the various task count metrics, so you can filter them by task type #20123

Contributor extensions

Iceberg

Apache Iceberg ingestion now supports an opt-in vectorized reader backed by iceberg-arrow. Enable it with "useArrowReader": true; tune batching with arrowBatchSize (default 1024). The Arrow path supports Parquet data files, dictionary-encoded values, schema evolution, time-travel snapshots, column projection, and predicate pushdown. Snapshots containing positional or equality delete files aren't supported, and Druid rejects them with an error; use the standard delete-aware reader for those snapshots. The Arrow path is not splittable, so keep it disabled where parallel batch throughput is required. The default reader remains unchanged, and both paths coexist under the same Iceberg input-source type.

#19510

Open Policy Agent

Added the druid-opa-authorizer contributor extension, which enables authorization through Open Policy Agent (OPA). You can write Rego policies to control access to Druid resources based on user identity, action, and resource type. The extension supports both basic authentication and LDAP authentication workflows.

#19670

Upgrade notes and incompatible changes

Upgrade notes

Netty 4.2

Druid now uses Netty 4.2 for its HTTP client. Previously, Druid used Netty 3. This is a substantive change in memory characteristics:

  • Direct memory: The HTTP client now consumes some direct memory. Operators running close to their -XX:MaxDirectMemorySize budget may need to increase it.
  • Pooled buffers: Buffers are now pooled, so resident memory may appear higher than under Netty 3, even when idle. This is normal for a pooled allocator.

There are new configs that control the behavior of the HTTP client: druid.global.http.connectTimeout, druid.global.http.allocator, druid.broker.http.connectTimeout, and druid.broker.http.allocator. You can also set some Netty configs directly.

#19567

Dart workers

Dart workers on Historicals are now announced and discovered by the Broker with a new DartWorkerService mechanism. If you've enabled Dart, follow the standard rolling upgrade order: during an upgrade, restart Historicals before the Broker; during a downgrade, restart the Broker before Historicals.

#20508

Protobuf ingestion

  • Protobuf ingestion now keeps the default value of proto3 fields that are not marked optional. Previously an int32 field holding 0 (or a string holding "", a bool holding false, an enum holding its first value) was ingested as null. Such fields now ingest as their actual value.

#20485

Fixed JDK HTTP client failures caused by query response trailers

Brokers and Historicals now send the X-Druid-Response-Complete and X-Error-Message HTTP trailers only when the request includes the TE: trailers header. Clients that read these trailers must send that header. This fixes query failures (java.net.ProtocolException: CR expected) in java.net.http.HttpClient-based clients, including the Druid JDBC driver, on JDK 17.0.21, 21.0.13 and 25.0.5.

#20717

Typed query context parameters

Common query context parameters listed in Query context reference and Scan queries are now typed QueryContextParameter descriptors. The following changes apply to the query context handling:

  • Integer and long parameters reject values that aren't exact integers instead of truncating or wrapping them. For example 1.5, and values out of range. Integral values like 5.0 or "5.0" are still accepted. Invalid values now fail with an error message: "Query context parameter [x] should be ...."
  • A null in the query context, or SET x = NULL, no longer clears a default configured with druid.query.default.context or the Broker dynamic configuration. The default applies as if the key were not set. Previously the null reached readers, which used the built-in default. To override a configured default, set an explicit value.
  • maxSubqueryBytes is a typed string parameter that accepts auto, disabled or a non-negative number of bytes, as the subquery guardrail does. Invalid values such as SET maxSubqueryBytes = TRUE now fail during validation, and numbers such as 1e9 are now accepted.

[#20555][https://github.com//pull/20555/]

Java home variable

Middle Managers now honor JAVA_HOME when using the bin/java script, which should be present in the working directory.

Druid uses the following logic

  • If you specified a command (such as if config.getJavaCommand is not null), use that.
  • If bin/run-java exists, use that
  • Use java

#19709

Quality of service filtering for Coordinator

You can now enable QoS filtering for the Coordinator to prevent a burst of slow API requests from exhausting the thread pool and blocking health checks.

Use druid.coordinator.server.maxConcurrentRequests to configure this feature (default: max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8))). Set this property to 0 to turn it off.

Leader-status endpoints are excluded from the cap so load balancers/monitors stay unaffected.

#19271

Incompatible changes

Permissions changes for SQL-based ingestion

The table visibility changes affect the permissions required for SQL-based ingestion. INSERT and REPLACE now require READ access (in addition to WRITE) on the target table.

To restore the previous behavior, set druid.sql.planner.authorizeTableVisibility to false.

#20075

Alerts on task/run/time

The statsd and prometheus emitters now emit the taskStatus dimension on task/run/time. This is a breaking change for dashboards and alerts on task/run/time:

  • statsd (non-DogStatsD mode): Dimension values are joined into the dotted metric name, so druid.overlord.task.run.time.DATASOURCE.TASKTYPE becomes druid.overlord.task.run.time.DATASOURCE.TASKSTATUS.TASKTYPE. Queries against the old name stop receiving data.
  • statsd (DogStatsD mode): A taskStatus tag is added. Existing queries keep working and aggregate over it.
  • prometheus: A taskStatus label is added. Update any queries that select on an exact label set and any recording rules that assume the old label set.

To keep the previous behavior, pin the old dimension list with druid.emitter.statsd.dimensionMapPath or druid.emitter.prometheus.dimensionMapPath.

#20122

SysMonitor removed

SysMonitor has been removed after being deprecated in Druid 27. Use OshiSysMonitor instead.

#20200

MariaDB Connector/J 3.x changes connection URI behavior

The MariaDB JDBC connector has been upgraded from from 2.7.3 to 3.5.10, which has the following impact:

  • For druid.metadata.storage.connector.connectURI with org.mariadb.jdbc.Driver, use jdbc:mariadb:. A legacy jdbc:mysql: URI remains usable only if it includes permitMysqlScheme. For example, jdbc:mysql://<host>/druid?permitMysqlScheme, because metadata storage passes the URI directly to the driver.
  • For MariaDB-backed SQL input sources and JDBC lookups with the default druid.access.jdbc.enforceAllowedProperties=true, use jdbc:mariadb:. Druid's JDBC property validation does not route jdbc:mysql: to the Connector/J 3.x parser, and permitMysqlScheme is not in the default druid.access.jdbc.allowedProperties, Adding the option alone is not sufficient.
  • Users who continue supplying Connector/J 2.x do not need to change existing jdbc:mysql: or jdbc:mariadb: URIs. MySQL Connector/J users are unaffected.
  • Images rebuilt from distribution/docker/Dockerfile.mariadb now install Connector/J 3.5.10 and must follow the 3.x guidance above.

#20036

Developer notes

Web console code editor

The web console's code editors moved from Ace (ace-builds and react-ace) to Code Mirror 6. All editors now share a single CodeEditor component, including the workbench, the explore view's SQL inputs, JSON inputs, spec and value dialogs, history, and explain. Two Lezer grammars, one for DruidSQL and one for Hjson, drive the highlighting and the completions.

#20507

Dependency updates

Druid 39.0.0 includes a variety of updates to dependencies such as Calcite. For a full list, see the Area - Dependencies label in the milestone.

Don't miss a new druid release

NewReleases is sending notifications on new releases.