Apache Druid 39.0.0 contains over 680 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 42 contributors.
See the complete set of changes for additional details, including bug fixes.
Review the upgrade notes and incompatible changes before you upgrade to Druid 39.0.0.
If you are upgrading across multiple versions, see the Upgrade notes page, which lists upgrade notes for the most recent Druid versions.
Important features, changes, and deprecations
This section contains important information about new and existing features.
Java
Support for Java 21 has been dropped. You must upgrade to Java 25.
DDL statements for the catalog (experimental)
Druid now supports CREATE and ALTER statements to manage metadata in the catalog. Use them to declare a table's columns, time partitioning, clustering, aggregate projections, and physical layout, instead of setting these in each ingestion spec.
This feature is off by default. To enable it, set druid.sql.planner.enableCatalogDdl to true.
For more information, see Catalog.
#19830 #20377 #20452 #19770 #19711
Kafka header-based filtering
Kafka supervisors can now filter records by their headers. This reduces ingestion CPU and memory use when you only need some of a topic's records.
To use the filter, add headerBasedFilterConfig to the supervisor's ioConfig. Use an in filter where dimension is the header name and values lists the header values to ingest. Records whose header value isn't in the list are dropped, and offsets still advance.
Filtering is permissive: records are ingested if they don't have the header, if the header value is null, empty, or can't be decoded, or if evaluation fails.
The new ingest/events/filtered metric reports how many records header filtering dropped.
Typed query context parameters
Common query context parameters listed in Query context reference and Scan queries are now typed QueryContextParameter descriptors. The following changes apply to the query context handling:
- Integer and long parameters reject values that aren't exact integers instead of truncating or wrapping them. For example
1.5, and values out of range. Integral values like 5.0 or "5.0" are still accepted. Invalid values now fail with an error message: "Query context parameter [x] should be ...." - A null in the query context, or SET x = NULL, no longer clears a default configured with
druid.query.default.contextor the Broker dynamic configuration. The default applies as if the key were not set. Previously the null reached readers, which used the built-in default. To override a configured default, set an explicit value. maxSubqueryBytesis a typed string parameter that accepts auto, disabled or a non-negative number of bytes, as the subquery guardrail does. Invalid values such asSET maxSubqueryBytes = TRUEnow fail during validation, and numbers such as1e9are now accepted.
Druid JDBC driver
Druid now provides its own JDBC driver that uses the /druid/v2/sql/ endpoint. We recommend that you switch from the Avatica driver to this driver. The new driver provides the following benefits: driver-side connection management that doesn't require sticky load balancing, streaming results, support for SET statements, and support for non-native SQL engines.
Netty 4.2
Druid now uses Netty 4.2 for its HTTP client. Previously, Druid used Netty 3. This is a substantive change in memory characteristics:
- Direct memory: The HTTP client now consumes some direct memory. Operators running close to their
-XX:MaxDirectMemorySizebudget may need to increase it. - Pooled buffers: Buffers are now pooled, so resident memory may appear higher than under Netty 3, even when idle. This is normal for a pooled allocator.
There are new configs that control the behavior of the HTTP client: druid.global.http.connectTimeout, druid.global.http.allocator, druid.broker.http.connectTimeout, and druid.broker.http.allocator. You can also set some Netty configs directly.
Quality of service filtering for Coordinator
You can now enable QoS filtering for the Coordinator to prevent a burst of slow API requests from exhausting the thread pool and blocking health checks.
Use druid.coordinator.server.maxConcurrentRequests to configure this feature (default: max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8))). Set this property to 0 to turn it off.
Leader-status endpoints are excluded from the cap so load balancers/monitors stay unaffected.
Functional area and related changes
This section contains detailed release notes separated by areas.
Ingestion
- Added support for RE2/J regex for the
regexinput format #19514 - Changed the S3 client to use the AWS-recommended standard retry policy. You can customize the retry strategy using
druid.s3.retryModeanddruid.s3.maxAttempts#19892 - Protobuf ingestion now keeps the default value of
proto3fields that are not marked optional. Previously anint32field holding0(or a string holding "", aboolholding false, anenumholding its first value) was ingested as null. Such fields now ingest as their actual value. #20485
SQL-based ingestion
- Workers now only pass the necessary worker context instead of the entire worker context to the task context. #20650
Streaming ingestion
Ingesting from selected partitions
Kafka supervisors now accept ioConfig.partitionIds, which lets you ingest only selected partitions of a topic. For example, you can use this for independent diagnostic ingestion.
Cost-based autoscaler improvements
- You can now simulate the cost-based autoscaler for Kafka to help determine the optimal task count for different amounts of lag. #19687
- You can now set a critical lag threshold for the cost-based autoscaler. When lag reaches 75% of the threshold, the autoscaler bypasses the scale-up boundary. When lag reaches 100%, the autoscaler scales up to the maximum task count. #19655
- You can now choose to configure cooldown for the cost-based autoscaler for a supervisor in one of 2 ways:
- High sensitivity: Set
minScaleUpDelay / minScaleDownDelay < taskDuration. For example ifminScaleUpDelay = taskDuration / 2, there would be at most 1 scaling event between 2 task rollovers - Low sensitivity : Set
minScaleUpDelay / minScaleDownDelay > taskDuration. In this case, cooldown is effectively ignored and the scaling happens on task rollover only.
#20706
- High sensitivity: Set
- The cost-based autoscaler now uses running averages instead of instantaneous values of task metrics to make decisions for the cost based auto-scaler. This leads to smoother scaling but may sometimes come at the cost of slightly delayed responses to change in metric trends that occur towards the end of the task lifecycle. For higher sensitivity to changing trends, set
minScaleUpDelayandminScaleDownDelayto be smaller thantaskDuration.
Other streaming ingestion improvements
- Fixed a bug where a Kafka indexing task would report a poll-idle-ratio of 1.0 (i.e. fully idle) before any sample was actually available, which could skew the supervisor autoscaler's averaged idle ratio across tasks. #20783
Querying
Query planning time
You can now limit how long the Broker spends planning a SQL query with the new druid.sql.planner.maxPlanningTimeMs runtime property. You can also set it per query using the maxPlanningTimeMs query context parameter. When planning exceeds the configured budget, the query fails with an HTTP 504 Query timeout error instead of tying up a Broker thread. The default value is 0, which disables the timeout.
Fixed JDK HTTP client failures caused by query response trailers
Brokers and Historicals now send the X-Druid-Response-Complete and X-Error-Message HTTP trailers only when the request includes the TE: trailers header. Clients that read these trailers must send that header. This fixes query failures (java.net.ProtocolException: CR expected) in java.net.http.HttpClient-based clients, including the Druid JDBC driver, on JDK 17.0.21, 21.0.13 and 25.0.5.
Other querying improvements
- You can now use Dart to query metadata tables such as those in the
sysschema #19952 CAST(x AS LONG)is now accepted as an alias forCAST(x AS BIGINT)in Druid SQL #20402- Added
datasourcefilter pushdown tosys.segmentstable #19718 - Added framing support to the FIRST_VALUE and LAST_VALUE window functions #19301
- Improved the performance of timeseries, timeBoundary, and scan queries that run in descending order #19947
- Improved the performance of COALESCE #18400
- Improved S3 API usage efficiency for MSQ tasks using durable storage and MSQ export #19893
- Improved validation for catalog types and parsing for native type strings #19769
- Upgraded to Calcite 1.42.0 #19976 #19370
Cluster management
Kubernetes node labels
The druid-kubernetes-extensions extension now supports a k8sNodeLabel dynamic config provider, which resolves configuration values from the labels of the Kubernetes node a Druid process runs on.
Peon pod template
You can now retrieve the Peon pod template names currently configured on the Overlord using the new GET /druid/indexer/v1/k8s/taskrunner/podTemplates endpoint. It returns the name for each configured template. The endpoint requires CONFIG read permission and returns 404 when the Overlord is not using the custom template pod adapter.
Environment variables
The environment dynamic config provider now omits variables that are not set in the environment instead of producing a null value that fails consumers such as the Kafka supervisor with a NullPointerException. A skipped variable is logged at info level.
Connection pools
You can now configure how the Druid connection pool tracks connection demand using druid.global.http.poolImplementation.
The default is retaining. With retaining, the pool keeps every connection it opens and replaces each stale or broken connection with a new one, so the pool stays at its high-water mark.
HTTP response headers
Druid can now optionally return X-Druid-Server, X-Druid-Service, and X-Druid-Version on HTTP responses, allowing operators to identify the process and version that served a direct or Router-proxied request. These headers expose cluster topology and version information, so only enable them if all clients are permitted to receive it. To enable them, set druid.server.http.enableResponseIdentityHeaders to true.
Improved LDAP support
Added support for LDAP group-based authorization on servers that do not return the memberOf attribute (such as OpenLDAP). Configure groupBaseDn and groupSearch on the LDAP credentials validator to enable a reverse group lookup that resolves group memberships automatically.
Table visibility
Users who query tables they don't have READ permission for now see a "not found" error instead of a "forbidden" error.
To restore the previous behavior, set druid.sql.planner.authorizeTableVisibility to false.
Additionally, when set to true, INSERT and REPLACE require both READ and WRITE access to the target table. If this property is set to false, they only require WRITE permission. Regardless of the value of this property, tables only appear in INFORMATION_SCHEMA if the user has READ access to them.
This is a breaking change that affects the permissions you need. For more information, see permissions changes for SQL-based ingestion.
Java home variable
Middle Managers now honor JAVA_HOME when using the bin/java script, which should be present in the working directory.
Druid uses the following logic
- If you specified a command (such as if
config.getJavaCommandis not null), use that. - If
bin/run-javaexists, use that - Use java
Add QoS filtering for Coordinator endpoints
Added configurable QoS filtering to the Coordinator to prevent a burst of slow API requests from exhausting the thread pool and blocking health checks .
Added a new configuration: druid.coordinator.server.maxConcurrentRequests (default: max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8)), set <= 0 to disable).
Leader-status endpoints are excluded from the cap so load balancers and monitors remain unaffected.
Other cluster management improvements
- Added a
KILLtask lock type that can be used by embedded kill tasks so that theUnusedSegmentsKillerdoes not skip intervals with active ingestions #19921 - Added support for PostgreSQL to the
export-metadatatool #19698 - Peons running tasks that do not serve queries (for example, batch, compaction, and MSQ tasks) now ignore
druid.server.http.unannouncePropagationDelayand shut down without waiting. Real-time streaming tasks still honor the delay #20444 - Improved port binding. Druid nodes can now advertise a different plaintext port for peer discovery while binding Jetty to the actual service port #19598
- Reduced the heap footprint of loaded segments #20467
- Fixed the
shutdownAllTasksOverlord API (POST /druid/indexer/v1/datasources/{datasource}/shutdownAllTasks), which returned a 404 error since Druid 32.0.0. #20824
Data management
Skip intervals during compaction
Auto-compaction now supports a skipIntervals setting in the datasource compaction config, allowing specific intervals to be excluded from compaction in addition to the existing skipOffsetFromLatest behavior. This is useful, for example, to exclude an interval that a backfill job is actively writing to, or one that consistently fails compaction.
Better visibility for large segment files
Druid now reports when your range or hash partitioning settings produce overly large segment files (hot partitions) in a time chunk. Native batch ingestion with hash or range partitioning now logs a warning and emits the ingest/segments/oversized metric. The metric counts published segments with more than double maxRowsPerSegment rows, or more than triple targetRowsPerSegment for range partitioning. The task completion report also includes this count as oversizedSegments.
Faster segment searches
You can now speed up segment searches in the in-memory timeline by enabling fast interval search. This feature uses an interval tree index to organize the timeline in memory for faster loading and searching of segments.
To use this feature, set druid.segment.timeline.fastIntervalSearch to true.
Partial segment loading
Druid now supports partial segment loading for virtual storage (on by default). To turn it off, set either of the following to false:
- the runtime property
druid.indexer.task.virtualStoragePartialDownloadsEnabled - the query context parameter
virtualStoragePartialDownloadsEnabled
Other data management improvements
- Added support for MD5 request checksums for S3-compatible storage. To enable this feature, set
druid.s3.enableLegacyMd5totrue#20055 - Added support for pushing, pulling, and killing segments in Azure and Google Cloud deep storage without zip compression #20454 #20327 #20382 #20380
- Improved the loading algorithm to better disperse requests across Historical tiers when using virtual storage #20155
- Improved the efficiency of embedded kill tasks #19772
Metrics and monitoring
Tier storage metrics
The tier storage metrics now provide better observability for Historicals, with or without virtual storage. Two metrics, tier/required/capacity and tier/total/capacity, are deprecated and will be removed in a future release. Use tier/storage/required and tier/storage/capacity instead.
task/run/time metric
The statsd and prometheus emitters now emit the taskStatus dimension on task/run/time. This is a breaking change. For more information, see alerts on task/run/time.
Other metrics and monitoring improvements
- Peon task metrics for clusters that use the Kubernetes Overlord extension now include a
podTemplatedimension. This makes it easier to monitor clusters that map different tasks to different pod templates #20001 - Added
mergeBuffer/maxSpillProximitywhich measures how close any single groupBy query came to spilling during the emission period, as a fraction in[0.0, 1.0]#19627 - Added
taskTypeto the various task count metrics, so you can filter them by task type #20123
Contributor extensions
Iceberg
Apache Iceberg ingestion now supports an opt-in vectorized reader backed by iceberg-arrow. Enable it with "useArrowReader": true; tune batching with arrowBatchSize (default 1024). The Arrow path supports Parquet data files, dictionary-encoded values, schema evolution, time-travel snapshots, column projection, and predicate pushdown. Snapshots containing positional or equality delete files aren't supported, and Druid rejects them with an error; use the standard delete-aware reader for those snapshots. The Arrow path is not splittable, so keep it disabled where parallel batch throughput is required. The default reader remains unchanged, and both paths coexist under the same Iceberg input-source type.
Open Policy Agent
Added the druid-opa-authorizer contributor extension, which enables authorization through Open Policy Agent (OPA). You can write Rego policies to control access to Druid resources based on user identity, action, and resource type. The extension supports both basic authentication and LDAP authentication workflows.
Upgrade notes and incompatible changes
Upgrade notes
Netty 4.2
Druid now uses Netty 4.2 for its HTTP client. Previously, Druid used Netty 3. This is a substantive change in memory characteristics:
- Direct memory: The HTTP client now consumes some direct memory. Operators running close to their
-XX:MaxDirectMemorySizebudget may need to increase it. - Pooled buffers: Buffers are now pooled, so resident memory may appear higher than under Netty 3, even when idle. This is normal for a pooled allocator.
There are new configs that control the behavior of the HTTP client: druid.global.http.connectTimeout, druid.global.http.allocator, druid.broker.http.connectTimeout, and druid.broker.http.allocator. You can also set some Netty configs directly.
Dart workers
Dart workers on Historicals are now announced and discovered by the Broker with a new DartWorkerService mechanism. If you've enabled Dart, follow the standard rolling upgrade order: during an upgrade, restart Historicals before the Broker; during a downgrade, restart the Broker before Historicals.
Protobuf ingestion
- Protobuf ingestion now keeps the default value of
proto3fields that are not marked optional. Previously anint32field holding0(or a string holding "", aboolholding false, anenumholding its first value) was ingested as null. Such fields now ingest as their actual value.
Fixed JDK HTTP client failures caused by query response trailers
Brokers and Historicals now send the X-Druid-Response-Complete and X-Error-Message HTTP trailers only when the request includes the TE: trailers header. Clients that read these trailers must send that header. This fixes query failures (java.net.ProtocolException: CR expected) in java.net.http.HttpClient-based clients, including the Druid JDBC driver, on JDK 17.0.21, 21.0.13 and 25.0.5.
Typed query context parameters
Common query context parameters listed in Query context reference and Scan queries are now typed QueryContextParameter descriptors. The following changes apply to the query context handling:
- Integer and long parameters reject values that aren't exact integers instead of truncating or wrapping them. For example
1.5, and values out of range. Integral values like 5.0 or "5.0" are still accepted. Invalid values now fail with an error message: "Query context parameter [x] should be ...." - A null in the query context, or SET x = NULL, no longer clears a default configured with
druid.query.default.contextor the Broker dynamic configuration. The default applies as if the key were not set. Previously the null reached readers, which used the built-in default. To override a configured default, set an explicit value. maxSubqueryBytesis a typed string parameter that accepts auto, disabled or a non-negative number of bytes, as the subquery guardrail does. Invalid values such asSET maxSubqueryBytes = TRUEnow fail during validation, and numbers such as1e9are now accepted.
[#20555][https://github.com//pull/20555/]
Java home variable
Middle Managers now honor JAVA_HOME when using the bin/java script, which should be present in the working directory.
Druid uses the following logic
- If you specified a command (such as if
config.getJavaCommandis not null), use that. - If
bin/run-javaexists, use that - Use java
Quality of service filtering for Coordinator
You can now enable QoS filtering for the Coordinator to prevent a burst of slow API requests from exhausting the thread pool and blocking health checks.
Use druid.coordinator.server.maxConcurrentRequests to configure this feature (default: max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8))). Set this property to 0 to turn it off.
Leader-status endpoints are excluded from the cap so load balancers/monitors stay unaffected.
Incompatible changes
Permissions changes for SQL-based ingestion
The table visibility changes affect the permissions required for SQL-based ingestion. INSERT and REPLACE now require READ access (in addition to WRITE) on the target table.
To restore the previous behavior, set druid.sql.planner.authorizeTableVisibility to false.
Alerts on task/run/time
The statsd and prometheus emitters now emit the taskStatus dimension on task/run/time. This is a breaking change for dashboards and alerts on task/run/time:
statsd(non-DogStatsD mode): Dimension values are joined into the dotted metric name, sodruid.overlord.task.run.time.DATASOURCE.TASKTYPEbecomesdruid.overlord.task.run.time.DATASOURCE.TASKSTATUS.TASKTYPE. Queries against the old name stop receiving data.statsd(DogStatsD mode): AtaskStatustag is added. Existing queries keep working and aggregate over it.prometheus: AtaskStatuslabel is added. Update any queries that select on an exact label set and any recording rules that assume the old label set.
To keep the previous behavior, pin the old dimension list with druid.emitter.statsd.dimensionMapPath or druid.emitter.prometheus.dimensionMapPath.
SysMonitor removed
SysMonitor has been removed after being deprecated in Druid 27. Use OshiSysMonitor instead.
MariaDB Connector/J 3.x changes connection URI behavior
The MariaDB JDBC connector has been upgraded from from 2.7.3 to 3.5.10, which has the following impact:
- For
druid.metadata.storage.connector.connectURIwithorg.mariadb.jdbc.Driver, usejdbc:mariadb:. A legacyjdbc:mysql:URI remains usable only if it includespermitMysqlScheme. For example,jdbc:mysql://<host>/druid?permitMysqlScheme, because metadata storage passes the URI directly to the driver. - For MariaDB-backed SQL input sources and JDBC lookups with the default
druid.access.jdbc.enforceAllowedProperties=true, usejdbc:mariadb:. Druid's JDBC property validation does not routejdbc:mysql:to the Connector/J 3.x parser, andpermitMysqlSchemeis not in the defaultdruid.access.jdbc.allowedProperties, Adding the option alone is not sufficient. - Users who continue supplying Connector/J 2.x do not need to change existing
jdbc:mysql:orjdbc:mariadb:URIs. MySQL Connector/J users are unaffected. - Images rebuilt from
distribution/docker/Dockerfile.mariadbnow install Connector/J 3.5.10 and must follow the 3.x guidance above.
Developer notes
Web console code editor
The web console's code editors moved from Ace (ace-builds and react-ace) to Code Mirror 6. All editors now share a single CodeEditor component, including the workbench, the explore view's SQL inputs, JSON inputs, spec and value dialogs, history, and explain. Two Lezer grammars, one for DruidSQL and one for Hjson, drive the highlighting and the completions.
Dependency updates
Druid 39.0.0 includes a variety of updates to dependencies such as Calcite. For a full list, see the Area - Dependencies label in the milestone.