What Changed

This release delivers significant improvements across the Multi-stage Query Engine (new UNNEST support, enriched joins, aggregation rewrites), Upsert (offline table support, commit-time compaction, xxhash compression), a new Federation / Multi-Cluster Routing framework, Kafka 4.x support, real-time auto-reset, Time Series Engine enhancements, new indexing capabilities (N-gram, IFST, combined Lucene), Query Resource Isolation, and numerous performance optimizations, security hardening, and bug fixes. It includes over 1100 commits from the community.

Multi-stage Query Engine Enhancements

UNNEST / CROSS JOIN UNNEST Support #17168

The multi-stage engine now natively supports array unnesting via CROSS JOIN UNNEST(...), with optional WITH ORDINALITY. This enables flattening array columns in queries without workarounds, a frequently requested SQL feature.

Enriched Join Operator #16123

The MSE join operator has been enriched with improved execution strategies, enhancing join performance and flexibility for complex multi-table queries.

MSE Lite Mode Improvements

Disable joins in Lite Mode: Added a configuration to explicitly disallow joins in MSE Lite Mode for workloads that should remain scatter-gather only. #17030
Fan-out adjusted limit: Lite Mode now supports adjusted limits based on fan-out to reduce data transferred. #16822
Push collation from Exchange to Sort: Optimizes Lite Mode queries by pushing collation requirements into sort operators. #16551

Physical Optimizer for Logical Tables #17447

Extended the Physical Optimizer to support logical tables, enabling exchange simplification and optimized execution plans for queries against logical table abstractions.

Other MSE Improvements

Row() value expression support in comparisons. #17317
Add maxRowsInJoin, maxRowsInWindow, numGroups to query response metadata. #17784
Broker config to support configuring planner rules disabled by default. #17258
Query option to exclude virtual columns from schema. #17047
Fix non-deterministic hash-distributed exchange routing. #17323
Sort servers for deterministic workerId-to-server mapping across stages. #17342
Always use singleton worker for final sort stage to ensure hotspot server rotation. #17347
Reset gRPC connection backoff when server is re-enabled. #17466
Improved error handling and error messages for leaf stage errors. #17306, #17207, #17161
New MSE metrics for better observability. #17419
MSE latency metric for multi-stage queries. #16398
Support tracing of planner rule productions. #16581
Added support for UNION (distinct) set operations in addition to the existing UNION ALL, enabling more performant deduplication of results from combined queries. #16570

Federation / Multi-Cluster Routing (New)

Pinot 1.5.0 introduces a new federation framework enabling a single broker to route queries across multiple independent Pinot clusters. This is a foundational capability for organizations operating multiple Pinot clusters that need unified query access.

Multi-cluster routing for SSE queries. #17439
Multi-cluster routing for MSE queries. #17444
Federated broker query support. #17296
MultiClusterHelixBrokerStarter for starting broker in federated mode. #17421
Multi-cluster QuickStart for easy setup and testing. #17581
Warnings for unavailable remote clusters in BrokerResponse. #17510
TLS routing support for logical tables. #17663
Physical optimizer support for multi-cluster routing. #16899
Disallow multi-cluster routing for physical tables (logical tables only). #17731

Logical Table Enhancements

Building on the logical table support introduced in 1.4.0, this release adds significant improvements:

Logical table management UI in the Controller. #17878
Logical tables panel in Query Console. #17199
QuickStart for logical tables. #17166
Pluggable LogicalTableConfig serializer/deserializer. #17678
Logical table TLS integration tests. #17683

Upsert and Dedup Enhancements

Upsert Support for Offline Tables #17789

Extends upsert (primary-key deduplication) to OFFLINE tables, a capability previously limited to REALTIME tables. This enables batch-ingested data to leverage primary-key-based deduplication with the same semantics, simplifying batch-only architectures that need upsert guarantees without requiring streaming infrastructure.

Commit Time Compaction #16344

A performance optimization that removes invalid and obsolete records during segment commit, before the segment becomes immutable. This addresses a fundamental challenge where committed segments contain significantly more physical records than logically valid records, reducing storage and improving query performance. Support for commit time compaction with column-major build was also added. #16769

xxHash for Primary Key Compression #17253

Introduces xxHash (xxh_128) as an option for upsert primary key compression. xxHash provides lower memory footprint compared to MD5 with significantly lower collision probability, reducing upsert memory usage.

Persist QueryableDocIds on Disk #16517

QueryableDocIds are now persisted on disk alongside validDocIds snapshots, which is essential for supporting disk-based upsert implementations and faster segment preloading.

Deterministic Data-Only CRC #17264, #17380

Added a data-only CRC computation (excluding metadata) for new segments, persisted to ZK. This is used for realtime committing segments during replace operations to ensure deterministic segment identification regardless of metadata differences across replicas.

Other Upsert/Dedup Improvements

Revert Upsert Metadata of a segment during inconsistencies. #17324
Avoid inconsistencies among replicas during Upsert Compaction Tasks. #17696
Consensus check before selecting segment for compaction or deletion. #17352
Add checks to prevent updating upsert/dedup configs after table creation. #17645
TIMESTAMP support as dedupTimeColumn. #17200
New minNumSegmentsPerTask property for UpsertCompactMerge task. #17104
Make dedup tables use StrictRealtimeSegmentAssignment with multi-tier support. #17154
Optimize previous record location key handling for upsert metadata revert. #17503
Add metric to detect when server is ahead of ZK committed offset. #17139

Real-Time Ingestion

Kafka 4.x Client Support #17633

Added a new pinot-kafka-4.0 stream ingestion module with Kafka 4.1.1 client support. This module uses KRaft mode (no ZooKeeper dependency) and provides full compatibility with Kafka 4.x clusters.

Kafka Subset-Partition Ingestion #17587

Tables can now consume only a subset of Kafka topic partitions, configured via stream.kafka.partition.ids. This enables flexible partition assignment for use cases where a single table doesn't need all partitions.

Remove Kafka 2.0 Plugin #17602

The deprecated Kafka 2.0 plugin has been removed. All ingestion is now standardized on Kafka 3.x (and 4.x) clients. The default Kafka consumer was also changed to Kafka 3. #16858

Auto Reset Offset During Ingestion Lag #16492, #16692, #16724

A three-part feature that enables automatic offset reset when ingestion lag exceeds configurable thresholds (by offset count or time). During segment commit, if lag exceeds the threshold, the new segment starts from the latest offset instead of next offset. Part 2 introduces topic "inactive" status, and Part 3 adds a backfill manager and handler interface for recovering skipped data.

Pause Consumption in Batches #17194

For tables with large numbers of partitions, pausing consumption can overload the controller as all consumers rush to commit. This adds batch configuration parameters to the pause consumption API, spreading the load.

Real-Time Table Replication Across Clusters #17235

First part of enabling real-time table replication between two Pinot clusters. Introduces functionality for creating a new real-time table with consuming segment watermarks from a source table's ZK metadata.

Other Ingestion Improvements

Fix all race conditions from stream consumer. #17089
Improve consumer recreation retry policy. #17062
End-to-end ingestion delay metric. #17163
Freshness checker uses minimum ingestion lag. #17598
Skip freshness check for streams not supporting offset lag. #17560
Fix multi-topic ingestion routing table errors. #17810
Fix incorrect stream partition id for multi-stream realtime consumption. #17953
Confluent library bumped to 7.9.5 to match Kafka 3.9.1. #17614
SimpleAvroMessageDecoder: support header-prefixed payloads (e.g., Confluent magic+schemaId). #17077

Time Series Engine Enhancements

Timeseries Engine is now GA in Pinot 1.5.0; building on the beta time series engine from 1.4.0:

Delta/DeltaDelta Storage Encoding #15258

Added Delta and DeltaDelta compression codecs for time series storage, improving compression ratios for monotonically increasing or slowly changing numeric data.

Query Execution Improvements

Query options and explain plan support in Controller UI. #17511
End-to-end query statistics propagation. #17170
Logical explain plan support. #17484
Partial results support. #17278
Query event listeners. #17464
M3QL JavaCC parser to replace custom tokenizer. #17192
POST /query/timeseries as BrokerResponse-compatible API. #16531
Apache ECharts visualization in Controller UI. #16390
Timeseries language endpoint and UI integration. #16424
Query stats panel in timeseries UI. #17423
DeltaDelta/Delta compression table config validation. #17252

Aggregation Function Improvements

ANY_VALUE Support #16678

New ANY_VALUE aggregation function that returns an arbitrary value from a column for each group. Useful for including columns in SELECT with a 1:1 mapping to GROUP BY columns without forcing them into the GROUP BY clause.

Smart Distinct Count Functions

DistinctCountSmartHLL: Improved for dictionary-encoded columns, automatically choosing exact counting for low cardinality and HLL++ for high cardinality. #17411, #17011
DistinctCountSmartULL: Smart distinct count backed by UltraLogLog (ULL) with configurable promotion threshold from exact set to ULL sketch. #16605

Automatic Aggregation Rewrites

The query engine now automatically rewrites aggregation functions to type-specific optimized variants:

MIN/MAX on string columns rewrites to MINSTRING/MAXSTRING. #16980
MIN/MAX/SUM on long columns rewrites to MINLONG/MAXLONG/SUMLONG. #17058
New SumIntAggregationFunction for optimized INT column aggregation. #16704
LONG-specific MIN/MAX/SUM aggregation functions. #17001
Dictionary/metadata based optimization for MINSTRING/MAXSTRING. #16983
Comprehensive AggregationOptimizer framework for multiple functions. #16399

Multi-Value Aggregation Consolidation #17519, #17109

Multi-value aggregation functions (e.g., COUNTMV, SUMMV) have been consolidated into regular single-value aggregation functions, simplifying the user experience.

Star-Tree Index on MV Columns #16836

Star-tree index now supports aggregation on multi-value columns (e.g., AVGMV), enabling pre-aggregated queries on MV data.

Other Aggregation Improvements

AVG window function value aggregator. #17268
ListAggMv (multi-value list aggregation) with distinct variant. #17155
ArrayAgg on multi-value columns. #17153
Handle empty tables for various aggregation functions. #17750

New Scalar Functions and Operators

INITCAP: Capitalizes the first letter of each word. #17642
IP Address Functions: Suite of IP address manipulation functions. #17127
Logical Functions: AND, OR, NOT, IF, and related boolean logic functions. #17189
RAND with determinism: Scalar functions now advertise determinism; RAND supports both runtime-only and seeded modes. #17208
arrayPushFront / arrayPushBack: Append or prepend elements to arrays. #17140
ARRAYHASANY: Check if two arrays share any common element. #17156
Ngrams MV UDFs: Generate n-grams from text as multi-value columns. #16671
String functions for regex and distance: New string manipulation functions. #17372
jsonExtractKey depth control and dot notation. #16306
Sanitization, Special, and Time column transformers. #17740
LIKE predicate now case-insensitive. #16568
REGEXP_LIKE enhancements: Scan dictionary when small; configure switch for dictionary vs raw scan. #16478, #16879
StringFunctions.replace() enhanced. #16528

Indexing and Storage

N-gram Filtering Index #16364

New realtime N-gram filtering index that efficiently prunes non-matching strings before final regex validation. Benchmarks show 100x matching speed improvement compared to brute-force regex matching. The index extracts N-grams of configurable length and builds posting lists for each N-gram.

IFST Index for Case-Insensitive Regex #16276

New Insensitive FST (IFST) index type alongside the existing FST for case-insensitive regex matching. A third parameter in REGEXP_LIKE specifies case sensitivity.

Minimum Should Match in Lucene Text Search #16650

Added OpenSearch-compatible minimum_should_match support for TEXT_MATCH queries, enabling control over how many terms must match in boolean queries. Supports both integer and percentage-based specifications.

Combined Lucene Text Index Files #16688

Lucene text index files can now be combined and merged into the columns.psf segment file, reducing the number of files per segment and improving storage management.

Column Major Segment Build #16727

New feature to build segments in columnar fashion for columnar input data sources (e.g., Pinot segments, Parquet files). This significantly improves segment rebuild performance for minion operations like schema or index config changes.

Other Indexing Improvements

Raw bitmap inverted index creator and reader. #17060
Enable JSON index for MAP data type. #16808
Match prefix phrase query Lucene parser. #16476
NoDict hybrid cardinality: exact counting up to 2,048, then HLL++ with +10% cap. #17186
Use v4 as default raw index version. #16943
Lucene doc ID mapping for offline segments when storeInSegmentFile is true. #17437
Make ForwardIndexReader pluggable. #16363
Don't fail segment build if star-tree index build fails; skip and support rollback. #17028
Star-tree validation for * column for non-COUNT functions. #17008
Rebuild H3 index on segment reload if resolution config is updated. #16953

Query Performance Optimizations

Early short circuit with AND/OR optimizations: Skip evaluation of remaining predicates when result is already determined. #16583
DescDocIdSetOperator: Improves ORDER BY DESC performance on sorted columns by scanning in reverse order. #16789
Trim group when orderBy = groupBy: Sort-aggregate and pair-wise merge optimization when ORDER BY keys match GROUP BY keys, trimming to LIMIT per segment. #16308
Skip per-row convertTypes() in FunctionOperand when types already match. #17730
Reduce array copying for non-equi join conditions in LookupJoinOperator. #17542
Force colocated join when is_colocated_by_join_keys is provided for semi-join. #17273
Optimized cloud file listing with per-page filtering and early termination in PinotFS. #17847
JSON message decoding optimization: Parse directly to Map, avoiding intermediate JsonNode. #17485
Optimize ProtoBufRecordExtractor with field descriptor caching. #17593
Optimize group ID generation by reducing memory allocation. #16798
Reduce memory footprint for MailboxContentObserver and ReceivingMailbox. #16872
Optimise fetchPartitionGroupIdToSmallestOffset in RealtimeSegmentManager. #17712
Optimise index stats collector for no-dict columns. #16845
Disable JOIN_PUSH_TRANSITIVE_PREDICATES optimization rule by default. #17181
Disable Calcite's AggregateUnionAggregate planner rule by default. #16515

Query Resource Isolation

Workload Scheduler #16018

An improvement over Binary Workload Scheduler that verifies CPU/memory budgets at admission time. Uses WorkloadBudgetManager.canAdmitQuery() before scheduling, with O(1) atomic-counter budget checks adding negligible overhead.

Cost-Split Support #16672

Workload budgets (CPU/memory) can now be split across tables/tenants with support for real-time splits through custom overrides per propagation scheme (e.g., CONSUMING vs COMPLETED stages).

Other Resource Isolation Improvements

Queue-based throttling for ThrottleOnCriticalHeapUsageExecutor (graceful degradation instead of immediate rejection). #16409
Byte-based throttling. #16572
Queries self-terminate in panic mode. #16380
QUERIES_THROTTLED metric. #16676
Workload stats collection interface. #16340

Security and TLS

TLS protocol allowlist: Allow configuring permitted TLS protocols and hardened JVM SSL initialization. #17776
Centralized SSL context: Set SSL context in Controller/Broker/Server contexts for consistent TLS configuration. #17358
Runtime TLS diagnostics for gRPC and HTTPS connections. #17559
Access control for segment download API. #17508
MSE gRPC channel authorization. #16475
@authenticate annotation on critical endpoints. #17552
SPI-based token resolver for custom audit identity resolution. #17658
Audit logging system: Comprehensive audit logging support with cluster config listener, response auditing, request body handling with size limits, URL filter patterns, and component-specific config prefixes. #16616, #16649, #16851, #16747, #16807, #16823, #16766
SAS token authentication for Azure Data Lake Storage (ADLS). #16343
Validate gRPC broker connections on connect. #17835
Refresh AWS credentials at runtime. #16553
Auth for revert replace segment. #16434

Minion / Task Framework

Table-level distributed locking for atomic minion task generation across controllers. #16857
Configurable max tasks per minion instance. #16981
Cluster-level max subtasks configuration. #16571
Subtask timing metrics. #17190
Task data cleanup on table deletion. #16307
Tag minion as Drained. #17375
GET /minions/status API endpoint. #17475
API for all minion tasks and summaries. #17330
Bound task queue size to prevent ZK bloat. #17741
LaunchBackfillIngestionJob for complete backfill support (handles fewer segments than original). #16890
Metadata push mode in BaseSingleSegmentConversionExecutor. #17632
PurgeTask auto-deletes empty segments. #16368
Isolate long-running minion task API resources on controller. #17494
Minion task metrics fixes. #17128

Rebalance and Cluster Management

Disaster Recovery Mode as Controller/Cluster config. #17243
Tenant Rebalance Cancellation. #16886
TenantRebalanceChecker for improved monitoring. #16455
Peer-download enabled table rebalance handling. #16341
Helix constraint on state transition messages. #16933
StateTransitionThreadPoolManager to override executor in SegmentOnlineOfflineStateModelFactory. #17453
Resume partially offline replicas in controller validation. #17754
Configurable server segment predownload. #17287
Batch delete deleted directory segments. #16848
Async segment refresh processing when enabled. #16931
Rebalance option to disable summary. #16615
Enable deep store upload retry for Pauseless by default. #17241
Fix Fault-Domain-Aware Instance Assignment failing rebalance. #17799
Fix rebalance logic to treat COMMITTING segments as CONSUMING. #16348

Observability and Metrics

Query Fingerprint: Configurable AST-based query fingerprinting that replaces literals with ? to create canonical query representations. Enriches logging via MDC and structured logs. Includes a POST endpoint for debugging. #17177
FlameGraph visualization for MSE query stage stats with clock time and allocation modes. #17263
Memory and GC time tracking in query execution statistics. #16576
SLA-style per-query error metrics on broker. #17457
Query response size metrics on broker and server. #17412, #17710
Segment download latency metric. #16375
Netty and gRPC memory metrics (total max and used). #16939
Zone failure tolerant segments metric on broker. #16334
Segment reload failure tracking with in-memory status cache and segment-level details. #17099, #17234
Easier to read query stats. #16794
Audit log performance metrics. #16869

Connectors and Clients

Pinot CLI #17029

New pinot-cli module providing a modern interactive and batch command-line client for Apache Pinot. Features include multi-line SQL REPL with history, multiple output formats (CSV/TSV/JSON/ALIGNED/VERTICAL/MARKDOWN), config via properties file, and support for extra HTTP headers.

Comprehensive Pinot Admin Client #17040

New admin client in pinot-java-client providing complete programmatic access to all Pinot controller REST endpoints. Includes 6 service clients (Table, Schema, Instance, Segment, Tenant, Task), authentication support, async operations, and comprehensive error handling.

Other Client Improvements

Cursor-based pagination support in Java client. #16782
Apache Arrow format decoder for stream data ingestion with 20-30% Kafka data volume reduction. #17031
X-Correlation-Id header in Java client HTTP requests. #17863
Spark connector improvements: Pinot Proxy, unified secureMode, comprehensive gRPC support. #16666
Fix Pinot JDBC driver metadata version retrieval. #17275
Fix pinot-driver broker cache permission. #17092
Use /table/livebrokers/ API in Spark connector. #14802

Controller UI Improvements

Logical table management UI. #17878
Primary key display in schema UI. #17566
Instance liveness vs health distinction. #17450, #17248
Minion task stats and status filtering. #16521
Package Versions section in cluster manager. #16534
Time zone support and segment start/end time in summary. #16489
Webpack 5 and TypeScript 5.8 migration. #16365
Query UI partial result flag fix. #17368
Fix table/schema deletion from UI. #17294
Fix Zookeeper Browser loading and display in QuickStart. #16440, #17490
Fix segment search bar width. #17520
Fix table type input always disabled. #17052
Rebalance UI flow fixes. #16545
Header links next to cluster name. #17499

Other Notable Features

ignoreMissingSegments query option: Skip SERVER_SEGMENT_MISSING errors with broker auto-config. #16556
$partitionId virtual column: Access the partition ID as a virtual column in queries. #16721
Query routing strategy pluggable. #17364
Pluggable table samplers with precomputed broker routing entries. #17532
Storage quota check for batch segment upload. #17512
Disk utilization check extended to offline segment upload. #17579
PinotFS streamed untar for segment download. #17586
Global pinot.md5.disabled switch to enforce MD5 guards. #17800
Custom map metadata in segment metadata and upload. #17055
Segment tier movement API: Select all segments at once for tier migration. #17812
Custom retention time for untracked segments. #16719
Prune option for lineage operations. #16310
Enhanced /tableConfigs/validate with cluster-aware validations. #16675
Deterministically order dimension-table segments during lookup loading. #17536
Enforce broker instance tags at startup. #17786
Updated timezone support with more timezones. #17543
Segment reload with force download for REALTIME tables. #17400
Upload from server to deep store endpoint. #17197
MAP type support in derived columns during segment reload. #17113
MAP data type support in GenericRowSerializer/Deserializer for file ingestion. #16843
Change default to DISALLOW_ALWAYS for dropOutOfOrderRecord tables. #17671
Move DimTable disablePreload to cluster config-based default. #17313
Backward compatible class name registration API. #17772
Consuming segment check for retention check. #17117
Correct HTTP response codes for errors. #17341
Enable auto reset of error segments by default. #15736
Helm chart: PDB for minion and minion-stateless. #17538
Replace Docker-based Kafka with embedded KRaft broker for integration tests. #17790

Notable Bug Fixes

Fix Broken Lucene Query Tracking and Cancellation for OOM Protection. #17884
Fix NOT TEXT_MATCH false positives on consuming segments. #17880
Fix incorrect stream partition id for multi-stream realtime consumption. #17953
Fix double-unescaping of SQL string literals in single-stage engine. #17438
Fix broker write failure handling in ServerChannels. #17861
Fix server outbound write failure creating zombie channels. #17845
Fix QueryServer channel cleanup on close. #17854
Fix negative reload estimated time and year 2088 rebalance timestamp. #17833
Fix ResponseStoreCleaner bugs causing unbounded response store growth. #17622
Fix funnel aggregation worker threads not responding to timeout/cancellation. #17692
Fix off-heap memory spike issue. #17489
Fix SSE query optimization regression for hybrid tables. #17751
Fix bug with COUNT(col) when using star-tree index with null handling enabled. #17106
Fix null handling issue for string to timestamp casts. #16885
Fix basic null handling issue in MSE. #16658
Fix null values in JsonExtractScalarTransformFunction with default value support. #16683
Handle null values gracefully in SSE post aggregation functions. #16711
Fix CAST evaluation with literal-only operands in MSQE. #16421
Fix AWS SDK credential provider thread leak in S3PinotFS. #17869
Fix S3PinotFS URL Encoding for S3-compatible storage. #17691
Improve GcsPinotFS deleteBatch resiliency. #17713
Fix DirectOOMHandler string matching bug for Netty direct memory OOM. #17684
Fix early termination on MSE operator. #16696
Fix broker request id generator to avoid generating same id. #16661
Fix key map cleanup in group key generator. #16982
Fix lead/lag window functions filling beyond num rows. #17348
Fix truncate(value) signed-zero and non-finite behavior. #17677
Fix roundDecimal() for null FLOAT values. #15377
Handle NaN and +/- Infinity in ROUND_DECIMAL. #16993
Fix complex field ser/de. #17685
Fix metadata APIs to report primary key counts for dedup tables. #17736
Fix alias handling for empty response on SSE queries with post aggregation. #17815
Respect column aliases in empty query responses. #17143
Fix non-daemon threads blocking JVM shutdown in RenewableTlsUtils. #17221
Fix fsm creation when segment is in committing state. #16847
Fix NPE in TableRebalancer. #17723
Fix retry of partial table deletion. #16835
Fix real-time ingestion metrics. #16783
Fix multi-stage query stats serde compatibility issue caused by UNNEST. #17590
Ensure proper exception returned when query is cancelled/killed. #17009
Fix query cancellation on op-chain not scheduled. #16625
Fix debug/routingTable always returning same replica-group. #17403
Handle mixed-type map key values by promoting to string. #17722
Fix input format resolution inconsistency between PluginManager and RecordReaderFactory. #17601
Fix TextIndexConfig asymmetric serialization. #17628
Fix Schema/TableConfigs serialization with JsonValue annotation. #17558
Fix Segment Partition Tracking for Multi-Topic Ingestion. #17217
Fix wrong interval for OOM protection. #17753

apache/pinot release-1.5.0-rc0 Apache Pinot Release 1.5.0 RC0 on GitHub