Apache Pinot 1.6.0 RC1

Changes since RC0

• Lucene rollback recovery: rebuild incompatible text and HNSW indexes from retained original forward-index values during segment loading. Compatible indexes are reused; missing recovery sources fail before replacing the index. Lucene remains 9.12.0. Install this recovery release on the rollback target before a later Lucene upgrade, and retain the original forward indexes. Rebuilding adds CPU/I/O and loading time. Backport #19976, upstream #19900.
• Legacy table-config notices: record deprecation versions, replacement guidance and removal intent for 16 legacy properties. The backport changes annotations/Javadoc; runtime acceptance and JSON compatibility remain unchanged. Backport #19978, upstream #18411.
• Approximation reporting: replace RC0's approximateFunctionApplied response flag with approximatedResult. It reports when a SMART aggregation actually produces an approximate result, including explicit SMART queries, rather than whether a rewrite occurred. The corresponding broker meter is BROKER_RESPONSES_WITH_APPROXIMATED_RESULT, and the query-log field is approximatedResult. Always-approximate functions do not set this flag. The old names appeared only in the unreleased 1.6 cycle. #19972.
• Conditional aggregate correctness: run the multi-stage approximate rewrite after CASE-to-FILTER conversion to preserve NULL handling for conditional distinct counts when approximation parameters are configured. Automatic approximate rewriting remains disabled by default. Backport #19977, upstream #19974.

SMART aggregation mixed-version restrictions

The approximation-reporting change adds DataTable metadata and StatMap keys that older readers cannot safely skip. The risk occurs when a SMART aggregation actually turns approximate, including a SMART function explicitly written in a query. Leaving automatic rewriting disabled does not protect explicit SMART queries.

• Upgrade brokers before servers for the DataTable path. An old broker receiving new final-result metadata can misread subsequent metadata or fail with BufferUnderflowException.
• Broker-first ordering alone does not make arbitrary server canaries or downgrades safe. In the multi-stage path, an older StatMap reader can drop that query's statistics, including numGroupsLimitReached, on encountering a new ordinal. Avoid approximation-producing SMART execution on paths where older components would receive the new metadata; do not assume arbitrary canary/downgrade combinations are supported.
• Older servers returning final results do not emit the new metadata. During that transition, a newer broker can report approximatedResult=false for a result produced from a sketch. Treat the flag as incomplete until relevant servers are upgraded.

These are known compatibility limits of #19972, not fixed by the CASE-ordering backport. Plan the rollout and query controls accordingly.

Apache Pinot 1.6.0 release notes

Changes since Apache Pinot 1.5.1.

Highlights

• SQL DDL: manage tables with CREATE TABLE, DROP TABLE, SHOW TABLES, and
  SHOW CREATE TABLE through the controller's /sql/ddl endpoint. Materialized
  views gain CREATE/SHOW/DROP DDL and single-stage query-rewrite support.
  #18241
  #18544
  #18529
• Analytics: GROUPING SETS, ROLLUP, and CUBE are supported in both query
  engines. Multi-stage execution also gains streaming distinct execution
  and lookup-join planning improvements.
  #18662
  #18817
  #19419
  #18158
• Vector search: add IVF_FLAT, IVF_PQ, and IVF_ON_DISK backends alongside
  HNSW, with filter-aware search, radius queries, and backend tuning controls.
  #17994
  #18090
  #18119
• Data handling: introduce OPEN_STRUCT for evolving nested data, a logical
  UUID type, Arrow IPC batch ingestion, and BSON input support.
  #18368
  #19040
  #18869
  #17871
  #18950
• Reliability: improve null-aware filtering, multi-stage buffer cleanup
  during failure/cancellation, and transactional Kafka offset handling.
  #19440
  #19462
  #19161

Upgrade notes and removals

• Pinot services now require Java 25 or newer to build and run. Java/JDBC
  clients and the explicitly pinned SPI/common artifacts continue to target
  Java 11 bytecode. Update service JVMs and check custom service plugins
  before upgrading.
  #19014
• The default Scala binary version changes to 2.13. Official Spark 3
  artifacts, including pinot-spark-3-connector, are compiled for Scala 2.13;
  their Maven artifact IDs have no Scala-version suffix. These artifacts
  require Java 25 and Spark built for Scala 2.13. Scala 2.12 requires a custom
  source build using the scala-2.12 profile.
  #18100
• Native text/FST indexes and TEXT_CONTAINS are removed. Migrate affected
  configurations and queries to supported Lucene text/FST facilities and
  TEXT_MATCH, reviewing the query semantics before upgrading.
  #17998
• Spark 2 and Presto integrations, the deprecated Kafka high-level consumer,
  and several APIs deprecated in releases through 1.3.0 are removed. Recompile custom
  integrations against 1.6.0 and replace removed API calls.
  #17950
  #17947
  #18931
  #19139
  #19140
  #19141
  #19142
  #19143
• New and updated schemas submitted through the controller reject deprecated
  TimeFieldSpec (fieldType=TIME); existing stored schemas can still load.
  Use DateTimeFieldSpec (fieldType=DATE_TIME).
  #18502
• Legacy controller periodic-task aliases are removed. For example, replace
  controller.retention.frequencyInSeconds with
  controller.retention.frequencyPeriod and a duration value such as 6h.
  8dec968
• Kafka's raw auto.offset.reset property now defaults to earliest when
  omitted. This controls recovery from unavailable Kafka offsets and is
  separate from Pinot's stream.kafka.consumer.prop.auto.offset.reset initial
  partition-offset setting. Set the raw property explicitly if another
  recovery behavior is required.
  #19359
• Single-stage queries no longer silently ignore HAVING; unsupported QUALIFY
  clauses are rejected. Review queries that relied on the earlier behavior.
  #19554
  #19553

Candidate artifacts

Candidate commit: affe120ac8e858ace266961931f0c4dabfe21704.

• Source distribution, signature, SHA-512.
• Binary distribution, signature, SHA-512.
• Maven staging repository.

Signing key: F6C06B50BFB7A502B1F95901FE4489E8BC6B342D (Apache Pinot KEYS).

RC1 validation

• The source archive matches the candidate and rebuilt successfully outside Git on JDK 25, with tests skipped. Both the source-built distribution and the signed binary passed the baseballStats demo: COUNT(*) = 97889 in each query engine.
• Source/binary signatures and SHA-512 checks passed. Maven stage orgapachepinot-1085 is closed; all 387 primary/classifier artifacts matched the installed outputs, and all 387 detached signatures and uploaded checksum sidecars passed verification.
• Java 11 client verification passed all 18 checks locally and in hosted CI.
• Hosted unit tests reported 30,259 tests, including 32 skipped; integration tests reported 1,776 tests, including 3 skipped. Both completed with zero failures/errors. Hosted quickstarts passed.
• Supplemental local rolling upgrade/rollback checks against the official 1.5.1 binary passed all 3,332 single-stage and 315 multi-stage query evaluations with a private operational adapter. These standard fixtures do not cover SMART-specific queries; the mixed-version restrictions above still apply.
• Hosted rolling upgrade/rollback checks against Apache Pinot 1.5.1 passed 3,332/3,332 single-stage queries and 315/315 multi-stage queries. These exact-head workflows ran on the release manager's fork, using the same Apache source commits and standard fixtures; all checker, old-version, and candidate builds succeeded.