apache/lucene releases/lucene/10.6.0

10.6.0

on GitHub

Highlights

• Doc-values updates:
  • Incremental updates are written as sparse delta overlays instead of rewriting the whole column. This is on by default, and
    IndexWriterConfig#setMaxDocValuesOverlays(0) disables it.
  • SORTED_NUMERIC fields can now also be updated in place (with a single value).
• Vector search:
  • New sandboxed de-duplicating HNSW formats that store each distinct vector once.
  • Faster HNSW merge-reuse, with graph repair running in parallel.
  • Merges can be aborted promptly (IndexWriter#rollback no longer blocks on a full graph build).
  • Vectorized uint4 unpacking for scalar quantization.
• Merge I/O: merges open their own file mappings and advise sequential reads. This leaves the access advice that searches use unchanged.
• Query performance:
  • Faster bulk scoring, including MUST_NOT handling and MaxScoreBulkScorer filtering.
  • Faster doc-values range queries (SIMD, GCD-aware).
  • Faster string sorting.
  • Bulk live-docs masking.

New features

• StableTflSimilarity estimates term rarity from term length and document length instead of corpus statistics.
• CaseFoldingFilter does Unicode simple case folding without the ICU module.
• Thai analysis additions: user dictionary and char, normalization and repeat filters.
• BinaryDocValues#binaryValues for bulk retrieval.

Notable fixes

• TopFieldCollector returned wrong results when a MultiReader combined indexes with different index sorts.
• DocValuesRangeIterator.docIDRunEnd() returned wrong run boundaries, so range and ordinal-set queries could skip or mis-score documents.
• BitSetIterator#docIDRunEnd() had quadratic cost over long runs of set bits, which has slowed DenseConjunctionBulkScorer since 10.5.0.
• Byte-vector dot products and square distances stayed several times slower for the life of the JVM once both signed and unsigned variants had been used.
• The HNSW merge base graph was chosen by comparing the wrong counts. It now compares live vector counts.
• A join-score float accumulation bug caused intermittent failures.
• IBSimilarity with DistributionSPL could produce negative scores.

API changes

• ReaderUtil.partitionByLeaf is redesigned. It now takes an int[] of doc IDs and returns a PartitionedHits record.
• Added KnnVectorsReader.getVectorCount.