opensearch-project/OpenSearch 2.10.0

on GitHub

2023-09-08 Version 2.10.0 Release Notes

[2.10]

Added

• Add server version as REST response header #6583
• Start replication checkpointTimers on primary before segments upload to remote store. (#8221#8221)
• Introduce new static cluster setting to control slice computation for concurrent segment search. (#8847)
• Add configuration for file cache size to max remote data ratio to prevent oversubscription of file cache (#8606)
• Disallow compression level to be set for default and best_compression index codecs (#8737#8737)
• [distribution/archives] [Linux] [x64] Provide the variant of the distributions bundled with JRE (#8195#8195)
• Prioritize replica shard movement during shard relocation (#8875)
• Introducing Default and Best Compression codecs as their algorithm name (#9123)
• Make SearchTemplateRequest implement IndicesRequest.Replaceable (#9122)
• [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them (#9223)
• [Remote Store] Add Segment download stats to remotestore stats API (#8718)
• [Remote Store] Add remote segment transfer stats on NodesStats API (#9168 #9393 #9454)
• [Segment Replication] Support realtime reads for GET requests (#9212)
• Allow test clusters to run with TLS (#8900)
• Add jdk.incubator.vector module support for JDK 20+ (#8601)
• [Feature] Expose term frequency in Painless script score context (#9081)
• Add support for reading partial files to HDFS repository (#9513)
• [Remote Store] Rate limiter integration for remote store uploads and downloads(#9448)
• [BWC and API enforcement] Decorate the existing APIs with proper annotations (part 1) (#9520)
• Add support for extensions to search responses using SearchExtBuilder (#9379)
• [Remote State] Create service to publish cluster state to remote store (#9160)
• Core crypto library to perform encryption and decryption of source content (#8466)
• Expose DelimitedTermFrequencyTokenFilter to allow providing term frequencies along with terms (#9479)
• APIs for performing async blob reads and async downloads from the repository using multiple streams (#9592)
• Add concurrent segment search related metrics to node and index stats (#9622)
• Add average concurrency metric for concurrent segment search (#9670)
• Introduce cluster default remote translog buffer interval setting (#9584)
• Added encryption-sdk lib to provide encryption and decryption capabilities (#8466 #9289)
• [Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level (#9709)
• Added crypto-kms plugin to provide AWS KMS based key providers for encryption/decryption. (#8465)
• [Remote state] Integrate remote cluster state in publish/commit flow (#9665)
• [Remote Store] Changes to introduce repository registration during bootstrap via node attributes. (#9105)
• [Remote state] Auto restore index metadata from last known cluster state (#9831)

Dependencies

• Bump org.apache.logging.log4j:log4j-core from 2.17.1 to 2.20.0 (#8307)
• Bump io.grpc:grpc-context from 1.46.0 to 1.57.1 (#8726, #9145)
• Bump com.netflix.nebula:gradle-info-plugin from 12.1.5 to 12.1.6 (#8724)
• Bump commons-codec:commons-codec from 1.15 to 1.16.0 (#8725)
• Bump org.apache.zookeeper:zookeeper from 3.8.1 to 3.9.0 (#8844, #9146)
• Bump org.gradle.test-retry from 1.5.3 to 1.5.4 (#8842)
• Bump com.netflix.nebula.ospackage-base from 11.3.0 to 11.4.0 (#8838)
• Bump com.google.http-client:google-http-client-gson from 1.43.2 to 1.43.3 (#8840)
• OpenJDK Update (July 2023 Patch releases) (#8869)
• Bump hadoop libraries from 3.3.4 to 3.3.6 (#6995)
• Bump com.gradle.enterprise from 3.13.3 to 3.14.1 (#8996)
• Bump org.apache.commons:commons-lang3 from 3.12.0 to 3.13.0 (#8995)
• Bump com.google.cloud:google-cloud-core-http from 2.21.0 to 2.21.1 (#8999)
• Bump com.maxmind.geoip2:geoip2 from 4.0.1 to 4.1.0 (#8998)
• Bump org.apache.commons:commons-lang3 from 3.12.0 to 3.13.0 in /plugins/repository-hdfs (#8997)
• Bump netty from 4.1.94.Final to 4.1.96.Final (#9030)
• Bump com.google.jimfs:jimfs from 1.2 to 1.3.0 (#9080)
• Bump io.projectreactor.netty:reactor-netty-http from 1.1.8 to 1.1.9 (#9147)
• Bump org.apache.maven:maven-model from 3.9.3 to 3.9.4 (#9148)
• Bump com.azure:azure-storage-blob from 12.22.3 to 12.23.0 (#9231)
• Bump com.diffplug.spotless from 6.19.0 to 6.20.0 (#9227)
• Bump org.xerial.snappy:snappy-java from 1.1.8.2 to 1.1.10.3 (#9252)
• Bump com.squareup.okhttp3:okhttp from 4.9.3 to 4.11.0 (#9252)
• Bump com.squareup.okio:okio from 2.8.0 to 3.5.0 (#9252)
• Bump com.google.code.gson:gson from 2.9.0 to 2.10.1 (#9230)
• Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 (#9228)
• Bump snakeyaml from 2.0 to 2.1 (#9269)
• Bump aws-actions/configure-aws-credentials from 1 to 2 (#9302)
• Bump com.github.luben:zstd-jni from 1.5.5-3 to 1.5.5-5 (#9431
• Bump netty from 4.1.96.Final to 4.1.97.Final (#9553)
• Bump io.grpc:grpc-api from 1.57.1 to 1.57.2 (#9578)
• Add Encryption SDK dependencies (#8466)

Changed

• Default to mmapfs within hybridfs (#8508)
• Perform aggregation postCollection in ContextIndexSearcher after searching leaves (#8303)
• Make Span exporter configurable (#8620)
• Perform aggregation postCollection in ContextIndexSearcher after searching leaves (#8303)
• [Refactor] StreamIO from common to core.common namespace in core lib (#8157)
• [Refactor] Remaining HPPC to java.util collections (#8730)
• Remote Segment Store Repository setting moved from index.remote_store.repository to index.remote_store.segment.repository and cluster.remote_store.repository to cluster.remote_store.segment.repository respectively for Index and Cluster level settings (#8719)
• Change InternalSignificantTerms to sum shard-level superset counts only in final reduce (#8735)
• Exclude 'benchmarks' from codecov report (#8805)
• Create separate SourceLookup instance per segment slice in SignificantTextAggregatorFactory (#8807)
• Replace the deprecated IndexReader APIs with new storedFields() & termVectors() (#7792)
• [Remote Store] Add support to restore only unassigned shards of an index (#8792)
• Add safeguard limits for file cache during node level allocation (#8208)
• Performance improvements for BytesRefHash (#8788)
• Add support for aggregation profiler with concurrent aggregation (#8801)
• [Remove] Deprecated Fractional ByteSizeValue support #9005 (#9005)
• Add support for aggregation profiler with concurrent aggregation (#8801)
• [Remote Store] Restrict user override for remote store index level settings (#8812)
• [Refactor] MediaTypeParser to MediaTypeParserRegistry (#8636)
• Make MultiBucketConsumerService thread safe to use across slices during search (#9047)
• Removed blocking wait in TransportGetSnapshotsAction which was exhausting generic threadpool (#8377)
• Adds support for tracing runnable scenarios (#8831)
• Change shard_size and shard_min_doc_count evaluation to happen in shard level reduce phase (#9085)
• Add attributes to startSpan methods (#9199)
• [Refactor] Task foundation classes to core library - pt 1 (#9082)
• Add support for wrapping CollectorManager with profiling during concurrent execution (#9129)
• Add base class for parameterizing the search based tests #9083 (#9083)
• Add support for wrapping CollectorManager with profiling during concurrent execution (#9129)
• Rethrow OpenSearch exception for non-concurrent path while using concurrent search (#9177)
• Improve performance of encoding composite keys in multi-term aggregations (#9412)
• Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility (#9262)
• Fix sort related ITs for concurrent search (#9177
• [Remote Store] Implicitly use replication type SEGMENT for remote store clusters (#9264)
• Add support to use trace propagated from client (#9506)
• Separate request-based and settings-based concurrent segment search controls and introduce AggregatorFactory method to determine concurrent search support (#9469)
• [Remote Store] Rate limiter integration for remote store uploads and downloads(#9448)
• [Remote Store] Implicitly use replication type SEGMENT for remote store clusters (#9264)
• Redefine telemetry context restoration and propagation (#9617)
• Use non-concurrent path for sort request on timeseries index and field(#9562)
• Added sampler based on Blanket Probabilistic Sampling rate and Override for on demand (#9621)
• Decouple replication lag from logic to fail stale replicas (#9507)
• Improve performance of rounding dates in date_histogram aggregation (#9727)
• [Remote Store] Add support for Remote Translog Store stats in _remotestore/stats/ API (#9263)
• Removing the vec file extension from INDEX_STORE_HYBRID_NIO_EXTENSIONS, to ensure the no performance degradation for vector search via Lucene Engine.(#9528))
• Cleanup Unreferenced file on segment merge failure (#9503)
• Move zstd compression codec to external custom-codecs repository (#9422
• [Remote Store] Add support for Remote Translog Store upload stats in _nodes/stats/ API (#8908)
• [Remote Store] Removing feature flag to mark feature GA (#9761)

Removed

• Remove provision to create Remote Indices without Remote Translog Store (#8719)

Fixed

• Fix flaky ResourceAwareTasksTests.testBasicTaskResourceTracking test (#8993)
• Fix null_pointer_exception when creating or updating ingest pipeline (#9259)
• Fix memory leak when using Zstd Dictionary (#9403)
• Fix condition to remove index create block (#9437)
• Add support to clear archived index setting (#9019)
• Fix range reads in respository-s3 (9512)
• [Segment Replication] Fixed bug where replica shard temporarily serves stale data during an engine reset (#9495)
• Disable shard/segment level search_after short cutting if track_total_hits != false (#9683)
• [Segment Replication] Fixed bug where bytes behind metric is not accurate (#9686)
• Fix broken backward compatibility from 2.7 for IndexSorted field indices (#10045)