opensearch-project/OpenSearch 2.12.0

on GitHub

2024-02-09 Version 2.12.0 Release Notes

[2.12.0]

Added

• [Admission control] Add Resource usage collector service and resource usage tracker (#9890)
• [Admission control] Add enhancements to FS stats to include read/write time, queue size and IO time (#10541)
• [Remote cluster state] Change file names for remote cluster state (#10557)
• [Search Pipelines] Add request-scoped state shared between processors (and three new processors) (#9405)
• Per request phase latency (#10351)
• [Remote Store] Add repository stats for remote store(#10567)
• [Remote cluster state] Upload global metadata in cluster state to remote store(#10404)
• [Remote cluster state] Download functionality of global metadata from remote store (#10535)
• [Remote cluster state] Restore global metadata from remote store when local state is lost after quorum loss (#10404)
• [Remote cluster state] Make index and global metadata upload timeout dynamic cluster settings (#10814)
• Add search query categorizer (#10255)
• Per request phase latency (#10351)
• Add cluster state stats (#10670)
• Remove ingest processor supports excluding fields (#10967, #11983)
• [Tiered caching] Enabling serialization for IndicesRequestCache key object (#10275)
• [Tiered caching] Defining interfaces, listeners and extending IndicesRequestCache with Tiered cache support (#10753)
• [Remote cluster state] Restore cluster state version during remote state auto restore (#10853)
• Update the indexRandom function to create more segments for concurrent search tests (10247)
• Add support for query profiler with concurrent aggregation (#9248)
• Introduce ConcurrentQueryProfiler to profile query using concurrent segment search path and support concurrency during rewrite and create weight (10352)
• Implement on behalf of token passing for extensions (#8679)
• Provide service accounts tokens to extensions (#9618)
• [Streaming Indexing] Introduce new experimental server HTTP transport based on Netty 4 and Project Reactor (Reactor Netty) (#9672)
• Enable must_exist parameter for update aliases API (#11210)
• Add back half_float BKD based sort query optimization (#11024)
• Request level coordinator slow logs (#10650)
• Add template snippets support for field and target_field in KV ingest processor (#10040)
• Allowing pipeline processors to access index mapping info by passing ingest service ref as part of the processor factory parameters (#10307)
• Add experimental SIMD implementation of B-tree to round down dates (#11194)
• Make number of segment metadata files in remote segment store configurable (#11329)
• Allow changing number of replicas of searchable snapshot index (#11317)
• Adding slf4j license header to LoggerMessageFormat.java (#11069)
• [BWC and API enforcement] Introduce checks for enforcing the API restrictions (#11175)
• Maintainer approval check (#11378)
• Create separate transport action for render search template action (#11170)
• Add additional handling in SearchTemplateRequest when simulate is set to true (#11591)
• Introduce cluster level setting cluster.index.restrict.replication.type to prevent replication type setting override during index creations(#11583)
• Add match_only_text field that is optimized for storage by trading off positional queries performance (#6836)
• Add copy ingest processor (#11870)
• Introduce new feature flag "WRITEABLE_REMOTE_INDEX" to gate the writeable remote index functionality (#11717)
• Bump OpenTelemetry from 1.32.0 to 1.34.1 (#11891)
• Add remove_by_pattern ingest processor (#11920)
• Support index level allocation filtering for searchable snapshot index (#11563)
• Add org.opensearch.rest.MethodHandlers and RestController#getAllHandlers (11876)
• New DateTime format for RFC3339 compatible date fields (#11465)
• Add community_id ingest processor (#12121)
• Add support for Google Application Default Credentials in repository-gcs (#8394)
• Remove concurrent segment search feature flag for GA launch (#12074)
• Enable Fuzzy codec for doc id fields using a bloom filter (#11022)
• [Metrics Framework] Adds support for Histogram metric (#12062)
• Support for returning scores in matched queries (#11626)

Dependencies

• Bumps jetty version to 9.4.52.v20230823 to fix GMS-2023-1857 (#9822)
• Bump Lucene from 9.7.0 to 9.8.0 (10276)
• Bump commons-io:commons-io from 2.13.0 to 2.15.1 (#10294, #11001, #11002, #11446, #11554, #11560, #11796)
• Bump com.google.api.grpc:proto-google-common-protos from 2.10.0 to 2.25.1 (#10208, #10298)
• Bump com.netflix.nebula.ospackage-base from 11.4.0 to 11.8.0 (#10295, #11630, #12167)
• Bump org.apache.zookeeper:zookeeper from 3.9.0 to 3.9.1 (#10506)
• Bump de.thetaphi:forbiddenapis from 3.5.1 to 3.6 (#10508)
• Bump org.codehaus.woodstox:stax2-api from 4.2.1 to 4.2.2 (#10639)
• Bump org.bouncycastle:bc-fips from 1.0.2.3 to 1.0.2.4 (#10297)
• Bump com.google.http-client:google-http-client from 1.43.2 to 1.43.3 (#10635)
• Bump com.squareup.okio:okio from 3.5.0 to 3.7.0 (#10637, #11632)
• Bump org.apache.logging.log4j:log4j-core from 2.20.0 to 2.22.1 (#10858, #11000, #11270, #11695)
• Bump aws-actions/configure-aws-credentials from 2 to 4 (#10504)
• Bump stefanzweifel/git-auto-commit-action from 4 to 5 (#11171)
• Bump actions/github-script from 6 to 7.0.1 (#11271, #12166)
• Bump jackson and jackson_databind from 2.15.2 to 2.16.0 (#11273)
• Bump netty from 4.1.100.Final to 4.1.106.Final (#11294, #11775), #12034)
• Bump com.avast.gradle:gradle-docker-compose-plugin from 0.16.12 to 0.17.6 (#10163, #11692)
• Bump com.squareup.okhttp3:okhttp from 4.11.0 to 4.12.0 (#10861)
• Bump org.apache.commons:commons-text from 1.10.0 to 1.11.0 (#11344)
• Bump reactor-netty-core from 1.1.12 to 1.1.15 (#11350), (#12042)
• Bump com.gradle.enterprise from 3.14.1 to 3.16.2 (#11339, #11629, #12056)
• Bump actions/setup-java from 3 to 4 (#11447)
• Bump commons-net:commons-net from 3.9.0 to 3.10.0 (#11450)
• Bump org.apache.maven:maven-model from 3.9.4 to 3.9.6 (#11445)
• Bump org.apache.xmlbeans:xmlbeans from 5.1.1 to 5.2.0 (#11448)
• Bump logback-core and logback-classic to 1.2.13 (#11521)
• Bumps jetty version from 9.4.52.v20230823 to 9.4.53.v20231009 (#11539)
• Bump org.wiremock:wiremock-standalone from 3.1.0 to 3.3.1 (#11555)
• Bump org.apache.commons:commons-compress from 1.24.0 to 1.25.0 (#11556)
• Bump actions/stale from 8 to 9 (#11557)
• Bump com.netflix.nebula:nebula-publishing-plugin from 20.3.0 to 21.0.0 (#11671)
• Bump commons-cli:commons-cli from 1.5.0 to 1.6.0 (#10996)
• Bump com.maxmind.geoip2:geoip2 from 4.1.0 to 4.2.0 (#11559)
• Bump org.apache.commons:commons-lang3 from 3.13.0 to 3.14.0 (#11691)
• Bump com.maxmind.db:maxmind-db from 3.0.0 to 3.1.0 (#11693)
• Bump net.java.dev.jna:jna from 5.13.0 to 5.14.0 (#11798)
• Bump lycheeverse/lychee-action from 1.8.0 to 1.9.1 (#11795, #11887)
• Bump Lucene from 9.8.0 to 9.9.2 (#11421), (#12063)
• Bump com.networknt:json-schema-validator from 1.0.86 to 1.2.0 (#11886, #11963)
• Bump com.google.api:gax-httpjson from 0.103.1 to 2.42.0 (#11794, #12165)
• Bump com.google.oauth-client:google-oauth-client from 1.34.1 to 1.35.0 (#11960)
• Bump com.diffplug.spotless from 6.23.2 to 6.25.0 (#11962, #12055)
• Bump com.google.cloud:google-cloud-core from 2.5.10 to 2.30.0 (#11961)
• Bump reactor-core from 3.5.11 to 3.5.14 (#12042)
• Bump org.apache.shiro:shiro-core from 1.11.0 to 1.13.0 (#12200)
• Bump com.google.http-client:google-http-client-jackson2 from 1.43.3 to 1.44.1 (#12059)
• Bump peter-evans/create-issue-from-file from 4 to 5 (#12057)
• Bump org.gradle.test-retry from 1.5.4 to 1.5.8 (#12168)
• Bump org.apache.kerby:kerb-admin from 1.0.1 to 2.0.3 (#12194)

Changed

• Mute the query profile IT with concurrent execution (#9840)
• Force merge with only_expunge_deletes honors max segment size (#10036)
• Add the means to extract the contextual properties from HttpChannel, TcpCChannel and TrasportChannel without excessive typecasting (#10562), (#11751)
• Introduce new dynamic cluster setting to control slice computation for concurrent segment search (#9107)
• Search pipelines now support asynchronous request and response processors to avoid blocking on a transport thread (#10598)
• [Remote Store] Add Remote Store backpressure rejection stats to _nodes/stats (#10524)
• [BUG] Fix java.lang.SecurityException in repository-gcs plugin (#10642)
• Add telemetry tracer/metric enable flag and integ test. (#10395)
• Performance improvement for Datetime field caching (#4558)
• Add instrumentation for indexing in transport bulk action and transport shard bulk action. (#10273)
• Disallow removing some metadata fields by remove ingest processor (#10895, #11607)
• Performance improvement for MultiTerm Queries on Keyword fields (#7057)
• Refactor common parts from the Rounding class into a separate 'round' package (#11023)
• Performance improvement for date histogram aggregations without sub-aggregations (#11083)
• Apply the fast filter optimization to composite aggregation of date histogram source (#11505)
• Disable concurrent aggs for Diversified Sampler and Sampler aggs (#11087)
• Made leader/follower check timeout setting dynamic (#10528)
• Improved performance of numeric exact-match queries (#11209)
• Change error message when per shard document limit is breached (#11312)
• Improve boolean parsing performance (#11308)
• Interpret byte array as primitive using VarHandles (#11362)
• Automatically add scheme to discovery.ec2.endpoint (#11512)
• Restore support for Java 8 for RestClient (#11562)
• Add deleted doc count in _cat/shards (#11678)
• Capture information for additional query types and aggregation types (#11582)
• Use slice_size == shard_size heuristic in terms aggs for concurrent segment search and properly calculate the doc_count_error (#11732)
• Added Support for dynamically adding SearchRequestOperationsListeners with SearchRequestOperationsCompositeListenerFactory (#11526)
• Ensure Jackson default maximums introduced in 2.16.0 do not conflict with OpenSearch settings (#11890)
• Extract cluster management for integration tests into JUnit test rule out of OpenSearchIntegTestCase (#11877), (#12000)
• Workaround for https://bugs.openjdk.org/browse/JDK-8323659 regression, introduced in JDK-21.0.2 (#11968)
• Updates IpField to be searchable when only doc_values are enabled (#11508)
• [Query Insights] Query Insights Framework which currently supports retrieving the most time-consuming queries within the last configured time window (#11903)
• [Query Insights] Implement Top N Queries feature to collect and gather information about high latency queries in a window (#11904)
• Add override support for sampling based on action (#9621)
• Added custom sampler support based on transport action in request (#9621)
• Disable concurrent search for composite aggregation(#12375)

Removed

• Remove deprecated classes for Rounding (#10956)

Fixed

• Fix failure in dissect ingest processor parsing empty brackets (#9225)
• Fix class_cast_exception when passing int to _version and other metadata fields in ingest simulate API (#10101)
• Fix Segment Replication ShardLockObtainFailedException bug during index corruption (10370)
• Fix some test methods in SimulatePipelineRequestParsingTests never run and fix test failure (#10496)
• Fix passing wrong parameter when calling newConfigurationException() in DotExpanderProcessor (#10737)
• Delegating CachingWeightWrapper#count to internal weight object (#10543)
• Fix per request latency last phase not tracked (#10934)
• Fix SuggestSearch.testSkipDuplicates by forcing refresh when indexing its test documents (#11068)
• [BUG] Fix the thread context that is not properly cleared and messes up the traces (#10873)
• Handle canMatchSearchAfter for frozen context scenario (#11249)
• Fix the issue with DefaultSpanScope restoring wrong span in the TracerContextStorage upon detach (#11316)
• Remove shadowJar from lang-painless module publication (#11369)
• Fix remote shards balancer and remove unused variables (#11167)
• Fix parsing of flat object fields with dots in keys (#11425)
• Fix bug where replication lag grows post primary relocation (#11238)
• Fix noop_update_total metric in indexing stats cannot be updated by bulk API (#11485,#11917)
• Fix for stuck update action in a bulk with retry_on_conflict property (#11152)
• Fix template setting override for replication type (#11417)
• Fix Automatic addition of protocol broken in #11512 (#11609)
• Fix issue when calling Delete PIT endpoint and no PITs exist (#11711)
• Fix tracing context propagation for local transport instrumentation (#11490)
• Fix parsing of single line comments in lang-painless (#11815)
• Fix memory leak issue in ReorganizingLongHash (#11953)
• Prevent setting remote_snapshot store type on index creation (#11867)
• [BUG] Fix remote shards balancer when filtering throttled nodes (#11724)
• Add advance(int) for numeric values in order to allow point based optimization to kick in (#12089)