opensearch-go v5.2.0

UPGRADE NOTE - v5.2.0 sends request bodies under 4 KiB uncompressed.

• A client with a Compressor or CompressRequestBody now compresses only request bodies of 4096 bytes or more. v5.1.3 compressed every body. Bulk bodies are far larger and are compressed as before, and a body whose length is not known is always compressed.
• To compress every body as v5.1.3 did, set CompressMinBytes to a negative value. A positive value sets a different limit, such as 1024 for a slow or metered link.

Before/after is in UPGRADING_V5.md.

v5.2.0 adds zstd request compression for OpenSearch 3.0.0 and later through a new CompressorTable, which picks the request compressor by server version. With compress.ZstdTable, the client sends zstd only to nodes it knows run OpenSearch 3.0.0 or later, judged from the node's version, and gzip to the rest. It also fixes shard routing on indices with routing_partition_size above 1 and stops a canceled request from demoting a healthy node. The osapilint v4 -> v5 migration now reports more fields that changed type.

Requires Go 1.26.0 or later.

Full Changelog: v5.1.3...v5.2.0

What's new

zstd request compression

compress.ZstdTable(level) sends zstd to OpenSearch 3.0.0 and later, and gzip at level 6 to older or unknown versions. Set it as opensearch.Config.CompressorTable, which picks the request Compressor by server version; it cannot be combined with Compressor.

table, err := compress.ZstdTable(compress.DefaultLevel)
if err != nil {
	return err
}
client, err := opensearch.NewClient(opensearch.Config{CompressorTable: table})

Bodies under CompressMinBytes (4096 by default) are sent uncompressed, since compressing them costs more than it saves on most links; measurements are in compress/README.md. Compressed bodies also allocate less: the output buffer is sized from the last body's ratio instead of grown from empty.

compress is a nested module and is not tagged; get it at the v5.2.0 commit: go get github.com/opensearch-project/opensearch-go/compress/v5@5de4b8cb356043e6027c2606aec1a2b7e99d51b5

Shard routing with routing_partition_size

On an index with routing_partition_size above 1, the server places a document by its routing value and an offset taken from its ID, so a routing value alone does not name the shard. The client used to ignore the offset, so it could send a request to a node that does not hold the document's shard. Discovery now reads the setting, and shard-exact routing, bulk placement and direct mget add the offset.

• LookupShard returns ErrDocumentIDRequired, which wraps ErrShardMapUnknown, on such an index. Transport.LookupDocShard and ShardMap.ShardForDocument take the routing value and the ID together. shardhash.ForDocument, shardhash.ForRoutingOffset, ShardMap.RoutingPartitionSize and the optional opensearchutil.DocShardRouter interface are new.
• The bulk indexer sends a document without an ID without a shard placement, and keeps placing the index's other documents.
• A request with ?routing= and no document ID, such as a search, goes to the node holding most of the routing value's partition shards.
• With comma-separated routing values, each target shard is now counted once when the coordinator is picked. A shard reached by more than one value used to be counted once per value.

The optional ShardLookupObserver interface reports each lookup that cannot name a node through OnShardLookupMiss and a ShardLookupMissEvent. RouteEvent.Err holds the reason a request fell back from shard-exact routing.

These changes are in #1292.

Other changes

• Add OpenSearch 3.9.0 to the CI compatibility matrix and make it the default integration test version, replacing 3.8.0 (#1257)

Fixes

• fix(opensearchtransport): an attempt that failed while the caller's context was canceled or past its deadline demoted the node, invalidated its shard placement and was retried. These failures now leave the node available, including on seed fallback. By @Kirollbas in #1254, fixing #1253
• fix(opensearchtransport): a disabled HealthCheckTimeout (< 0) made seed startup health checks and node-stats polls fail at once. Both now run without a deadline. By @iprithv in #1273
• fix(opensearchtransport): with response-body logging on, a mid-read failure in duplicateBody left the original HTTP body open, which could leak a keep-alive connection. By @iprithv in #1274
• fix(opensearchtransport): ColorLogger and CurlLogger panicked when logging a request that never got a response, such as a routing failure. ColorLogger now prints ERROR. By @iprithv in #1289
• fix(signer): the AWS SDK v2 signer's SignRequest sometimes signed a request whose context was already done, because the credentials cache can answer without checking it. It now returns the context error (#1292)
• fix(osapilint): the v4 -> v5 hop renames IndicesRefreshReq.Index, IndicesCloseReq.Index and IndicesOpenReq.Index to Indices, where it used to abort with an unclassified-field error. The close and open fields change from string to []string and are reported MANUAL. By @iprithv in #1275
• fix(osapilint): the v4 -> v5 hop renames SearchParams.Query and UpdateByQueryParams.Query to Q, and reports SearchParams.MinCompatibleShardNode and UpdateByQueryParams.VersionType as MANUAL. It used to abort with an osapilint bug error on code that set them (#1256)
• fix(osapilint): a field whose type changed without becoming a pointer, such as SearchParams.From (*int -> int) or ExpandWildcards (string -> []string), is reported MANUAL wherever it is set or read. These sites used to pass and fail later at build time. Every MANUAL line now ends with the site's position (at file.go:line:col) (#1293)
• fix(osgen): generate the same types under kin-openapi 0.152.0 as under 0.151.0, so the MLGetStatsResponse index status fields stay *string and BulkByScrollTaskStatusOrException keeps its v5.0.0 accessor and constructor names (#1258)
• Fix flaky shard-routing integration tests in opensearchutil, which waited for one warm-up routing value to be placed rather than every primary they used (#1291)

Breaking changes

• Request bodies shorter than CompressMinBytes, 4096 bytes by default, are sent uncompressed. Set CompressMinBytes to a negative value to compress every body (#1286)

Dependencies

• Update the toolchain line to go1.26.10 in the root and all nested modules. The go directive stays 1.26.0, so consumers are not affected (#1256)
• Bump github.com/aws/aws-sdk-go-v2 to 1.47.5 and aws-sdk-go-v2/config to 1.33.13 (#1246, #1276, #1278)
• Bump github.com/getkin/kin-openapi from 0.151.0 to 0.152.0 in cmd/osgen (#1258)
• Pin every nested module to the v5.2.0 root module (#1295)

See CHANGELOG.md for the full list.

Contributors

Thanks to everyone who contributed to this release: @iprithv, @Kirollbas, @sean- and @ryanyuan.

New Contributors

• @Kirollbas (#1254)