opensearch-go v5.1.2

UPGRADE NOTE - v5.1.2 has two breaking changes.

• Transport retries now wait before they are sent. When RetryBackoff is not set, a retry waits a random delay of up to 100ms, doubled for each retry after the first (exponential backoff with full jitter). v5.1.1 retried at once. A new RetryBudget (default 5s, negative for no bound) caps the total delay across one request's retries, including delays from a RetryBackoff you set. A retry that would exceed it is not sent.
• The opensearchutil names added in v5.1.0 are renamed to describe the bulk client rather than an indexer, and to name the unit they measure. For example, ErrBulkIndexerClosed is now ErrBulkClientClosed, NodeStats is BulkTargetStats, GetNodeStats is GetTargetStats and OPENSEARCH_GO_BULK_NODE is OPENSEARCH_GO_BULK_TARGET. The old names are removed, not deprecated.

Before/after for both is in UPGRADING_V5.md.

v5.1.2 fixes opensearchutil.BulkIndexer throughput when it buffers by shard. v5.1.1 held each node to the transport's write pool window. That window starts at one request, changes only when node statistics are polled, and never exceeds the pool's thread count, so the indexer kept far fewer requests in flight than a node could take. This release also stops the transport from treating a busy write pool as congested and from dropping a node from routing after a single 429.

Requires Go 1.26.0 or later.

Full Changelog: v5.1.1...v5.1.2

Highlights

Faster bulk indexing

In a benchmark of bulk indexing with several concurrent producers, v5.1.2 indexed 1.4x to 3.3x as fast as v5.1.1:

With write queue_size 2, v5.1.2 was also 1.7x as fast as v5.0.3 with the caller retrying rejected items.

The benchmark was run by @iamrajiv. Results on your cluster depend on its size, shard layout, document shape and concurrent load.

Fixes

• opensearchutil.BulkIndexer allows up to NumWorkers requests in flight per node when it buffers by shard, or GOMAXPROCS when NumWorkers is 0. A 429 backs a node off by shrinking its byte window. The write pool window still picks the node for documents that belong to no shard (#1231)
• opensearchutil.BulkIndexer no longer freezes a node's byte window and its shard buffers for as long as the transport reports the node write-saturated. That report can last many seconds, so a window shrunk by a 429 could not grow back. The window and buffers now pause their growth for a few flushes, as after a 429, and then grow again (#1231)
• opensearchtransport.Connection.WritePool reports the transport's initial window, four requests per allocated processor, for a node whose pool sizes have not arrived yet. It used to report one request (#1231)
• opensearchtransport no longer reads a busy FIXED thread pool, such as write or get, as congested. For a pool that reports no wait time, any queued work with every thread busy used to halve the node's routing window, though that is how a well-fed pool looks. Such a pool is now congested when its queue reaches half its capacity or grows by more than its thread count between statistics polls (#1231)
• opensearchtransport no longer removes a node from routing when its thread pool is marked overloaded. A single 429, or a statistics poll that saw the pool reject work from any client, used to take the node out for 5 to 30 seconds and push its traffic onto the other copies until they refused too. A refusal now halves the pool's window, which lowers the node's score. A retry goes to a candidate the request has not tried in its current pass, and once all have been tried, a new pass starts with each candidate's score raised by its earlier tries (#1231)

Breaking changes

See UPGRADING_V5.md for before/after.

• Transport retries wait before they are sent, bounded by the new RetryBudget. With the default router, which retries a 429, every client a node refused used to retry at once, up to six times, against the same few nodes. MaxRetries and HealthCheckMaxRetries still default to 6 (#1231)
• opensearchutil names added in v5.1.0 are renamed (#1231):
  • ErrBulkIndexerClosed, ErrIndexerCanceled, ErrIndexerLayer -> ErrBulkClientClosed, ErrBulkClientCanceled, ErrClientLayer
  • LayerIndexer -> LayerClient. Its String(), and the layer label the osprom and osotel bulk observers record, is client
  • BulkIndexerStats.BudgetFill -> Utilization
  • NodeSizing, NodeUnset, NodeAIMD, NodeFixed, BulkSizing.Node, BulkIndexerConfig.NodeBytes -> TargetSizing, TargetUnset, TargetAIMD, TargetFixed, BulkSizing.Target, TargetBytes
  • OPENSEARCH_GO_BULK_NODE -> OPENSEARCH_GO_BULK_TARGET
  • NodeStats, GetNodeStats, JobNodeStats -> BulkTargetStats, GetTargetStats, JobTargetStats
  • BulkIndexerStats.NodeWindow, NumNodes, FlushNode -> TargetWindow, NumTargets, FlushTarget
  • Error messages say "bulk client"

Dependencies

• Bump github.com/prometheus/client_golang in osprom to v1.25.0 (#1225)
• Bump go.opentelemetry.io/otel/sdk/metric in osotel to v1.47.0 (#1223)
• Bump github.com/getkin/kin-openapi in cmd/osgen to v0.151.0 (#1222)

See CHANGELOG.md for the full list.

Contributors

Thanks to everyone who contributed to this release: @ryanyuan and @sean-.

Thanks to @iamrajiv for finding the write pool window limit and benchmarking the fix.