nats-io/nats-server v2.4.0

Release v2.4.0

on GitHub

Changelog

Notice for JetStream Users

With the latest release of the NATS server we have fixed bugs around queue subscriptions and have restricted undesired behavior that could be confusing or introduce data loss by unintended/undefined behavior of client applications. If you are using queue subscriptions on a JetStream Push Consumer or have created multiple push subscriptions on the same consumer, you may be affected and need to upgrade your client version along with the server version. We’ve detailed the behavior with different client versions below.

With a NATS Server prior to v2.4.0 and client libraries prior to these versions: NATS C client v3.1.0, Go client v1.12.0, Java client 2.12.0-SNAPSHOT, NATS.js v2.2.0, NATS.ws v1.3.0, NATS.deno v1.2.0, NATS .NET 0.14.0-pre2:

• It was possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is not correct since each instance will receive the same copy of a message and acknowledgment is therefore meaningless since the first instance to acknowledge the message will prevent other instances to control if/when a message should be acknowledged.
• Similar to the first issue, it was possible to create many different queue groups for one single JetStream consumer.
• For queue subscriptions, if no consumer nor durable name was provided, the libraries would create ephemeral JetStream consumers, which meant that each member of the same group would receive the same message than the other members, which was not the expected behavior. Users assumed that 2 members subscribing to “foo” with the queue group named “bar” would load-balance the consumption of messages from the stream/consumer.
• It was possible to create a queue subscription on a JetStream consumer configured with heartbeat and/or flow control. This does not make sense because by definition, queue members would receive some (randomly distributed) messages, so the library would think that heartbeat are missed, and flow control would also be disrupted.

If above client libraries are not updated to the latest but the NATS server is upgraded to v2.4.0:

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. Since the check is performed by the library (with the help of a new field called PushBound in the consumer information object set by the server), this mis-behavior is still possible.
• Queue subscriptions will not receive any message. This is because the server now has a new field DeliverGroup in the consumer configuration, which won’t be set for existing JetStream consumers and by the older libraries, and detects interest (and starts delivering) only when a subscription on the deliver subject for a queue subscription matching the “deliver group” name is found. Since the JetStream consumer is thought to be a non-deliver-group consumer, the opposite happens: the server detects an core NATS queue subscription on the “deliver subject”, therefore does not trigger delivery on the JetStream consumer’s “deliver subject”.

The 2 other issues are still present because those checks are done in the updated libraries.

If the above client libraries are update to the latest version, but the NATS Server is still to version prior to v2.4.0 (that is, up to v2.3.4):

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is because the JetStream consumer’s information retrieved by the library will not have the PushBound boolean set by the server, therefore will not be able to alert the user that they are trying to create multiple subscription instances for the same JetStream consumer.
• Queue subscriptions will fail because the consumer information returned will not contain the DeliverGroup field. The error will be likely to the effect that the user tries to create a queue subscription to a non-queue JetStream consumer. Note that if the application creates a queue subscription for a non-yet created JetStream consumer, then this call will succeed, however, adding new members or restarting the application with the now existing JetStream consumer will fail.
• Creating queue subscriptions without a named consumer/durable will now result in the library using the queue name as the durable name.
• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

For completeness, using the latest client libraries and NATS Server v2.4.0:

• Trying to start multiple non-queue subscriptions instances for the same JetStream consumer will now return an error to the effect that the user is trying to create a “duplicate subscription”. That is, there is already an active subscription on that JetStream consumer.
  It is now only possible to create a queue group for a JetStream consumer created for that group. The DeliverGroup field will be set by the library or need to be provided when creating the consumer externally.
• Trying to create a queue subscription without a durable nor consumer name results in the library creating/using the queue group as the JetStream consumer’s durable name.
• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

Note that if the server v2.4.0 recovers existing JetStream consumers that were created prior to v2.4.0 (and with older libraries), none of them will have a DeliverGroup, so none of them can be used for queue subscriptions. They will have to be recreated.

Go Version

• 1.16.7: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Domain to the content of a PubAck protocol (#2432, #2434)
  • PushBound boolean in ConsumerInfo to indicate that a push consumer is already bound to an active subscription (#2438)
  • DeliverGroup string in ConsumerConfig to specify which deliver group (or queue group name) the consumer is created for (#2438)
  • Warning log statement in situations where catchup for a stream resulted in an error (#2444)
• Monitoring:
  • Ability for normal accounts to access scoped connz information (#2437)
• Operator option resolver_pinned_accounts to ensure user are signed by certain accounts (#2461)

Changed

• JetStream:
  • ConsumerInfo's Delivered and AckFloor are now SequenceInfo instead of SequencePair. SequenceInfo contains Last which represents the last active time (in UTC) (#2462)
  • Delivery of messages for consumers will now be subject to proper group information. Older clients may fail to receive messages on a queue subscription if the library has not been updated to specify the DeliverGroup (#2438)

Improved

• Make error message actionable when adding operator and leaf nodes. Thanks to @alsuren for the contribution (#2449)
• JetStream:
  • File utilization when using a Jetstream stream as a KeyValue store (#2456)
  • Encryption at rest with message expiration on server restart (#2467)
  • Enable global account on non JetStream servers in mixed mode (#2473)

Fixed

• JetStream:
  • Stream delete can fail for non empty directory (#2418)
  • Possible panic for concurrent stream remove and consumer create (#2419)
  • Simplified flow control and avoid stalls due to message loss (#2425)
  • Consumer info max_msgs_per_subject defaults to 0, but should be -1 (#2426)
  • Creating a consumer with a max_waiting value and a deliver subject was returning the wrong error message (#2427)
  • Assign default to max_ack_pending when AckExplicit or AckAll (#2428)
  • Subscriptions for internal clients for JetStream consumers were not properly removed, resulting in possible subscriptions leak/high memory usage (#2439)
  • Expiration of messages during a server restart could lead to clients reporting errors after reconnect when trying to send new messages (#2452)
  • Added additional checks for failures during filestore encryption (#2453)
  • Processing of a publish ACK could cause a server panic (#2460)
  • Consumer's number of pending messages for multiple matches and merging (#2464)
  • Deadlock due to lock inversion when creating a RAFT group. Thanks to @lokiwins for the report (#2471)
• Leafnode:
  • User authorization issue when JetStream is involved. Thanks to @wchajl for the report (#2430)
  • A remote websocket connection with wss:// scheme but no tls{} configuration block would be attempted as non TLS connection, resulting on an "invalid websocket connection" in the log of the server attempting to create the remote connection (#2442)
  • Wrong permission checks prevented messages flow. Thanks to @rbboulton for the report (#2455, #2470)
  • Daisy chained subject propagation issue. Thanks to @KimonHoffmann for the report (#2468)
• Possible subscription leak with the use of "AutoUnsubscribe" (#2421)
• Prevent JWT claim updates from removing system imports (#2450, #2451)
• Error print when adding back existing system imports (#2466)
• Build on OpenBSD-6.9. Thanks to @miraculli for the contribution (#2472)

Complete Changes

v2.3.4...v2.4.0