redis/lettuce 6.0.0.RELEASE on GitHub

The Lettuce team is delighted to announce the availability of Lettuce 6.

Lettuce 6 aligns with Redis 6 in terms of API and protocol changes. Both protocols, RESP and RESP3 are supported side-by-side defaulting to RESP.

Most notable changes that ship with this release are:

RESP3 support
ACL Authentication with username/password
Asynchronous Cluster Topology Refresh
Client-side caching support
Registration of push message listeners
Configuration files for GraalVM Native Image compilation
Kotlin Coroutine API
Redesign command latency metrics publishing
API cleanups/Breaking Changes

Lettuce 6 supports Redis 2.6+ up to Redis 6.x. In terms of Java runtime, Lettuce requires at least Java 8 and works with Java 15.

Thanks to all contributors who made Lettuce 6.0.0.RELEASE possible.
Lettuce requires a minimum of Java 8 to build and run and is compatible with Java 15. It is tested continuously against the latest Redis source-build.

Documentation

Reference documentation: https://lettuce.io/core/6.0.0.RELEASE/reference/
Javadoc: https://lettuce.io/core/6.0.0.RELEASE/api/

RESP3 Support

Redis 6 ships with support for a new protocol version. RESP3 brings support for additional data types to distinguish better between responses. The following response types were introduced with RESP3:

Null: a single null value replacing RESP v2 *-1 and $-1 null values.
Double: a floating-point number.
Boolean: true or false.
Blob error: binary-safe error code and message.
Verbatim string: a binary-safe string that is typically used as user message without any escaping or filtering.
Map: an ordered collection of key-value pairs. Keys and values can be any other RESP3 type.
Set: an unordered collection of N other types.
Attribute: Like the Map type, but the client should keep reading the reply ignoring the attribute type, and return it to the client as additional information.
Push: Out-of-band data.
Streamed strings: A large response using chunked transfer.
Hello: Like the Map type, but is sent only when the connection between the client and the server is established, in order to welcome the client with different information like the name of the server, its version, and so forth.
Big number: a large number non-representable by the Number type

Lettuce supports all response types except attributes. Push messages are only supported for Pub/Sub messages.

The protocol version can be changed through ClientOptions which disables protocol discovery:

ClientOptions options = ClientOptions.builder().protocolVersion(ProtocolVersion.RESP2).build();

Future versions are going to discover the protocol version as part of the connection handshake and use the newest available protocol version.

ACL Authentication

Redis 6 supports authentication using username and password. Lettuce's RedisURI adapts to this change by allowing to specify a username:

redis://username:password@host:port/database

Using RESP3 or PING on connect authenticates the connection during the handshake phase. Already connected connections may switch the user context by issuing an AUTH command with username and password:

StatefulRedisConnection<String, String> connection = client.connect();
RedisCommands<String, String> commands = connection.sync();
commands.auth("username", "password");

Asynchronous Cluster Topology Refresh

Cluster Topology Refresh was in Lettuce 4 and 5 a blocking and fully synchronous task that required a worker thread. A side-effect of the topology refresh was that command timeouts could be delayed as the worker thread pool was used for timeout tasks and the topology refresh. Lettuce 6 ships with a fully non-blocking topology refresh mechanism which is basically a reimplementation of the previous refresh mechanism but using non-blocking components instead.

Server-assisted Client-side caching support

Redis can notify clients about cache invalidations when you use Redis as a read-through cache.
That is when applications keep a local copy of the cached value and use Redis to back up the local cache before.
When a cache values gets changed (that were fetched from Redis), then Redis notifies interested clients so they can invalidate their near-cache and potentially fetch the changed value.

Redis 6 allows for tracking clients and sending push messages using RESP3 push messages.
Lettuce provides a CacheFrontend that can be used to interact with a cache.

Client-side caching assumes a near cache that can be queried, updated and evicted for individual keys.
Cache values are represented as strings and the entire functionality is exposed through a CacheFrontend.

See the following example that uses a ConcurrentHashMap as near cache and outlines the interaction between involved parties:

Map<String, String> clientCache = new ConcurrentHashMap<>();

RedisCommands<String, String> otherParty = redisClient.connect().sync();

// creates a key
otherParty.set(key, value);

StatefulRedisConnection<String, String> myself = redisClient.connect();
CacheFrontend<String, String> frontend = ClientSideCaching.enable(CacheAccessor.forMap(clientCache), myself,
        TrackingArgs.Builder.enabled().noloop());

// Read-through into Redis
String cachedValue = frontend.get(key);
assertThat(cachedValue).isNotNull();

// client-side cache holds the same value
assertThat(clientCache).hasSize(1);

// now, the key expires
commands.pexpire(key, 1);

// a while later
Thread.sleep(200);

// the expiration reflects in the client-side cache
assertThat(clientCache).isEmpty();

assertThat(frontend.get(key)).isNull()

Lettuce ships out of the box with a CacheAccessor for the Map interface.
You can implement a CacheAccessor for your cache if it doesn't implement the Map interface.
Client-side caching support is only supported when using RESP3. The Pub/Sub mode isn't supported through ClientSideCaching and we don't support Pub/Sub redirection.
Since push messages are node-local, client-side caching is supported only on Redis Standalone setups.
Master/Replica or Clustering operating modes are not supported as multi-node operations and connection failover impose severe complexity onto key tracking.

Registration of push message listeners

Registration of push message listeners completes Lettuce's RESP3 support.
You can register PushMessage listeners on Standalone and Redis Cluster connections by implementing PushListener respective RedisClusterPushListener:

connection.addListener(message -> {

    if (message.getType().equals("invalidate")) {
        invalidations.addAll((List) message.getContent(StringCodec.UTF8::decodeKey).get(1));
    }
});

Using push messages with Redis Cluster are subject to node-locality, therefore RedisClusterPushListener provides access to the RedisClusterNode from which the push message originated.

The content of push messages may consist of arbitrary data structures and vary across various push message types.
PushMessage exposes the message type and access to its content as List<Object>. Bulk content can be decoded into more specific data types.

Kotlin Coroutine API

Kotlin users can now use a Coroutine API that exposes suspended methods and uses Flow where appropriate.
The API can be obtained through an extension function for Standalone, Cluster, and Sentinel connections exposing the appropriate API.

val api: RedisCoroutinesCommands<String, String> = connection.coroutines()

val foo1 = api.set("foo", "bar")
val foo2 = api.keys("fo*")

Additionally, we ship two extensions that simplify transactional usage by providing a multi closure:

val result: TransactionResult = connection.async().multi {
    set("foo", "bar")
    get("foo")
}

The API is marked experimental and requires opt-in through @ExperimentalLettuceCoroutinesApi to avoid Compiler warnings.
We expect further evolution of the API towards a more Flow-oriented API where now List is returned to enable streaming of large responses.

Redesign command latency metrics publishing

This is a mostly internal change that switches from CommandLatencyCollector to the newly introduced CommandLatencyRecorder interface.
Unless you're implementing or configuring CommandLatencyCollector yourself, you should not see any changes.

This change is motivated by support for libraries that provide latency observability without publishing metrics to the EventBus.

API cleanups/Breaking Changes

With this release, we took the opportunity to introduce a series of changes that put the API into a cleaner shape.

Redesign command latency metrics publishing #1409
Remove JavaRuntime class and move LettuceStrings to internal package #1329
Remove Spring support classes #1358
Replace io.lettuce.core.resource.Futures utility with Netty's PromiseCombiner #1283
XGROUP DELCONSUMER should return pending message count #1377 (xgroupDelconsumer(…) now returns Long)
Change hgetall return type from Mono to Flux #1434
Script Commands: eval, digest, scriptLoad methods now only accept String and byte[] argument types. Previously digest and scriptLoad accepted the script contents as Codec value type which caused issues especially when marshalling values using JSON or Java Serialization. The script charset can be configured via ClientOptions (ClientOptions.builder().scriptCharset(StandardCharsets.US_ASCII).build();), defaulting to UTF-8.
Connection: Removal of deprecated timeout methods accepting TimeUnit. Use methods accepting Duration instead.
Async Commands: RedisAsyncCommands.select(…) and .auth(…) methods return now futures instead if being blocking methods.
Asynchronous API Usage: Connection and Queue failures now no longer throw an exception but properly associate the failure with the Future handle.
Master/Replica API: Move implementation classes from io.lettuce.core.masterslave to io.lettuce.core.masterreplica package.
Internal: Removal of the internal LettuceCharsets utility class.
Internal: Reduced visibility of several protected fields in AbstractRedisClient (eventLoopGroups, genericWorkerPool, timer, clientResources, clientOptions, defaultTimeout).
Internal: Consolidation of Future synchronization utilities (LettuceFutures, RefreshFutures, Futures).
Deprecate reactive StreamChannel methods #1434
Rename StatefulRedisConnection.suspendable() to .coroutines() and command interfaces accordingly #1436

Commands

Add support for STRALGO #1280
Add support for LPOS #1320

Enhancements

Use channel thread to enqueue commands #617
Redesign connection activation #697
Add support for RESP3 #964
Consolidate Future utils #1039
Make RedisAsyncCommands.select() and auth() async #1118 (Thanks to @ikkyuland)
Allow client to pick a specific TLS version and introduce PEM-based configuration #1167 (Thanks to @amohtashami12307)
Optimization of BITFIELD args generation #1175 (Thanks to @ianpojman)
Add mutate() to SocketOptions #1193
Add CLIENT ID command #1197
Lettuce not able to reconnect automatically to SSL+authenticated ElastiCache node #1201 (Thanks to @chadlwilson)
Add support for AUTH with user + password introduced in Redis 6 #1202 (Thanks to @tgrall)
HMSET deprecated in version 4.0.0 #1217 (Thanks to @hodur)
Allow selection of Heap or Direct buffers for CommandHandler.buffer #1223 (Thanks to @dantheperson)
Use domain specific value object as return type for xpending. #1229 (Thanks to @christophstrobl)
Support JUSTID flag of XCLAIM command #1233 (Thanks to @christophstrobl)
Add support for KEEPTTL with SET #1234
Add support for RxJava 3 #1235
Retrieve username from URI when RedisURI is built from URL #1242 (Thanks to @gkorland)
Introduce ThreadFactoryProvider to DefaultEventLoopGroupProvider for easier customization #1243 (Thanks to @apilling6317)
Add template method for EventLoopGroup creation #1273 (Thanks to @konstantin-grits)
Add support for Client-side caching #1281
Registration of push message listeners #1284
Add charset option to ScanArgs.match(…) #1285 (Thanks to @gejun123456)
Allow for more customisation of the tracing span #1303 (Thanks to @JaidenAshmore)
Support for GraalVM Native Images #1316 (Thanks to @ilopmar)
Consider topology updates for default Cluster connections #1317 (Thanks to @be-hase)
SSL handshake doesn't respect timeouts #1326 (Thanks to @feliperuiz)
Reduce RedisStateMachine bytecode size #1332 (Thanks to @hellyguo)
Feature request: add a cluster-capable version of flushallAsync #1359 (Thanks to @jchambers)
BoundedAsyncPool object is ready to be manipulated with even though a connection is not created yet #1363 (Thanks to @little-fish)
Introduce DecodeBufferPolicy to reduce memory usage #1314 (Thanks to @Shaphan)
Kotlin Coroutine API #1387 (Thanks to @sokomishalov)
Add support for aarch64 #1396 (Thanks to @odidev)

Fixes

Commands Timeout ignored/not working during refresh #1107 (Thanks to @pendula95)
StackOverflowError in RedisPublisher #1140 (Thanks to @csunwold)
Incorrect access on io.lettuce.core.ReadFrom.isOrderSensitive() #1145 (Thanks to @orclev)
Consider ReadFrom.isOrderSensitive() in cluster scan command #1146
Improve log message for nodes that cannot be reached during reconnect/topology refresh #1152 (Thanks to @drewcsillag)
BoundedAsyncPool doesn't work with a negative maxTotal #1181 (Thanks to @sguillope)
TLS setup fails to a master reported by sentinel #1209 (Thanks to @ae6rt)
Lettuce metrics creates lots of long arrays, and gives out of memory error. #1210 (Thanks to @omjego)
CommandSegments.StringCommandType does not implement hashCode()/equals() #1211
Unclear documentation about quiet time for RedisClient#shutdown #1212 (Thanks to @LychakGalina)
StreamReadOutput in Lettuce 6 creates body entries containing the stream id #1216
Write race condition while migrating/importing a slot #1218 (Thanks to @phyok)
randomkey return V not K #1240 (Thanks to @hosunrise)
ConcurrentModificationException iterating over partitions #1252 (Thanks to @johnny-costanzo)
Replayed activation commands may fail because of their execution sequence #1255 (Thanks to @robertvazan)
Fix infinite command timeout #1260
Connection leak using pingBeforeActivateConnection when PING fails #1262 (Thanks to @johnny-costanzo)
Lettuce blocked when connecting to Redis #1269 (Thanks to @jbyjby1)
Stream commands are not considered for ReadOnly routing #1271 (Thanks to @redviper)
Write race condition while migrating/importing a slot #1218 (Thanks to @phyok)
PauseDetector acquisition hang in DefaultCommandLatencyCollector #1300 (Thanks to @ackerL)
NullPointerException thrown during AbstractRedisAsyncCommands.flushCommands #1301 (Thanks to @mruki)
xpending(K, Consumer, Range, Limit) fails with ERR syntax error using Limit.unlimited() #1302 (Thanks to @nagaran1)
Remove duplicated command on asking #1304 (Thanks to @koisyu)
ArrayOutput stops response parsing on empty nested arrays #1327 (Thanks to @TheCycoONE)
Synchronous dispatch of MULTI returns null #1335 (Thanks to @tzxyz)
RedisAdvancedClusterAsyncCommandsImpl scriptKill is incorrectly calling scriptFlush #1340 (Thanks to @azhukayak)
RedisAdvancedClusterAsyncCommands.scriptKill now calls scriptKill instead of scriptFlush #1341 (Thanks to @dengliming)
Lingering topology refresh connections when using dynamic refresh sources #1342 (Thanks to @tpf1994)
Wrong cast in StringCodec may lead to IndexOutOfBoundsException #1367 (Thanks to @dmandalidis)
xpending(key, group) fails without pending messages #1378
Sentinel authentication failed when using the pingBeforeActivateConnection parameter #1401 (Thanks to @viniciusxyz)
RedisURI.toString() should not reveal password #1405
MasterReplica.connect(…) doesn't consider username with Redis 6 #1406
LPOS command sends FIRST instead of RANK #1410 (Thanks to @christophstrobl)
Fix **/ for closing comments #1416
Correctly report isDone if Command completed with completeExceptionally #1433

Other

Refactor script content argument types to String and byte[] instead of V (value type) #1010 (Thanks to @danielsomekh)
Render Redis.toString() to a Redis URI #1040
Pass Allocator as RedisStateMachine constructor argument #1053
Simplify condition to invoke "resolveCodec" method in AnnotationRedisCodecResolver #1149 (Thanks to @machi1990)
Encode database in RedisURI in path when possible #1155
Remove LettuceCharsets #1156
Move SocketAddress resolution from RedisURI to SocketAddressResolver #1157
Remove deprecated timeout methods accepting TimeUnit #1158
Upgrade to RxJava 2.2.13 #1162
Add ByteBuf.touch(…) to aid buffer leak investigation #1164
Add warning log if MasterReplica(…, Iterable) contains multiple Sentinel URIs #1165
Adapt GEOHASH tests to 10 chars #1196
Migrate Master/Replica support to the appropriate package #1199
Disable RedisURIBuilderUnitTests failing on Windows OS #1204 (Thanks to @kshchepanovskyi)
Provide a default port(DEFAULT_REDIS_PORT) to RedisURI's Builder #1205 (Thanks to @hepin1989)
Update code for pub/sub to listen on the stateful connection object. #1207 (Thanks to @judepereira)
Un-deprecate ClientOptions.pingBeforeActivateConnection #1208
Use consistently a shutdown timeout of 2 seconds in all AbstractRedisClient.shutdown methods #1214
Upgrade dependencies (netty to 4.1.49.Final) #1161, #1224, #1225, #1239, #1259
RedisURI class does not parse password when using redis-sentinel #1232 (Thanks to @kyrogue)
Reduce log level to DEBUG for native library logging #1238 (Thanks to @DevJoey)
Reduce visibility of fields in AbstractRedisClient #1241
Upgrade to stunnel 5.56 #1246
Add build profiles for multiple Java versions #1247
Replace outdated Sonatype parent POM with plugin definitions #1258
Upgrade to RxJava 3.0.2 #1261
Enable Sentinel tests after Redis fixes RESP3 handshake #1266
Consolidate exception translation and bubbling #1275
Reduce min thread count to 2 #1278
Upgrade dependencies #1305
Add FAQ section to reference docs #1307
Rename master branch to main #1308
Consistently use Javadoc wording in BoundedPoolConfig.Builder #1337 (Thanks to @maestroua)
Upgrade to Reactor Core 3.3.8.RELEASE #1353
Upgrade to netty 4.1.51.Final #1354
Consistently translate execution exceptions #1370
Upgrade to RxJava 3.0.5 #1374
Upgrade to Commons Pool 2.8.1 #1375
Upgrade to Reactor 3.3.9.RELEASE #1384
Upgrade to Project Reactor 3.3.10.RELEASE #1411
Upgrade to netty 4.1.52.Final #1412
Upgrade test/optional dependencies #1413
Explicit AWS and AZURE compatibility on README #1425 (Thanks to @raphaelauv)
Revisit synchronized blocks #1429
Upgrade dependencies #1431