apache/pulsar v2.8.4

on GitHub

Broker

• [improve][broker] Do not print the log that the topic does not exist #13535
• [fix][broker] Fix NPE when nextValidLedger is null #13975
• [fix][broker] Fix NPE when ReplicatedSubscriptionsController sends marker message with deduplication enabled #14017
• [fix][broker] Fix avg-messagePerEntry metrics for consumer #14330
• [improve][broker] Validate rack name when setting the bookie rack info #14336
• [improve][broker] Optimize find nics process. #14340
• [fix][broker] Fix NPE when subscription is already removed #14363
• [fix][broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions #14367
• [improve][broker] Optimize RawReader#create when using Compactor #14447
• [fix][broker] Fix the latest message of the compacted topic cannot be read #14449
• [fix][broker] Fix producerFuture not completed in ServerCnx#handleProducer #14467
• [fix][broker] Fix NPW in ManagedLedgerImpl #14481
• [improve][broker] Support shrink in ConcurrentLongHashMap #14497
• [fix][broker] Fixed wrong behaviour caused by not cleaning up topic policy service state. #14503
• [improve][broker] Optimize memory usage: support to shrink for pendingAcks map #14515
• [fix][broker] Cancel offload tasks when managed ledger closed #14744
• [improve][broker] Reduce unnecessary expansions for ConcurrentLong map and set #14562
• [fix][broker] Fix metadata store deadlock when checking BacklogQuota #14634
• [improve][broker] Support shrink for map or set #14663
• [fix][broker] Fix lost message issue due to ledger rollover #14664
• [fix][broker] Set splitNamespaceBundle with readonly=false #14680
• [fix][broker] Fixed duplicated delayed messages when all consumers disconnect #14740
• [fix][broker] Fix wrong state for non-durable cursor #14869
• [fix][broker] Improve error logging for topic not found #14892
• [fix][broker] Fix topic policy reader close bug #14897
• [fix][broker] Return if reset in progress #14978
• [fix][broker] Fix potential NPE in Replicator #15003
• [fix][broker] Fix race condition in updating lastMarkDeleteEntry field #15031
• [fix][broker] Fix potential to add duplicated consumer #16826
• [fix][broker] Follow up on race condition fixes in ManagedCursorImpl #15031 #15067
• [cleanup] [broker] Remove useless code to avoid confusion in OpReadEntry#checkReadCompletion #15104
• [fix][broker] Cancel fencedTopicMonitoringTask when topic closed normally. #15202
• [fix][broker] Fix MessageDeduplication#inactiveProducers may not be persistence correctly #15206
• [fix][broker] Fix race condition between timeout and completion in OpAddEntry #15233
• [fix][broker] Fix problem at RateLimiter#tryAcquire #15306
• [improve][broker] Use shrink map for message redelivery. #15342
• [improve][broker] Support shrink for ConcurrentSortedLongPairSet #15354
• [improve][broker] Close connection if a ping or pong message cannot be sent #15382
• [cleanup][broker] Fix typo in enum name and handle closing of the channel properly since writeAndFlush is asynchronous #15384
• [fix][broker] Avoid heartbeat topic to offload. #15393
• [fix][broker] Fix deadlock in broker after race condition in topic creation failure #15570
• [fix][broker] fix calculation in getNumberOfEntriesInStorage #15627
• [fix][broker] Fix NPE when set AutoTopicCreationOverride #15653
• [fix][broker] Fix creating producer failure when set backlog quota #15663
• [fix][broker] Fix NPE when put value to RangeCache #15707
• [fix][broker] Fix can not enable system topic if AutoUpdateSchemaEnabled=false #15759
• [fix][broker] Fix wrong unit of NIC speed on Linux #15770
• [fix][broker] Fix NPE in MessageDeduplication #15820
• [fix][broker] Fix NPE when ledger id not found in OpReadEntry #15837
• [fix][broker] When skipping updating mark delete position, execute callback with executor to prevent deadlock #15971
• [fix][broker] Fix NPE when drop backlog for time limit. #16235
• [fix][broker]Fix getInternalStats occasional lack of LeaderInfo again #16238
• [fix][broker] Fix RawReader out of order #16390
• [fix][broker] Skip reading more entries for a pending read with no more entries #16400
• [fix][broker] Fix No such ledger exception #16420
• [fix][broker] Fix RawReader hasMessageAvailable returns true when no messages #16443
• [fix][broker] Fix stats-internal with option -m cause active ledger recover then close #16662
• [fix][broker] Fix passing incorrect authentication data #16840
• [fix][broker] ManagedCursor: mark delete no callback when create meta-ledger fail #16841
• [improve][broker] Avoid reconnection when a partitioned topic was created concurrently #16867
• [improve][broker] Recycle OpReadEntry in some corner cases #16869
• [fix][broker] Fix compaction subscription acknowledge Marker msg issue. #16918

Security

• [fix][sec] Use provider's canLookupAsync method for AuthorizationService #11777
• [improve][sec] Optimize the logic of allowing namespace operation #13090
• [improve][sec] Full-support set SSL provider, ciphers and protocols #13740
• [fix][sec] Role with namespace produce authz can also get topics #13773
• [improve][sec] Full-support SSL provider, ciphers, and protocols for broker service and proxy service #15034
• [improve][sec] Add load multiple certificates support #14798
• [fix][sec] Fix handling single role and non-jwt-token in MultiRolesTokenAuthorizationProvider #14857
• [fix][sec] Use tlsCertRefreshCheckDurationSec instead of 0 for refresh value #15075
• [improve][sec] Skip unnecessary DNS resolution when creating AuthenticationDataHttp instance #15221
• [improve][sec] Improve skipping of DNS resolution when creating AuthenticationDataHttp instance #15228
• [fix][sec] Fix MultiRolesTokenAuthorizationProvider authorize issue #15454
• [fix][sec] Fix grant all permissions but can't list topic #15501
• [improve][sec] Switch to rely on Netty for Hostname Verification #15824
• [fix][sec] Fix create client with TLS config #16014
• [fix][sec] Avoid AuthenticationDataSource mutation for subscription name #16065
• [improve][sec] Improve get the basic authentication config #16526
• [fix][sec] Fix multiple roles authorization #16645
• [fix][sec] Add timeout of sync methods and avoid call sync method for AuthoriationService (#15694) #16831
• [improve][sec] Support for get token from HTTP params #16871
• [improve][sec] Adapt basic authentication configuration with prefix #16935

Admin

• [improve][admin] Add corresponding get command for namespace #12322
• [fix][admin] Fix deleting namespace will fail when system topics exist #15040
• [fix][admin] Fix validateGlobalNamespaceOwnership wrap exception issue #14612
• [fix][admin] Fix wrong prompt exception when getting the non-persistent topic list without GET_BUDNLE permission #14638
• [fix][admin] Fixed 404 error msg not returned correctly using http lookup #14677
• [improve][admin] Provide an accurate error message when set autoTopicCreation #14684
• [fix][admin] Fix cannot delete namespace with system topic #14730
• [fix][admin] Fix NPE in PulsarAdminBuilder when the service is not set #14769
• [fix][admin] Fix missing response type in swagger definitions #16022

Function

• [fix][fn] Fixing get functions for output topic and serde classname #14103
• [improve][fn] Pass configured metricsPort to k8s runtime #14502
• [fix][fn] fix some IOExceptions when create functions from package URL #14553
• [fix][fn] handle NPE when getLeader returns null #15058
• [improve][fn] Add KeyStore support in WebSocket, Function Worker HTTPS Servers #15084
• [fix][function] Check executor null when close the FileSource #15247
• [improve][fn] Provide default error handler for function log appender #15728
• [fix][function] Fix error when user starts with the pulsar functions local runner #16565
• [fix][function] Fix python instance not process zip file correctly #16697

IO

• [fix][io] throw exceptions when Kafka offset backing store failed to start #14491
• [fix][io] Handle Kafka sinks that return immutable maps as configs #14780

SQL

• [fix][sql] Fix PulsarRecordCursor deserialize issue. #14615
• [fix][sql] Fix messageQueue release message issue. #16155

Offloader

• [fix][offloader] Fix incorrect entryId in warn log when reading entry from tiered storage #14685
• [improve] TieredStorage: add debug information #14907
• [feat][offloader] Add pure S3 provider for the offloader #15710
• [fix][offloader] Fix setManagedLedgerOffloadedReadPriority not work. #16436

Proxy

• [improve][proxy] Log warning when opening connection to broker fails #14710
• [refactor][proxy] Refactor Proxy code and fix connection stalling by switching to auto read mode #14713
• [improve][proxy] Configure Netty DNS resolver to match JDK DNS caching setting, share DNS resolver instance in Proxy #15219
• [fix][proxy] Fix proxy connection leak when inbound connection closes while connecting is in progress #15366
• [fix][proxy] Fix DNS server denial-of-service issue when DNS entry expires #15403
• [improve][proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler #15415
• [fix][proxy] Do not preserve host when forwarding admin requests. #16342
• [fix][proxy] Fix client service URL #16834

Monitor

• [improve][monitor]: add metrics for pulsar web service thread pool #14742
• [improve][monitor] add message ack rate metric for consumer #15674
• [fix][monitor] Fix bug: fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true #16343
• [improve][monitor] Expose topic level storage write and read rate metrics #16855

CLI

• [fix][cli] Remove the trust certs check #14764

Test & Other

• [improve][test] Use Awaitility to replace Thread.sleep for pulsar-zookeeper-utils module. #11017
• [fix][test] Fix flaky test in PersistentStreamingDispatcherBlockConsumerTest #12943
• [fix][build] Fix NoClassDefFoundError: com/google/inject/AbstractModule in pulsar-io/batch-data-generator and Jcloud offloader #14150
• [fix][test] Fix flaky AdminApiSchemaTest#testSchemaInfoApi #14508
• [fix][test] Fix NamespacesTest execution order #14552
• [fix][test] Fix AdminApiTest.testNamespaceSplitBundleConcurrent #14565
• [fix][test] Fix flaky test MemoryLimitTest#testRejectMessages (#14220) #14628
• [fix][ci] Upgrade zlib version to 1.2.12 #14964
• [improve][build] Ignore case when obfuscating passwords in configuration scripts #15077
• [fix][build] Fix LICENSE files for branch-2.8 #15261
• [Improve][doc] Add config of IO and acceptor threads in proxy #15340
• [improve][build] Configure DLog Bookie, Pulsar, and Admin clients via pass-through config #15818
• [improve][build] Allow pulsar_tool_env.sh PULSAR_MEM to be Overridden #15868
• [fix][test] TieredStorageConfigurationTests - clear system properties #15957
• [fix][doc] Update/fix Swagger Annotation for param: authoritative #16222
• [improve][test] Verify the authentication data in the authorization provider #16900

Dependency Updates

• [improve][build] Upgrade snakeyaml version to 1.30 #13722
• [fix][build] Remove --illegal-access errors resulting from Google Guice (upgrade to 5.0.1 and JClouds to 2.4.0) #13810
• [fix][build] Remove --illegal-access errors resulting from Google Guice - Pulsar IO, Offloaders and Pulsar SQL - Bump Guice to 5.1.0 #14300
• [fix][build] Upgrade jackson and jackson-databind (2.13.2.1) to get rid of CVE-2020-36518 #14871
• [improve][build] Remove log4j for CVE-2022-23307 #15109
• [improve][build] Upgrade Netty to 4.1.76.Final, Netty Tcnative, grpc and protobuf #15212
• [improve][build] Use grpc-bom to align grpc library versions #15234
• [improve][build] Upgrade BookKeeper to 4.14.5 #15581
• [improve][build] Upgrade Netty to 4.1.77.Final and netty-tcnative to 2.0.52.Final #15646
• [improve][build] Upgrade JClouds to 2.5.0 [#15649](#15649
• [fix][build] Tiered storage: Upgrade Hadoop to 3.3.3 to get rid of CVE-2022-26612 #15660
• [fix][build] Upgrade to Jetty to 9.4.48.v20220622 to get rid of CVE-2022-2047 #16520
• [improve][broker] Upgrade log4j2 version to 2.18.0 (#16884) #16914

Java Client

• [improve][java] Improve consumer listener logic #13273
• [fix][java] Fix ConsumerBuilderImpl#subscribeAsync blocks calling thread #14614
• [fix][java] Fix partitionsAutoUpdateFuture never completes #14625
• [fix][java] Fix race condition in consumer redelivery #14687
• [improve][java] Process maxRedeliverCount is 0 of DeadLetterPolicy #14706
• [improve][java] Avoid timer task run before previous subscribe complete #14818
• [fix][java] Fix potentially unfinished CompletableFuture in doReconsumeLater #14947
• [fix][java] ConsumerBuilderImpl can not set null to deadLetterPolicy. #14980
• [improve][java] Add test to ensure the message order in listener callbacks #15049
• [fix][java] Fix performance regression with message listener #15162
• [fix][java] Fix wrong behavior of deduplication for key based batching #15413
• [improve][java] improve logic when ACK grouping tracker checks duplicated message id #15465
• [fix][java] Remove consumer when close consumer command is received #15761
• [fix][java] Fix conversion of TimestampMillisConversion has no effect when Jsr310Conversion is enabled #15863
• [fix][java] Remove producer when close producer command is received #16028
• [fix][java] Fix thread safety issue of LastCumulativeAck #16072
• [improve][java] Send CloseConsumer on client timeout #16616
• [fix][java] Fix ReconsumeLater will hang up if retryLetterProducer exception #16655
• [fix][java] Fix load trust certificate #16789
• [fix][java] Remove redundant check for chunked message TotalChunkMsgSize in ConsumerImpl #16797
• [fix][java] Forget to update memory usage when message is invalid #16835
• [fix][java] Fix PatternTopicsChangedListener blocked when topic removed #16842
• [fix][java] Fix MaxQueueSize semaphore release leak in createOpSendMsg #16915

C++ Client

• [improve][cpp] Expose getLastMessageId in the Reader API #11723
• [fix][cpp] Fix hasMessageAvailable returns wrong value for last message #13883
• [fix][cpp] Fix thread safety issue for multi topic consumer #14380
• [fix][cpp] Fix wrong unit of Access Token Response's expires_in field #14554
• [fix][cpp] Handle exception in creating socket when fd limit is reached #14587
• [cleanup][cpp] Add braces around initialization of subobject #14735
• [fix][cpp] Fix producer is never destructed until client is closed #14797
• [fix][cpp] Fix flaky tests about reference count #14854
• [fix][cpp] Fix segmentation fault when creating socket failed #14834
• [fix][cpp] Fix the race condition of connect timeout task #14823
• [fix][cpp] Fix send callback might not be invoked in key-based batching #14898
• [fix][cpp] Fix connection is not closed when broker closes the connection to proxy #15009
• [fix][cpp] Fix single message metadata not set correctly #15072
• [fix][cpp] Fix UnknownError might be returned for a partitioned producer #15161
• [improve][cpp] Remove the flaky and meaningless tests #15271
• [improve][cpp] Wait until event loop terminates when closing the Client #15316
• [fix][cpp] Avoid race condition causing double callback on close #15508
• [fix][cpp] Generate correct well-known OpenID configuration URL #15928
• [fix][cpp] Rename function name: pulsar_producer_configuration_set_crypto_failure_action #16031
• [cleanup][cpp] Clean up C++ client curl configuration #16064
• [fix][cpp] Fix the close of Client might stuck or return a wrong result #16285
• [improve][cpp] Fix flaky C++ ClientTest.testWrongListener #16510

Python Client

• [fix][python] Fix generated Python protobuf code not compatible with latest protobuf package #15846

WebSocket Client

• [fix][ws] Fix MultiTopicReader#getConsumer ClassCastException 15534