apache/pulsar v2.9.2

on GitHub

Security

• Getting rid of CVEs in Solr connector #13822
• Getting rid of CVEs in InfluxDB connector #13821
• Getting rid of CVEs in batch-data-generator #13820
• Getting rid of CVEs brought in with aerospike #13819
• [owasp] Suppress false positive Avro CVE-2021-43045 #13764
• [security] Upgrade protobuf to 3.16.1 to address CVE-2021-22569 #13695
• [security] Upgrade Jackson to 2.12.6 #13694
• [security] Upgrade Log4j to 2.17.1 to address CVE-2021-44832 #13552
• [security] Upgrade to Log4J 2.17.0 to mitigate CVE-2021-45105 #13392
• Cipher params not work in KeyStoreSSLContext #13322
• Use JDK default security provider when Conscrypt isn't available #12938
• [authorization] Return if namespace policies are read only #12514
• [security] Upgrade Postgre driver to 42.2.25 to get rid of CVE-2022-21724 #14119
• Bump slf4j from 1.7.25 to 1.7.32 #13595
• [security] Upgrade Netty to 4.1.72 - CVE-2021-43797 #13328

Broker

• Fix lost message issue due to ledger rollover. #14664
• [tests] Fix thread leak in MLTransactionMetadataStore #14524
• Fix npe in ManagedLedgerImpl #14481
• [broker] Fix Future.join() causing deadlock. #14469
• [broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions #14367
• Validate rack name #14336
• [pulsar-broker] Fix avg-messagePerEntry metrics for consumer #14330
• Fix the wrong parameter in the log. #14309
• Fix batch ack count is a negative issue. #14288
• Bug fix: IllegalArgumentException: Invalid period 0.0 to calculate rate #14280
• Clean up individually deleted messages before the mark-delete position #14261
• If mark-delete operation fails, mark the cursor as "dirty" #14256
• Fixed detecting number of NICs in EC2 #14252
• Remove log unacked msg. #14246
• [broker] Change broker producer fence log level #14196
• [issue 14105] Avoid creating any topics in NamespaceService#checkTopicExists during topic lookup. #14134
• Fix NPE of cumulative ack mode and incorrect unack message count #14021
• Fixes NPE - ReplicatedSubscriptionsController send marker message when enable deduplicated. #14017
• KeyShared stickyHashRange subscription: prevent stuck subscription in case of consumer restart #14014
• Add null check to workaround NPE in unit tests with Mockito/PowerMock #14006
• Fixed handling of consumers with equal names on on key shared selector with consistent hashing #13991
• Trim configuration value string which contains blank prefix or suffix #13984
• [broker] WaitingCursors potential heap memory leak #13939
• [broker] Fix read schema compatibility strategy priority #13938
• [branch-2.9] Fix SystemTopicBasedTopicPoliciesService NPE issue. #13840
• NPE when get isAllowAutoUploadSchema #13831
• [broker] Fix topic produced through REST not support Authorization #13771
• Fix call sync method in async rest API for internalGetSubscriptionsForNonPartitionedTopic #13745
• Fix the deadlock while using zookeeper thread to create ledger #13744
• [broker] Fix inefficient forEach loop #13742
• [issuse 13640][broker] Fix non persistent topic subscription error. #13685
• Fix invalid rack name cause bookie join rack failed #13683
• Release old bundle from ownership cache when operator split bundle #13678
• Avoid call sync method in async rest API for force delete subscription #13668
• Avoid call sync method in async rest API for delete subscription #13666
• Fix getInternalStats occasional lack of LeaderInfo #13656
• [issue 13479] Fixed internal topic effect by InactiveTopicPolicy. #13611
• Fix bug :Infinity value for CPU or Bandwidth usage #13609
• Change ContextClassLoader to NarClassLoader in BrokerInterceptor #13589
• Fix NPE when unloading namespace bundle #13571
• [issue #13439][backlogQuota] Allow both limit and limitsize be null #13557
• [pulsar-broker] The log prints NamespaceService#isServiceUnitActive exception stack information. #13553
• Update log content #13540
• Healthcheck v2 failed #13525
• Change ContextClassLoader to NarClassLoader in AdditionalServlet #13501
• Optimize the debug log that affects performance, and unify the style #13498
• Fix(Auth): Fix multi roles authz cannot handle empty roles case #13477
• Fixes the NPE in system topics policies service #13469
• [broker] Fix race conditions in closing producers and consumers #13428
• [issue 12726] [broker] Fix deadlock in metadata-store callback thread for branch 2.9 #13426
• [broker] Remove check resource when delete failure domain #13421
• [broker] Fix create the dynamic configuration resource if not exist #13420
• Fix batch message ack does not decrease the unacked-msg count. #13383
• [issue 13194][pulsar-broker] Fix dead loop in BacklogQuotaManager.dropBacklogForTimeLimit #13249
• [broker] Modify return result of NamespacesBase#internalGetPublishRate #13237
• [broker] Optimize ManagedLedger Ledger Ownership Check #13222
• Fix shedding heartbeat ns #13208
• Fix when deleting topic with NotFoundException, do not return to client. #13203
• [pulsar-broker] Clean up active consumer on already closed connection #13196
• Update cursor last active timestamp when reseting cursor #13166
• Use current resourceUsage value as historyUsage when leader change in ThresholdShedder #13136
• [broker] Don't attempt to delete pending ack store unless transactions are enabled #13041
• [issue 13004] Fix race condition in ResourceLockImpl#revalidate #13006
• [broker] Fix NPE in PersistentTopic.checkSubscriptionTypesEnable #12961
• Fix wrong isEmpty method of ConcurrentOpenLongPairRangeSet #12953
• [broker] Correct param of delete method for v1 topic #12936
• [broker] Change create topic return error to Status.BAD_REQUEST #12919
• [broker] Fix topic policy listener deleted by mistake. #12904
• Remove readerCaches and close reader when exception occurs in SystemTopicBasedTopicPoliciesService #12873
• Fix deleting tenants with active namespaces with 500. #12848
• [broker] Bug Fix: topic policy is not properly init if namespace is loaded first. #12833
• [issue 12757][broker] Add broker config isAllowAutoUpdateSchema #12786
• Fix TopicPoliciesCacheNotInitException issue. #12773
• The problem of two exception handling #12744
• [pulsar-broker] Handle lookup redirect for V1-topics with different cluster #12743
• [issue 12723] Fix race condition in PersistentTopic#addReplicationCluster #12729
• Remove catching NamingException. #12725
• Even if always compatible is set, Consumers cannot be created #12721
• Fix znode leakage caused by deleting tenant #12711
• Remove unused listeners if it have no listeners. #12654
• [broker] Optimize exception information for schemas #12647
• Support retry when creating reader of Topic Policies #12622
• Fix String should use equals but not ==. #12619
• Fix 12614, waitingForPingResponse needs to be modified with volatile for concurrent sence #12615
• [test] Cleanup ProxyPublishConsumeTest #12607
• Fix additional servlets nar might extract to null directory #12585
• Fix log typo in NamespaceService#checkHeartbeatNamespace #12582
• Clean up the metadata of the non-persistent partitioned topics. #12550
• [pulsar-broker] Add git branch information for PulsarVersion #12541
• Websocket should pass the encryption context to the consumers #12539
• The count of topics on the bundle is less than 2, skip split #12527
• Remove redundant null check for getInternalListener #12474
• Optimize the code: remove extra spaces #12470
• Fix wrong property name in NamespaceIsolationDataImpl#secondary #12433
• Fix the null point caused by deleting the system topic policy #12367
• Future completed twice in the method of impl.MLPendingAckStore#closeAsync #12362
• Fix the race of delete subscription and delete topic #12240
• Fix delete authentication policies when delete topic. #12215
• Fix update ledger list to znode version mismatch failed, ledger not delete #12015
• Fix issues 11964, deadlock bug when use key_shared mode #11965
• Fix validateGlobalNamespaceOwnership wrap exception issue. #14269
• Fix reader skipped remaining compacted data during the topic unloading. #13629
• Return message ID from compacted ledger while the compaction cursor reach the end of the topic #13533
• [broker] Fix getting the last message-id from an empty compact ledger #13476
• Fix lost compaction data due to compaction properties missed during reset-cursor #12698
• [Compaction] Do not move the non-durable cursor position when trimming ledgers while topic with compaction #12602
• Fix can't read the latest message of the compacted topic #14449
• Fix skips compacted data for reader/consumer #12464
• [broker] Fixed wrong behaviour caused by not cleaning up topic policy service state. #14503
• [broker] Avoid thread deadlock problem when creating topic policy reader #13837
• [broker] Bug Fix: topic policy is not properly init if namespace is loaded first. #13422
• [schema] Fix parse BigDecimal #14019
• Fix: bug when allAll bucket #13467
• [authorization] Support GET_BACKLOG_SIZE topic op after enable auth #12850
• Reduce the readFailureBackoff time #12444
• [config] Add readWorkerThreadsThrottlingEnabled to conf/bookkeeper.conf #12666

Transaction

• [transaction] Fix cursor readPosition is bigger than maxPosition in OpReadEntry #14667
• Fix MLTransactionMetadataStore.update async fail #14532
• [transaction] Fix end transaction at state of timeout #14370
• [transaction] Optimize testEndTBRecoveringWhenManagerLedgerDisReadable #14303
• [transaction] delete changeMaxReadPositionAndAddAbortTimes when checkIfNoSnapshot #14276
• [transaction] Adopt single thread pool in TC #14238
• [transaction] Fix send normal message can't change MaxReadPosition #14192
• [transaction] Optimize transaction timeout #14172
• [transaction] Fix subscription ack transaction marker. #14170
• [client][transaction] Fix OpBase.callback is not called in TransactionMetaStoreHandler#failPendingRequest #14133
• Fixed deadlock on txn semaphore permit exhaustion #14131
• Fix unack message count for transaction Ack while disabled batch index ack #14071
• [transaction] Fix individual ack with transaction decrease unAckMessageCounnt #14020
• [transaction] Resolve the performance bottleneck of TransactionBufferHandle #13988
• [transaction] RetryException should not be return #13828
• [transaction] Fix recover of TransactionBuffer #13739
• [transaction] Duplicate TYPE of Prometheus metrics #13704
• Remove txn add partition info log #13670
• Fix no response to client when handleSubscribe because PendingAckHandleImpl init fail #13655
• [transaction] PendingAckHandleImpl handle isInCacheRequest #13481
• [branch-2.9] [transaction] Stop TB recovering with exception #13425
• Optimize transaction FieldUpdater to static final #13396
• [transaction] Delete the redundant code #13327
• [transaction] Allow transaction to be committed or aborted in the state of aborting or committing. #13323
• [transaction] Remove request if can not send #13308
• [transaction] Fix generate transactionId some comments #13234
• [transaction] Fix transaction sequenceId generate error. #13209
• [transaction] No TransactionCoordinatorNotFound, but automatic reconnect #13135
• [transaction] Add a check for uninitialized PendingAck #13088
• Handle exception double #12881
• [transaction] Stop TP replaying with Exception #12700
• [transaction] Stop TB recovering with exception #12636
• [transaction] Txn client check timeout #12521
• [transaction] Fix maxReadPosition with normal publish #12386
• [transaction] Fix bugs, Exception thrower by TB::appendBufferToTxn must be ManagedLedgerException. #12376
• [transaction] Merge transactionBuffer exception into a class #12358
• [transaction] Add transaction perf #11933
• [transaction] Adopt Single_thread to handle TcClient connecting #13969
• [transaction] Stop TC replaying with exception #12705
• [transaction] Fix performance #13253

Admin

• [branch-2.9] Fix Broker HealthCheck Endpoint Exposes Race Conditions. #14658
• [branch-2.9] Fix delete namespace issue. #14657
• Make sure policies.is_allow_auto_update_schema not null #14409
• Fix delete namespace issue. #14215
• [issue 13551] FIxed NPE when reset-cursor at a non-existent topic (RestException without cause) #13573
• [transaction] Add transaction admin #13447
• [pulsar-admin] Print topic internal info as formatted json #12709
• [pulsar-admin] Add --all option to get all version schema of topic #12535
• [pulsar-admin] Modify exception of set-properties for namespace #12436
• [admin] Get schema validation enforce add applied. #12349
• [pulsar-admin] Add corresponding get command for namespace #12322
• Pulsar admin exposes secret for source and sink #13059
• [pulsar-admin] Perfect judgment conditions of pulsar-admin #12315

Proxy

• [proxy] Fix port exhaustion and connection issues in Pulsar Proxy #14078
• Allow config of IO and acceptor threads in proxy #14054
• [proxy] Prevent leak of unreleased lookupRequestSemaphore permits #13812
• [proxy] Fix issue when Proxy fails to start and logs about an uncaught exception #13171
• Add error log when new jetty client exception #12840
• [proxy] Remove unnecessary Pulsar Client usage from Pulsar Proxy #13836

Client

• Fix ConsumerBuilderImpl#subscribeAsync blocks calling thread. #14433
• Fix adding message to list potential issue #14377
• [pulsar-client] Fix send to deadLetterTopic not working when reach maxRedeliverCount #14317
• [client] Fix time unit mismatch in errMsg when producer send fails. #14299
• Fix PersistentAcknowledgmentsGroupingTracker set BitSet issue. #14260
• Check getTlsTrustStorePath NPE when user forget to set it. #14253
• [issue 12262][client] Fix consume failure when BatchReceivePolicy#maxNumBytes < message size #14139
• Use scheduleWithFixedDelay instead of scheduleAtFixedRate for java producer batch timer #14125
• Add a default timeout for OAuth2 Metadata Resolver #14056
• [client] Fix send chunking message failed when ordering key is set. #13699
• [issue 13688][client] Fix send chunking message failed when encryption enabled #13689
• Fixed Producer semaphore permit release issue #13682
• [pulsar-client] Fix the wrong multi-topic has message available behavior #13634
• Use PulsarByteBufAllocator to allocate buffer for chunks #13536
• Fix resources leak when create producer failed #13505
• Fix semaphore and memory leak when chunks failed to enqueue #13454
• Fix invalid setting of enabled ciphers to fix warning from BoringSSL #13435
• [pulsar-client] Fix multi topic reader has message available behavior #13332
• [java client] Improve consumer listener logic #13273
• [client] Fix memory leak when message payload processor is configured #13233
• [producer] Change the time units from ns to ms #13057
• Use sendAsync instead of send when produce message to retry topic. #12946
• [java client] Avoid IllegalStateException in ClientCnx debug logs #12899
• [pulsar-client] Fix pending queue-size stats for batch messages #12704
• Fix invalid firstSentAt in log message when timeout first time #12588
• [pulsar-client] Add conf backoff values #12520
• Update producer stats when producer close #12500
• Fix a typo in UnAckedMessageTracker #12467
• Fix the retry topic's REAL_TOPIC & ORIGIN_MESSAGE_ID property #12451
• Change the producer fence error log to debug level #12447
• Add log error tracking for semaphore count leak #12410
• Fix race condition of OpSendMsgQueue when publishing messages #14231

Client C++

• [c++] Fix wrong unit of Access Token Response's expires_in field #14554
• [c++] Fix thread safety issue for multi topic consumer #14380
• [c++] Fix pulsar client cpp build fail in gcc-4.8.5 #14053
• [c++] Fix hasMessageAvailable returns wrong value for last message #13883
• [c++] Fix in macOS CMake might find error boost-python libs path #13193
• [c++] Fix libcurl miss auth header when broker return 307 #13112
• [c++] Define and expose PULSAR_VERSION macro #12769
• [C++] Fix request timeout for GetLastMessageId doesn't work #12586
• [capi] Support setting priority for consumers #12526
• [c++] Fixed connection read error logging #12492
• [c++] Use weak ref to ClientConnection for timeout task #12409
• [c++] Fix Version.h not found when CMake binary directory is customized #13324
• [c++] Fix GCC compilation failure caused by warning macro #14402
• Fix frequent segmentation fault of Python tests by refactoring ExecutorService #12427

Client Python

• Apply clang-format check for python wrapper #13418
• [python schema] Python schema support custom Avro configurations for Enum type #12642
• [issue #12486][python client] JsonSchema encoding is not idempotent #12490
• [issue #12485][python client] cannot use any values that evaluates to False #12489
• [python client] Python client support using custom Avro schema definition #12516

Client WebSocket

• [websocket] Fix ClassCastException when user create MultiTopicReader. #14316
• Add debug log for WebSocket. #12458

CleanUp

• [clean up] Remove unnecessary boxing unboxing #12790
• Optimize if statement #12741
• Remove unnecessary string operation #12719
• Remove redundant placeholders #12717
• Remove unused code in PersistentTopic.java #12715
• Remove unnecessary powermock annotation #12713
• [broker] Remove useless method "PersistentTopic#getPersistentTopic" #12655
• Optimization：remove unused isTxnMessage #12472
• Optimize SecurityUtility code flow #12431
• Remove redundant code #12424

Doc

• [doc] Fix doc for the wrong default value of maxPendingChunkedMessage #14144
• Update notice year. #13653
• Fix java doc for MultipleListenerValidator #12389

Pulsar SQL

• [pulsar sql] Fix PulsarRecordCursor deserialize issue. #14379
• [pulsar sql] support protobuf/timestamp #13287
• [pulsar sql] Handle message null schema version in PulsarRecordCursor #12809
• [pulsar sql] Support query chunked messages feature in Pulsar SQL #12720
• [pulsar sql] Pulsar SQL support query big entry data #12448
• Fix time field use error #12249

Dependency

• Bump netty version to 4.1.74.Final #14257
• Upgrade commons-cli to 1.5.0 #14094
• Upgrade Gson version 2.8.6 to 2.8.9 #13610
• [presto] Fix issue of mixed SLF4J library versions in presto libs #13603
• Remove the unused junit4 dependency in managed ledger #13514
• Upgrade to BookKeeper 4.14.3 #12760
• Some depdency in integration tests scope should be test #12696
• Upgrade debezium to 1.7.1 #12644
• Upgrade BK to 4.14.4 and Grpc to 1.42.1 #13714

Test

• [flaky-test] BacklogQuotaManagerTest#testProducerExceptionAndThenUnblockSizeQuota #14213
• [flaky-test] BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuotaWithEmptyLedger #13724
• [test] Fix flakiness issue when spying ServerCnx #13608
• Fix flaky test for BrokerServiceLookupTest.testModularLoadManagerSplitBundle #13159
• [issue 12883] Fix flaky caused by message receive timeout in testBlockBrokerDispatching #12954
• [issue 12883] Fix flaky test in PersistentStreamingDispatcherBlockConsumerTest #12943
• [issue 12897] Fix flaky test in testReplicatorProducerName #12898
• JavaInstanceTest should be AssertEquals #12836
• Flaky tests: ElasticSearchClientTests tests time out #12694
• Close Zk database on unit tests #12649
• [transaction] Optimize the tests extends from TransactionTestBase #12560
• Broker resource group test optimize fail msg #12438
• [test] Fix some tests not enabled in integration tests #12417
• Fix windows test path probleam #12398
• Fix ProxyServiceStarterTest flaky tests #12344
• [flaky-test] AdminApiTest.testNamespaceSplitBundleConcurrent #14565
• Add test to ensure correct zk children cache invalidation #14178
• [test] Upgrade Mockito to latest stable 3.x version, 3.12.4 #13622
• [test] Add @test annotation to test methods #12640

Build

• [owasp] Add suppression for pulsar-package-bookkeeper-storage #13451
• [owasp] Suppress false positive check for netty-tcnative-classes #13364

Tool

• Fix NPE in cmdTopics #13450
• Add missed import #13037
• [pulsar-perf] Support listenerThreads configuration. #12892
• Improve exception info for invaild time-related option #12828
• [docs] Fix doc for pulsar-admin bookies cmd #12542

Function

• Fix(functions): Missing runtime set in GoInstanceConfig #13031
• Remove corrupt visible testing annotation in function workers #12621
• [function] Pulsar admin exposes secrets for function #12950

CLI

• Feat(cli): support autorecovery service in pulsar cli #12985
• [cli] Enable CLI to publish non-batched messages #12641

CI

• [ci] Replace deprecated "adopt" OpenJDK distribution with "temurin" in GitHub Actions config #12945

Connector

• [es-sink] Use topic name as the index name if indexName is not configured #13064
• Don't create AvroData for each KafkaSourceRecord #12859
• [elasticsearch sink] Correct @FieldDoc defaultValue for some fields #12697

Tiered Storage

• Add retry to tolerate the offload index file read failure #12452
• Fix the read performance issue in the offload readAsync #12443