apache/pulsar v3.1.0 on GitHub

Important notice

Fix the reason label of authentication metrics #20030

PIPs

PIP-275: Introduce topicOrderedExecutorThreadNum to deprecate numWorkerThreadsForNonPersistentTopic #20504
PIP-255: Make the partition assignment strategy pluggable #20537
PIP-278: Support pluggable topic compaction service #20624

Broker

Fix MessageDeduplication throw NPE when enabling broker deduplication and disabling namespace deduplication #20905
In replication scenarios, remote consumers could not be registered if there were no message sent #20888
Fix IndexOutOfBoundsException in the CompactedTopicUtils #20887
Add annotation for topic compaction strategy #20858
Fix inconsensus namespace policies by getPoliciesIfCached #20855
Pass bytesToRead when reading compacted entries #20850
Fix the built-in admin failed to delete a topic with a custom authz provider #20848
Avoid print redirect exception log when getting a list from bundle #20846
Inconsistent behaviour for topic auto_creation #20843
Follow up 4196 use PulsarByteBufAllocator handle OOM #20837
Add broker filter sync method back to guarantee the API compatibility #20826
Fix direct memory leak by delayed index OutOfDirectMemory #20823
Avoid infinite bundle unloading #20822
Add the MessageExpirer interface to make code clear #20800
Broker failed to load v1 namespace resources cache #20783
Cannot receive any messages after switching to a standby cluster #20767
Fix get topic policies as null during clean cache #20763
Fix typo in ServerCnx.java #20762
Avoid throwing RestException in BrokerService #20761
Call ServerCnxcloseProducer from correct thread #20747
Ignore openIDTokenIssuerTrustCertsFilePath conf when blank #20745
Support pluggable topic compaction service - part2 #20718
Revert "Skip loading broker interceptor when disableBrokerInterceptors is true 20422" #20710
Gracefully shutdown does not work with admin cli in standalone #20709
Improve CompactedTopicImpl lock #20697
Added the skipped message handler for ServiceUnitStateChannel #20677
Make ExtensibleLoadManagerImpl's broker filter pure async #20666
Add logs for topics or subscriptions when enabling rate limit #20660
Make get list from bundle Admin API async #20652
Fix the publish latency spike from the contention of MessageDeduplication #20647
Support pluggable topic compaction service - part1 #20645
Fix get owned service units NPE #20625
Topic policy cannot work well if the replay policy message has any exceptions #20613
Fix the publish latency spike issue with a large number of producers #20607
AuthenticationFilter.doFilter #20598
Fix NPE when resetting Replicator's cursor by position. #20597
Add consumer-id into the log when doing subscribe #20568
Release orphan replicator after topic closed #20567
New load balancer system topic should not be auto-created now #20566
Remove unused cache executor in PulsarService #20563
Handle get owned namespaces admin API in ExtensibleLoadManager #20552
Handle heartbeat namespace in ExtensibleLoadManager #20551
Validate authz earlier in delete subscription logic #20549
Should not throw NotFoundException when metadata already exists #20539
REST Client Producer fails with TLS only #20535
Fix redirect loop when using ExtensibleLoadManager and list in bundle admin API #20528
Restore solution for certain topic unloading race conditions #20527
Emit the namespace bundle listener event on extensible load manager #20525
Choose random thread for consumerFlow in PersistentDispatcherSingleActiveConsumer #20522
Disable EntryFilters for system topics #20514
Support get/remove permissions for AuthorizationProvider #20496
Improve the efficiency of checking message deletion #20490
Reformat property in generateResponseWithEntry #20481
Add metric prefix for topic_load_times #20472
Fix return the earliest position when query position by timestamp. #20457
Support revoking permission for AuthorizationProvider #20456
Replace String.intern() with Guava Interner #20432
Remove ineffective solution for reducing GC pressure #20428
Skip loading broker interceptor when disableBrokerInterceptors is true #20422
Avoid PersistentSubscription.expireMessages logic check backlog twice #20416
Change limitStatsLogging config default value to true #20409
PIP-255 Part-1: Add listener interface for namespace service #20406
Use immutable base cursor properties #20400
Fix partitioned __change_events topic is policy topic #20392
Do not expose bucketDelayedIndexStats #20383
Fix flaky test - testClusterMigrationWithReplica… #20379
Change some static fields referencing mutable objects to ordinary instance fields in BrokerService #20372
Fix pre-create non-partitioned system topics for load balance extension #20370
Fix memory leak when rebatchMessage #20369
Invalidate metadata children cache after key deleted #20363
Fix managedLedger.getConfig().getProperties().putAll(properties) NPE #20361
Update rest endpoint method names to avoid swagger conflict #20359
Fix broker load manager class filter NPE #20350
Copy BrokerEntryMetadata when rebatchMessage #20337
In Key_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck #20335
When creating a namespace anti-affinity group with the default local policies, use the bundle size supplied by the user. #20327
Fix skip message API when hole messages exist #20326
Gracefully shut down load balancer extension #20315
Fix class name typo PrecisPublishLimiter to "Precise" #20310
Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion #20304
Fix NPE cause by topic publish rate limiter. #20302
Remove Streaming Dispatcher Code #20279
Fix UnsupportedOperationException when updating topic properties. #20261
Fix ledger cachemiss size metric #20257
Fix default bundle size used while setting bookie affinity #20250
Fix the behavior of delayed message in Key_Shared mode #20233
Fix infinite ack of Replicator after topic is closed #20232
Producer created by replicator is not displayed in topic stats #20229
Use bulk operation instead of iteration #20219
Fix Return value of getPartitionedStats doesn't contain subscription type #20210
Improve knownBrokers update in ModularLoadManagerImpl #20196
Skip split bundle if only one broker #20190
Fix deadlock of metadata store #20189
Fix RoaringBitmap.contains can't check value 65535 #20176
Release EntryBuffer after parsing proto object #20170
Remove useless field in the DelayedMessageIndexBucketSegment.proto #20166
Optimization protobuf code in the bucket delayed tracker #20158
Move bitmap from lastMutableBucket to ImmutableBucket #20156
Producer/Consumer should call allowTopicOperationAsync #20142
Fix entry filter feature for the non-persistent topic #20141
Optimize delayed metadata index bitmap #20136
Fast fix infinite HTTP call getSubscriptions caused by wrong topicName #20131
Make timer execute immediately after load index #20126
Cache LedgerHandle in BookkeeperBucketSnapshotStorage #20117
Pin AppendIndexMetadataInterceptor to field in ManagedLedgerInterceptorImpl #20112
Close temporary open ledger in BookkeeperBucketSnapshotStorage #20111
Fix use error TimeUnit to record publish latency #20074
Fix the reason label of authentication metrics #20030
Save createIfMissing in TopicLoadingContext #19993
Log resource usage rate of brokers that need to be offloaded in ThresholdShedder #19983
Fix getPartitionedStats miss subscription's messageAckRate #19870
Fix wrong logic of method TopicName.getPartition(int index) #19841
Fix Update contains no change error when using --update-auth-data flag to update function/sink/source #19450
Add topic_load_failed metric #19236
Support getStats/update partitioned topic with -partition- #19235
Do not filter system topics while shedding. #18949
If ledger lost, cursor mark delete position can not forward #18620
Do not grant permission for each partition to reduce unnecessary zk metadata #18222
Support cgroup v2 by using jdk.internal.platform.Metrics in Pulsar Loadbalancer #16832

Pulsar IO and Pulsar Functions

[IO] Not restart instance when kafka source poll exception. #20795
[IO] Add notifyError method on PushSource. #20791
[IO] elastic-search sink connector not support JSON.String schema #20741
[IO] Close the Kafka source connector got stuck #20698
[IO] Close the Kafka source connector if there is uncaught exception #20424
[IO] add protobuf ByteString to pulsar-io jdbc core #20259
[IO] KCA: handle kafka sources that use commitRecord #20121
[IO] Fix the wrong format of the logs #20907
[IO] Remove useless configuration for Pulsar SQL #20605
[IO] Some optimized points in 9224 #19027
[FN] Exit JVM when the main thread throws exceptions #20689
[FN] TLS args admin download command use zero arity #20513
[FN] Optimize string concatenation in user metrics gen #20499
[FN] Support customizing TLS config for function download command #20482
[FN] enable Go function token auth and TLS #20468
[FN] Reset idle timer correctly #20450
[FN] Add stateStorageURL and pulsarWebService URL to go InstanceConfig #20443
[FN] Go functions must retrieve consumers by non-partitioned topic ID #20413
[FN] Go functions need to use static grpcPort in k8s runtime #20404
[FN] Correct TLS cert config translation from broker to fn worker #20297
[FN] JavaInstanceStarter --tls_allow_insecure default to false #20267
[FN] Allow unknown fields in connectors config #20116
[FN] Use functions classloader in TopicSchema.newSchemaInstance() to fix ClassNotFoundException when using custom SerDe classes. (targeted for master) #20115
[FN] Upgrade Kotlin version from 1.4.32 to 1.8.20. (targeted for master) #20114
[FN] Check user metric len before iterating #20021
[FN] Support multiple input topics for Go runtime #20000
[FN] Make pulsar-admin support update py/go with package url #19897
[FN] Fix JavaInstanceStarter inferring type class name error #19896
[FN] Fix function update error #19895
[FN] Make KubernetesRuntime translate characters in function tenant, namespace, and name during function removal to avoid label errors #19584

Observability

[Metrics] Add JVM start time metric #20381
[Metrics] Topic with double quote breaks the prometheus format #20230

CLI

[Admin] Set ns level backlog quota does not take effect if retention exists #20690
[Admin] Remove duplicate topics name when deleteNamespace #20683
[Admin] Make admin.cli throw exception instead of silent catch #20530
[Admin] Pretty print bookies racks-placement command output #20516
[Admin] Make failed bin/pulsar-admin source command exit with code 1 (failed) instead of 0 (success) #20503
[Admin] Using AuthorizationService to revoke the permissions #20478
[Admin] Return BAD_REQUEST on cluster data is null for createCluster #20346
[Admin] Enhancing Transaction Buffer Stats and Introducing TransactionBufferInternalStats API #20330
[Admin] Fix examine messages if total message is zero #20152
[Admin] Make subscriptions on NonPersistentTopic non-durable #19741
[Admin] Report earliest msg in partitioned backlog #19465
[CLI] Fix the variable producerName of BatchMsgContainer is null #20819
[CLI] Fix error messages in CmdNameSpaces and CmdTopics #20701
[CLI] Dedicated docs tools #20700
[CLI] pulsar-client produce: allow to produce AVRO message from strings #20654
[CLI] Client: allow to specify KeyValue's schema key (Avro Key support) #20447
[CLI] Fulfill add-opens to function-localrunner also #20417
[CLI] Allow pulser-client consume create a replicated subscription #20316
[CLI] Add --cleanupSubscription to pulsar-admin #20028
[CLI] Fix logging noise while admin tool exit #19884
[CLI] PerformanceProducer do not produce expected number of messages #19775
[CLI] Pulsar-perf: refactor to reduce code duplication #19279

Others

[TXN] Add log when can't send TcClientConnectRequest to broker #20880
[TXN] Use PulsarResource check for topic existence instead of brokerservice.getTopic() #20569
[TXN] Unwrap the completion exception. #20396
[TXN] Implement compatibility for transaction buffer segmented snapshot feature upgrade #20235
[TXN] Fix transaction is not aborted when send or ACK failed #20055
[TXN] fix consumer can receive aborted txn message when readType is replay #19815
[ML] cleanup numManagedLedgerWorkerThreads #20454
[ML] Fix ledger left in OPEN state when enabling inactiveLedgerRollOverTimeMs #20276
[ML] There are two same-named managed ledgers in the one broker #18688
[META] Bookie Info lost by notification race condition. #20642
[META] Adding the missed bookie id in the registration manager #20641
[META] Upgrade to jetcd to 0.7.5 #20339
[META] Switch to the metadata store thread after zk operation #20303
[META] deadlock of zkSessionWatcher when zkConnection loss #20122
[META] Support set metadata size threshold for compression #19561
[WS] Add cryptoKeyReaderFactoryClassName into the file websocket.conf #20840
[WS] Remove unnecessary ping/pong implementation #20733
[WS] Use allowTopicOperationAsync for authz checks #20299
[WS] Use async auth method to support OIDC #20238
[MISC] Stop logging full stack trace when SecurityUtility.loadConscryptProvider() fails to find Conscrypt with an unknown error #20705
[MISC] Fix typo in pip template #20556
[MISC] Use ubuntu 22.04 for Pulsar images #20475
[MISC] Upgrade Netty to 4.1.93.Final #20423
[misc] Consolidate certs in broker (and some proxy) tests #20353
[MISC] Catch TimeoutException when logging about time outs #20349
[Schema] Only handle exceptions when there has #20730
[Schema] Add admin cli for testCompatibility #19974
[Build] Fix generating javadoc issue #20922
[build] Do not bundle swagger dependencies #20687
[Build] Build function-go in golang image #20665
[Build] Configure git-commit-id-plugin to skip git describe #20550
[Build] Fix the pulsar-all image may use the wrong upstream image #20435
[Build] Fix publish image script #20305
[Build] Capture local build scans on ge.apache.org to benefit from deep build insights #20187

Library updates

Upgrade Guava to 32.1.1 to address CVE-2023-2976 #20699
Upgrade snappy-java to address multiple CVEs #20604
Upgrade Jetty to address CVEs #20461
Suppress etcd CVE warnings #20342
Upgrade PyYaml version to 6.0.1 #20831
Upgrade jackson version to 2.15.0 for CVE-2022-1471 #20177
Upgrade prometheus client_golang to v1.12.2 to fix CVE-2022-21698 #20579
Upgrade Netty to 4.1.93.Final #20423
Upgrade to jetcd to 0.7.5 #20339
Upgrade Kotlin version from 1.4.32 to 1.8.20. (targeted for master) #20114

For the complete list, check the Pulsar 3.1.0 changelog.