redpanda-data/redpanda v24.3.1 on GitHub

Features

Added support for Iceberg Topics (various improvements below)
New REST API for mounting/unmounting topics by @mmaslankaprv in #23167
adds rpk cluster storage topic mount, unmount, list-mount, status-mount, cancel-mount by @gene-redpanda in #23575
Add leadership pinning: ability to set preferred racks for topic partition leaders. To configure, set redpanda.leaders.preference topic config property or default_leaders_preference cluster config property. by @ztlpn in #23691
Enable node_local_core_assignment feature by default by @ztlpn in #23453
Adds Schema Registry support for the JavaScript Data Transforms SDK by @oleiman in #21491
Adds list-mountable to allow listing mountable topics by @gene-redpanda in #23924
Adds the topic property delete.retention.ms, as well as the cluster property tombstone_retention_ms. Configuring these allow for the removal of tombstone records in compacted topics with tiered storage disabled in redpanda. by @WillemKauf in #23662
Schema Registry: Support normalize=true by @BenPope in #22519
Schema Registry: added support for the "verbose" query parameter on the schema compatibility checker endpoint by @pgellert in #22877
Schema Registry: verbose compatibility error reporting is now supported for JSON as well by @pgellert in #23208
#17984 Adds a new broker configuration transaction_max_timeout_ms. The configuration controls the maximum allowed user set timeout for transactions. If a client requested transaction timeout exceeds this configuration, the broker will return
an error during transactional producer initialization. This guardrail prevents hanging transactions from blocking consumer progress. The default value is 15mins. by @bharathv in #21504
rpk: Add rpk registry mode to manage the schema registry mode. by @r-vasquez in #22675
rpk: supports triggering on-demand partition balancer by @daisukebe in #22855
Added support for using PKCS#12 files for TLS services by @michael-redpanda in #21313
Adds admin API endpoint for enterprise feature info GET /v1/features/enterprise by @oleiman in #23314
A new metric (cluster_features_enterprise_license_expiry_sec) is added for easier monitoring of the enterprise license's expiry time. by @pgellert in #23367
After the cluster is first formed, a trial license is automatically loaded to provide an evaluation period of enterprise features. by @pgellert in #23893

Improvements

--regex flag in rpk topic describe now supports internal topics. by @r-vasquez in #23487
A number of optimizations to local storage compaction. by @WillemKauf in #23380
Add an LRU caching layer to Rust transform SDK Schema Registry client by @oleiman in #19859
Add support for differentiating tombstone records from empty-string value records in rpk produce and rpk consume. by @WillemKauf in #23264
Added support for Metadata API v8 by @michael-redpanda in #22669
Added vectorized_kafka_rpc_connections_rejected_rate_limit metric which counts incoming Kafka connections rejected due to the connection rate limit (if set), analogously to the existing vectorized_kafka_rpc_connections_rejected metric which counts rejected connections due to the hitting the open connection limit. by @travisdowns in #22803
Adds a shard label to some consumer group metrics. by @ballard26 in #23339
Adds support for setting schema registry connection parameters in the rpk stanza of redpanda.yaml. by @andrewstucki in #24017
Adds the cloud_storage_backend::oracle value, and helps the s3_client properly configure for OCI storage. by @WillemKauf in #22902
Adds the ability to configure Node UUID and ID overrides at broker startup. by @oleiman in #22972
Allow rpk cluster self-test start to run, even in a cluster with mixed versions of redpanda (before and after cloudcheck addition in 24.2.x). by @WillemKauf in #21370
Allows DeleteRecords requests from Kafka clients or rpk topic trim-prefix to be called with truncation_offset <= start_offset without returning an error. The request is instead treated as a no-op. by @WillemKauf in #22905
Allows the self-test to be completely compatible with a mixed version cluster, in the case of a rolling upgrade. by @WillemKauf in #22831
Deprecate leader_balancer_mode cluster config property. by @ztlpn in #23780
Implements @redpanda-data/transform-sdk-sr.SchemaFormat for the WASM Transforms JS module by @oleiman in #23164
Improve handling of boolean property values during a CreateTopics request by making parsing case-insensitive. by @WillemKauf in #23682
Improve handling of boolean values during a CreateTopics request by no longer silently ignoring an invalid value, instead throwing a configuration error. by @WillemKauf in #23682
Improve handling of certain invalid topic configuration parameters that would lead to a timeout failure instead of a graceful error code during a CreateTopics request. by @WillemKauf in #23682
Improve property configuration descriptions. by @Deflaimun in #23347
Minimizes data loss in recovery scenarios by @mmaslankaprv in #24071
Reduce the memory overhead of many small segments. by @rockwotj in #22962
Return core assignments from health report in /v1/cluster/partitions admin API output. by @ztlpn in #22695
Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in #22798
Schema Registry: Improve AVRO Normalization by @BenPope in #22519
Schema Registry: now reports more specific error messages for Avro and Protobuf schemas when they are incompatible with earlier schemas. by @pgellert in #22958
Set the default value of topic_partitions_reserve_shard0 to zero. This means that we no longer weight shard 0 as if it has 2 more partitions than it actually has, leading to more even partition distribution in cases where the total number of partitions is close to the vCPU count. by @travisdowns in #22841
The command line is now printed to the log at startup by the Redpanda process. by @travisdowns in #22826
Upgrade data transforms tinygo compiler to version 0.34.0 by @rockwotj in #23969
#17682 Schema Registry: Remove spurious log entry: No syntax specified for the proto file by @BenPope in #22633
#21536 rpk topic describe-storage can be used now with internal topics. by @r-vasquez in #22338
#22333 rpk debug bundle: include the result of uname -a by @JFlath in #22334
#22666 Allows users to query the value of a cluster property with rpk cluster config get using either the original property name, or any of its aliases. Whereas before, rpk cluster config get using a property's aliased name would return a Property {} not found result. by @WillemKauf in #22674
#23038 Schema Registry: Tolerate null metadata and ruleSet when registering or querying a schema by @BenPope in #23044
rpk registry schema get will now fail if is incorrectly invoked using the --print-schema and --format flag. by @r-vasquez in #23886
fixed large allocation in Raft implementation by @mmaslankaprv in #24002
fixed spurious warning log entries that may be altering users by @mmaslankaprv in #22752
optimized memory usage with large core count nodes by @mmaslankaprv in #22977
rpk cluster license info now shows if your loaded license is expired by @r-vasquez in #23636
rpk cluster license info: now includes more information about your license status and possible violations. by @r-vasquez in #23744
rpk connect will be downloaded on the first run. by @r-vasquez in #23117
rpk will now warn you when you are running enterprise features without a license. by @r-vasquez in #23472
rpk: redpanda admin brokers list exposes Host/Port/Rack/UUID additionally by @daisukebe in #23595
rpk: add --format JSON/YAML support to rpk topic list. by @gene-redpanda in #23806
rpk: introduce rpk connect install, rpk connect upgrade, and rpk connect uninstall, manual ways to manage rpk connect versioning. by @r-vasquez in #23117
tbd by @nvartolomei in #23381
with this improvement the force reconfigured partitions will be backfilled from Tiered Storage even if they lost all local data by @mmaslankaprv in #22908