redpanda-data/redpanda v23.1.1 on GitHub

New Features

Azure Blob Storage Support

#7556 Redpanda gains support for using Azure Blob Storage with Tiered Storage. In order to point Redpanda at your ABS container, set the following new cluster configs: cloud_storage_azure_storage_account, cloud_storage_azure_container, cloud_storage_azure_shared_key. by @VladLazar in #8086
Azure shared key is now reloadable without a restart via cloud_storage_azure_shared_key. This makes rotating keys more convenient. by @VladLazar in

Kerberos Authentication

Support GSSAPI (Kerberos) as a SASL mechanism on the Kafka API. by @BenPope in #8092

Automatic Topic Recovery from Cloud Storage

Add rpk cluster storage recovery sub-command to rpk. The command can start automated topic recovery from an archival bucket and poll the status of an on-going recovery process by @Scandiravian in #8836

Ingress/Egress Rate Limits per Node

#7855 Throughput limit can now be applied at node scope. Each individual node will limit the overall ingress throughput of all Kafka client connections it serves to the number specified by kafka_throughput_limit_node_in_bps cluster property; same can be done on the egress side using kafka_throughput_limit_node_out_bps. by @dlex in #8335
Node Self-Test Tool
Introduces new self-tests for redpanda allowing users to run mini-benchmarks on disk and network hardware to verify expected performance by @graphcareful in #8172
Self test rpk integration by @graphcareful in #8398

Kafka API / Feature Support

#8191 Added support for AlterPartitionReassignments API and ListPartitionReassigments API by @NyaliaLui in #7685
Support for OffsetDelete introduced. by @graphcareful in #8669
#8348 If an offset was committed prior to upgrade then it is not automatically reclaimed under any policy. To remove such offsets use the Kafka Delete Offset API. by @dotnwat in #8634
#8348 If an offset was committed prior to upgrade then it is not automatically reclaimed under any policy. To remove such offsets use the Kafka Delete Offset API. by @dotnwat in #8634
#8348 If an offset was committed prior to upgrade, and later re-committed / written after upgrade then behavior is identical to a committed offset that was initially written after the upgrade. Namely, normal offset retention policy is active. by @dotnwat in #8634
#8348 Legacy systems must explicitly enable offset retention following an upgrade. It is important to note the behavior of the system in this case. Offsets committed after upgrade, but before offset retention is enabled, will behave as if retention had always been enabled. For example, if an offset were committed the same day as an upgrade with default retention time of 7 days, and then legacy support was enabled 5 days later, the offset would be reclaimed 2 days after. by @dotnwat in #8634
#4438 Implement a Kafka-compatible segment.ms topic property. This controls the maximum time between the first write to a segment, and later closing the segment. This is sometimes useful when one needs topic retention limits to apply within a known time period, rather than at the next segment roll. This is disabled by default. by @andijcr in #8047
#4438 Add log_segment_ms_min and log_segment_ms_max cluster configuration properties. These limits are applied to segment.ms and log_segment_ms, to protect systems from pathologically small segment roll periods. by @andijcr in #8047
#4438 Add log_segment_ms cluster configuration property, which is the default value for segment.ms on topics that do not override it. This is also accessible via the Kafka API under its Kafka-compatible alias log.roll.ms. The lowest permitted value for this property is 60 seconds. by @andijcr in #8047
#7624 Adds Kafka group offset retention policy. Enabled by default in v23.1 and later. Clusters created prior to v23.1 and upgraded to v23.1 will have the feature disabled by default to retain an effective infinite retention policy. by @dotnwat in #8158
Support DescribeTransactions Kafka API request by @VadimPlh in #7469
Support ListTransactions Kafka API request by @VadimPlh in #7469
#7827 Updates Redpanda to use the same error code for an expired transaction as Kafka does by @rystsov in #7909

Security

Add secret store configuration to Redpanda console by @RafalKorepta in #8543

Administrative Tools & Guardrails

Add POST v1/partitions/rebalance API to manually trigger partition rebalancing by @mmaslankaprv in #7367
Add vectorized_cluster_partition_num_with_broken_rack_constraint metric (total number of partitions that don't satisfy the rack awareness constraint) by @ztlpn in #7432
Added a tool allowing changing partition count in __consumer_offsets topic by @mmaslankaprv in #7604
Added fetch throttling mechanism. target_fetch_quota_byte_rate - target fetch size quota byte rate for client (bytes per second). Disabled by default by @ZeDRoman in #7658
#9023
Certain topics are now protected from being produced to by Kafka clients, using a new kafka_noproduce_topics cluster configuration property. The default value of this property is ['__audit'] by @ballard26 in #7614
Certain topics are now protected from deletion and configuration changes by Kafka clients, using a new kafka_nodelete_topics cluster configuration property. The default value of this property is ['__consumer_offsets', '__redpanda_e2e_probe', '_schemas'] by @ballard26 in #7614
Cluster configuration properties are added to control node isolation detection: node_isolation_raft_timeout and node_isolation_heartbeat_timeout. Both are 3 seconds by default. by @VadimPlh in #7829
New command rpk group offset-delete will be added to rpk by @graphcareful in #8757
REST Proxy: A health check is now available at :8082/status/ready by @BenPope in #7080
Redpanda nodes now automatically decline Kafka API requests when they detect that they are isolated from the cluster by @VadimPlh in #7829
Schema Registry: A health check is now available at :8081/status/ready by @BenPope in #7080
Wait for under replicated partition to catch up by @RafalKorepta in #7594
#5321 Adding new redpanda setting (max_concurrent_producer_ids) to control how many sessions for transaction/idempotency will be saved by @VadimPlh in #7057
#6569 Avoids infinite crash loop by detecting unclean shutdowns of the process and enforcing a limit on such shutdowns, enforced via node configuration --crash_loop_limit. Once the limit is reached, operator should remove the tracker file manually to make startup progress. The tracking automatically resets if node configuration changed or if 1 hour elapsed since the last crash. by @bharathv in #8122
#8268 rpk: Introduce a new command rpk redpanda admin brokers decommission-status <node-id> by @daisukebe in #8376
#7874 new admin API GET /v1/brokers/<id>/decommission by @mmaslankaprv in #8167

Improvements

Small adjacent segments in Tiered Storage are now periodically merged together in the background to improve efficiency by @Lazin in #8238
Improve memory efficiency of the tiered storage by @Lazin in #7366
Improved tail latency when one partition has many segments in cloud storage by @jcsp in #7751
A frequent "No segments to collect for ntp" log message is now reduced to DEBUG severity. by @jcsp in #8572
A new admin API /v1/partitions/local_summary reports the count, leaderless count and under-replicated count for partitions with replicas on the node serving the request. This replaces the use of the prometheus endpoint for programs that previously used that to read the vectorized_cluster_partition_under_replicated_replicas metric. by @jcsp in #9011
A new partition metric redpanda_cloud_storage_segments_metadata_bytes is added that tracks the amount of memory consumed by segment metadata managed by the remote partition. by @andrwng in #8304
Add --superusers-prefix flag in the operator CLI by @pvsune in #7178
Added a new field called additionalCommandlineArguments under the configuration block that allows additional command line arguments to be passed to the redpanda command line. by @koikonom in #8401
Additional safety checks on the version compatibility of nodes joining a cluster. Please ensure that when adding nodes to a cluster, the new nodes are from the same feature series, or at most one feature series newer (e.g. adding a 23.1.1 node to a cluster of 22.3.x nodes). Nodes from an older feature version will not be permitted to join a cluster that has previously been upgraded fully to a newer feature version. by @jcsp in #8282
Before deleting the POD maintenance mode is configured (not within 120 second lifecycle hook) by @RafalKorepta in #7530
Better UX when auto creating topics, creation waits for the topic leaders to be assigned and ready to accept requests. by @bharathv in #7731
Bumps API version by one to enable flex support for add_partitions_to_txn, offset_commit, delete_groups, describe_groups, end_txn_request, find_coordinator, heartbeat, incremental_alter_configs, join_group, leave_group, list_groups, offset_for_leader_epoch, alter_configs, create_acls, delete_acls, describe_acls, describe_log_dirs, and sasl_authenticate APIs by @graphcareful in #7900
Call GetLicenseInfo in the operator before SetLicense to add extra check and avoid unnecessary logs in Redpanda by @pvsune in #7272
Client authentication errors are now logged at WARN severity rather than ERROR by @jcsp in #8087
Consumer group related errors are now logged at a higher level (info) by @dlex in #8192
Disk space checks are now more frequent, running every 1 second. This makes redpanda_storage_free_space_alert update more promptly when the system starts to run low on disk space. by @jcsp in #5252
Handle user provided timestamps on the timequery read path more gracefully. by @VladLazar in #8509
Improve tiered-storage reliability in presence of inconsistencies in metadata by @Lazin in #7502
Improved handling of stop condition based on lack of partition balance improvement by @mmaslankaprv in #7408
Improved heuristic for determining the maximum timestamp on index truncation. This should result
in more precise time retention enforcement. by @VladLazar in #8509
In large cluster with a lot of partitions and CPU count the response might not be returned in 2 second timeout. by @RafalKorepta in #9016
Make tiered-storage robust in presence of inconsistencies in data in the object store by @Lazin in #8810
Metrics are added to the /public_metrics endpoint for the tiered storage read cache by @jcsp in #8874
Migration code for upgrading directly from 21.x Redpanda versions is removed. by @jcsp in #7507
More robust stop condition for finishing partition movement. by @mmaslankaprv in #8314
Pod name included during rolling update procedure by @RafalKorepta in #8649
Redpanda may now be run with an empty list for the kafka_api node configuration property, enabling starting a cluster without client connectivity during initial configuration/bootstrap. by @jcsp in #5126
Redpanda now shuts down more quickly if it is signaled while still starting up. by @jcsp in #5707
Redpanda will abort immediately with a diagnostic when a hard memory allocation failure occurs (a hard failure is an allocation one which cannot be satisfied even by reclaiming memory from in-memory caches). by @travisdowns in #9073
Redpanda will now return a retriable status when its internal RPC subsystem is bootstrapping. by @andrwng in #8482
Schema Registry: Improve performance by disabling batching unless explicitly set by @BenPope in #7744
Sped up time queries by using indexing. by @VladLazar in #8509
The /v1/features Admin API endpoint now includes an original_cluster_version reflecting the logical version at which the cluster was created. by @jcsp in #4996
The seed_servers node configuration may now be identical on each node to form a cluster, even when empty_seed_starts_cluster is true (the default). by @andrwng in #7390
Updates inside kubernetes environment the Redpanda update will be more
safe as operator now doesn't consider updates if cluster does not report
healthy status. by @RafalKorepta in #7528
When loading redpanda.yaml, if an 'unknown control escape character' error occurs, a hex dump of the configuration file will be logged. This improves diagnosability of rare issues loading the file (#3798) by @jcsp in #8552
Writes to the tiered storage cache now run with a lower I/O priority than writes from Kafka producers: this may improve performance in some situations on systems using tiered storage. by @jcsp in #8651
#3273 Read replicas behave more robustly in rare cases where leadership changes on the writer cluster could previously cause partition high watermark to appear to temporarily jump backwards. by @jcsp in #7707
#4170 Updates DescribeConfigs Kafka API to support KIP-569 by @NyaliaLui in #7543
#4950 Controller log replay is more resilient to unexpected removal of log on disk. by @jcsp in #5703
#6902 Change HTTP error code when leadership transfer can't proceed due to recovery from 504 to 503 by @dlex in #7297
#7208 An ERROR log message on S3 upload failures is downgraded to WARN, as this situation has been observed to occur transiently in normal operation on AWS. by @jcsp in #7468
#7218 Improved algorithm rebalancing partitions after node addition. by @mmaslankaprv in #7511
#7222 Cleaned up comments. by @vrazdalovschi in #8600
#7288 Fix const in connection_rate_limit_test by @VadimPlh in #7648
#7420 Upon completion of rpk cluster config edit or rpk cluster config set, print ‘ Cluster needs to be restarted. See more details with 'rpk cluster config status' if restart is required: by @daisukebe in #7421
#7451 Improved compatibility with tools using AlterConfigs API. Tiered storage topic properties now report DEFAULT_CONFIG state if their current value matches the cluster defaults. by @jcsp in #7602
#7490 Tiered storage topic recovery now uses the retention.local.target settings to bound how much data is restored to local disk, rather than the total retention settings. by @jcsp in #7493
#7539 Internal metrics archival_service_[stop|start]_archiving_ntp are no longer used, and are removed from /metrics API output. by @jcsp in #7547
#7539 Tunable cluster configuration property Unused configuration property is no longer used and is deprecated. by @jcsp in #7547
#7556 The existing vectorized_s3_client.* metrics have been renamed to vectorized_cloud_client.* in order to match the naming on the public_metrics endpoint. by @VladLazar in #8086
#7629 improved startup time and eliminated reactor stalls with large number of consumer groups by @mmaslankaprv in #7957
#7641 Add --max-message-bytes to rpk topic produce to be able to override the max batch size franz-go enforces behind the scenes. by @daisukebe in #7646
#7650 Add new tags to AWS S3 objects by @Lazin in #7654
#7656 Adds a new endpoint to the schema registry that clears the in-memory data. by @NyaliaLui in #7765
#7665 Improve speed when recovering a large topic in some situations. by @ballard26 in #7782
#7761 Set group_initial_rebalance_delay=0 in dev container mode. The default setting adds a 3s delay when using consumers groups in unit tests and is a bad UX. by @bharathv in #7784
#7874 no need to wait for all reallocations to finish before decommissioning a node by @mmaslankaprv in #7862
#7936 Log lines with s3 response header are now on one line only by @andijcr in #7947
#7975 Non terminal error code invalid_tx_state is returned when committing/aborting transactions that haven't begun. by @bharathv in #7996
#7984 Transient "Transport endpoint not connected" errors to S3 are no longer logged at ERROR severity. by @jcsp in #8018
#7990 validation of segments.bytes property according to configured min/max values by @mmaslankaprv in #8006
#8046 Topic deletion now removes local data more reliably in situations where a tiered storage topic experiences an inability to connect to an object storage backend. by @jcsp in #8090
#8170 The default log segment size is now 128MiB, decreased from the previous 1GiB. by @jcsp in #8180
#8301 better handling of nodes decommission/recommission by @mmaslankaprv in #8393
#8362 Made it safe to restart decommissioned node without a worry that it would rejoin the cluster by @mmaslankaprv in #8547
#8496 Cluster configuration property cloud_storage_graceful_transfer_timeout_ms is added, with a default of 5000ms. Setting this property to null disables the new behavior of waiting for uploads to complete before transferring leadership. by @jcsp in #8560
#8496 Partition leadership transfers now wait for tiered storage uploads to finish, resulting in a lower probability of orphan objects in the object storage bucket. These objects are not a data integrity issue, but could result in a small amount of extra space used. by @jcsp in #8560
#8724 easier debugging of long reads during recovery by @mmaslankaprv in #8741
rpk cluster health now reports under replicated partitions, and considers the cluster unhealthy if any partitions are under replicated. by @jcsp in #9011
added reading consumer offsets topic capability to Redpanda offline log viewer tool by @mmaslankaprv in #7603
more eager upgrade of configuration to new safer version by @mmaslankaprv in #8420
more robust partition movements related with decommissioning/recommissioning nodes that are unavailable by @mmaslankaprv in #8322
schema_registry: AVRO: Support custom attributes on fields (fix a regression since v22.2.x) by @BenPope in #7330
schema_registry: If _schemas topic is missing, return a 500 status code. by @BenPope in #7555
treat crash_loop_limit as node configuration option in configuration drift detection by @RafalKorepta in #8312
dropped support for data policies i.e. redpanda internal v8 script executor by @mmaslankaprv in #7724

Bug Fixes

Added missing permissions to the operator helm chart's ClusterRolebindings template. by @joejulian in #7484
An issue is fixed where shutdown or internal housekeeping operations could fail to progress on a node that had been out of contact with peers for some time, when partitions are also not being written to. by @jcsp in #8271
Files left over from aborted compactions will now be cleaned up more robustly. by @andrwng in #7912
Fix a rare issue in topic recovery, where if topic retention was configured to a very small size, subsequent consumer reads could hang. by @jcsp in #8843
Fix an issue that can cause redpanda to send ill-formed DeleteObjects request to AWS S3 by @Lazin in #8004
Fix an issue that prevented 22.3.x nodes from starting when recent controller log contained commands to manipulate __consumer_offsets partitions by @dlex in #7406
Fix automatic partition rebalancing when multiple nodes are added to the cluster before previous rebalancing finished by @mmaslankaprv in #7367
Fix bug where the request size of a disk self test was 512K. Has been changed to 4k. by @graphcareful in #8737
Fix flakiness of operator e2e tests. by @RafalKorepta in #8648
Fix for unreleased feature #7820 by @alenkacz in #8057
Fix hang during shutdown when tiered storage is actively used by @Lazin in #8554
Fix raft_recovery group name by @mmaslankaprv in #7267
Fix timequery in read-replica by @Lazin in #7938
Fixed an incorrect log message indicating the success or failure of archival metadata updates. by @andrwng in #8616
Fixed bug where rpk sends incorrect test duration to admin_server by @graphcareful in #8695
Fixed issue that may caused heartbeats to be missed in loaded clusters by @mmaslankaprv in #8781
Fixed starting redpanda after executing at least 100 node operations by @mmaslankaprv in #8861
Fixes Cluster can't be deleted when licenseRef is not found due to introduction of Cluster finalizers by @pvsune in #7417
Fixes a bug where licenses could not be re-uploaded. by @graphcareful in #8195
Fixes a segfault in partition shutdown caused by incorrect use of intrusive lists. by @bharathv in #7824
Fixes bug that would start slightly more network test runs then necessary by @graphcareful in #8695
Fixes hang on shutdown bug due to active self-test(s) by @graphcareful in #8695
Improve robustness of Schema Registry and HTTP Proxy under several connection errors. by @BenPope in #7735
Node ID assignment will now use node IDs higher than those used by the initial seed servers, avoiding a potential node ID re-use in cases where a seed server is fully decommissioned before adding a new node. by @andrwng in #7821
Removes duplicate responses in OffsetFetch requests by @graphcareful in #8689
Return a correct node_id in moving_to when invoking GET /v1/brokers/<id>/decommission by @daisukebe in #8347
Schema Registry: Improve protobuf support for recursive imports by @BenPope in #8789
This fix will prevent users from configuring any even value to the cluster configuration property, default_topic_replications by @daisukebe in #7244
#5558 Fix cross-shard allocator manipulation by @travisdowns in #7351
#5994 Fixes Pandaproxy and schema registry integration with redpanda when TLS is enabled on internal listener by @alenkacz in #7820
#6846 Made list of nodes returned by /brokers and /config_status endpoints consistent by @mmaslankaprv in #8181
#7217 Fixes frequent unknown server errors in the transactional API by @rystsov in #7591
#7043 An issue is fixed where after certain disaster recovery procedures (tiered storage topic recovery), data in aborted transactions within a small range of offsets could become visible to consumers. by @jcsp in #7819
#7222 Pod distribution budget version v1 is used now in the operator by @vrazdalovschi in #8600
#7256 Improve enforcement of storage_target_replay_bytes by ensuring the limit
is shared amongst all shards. by @VladLazar in #8433
#7521 transactions: preserve monotonicity of LSO by @bharathv in #7544
#7526 Fix recommissioning stop condition that might prevent the recommission operation from finishing by @mmaslankaprv in #7533
#7539 A race is fixed where tiered storage topics could leave a small number of objects behind in the cloud storage bucket after the topic was deleted. by @jcsp in #7547
#7637 In some cases, Redpanda could hang on EIOs from the underlying storage device. This behavior has changed to terminate redpanda with an assertion on EIO, in anticipation of the node/drive requiring replacement. by @jcsp in #7633
#7647 Fix integer overflow in compaction index footer that could lead to hangs when trying to compact large segments. by @ztlpn in #7687
#7691 rpk: --offset option parses duration properly, e.g. --offset @2022-02-14:1h by @daisukebe in #7971
#7708 Fixed an issue wherein fetching at an offset could incorrectly return no results if the partition manifest were truncated. by @andrwng in #8110
#7715 Pod distribution budget version v1 is used now in the operator by @RafalKorepta in #8313
#7748 Redpanda will now correctly honor negative values to mean an infinite value for the retention.ms and retention.bytes topic configurations. by @andrwng in #7747
#7816 fixes pathological case in which a controller log may grow very large when partition op is being retried by @mmaslankaprv in #7789
#7818 Fix operator panic during decommissioning logging by @RafalKorepta in #7828
#7895 Mechanism for cleaning partition orphan files by @ZeDRoman in #8396
#7905 Fixes occasional InvalidTxnState on adding partition or group by @rystsov in #7807
#7921 Fix setting redpanda.remote.delete topic property via the legacy AlterConfig API (this API was deprecated in Kafka 2.3.0, but is supported for backward compatibility) by @jcsp in #7922
#7955 Fix a rare bug due to which aborted transactions would not be included in the result of fetch requests
that hit cloud storage. by @VladLazar in #8926
#9058 Fixes a bug where the operator would try to reconfigure the cluster every minute. by @joejulian in #9075
#8031 Fix bug with movement on node addition of partition that is already moving by @ZeDRoman in #8182
#8045 Fix archival metadata STM replication error in read replicas by @Lazin in #8051
#8069 map replicated_entry_truncated error to 503 http error code by @ZeDRoman in #8494
#8091 fixed spurious batch parse error messages emitted while reading by @mmaslankaprv in #8409
#8203 An issue is fixed that could prevent clusters forming successfully when network disruptions happened during initial startup of the founding nodes. by @jcsp in #8225
#8218 fixed incorrectly tracked previous decommission update that in some rare situations may lead to assertion being triggered in Redpanda by @mmaslankaprv in #8245
#8236 Fixes availability issue which could led to stuck transactions by @rystsov in #8566
#8290 fixed stopping producer when moving partitions by @mmaslankaprv in #8387
#8291 Fix a race between the time-query and truncation logic which caused time-queries to return
offsets marked for deletion/compaction. by @VladLazar in #8900
#8331 Fixed a hang at shutdown when using tiered storage. by @andrwng in #8587
#8486 Fixes availability issue which could led to stuck transactions by @rystsov in #8624
#8511 Fix reactor stalls on partitions with large number of segments in the cloud storage by @Lazin in #8620
#8745 A bug is fixed where background leadership balancing could unexpectedly interrupt transactional workloads. by @jcsp in #8941
#8758 An issue is fixed where a cluster might not activate new functionality after an upgrade if a controller leadership change happened at a particular point during upgrade by @jcsp in #8902
#8945 Fixed an issue that could prevent uploads after all cloud segments have been garbage collected. by @andrwng in #8968
#9014 The type of metrics redpanda_storage_disk_free_bytes and redpanda_storage_disk_total_bytes is corrected from by @jcsp in #9020
fix situations in credentials are not refreshed leading to expired token errors by @mmaslankaprv in #8599
https:// clients which send a "Connection: none" header over a TLS connection no longer experience premature connection closure. by @jcsp in #8022
schema_registry: Explicitly set _schemas topic to retain forever. by @BenPope in #7553
storage: protect optional access by @abhijat in #7655

Full Changelog: v22.3.13...v23.1.1