redpanda-data/redpanda v26.1.1 on GitHub

Features

Cloud Topics: add redpanda.storage.mode by @WillemKauf in PR #29352
cloud_topics_reconciliation_parallelism: Controls the maximum number of L1 objects the reconciler can build concurrently per shard. Defaults to 8. The per-shard memory reserved for reconciliation is part_size × parallelism. by @wdberkeley in #29671
cloud_topics_reconciliation_upload_part_size: Controls the multipart upload part size used when the reconciler builds L1 objects. Defaults to 5 MiB (the minimum allowed by cloud object storage providers). Together with the parallelism setting, determines the per-shard memory reserved for reconciliation. by @wdberkeley in #29671
Cloud Topics: Level Zero GC delete pipeline by @oleima In PR #29050
Cloud Topics: Sharded level zero GC workers by @oleiman In PR #29088
Cloud Topics: Object path random prefix by @oleiman in PR #29066
Cloud Topics: Introduce Admin API for Level 0 GC by @oleiman in PR #29388
ct: describe-storage impl for cloud topics by @WillemKauf In PR #29864
ct: cloud topics compaction configs by @WillemKauf in PR #29287
ct: hook cloud topics recovery into whole cluster restore by @andrwng in PR #29515
Cloud Topics: Admin RPCs GetEpochInfo and AdvanceEpoch by @oleiman in PR #29515
Cloud Topics: Epoch advance frontend by @oleiman in PR #29536
Cloud Topics: Metrics for L0 GC by @oleimanin PR #29556
Cloud Topics: Level Zero GC force reset (escape hatch) by @oleiman in PR #29774
Cloud Topics: L0 GC Safety Monitor by @oleimangroup principal support by @michael-redpanda in PR #28906
group role authz by @michael-redpanda in PR #29259
Consider group membership for authz checks by @michael-redpanda in PR #29157
admin: add group role members to v2 RBAC API by @nguyen-andrew in PR #29236
Process group claims from OIDC tokens by @michael-redpanda in PR #29000
security/role: Add group role member type by @nguyen-andrew in PR #29217
Support Group ACLs in Schema Registry by @michael-redpanda in PR #29476
rpk security group assign <group> --role <role> by @graham-rp in #29738
rpk security group describe <group> by @graham-rp in #29738
rpk security group list by @graham-rp in #29738
rpk security group unassign <group> --role <role> by @graham-rp in #29738
User-based client quotas. by @IoannisRP in #29425
Allow leadership pinning to racks in priority order by @joe-redpanda in #29366
Add --schema-context flag to rpk registry to scope all registry operations to a specific context by @c-julin in #29689
Add rpk registry context list and rpk registry context delete commands for managing schema registry contexts by @c-julin in #29689
schema_registry: Support storing and retrieving Metadata Properties alongside the Schema. by @BenPope in #28845
rpk now supports Schema Registry metadata properties. by @r-vasquez in #29431
Adds support for SASL/PLAIN authentication for Shadow Linking by @michael-redpanda in #28708
Iceberg: Support customizing Iceberg catalog namespace (defaults to redpanda) at cluster level. by @nvartolomei in #29113
Iceberg: Add $ref keyword support to JSON Schema integration. Limited to internal references and known keywords for now. by @nvartolomei in #29272
Support JSON Schema additionalProperties with object sub-schemas, mapping them to Iceberg map types with string keys. by @nvartolomei in #29735
Iceberg: JSON Schema to Iceberg conversion now supports oneOf for expressing nullable fields (oneOf: [T, null]). by @nvartolomei in #29590
Remote Read Replicas now support cross-region access by specifying region and endpoint overrides in the read replica topic's bucket property (e.g. bucket-name?region=us-east-1&endpoint=s3.us-east-1.amazonaws.com), enabling RRR to read from buckets in different regions on AWS S3, and other S3 compatible backends.
Allow nodes to automatically decommission after a certain timeout by @joe-redpanda in #28946
Adds native support for buf's protovalidate to Schema Registry by @michael-redpanda in #29404
Coordinate transaction end markers removal by @bashtanov in PR #29043
rpk cluster health: high_disk_usage_nodes is now reported. by @r-vasquez in #29480
adds the cloud_storage_prefetch_segments_max cluster config which can be used to enable small segment prefetching in cloud storage. by @ballard26 in #29496
#29347 Added delete_topic_enable cluster configuration property to globally enable or disable topic deletion via the Kafka API. When set to false, all topic deletion requests are rejected with error code TOPIC_DELETION_DISABLED (73). Default is true for backward compatibility. This setting works independently of kafka_nodelete_topics, which continues to protect specific topics regardless of this setting. by @michael-redpanda in #29365
Added OIDC security to Admin API v2 with identity resolution, cluster-wide key refresh, and session revocation capabilities. Administrators can now manage OIDC authentication security through the new v2 endpoints with proper cluster coordination and error handling. by @nguyen-andrew in #28973
Added SCRAM credential management in Admin API v2, including create, list, get, update, and delete SCRAM credentials. by @nguyen-andrew in #29174
Added role management in Admin API v2, including create, list, get, and delete roles, and manage role members. by @nguyen-andrew in #29072
rpk shadow now has visual feedback on long-running operations. by @r-vasquez in #29060
rpk: supports now RPK_PROFILE environment variable to select your current profile. by @r-vasquez in #29195
rpk: now have --ignore-profile that makes rpk ignore the rpk.yaml and redpanda.yaml config files. by @r-vasquez in #29215
rpk: rpk can now be used to create Redpanda roles in Redpanda Cloud using the rpk security role command. by @r-vasquez in #29162
Rpk can now generate a dashboard for monitoring Redpanda Serverless clusters via rpk generate grafana-dashboard --dashboard serverless. by @nicolaferraro in #28858

Bug Fixes

Addresses an issue if the OIDC service fails to communicate with the IdP causing the prevention of controller leadership election by @michael-redpanda in #28608
Adds better validation to arrays and strings being read off the wire from kafka clients. by @WillemKauf in #29208
Attempt to fail over a topic that isn't being Shadowed now returns a 404 rather than a 409 by @michael-redpanda in #28737
Avoids runaway replicators after topics got deleted. Additionally improve some observability related to shadow requests. by @bharathv in #28825
Compatibility lookups on non-existent subjects can now fall through to context-level resolution with defaultToGlobal=true instead of always returning an error. by @nguyen-andrew in #29713
Do not wait for EOF from OIDC server on TLS connection shutdown. by @pgellert in #28425
Fix a cache collision in in-memory spillover manifest cache which could lead to new topic reading data from the old topic with the same name for a short period of time. by @nvartolomei in #29068
Fix a rare bug where adjacent segment merging in tiered storage fails with Candidate creation error: candidate creation error: failed to seek end offset which could result in degraded tiered storage read performance. by @nvartolomei in #28602
Fix an issue where an unhandled exception in archival could lead to an input_stream asserting in its destructor. by @oleiman in #28886
Fixed a bug where if a user updates a Shadow Link with the same password the password set at timestamp was cleared. Now if they use the same password, the timestamp is kept at its previous value by @michael-redpanda in #28855
Fixed a rare bug that causes timequeries against partitions using tiered storage to incorrectly return no result when the partition's local log is empty but retains an active segment. by @wdberkeley in #28642
Fixed admin API service restart to handle pandaproxy and schema_registry on the correct shard. by @pgellert in #29519
Fixed an issue where Kafka fetch session responses could omit partitions with changed metadata (high watermark, log start offset, last stable offset) when the fetch retried internally due to min_bytes not being satisfied. by @pgellert in #29304
Fixed buggy race detection logic in archival segment collection that could miss races between collection and compaction. by @oleiman in #29721
Fixed cluster link replication attempting prefix truncation on partitions that do not support it (e.g., cloud topics) by @michael-redpanda in #29274
Fixed file cleanup issue in datalake::record_multiplexer. by @ballard26 in #29249
Fixed shadow link replicating all historical data when start_at_timestamp is set to a timestamp beyond the end of the source partition log. The link now falls back to the last stable offset and replicates only new data. by @nguyen-andrew in #29861
Fixes a bug in XML parsing for S3 and ABS clients which could lead to partial reads of XML responses by @WillemKauf in #28552
Fixes a bug in which tristate topic properties would be considered disabled if set with the value 0 during a CreateTopic or AlterConfig request. Now, topic properties set with 0 will reflect that value exactly. This affects the following topic properties: segment.ms, retention.bytes, retention.ms, retention.local.target.bytes, retention.local.target.ms, initial.retention.local.target.bytes, initial.retention.local.target.ms, delete.retention.ms, min.cleanable.dirty.ratio. by @WillemKauf in #28811
Fixes a bug in which a segment could be incorrectly marked as having finished self compaction during a race with a log shutdown by @WillemKauf in #28730
Fixes a bug where truncation does not propagate to sink if source start offset has moved ahead of sink high watermark. by @bharathv in #29358
Fixes a long-standing bug in which compaction in the presence of leadership transfers/restarts could punch a hole in idempotency. Compaction is now disallowed from compacting the last batch for an idempotent producer. by @WillemKauf in #29001
Fixes a possible cause of offset translator inconsistency arising from incorrect truncation of the log. by @bharathv in #29849
Fixes a race in http client during shutdown by @rockwotj in #28438
Fixes an issue where kafka clients that don't set a max_wait_ms (such as the schema registry client) would timeout immediately and not return any data. by @rockwotj in #29708
Fixes duplicate message writes in Iceberg when disk or memory reservation errors occur. by @ballard26 in #29251
Fixes incorrect reporting of partition movement progress in decommission/reconfiguration status output. by @bharathv in #29047
Iceberg: Allow bucket names with dots. Also, reject URIs returned by catalog that look like s3://bucket.s3.amazonaws.com/path/to/object. by @nvartolomei in #29071
Iceberg: Fix incorrect duplicate keyword detection in JSON Schema that only caught adjacent duplicates. by @nvartolomei in #29592
Prevent a slew of races between operations in the storage layer that could lead to attempting to append to a closed segment or a failed assert by @WillemKauf in #28472
Prevent a very narrow race condition in the partition_balancer_backend. by @WillemKauf in #28460
Reduces a window in which Redpanda may end up missing data when using the object_storage catalog for Iceberg Topics. Reminder: the object_storage is generally unsafe and should not be used in production. by @andrwng in #28414
Repair tiered-storage manifests with misaligned archive offsets during archiver startup. This unblocks retention when such a manifest was present. by @nvartolomei in #29831
Resolve deadlock in force_reconfigure edge case by @joe-redpanda in #29386
Schema Registry: Fixes a bug in Avro schema compilation where external references were not properly resolved from the schema store, causing compilation failures, compatibility check failures or server-side schema id validation failures for schemas with reference dependencies. by @pgellert in #28780
Updated C-Ares to 1.34.6 to address CVE-2025-62408 by @michael-redpanda in #29036
Upgrade libxml2 to v2.15.2 to fix CVE-2026-0990 (Uncontrolled Recursion via xmlCatalogXMLResolveURI() in XML catalog processing). by @tyson-redpanda in #29788
Variety of fixes to reduce noise in log concerning internal kafka client authentication and authorization by @michael-redpanda in #28852
#29189 Fix the partition_response for a produce request to a LogAppendTime topic to accurately reflect the timestamp_type. by @WillemKauf in #29286
deflake ManyTopicsTest.test_decommission_node_unsafely by @joe-redpanda in #28518
deflake offset for leader epoch by @joe-redpanda in #28389
firm up start offset update logic by @joe-redpanda in #28309
fix dangling reference in many partitions edge case by @joe-redpanda in #29909
guarantee invalid lso gets converted to a retryable error when relevant by @joe-redpanda in #29112
makes CFR cleanup manual to avoid nodewise recovery race by @joe-redpanda in #28307
prevents state machine from loading local snapshot with offsets greater than log dirty offset by @mmaslankaprv in #28890
pulls in seastar fix for ossl by @joe-redpanda in #28503
schema_registry/swagger: Fix type for version in GET /compatibility/subjects/{subject}/versions/{version} by @BenPope in #28297

Improvements

rpk profile edit now documents each available field for an rpk profile by @r-vasquez in #29309
rpk shadow config generate: now supports --print-template along with --for-cloud by @r-vasquez in #29098
Snapshot id correlated with table update transaction is now available in Datalake REST API by @mmaslankaprv in #29180
Cluster health report now includes node IDs of nodes that exceed the disk usage reporting thresholds. by @bharathv in #29136
rpk: WCR supports --cluster-uuid-override by @daisukebe in #29153
Add rpk profile validate to check for your profile correctness. by @r-vasquez in #29357
Add batch delete support for ABS cloud storage clients by @oleiman in #29143
Add batch delete support for GCS cloud storage clients by @oleiman in #29246
Added IO queue configuration metrics exposing iotune rates per Seastar IO queue by @travisdowns in #29893
Added host_metrics_info gauge with labels describing device resolution and configuration state by @travisdowns in #29893
Adds automatic retry with backoff for transient HTTP connection errors in Cloud API and S3 package download calls, reducing intermittent CDT test failures caused by RemoteDisconnected errors. by @cjayani in #29238
Adds commons sense validation to intervals relevant to partition balancing by @joe-redpanda in #29282
Aliases the following enum variants: by @michael-redpanda in #28807
- ACLPattern.ACL_PATTERN_PREFIX to ACLPattern.ACL_PATTERN_PREFIXED
- PatternType.PATTERN_TYPE_PREFIXED to PatternType.PATTERN_TYPE_PREFIX
Archive garbage collection now deletes segments in bounded batches (controlled by cloud_storage_gc_max_segments_per_run, default 300) instead of attempting to delete the entire backlog in a single housekeeping run. This prevents unbounded object storage delete requests on partitions with large expired archives, ensures each run commits incremental progress, and automatically schedules fast follow-up runs until the backlog is fully drained. by @nvartolomei in #29870
Better prioritize compaction of __consumer_offsets partitions when heavy-weight compactions are present on a broker. by @WillemKauf in #29382
Bump ListOffsets API support to v6 by @michael-redpanda in #28722
Calling ListBrokers in the Admin v2 API doesn't require all brokers to be up. by @rockwotj in #29133
Connection errors now report the resolved address where possible. by @BenPope in #29201
Fixes a vulnerability in aioboto3 by pinning the urllib3 dependency. by @tyson-redpanda in #28907
Iceberg manifest serialization now handles all data_file fields from the v2 spec, ensuring full compatibility/no optional metadata loss during merge_append_action manifest rewriting. by @nvartolomei in #29680
Improve cluster linking shadow topic prefix truncation error message. by @pgellert in #29289
Improve performance of Shadow Link when scaling to thousands of topics but making internal data structures copy on write by @michael-redpanda in #28956
Improve timeout logic for failover link to not timeout when processing thousands of shadow topics by @michael-redpanda in #28986
Improved default TLS Cipher Suites - Remove RSA key-exchange, and CBC, CCM by @BenPope in #27912
Increase frequency of topic reconciliation loop to reduce time it takes to failover topics by @michael-redpanda in #28986
Leader balancer improved to issue less moves to reach balanced state by @bashtanov in #29566
Low-level metrics for the segment appender are not available, such as bytes requested/written, IOs issued, etc. This lets us calculate various types of write amplification at the appender level and otherwise monitor the appender behavior. by @travisdowns in #29395
Minor improvement to speed up feature activation after major version upgrades. by @pgellert in #29824
Minor improvements to http client for better connection reuse in certain edge cases. by @nvartolomei in #28788
On rpk generate app, set go as the default language. by @paulohtb6 in #28246
Partition balancer reallocation failures now includes moves which lost source quorum by @joe-redpanda in #28751
Reduce log noise by @michael-redpanda in #28986
Reduces oscillating behavior in partition balancing by @joe-redpanda in #29241
Remove a copy of a potentially large datastructure when starting a transaction by @rockwotj in #29834
Reply HTTP 400 not 500 to admin endpoints when the underlying error is cluster::errc::topic_already_exists. It deflakes data migration tests, and is consistent with existing HTTP 400 on cluster::errc::topic_not_exists. by @bashtanov in #29630
Resolve deadlock in force_reconfigure edge case by @joe-redpanda in #29279
SCRAM credentials now track password_set_at timestamps, exposed via admin API v2. This enables clients like the Kubernetes operator to verify credential state changes have propagated and perform accurate reconciliation. by @nguyen-andrew in #29328
Schema Registry: Optimise compatibility checking for json to allow more nested schemas. by @BenPope in #29290
Scoped host diskstat metrics to the data and cache directory devices, reporting partition-level and whole-disk I/O separately by @travisdowns in #29893
Shadow Linking: allow creating shadow links with TLS settings where the CA/key/cert strings exceed 128KiB. by @pgellert in #29718
The message printed by rpk config export is now written to stderr instead of stdout, preventing it from interfering with the exported configuration when stdout is used as the output destination. i.e. rpk config export --all -f /dev/stdout by @simon0191 in #28760
The restart is no longer required if the cloud storage auth token is invalidated. by @Lazin in #28556
Updated default console image version in rpk container commands to v3.3.0 by @vbotbuildovich in #28635
Updated default console image version in rpk container commands to v3.3.2 by @vbotbuildovich in #28779
Updated default console image version in rpk container commands to v3.4.0 by @vbotbuildovich in #29151
Updated default console image version in rpk container commands to v3.5.1 by @vbotbuildovich in #29473
Upgrade libxml2 to latest minor version by @tyson-redpanda in #29423
#15206 rpk redpanda admin config log-level set: Added --help-loggers flag to display the list of available loggers without setting any log level. by @r-vasquez in #29601
#15206 rpk redpanda admin config log-level set: now dynamically discovers available loggers from the local Redpanda binary or the Admin API. by @r-vasquez in #29601
fix shard placement table race condition by @joe-redpanda in #28575
kafka/server: Fix a memory leak when the keytab cannot be found. by @BenPope in #28447
kafka/server: Fix a memory leak when the keytab cannot be found. by @BenPope in #28469
kafka/server: Fix rare uncaught exceptional future on connection shutdown by @BenPope in #28445
none by @travisdowns in #29560
partitions with an in-flight move and original replica quorum loss will now be reported as immutable by @joe-redpanda in #28393
potentially better behavior when working with some hardware RAID controllers and portworx by @mmaslankaprv in #28249
prevent partition balancer oscillations by @joe-redpanda in #29554
quality of life logging for shard_placement_table_test by @joe-redpanda in #28697
reduce test failures on data_migrations_api_test by @joe-redpanda in #28325
rpk cluster config set: Poll operation status before returning by @paulzhang97 in #28553
The rpk cluster config set command for Redpanda Cloud clusters now polls for operation completion and displays real-time progress, eliminating the need to manually check status with rpk cluster config status in most cases.
What changed:
- The command now waits up to 10 seconds (configurable via --timeout) for configuration updates to complete
- Progress indication shows elapsed time: Processing configuration... (3s elapsed)
- Clear success/failure messages upon completion
- For operations taking longer than the timeout, the command displays the operation ID for manual status checking
Example: $ rpk cluster config set kafka_connections_max_per_ip 12345 Processing configuration... (3s elapsed) Configuration update completed successfully. Operation ID: abc123
New flag:
- --timeout: Maximum time to wait for operation completion before displaying the operation ID (default: 10s)
This improvement reduces the need to manually check operation status for quick configuration changes while maintaining backward compatibility for longer-running operations.
rpk profile edit, rpk profile edit-globals, rpk shadow update now report when an updated file contains unknown fields. by @r-vasquez in #29245
rpk profile prompt: add 'raw' as a modifier option to our prompt parsing. by @r-vasquez in #29219
rpk redpanda admin brokers decommission: --force flag has been deprecated and renamed to --skip-liveness-check. by @r-vasquez in #29369
rpk shadow create [cloud]: now rpk validates if the secret exists in the Shadow cluster before sending a request to create by @r-vasquez in #29077
rpk shadow describe now shows Schema Registry sync mode. by @r-vasquez in #29023
rpk shadow describe now supports the --format json/yaml flag. by @r-vasquez in #29111
rpk shadow is now supported in Redpanda Cloud by @r-vasquez in #28748
rpk will warn users when they interact with a cloud cluster using an rpk profile that isn't properly configured for cloud. by @r-vasquez in #28594
rpk: add --if-not-exists flag to topic create by @nvartolomei in #28193
rpk: consume warns when requested end offset is lower than available start offset by @daisukebe in #28979