redpanda-data/redpanda v23.3.1

on GitHub

Features

• Add recovery mode - an option to start redpanda in "metadata-only" mode, skipping loading user partitions and allowing only metadata operations to enable recovery from fatal misconfiguration or resource exhaustion situations. Enabled by the recovery_mode_enabled node config property. by @ztlpn in #14236
• Add broker support for SASL reauthentication. To enable, config connections_max_reauth_ms > 0. by @oleiman in #13822
• Adds cloud storage scrubbing capabilities to Redpanda. In brief, the scrubber runs in the background and verifies the integrity of the cloud storage metadata and the existence of data referenced by it. When an issue is discovered, the redpanda_cloud_storage_anomalies will increment its counters based on the anomaly type. Per partition anomalies can be queried via the v1/cloud_storage/anomalies/ admin API endpoint. by @VladLazar in #13253
• Add per-shard download throughput limit to tiered-storage by @Lazin in #13552
• Added a Golang SDK for Data Transforms. by @rockwotj in #12322
• Added the ability do disable whole topics or specific topic partitions. Disabled partitions are ignored by Redpanda and producing/consuming to them is impossible. This is useful when only a handful of partitions prevent the cluster from becoming healthy (e.g. if the data in them is corrupted in a way that causes Redpanda to crash). by @ztlpn in #15141
• Adds an admin API to reset the crash loop prevention counter. Additionally the tracking metadata is reset every time the broker boots up in recovery mode. by @bharathv in #15064
• Adds node wise partition recovery functionality. Given an input set of node ids, force reconfigures all partitions that would lose majority if the specified set of nodes are dead. Intended to bulk recover partitions that are stuck when majority of brokers hosting their replicas are dead and irrecoverable. by @bharathv in #13943 and
  #15394
• Admin API: Support OpenID connect Authentication @BenPope in #14378
• Allow template prefix in advertised addresses by @RafalKorepta in #14177
• K8s: Allow to move pods into new nodes if needed, and allow down scaling Redpanda custom resource. by @alejandroEsc in #12847
• Force reconfiguration now supports recovering from an ‘all replicas lost’ scenario. The replicas are reset with empty logs (no data) on the (broker, shard) destinations passed in the reconfiguration command. by @bharathv in #13661
• HTTP Proxy: Support OpenID connect authentication by @BenPope in #14378
• Adds fast partition movement to speed up cluster scaling operations by @mmaslankaprv in #14305
• Implements audit logging capabilities for the admin server. To configure set audit_enabled to True and ensure one of the auditing enabled type in audit_enabled_event_types is set to management by @graphcareful in #14158
• Metadata backup: In addition to topic data, Redpanda will now upload cluster-related metadata, such as cluster configs, security settings, and consumer offsets into cloud storage. This metadata can be restored alongside topic data, allowing for the functional recovery of the entire cluster. by @andrwng in #15188
• Introduce Prometheus metrics for tracking TLS cert/CA expiration by @oleiman in #13477
• Kafka API: Adds SASL/OAUTHBEARER authentication to the Kafka API by @BenPope in #14378
• Redpanda is now compatible with Azure Storage Accounts that enable Hierarchical Namespaces. by @VladLazar in #12632
• K8s: Redpanda Node ID will be reported as label and annotation in Pod metadata by @RafalKorepta in #12524
• Schema Registry: Support OpenID connect by @BenPope in #14378
• You can decode now schema registry encoded messages (AVRO or Protobuf) using rpk topic consume --use-schema-registry by @r-vasquez in #14498
• #12912 Adds rpc_client_connections_per_shard cluster property that allows for the number of clients a broker opens to a given peer to be user configurable. by @ballard26 in #12906
• #12934 #13617 Time-based retention now uses broker-based timestamps for determining when to purge new data. This reduces the risk of retention not removing segments when a misbehaving client produces messages with incorrect timestamps (e.g. a timestamp in the future). by @andijcr in #12991
  • The configuration option storage_ignore_timestamps_in_future_secs is retained to deal with bad segments produced before v23.3
  • This changes the behavior for messages with a timestamp in the past. Before, retention would use this timestamp to delete data. now, the retention window starts when the message arrives in the broker.
• #12989 Schema Registry: Support GET /schemas/ids/{id}/subjects by @oleiman in #13020
• #13229 Schema Registry: Add DELETE /config/{subject} endpoint by @oleiman in #13557
• #14191 Audit Kafka API authentication and authorization events by @michael-redpanda in #14452
• #14394 rpk: adds rpk cluster partitions enable/disable: now users can disable or enable partitions of a topic using rpk. by @r-vasquez in #14909
• #14483 rpk debug bundle: now you can use the --partition flag to request additional debugging information from the admin API for the provided partitions. by @r-vasquez in #15136
• #9128 K8s: New Topic Custom Resource that can manage single topic by @RafalKorepta in #11208
• #9205 rpk: new 'rpk cluster partitions move' command to reassign replicas by @daisukebe in #13684
• #9205 rpk: new ‘rpk cluster partitions move-status’ to show ongoing partition movements by @daisukebe in #13258
• adds rpk redpanda mode recovery command to put redpanda into recovery mode. by @r-vasquez in #14431
• ability to change number of partitions in tx manager topic by @mmaslankaprv in #15121
• adds support to the Cluster API (deprecated) for a user-specified ServiceAccount by @joejulian in #12864
• adds support to the Console API (deprecated) for a user-specified ServiceAccount by @joejulian in #13120
• crash_loop_limit now defaults to 5. If a broker uncleanly shutdowns for 5 times back to back, it is considered to be in a crash loop mode and Redpanda refuses to start up and may need manual intervention. This enforcement is disabled in developer mode and rpk's dev-container mode. by @bharathv in #13431
• rpk introduces rpk cluster partitions list which lets users query cluster-level metadata of all partitions in the cluster. by @r-vasquez in #14862
• rpk now has a cluster txn command space by @twmb in #7557
• rpk now supports encoding messages with a given schema stored in the schema registry when producing via rpk topic produce. by @r-vasquez in #13543
• rpk: Introduce a new flag --label-selector to rpk debug bundle and have a way to filter the bundled resources by label. by @r-vasquez in #15185
• rpk: new rpk registry command to manage schema registry with rpk by @r-vasquez in #12669
• PR #14283 Compaction lossy hash by @dotnwat
• PR #14328 storage: implement sliding window compaction by @andrwng
• PR #14368 storage: utilities to implement sliding window compaction by @andrwng
• PR #14912 storage: use hash map in compaction by @andrwng
• PR #15269 storage: limit and account for memory used for compaction by @andrwng
• PR #15404 storage: further improvements to windowed compaction short circuit by @andrwng