v2.8.2

NEW FEATURES

• add clickhouse.skip_restore_non_sharded_replicated_tables (env CLICKHOUSE_SKIP_RESTORE_NON_SHARDED_REPLICATED_TABLES, default false), clickhouse.skip_check_non_sharded_replicated_tables_cluster (env CLICKHOUSE_SKIP_CHECK_NON_SHARDED_REPLICATED_TABLES_CLUSTER, default {cluster}) and clickhouse.always_sharded_macros (env CLICKHOUSE_ALWAYS_SHARDED_MACROS, default ["shard"]), when true restore, restore_remote and restore --dry-run restore the data of non-sharded replicated tables (ZooKeeper path in the backup DDL resolves to the same value on every shard, macro values are read with cluster(..., system.macros), {database}, {table}, {uuid} are the same for the table everywhere) only on the first replica of the first shard of the cluster from system.clusters, every other node restores only their schema and gets the data through replication, so the same config works on every node, a macro listed as sharded with the same value on all shards is reported with a warning, such tables are never rebound to default_replica_path, not supported with use_embedded_backup_restore and restore_cloud, fix #1181
• use_embedded_backup_restore_cluster now orchestrates the whole cluster instead of only adding ON CLUSTER to the BACKUP / RESTORE SQL: create, create_remote, restore, restore_remote on the initiator node send the same command with the new --embedded-on-cluster-worker flag to every other node of the cluster through its system.backup_actions table (clickhouse-backup server with api.create_integration_tables: true is required on every node) and wait for them, a worker skips the BACKUP / RESTORE SQL and only describes, uploads, downloads and fixes its own shards/N/replicas/M/ part of the shared destination, per table metadata/<db>/<table>.json moved under shards/N/replicas/M/ next to the .sql of each node (such backups are tagged embedded,cluster=<name>, older backups without the tag keep the flat layout), .backup, metadata.json, rbac, configs and named collections stay on the initiator, delete local is propagated to the other nodes as delete local --embedded-on-cluster-worker, so delete accepts the flag too, a single node cluster behaves as before, fix #928
• add azblob.chunk_size (env AZBLOB_CHUNK_SIZE, default 0), a fixed upload block size used when ceil(file size / chunk_size) <= max_parts_count, clamped to [2MiB, 4000MiB], 0 keeps the automatic file size / max_parts_count clamped to [2MiB, 10MiB], brings back the explicit size removed together with AZBLOB_BUFFER_SIZE in v2.6.6, fix #1142
• back up and restore Kafka offsets of Kafka engine tables with kafka_keeper_path / kafka_replica_name (StorageKafka2, ClickHouse 24.8+), previously only the schema was backed up and the restored table re-read from the consumer group offset (lost rows) or from earliest (duplicates); create detaches such tables before FREEZE so the offsets match the frozen data of the MV target (DETACH PERMANENTLY in ENGINE=Replicated databases), attaches them back after FREEZE (also on error and cancel) and stores the Keeper subtree without ephemeral nodes in kafka_keeper field of metadata/<db>/<table>.json; restore writes the offsets to Keeper right before CREATE (after the --rm DROP), when the path already exists without active replicas the offsets are overwritten with a warning, when it has active replicas they are left untouched with a warning; --restore-database-mapping / --restore-table-mapping rewrite kafka_keeper_path, a mapped restore with the same kafka_group_name next to a live source on ClickHouse < 25.6 fails before any Keeper write, because consumer group rebalance would steal the partitions of the source; between restore --schema and restore --data (Altinity ACM restore sequence) consumption is held by persistent partition lock nodes, which the data phase removes, add --skip-kafka-keeper-hold to restore / restore_remote (REST skip_kafka_keeper_hold) to disable it, embedded backups only log a warning, fix #1120
• list remote and restore_remote auto-detect native ClickHouse Cloud backups (BACKUP ... TO S3 / AzureBlobStorage with <name>/.backup and without metadata.json) under s3.path / azblob.path, such backups are listed with data_format: cloud and size instead of broken (can't stat metadata.json), clean_remote_broken and backups_to_keep_remote retention no longer delete them, restore_remote <name> (CLI and POST /backup/restore_remote/{name}) switches to restore_cloud with --tables, --partitions, --skip-empty-tables, --rm, --dry-run and restore_schema_on_cluster support, options which restore_cloud can't apply and download fail explicitly; the base backup chain of incremental cloud backups is resolved from <base_backup> of each .backup (verified by base_backup_uuid) and passed to RESTORE with the configured credentials instead of the ones embedded in the manifest, a base in the same bucket and path is shown as +<base> in list remote and protected by the delete remote guard, fix #1574
• add general.rbac_restore_always, general.config_restore_always, general.named_collections_restore_always (env RBAC_RESTORE_ALWAYS, CONFIG_RESTORE_ALWAYS, NAMED_COLLECTIONS_RESTORE_ALWAYS, default false), restore-side counterparts of *_backup_always, act as --rbac / --configs / --named-collections for restore and restore_remote (including --streaming), disabled by default because these restores overwrite existing objects and run restart_command
• add general.download_transfer_concurrency (env DOWNLOAD_TRANSFER_CONCURRENCY, default 0 = download_concurrency^2, negative keeps the legacy per-table limit), one transfer budget per download shared by every part, archive and required part of all tables, so a single big table can use all the connections at the tail of a download, for FTP the budget is capped by 2*ftp.concurrency; add general.download_table_order (env DOWNLOAD_TABLE_ORDER, largest_first (default) or metadata), only the download dispatch order changes, restore order is not affected, fix #1591
• add general.file_transfer_concurrency (env FILE_TRANSFER_CONCURRENCY, default 1), with compression_format: none files of one data part are transferred by several streams which only take slots of the shared download_transfer_concurrency (download) / upload_concurrency^2 (upload) budget left idle by other parts, so the total number of transfers never grows above the budget, fix #1454
• add gcs.ranged_download_concurrency (env GCS_RANGED_DOWNLOAD_CONCURRENCY, default 0 = disabled), gcs.ranged_download_min_size (default 64MiB), gcs.ranged_download_chunk_size (default 32MiB) and gcs.ranged_download_max_buffers (default 64), archives larger than ranged_download_min_size are read by concurrent byte range requests and fed to decompression in order, without a temporary file, memory is capped process-wide by ranged_download_max_buffers * ranged_download_chunk_size, works with download_max_bytes_per_second, thanks @minguyen9988, fix #1592

IMPROVEMENTS

• GCS: all pooled clients share one cached OAuth2 token, previously each client fetched its own token, so a cold delete remote with gcs.delete_concurrency: 50 opened about 50 parallel TLS handshakes to oauth2.googleapis.com and could fail with auth: cannot fetch token: ... TLS handshake timeout, and every retry after a delete error repeated it
• download with clickhouse.force_rebalance spreads parts round-robin across local JBOD disks whose free space differs by less than 1% of total space (typically several disks on the same filesystem), previously least_used picked the disk with the largest free space from statvfs noise and all parts of a download could land on a single disk
• Docker image: base image changed from alpine:3.23 to alpine:latest and pre-installed packages are upgraded with apk upgrade, the alpine:3.23 base shipped libcrypto3 / libssl3 3.5.7-r0 with CVE-2026-63073 and CVE-2026-75803, thanks @dfdemar, fix #1588
• update go dependencies: aws-sdk-go-v2, azure-sdk-for-go, cloud.google.com/go/storage 1.69.0, clickhouse-go/v2 2.49.0, urfave/cli/v3 3.14.0, golang.org/x/crypto 0.58.0, klauspost/compress 1.20.3 and others

BUG FIXES

• the SQL query log no longer prints passwords embedded in URLs: pass= / password= query string parameters and user:password@ are written as [MASKED], with api.create_integration_tables: true and api.password set every server start logged the DDL of system.backup_actions, system.backup_list and system.backup_version with the API password in plain text inside the URL() engine argument, at info level with the default clickhouse.log_sql_queries: true, the query sent to ClickHouse is unchanged, fix #1630
• s3.chunk_size is now also used when ceil(file size / chunk_size) is exactly s3.max_parts_count, previously it fell back to file size / max_parts_count, ReadMe.md now shows the real default 5242880 and the semantics of s3.chunk_size, raise it to upload big files as fewer bigger parts instead of max_parts_count=1 workaround, each multipart upload allocates about (s3.concurrency + 2) * chunk_size bytes, s3.chunk_size and azblob.chunk_size also apply to uploads into object disks during restore, fix #1142
• watch --env=KEY=VALUE and server --env=KEY=VALUE no longer lose the --env overrides, config reload on every watch iteration (also inside server --watch and POST /backup/watch) read only the config file and the real environment, so e.g. --env=REMOTE_STORAGE=ftp turned back into the config file value and backups went to the wrong remote storage, and server never applied its --env to its own config at all (startup, POST /restart, SIGHUP and every API handler reload); --env is now applied to the process environment once at startup and kept for the whole process lifetime, so /backup/actions commands and operations resumed after restart (api.complete_resumable_after_restart) inherit the server --env too, --env passed inside a /backup/actions command still wins and is restored after the command; a duplicated --env key keeps the last value; server now exits on an invalid config (including a bad --env value) instead of retrying every 5 seconds forever, only the ClickHouse connection is still awaited; general.disable_environment_override: true still ignores --env
• --resume (and use_resumable_state: true) now starts from scratch when the config which defines where the resumable state keys point to changed since the interrupted run: remote_storage, the active storage location (s3 endpoint/bucket/path, gcs endpoint/bucket/path, azblob endpoint_suffix/account_name/container/path, cos url/path, ftp address/path, sftp address/port/path; path is not used by create), object_disk_path, the customer-provided encryption key (s3.sse_customer_key, gcs.encryption_key, azblob.sse_key, only its digest is stored), compression_format for upload / create_remote, and restore_database_mapping / restore_table_mapping for restore / restore_remote --streaming; the state keys are relative to the storage path, so upload --resume after changing s3.path used to skip every file already uploaded to the old path; tuning concurrency, retries, credentials or log_level still keeps the resume; the discarded state is logged as a warning with the changed parameters only; ResumeOperationsAfterRestart now passes --restore-database-mapping / --restore-table-mapping back to the resumed restore / restore_remote --streaming. Upgrade note: resumable states written by previous versions have no config fingerprint and are discarded on the first --resume after upgrading, so an interrupted operation restarts from scratch once, fix #1010
• watch no longer treats a failed backups_to_keep_remote / backups_to_keep_local retention or --delete-source after a completely uploaded backup as a failed upload: the new backup becomes the base of the next increment (previously the next increment was diffed from a stale base) and the failure is not counted towards too many errors ... abort watching, so a permanently failing post-upload step no longer stops watch, it is reported only by the error log, /backup/status, system.backup_actions and the callback, with api.enable_metrics the failed_create_remotes counter keeps growing and last_create_remote_status stays 0 while new backups are uploaded; all post-upload steps now run even when a previous one failed and their errors are reported together; with --schedule the full_type: rebase and delete_previous_cycle steps also proceed, a failed rebase is reported together with the post-upload error and the backup is not used as the base; use_embedded_backup_restore_cluster initiator waits for the worker nodes before the backup is used as the base, a post-upload failure on a worker node is still counted as a failed upload; create_remote --streaming no longer reports such a complete backup as partial; one-shot create_remote and upload still exit with a non-zero code, fix #1535
• restore --schema --rm without restore_schema_on_cluster on a replica of ENGINE=Replicated database, when the database schema was already restored from another replica (ACM restore sequence and How to back up / restore a sharded cluster in Examples.md), no longer fails with code: 529 ... Cannot enqueue query on this replica, because it has replication lag, code: 36 ... It's not allowed to specify explicit zookeeper_path and replica_name or Table ... already exists: local DROP DATABASE of a Replicated database is skipped (it removed only the current replica, which then recovered from the database DDL log concurrently with the following DROP/CREATE TABLE, tables are still dropped and their DROP TABLE is replicated), SYSTEM SYNC DATABASE REPLICA is executed before and after dropping tables (ClickHouse 22.5+), and {uuid} macro of Replicated*MergeTree path is no longer replaced with the backup table UUID and the replication path is no longer rebound to default_replica_path inside a Replicated database, fix #1611
• backup and restore of parts on Azure object disks (azure_blob_storage) now accept every disk configuration which clickhouse-server accepts, any Azure disk in the config which didn't match <storage_account_url> + <account_name> + <account_key> failed every backup with parts on any object disk: a disk without <account_key> (<use_workload_identity>true</use_workload_identity> or a managed identity) is accessed with DefaultAzureCredential, the same way as azblob.use_managed_identity: true, <connection_string> (account key, SAS, BlobEndpoint or EndpointSuffix), <endpoint> with <endpoint_contains_account_name> and <endpoint_subpath>, <container> and <account_name> taken from the URL are supported, fix #1604
• delete local|remote and backups_to_keep_remote retention of backups created with use_embedded_backup_restore: true and embedded_backup_disk on an object disk (s3, azure_blob_storage) now delete the native BACKUP objects once both the local and the remote copy are gone: delete remote never detected the backup as embedded (the cleanup ran over object_disk_path and removed nothing), files of the native backup named *.json (e.g. serialization.json) were skipped, tar/gzip data archives, access, configs and named_collections were parsed as object pointers and failed the cleanup; note delete remote now removes the objects even when another host still keeps a downloaded local copy of the same backup, the same way delete local already did, contributed by @cryptoque, fix #1590
• list and tables text output is now written to stdout as a single write, so a log line on stderr can no longer land in the middle of a backup row when both streams are merged (docker exec, 2>&1), the same fix json / yaml / csv / tsv formats got in v2.8.1
• restore / restore_remote with --restore-database-mapping / --restore-table-mapping of tables on object disks (s3, azure_blob_storage, including cache and encrypted wrappers) no longer leaves the restored copy sharing object storage keys with the source table, so DROP TABLE|DATABASE ... SYNC of the copy no longer deletes the objects of the source (Code: 499 ... The specified key does not exist (S3_ERROR) on the source afterwards): the key rewrite was skipped when the disk endpoint key prefix contained the backup name (e.g. backup regression on the disk https://s3/bucket/regression-test/live/), and the metadata files were linked into detached / the table directory before the keys were rewritten, so an interrupted restore left the copy pointing to the source keys; the rewritten key now also carries a hash of the destination table (<dir>_<backup_name>_<crc32(db.table)>/<object>), so several copies of one backup restored next to each other never share objects, the source key of the copy is always derived from the backup, so the same local backup can be restored several times, and metadata files are written via temp file + rename, so a repeated restore never modifies the parts of an already restored table through their shared hardlink, fix #1568
• upload --resume no longer writes an incomplete manifest.bolt.gz: files of parts, archives and table metadata which the resumable state marks as already processed were skipped without being recorded in the manifest, so the manifest of a resumed upload listed only the files of the last attempt, overwrote the fuller one and download / restore_remote fell back to a Walk (ListObjectsV2) per skipped part. Skipped files are now recorded from the local backup without any extra remote listing; with --delete-source (create_remote) the local part files are already removed, so parts skipped on resume stay out of the manifest and still use the Walk fallback, fix #1569
• SFTP upload of compressed archives (compression_format other than none) now keeps sftp.concurrency writes in flight per file, previously the archive stream from the compressor has no known size, so every 32KiB write waited for the previous one (about 8MB/s per file at 4ms RTT), fix #1607
• SFTP download with download_max_bytes_per_second now keeps concurrent reads per file, the rate limiter wrapper hid WriteTo of the SFTP file, so every 32KiB read waited for the previous one and the download stayed at about 75% of the limit, fix #1607
• GCS ranged download (gcs.ranged_download_concurrency) with download_max_bytes_per_second now reaches the limit instead of about 80% of it, the limit is applied to each range request on the wire instead of the in-order consumer, which waited idle until the first chunks arrived and never made up the lost time, fix #1607
• change default gcs.ranged_download_min_size from 256MiB to 64MiB, with gcs.ranged_download_concurrency enabled archives of 64-256MiB, typical for big parts, were downloaded by a single stream and stayed the tail of the download, fix #1607
• GCS with force_http: true or disable_http2: true no longer opens a new TCP connection for most requests, the HTTP/1.1 transport kept only 1 (64 with disable_http2) idle connection, so a download with many concurrent transfers left closed connections in TIME_WAIT, exhausted the ephemeral port range and failed with connect: cannot assign requested address, fix #1607
• security: download / restore_remote extraction of compressed archives rejects entries whose path escapes the target directory (../, tar-slip, CWE-22) and non-regular entries (symlinks, hardlinks), previously an archive planted in remote storage could write arbitrary files on the ClickHouse host, thanks @tpq2030
• S3 NoSuchUpload is no longer classified as a permanently missing object, so the upload retries with a new multipart upload instead of aborting, NoSuchKey and allow_missing_files_on_download behave as before, thanks @sanasol, fix #1579
• create_remote and upload --delete-source on CIFS/SMB mounts (Azure Files) no longer fail with directory not empty when local retention (backups_to_keep_local: -1) removes the just uploaded backup, the resumable <command>.state2 file is now closed before retention, previously it was still open and CIFS deferred its deletion, GetLocalBackups also closes the backup directory handle when Readdirnames fails, fix #1599
• ClickHouse connection retries no longer leak connection pools, the pool whose Ping failed (wrong password, failed native handshake) is closed before the next retry or before returning the error, previously every retry left its drain goroutine and ticker running, thanks @assimelha
• GCS with gcs.encryption_key falls back to reading not encrypted objects again, GCS returns the reason resourceNotEncryptedWithCustomerEncryptionKey in lowerCamelCase and the check compared it case-sensitively with the UpperCamelCase spelling, so these requests failed
• clean --all keeps shadow/increment.txt, previously it was removed with the rest of shadow/ and concurrent FREEZE queries which recreate it could leave it empty, after that every FREEZE failed with code: 32, message: File /var/lib/clickhouse/shadow/increment.txt is empty
• cleanup of shadow directories recorded in freezes.tmp uses ALTER TABLE ... UNFREEZE WITH NAME when the table is known and SYSTEM UNFREEZE WITH NAME only for dropped tables, previously SYSTEM UNFREEZE was always used on 22.6+ and failed with code: 344, message: Support for SYSTEM UNFREEZE query is disabled when the enable_system_unfreeze server setting is not enabled
• create interrupted by SIGINT / SIGTERM removes its partial local backup again, the cleanup used the already canceled context and failed with context canceled, leaving the backup directory behind