v2.8.1

NEW FEATURES

• add clickhouse.drop_replica_if_exists (env CLICKHOUSE_DROP_REPLICA_IF_EXISTS, CLI --drop-replica-if-exists for restore/restore_remote, also accepted as the drop_replica_if_exists query argument by POST /backup/restore and POST /backup/restore_remote) — when the replica entry <replica_path>/replicas/<replica_name> of a restored Replicated*MergeTree still exists in ZooKeeper but no local table uses that path, execute SYSTEM DROP REPLICA '<replica_name>' FROM ZKPATH '<replica_path>' and restore the table with the ORIGINAL replication path from the backup, instead of the default behavior which silently rebinds the table to clickhouse.default_replica_path / clickhouse.default_replica_name; requires ClickHouse 20.4+, the restore fails when the drop is impossible (older server, or a local table still occupies the path), the statement is executed without ON CLUSTER because the path and replica name are resolved with the macros of the current node; MUST stay false during a concurrent multi-replica restore, fix #1162
• add --streaming flag to create_remote, restore_remote and watch, --watch-streaming flag to server and the streaming query argument to POST /backup/create_remote, POST /backup/restore_remote and POST /backup/watch — process the backup table by table: create_remote --streaming runs freeze -> upload -> remove local copy per table, restore_remote --streaming runs download -> attach -> remove local copy per table, so only the tables in flight occupy local disk space and backups bigger than the free local space can be uploaded or restored; the local backup is not kept afterwards; interrupted runs are continued with --resume via the dedicated create_upload_streaming.state2 / download_restore_streaming.state2 state files (add these command names to api.complete_resumable_after_restart_commands to resume them after an API server restart); not available with use_embedded_backup_restore: true or remote_storage: custom, fix #780
• add restore_cloud command and POST /backup/restore_cloud REST API endpoint — restore a ClickHouse Cloud native S3 backup (created via external backup / Bring Your Own Backup) onto self-managed ClickHouse: reads the .backup manifest directly from S3 (including data_file_name_generator: Checksum blob layout), rewrites ENGINE = Shared databases to Atomic and Shared*MergeTree tables to the matching Replicated*MergeTree (adding --replicated-zk-path/--replicated-replica arguments when Cloud DDL has none), applies the DDL and runs RESTORE TABLE ... FROM S3(...) / AzureBlobStorage(...) with allow_different_database_def=1, allow_different_table_def=1; source storage is S3-compatible (AWS, GCS over s3->endpoint: https://storage.googleapis.com with HMAC keys, MinIO) or AzureBlobStorage (when --container / --azblob-restore-url is passed or general->remote_storage: azblob); supports --tables filter, --base-prefix for incremental backups, --s3-restore-url / --azblob-restore-url overrides, --skip-empty-tables, --continue-on-error, --drop, --parallel and --dry-run; note: backups containing Packed parts (data.packed) require ClickHouse 26.8+ on the target server, fix #1508
• add --dry-run flag to create, create_remote, upload, download, restore, restore_remote, delete commands and the dry-run (dry_run) query argument to the matching REST API endpoints — instead of performing the operation, report the number of tables, parts and the data size (split into local-disk and object-disk bytes, human readable in the log) which would be processed, honoring --tables, --partitions, --diff-from/--diff-from-remote, --schema and the other filters; create --dry-run additionally forecasts the disk space the backup hardlinks will occupy over time (bytes held in parts younger than 1/7/30 days approximate what background merges will rewrite within that horizon, up to the full local data size in the worst case) and delete --dry-run reports the dependent incremental backups; API dry-run requests execute synchronously, are allowed even when another operation is in progress regardless of api.allow_parallel, and return the report as JSON in the response body; every dry-run also fills the new result field of its status row, so the report is visible via GET /backup/status, GET /backup/actions and the system.backup_actions table (which gained a result String column, manually created tables need it added or input_format_skip_unknown_fields=1); estimation limits: create doesn't subtract --diff-from-remote increments, create_remote/upload can't know the compressed size before compression, parts of backups made by versions older than 2.7.0 have no per-part sizes and are counted in unknown_size_parts, fix #1012
• add general.callback_url (env CALLBACK_URL) and general.callback_timeout (env CALLBACK_TIMEOUT, default 5s) — HTTP POST completion notification for API, one-shot CLI commands, and each watch iteration; API ?callback= overrides the global URL when non-empty; the payload keeps the existing status/error/operation_id fields and adds command and duration; status is now also cancel when the operation was killed via /backup/kill; callbacks are sent asynchronously, failures are logged and never change the backup result; one-shot CLI commands and watch iterations are now registered in /backup/status like API operations, fix #1481
• add general.status_history_size (env STATUS_HISTORY_SIZE, default 1000) — upper bound on how many finished operations are kept in the in-memory status list exposed by /backup/status and system.backup_actions; needed because watch now records one operation per iteration, so the history would otherwise grow for as long as the process lives; operations still running are never dropped, whatever their age
• delete local|remote <backup_name> and POST /backup/delete/{where}/{name} now refuse to delete a backup which other backups require via required_backup and report the dependent backup names, instead of silently breaking the incremental backups chain (the breakage surfaced only later, when a descendant was downloaded or restored, and for object disks the descendant required parts blobs were deleted together with the parent); pass --force (force=1 for the API) to get the old behavior, or set general.rebase_during_delete: true (env REBASE_DURING_DELETE, default false) to rebase every dependent increment first (same as the rebase command) so the chain stays restorable — rebase copies the deleted backup parts into its dependents, so deletion time grows with the copied data size and a rebase failure aborts the delete. backups_to_keep_local/backups_to_keep_remote retention is not affected, fix #1493
• restore --rbac now converts RBAC objects between local_directory and replicated user directories, when the target server has only one of the two shapes: a Keeper dump from the backup is written as <access_control_path>/<uuid>.sql files when the target has no replicated user directory, and backup *.sql files are written to Keeper (<zookeeper_path>/uuid/<uuid> plus the <zookeeper_path>/<type char>/<name> index znode) when the target has no local_directory user directory; previously the half which had no matching storage was silently lost while the restore still reported success; restoring a backup with RBAC objects onto a server which has neither user directory now fails explicitly, fix #881

IMPROVEMENTS

• support tables which declare their storage inline in the DDL via SETTINGS disk = disk(...) instead of a storage_policy (ClickHouse 23.2+, tested from 24.8), including nested cache and encrypted wrappers over an object storage disk; such disks are registered by ClickHouse under a generated __tmp_internal_<hash> name (or under name = '...') and are absent from preprocessed_configs/config.xml, so create/upload/download/restore used to fail with __tmp_internal_... is not present in object_disk.DisksCredentials; object storage credentials are now taken from the table DDL and the target disk is resolved again after the schema is restored, so the generated name may differ between the source and the target server. Note that the DDL stored in the backup metadata contains the secret_access_key in plain text, same as for S3/MySQL engine tables (#640), and that clickhouse-backup needs to run on the ClickHouse host (to read /var/lib/clickhouse/metadata/<db>/<table>.sql) or the server needs display_secrets_in_show_and_select=1, because SHOW CREATE TABLE masks disk(...) arguments as '[HIDDEN]', fix #943
• fail fast instead of burning the whole retries_on_failure x retries_duration backoff budget per file when a remote object is permanently missing (S3 NoSuchKey/404, GCS 404, Azure BlobNotFound, FTP/SFTP not-found) during download, restore_remote and other retried remote operations; add general.allow_missing_files_on_download (env ALLOW_MISSING_FILES_ON_DOWNLOAD, default false), --allow-missing-files CLI flag for download/restore_remote and the allow_missing_files query argument for POST /backup/download and POST /backup/restore_remote — salvage mode which skips data parts missing on remote storage with an error-level log and a final summary, drops them from local table metadata and lets the intact tables/parts of a partially corrupted backup be restored, metadata files are never skipped, fix #1456
• add s3.delete_batch_fallback_to_single (env S3_DELETE_BATCH_FALLBACK_TO_SINGLE, default true) and s3.delete_batch_min_size (env S3_DELETE_BATCH_MIN_SIZE, default 0) — when a whole DeleteObjects batch fails (some S3-compatible gateways such as DigitalOcean Spaces / Ceph RGW reset the response stream when the batch contains large objects, so retrying the same batch never succeeds and blocks retention and the watch loop), the batch is split in halves down to delete_batch_min_size and finally its objects are deleted one by one with DeleteObject, s3.delete_concurrency in parallel; general.delete_batch_size is now validated (1..1000 for s3), fix #1532
• download --hardlink-exists-files no longer does per-part filesystem and ClickHouse lookups, which dominated the download time on servers holding many local backups or many parts. Two changes: the shadow directories of the local backups are now indexed once per download run from their table metadata, so finding a hardlink candidate for a backup carrying legacy CRC64 checksums costs one map lookup plus one stat instead of a filepath.Glob over <disk>/backup/*/shadow/... (which ran twice per part, once for the free space check and once for the download itself); and the hash_of_all_files lookup in system.parts is now read once per table in chunks of 1000 hashes instead of one SELECT per part. Both paths keep their previous results: an indexed candidate is still verified by the CRC64 of its checksums.txt before being hardlinked, a local backup which can't be indexed (broken backup, unreadable metadata) marks the index incomplete and restores the old glob, and a system.parts candidate which a merge removed after the snapshot was taken is detected before hardlinking and re-resolved by a single live query for that part, fix #1457

BUG FIXES

• FTP: MkdirAll now treats an already existing directory as success and remembers it in its directory cache, instead of logging MkdirAll MakeDir(...) return error: 550 "...: File exists" and re-issuing MKD for the same directories for every single uploaded object (hundreds of serialized round-trips per create/upload of an object disk backup); a MakeDir failure for any other reason is no longer swallowed, it now fails the upload with the real error instead of letting STOR fail later with a confusing 550 ... No such file or directory. A failed STOR also drops the cached directories of its path and is no longer classified as a permanently missing object, so it is retried (re-reading the source object) instead of aborting the whole backup on the first transient FTP error, fix the TestFTP CI failure of #1559
• download / restore_remote of an incremental backup no longer fail with <part> not found on <required_backup> and all required backups sequence when the local copy of an intermediate backup of the chain lists only a subset of its parts; the local backup/<name>/metadata/<db>/<table>.json of a required backup could have been narrowed by an earlier download --partitions=..., by allow_missing_files_on_download or by disk rebalancing, but it was trusted as the description of which backup of the chain physically holds a part, so the resolution stopped at that backup instead of recursing to its own required backup; the required backup table metadata is now always read unfiltered from remote storage and the local file is neither read nor overwritten, fix #1045
• download of an incremental backup no longer fails with filepath.Walk: Link in walk: link ...: file exists when a local backup already holds a copy of a required part from an earlier download: the required backup that copy was hardlinked from can be removed by the partial-required-backup cleanup and downloaded again later, which leaves the local copy pointing at the old inode; the freshly downloaded copy is now hardlinked over the leftover instead of aborting the download. A resumed download which already recorded that part in download.state2 still skips the re-download and keeps the leftover as it is, fix #1045
• restore --rbac / restore --rbac-only no longer fails with zk: node already exists or zk: version conflict when several replicas which share a replicated user directory restore the same backup at the same time (typical when restore_remote --rbac is fired via the REST API on every node of a cluster) — the Keeper writer used a non-atomic get-then-create/set and treated a concurrent writer of a byte identical znode as an error, znodes are now written with an unconditional set-or-create and dropExistsRBAC tolerates znodes already removed by another replica, so every replica exits successfully and the final RBAC state is the same everywhere; rbac_conflict_resolution: fail still fails on the second replica by design, and for replicated-only clusters set clickhouse: restart_command: "sql:SYSTEM RELOAD USERS" so the concurrent restores don't restart every replica at once, fix #1048
• create no longer leaves shadow/<uuid> orphans forever when the process is killed (OOM, SIGKILL, pod eviction) between FREEZE and UNFREEZE: every FREEZE ... WITH NAME <uuid> is now recorded in <backup_name>/freezes.tmp (bbolt) until the table is unfrozen, and clean, delete local, clean_local_broken and the local retention unfreeze the recorded shadow of backups which are not processed by a live clickhouse-backup process (pid file or freezes.tmp lock) via SYSTEM UNFREEZE WITH NAME (22.6+), ALTER TABLE ... UNFREEZE WITH NAME (21.4+) or direct removal; clean is safe by default now and no longer wipes the whole shadow/ (which destroyed the data of a concurrent create and manual FREEZE), clean --older-than=24h additionally removes unrecorded shadow/* directories not modified for that long (orphans of previous versions), clean --all keeps the old wipe-everything behavior, clean --dry-run only logs; the cleanup waits until the FREEZE queries of the interrupted command left system.processes and re-checks that shadow/<uuid> stays removed, otherwise the server recreated it right after the removal; POST /backup/clean accepts older_than, all and dry_run query parameters and clean --older-than=24h --all --dry-run works via POST /backup/actions, fix #1563
• server no longer subscribes SIGINT to the SIGHUP reload channel, so Ctrl+C stops the server directly instead of sometimes reloading the config and canceling all commands twice before stopping
• SIGINT/SIGTERM now also stop a command waiting in the infinite ClickHouse reconnect loop (added for #857): the reconnect gives up as soon as the process is asked to terminate instead of retrying every 5 seconds forever, so kill and timeout no longer hang on an unreachable or incompatible server
• one-shot CLI commands handle SIGINT/SIGTERM: the first signal cancels the running command so create unfreezes its shadow and unwinds, the second one exits immediately; previously only server and watch handled signals and a kill -TERM clickhouse-backup create left shadow/<uuid> behind, fix #1563
• restore no longer hardlinks default disk parts into an object disk metadata tree (ATTACH PART then failed with code: 27 ... Cannot parse input: expected '\n' before: 'checksums format version: 4') when clickhouse-server is started with a relative <path>./</path> (ClickHouse 25.x and older report relative paths in system.disks.path and system.tables.data_paths, 26.8 canonicalizes them). Such relative disk paths are now resolved against clickhouse.disk_mapping["default"], which is the clickhouse-server working directory, and system.tables.data_paths get exactly the same substitution, so a data path always matches its own disk; when the paths can't be resolved because disk_mapping is not set, create/upload/download/restore fail up-front with a message naming the relative disks instead of silently corrupting the restore. The same root now also resolves the relative system.databases.data_path / system.tables.metadata_path (previously a leading / was forced onto them, so preprocessed_configs, the RBAC access directory and flags/force_drop_table were looked up at the filesystem root and create failed with can't parse config.xml from /preprocessed_configs/config.xml) and a relative access_control_path. Additionally a table data path which matches no disk path at all no longer overwrites the default disk entry of a data path which really matched it, and a relative clickhouse.disk_mapping value is now rejected at config validation instead of silently disabling the resolution, fix #1121
• restore no longer merges data into a leftover detached/<part> directory: parts are now hardlinked into a clickhouse_backup_tmp_<part> directory and renamed into place, so ClickHouse never observes a half-built part, and a same-named detached/<part> left by a previous failed restore or a manual ALTER TABLE ... DETACH PARTITION is renamed to detached/ignored_<part> (logged at warn) on every local disk of the destination table before the restore, because ATTACH PART takes the first disk of the storage policy which has such a directory and a stale copy on another disk shadowed the freshly restored one (it failed with No columns.txt in part or silently attached stale data); the renamed directory keeps a manually detached part visible in system.detached_parts and re-attachable, while ATTACH PARTITION skips it; parts on plain/plain_rewritable object disks are restored on the bucket level and are not covered; download of an incremental backup no longer aborts with non empty `files` can't find disk when the rebalanced disk carries only required parts (they are uploaded with the base backup, so they legitimately have no files entry in the increment); the error raised when the destination table storage policy does not contain a disk the backup needs now names the table, the part and the disk and points at SELECT * FROM system.storage_policies instead of dumping a Go map, fix #1034
• upload / create_remote no longer keep the relative paths of every file of a table in memory for the whole upload — splitFilesByName walked all parts of the table up front and the resulting list (which also retained the full .../shadow/<db>/<table>/<disk>/ prefix of every path via strings.TrimPrefix) lived until the last part was uploaded, so tables with 1000+ columns and thousands of Wide parts (millions of files) drove clickhouse-backup server to 4 GB of live heap and 10 GB RSS; with upload_by_part: true (default) the files of a part are now listed inside its upload goroutine right before it is archived, so at most upload_concurrency part file lists are alive, and parts already recorded in the resumable state skip the directory walk; upload_by_part: false still lists all files (it has to split by max_file_size) but detaches the relative path from the full path (~3x less memory per file), fix #1550
• a zero-byte "folder placeholder" object whose key equals the remote path with a trailing slash (created by cloud consoles "Create folder", aws s3api put-object --key <prefix>/ and similar external tools, never by clickhouse-backup itself) was listed as a broken backup with an empty name, and backups_to_keep_remote retention (also clean_remote_broken) deleted that entry with path.Join("", ...) collapsing to the remote path root, wiping every remote backup; list remote now skips such nameless entries with a warning and remote deletion refuses an empty backup name, fix #1524
• create on a plain_rewritable disk no longer fails with NoSuchKey when a __meta/<token>/prefix.path object is deleted between the __meta/ listing and its read (a background merge or a concurrent DROP removes directories on the live disk); the vanished token is skipped the same way ClickHouse itself would not see the directory when building its path map at startup, any other read error still fails the backup
• flags placed after positional arguments are no longer silently ignored — download my_backup --tables db.huge_table downloaded the whole backup because urfave/cli v1 stops flag parsing at the first positional argument, and the same applied to every command flag after a backup name (upload x --delete-source, restore x --rm, ...); the CLI migrated from urfave/cli v1 to v3, which parses flags at any position, and global flags (--config/-c, --env) now also work both before and after the command name; v1 behavior is preserved where it matters: comma-containing values of --partitions, --schedule, --env are not split into multiple values (v3 default splitting is disabled), flag names, aliases, defaults and CLICKHOUSE_BACKUP_CONFIG stay unchanged, and each in-process API re-entry builds a fresh command tree so parallel API commands can't leak flag values into each other; visible differences: --help output renders typed placeholders (--tables string instead of --tables value), moves global flags into a per-command GLOBAL OPTIONS: section, and an explicit --rebind-replica-path-if-exists=false no longer overrides rebind_replica_path_if_exists: true from the config file, fix #1459
• create --schema and --rbac-only / --configs-only / --named-collections-only backups no longer require remote storage when tables live on an object disk — the remote backup destination was created and connected even though no data is copied, so a local metadata-only backup failed with storage type 'none' is not supported when remote_storage: none, fix #1517
• GET /backup/kill and kill in POST /backup/actions now actually cancel commands which were started through POST /backup/actions — such commands re-enter the CLI app in process with --command-id passed before the command name, so the value lands in the application flag set, but every command action read it with c.Int("command-id"), which in urfave/cli v1 only looks at the command's own flag set (each command re-declares the application flags via Flags: append(cliapp.Flags, ...), so it saw the -1 default). The command therefore ran with a fresh background context instead of the one owned by its status row and ignored cancellation; broken since 0a26ee69 (v2.1.0), which moved command-id from the commands to the application but kept the c.Int lookups
• drop a stale upload.state2 when upload --resume runs for a backup which doesn't exist on remote storage anymore — the resumable state survives a successful upload and is removed only together with the local backup, so create_remote --resume + delete remote + upload --resume skipped every data file and uploaded a backup containing metadata.json only, which looked valid in list remote; an interrupted upload still leaves the backup folder on remote, so a real resume is not affected, fix #1492
• keeper.Connect (used to back up and restore user_directories.replicated RBAC objects and named_collections_storage of a keeper* type) failed with /zookeeper/node not exists in .../preprocessed_configs/config.xml when the <zookeeper> section spells its endpoints with any element name other than exactly node. ClickHouse itself accepts every child whose name starts with node (<nodes>, <node1>, ...), and the ClickHouse Kubernetes operator renders its zookeeper: nodes: [...] YAML config as repeated <nodes> elements, so on such clusters create with the default rbac_backup_always: true / named_collections_backup_always: true died via log.Fatal on every run and server --watch with watch_is_main_process: true restarted forever without producing a single backup; the parser now mirrors the server rule and the error messages name the element which was actually found. Keeper digest auth is now also taken from <zookeeper><identity>, the element ClickHouse itself reads, in addition to the <digest> element this tool looked for before, and the new clickhouse.keeper_identity (env CLICKHOUSE_KEEPER_IDENTITY) setting supplies it when the server config hides the value from preprocessed_configs (hide_in_preprocessed, from_env, which the ClickHouse Kubernetes operator does) — without it the connection succeeded and every read failed with zk: not authenticated