v2.8.0
NEW FEATURES
- add
rebasecommand andPOST /backup/rebase/{name}API endpoint — copyrequiredparts from therequired_backupchain into a remote incremental backup via server-sideCopyObject(with streaming fallback) and remove therequired_backupdependency, so the incremental backup becomes a full one without re-uploading data from the ClickHouse host; per-table parallelism is controlled bygeneral.rebase_concurrency(envREBASE_CONCURRENCY, default =download_concurrency); requires backups made withupload_by_part: trueand the samecompression_formatacross the chain, fix #1344, #1444 - add
general.rebase_before_remove_old_remote(envREBASE_BEFORE_REMOVE_OLD_REMOTE, defaultfalse) — makesbackups_to_keep_remotea strict limit: when deletion of old remote backups is blocked byrequired_backuplinks from kept backups, the oldest kept increment is rebased first (same as therebasecommand) so the whole out-of-window chain becomes deletable; rebase failure is not fatal and falls back to the legacy keep-required behavior - add
--scheduletowatchcommand,general.watch_schedulesconfig section (envWATCH_SCHEDULES,;-separated) and theschedulequery argument toPOST /backup/watch— named cron-driven backup chains inname=<name>,full=<cron>[,increment=<cron>][,full_type=create|rebase][,delete_previous_cycle=true|false]format (standard 5-field cron, optional leading seconds field,@every/@dailydescriptors), can be specified multiple times, mutually exclusive with--watch-interval/--full-interval;nameis added as a prefix towatch_backup_name_templateto isolate backup chains,full_type=rebasecreates the scheduled full backup as increment +rebase(server-side copy of the previous chain instead of a full re-upload),delete_previous_cycle=truedeletes all older backups of the chain after a successful full backup, fix #1354 - add
--list-partsand--partitions(--list-partitions) flags totablescommand — additionally list every physical part (name,partition_id, size) or the distinct partitions (partition_id,partition, parts count, size) for each table; against the live server the data comes fromsystem.parts, against--local-backup/--remote-backuppart names come from backup metadata (partition_idderived from the part name), fix #959, #1333 - add
--rbac,--configs,--named-collectionsflags toserver --watch— the server command never passed them through, so watch mode backed up only data even though the standalonewatchcommand and the REST API endpoints already supported them, fix #955 - add
rebalancecommand andPOST /backup/rebalance/{name}API endpoint (with--tablesand--dry-runsupport) — move data parts inside a local backup between disks: hardlink from the live part whensystem.partsshows it on another disk; parts absent fromsystem.partsare copied to the storage-policy disk of the same type with the most free space (unreserved_space - keep_free_spacefromsystem.disks), parts on a disk which is absent or left the policy always move; parts on object disks are skipped, fix #1024 - add
gcs.allow_multipart_upload(envGCS_ALLOW_MULTIPART_UPLOAD, defaultfalse, experimental) withgcs.upload_concurrencyandgcs.multipart_upload_min_size(default 1GB) — files bigger thanmultipart_upload_min_sizeupload to GCS as multiple parallel parts (part size ischunk_size) via gRPC client and compose into the final object; not compatible withendpoint,force_http,encryption_key, fix #1028 - add
gcs.allow_multipart_download(envGCS_ALLOW_MULTIPART_DOWNLOAD, defaultfalse) withgcs.download_concurrency— download each file as parallel range reads (part size ischunk_size) into a temporary file, mirrorss3.allow_multipart_download, fix #1028 - add
general.disable_environment_override(settable ONLY in the config file, no environment variable name on purpose, defaultfalse) — whentrue, config values come only from the config file: all environment variables and the--envCLI flag are ignored during config loading; protects against accidental overrides such as Kubernetes service-discovery variables, fix #1079
IMPROVEMENTS
- write a gzip-compressed bbolt file manifest
manifest.bolt.gzduringuploadand read it duringdownload/restoreto eliminate the per-partWalk(ListObjectsV2and analogs) calls to remote storage that dominated restore time on backups with many parts; memory stays bounded even with millions of files (batched writes on upload, mmap-backed prefix seek on read), with graceful fallback toWalkwhen the manifest is missing or unreadable, and manifest failures are non-fatal for the backup, fix #1375, #1380 - store per-part
sizein table metadata duringcreateand check the required free disk space right after table metadata download, before downloading any data — the check respects--tables/--partitions, resolves sizes ofrequiredparts through the required-backup chain, and when--hardlink-exists-filesis used doesn't count parts which can be hardlinked from existing local parts (matched byhash_of_all_filesviasystem.partsor by CRC64 ofchecksums.txt), fix #1268 - fail fast on canceled context in the retrier — SIGTERM /
POST /backup/killno longer trigger a useless extra retry attempt with a misleadingWill wait and retrywarning per goroutine; during API server shutdown log which in-flight command andcancel_operation_timeoutthe server is waiting for instead of blocking silently
BUG FIXES
- fix silent truncation of compressed backup uploads to S3 when
s3.check_sum_algorithm: CRC32(Object Lock buckets): the multipart upload stopped at the declared size (sum of raw file sizes), dropping the tar headers/padding/trailer tail of the archive; the stream is now consumed until EOF, fix #1471 - fix
rbac_conflict_resolution: ignoreduringrestore --rbac— RBAC objects from the backup that already exist on the server are now really ignored: theiraccess/*.sqlfiles are not copied over the live ones and the correlated replicateduuidznodes and name-to-uuid mappings are not restored into Keeper, fix #1013