github Altinity/clickhouse-backup v2.8.0
2.8.0

3 hours ago

v2.8.0

NEW FEATURES

  • add rebase command and POST /backup/rebase/{name} API endpoint — copy required parts from the required_backup chain into a remote incremental backup via server-side CopyObject (with streaming fallback) and remove the required_backup dependency, so the incremental backup becomes a full one without re-uploading data from the ClickHouse host; per-table parallelism is controlled by general.rebase_concurrency (env REBASE_CONCURRENCY, default = download_concurrency); requires backups made with upload_by_part: true and the same compression_format across the chain, fix #1344, #1444
  • add general.rebase_before_remove_old_remote (env REBASE_BEFORE_REMOVE_OLD_REMOTE, default false) — makes backups_to_keep_remote a strict limit: when deletion of old remote backups is blocked by required_backup links from kept backups, the oldest kept increment is rebased first (same as the rebase command) so the whole out-of-window chain becomes deletable; rebase failure is not fatal and falls back to the legacy keep-required behavior
  • add --schedule to watch command, general.watch_schedules config section (env WATCH_SCHEDULES, ;-separated) and the schedule query argument to POST /backup/watch — named cron-driven backup chains in name=<name>,full=<cron>[,increment=<cron>][,full_type=create|rebase][,delete_previous_cycle=true|false] format (standard 5-field cron, optional leading seconds field, @every/@daily descriptors), can be specified multiple times, mutually exclusive with --watch-interval/--full-interval; name is added as a prefix to watch_backup_name_template to isolate backup chains, full_type=rebase creates the scheduled full backup as increment + rebase (server-side copy of the previous chain instead of a full re-upload), delete_previous_cycle=true deletes all older backups of the chain after a successful full backup, fix #1354
  • add --list-parts and --partitions (--list-partitions) flags to tables command — additionally list every physical part (name, partition_id, size) or the distinct partitions (partition_id, partition, parts count, size) for each table; against the live server the data comes from system.parts, against --local-backup/--remote-backup part names come from backup metadata (partition_id derived from the part name), fix #959, #1333
  • add --rbac, --configs, --named-collections flags to server --watch — the server command never passed them through, so watch mode backed up only data even though the standalone watch command and the REST API endpoints already supported them, fix #955
  • add rebalance command and POST /backup/rebalance/{name} API endpoint (with --tables and --dry-run support) — move data parts inside a local backup between disks: hardlink from the live part when system.parts shows it on another disk; parts absent from system.parts are copied to the storage-policy disk of the same type with the most free space (unreserved_space - keep_free_space from system.disks), parts on a disk which is absent or left the policy always move; parts on object disks are skipped, fix #1024
  • add gcs.allow_multipart_upload (env GCS_ALLOW_MULTIPART_UPLOAD, default false, experimental) with gcs.upload_concurrency and gcs.multipart_upload_min_size (default 1GB) — files bigger than multipart_upload_min_size upload to GCS as multiple parallel parts (part size is chunk_size) via gRPC client and compose into the final object; not compatible with endpoint, force_http, encryption_key, fix #1028
  • add gcs.allow_multipart_download (env GCS_ALLOW_MULTIPART_DOWNLOAD, default false) with gcs.download_concurrency — download each file as parallel range reads (part size is chunk_size) into a temporary file, mirrors s3.allow_multipart_download, fix #1028
  • add general.disable_environment_override (settable ONLY in the config file, no environment variable name on purpose, default false) — when true, config values come only from the config file: all environment variables and the --env CLI flag are ignored during config loading; protects against accidental overrides such as Kubernetes service-discovery variables, fix #1079

IMPROVEMENTS

  • write a gzip-compressed bbolt file manifest manifest.bolt.gz during upload and read it during download/restore to eliminate the per-part Walk (ListObjectsV2 and analogs) calls to remote storage that dominated restore time on backups with many parts; memory stays bounded even with millions of files (batched writes on upload, mmap-backed prefix seek on read), with graceful fallback to Walk when the manifest is missing or unreadable, and manifest failures are non-fatal for the backup, fix #1375, #1380
  • store per-part size in table metadata during create and check the required free disk space right after table metadata download, before downloading any data — the check respects --tables/--partitions, resolves sizes of required parts through the required-backup chain, and when --hardlink-exists-files is used doesn't count parts which can be hardlinked from existing local parts (matched by hash_of_all_files via system.parts or by CRC64 of checksums.txt), fix #1268
  • fail fast on canceled context in the retrier — SIGTERM / POST /backup/kill no longer trigger a useless extra retry attempt with a misleading Will wait and retry warning per goroutine; during API server shutdown log which in-flight command and cancel_operation_timeout the server is waiting for instead of blocking silently

BUG FIXES

  • fix silent truncation of compressed backup uploads to S3 when s3.check_sum_algorithm: CRC32 (Object Lock buckets): the multipart upload stopped at the declared size (sum of raw file sizes), dropping the tar headers/padding/trailer tail of the archive; the stream is now consumed until EOF, fix #1471
  • fix rbac_conflict_resolution: ignore during restore --rbac — RBAC objects from the backup that already exist on the server are now really ignored: their access/*.sql files are not copied over the live ones and the correlated replicated uuid znodes and name-to-uuid mappings are not restored into Keeper, fix #1013

Don't miss a new clickhouse-backup release

NewReleases is sending notifications on new releases.