zpaqfranz 65.17

This is the release of Proxmox.
Until now, on Proxmox, zpaqfranz saved only
the disks of the VMs that were files in /var/lib/vz. Of the other VMs, just the
.conf, with an exit code that looked like a backup. Containers: nothing.

Now every guest, VM or container, on every kind of storage, in two ways, each
one with its restore. A version each run, deduplicated.

zpaqfranz proxvzdump    /backup -all
zpaqfranz proxvzrestore /backup/proxvzdump-qemu-100.zpaq 999 -storage local-zfs

zpaqfranz zfsproxbackup  /backup -all
zpaqfranz zfsproxrestore /backup/zfsproxbackup-qemu-100.zpaq 999 -storage local-zfs

• proxvzdump, proxvzrestore: vzdump straight into zpaqfranz, no
  temporary file, any storage, running guests too. The restore is simple: one
  command, qmrestore or pct restore fed by a pipe.
• proxvzdump -proxmox: two runs of a guest give the same bytes. A run with
  nothing changed costs KB, not MB. See the note below before you use it.
• zfsproxbackup, zfsproxrestore: rewritten. The disks read as they are,
  from a snapshot where the storage has one (ZFS, LVM-thin, btrfs). A run with
  nothing changed costs KB.
• work proxmox: a small manager for a Proxmox of the LAN, a menu by
  numbers, from any zpaqfranz, Windows too.
• work prox user, work prox start: a user and a token that can do one
  thing, start one guest. A client that uses them needs no libcurl.

Restoring ZFS streams: go slowly

The restore of ZFS streams and volumes (zfsreceive, zfsrestore,
zfsproxrestore) needs more testing. It works on the test machines, it has not
yet been through what real machines will do to it.

And a restore is not a read. What it replaces is gone, and if something goes
wrong halfway there is no key that brings it back. So, with care, in a safe
place:

• Try it first on a machine you can afford to lose, one you can roll back. Not
  on the host that matters, not for the first time on the day you need it.
• Restore to a new dataset or a new vmid, not over the original. Look at
  what came out. Only then decide.
• Before replacing anything, have another copy of it, made in another way.
• zfsreceive and zfsrestore write a script: read it before running it.
• -force is the switch that puts a restore on top of what is there. It does
  not belong in a cron.

Details in point 6 of part one.

-proxmox, in plain words

Without -proxmox the stream of vzdump stays whole: what goes in the archive
is what vzdump wrote, byte by byte. What comes out is the file Proxmox knows.

With -proxmox the stream is changed on its way in, so that two runs look
alike. It is still a good VMA or a good tar, and in every test made so far the
guest that comes back is the same. But it is no longer the file of vzdump.

So leaving it out is the safer choice, until -proxmox has had more checks and
more tests. The price is space: MB each run instead of KB. Nothing else.

And:

• mysqldump rewritten, mysqlrestore new. No more script behind it. Exit
  code 0 only if every dump is whole: before, a wrong password, no database at
  all and a dump cut in half were all "OK".
• The zfs commands reviewed, on a real pool: one archive per dataset, a
  second run takes only the new snapshots, Franzen, datasets at any depth, -ttl.
• r -paranoid: what is copied is read back from the media, by pieces, during
  the copy.
• -m8, the default, no longer makes a file larger. A JPEG in a small archive
  paid for a decoder it did not use.
• -m 5, -t 2 with the space work. A wrong command line stops with exit
  code 2, it no longer shows the help with exit code 0.
• Bugfix: many

Part one, what is new, for the user

1. Proxmox: which one
2. proxvzdump and proxvzrestore
3. proxvzdump -proxmox: KB, not MB
4. zfsproxbackup: the disks as they are
5. zfsproxrestore
6. Restoring ZFS streams: the care it takes
7. work proxmox, work prox
8. The zfs commands
9. mysqldump, mysqlrestore
10. r -paranoid
11. -m8 and what does not compress
12. The command line
13. Bugfix

1. Proxmox: which one

In doubt, proxvzdump without -proxmox: the file of vzdump, unchanged, and
the tools of Proxmox to bring it back. It is the plain road, and the one with
the least new code on it.

Which binary. None has Proxmox in its name. These two were run on a Proxmox VE 9.2,
as they are, autotest and proxvzdump -proxmox: zpaqfranz_oracle79 (the old glibc
it wants is there everywhere, the first choice) and zpaqfranz_nas_x86_64 (static, no
Franzen). Or build it on the node:

apt install build-essential
g++ -O3 -Dunix zpaqfranz.cpp -o zpaqfranz -pthread -ldl

2. proxvzdump and proxvzrestore

zpaqfranz proxvzdump /backup 100 200                 a VM and a container
zpaqfranz proxvzdump /backup -all -not 300           every guest but one
zpaqfranz proxvzdump /backup -all -key pippo         encrypted
zpaqfranz proxvzdump /backup/all.zpaq -all           all in one file

• A folder (it must be there, 79556!): one archive per guest,
  proxvzdump-qemu-100.zpaq, proxvzdump-lxc-200.zpaq. If you delete a guest, you
  delete its backups in one go.
• A file (.zpaq, .franzen): every guest in it, the best dedup.
• Each run is one version, with a comment: proxvzdump qemu 100 web 2026-10-31_20:14:22.
  zpaqfranz i archive lists them.
• -key, -franzen, or both.
• A run is good only if vzdump says 0, a says 0 or 1, and the archive grew.
  If not, the archive goes back to what it was, the last lines of the log of
  vzdump are shown, exit code 2.

The restore:

zpaqfranz proxvzrestore /backup/proxvzdump-qemu-100.zpaq -force            VM 100 back, as it was
zpaqfranz proxvzrestore /backup/proxvzdump-qemu-100.zpaq 999 -storage s1   a new VM 999, with a new MAC
zpaqfranz proxvzrestore /backup/proxvzdump-lxc-200.zpaq -until 3 -force    the version 3 of the CT 200
zpaqfranz proxvzrestore /backup/all.zpaq 100 -force                        from a file with many
zpaqfranz proxvzrestore /backup/all.zpaq 100 999                           from a file with many, as a new one

No temporary file: the stream goes from the archive to qmrestore or pct restore
by a pipe. A vmid that is already there wants -force, and -force means
that guest is overwritten. Without a good reason, give a new number.

3. proxvzdump -proxmox: KB, not MB

vzdump writes a different file each time, even if the guest did not move. To a
deduplicator every difference costs. What each run costs, in the archive, with
nothing changed in the guest:

With something changed you pay for what changed: 15 MB new, +15.2 MB.

zpaqfranz proxvzdump /backup -all -proxmox

• proxvzrestore has no such switch. It reads both kinds, and an archive can
  hold both. An older zpaqfranz reads them too.
• The first run with -proxmox on an old archive pays once, a whole increment
  (10 MB for that VM, 72 MB for that container). After that, KB.
• If the stream is not what it should be, nothing goes through: the run fails and
  the archive stays as it was.

The note again, because it matters. Without -proxmox the archive holds the
stream of vzdump as it is, whole, untouched. With it, the stream is rewritten
on its way in. It has been tried on ten guests, on every storage, and every
restore was the same. But ten guests on a test machine are not the world, and the
safer choice, until there are more checks and more tests, is to leave it
out and pay the MB.

4. zfsproxbackup: the disks as they are

Rewritten, with a new syntax. The old one is gone, and the old archives can
still be restored.

zpaqfranz zfsproxbackup /bak 200 300                 two guests
zpaqfranz zfsproxbackup /bak -all -not 300           every guest but one
zpaqfranz zfsproxbackup /bak -all -key pippo         encrypted
zpaqfranz zfsproxbackup /bak/all.zpaq -all           all in one file, the best dedup

Folder or file as in proxvzdump: zfsproxbackup-qemu-ID.zpaq,
zfsproxbackup-lxc-ID.zpaq. No vzdump here: a run is a version with the .conf
and every volume of the guest, read from a snapshot taken on the spot.

• A container on a ZFS subvolume goes file by file, with -tar: owners, links,
  everything. On a raw image, rootfs.raw.
• With agent: 1 the file systems of the guest are frozen during the snapshot. If
  it fails, or there is no agent, the backup is what a sudden power off would
  leave, and it says so.
• Nothing changed: +37 KB for a VM, from +6 KB to +58 KB for a container. One run of
  a container, once, cost +1 MB: it could not be made to happen again. 15 MB new:
  +15.2 MB.
• What a dead run left behind (snapshots, clones, with our names) is cleaned by the
  next one.

5. zfsproxrestore

zpaqfranz zfsproxrestore /bak/zfsproxbackup-qemu-200.zpaq 999 -storage local-zfs    as a new VM 999
zpaqfranz zfsproxrestore /bak/zfsproxbackup-qemu-200.zpaq -until 3 -force           the version 3
zpaqfranz zfsproxrestore /bak/all.zpaq 200 -force                                   from a file with many

Read the next point first.

• The volumes are always new ones. The data go there, then the config, and
  the old volumes go away only if all of this went well. If it fails, the new
  ones are removed and the guest is as it was.
• A guest that is running is refused. With -force too.
• Before each write the target is checked: see
  part three.
• -storage S: another storage. Disks moved from a kind of storage to another
  were tried, in both directions.
• Old archives: as ever, VMIDs, -not, -kill, and their paths are checked
  before anything is extracted.

6. Restoring ZFS streams: the care it takes

The backup is the easy half. If zfsbackup fails you have a failed backup, and
your data are where they were. A restore does the opposite: it takes the place of
something, and that something does not come back.

The restore of ZFS streams and volumes needs more testing: zfsreceive,
zfsrestore, zfsproxrestore. What has been tried is in
How it was tested: test pools, test VMs, a single node. No
production host, no pool of real size, no failure in the middle of a large restore.

So, operate with care, and in a safe place:

1. A place where a mistake costs nothing. A test VM, a host you can roll back.
   Learn what the command does there, not on the machine you are trying to save.
2. A new name. A new dataset, a new vmid. The restored data sit next to
   the original, and you compare.
3. Check what came out. Boot the VM, enter the container, zfs diff, compare
   a few files you know. An exit code 0 is a good sign, not a proof.
4. Another copy. Before touching the original, a zfs snapshot, a vzdump of
   your own. It takes a minute.
5. -force last, by hand, looking at the screen. It is not a switch for a cron.
6. Space. New volumes are made next to the old ones: the storage must hold both.

If something does not look right, stop, and report it. Those reports are the
further testing this needs.

7. work proxmox, work prox

zpaqfranz work proxmox 192.168.1.2

A small manager for a Proxmox of the LAN, from any zpaqfranz, Windows too. A menu
by numbers: guests and their disks, storages (as df -h), start, halt, stop,
users, backup check. Only the API, no ssh, no libcurl. The certificate is shown
before the password is sent; to skip the question give its SHA-256.

zpaqfranz work prox user 101
zpaqfranz work prox start 192.168.1.2 101 "zpaqstart-101@pve!start=uuid" 4FCEE2BD...

work prox user prints the text for the console of Proxmox: a user and a token
that can only start the guest 101, and the line to give to the client.
work prox start uses them, and waits for the guest to run. Say, a NAS or a
backup machine that must wake up a VM, and must not be able to do anything else.

8. The zfs commands

Reviewed on a real pool, with snapshots "by the bucket" as zfSnap and
zfs-auto-snapshot make them.

• zfsadd: one dataset, one archive. Each version has a comment with its
  snapshot, and a second run takes only the new ones: put it in a cron.
• zfsbackup: any depth (tank/a/b), -franzen, -key -franzen, -hashdeep
  (it had never worked). A run that fails now says so: the snapshot of that run is
  destroyed and the archive goes back to what it was. Before, every failure was
  "seems OK", exit code 0. -kill destroys the previous snapshot only after a
  success.
• zfsreceive, zfsrestore: any depth, and the script stops at the first
  zfs receive that fails. See point 6.
• zfspurge, zfslist, zfssize: -ttl, only what is past the TTL in its
  zfSnap name (--1h, --7d, --2w, --1m, --1y).
• zfspurge with a filter "--1h" was taken for an unknown switch, and the
  script destroyed every snapshot. Fixed. If you have old scripts made
  this way, read them.

9. mysqldump, mysqlrestore

zpaqfranz mysqldump /bak -u root -p pluto                       one archive per database
zpaqfranz mysqldump /bak/all.zpaq -u root -p pluto -only "sales*"   all in one file
zpaqfranz mysqlrestore /bak/mysqldump-shop.zpaq shop_old        back as a new database
zpaqfranz mysqlrestore /bak/mysqldump-shop.zpaq -force          over the one that is there
zpaqfranz mysqlrestore /bak/all.zpaq shop -until 3 -force       a version, from a file with many

• Exit code 0 only if every dump is whole: mysqldump said 0, its last line is
  there, the archive grew. If not, the archive is cut back, the next database
  goes on, exit code 2. -force keeps what could be read, exit code 1.
• With routines, events, triggers. --single-transaction where all is InnoDB,
  locks where it is not: the server stays up.
• The password is on no command line of a child: it goes in a temporary file only
  you can read.
• -only, -not: as the help always said, a text contained or a mask.
• mysqlrestore is new: a database back, under a new name if you want. -force
  replaces the one that is there. -force -kill is for the database mysql, users
  and grants of the server. The same care as point 6: a new name first.

Tried on MariaDB 11.4, 11.8, 13.0 and MySQL 9.7, on Linux, FreeBSD, OpenBSD, Windows.

10. r -paranoid

zpaqfranz r c:\d0 k:\d1 -kill -paranoid

During the copy, by pieces, each one read back from the media and compared, and
written again if it differs, as cp -verify. -n X for pieces of X%. Files already
equal are not read.

r is a mirror, the same idea as robocopy. -append now appends only if a quick
test says that what is there is good: a file on the slave that is larger or
different is written again, whole.

11. -m8 and what does not compress

-m8 stores in the block the zstd decoder any zpaq can run, about 5.5 KB. It did so
even when zstd had saved less than that: JPEG, ZIP, small blocks came out larger
than they went in, and repack could make an archive grow. Now such a block is
stored plain.

Text and source code: the same as before.

12. The command line

• -m 5, -t 2 work with the space, in the .ini too. Before, -m was
  ignored and the 5 alone showed the help: repack a.zpaq -all -to b.zpaq -m 1 && rm a.zpaq, with no new archive and exit code 0, deleted the source.
• An argument out of place: 00569!, nothing done, exit code 2.
• A command that does not exist, or one without its archive: 00572!,
  00573!, nothing done, exit code 2. h, ?, /? are still the help, exit
  code 0.

13. Bugfix

Many, in many commands. They are not listed here, or this file would never end 😄

Part two, the details, for power users

1. What proxvzdump runs

For each guest, with system(), so you see the progress of both:

( vzdump ID --stdout --compress 0 --mode snapshot --performance max-workers=1 2>log ; echo $? > rc1 )
  | zpaqfranz a archive vzdump-qemu-ID.vma -stdin -comment "proxvzdump qemu ID name date" ; echo $? > rc2

max-workers=1 for the VMs: with more workers the blocks of a VMA come in another
order at every run (two backups of a VM that was off: 570 MB different out of 670).
-verbose shows the real line. Passwords travel in FRANZKEY and FRANZFRANZEN,
not on a command line. An archive that failed goes back with a cut to the old size,
or is deleted if it was new.

A VM of 2 GB, 600 MB used, 30 MB changed a day: 40 MB a day in the archive. The .zst
of vzdump: 300 MB a day.

2. -proxmox: what changes in the stream

Why the safer choice is to leave it out. Without the switch nobody touches the
stream: if something is strange, it is strange in vzdump too, and you can
hand the file to anybody. With the switch one more program stands in the middle.
work proxmoxvma can be run by hand: a VMA from stdin to stdout, it is the filter
of proxvzdump, and vma verify can look at what comes out.

3. zfsproxbackup, zfsproxrestore: the names in the archive

All of them sit under qemu-101/ or lxc-201/, in an archive per guest too. So they
can be extracted by hand with x, with nothing of Proxmox around.

4. What can change for a script

• zfsproxbackup: new syntax, zfsproxbackup <folder or file> vmid...|-all.
• zfsadd: a selection of more datasets is refused, 79545!.
• zfspurge: an unknown option is 79533!, exit code 2, nothing done.
• zfsbackup, mysqldump: a failure is now exit code 2. It was 0.
• A wrong command line: exit code 2. It was 0, with the help.
• isopen is gone on *nix.

5. The messages

Part three, lo spiegone, for developers

1. Why two vzdump are not alike

Each cause was measured, byte by byte, between two streams of the same guest.

• VM, the UUID. A VMA has a UUID in the header and in each extent, one every ~3.8
  MB, each with its MD5. A new UUID each run: one fragment each, ~11 MB.
• VM, the order. With more than one worker the blocks come in another
  order: max-workers=1.
• Container on ZFS, the PID. Not atime and ctime, as it looks: with them out the
  same 56 MB. ZFS compresses, so st_blocks is below the size, --sparse of
  vzdump takes the file for sparse, and tar writes it in its sparse format, with
  the name ./var/log/GNUSparseFile.<PID>/dpkg.log. 4,578 names of 23,600 changed at
  every run, and it is fixed in the source of tar, no option. --sparse-version=0.0
  keeps the real name.
• Container running, no snapshots. Suspend mode, rsync to a new folder: readdir
  in another order, 23,577 names of 23,586 out of place. --sort=name.
• atime. Nothing between two runs in a row, but in a real day: 4,000 files read,
  +21 MB. tar restores neither atime nor ctime: delete=atime,delete=ctime.

TAR_OPTIONS works because vzdump leaves it in the environment, and
lxc-usernsexec too.

2. The filter of the VMA

add() is not touched: work proxmoxvma -stdout is a process of its own, in the
pipe. Fixed UUID (zpaqfranzproxmox), MD5 recomputed, date of the header kept.

header  13,312 bytes: uuid at 8, ctime at 24, md5 at 32, header_size at 56
extent  512 bytes: block_count at 6, uuid at 8, md5 at 24, 59 blockinfo
block   4 KB

It checks magic, UUID, the MD5 of the header and of each extent before rewriting.
It cannot let an anomaly through, because after it the UUID is the same everywhere:
so anomaly = failed backup, its exit code in a third file, archive back. The layout
was checked on real streams, it is not in a document we can lean on: a future VMA
that differs must stop here, not pass.

3. Where a restore may write

A path computed from the output of another program is never to be trusted: an empty
one, plus a /, is the root of the host. zpb_destinazione is called before every
write of zfsproxrestore, and again on a folder at the last moment.

• the volume belongs to this guest, in its id and in its path;
• an absolute path, no .., no quotes;
• never /, /etc, /usr, /bin, /sbin, /lib, /lib64, /boot, /proc, /sys,
  /root, also after realpath;
• a device: a block device, not mounted;
• a file: a real file, under images/ID/;
• a folder: a file system of its own, and empty.

The old-format branch reads the paths in the archive before extracting them.
guardie_prova.sh puts a fake pvesm in the PATH, with wrong paths: 16 refused of
16, root untouched.

-tar is what makes a container come back: without it 3,187 symlinks and the
owners (100000) were lost. pct restore from a pipe cannot read the config first,
so proxvzrestore takes it from the tar, its first file (--occurrence=1), for
--rootfs, --mpN and --unprivileged: without that last one, an unprivileged
container came back privileged.

4. HTTPS with no libcurl

postami::https_chiama: the TLS already in the source for the mail, one request,
with a body and PUT, the same on Windows and *nix. Login with a ticket. The state of
a guest from /nodes/N/qemu and /lxc, because /cluster/resources lags ~10 s.
Deleting a zpaqstart user removes its ACL first, or Proxmox goes on printing
ignore invalid acl token before every output, and any output parsed from then on
is shifted by one line.

5. mysqldump: two children and a pump

The process reads from the child mysqldump and writes to the child a -stdin:
two real exit codes, and it sees the last line, -- Dump completed. On Windows
main() turns every \ of argv into /, so the password is read again from
GetCommandLineW.

A measure, not to do it again: one row per INSERT is 19% smaller in the archive,
and the restore takes 134 s instead of 9 s. Extended inserts stay.

6. How it was tested

• Proxmox VE 9.2.2, one node, root on ZFS. Ten guests: VMs on a zvol, on a second
  pool, on raw, qcow2 and vmdk files, on LVM-thin, LVM, btrfs, ext4; containers on a
  ZFS subvolume, on a raw image, on LVM-thin. Each one saved, restored as a new guest and
  compared: disks byte by byte (qemu-img compare, cmp), containers by entries,
  owners, modes, times to the nanosecond, xattr, hard links. Also a running VM, -until,
  -force, a move between storages, AES, AES + Franzen, a wrong password, a damaged
  VMA, a locked VM, a guest restored and booted to its login.
• This very source, on that host: proxvzdump with and without -proxmox,
  zfsproxbackup, two runs each, t on each archive, then restores as new guests. 19
  checks of 19. Disks the same byte by byte. A container against its original, 23,626
  entries: the 24 sockets (tar does not keep them), and in proxvzrestore also
  etc/vzdump, which vzdump adds.
• The guards of zfsproxrestore: a fake pvesm, 16 wrong paths, 16 refused.
• work proxmox 37 of 37, work prox 17 of 17, from Windows to that host.
• The zfs commands: FreeBSD 14.2, a pool of its own, 121 checks (verifica.sh,
  verifica2.sh), in clear, AES, Franzen, AES + Franzen.
• mysqldump, mysqlrestore: 119 checks on MariaDB 11.8 in a container, 64 on
  OpenBSD, 68 on Windows 11, 17 on MySQL 9.7 on FreeBSD.
• The review of the commands, tortura/comandi and tortura/work, POSIX, the
  same scripts on each system: 179 and 122 on Windows, 154 and 122 on Fedora, 147 and
  118 on FreeBSD; tortura/work 118 on macOS too.
• *Every nix build, from this source: ESXi (gcc 3.4.6), Oracle Linux 7.9 (gcc
  4.8.5, and the -DANCIENT line), Solaris 11.4 (gcc 7.3), the eleven NAS binaries
  (musl), FreeBSD 14.2 (clang, static and with mount), OpenBSD 7.9 (clang), macOS 12
  (x86_64, arm64, macFUSE, FUSE-T), openSUSE, Arch (gcc 15), Fedora 44 (gcc 16, -O1 and
  -O3, with mount, and the syntax with -std=c++11, -DANCIENT, -DNAS), Proxmox:
  0 errors, the warnings are the usual ones. autotest -all OK; autotest on ESXi,
  Solaris and the two x86 NAS.
• tortura/tutti/fumo16.sh 10 of 10 on ESXi, Oracle, Solaris, NAS x86_64, FreeBSD,
  OpenBSD, macOS, openSUSE, Arch, Fedora. nix_controlc.sh 6 of 6 on each system
  with a handler (not ESXi, not Solaris).
• Not run: the ARM binaries for the NAS and the arm64 half of the macOS binary are
  built, not run.

7. Known limits and open points

• Restoring ZFS streams and volumes: more testing needed, see point 6 of part one.
• -proxmox: more testing needed, see the note at the top. Never tried: a VM
  written hard during its backup (blocks out of order change the block info, it will
  cost more than KB), a real day of backups, a container of a real system.
• A VMA has no checksum on the data of a disk, only on its headers: 1 MB zeroed in
  the middle of an extent goes through, the filter of -proxmox and vzdump alike.
  The hash of zpaqfranz protects what is in the archive from then on.
• zfsproxbackup: a container saved as a raw image cannot go on a zfspool
  (79605!, refused). Sockets are not stored by -tar.
• A container stored without snapshots wants the guest off (79587!).
• work proxmox: not tried on a cluster of many nodes. TFA is refused.
• A single node: Proxmox VE 9.2, root on ZFS. Clusters, Ceph, NFS, older
  versions of Proxmox: not tried.
• OpenZFS on Linux outside Proxmox and Solaris: the zfs commands were reviewed on
  FreeBSD and Proxmox only.
• mysqldump: clients 5.x, old servers, the download of -space, -bin: not tried.
• The free space is read as double on macOS and OpenBSD, and on FreeBSD where
  BLOCKSIZE is not in the environment (a login sets it, a cron may not): getbsize()
  gives blocks of 512 bytes and they are counted as 1024. work checkspace can say OK
  with half of what it says free. Way out, until it is fixed: BLOCKSIZE=1024.