zpaqfranz 65.21 -wim, -json and fixing
The big news is -wim: the daily image of a Windows drive reads only what
changed since the run before. On my PC the image of C: went from about 4
minutes and 17 seconds to about 22 seconds.
Then -json, an output made for programs: it is how catpaq (the GUI) now
talks with zpaqfranz. Also a version command, x that filters by date and
size, and three issues fixed (361, 362, 363).
In short:
- New:
-wim(Windows, NTFS): the blocks of an image that did not change are
not read. The restore does not change.-wimcheckreads them anyway and
compares. The whole story is in wim.md. - New:
-json: one JSON object per line, with the code of every message, the
progress and the exit code.pakka -jsonwrites its list in the same way.
x -listfrom filetakes the names (and where each one goes) from a file. - New:
version(or-v,--version): how this zpaqfranz was built, what is
inside, and where it runs. One line each, in plain words. xfilters by date and size:-datefrom/-dateto/-minsize/-maxsize
work inxtoo (as inl).-mindateand-maxdateare new names for the dates.-ifexistaccepts a file, not only a folder (with a file it always aborted).- New:
-timer X: how long-pausewaits, in the window opened for the
administrator rights too. - Long relative names on Windows (issue 363): files could be left out with
exit code 0. - A crash in
xandtwith old compilers and many threads (issue 362). updateon Windows (issue 361): the exit code now says whether the update
went well.
Part one, what is new, for the user
-wim: images that read only what changed-json: an output for programsversion- Filters by date and size in
x -ifexistwith a file-timer: how long-pausewaits- Long relative names on Windows (issue 363)
- A crash in
xandtwith many threads (issue 362) update: an exit code to trust (issue 361)
1. -wim: images that read only what changed
Windows, NTFS. You make an image of a drive (ac, or a X: -image) every day,
into the same archive. Until now, every run read the whole drive. Deduplication
stored only what changed, but it had to read everything to find it.
zpaqfranz ac z:\pc.zpaq -wim the whole system, as ac always does
zpaqfranz a z:\data.zpaq d: -image -wim one NTFS drive
zpaqfranz ac z:\pc.zpaq -wim -wimcheck now and then: read everything anyway, and compare
With -wim, zpaqfranz reads the $MFT of the shadow copy and gives every block of
2 MB of the image a signature: which files have data there, their state, where
their data is. A block whose signature did not change since the last run is
not read: its fragments of before go into the image in its place.
On my PC (C: with about 200 GB of data, NVMe) the daily image of C: went from about
4 minutes and 17 seconds to about 22 seconds, with about 6 GB read instead of 200.
- The first run reads everything, as without
-wim. From the second one on,
only the changes, plus the metadata of the file system, are read. - The restore does not change. The image in the archive is the usual one:
x
extracts it,xcrestores it,tchecks it. - The archive is not smaller.
-wimsaves time and disk reads, not space. - The state for the next run is inside the archive, as two small files next to
the image (image_C.wimandimage_C.wimd).-until Nuses the state of version
N. After arepack, everything is read again, once. - What changes without leaving a trace is always read: the metadata of NTFS (the
folder indexes above all), the registry hives and their logs, the logs of the ETW
sessions, and the files open for writing. Each of these rules comes from a case
-wimcheckcaught during the development.
-wimcheck reads the blocks that would be skipped, and compares them. If one is
different, zpaqfranz takes it from the disk (so this version is right) and says
where, in red. Please report it if it ever happens. Once a week is enough.
Start a new archive for -wim. With -wim the image is cut into fragments block
by block. On an archive made without it, the first -wim run stores most of the
image again, once.
The spiegone (#364)
2. -json: an output for programs
A GUI (catpaq) that reads the output of zpaqfranz had to guess: lines meant for
people, progress bars that redraw themselves with \r, errors mixed with the rest.
Now, with -json, stdout is one JSON object per line, and nothing else:
zpaqfranz t z:\docs.zpaq -json
{"ev":"start","proto":1,"ver":"65.21o","cmd":"t"}
{"ev":"msg","code":65430,"sev":"I","text":"VFILE-info: 2 version(s) of 2 are as they were written (number, offset, fragments, index)"}
{"ev":"msg","code":1302,"sev":"I","text":"0.000s (00:00:00,1.36MB,OS REPORT 14.84MB) (all OK)"}
{"ev":"end","rc":0,"sec":0.000}
| Line | What |
|---|---|
start
| the first one: the version of this format (proto), of zpaqfranz, the command
|
msg
| a message with its code. sev: E error (!), W warning ($), I information (:)
|
text
| a line with no code (a summary...) |
prog
| the progress: phase (dec reading the archive, ext extracting, prg, lst), percent, done, total, seconds left
|
end
| the last one: the exit code (the same as the one of the process) and the seconds |
A line that redraws itself is never written: its progress is a prog. The banner is
off. The format can grow: new things are only added, never changed, and a reader
should skip what it does not know.
pakka -json writes its list in the same way: a line for each version, then
one line per file (with -all, every version of it).
zpaqfranz pakka z:\docs.zpaq -json -all
...
{"ev":"count","files":4,"vers":2,"recs":4}
{"ev":"ver","n":1,"date":20261116200154,"add":4,"del":0,"bytes":8}
{"ev":"ver","n":2,"date":20261116200155,"add":1,"del":0,"bytes":9}
{"f":"src/a.txt","v":[1,20261116200154,4,2,20261116200154,9]}
...
In v, a triple for each version: number, date, size. A date of 0 means deleted. The
lines of the files are kept short on purpose: on a list of a million files, the JSON is
about 9% bigger than the classic one, and about 3% slower.
Also in pakka, with or without -json:
- an archive that cannot be opened gave an empty list and exit code 0. Now 66936, exit
code 2; - a wrong password ended with exit code 1. Now 66937, exit code 2.
x -listfrom thefile takes the names to extract from a file, not from the command
line (on Windows a command line cannot go over 32,767 characters: a GUI extracting
thousands of selected files hits it). One name per line, UTF-8:
src/a.txt
src/sub/b.txt
or, with a TAB, each name with its destination (its own -to):
src/a.txt<TAB>c:/restore/first.txt
src/sub/b.txt<TAB>d:/other/b.txt
A BOM, CRLF and empty lines are fine. All the lines the same way: with a destination or
without (66939). With the destinations, no other names and no -to on the command line
(66941). An empty list is refused (66940), because it would mean every file.
3. version
The banner of zpaqfranz says a lot in a line: v65.21o-JIT8,-L,HW SHA1/2,4,SFX64 55.1.
Only I know what each piece means. Just kidding, it's in the source code. Now:
zpaqfranz version (or zpaqfranz -v, zpaqfranz --version)
The program
name zpaqfranz (full version)
version 65.21o (2026-11-16)
executable C:/zpaqfranz/zpaqfranz.exe
size 9.891.372 bytes
SHA-256 AA789273A44A7F52B4C3C6B33417AD98A6DE9693FBC98F4BC34596431EA8B3C9
How it was built
compiler gcc 14.2.0, MinGW-w64 13.0.0, UCRT
C++ 201703 (C++17)
optimized yes
for Windows 64 bit, x86-64, 64 bit pointers
byte order little endian
-D in effect ZPAQFULL ZPAQLIBSODIUM ZPAQZSTD HWSHA2
What is inside
JIT yes: ZPAQL turned into x86 code, fast
zstd yes, zstd 1.5.7 (-m8)
encryption -key yes (AES-256)
Franzen yes, with franzsodium (inside); libsodium with -libsodium
hardware SHA SHA-1 and SHA-256 with the instructions of the CPU (-DHWSHA2): this CPU has them
...
Where it runs
system Windows 11 Pro 24H2 (build 26100.4202)
process 64 bit
administrator no
...
"yes" is green, "no" is yellow. Yellow too when something is there but runs slower here,
such as the hardware SHA on a CPU without those instructions. The SHA-256 of the executable
says which build it is, to the byte. When you report a problem, paste this.
-v and --version mean version only as the first argument. Anywhere else, -v is
an unknown switch, as before.
4. Filters by date and size in x
l has always filtered by date and size. x ignored those filters, and extracted
everything. Now:
zpaqfranz x z:\1.zpaq -to z:\y2025 -mindate 2025 -maxdate 2025 only the files of 2025
zpaqfranz x z:\1.zpaq -to z:\small -maxsize 10m only the files up to 10 MB
- Dates (
-datefrom,-dateto, or-mindate,-maxdate): as inl, for files and
folders. A line says how many were left out. - Sizes (
-minsize,-maxsize): files only. With a size filter the folders are not
entries of their own: an empty folder is not made, and the folders of the files taken
are. x -testand-lowramtoo. Not-repack. Not with-image, where-minsizekeeps
its old meaning (the step of the progress of the writes).
-mindate X and -maxdate Y are new names for -datefrom X and -dateto Y, the same
in every command. They pair with -minsize and -maxsize, so you do not have to remember
which name goes with which:
zpaqfranz dir c:\data /s -mindate 2025 -maxdate 2025 -minsize 1g
In dir, the filters already worked. One exception: with a hash (the duplicates,
dir /s -xxhash), a second filter, stricter than the first, left out the files of
exactly -minsize or -maxsize bytes, and ignored -maxsize when there was a
-minsize. It is gone.
5. -ifexist with a file
-ifexist X is the switch of the backups that should not run when something is missing,
such as a USB disk that is not plugged in:
zpaqfranz a z:\backup.zpaq c:\data -ifexist z:\this_is_the_backup_disk.txt
It worked with a folder only: with a file, the backup always aborted, with exit
code 2. Now X can be a file or a folder, in a and in the backups of zfsbackup,
zfsproxbackup, proxvzdump and mysqldump:
01853! Abort because -ifexist <<z:/this_is_the_backup_disk.txt>> is not there (no file, no folder)
On Windows, relative names and long names (over 260 characters) work too. A * is not a
pattern: -ifexist z:\*.txt looks for a file called *.txt.
6. -timer: how long -pause waits
-pause waits for a key at the end, or 10 seconds. Now:
zpaqfranz a z:\1.zpaq c:\data -pause -timer 3 3 seconds
zpaqfranz a z:\1.zpaq c:\data -pause -timer 0 no wait
zpaqfranz pause -timer 5 the same as pause -n 5
The countdown says the seconds left (Press any key to continue (or wait 3)).
It matters most in the window opened for the administrator rights. When zpaqfranz has
to run itself again as administrator (a with -vss or -image, q, cloud -image, the
mount of a .vhd), the new window ends with a -pause of 10 seconds, so you can read
what it says. The command line goes to that window as it was typed. With -timer 2 it
closes after 2 seconds; with -timer 0, at once.
-n does not change -pause. In the command pause, -timer X is the same as -n X
(and wins over it).
7. Long relative names on Windows (issue 363)
c:\a\very\deep\current\folder> zpaqfranz a ..\r.zpaq src
When src is relative and short, but its full path is over 260 characters, the files
under the long paths were not stored. There was no message and the exit code was 0
("all OK"); the folders were in the archive, empty. The checks for long paths measured
the name as it was written (src\..., short), not as Windows uses it (current folder +
src\..., too long).
Now they measure the full name. As for an absolute path: 66861 lists the folders, the
messages say to try -longpath, and the exit code is 1.
In the same place, on Windows, a x.zpaq . stored nothing ("total size, file/folder
count == zero", exit code 0). Now it stores the current folder, as a x.zpaq ./ does.
Thanks to YadeWira.
8. A crash in x and t with many threads (issue 362)
x and t of a Franzen archive could crash with "double free or corruption" (exit code
134), a few times in a hundred. Every decompression thread wrote the same global string
at the same time. With the std::string of the old libstdc++ (gcc before 5, or
-D_GLIBCXX_USE_CXX11_ABI=0), its reference count went wrong. With the newer one there
was no crash, but it was the same undefined behaviour.
Now that string is written under a lock, and only when it changes. On a build with the old
std::string: 0 crashes in 800 runs (before: a few in a hundred).
Thanks to YadeWira.
9. update: an exit code to trust (issue 361)
On Windows, zpaqfranz update writes a batch that puts the new exe in place. That batch
killed every zpaqfranz running, the one that started it too:
- the exit code was always 1 (the one of
taskkill /F), even when the update was good:
no use in a script; - the script went on while the batch was still moving and copying the exe.
Now the batch kills the other zpaqfranz as before, and skips this one. This one waits for
the new exe to be in place, checks it (SHA-256), and ends with:
03838: C:/zpaqfranz/zpaqfranz.exe is now the new one (65.21o) exit code 0
03839! C:/zpaqfranz/zpaqfranz.exe is NOT the new one: see the messages... exit code 2
The .old (the exe this process ran from) is deleted by the batch after it ends. On a file
system that does not let a running exe be renamed, after 10 seconds the batch kills this
one too, as before.
Thanks to cheebusjeebus.
Part two, the details, for power users
1. What can change for a script
xwith-datefrom,-dateto,-minsize,-maxsize(whichxused to ignore):
now they filter. A script that passed them by habit (the same line asl) now
extracts less.-ifexistwith a file: the backup runs (before: exit code 2, always).-pause:-timer Xsets its seconds.-nhas nothing to do with it.pakkaon an archive that cannot be opened, or with a wrong password: exit code 2
(before: 0 with an empty list, and 1).updateon Windows: exit code 0 or 2, after the new exe is in place (before: 1,
always, at once).- Windows,
aof a relative folder with a full path over 260 characters, without
-longpath: 66861 and exit code 1 (before: files missing, 0). - Windows,
a x.zpaq .: the current folder is stored (before: nothing, 0). dir /swith a hash and-minsize/-maxsize: the files of exactly that size are
there.-v/--versionas the first argument: theversioncommand.-wim: an archive made with-wimdiffers from one made without it (the image is
cut block by block). The image extracted is the same.
2. The switches
| Switch | What |
|---|---|
a -image -wim, ac -wim
| (Windows, NTFS) the blocks that did not change since the last run are not read |
a -image -wim -wimcheck, ac -wim -wimcheck
| the blocks to skip are read anyway, and compared |
-json
| the output for programs: one JSON object per line |
pakka -json
| the list too, in JSON |
x -listfrom thefile
| the names (and their destinations, after a TAB) from a file |
version, -v, --version
| how it was built, what is inside, where it runs |
-mindate X, -maxdate Y
| the same as -datefrom X, -dateto Y
|
x -datefrom -dateto -minsize -maxsize
| x filters by date and size (not with -image for the size)
|
-ifexist X
| X a file or a folder |
-timer X
| the seconds of -pause (default 10, 0 = no wait); in pause, the same as -n X
|
3. The messages
| Code | When |
|---|---|
01853!
| -ifexist: X is not there (no file, no folder)
|
03838, 03839!
| update (Windows): the exe is the new one; it is not
|
03879, 03880
| x: names left out by their date; files left out by their size
|
03930, 03931
| version
|
44018-44064
| -wim (only three lines by default, the rest with -verbose)
|
66936!, 66937!
| pakka: the archive cannot be opened; the password is not right
|
66938!-66942
| -listfrom: cannot be read; mixed lines; empty; destinations with other names or -to; how many names
|
Part three, for developers
1. How it was tested?
Nobody cares. Test by yourself 😄
2. Gone: -DDLL
The DLL build is no longer in the source. It could not be run twice in the same process:
the state of the program lives in global variables, never reset between two calls, and
exit() is everywhere. It also meant nothing on Linux and macOS. catpaq runs zpaqfranz as
a program and reads -json.
3. Known limits
-wimworks on NTFS read from a shadow copy, or from a locked volume. FAT32, exFAT, ReFS
and the rest are imaged as always (a line says so). With-noturbo,-t1,-verifyor
-collision, the image is made as without-wim.-wimwas tested on three Windows machines, mine included:-wimcheckis there to find
what they did not show.-jsonis made for catpaq.protois 1: new things will only be added.- The pause of the elevated window was timed on the relaunch of
a -vss. The relaunches of
cloud -imageand of the mount of a.vhduse the same-pause, but were not timed. -ifexist: no patterns (*,?).- On Solaris, gcc warns that a line of
versioncould be cut (utsnamehas fields of 257
bytes there). It is ansnprintf: at worst the text is shorter.
4. The binaries
Built from the same source and tried on each system (autotest, then the quick checks
above, then -ifexist and -timer):
| System | Binary |
|---|---|
| NAS (musl, static) | zpaqfranz_armv5, _armv7, _armv7_cortexa9, _armv7_cortexa15, _armv8, _cortexa53, _cortexa55, _cortexa57, _cortexa72, _nas_i686, _nas_x86_64
|
| ESXi | zpaqfranz_esx
|
| Solaris 11 | zpaqfranz_solaris
|
| macOS (Intel and Apple Silicon) | zpaqfranz_macos
|
| openSUSE | zpaqfranz_opensuse
|
| Oracle Linux 7.9 | zpaqfranz_oracle79
|
| OpenBSD | zpaqfranz_openbsd
|
| Arch | zpaqfranz_arch
|
| FreeBSD | zpaqfranz_freebsd
|
| Fedora | zpaqfranz_fedora
|
The NAS binaries for ARM were built, not run (here there is no ARM. Please send my one 😄 ). The same goes for the Apple Silicon half of zpaqfranz_macos, on an Intel Mac. The warnings of the compilers are
the known ones (most of them in code by others: Mbed TLS, libzpaq), plus the one of
Solaris above. As always, the pairs A53 = A55 and A57 = A72 are the same file: gcc makes
the same code for them (see NAS.md).