zpaqfranz 65.18

Two main things in this release.

Franzen no longer needs libsodium. The cryptography of Franzen is now built
into zpaqfranz: franzsodium. Nothing to install, nothing to download, no DLL.
libsodium is not thrown away: it stays, optional, as a second and independent
engine to check the first one against. This is cryptography, and it deserves the
utmost care.

The compression levels have names: -lean -fast -normal -good -better -ultra -max. A new scale, easy to remember. The zpaq models 1, 2 and 3 partly give way
to the modern ones, faster and with much less RAM; 4 and 5, the ones that
compress the most, stay.

zpaqfranz a backup.zpaq c:\data                              the default (-normal): leave it
zpaqfranz a backup.zpaq /share -lean                         a cheap NAS, little RAM
zpaqfranz a backup.zpaq c:\data -good                        a bit smaller, a bit slower
zpaqfranz a backup.zpaq c:\data -franzen MyLongPassword      encrypted, nothing to install

Which level? My advice

In the vast majority of cases you do not need to change the level. Leave
the default: today it is a very good balance between speed, resources used and
size of the archive.

-fast is indeed a bit faster, but not by much. It is worth it only if you
really want the fastest that zpaqfranz can give today.

-good has its use: a little slower, a little smaller. It is probably the best
choice after the default.

The levels above it differ in one key point: the speed of extraction. With
-better it is high. With -ultra and -max it is about the same as the speed
of compression, that is, slow. Those two are for data you expect to extract
rarely, or for small data.

In short:

Level	When
-lean	weak hardware, little memory (a cheap NAS...). On really minimal hardware add -t1 too
-fast	when you want top speed, at the cost of the size of the archive
-normal	the default, the one I recommend
-good	my second choice
-better	when you want to compress a lot, but you will extract often
-ultra	high compression, slow
-max	the smallest archive, with no compromise

And:

• -fast is a level now. The switch that stores the history for a fast l
  is -lfast (and -nolfast). If a script of yours writes -fast, read
  point 4.
• Franzen everywhere: 32-bit Windows, the NAS, ESXi, the "open" build, static
  binaries. Before, none of them had it.
• -franzen with no password is an error. Before, the archive was written in
  clear, with "all OK".
• The RAM in the last line is now the peak, and next to it the peak as the
  operating system saw it (OS REPORT).
• -turbo with few cores: faster, and with less RAM. The threads are counted
  on the cores the process is allowed to use.
• work proxmox audit: what it takes to rebuild a Proxmox node from an empty
  ISO, in two archives: one with the secrets, one without.
• -tar -not "X/*" keeps the folder X, empty, as tar does (issue 350).
• l -all shows # (changed) with every hash, not only with the "b" ones;
  the help says what each hash is (-sha3 is SHA3-256) (issue 351).
• Bugfix: pp and x -stdout could hang forever with little RAM; an archive
  name that is a folder; -stdin -noattributes stored the attributes anyway
  (issue 352); others.

From 65.17o to 65.18r: about 4,730 new lines (274,509 to 279,242).

Part one, what is new, for the user

1. The levels by name
2. Some numbers
3. The NAS: -lean
4. -fast is a level now: -lfast for the history
5. Franzen with no libsodium
6. libsodium, for a double check
7. The RAM in the last line
8. Threads: physical cores, affinity, -turbo
9. work proxmox audit
10. -tar -not "X/*"
11. The command line
12. Bugfix
13. The hashes: what they are, and # in l -all

1. The levels by name

What is behind the names:

• -m8 is zstd 1.5.7, level 3, in blocks of 16 MB, written in a way that any
  zpaq can extract: the decoder, in ZPAQL, travels inside the archive. zpaq 7.15
  extracts it too.
• -m80 is the same with blocks of 1 MB: a fraction of the RAM, an archive a
  few % bigger.
• -m9 is -m8h9: zstd at level 9.
• -m7 is -m86h19: zstd at level 19, in blocks of 64 MB. Slow to compress,
  fast to extract.
• -m6 is the -m4 of zpaq with three more (sparse) contexts, the "m4.5".
• -m5 is the strongest model of zpaq.

Why. The models 1, 2 and 3 of zpaq are beaten by the modern ones. -fast
against -m1, -good against -m2: faster, smaller, with a fraction of the
RAM. -better against -m3: smaller, with less RAM, about the same time to
compress (a bit less or a bit more, by the data), and an extraction more than
ten times faster. The numbers are in
point 2. Where zpaq is not beaten, at the top, its models
stay: -ultra (an enhanced -m4) and -max (-m5).

Less RAM: a road already begun. By default zpaqfranz runs one thread per
physical core; zpaq 7.15 one per logical core, twice as many on a CPU with SMT
(HyperThreading). Every thread has its own memory, so doubling them is not free:
on the numbers of point 2, -m2 takes 15.2 GB with zpaq 7.15 and 8.2 GB with
zpaqfranz. Other releases already cut the RAM in general (the index of an
archive kept in less memory, the batches of -turbo sized on the threads).
-lean is the next step: speed balanced against modest resources, above all
memory and CPU. It is for cheap NAS and the like.

Nothing is removed. -m0 ... -m9, and the long forms (-m8h19, -m86,
-m4...), work as before. The names are a shorter road. -store is the same as
before (no deduplication, no compression).

The rules:

• One level only, and not together with -m: -better -m3 stops at once
  with error 03995, exit code 2.
• What you write wins over what a level sets: -lean -t4 runs 4 threads,
  -lean -turbo uses -turbo, -lean -t1 one thread.
• -verbose says what a level is: -lean = -m80 -t2 -noturbo.
• In a -ini file a level counts as a -method: the command line wins over the
  file.
• Where there is no zstd (the ESXi binary, a build with -DANCIENT) the
  names are the zpaq models, in the same order: -lean -m1 -t2, -fast and
  -normal -m1, -good -m2, -better -m3, -ultra -m6, -max -m5.

2. Some numbers

Indicative: one PC, one run each, not a benchmark. Windows 11, AMD Ryzen 9
7950X3D (16 cores, 32 threads), zpaqfranz 65.18p, measured with franztime.
Archive = the size of the archive against the size of the data. RAM = the peak
of the process.

3.3 GB, a folder of programs and sources:

21.7 GB (an MSYS2 installation, the same folder, and the 11.8 GB disk of a
Debian 12 VM), against zpaq 7.15:

• -lean and -fast write the same archive. -lean gives up half of the
  speed for less memory and much less CPU: about 2 cores busy instead of 6-11.
• zpaq 7.15 runs 32 threads here, zpaqfranz 16: about half the RAM. With the
  heavy models, on a CPU with SMT, fewer threads also means more time (-m4: 715
  s against 479). -ht runs one thread per logical core, as zpaq 7.15 does.
• On 154 GB of virtual machines (no archive written, compression only): -fast
  1,425 MB/s, -normal 1,372 MB/s, -good 971 MB/s, -lean 407 MB/s with 107
  MB of RAM; zpaqfranz -m1 903 MB/s, zpaq 7.15 -m1 216 MB/s.

Extraction. Measured in October (65.11, the same PC), 6.6 GB in 116,703
files. Add = the archive written; test = the whole archive decompressed and
checked (t), about the cost of an extraction:

-better extracts as fast as -normal; -ultra and -max take as long to
extract as to compress.

3. The NAS: -lean

A cheap NAS has few cores, slow ones, and little RAM. -lean is made for it.

RAM, the peak from the system, 1.5 GB in 12,000 files (the NAS binary,
x86_64, in a VM):

-lean is -m80 with 2 threads: 31 MB, against 234-452 MB of -m1.

Speed, on a real NAS (ARM Cortex-A15, 4 cores, 65.17o): -m80 -t2 30 MB/s,
-m1 -t2 24 MB/s, -m1 -t4 28 MB/s. The ceiling there is 37 MB/s: -m0, the
deduplication alone, with no compression. -lean is faster than -m1, with a
fraction of the memory.

On the NAS binaries the default is still -m1, as before: on a NAS, write
-lean. On a really minimal one, -lean -t1.

4. -fast is a level now: -lfast for the history

Up to 65.17 -fast was something else: it stored the history inside the
archive, so that l is fast. Everyone reads -fast as a level, so:

• -fast is the level (-m80);
• the history is -lfast; -nolfast turns it off;
• -nofast still works, as the old name of -nolfast.

If a script writes -fast:

• on a PC (Windows, Linux, BSD, macOS) the history is stored anyway, it has been
  the default since 65.6. The script now gets -m80: faster, less RAM, an
  archive a few % bigger. To keep the old behavior, take -fast away;
• on the NAS and ESXi binaries the history is not the default: the script gets
  -m80 and no history. Write -lfast instead of -fast.

5. Franzen with no libsodium

Up to 65.17 Franzen needed libsodium: a DLL on Windows (extracted from
zpaqfranz-full.exe or downloaded), a package on Linux, BSD and macOS. Where it
could not be loaded, no Franzen: 32-bit Windows, the NAS, ESXi, the "open" build,
a static FreeBSD.

Now Franzen uses franzsodium, built into zpaqfranz:

• the nine functions Franzen needs, with the interface of libsodium:
  ChaCha20-Poly1305 (IETF), Argon2id, random bytes, wiping of memory;
• taken from Monocypher 4.0.2, plus ChaCha20 with SSE2 and AVX2 (AVX2 chosen
  at run time) and Poly1305 with 64-bit arithmetic where the compiler allows it;
  plain portable C everywhere else (ARM, 32 bit, old compilers);
• the same file format: a file written with franzsodium and one written with
  libsodium are the same thing. Each engine reads the files of the other, in
  both directions;
• one core encrypts and authenticates about 1.2 GB/s (Ryzen 9 7950X3D, AVX2), far
  more than an archive is written. The key derivation (Argon2id, 256 MB) takes
  0.44 s instead of 0.28 s, once per run.

Franzen everywhere: 32-bit Windows, the NAS (x86 and ARM), ESXi (gcc 3.4.6),
the "open" build (no downloads, no external libraries), static binaries. Same
commands, same files, the same bytes.

Which engine is at work: the "Creating/Appending Franzen" banner is cyan
with franzsodium, magenta with libsodium. With -verbose a line says it:
Franzen with franzsodium (ChaCha20 AVX2+SSE2, Poly1305 64 bit), not libsodium.

Franzen is explained from the beginning, with its file format and its limits, in
part three.

6. libsodium, for a double check

An engine of cryptography, even one taken from a well known library, has to be
checked by something else. libsodium is the reference. So it stays, optional, as
a second and independent implementation:

• -libsodium: Franzen with libsodium, loaded at run time, as up to 65.17.
  Without -libsodium on the command line it has no effect at all: libsodium
  is not looked for, not loaded, not downloaded.
• -sodiumcheck: every call is made by both engines and the results are
  compared byte by byte. A difference is an ERROR, exit code 2. What is written
  is what libsodium gave.
• -franzsodium asks for the built-in engine; it is the default anyway.

How franzsodium was checked against libsodium:

• the official test vectors of RFC 8439 (ChaCha20-Poly1305) and RFC 9106
  (Argon2id);
• 200,000 random encryptions and decryptions and 10,001 key derivations: the
  same bytes as libsodium 1.0.18 (Windows, 64 and 32 bit) and 1.0.22 (Fedora,
  FreeBSD, OpenBSD), refusals and errors too;
• AddressSanitizer and UndefinedBehaviorSanitizer, clean (Fedora, FreeBSD);
• the batteries of tests on archives (Franzen, AES + Franzen, repack,
  password, multipart backup) with -sodiumcheck always on: 374 checks, no
  difference;
• in a deterministic mode (fixed salt and nonces: for tests only), the same
  archive, to the byte, on ESXi, FreeBSD, the NAS (x86 and ARM), Oracle Linux
  7.9; Windows reads it with both engines.

What it has not had: an independent audit. Neither the format of Franzen nor
franzsodium has been reviewed by an outside cryptographer.

On 32-bit Windows, the NAS, ESXi, any build with -DANCIENT and the "open"
build there is no libsodium: -libsodium and -sodiumcheck stop at once with
error 00595, exit code 2.

7. The RAM in the last line

0.031s (00:00:00,8.32MB,OS REPORT 21.57MB) (all OK)

• The first number is the peak of the memory zpaqfranz counts by itself.
  Up to 65.17 it was what was left at the end of the run, and with -m1 an error
  in the count made it grow (3.53 GB shown on a NAS).
• OS REPORT is the peak of the process, as the operating system measures
  it: Windows, Linux, the BSDs, macOS. Not on Solaris and ESXi.
• The internal count does not see everything (zstd, the C++ library): when the
  two differ, OS REPORT is the true one.
• -verbose shows the pieces: heap, arrays, index, the peak.

8. Threads: physical cores, affinity, -turbo

• One thread per physical core, by default, as before; -ht for one per
  logical core.
• Only the cores the process may use: with taskset, start /affinity, a
  container with fewer CPUs, zpaqfranz counts the cores it is allowed to run on,
  not all of the machine. Windows and Linux (not FreeBSD and macOS). -verbose
  says it.
• -turbo with few threads: the batch of the parallel fragmenter is now
  sized on the threads and on the block. With few threads, faster and with less
  RAM. On a 1.5 GB piece of a VM disk, 4 threads: -m80 546 → 727 MB/s and 172
  → 54 MB; -m8 704 → 790 MB/s. The archive is the same, byte by byte.

9. work proxmox audit

Before an upgrade: what it takes to rebuild a Proxmox node from an empty ISO.

zpaqfranz work proxmox audit 192.168.1.10              by SSH, as root
zpaqfranz work proxmox audit localhost                 on the node itself, as root
zpaqfranz work proxmox audit -script audit.sh          the script, to read it or run it by hand
zpaqfranz work proxmox audit audit.tar                 the archives from that tar

• Reports of about 100 commands, /etc with /etc/pve, /root, keys, partition
  tables, the crontab.
• Two archives, a version each run: private_<node>.zpaq, everything; and
  public_<node>.zpaq with a .zip, the same without passwords, private
  keys, tokens, hashes: to read with someone else, or with an AI.
• On the node it only reads. The script can be written (-script) and read
  before anything runs.
• By SSH: host[:port], the password, the SHA256 of the host key (asked, or
  checked, before any password is sent). -ssh K for a key file.
• -key, -franzen: the private archive only. The public one is in clear.
• Also entry 17 of the menu of work proxmox.

10. -tar -not "X/*"

With -tar, -not "/dev/*" leaves out what is inside /dev, not /dev
itself: the folder is kept, empty, with its owner, mode and times, as tar --exclude does. A system restored from such a backup has its /dev, /proc,
/sys, /run again (issue 350, Oracle Linux 7.9 restored from SystemRescue).
-not "X" and -not "X/" still leave the folder out. Without -tar, nothing
changes.

11. The command line

• -franzen with no password (nothing after it, another switch after it, or
  an empty value, as -franzen "$PW" with PW not set): error 00599, exit code
  2, nothing done. The same for -franzen2. Up to 65.18h that -franzen was
  ignored, and the archive was written in clear, with "all OK".
• An archive name that is a folder (a z:\pippo\, a ., backup dir/, or
  a folder that exists): error 00597, exit code 2. Before, backup dir/ wrote
  dir/_00000001.zpaq.
• Two levels, or a level with -m: error 03995, exit code 2.
• -libsodium, -sodiumcheck where there is no libsodium: error 00595,
  exit code 2.

12. Bugfix

• pp and x -stdout hung forever with little RAM: the block the output
  was waiting for could not get memory, and nobody gave it any. Seen on a
  FreeBSD VM with 4 GB; with -ramsize 20MB it happened every time, on every
  system. With the default limit nothing changes.
• -franzen with no password: the archive in clear (see point 11).
• The RAM count: Array subtracted the elements instead of the bytes; the
  arrays of the extraction were never subtracted.
• ESXi: a static binary with __thread crashed when an archive with Franzen
  had nothing new to add (LinuxThreads, kernel 2.4).
• -stdin -noattributes stored the attributes anyway (issue 352): A.... on
  Windows, 0644 on *nix. Now nothing, as for a file added from the disk. The
  same for -image -noattributes.
• -sha1b never showed # in l: the name was looked for as SHA1-B, it
  is SHA-1B.
• -nochecksum was twice in h a, with two different texts.

13. The hashes: what they are, and # in l -all

# with every hash. In l -all a file is + when it is added, # when it
is changed, - when it is deleted. Up to 65.18q # came only from the "b"
hashes (-xxhashb, the default, -sha1b, -sha256b, -xxh3b, -md5b,
-blake3b, -sha3b): only their block in the archive has the byte added or
changed. With any other (-whirlpool, -sha3, -crc32, -highway*,
-nochecksum, -715, an archive of zpaq 7.15) a changed file was +.

Now l -all finds it by itself: a file is changed if, in the version before the
one where it is, it was there and not deleted. A file deleted and then added back
is + again. Nothing changes in the archive, so it works with the archives made
before too. The "b" hashes keep what their block says. Without -all, l
shows the last state of each file: there # still comes from the "b" hashes
only.

-whirlpoolb does not exist, and will not: WHIRLPOOL is 512 bit, the block of
the "b" hashes has room for 256. With l -all it is no longer needed.

What each hash is, now in h a:

"b" is the same hash, stored in binary: a smaller block in the archive (50
bytes instead of 76), with the creation date and the byte added/changed.

Part two, the details, for power users

1. The levels, as the source sees them

g_livelli[]: name, method, the words of the help. A level is resolved in
loadparameters, after every switch is read and before -m7/-m9 become their
zstd forms: from there on it is a -m like any other, the same archive, byte by
byte, as the method written by hand. For -lean only: -t2 if no -t was
written, -noturbo if no -turbo was written. -verbose prints 03996.

In a -ini the levels have the key of -method: the command line wins over the
file; inside the file, the last one wins, as for -m3 -m5.

2. The NAS and ESXi builds

The ANCIENT builds (the NAS, ESXi) keep the old defaults: -m1, no -turbo
(not compiled there), no history. The levels by name are there, and -lean is
what to use on a NAS. The 32-bit binaries use at most 2 threads by default: on a
4-core ARM -t4 uses all of them (-lean -t4).

ESXi, and any build with -DANCIENT that is not the NAS, has no zstd: no
-m7, -m8, -m9 (written by hand they are -m5, with a warning). There the
levels by name are the zpaq models, in the same order, from the lightest to the
strongest:

Every ANCIENT build has Franzen with franzsodium only, no libsodium: no
dlopen, so on an old glibc (before 2.34, where dlopen is in libdl) the
line of the notes links with no -ldl:

g++ -O3 -DANCIENT -Dunix zpaqfranz.cpp -o zpaqfranz -pthread

-DANCIENT is for compilers with no C++11. With gcc 4.8 and 5 (Oracle Linux,
CentOS, RHEL 7) the right line is the normal one, with -std=c++11 -ldl.

3. -libsodium, -sodiumcheck, ZPAQLIBSODIUM

• libsodium support is under #define ZPAQLIBSODIUM; it is not defined on
  32-bit Windows, in the ANCIENT builds (the NAS, ESXi, -DANCIENT) and in the
  "open" build. Comment it out to build without libsodium.
• Where -libsodium looks: Windows 64 bit, libsodium.dll next to
  zpaqfranz.exe, only if its SHA-256 is the expected one, otherwise extracted
  from zpaqfranz-full.exe, otherwise downloaded and checked against the same
  hash; Linux and BSD, the library of the system by dlopen (a static binary
  cannot load it); macOS, libsodium.dylib (Homebrew and MacPorts folders).
• -sodiumcheck: the first 20 differences are shown, then the count at the end
  and exit code 2; if all is equal, how many calls were compared.
• -franzsodium wins over -libsodium; -sodiumcheck wins over both.

4. What can change for a script

• -fast is -m80 now, not the history: see point 4 of part one.
• -franzen with no value: exit code 2 (before: 0, and the archive in
  clear).
• The number in the last line is the peak, not what was left at the end;
  ,OS REPORT X is added inside the parentheses where the system gives it.
• A level and -m together, two levels: exit code 2.
• An archive name that is a folder: exit code 2.
• Threads by default under taskset or a CPU limit: fewer than before
  (Windows, Linux).
• -tar -not "X/*": the folder X is in the archive now, empty.
• l -all: # where it showed +, for a changed file, with every hash.
• -stdin, -image with -noattributes: no attributes stored.
• The texts of the hashes in the help (h a, and in Italian and Russian):
  new, with the type and the bits.

5. The messages

Part three, lo spiegone: Franzen

Yes, the Franz-en-coder 😄

In short

Franzen is the encryption layer of zpaqfranz. Add -franzen and a password to the usual commands, and the archive exists only in encrypted form, as name.zpaq.franzen: no cleartext copy is left on the disk.

zpaqfranz a backup.zpaq c:\data -franzen MyLongPassword
zpaqfranz t backup.zpaq -franzen MyLongPassword
zpaqfranz x backup.zpaq -franzen MyLongPassword -to c:\restore

(backup.zpaq and backup.zpaq.franzen are both accepted as the name of the archive.)

• Modern, authenticated encryption: ChaCha20-Poly1305 for the data, Argon2id for the password.
• Every 64 KB block is sealed on its own: a wrong password or a changed byte is detected, never decrypted into silent garbage (the known limits are in the Threat Model).
• Append-only: new versions are added at the end, the start of the file never changes, so rsync --append sends only what is new.
• Nothing to install: since 65.18 the cryptography is built into zpaqfranz (franzsodium). libsodium is no longer needed.

Speed

With today's zpaqfranz, Franzen is faster than zpaq's classic AES-256 (-key), above all when writing an archive. From the fastest to the slowest:

1. Franzen alone (-franzen)
2. AES alone (-key)
3. AES + Franzen together (-key ... -franzen ...): two layers of encryption cost more than one.

The reason: zpaq's AES-256 is portable C code that does not use the AES instructions of modern CPUs. ChaCha20-Poly1305 was designed to be fast in software, and zpaqfranz's engine uses the vector units of the CPU (SSE2 on every x86-64, AVX2 when the CPU has it). One core encrypts and authenticates about 1.2 GB/s on a Ryzen 9 7950X3D, far more than the rate at which a compressed archive is written.

One cost is there on purpose: turning the password into a key (Argon2id) takes about half a second and 256 MB of RAM. That is what makes password guessing expensive. It happens once per run.

How safe is it?

Honestly, it is hard to say. The history of cryptography is full of schemes that looked sound and turned out to be flawed years later.

I think Franzen is robust, and I believe it is well implemented, and therefore reliable. But I cannot guarantee anything, just as there is no guarantee for zpaqfranz in general. Try it, and use it if you like it.

What this opinion rests on:

• Nothing home-made in the cryptography. Franzen uses standard, widely studied building blocks: ChaCha20-Poly1305 (RFC 8439, the same construction used by TLS 1.3 and WireGuard) and Argon2id (RFC 9106, winner of the Password Hashing Competition). What is zpaqfranz's own is the way they are put together: the file format.
• Checked against the references. The built-in engine passes the official test vectors of the RFCs, and it was compared with libsodium byte by byte: 200,000 random encryptions and decryptions and 10,001 key derivations gave exactly the same results, on Windows, Linux and BSD, 32 and 64 bit.
• The same bytes everywhere. In a deterministic test mode (fixed salt and nonces: for tests only, never for real data), archives written on very different systems (ESXi with a 2006 compiler, FreeBSD, NAS on x86 and ARM) are identical to the byte, and Windows reads them.
• Self-test. zpaqfranz franzen autotest runs the internal tests of the Franzen code; zpaqfranz autotest -all -to <empty folder> -franzen pippo runs the bigger suite with Franzen.

What it has not had: an independent audit. Neither the Franzen format nor franzsodium has been reviewed by an outside cryptographer. Its known limits are listed in the Threat Model.

What Franzen is for. Its goal is one: nobody without the password reads the data. It is not meant to be armored against every kind of cryptographic attack on the file itself, such as forging or rearranging its pieces. The format could be refined further (binding each block to its position, authenticating every byte of the header, a nonce of its own for the password check), but none of that would protect the data any better, and protecting the data is what I care about.

The engine: franzsodium (built in) and libsodium (optional)

Up to 65.17 Franzen needed libsodium, an external library (a DLL on Windows, a package on Linux, BSD and macOS). Since 65.18 Franzen uses franzsodium, built into zpaqfranz:

• the nine functions Franzen needs, with the same interface as libsodium: ChaCha20-Poly1305 (IETF), Argon2id, random bytes, wiping of memory;
• taken from Monocypher 4.0.2 (its license is in the source), plus ChaCha20 with SSE2 and AVX2 (AVX2 chosen at run time, if the CPU has it) and Poly1305 with 64-bit arithmetic where the compiler allows it; plain portable C everywhere else (ARM, 32 bit, old compilers);
• random bytes come from the operating system: RtlGenRandom on Windows, arc4random on macOS and the BSDs, getrandom (or /dev/urandom) on Linux;
• nothing to install, nothing to download, no DLL next to the executable. This also brings Franzen to builds that never had it: 32-bit Windows, NAS, ESXi and (since 65.18k) the "open" build.

The file format is the same. A file written with franzsodium and one written with libsodium are the same format: each engine reads the files of the other (tested in both directions).

Which engine is at work: the "Creating/Appending Franzen" banner is cyan with franzsodium and magenta with libsodium; with -verbose a line says it, for example Franzen with franzsodium (ChaCha20 AVX2+SSE2, Poly1305 64 bit), not libsodium.

Raw speed of the engines, one core, measured with the test bench on the same PC (Ryzen 9 7950X3D; Fedora in a virtual machine):

A recent libsodium is still faster on the bare cipher (its Poly1305 is vectorized). In an archive run the difference does not show: both are far faster than compression and disk. Key derivation (Argon2id, 256 MB) takes about 0.44 s with franzsodium against 0.28 s with libsodium 1.0.18, once per run.

libsodium, optional: -libsodium

• -libsodium makes Franzen use libsodium, loaded at run time, as zpaqfranz did up to 65.17. It is left for debugging and comparison, and over time it will probably be removed.
• If you do not put -libsodium on the command line, it has no effect at all: libsodium is not looked for, not loaded, not downloaded.
• -sodiumcheck (debug only): every call is made by both engines and the results are compared byte by byte; a difference is an ERROR (exit code 2). What is written is what libsodium gave.
• -franzsodium asks for the built-in engine (it wins over -libsodium). It is the default anyway.
• Where -libsodium can find the library:
  • Windows 64 bit: libsodium.dll in the folder of zpaqfranz.exe, taken only if its SHA-256 is the expected one; otherwise extracted from zpaqfranz-full.exe (it has the DLL inside), otherwise downloaded and checked against the same hash.
  • Linux, BSD: the system library (apt install libsodium23, pacman -S libsodium, pkg install libsodium...), found by dlopen. A static binary cannot load it.
  • macOS: libsodium.dylib (brew install libsodium; the Homebrew and MacPorts folders are searched).
  • 32-bit Windows, NAS, ESXi, any build with -DANCIENT, the "open" build: these builds have no libsodium support. There -libsodium and -sodiumcheck stop at once with an error (exit code 2), and nothing is done.
• For who compiles: libsodium support is under #define ZPAQLIBSODIUM; comment it out to build without it.

The "open" build of zpaqfranz (no downloads, no embedded files, no external libraries) has Franzen too since 65.18k, with franzsodium only: same commands, same files, the same bytes as the full build.

What Problem Does Franzen Solve?

When you encrypt a backup archive for cloud storage, you typically face three painful trade-offs:

1. Speed vs. Security: the encryption must not slow down a backup that writes hundreds of megabytes per second.
2. Integrity vs. Cost: zpaq's built-in AES-256 (CTR mode) hides the data, but it has no authentication of its own: it cannot tell a changed archive from a good one. To check an archive you test it (t): decrypt and decompress everything. That takes time and CPU.
3. Encryption vs. rsync: most encrypted formats rewrite the file header on every update, forcing tools like rsync to retransmit the entire file even if only a few megabytes were added.

Franzen addresses all three. It wraps any file (typically a .zpaq archive) in a layer of modern, block-based authenticated encryption that is:

• Parallel: every 64 KB block is cryptographically independent, so blocks can be encrypted, decrypted and checked by many threads at once.
• Rsync-friendly: the file header is immutable; new data is strictly appended; rsync --append transfers only the delta.
• Verifiable without decompressing: each block carries its own authentication tag, so the whole file can be checked with nothing written to disk.

Architecture

Independent Block Processing

Every 64 KB data block is encrypted on its own, with its own random nonce, and carries its own Poly1305 authentication tag. No block depends on another:

Block 0:  [Nonce₀ (12 B)] [Ciphertext₀ (64 KB + 16 B tag)]
Block 1:  [Nonce₁ (12 B)] [Ciphertext₁ (64 KB + 16 B tag)]
  ...
Block N:  [Nonceₙ (12 B)] [Ciphertextₙ (≤ 64 KB + 16 B tag)]   ← last block, variable size

• Multi-threading: the standalone franzen command, with -ssd -tN, splits the blocks among N threads. When an archive is read (t, x...), every thread of zpaqfranz decrypts the blocks it reads. When an archive is written (a), the blocks are encrypted in the order the archive is written, one stream: about 1.2 GB/s on one core.
• Random-access decryption: you can decrypt the last block of a 100 GB file without reading the preceding 99 GB.
• Granular integrity: a damaged block is detected and reported on its own; the other blocks still decrypt.
• What a tag does not cover: the position. The tag of a block covers its content, not its place in the file. Whole blocks swapped, repeated or moved are not noticed by Franzen itself (see the Threat Model).

Append-Only / Rsync-Friendly Design

Backup workflows often append incremental data to an existing archive. Tools like rsync rely on the beginning of a file remaining stable: if the header changes, rsync retransmits everything.

Franzen uses an immutable header strategy:

1. Header: written once at creation. Contains the Argon2id salt and cost parameters. It is never modified afterwards.
2. Footer: the mutable metadata (logical size, segment tracking, authentication tag) is stored at the end of the file.
3. Append behavior: when new data is added, the old footer is left in place as inert padding, new encrypted blocks are written after it, and a new footer is written at the new end of the file.

Result: rsync --append sees that the header and all existing blocks are unchanged, and transfers only the newly appended bytes. For a 100 GB archive that grew by 500 MB, only ~500 MB is transferred.

An update that does not finish (the process killed, the power gone, the disk full) leaves blocks after the last good footer and no footer at the end. Such a file is still read up to its last complete update, with a warning; the next a cuts the unfinished rest away and continues from there.

Segment Cache

Because multiple updates leave orphaned footers scattered through the file, Franzen reconstructs the logical layout at open time:

[HEADER (144 B)]
[Segment 1 blocks ... ]  [Footer₁ (84 B)]      ← orphaned
[Segment 2 blocks ... ]  [Footer₂ (84 B)]      ← orphaned
[Segment 3 blocks ... ]  [Footer₃ (84 B)]      ← active (end of file)

The active footer stores the total logical size and the physical offset where the most recent segment starts. Franzen walks backwards through the footers and builds a small table, one entry per update, that maps logical offsets to physical ones.

Use Cases

Two Passwords: the "Zero-Knowledge" Provider Model

Franzen can wrap an archive that is already encrypted with zpaq's AES-256:

zpaqfranz writes both layers in a single pass, into a single file:

zpaqfranz a backup.zpaq c:\data -key ClientSecret -franzen SharedWithProvider

The result is backup.zpaq.franzen (magic FRENZEN): AES inside, Franzen outside, two passwords, two salts, two key derivations. No cleartext .zpaq is written.

Workflow:

1. The client creates the archive with both passwords.
2. The client sends backup.zpaq.franzen to the provider (rsync --append for the following versions).
3. The client shares only the Franzen password with the provider.

Why this matters:

• End-to-end privacy: the provider can verify integrity without seeing the contents (it lacks the inner AES key).
• Server-side verification: the provider checks every block, with many threads, without ever seeing the data:

  zpaqfranz franzen test backup.zpaq.franzen -franzen SharedWithProvider -ssd -t8
  

• Defense in depth: if one of the two ciphers were someday broken, the other layer would still protect the data.

Interrupted Updates (and Ransomware)

• New data is always appended after the existing content. Previous segments are never overwritten.
• An interrupted update leaves every earlier block untouched: zpaqfranz reads the file up to its last complete update, and the next a continues from there. Only the unfinished update is lost.
• Ransomware: Franzen alone is not a defense (a program that can write to the archive can destroy it, like any other file). What protects is a copy that ransomware cannot reach: another machine, an offline disk, a provider with read-only snapshots. Franzen's append-only design makes keeping that copy up to date cheap: rsync --append sends only the new bytes.

Rapid Integrity Verification

Verifying a file does not require decompressing it or writing anything to disk:

1. Read each block's nonce and ciphertext.
2. Verify the Poly1305 tag (this needs the key: whoever verifies needs the Franzen password).
3. Throw the decrypted data away.

The work is split among the threads, so the check runs at the speed of the disk. franzen test checks the Franzen layer only; t tests the whole archive (decompression, hashes) and needs every password.

Using Franzen

With the archive commands

With -key too, give both passwords to every command (-key aespw -franzen pw).

What is written:

• -franzen alone, or with -key: only name.zpaq.franzen. The cleartext .zpaq is never opened.
• -chunk and backup: every part (and the index of a backup) is a Franzen file of its own; the parts share the salt of the first one.
• -debug when the archive is created: both name.zpaq and name.zpaq.franzen, to compare them. A test mode, not for real data.
• A write that fails on a Franzen file (disk full, I/O error) ends the run with an ERROR (exit code 2), never "all OK".

Standalone: the franzen command

Franzen can also encrypt any file on its own: a database dump, a disk image, a VM snapshot.

• Encrypt or decrypt is decided by the file: a Franzen file is decrypted, anything else is encrypted.
• Output names: file gives file.franzen, and back. If that name exists, file.encrypted.franzen or file.franzen.decrypted is used; if that one exists too, the command stops. Nothing is ever overwritten.
• Threads: one by default; -ssd turns on multi-threading, -tN sets how many.
• franzen test decrypts and checks every block, writing nothing. franzen ispasswordok reads only the header: it takes the same time whatever the size of the file.

Passwords

• The password goes right after -franzen: -franzen MyLongPassword. A -franzen with nothing after it, followed by another switch, or with an empty value (-franzen "$PW" with PW not set) is an error (exit code 2), and nothing is done. The same for -franzen2. (Up to 65.18h such a -franzen was ignored, and the archive was written in clear.)
• Opening an existing Franzen archive without -franzen (leave the switch out): zpaqfranz asks for the password on the console (no echo). With no console (scripts, cron) it stops with an error instead of waiting. -key without a value asks too.
• Keep passwords off the command line where you can: on Linux and macOS the arguments of a command are visible to all users via ps and end up in the shell history. In scripts use the environment variables FRANZFRANZEN (for -franzen) and FRANZKEY (for -key), and unset them right after; at the console, set +o history (bash) or a leading space with setopt HIST_IGNORE_SPACE (zsh).
• Length: up to 255 bytes. A longer -franzen password is refused when an archive is created.
• Strength: Argon2id slows guessing down a lot, but it cannot save a weak password. zpaqfranz franzen dump rates the password (length, character classes, entropy, estimated brute-force time) and flags anything under 8 characters.
• Keyfile (-keyfile, AES layer only): the content of a file (4 KB to 50 MB, at least 7.5 bit/byte of entropy: a JPEG, a .zip) is combined with the -key password: something you know plus something you have. zpaqfranz work keyfile file checks a file and shows its key, to keep on paper. Keep a copy of the keyfile: without it the data cannot be recovered, even with the password.

Cryptographic Foundations

• ChaCha20-Poly1305 (IETF, RFC 8439): confidentiality and tamper detection in one pass, a 16-byte tag per block. No lookup tables, so no cache-timing leaks. The data blocks are encrypted with no associated data; the header and the footers are authenticated as associated data.
• Argon2id (RFC 9106): memory-hard key derivation, so every guess costs a lot of RAM. zpaqfranz uses 3 passes and 256 MB, with a random 16-byte salt per file (a multipart archive: the salt of its first part); the key is derived once per run and kept for all threads and parts. The two parameters are stored in the header: change them, and the key comes out different and the password check fails.
• Nonces: a fresh random 12-byte nonce every time a block, a footer or the header is written, also when a block is rewritten in place. The same file encrypted twice with the same password gives completely different bytes (the salt is random too). With random 96-bit nonces the usual safety limit is about 2³² blocks per key: 256 TB with 64 KB blocks.

File Format Specification (Version 4)

Files written today are version 4. Version 3 (the same layout, before the FRENZEN magic existed) is still read. All numbers are little-endian.

Header (offset 0, 144 bytes, written once, never modified):

Data block (repeated): [nonce 12 B][ciphertext N B + Poly1305 tag 16 B], with N ≤ block_size; the last block of each update may be shorter (no padding). A full block takes 65,564 bytes on disk.

Footer (84 bytes, a new one at the end of every update; the old ones stay):

Overhead: 28 bytes per 64 KB block (0.043%), plus 228 bytes (header and footer), plus 84 bytes for every further update. A 1 GiB file written in one go: 458,980 bytes.

Threat Model

Franzen protects against:

• Unauthorized access to the data at rest (ChaCha20-Poly1305, with an Argon2id-derived key).
• Silent data corruption (bit rot): the tag of every block detects it.
• Altered bytes in the data blocks, in the footers and in the header fields that matter.
• Brute-force password guessing: 256 MB of RAM and 3 passes for every guess.
• Nonce reuse: a fresh random nonce for everything written.
• Interrupted writes: the previous updates stay intact and readable.

Franzen does not protect against:

• Block reordering. Whole blocks swapped, repeated or moved (also between the parts of a multipart archive, which share one key) are not noticed by Franzen. Inside a zpaq archive, zpaq's own hashes catch it (t, x). For a file encrypted with the standalone franzen command, keep a hash of the original if this matters.

• Older copies (rollback). An older, complete version of the file, or the file cut back to one of its earlier footers, is a valid Franzen file. To know you have the latest version, compare sizes or hashes with what you expect.

  These two are limits of integrity, not of confidentiality: in neither case can anyone read the data without the password (see What Franzen is for).

• A key in the memory of the running process. The key must be in RAM while data is encrypted and decrypted; a memory dump of the running process can contain it. -memzero wipes keys and buffers as soon as they are no longer needed (slower): it shortens the window, it does not close it.

• A compromised operating system. Whoever controls the system can read keys, plaintext and passwords as they are typed. No user-space encryption can defend against that.

• Weak passwords. Argon2id slows guessing down; it cannot make a weak password strong (see Passwords).

• Traffic analysis. File sizes, timestamps and I/O patterns stay visible: Franzen encrypts the content, not the existence of the data.

Part four, for developers

1. How it was tested

• franzsodium against libsodium, a bench of its own (tortura/sodio): the
  test vectors of RFC 8439 and RFC 9106; 200,000 random encryptions and
  decryptions and 10,001 key derivations compared with libsodium 1.0.18 (Windows,
  64 and 32 bit) and 1.0.22 (Fedora; FreeBSD and OpenBSD 50,000 + 2,501): the same
  bytes, refusals, errors and buffers too. AddressSanitizer and
  UndefinedBehaviorSanitizer clean on Fedora and FreeBSD. Where there is no
  libsodium (ESXi, the eleven NAS targets: x86 native, ARM under qemu), a
  "golden" file of cases written by libsodium (20,000; 2,000 under qemu): the
  same.
• zpaqfranz with the two engines: matrice.sh 59 of 59 and matrice2.sh 68
  of 68 (a, t, x, -until, repack, password -franzen2, l, i,
  dump, tw, AES + Franzen...) on Windows and FreeBSD, 56 and 68 on Fedora and
  OpenBSD; identici.sh (multipart backup with Franzen, written by one engine,
  read by the other, both ways) 376 of 376 on Windows; with -sodiumcheck always
  on, 374 of 374 on Windows, Fedora, FreeBSD, OpenBSD. A difference planted in the
  comparison: -sodiumcheck exit code 2 in a and in t.
• With no libsodium (nolib.sh, POSIX): 22 of 22 on 32-bit Windows, ESXi,
  NAS x86_64, i686, armv8 and armv7 (qemu), the "open" build; 19 of 19 on 64-bit
  Windows and a static FreeBSD. franzen autotest 62 of 62. A deterministic
  archive (-715, fixed salt and nonces): the same bytes (cksum 1953340381) on
  ESXi, FreeBSD, the four NAS, Oracle Linux 7.9; the one of ESXi extracted on
  Windows with both engines.
• The levels: livelli.sh, 18 of 18 on Windows: each name gives the archive
  of its -m, byte by byte; the 4 errors, exit code 2, no archive written; what
  is written wins (-lean -t4, -lean -turbo); -lfast, -nolfast,
  -nofast. -ini: 4 cases, and tortura/ini 69 of 69. The same, 7 of 7, with
  the NAS binary x86_64. Where there is no zstd (65.18q), livelli_nozstd.sh
  (POSIX) 19 of 19 on ESXi and on a -DANCIENT build of Oracle Linux 7.9: each
  name gives the archive of its zpaq model, byte by byte, with no "method >5";
  -verbose; -lean -t1; the errors; t of each archive. On ESXi, 3 MB with
  -lean: 0.11 s and 35 MB (with 65.18p, where it was -m5: 3.9 s and 178 MB).
• The RAM in the last line: OS REPORT against an outside measure, to the
  byte: PeakWorkingSet64 of PowerShell (Windows), time -l (FreeBSD, OpenBSD,
  macOS), VmHWM (Fedora, NAS x86_64 and i686).
• The batches of -turbo: the same archive (md5) for each method, with
  batches of 1 MB, automatic, 64 MB, -noturbo, -t2, -t4, -t16; then 41
  runs on a 10.3 GB VM disk, the same CRC for each method.
• The affinity: Windows start /affinity, Fedora taskset (1, 2, 3 cores),
  the NAS binary x86_64 and armv8 (under qemu).
• The hang of pp: -ramsize 20MB, -t1 to -t16, scattered fragments,
  Franzen; x -recover on a good and on a damaged archive gives the same bytes as
  before.
• -tar -not "X/*": t350.sh 43 of 43 on Oracle Linux 7.9 (28 of 44 with
  65.17o: the defect seen).
• work proxmox audit: Proxmox VE 9.2.2, by SSH from Windows and on the node;
  secrets planted (WireGuard keys, .my.cnf, sasl_passwd, a crontab with -p,
  tokens, PEM keys): 0 in the public archive.
• Issues 351 and 352 (65.18r), prova351.sh (POSIX), 21 checks: l -all
  with 15 hashes (the "b" ones, the others, -nochecksum, the default): a file
  added +, changed #, deleted -, added back +; -715 and -nolfast
  archives too; -stdin -noattributes the same attributes as a file added with
  -noattributes, without it as before; t; h a. 21 of 21 on Windows, Fedora,
  FreeBSD, OpenBSD, Oracle Linux 7.9 (also -DANCIENT), the NAS x86_64 and i686,
  ESXi, Solaris, macOS, openSUSE, Arch; 65.18q fails 13 of them.
• Every build, from this very source, 65.18r: Windows 64 bit (g++ 14.2,
  -O1 -Wall, and the syntax with -DANCIENT), Fedora 44 (gcc 16, -O1 and
  -O3, with mount, and the syntax with -std=c++11, -DANCIENT, -DNAS),
  FreeBSD 14.2 (clang, static and with mount), Oracle Linux 7.9 (gcc 4.8.5,
  -std=c++11, and the -DANCIENT line, now with no -ldl), OpenBSD 7.9 (clang,
  with mount), macOS 12 (x86_64, arm64, macFUSE, FUSE-T), ESXi (gcc 3.4.6),
  Solaris 11.4 (gcc 7.3), the eleven NAS binaries (musl), openSUSE, Arch (gcc 15),
  with and without mount: 0 errors, the warnings are the usual ones.
  autotest -all OK; autotest on ESXi, Solaris and the two x86 NAS.
  -helpcheck as before. tortura/tutti/fumo16.sh 10 of 10 on ESXi, Oracle,
  Solaris, NAS x86_64, FreeBSD, OpenBSD, macOS, openSUSE, Arch, Fedora;
  nix_controlc.sh 6 of 6 on each system with a handler (not ESXi, not
  Solaris).

2. Known limits and open points

• franzsodium has had no independent audit, nor has the format of Franzen.
  The limits of the format are in the Threat Model.
• The levels on a real NAS: -lean is -m80 -t2, measured on a real
  Cortex-A15 with 65.17o (point 3 of part one). Other NAS, other ARM: not
  measured.
• The arm64 half of the macOS binary and the ARM binaries for the NAS are
  built, not run on real hardware here (the ARM ones run under qemu-user).
• The affinity (point 8) is read on Windows and Linux; on FreeBSD and macOS
  the cores are counted as before.
• The free space is read as double on macOS and OpenBSD, and on FreeBSD where
  BLOCKSIZE is not in the environment (getbsize() gives blocks of 512 bytes,
  counted as 1024). work checkspace can say OK with half of what it says free.
  Way out, until it is fixed: BLOCKSIZE=1024. (Open since 65.17.)
• The internal RAM count does not see what zstd and the C++ library take,
  and free does not subtract: OS REPORT is the number to trust.