zpaqfranz 65.2

From 65.1a (2026-09-19) to 65.2k, one new command, a DEFLATE encoder written from scratch, and a long
list of Windows things that used to fail on filenames nobody should be allowed
to create in the first place.

The headline is zip: an archive can be turned into one single ZIP64 file,
which any machine on this planet can open, with no zpaqfranz at the other end.

Part one, what is new, for the user

1. zip: the archive into one single .zip
2. zip -deflate: the same .zip, three times smaller
3. zip -all: one folder per version, the way mount shows them
4. Names Windows cannot create, fixed while extracting
5. rd: deletions that get to the end
6. mount: one engine, an ETA, a lot less RAM
7. zpaqfranz-full: WinFsp without downloading anything
8. Wildcards on *nix, patterns on dir
9. Franzen-only archives
10. Smaller things

1. zip: the archive into one single .zip

zpaqfranz zip backup.zpaq

That is all. z:\backup.zpaq becomes z:\backup.zip, with everything the last
version holds inside it.

zip is the x command: same filters, same -until, same -range, same
everything. The only difference is where the bytes land, one single ZIP64 file
instead of thousands of files on a filesystem.

The point is a restore file that travels. The Linux, NAS and Mac filenames that
Windows cannot even create (: * ? " < > |, a trailing dot or space, CON,
LPT1, paths longer than 255) are perfectly legal inside a zip. So the
whole tree can be carried on a Windows box, or on an exFAT disk, handed over to
somebody who has never heard of zpaq, and unpacked back on the original system
exactly as it was.

zpaqfranz zip nas.zpaq -to z:\restore.zip              :: where you want it
zpaqfranz zip nas.zpaq /share/PUBLIC -to z:\pub.zip    :: only one folder
zpaqfranz zip nas.zpaq -until 100 -to z:\v100.zip      :: a specific version
zpaqfranz zip nas.zpaq -force                          :: overwrite the default name

Without -to the .zip is born beside the archive, same name, .zip extension,
and in that case (and only in that case) it must not exist yet, unless -force.

What ends up in there: symlinks (as real symlinks, with their target as data),
*nix permissions, the modification time with no timezone nonsense (Info-ZIP
UT field, one second resolution), empty folders, and files bigger than 4 GB.
Read-only by construction: the archive is never touched.

Nothing is compressed by default. The data inside a .zpaq is already
compressed, unpacking it and packing it again costs time, and the STORED zip is
what makes the whole thing cheap and fully multithreaded. If you do want it
smaller, that is the next chapter.

2. zip -deflate: the same .zip, three times smaller

zpaqfranz zip sources.zpaq -deflate -to z:\sources.zip

Real numbers, measured on 19 files of Pascal and C++ sources, 5.78 MB:

zip              5.783.027 =>             2.141.820 (37.04%, 3.47 MB saved)

On zpaqfranz.cpp alone (4.68 MB) it gets 37.6%, where gzip -1 gets 35.1%
and gzip -6 gets 30.0%. Throughput on one core is about 95 MB/s, everything
included (zpaq decompression, CRC-32, compression and writing).

It is method 8, plain DEFLATE, the one thing every unzip on earth has been able
to read since 1993. No new dependency: zpaqfranz does not link zlib and still
does not.

It is meant for sources, text, logs, configuration files, mailboxes, anything
that is still in its original form inside the archive. What is not worth it is
left alone: folders, symlinks, files under 64 bytes, and the extensions that
carry compressed data already (.zip .gz .7z .jpg .png .mp4 .mp3 .docx .xlsx .apk .jar and a few dozen more) stay STORED, so no CPU is wasted on them.

Even a file that looks compressible and is not cannot grow: on 200 KB of pure
random data the entry came out 22 bytes longer than the original, not the 12%
that a naive DEFLATE would have added.

-zip is an alias of -deflate, if you find it easier to remember.

3. zip -all: one folder per version, the way mount shows them

zpaqfranz zip backup.zpaq -all -to z:\every.zip

every.zip
   VER00000000\...   the archive as it was after the 1st add
   VER00000001\...   ... after the 2nd
   VER00000002\...   ... and so on

Exactly the folders the mount command shows, with exactly the same names and
the same numbering, so VER00000002 in the zip is VER00000002 on the mounted
drive and is what x -until 3 would write on disk.

Every folder is a full snapshot, not a delta: a file that never changed is
in all of them, a file deleted in version 3 is in the first two folders and
nowhere else. Which makes "give me the whole backup history in one file I can
hand to the lawyer" a single command.

The price is written on the tin, because a zip does not deduplicate anything:
an unchanged file is stored again in every single folder. So the damage is
measured before anything is created:

zip  -all  1.349 version folder(s) (VER00000000 ... VER00001348)
zip  -all  32.534.075 entries  7.552.157.500.930 (6.87 TB), about 15.15 GB of RAM
71401! zip: -all, not enough free space. -range/-until to write less, -space to bypass

That took 50 seconds on a 1,349 version archive and stopped there, without
eating the machine. Past 5 million entries it asks for -force as well.

-range is what makes it usable on a real archive: it picks which folders get
written, while the versions before them are read anyway, because that is what a
snapshot is made of.

zpaqfranz zip backup.zpaq -all -range 10:20 -to z:\10_20.zip
zpaqfranz zip backup.zpaq -all -range 1348  -to z:\last.zip

-deflate works together with -all, if you have the patience.

4. Names Windows cannot create, fixed while extracting

Extract a NAS backup on Windows and, until now, it died one file at a time with
ERROR_INVALID_NAME (123), "the filename, directory name, or volume label
syntax is incorrect". The names were perfectly legal where they were made and
are forbidden here: < > " | ? * : and the backslash, the control characters, a
name ending with a dot or a space, the DOS device names (CON, PRN, AUX,
NUL, COM1...LPT9).

Now x (and w) fix them on the fly, by default, and say how many were
changed. -nosanitize turns it off if you want the old behaviour.

The drive letter and the //?/ prefix survive the treatment, which is the part
that used to go wrong: a long path stored inside the archive, extracted -to
another long path, became //?/z:///?/z:/something and Windows refused it.

5. rd: deletions that get to the end

zpaqfranz rd z:\kajo -force -kill -space

Three changes, all of them about finishing the job.

It does not stop at the first failure. One locked file used to leave the
folder half deleted, with no clue about which object was the culprit. Now
everything else is deleted anyway, and at the end:

71345! NOT deleted     3
71346! First error     00000032 The process cannot access the file because it is being used by another process
71347! on <<z:/kajo/logs/current.log>>
71348: use -verbose to get every object that could not be deleted

Long paths and impossible names no longer need -longpath. Deleting does
not need any compatibility with anything, so rd always goes through the
extended //?/ syntax: a folder called acustica (with a trailing space),
which cannot even be opened by a normal Win32 call, is now removed.

Junctions and directory symlinks are removed, never followed. This one
mattered: walking into a junction meant erasing the files it points to,
somewhere else on the disk. Now the link is removed and its target is left
exactly as it was.

6. mount: one engine, an ETA, a lot less RAM

Much less memory on archives with many versions. A version used to be a
full copy of the file table. With 746 versions of a 455,000 files NAS backup
that is 340 million map nodes, tens of GB, std::bad_alloc long before the
drive letter appeared. A version is now a point in the history of each file, and
a file that never changes across 300 versions is stored once.

The long wait tells you how long. Building the directory trees of a
1,349 version archive used to print a line with no idea of the end:

 00:00:27  V00000367  sorting 72.871 files

and now:

  43.07%  00:00:40  eta 00:00:53  V00000581  sorting 72.871 files

One engine. The core engine (the memory-mapped scanner inherited from the
standalone zpaqfuse project, plain archives only) is gone from the options:
jidac reads everything it read, plus encrypted, Franzen and franzen-only
archives. -backend auto and -backend jidac are the same thing now. The
source of core has not been deleted, it is commented out line by line.

7. zpaqfranz-full: WinFsp without downloading anything

zpaqfranz-full.exe carries its dependencies inside itself and lays them down
when they are needed. The WinFsp installer is now one of them.

So on a machine with no WinFsp, mount finds the MSI inside the executable,
checks its SHA-256 and offers to install it, with no network at all:

WinFsp is not installed (or cannot be loaded)
winfsp-2.1.25156.msi extracted from this very executable (no download)
Installer verified (SHA-256 OK) C:\Users\...\Temp\winfsp-2.1.25156.msi

The plain zpaqfranz.exe keeps doing what it did before, downloading the
pinned installer and verifying it.

8. Wildcards on *nix, patterns on dir

On Windows the shell expands z:\foo\nas* before zpaqfranz sees it. On Linux
and BSD a quoted pattern arrived as it was written and matched nothing.
Now it is expanded internally, the same way:

zpaqfranz a backup.zpaq "/tmp/nas*"

And the dir command applies a pattern to the name, not to the whole path,
which is what you expect when you type

zpaqfranz dir /tmp/nas*

without having to fight with -only */nas*.

9. Franzen-only archives

An archive can now really be called pippo.franzen and have no cleartext
.zpaq side at all. Everything that looked for <archive>.franzen now
understands that, when the archive itself is the franzen file, there is nothing
else to look for and the cleartext file must never be opened.

10. Smaller things

• Windows seek: fseeko() reported a failure that had happened somewhere
  else, long before. A seek to offset 0xFFFFFFFF and a real error look the
  same to SetFilePointer(), and the only way to tell them apart is to clear
  the error code first. Now it does.
• copy with several sources counted and copied the same file once per
  source. Fixed, and the error message now says which file it was about.
• Windows error messages are no longer cut at 30 characters when they are
  printed on their own (the padding is there to keep the columns aligned in the
  error report, and only there).
• x -recover, heavy testing and zip no longer print one line per file
  for "no stored hash" and "all zero hash": on 78,000 files that was hundreds of
  lines of scroll for something the summary says in one number. -verbose
  brings the list back.

Part two, technical detail, lo spiegone

1. zip

~1,000 lines before a single byte of DEFLATE, all of them free functions plus
one struct, right before Jidac::extract(). The command letter is 'x':
zip sets flagzip= true and goes through the very same extract(), which is
why every filter, every -until, every -range works with no extra code.

Why STORED is not laziness

Everything is method 0, and that is what makes the extractor work unchanged:
the compressed size IS the uncompressed size, and it is known before
extracting anything. So zipprepare() walks dt once, computes the position
of every local header and of every byte of data, checks the free space, writes
all the local headers, and stores into each DT the absolute offset its data
has to go to (zipdataoffset, zipindex).

After that the extractor keeps doing exactly what it does on N separate files:
decompressThread seeks to offset + job.zipbase and writes. The zip is one
file that is never closed between entries (g_zipfp), the write mutex it
already had is enough, and the whole thing stays multithreaded.

The CRC-32 that nobody (re)computes

A zip entry needs the CRC-32 of the uncompressed data. Computing it again would
mean reading everything a second time. It is not computed at all: the extractor
already calculates the CRC-32 of every chunk it writes (g_crc32, the same
machinery -checksum uses to verify), so zipfinalize() groups those chunks by
filename, sorts them by position, and combines them with crc32_combine().
A hole that was never written (an all-zeros run the extractor skips) is combined
in as crc32zeros(n).

Names, symlinks, attributes

zipmakename() turns a stored name into a legal zip name: backslashes become
slashes, //?/ and //?/UNC/ prefixes go away, X:/something becomes
X/something, leading slashes are dropped, and .. is neutralized in four
passes so nothing can escape the extraction folder.

A symlink has no fragments at all inside the archive: the target sits in the
posix block. In the zip it becomes an entry whose data is the target itself,
with mode 0xA1FF (S_IFLNK | 0777) in the high 16 bits of the external
attributes and "version made by" saying Unix, which is what unzip restores as
a real symlink.

zipexternalattr() carries the *nix mode when the archive has one
(attr & 255 == 'u'), or a sensible 0755/0644 when it does not, plus the MS-DOS
directory bit. Dates go in twice: the MS-DOS pair (two seconds, no timezone,
1980 to 2107) and the Info-ZIP UT extra field with the real Unix time.

Zip64 is used per entry, on the uncompressed size, and for the end of central
directory when there are more than 65,534 entries or the directory does not fit
in 32 bits.

-deflate, the encoder

section: deflate, ~330 lines, no external library.

Minimal means fixed Huffman codes only (BTYPE=01): the tables are the ones
printed in RFC 1951, so there is no tree to build, no code lengths to transmit
and nothing to get wrong. zipdeflate_preparetabelle() builds them once,
already bit-reversed, because deflate writes a Huffman code starting from its
most significant bit and everything else from the least.

The matcher is a 3 byte hash ((b0<<16|b1<<8|b2) * 2654435761 >> 16), 64 K
buckets, a prev[] chain, greedy, at most 16 candidates per position, no lazy
matching. The positions a match jumps over are inserted into the table anyway,
because that is where the next match usually is.

Input is compressed 256 KB at a time and no match ever crosses a chunk: it
costs a whisker of ratio and it removes the sliding window, the rebasing of the
hash table and every bug that lives in there. The bit stream, on the contrary,
runs across the chunks, so what comes out is one single deflate stream closed by
an empty final block.

While the tokens are built their exact cost in bits is summed. If the fixed
block is not smaller than the same data in stored blocks (BTYPE=00, 5
bytes of header every 64 KB), the stored blocks are written instead. That is the
bound: incompressible data costs 5 bytes every 64 KB, not the 12% that fixed
Huffman codes would add to it.

s_deflate is built once and reused entry after entry with begin(FP):
head[] 256 KB, prev[] 1 MB, tokens 1 MB, staging buffers. About 2.6 MB, once.

-deflate, why it needed another engine

The parallel extract() cannot do it. Its whole design is the precomputed
layout described above, and a compressed entry does not know where it ends until
it has ended.

The sequential engine of extractstdout() does exactly the opposite: it walks
dt file by file and, for each file, its fragments in order, pulling the blocks
from a cache that a pool of workers fills ahead of it. That is a stream, and a
stream is what a deflate wants.

So zip -deflate goes there instead:

Jidac::extract()          arguments, read_archive, -all expansion, filters
   flagzip && flagdeflate -> extractstdout(0, "")   (dt is already loaded)
extractstdout()           worker pool, block cache, one file at a time
   zip_export_handler     header, deflate stream, crc, patch, central directory

extract_handler gained two virtual methods with an empty default
implementation, begin_file() and end_file(), so the partition, raw and
normal handlers do not even notice. In the main loop the change is three lines:
open the entry, close the entry, and one else handler->write_fragment(...)
where the file handle would have been.

zip_export_handler writes the local header with the CRC and the compressed
size at zero, feeds the fragments to s_deflate, and when the file is over
patches CRC, compressed size and uncompressed size back into the header (the
32 bit fields, or the 8+8 inside the zip64 extra, whose offset it remembered).
Entries come out in dt order, which is already sorted by name, so the central
directory needs no sorting.

zipgrande() decides zip64 on the uncompressed size plus the worst case a
deflate can add to it, so a layout decided before compressing is still right
after it. zipcsize() returns the uncompressed size when the method is 0, which
is why the STORED path did not have to change a single line.

Files under 64 bytes, folders, symlinks and the compressed extensions
(zipdeflate_inutile()) get method 0 and go straight out.

Since the data flows through the sequential engine, zip -deflate also
verifies the stored hashes while it compresses, for free, which the parallel
path does not do.

Validated by three independent decoders: Python zipfile (testzip()),
Info-ZIP unzip -t and 7z t, on sources, mixed data and a 93 MB file whose
SHA-256 after a round trip is the original one. Content, names, dates,
attributes and CRCs of a deflated .zip are identical to the STORED one of the
same archive.

-all, the version folders

read_archive(), when all > 0, already keys every entry it reads as
0001/name, one group per version, holding what that version did, a date of
0 being a deletion. That is the whole history.

zipexpandversions() replays it from the first version onwards and, at every
step, pours the live state into VER%08u/, which is
franzmount::mount_version_name() and is 0 based like it (internal version v
becomes folder v-1, because mount calls the first version VER00000000).

Two passes:

1. projection, O(changes) and not O(versions x files): the live byte count
   is kept up to date while the history is replayed, so nothing has to be walked
   twice. It prints folders, entries, bytes and an estimate of the RAM
   (500 bytes per entry, which on the 1,349 version test said 15.15 GB against
   the 15.61 GB actually measured), then refuses if the bytes do not fit on the
   destination (bypass -space) or if the entries are more than
   ZIPALLMAXENTRIES (5,000,000, bypass -force);
2. build, with a progress line every 200 ms.

insert(), never operator[]: a default constructed DT calls
franz_malloc() for its franz_block, and 32 million of those is not a good
idea. DT has no destructor (it is commented out in the source, on purpose), so
the copies share that pointer and nothing is ever freed twice.

-range cannot be allowed to filter the read, because the snapshot of version
300 is made of everything the 299 before it did. extract() puts it aside
around read_archive() and hands it to the expansion, which uses it to choose
which folders get written. Note that -range alone already forces all= 8, so
zip -range 3 now writes the full snapshot of version 3 and not its delta.

The expansion runs right before the extraction marking loop, because from there
on dt is walked to decide what to extract and iterators into it are taken
(block[].files), so nothing may be moved around any more.

2. Windows filenames

franzrootlength() says how many leading characters are the "root" and must be
left alone: 7 for //?/X:/, up to the share for //server/share/ and
//?/UNC/server/share/, 3 for X:/, 2 for X:, 1 for a *nix absolute path.

franzsanitizepath() fixes everything after the root and nothing else:

• a doubled root (//?/z:///?/z:/something, which is what a long path stored
  inside the archive glued to a long path -to produces) is collapsed, up to 8
  times;
• forbidden characters (< > " | ? * : and backslash) and control characters
  become _;
• with i_strict (that is -fixreserved) it also fixes what Windows accepts
  and then silently mangles: trailing dots and spaces, and the DOS device names.

Jidac::sanitizzawindows() rewrites the keys of dt with it, counts how many
changed, and is called by extract() and extractw() unless -nosanitize.
It never runs for zip, where those names are exactly what has to be preserved.

forcelongpath() is the other half: it turns any path into the extended //?/
form, expanding a relative one first, without caring about -longpath. Some
operations (erasing, for one) never need compatibility with anything.

3. rd

erredbarras() was rewritten around three ideas.

erredbarrasfallito() is called on every failure: it counts it (g_rd_errors),
remembers the first one and its object (g_rd_lasterror, g_rd_errorpath), and
returns, so the walk goes on. The report at the end prints the count, the
decoded Windows message of the first error and the object it was about.

Reparse points are checked before recursing:
(attributi & FILE_ATTRIBUTE_REPARSE_POINT). A junction or a directory symlink
is removed with RemoveDirectory on the link itself, which leaves the target
alone. Without this, deleting a folder could erase files on the other side of
the disk.

Names go through wtou() everywhere, instead of the old
std::string(w.begin(), w.end()) that truncated every non ASCII character, and
stermina() forces the extended syntax on the whole path before starting.

4. mount

The history of a file, instead of a copy per version

Both engines used to hold, per version, a map<string, record> of the whole
tree. 65.2 replaces it with the changes:

struct JidacChange { unsigned version; const JidacRec* rec; }; // rec NULL = deleted
typedef map<string, vector<JidacChange> > JidacHistMap;

one entry per file, and inside it only the versions that really touched it.
jidac_at(hist, v) is a binary search for the last change with
version <= v, jidac_touch() appends one (twice in the same version is an
overwrite). The records live in a deque, so pointers into it never move, and a
file that does not change across 300 versions is stored once, with its fragment
list stored once.

The core engine

Commented out, not deleted: from the Backend "core" (DISABLED) banner down to
the end of CoreBackend, 469 lines are the original source with a // in front
of each, verified identical to the original once the prefix is stripped.
CoreFragLoc, CoreStrWriter and mount_tree_add()/mount_tree_finish() stay
alive because jidac uses them.

mount_pick_backend() returns jidac for everything and keeps the original
choice in a comment; open_backend() has the core branch commented out and
says (available: jidac, auto); -backend core is refused as an unknown
backend.

The ETA of the tree building

MountScanProgress could only count bytes of the archive, which is the right
yardstick for the index scan and no yardstick at all for the phase that builds
the directory trees. It now also counts units of work, one unit being one
version's tree:

void begin_units(int64_t i_units);  // the phase counts versions, not bytes
void unit(int64_t i_done);          // i_done units are behind us
void subunit(double i_frac);        // how far the current one has got, 0..1

build_tree() feeds subunit(done/entries) while it walks, so a single version
mount (the default) has an ETA too, and Jidac::mount() calls unit(v) before
each prepare(v). Percentage and ETA are the same arithmetic as the byte based
ones, and the line becomes
43.07% 00:00:40 eta 00:00:53 V00000581 sorting 72.871 files.

In -catpaqmode the phase used to emit nothing at all (it returned early on
total_ <= 0); it now sends @SPK@EXT@ with percentage and ETA, and 0/0 in
the two byte counters, because the GUI formats those with FormatFileSize() and
a number of versions in there would read as bytes.

The totals walk that follows starts its own phase, so it does not inherit the
units of the one before it.

5. Resources, zpaqfranz-full

resources.rc stopped at 11 while fai.bat was already compressing the MSI
into 12.lib: windres ignored it, and the installer was not inside the
executable at all. The proof is arithmetic, the sum of 1.lib to 11.lib is
4,388,866 against a resources.o of 4,389,656, which is 790 bytes of COFF
overhead and no room for 1.28 MB of MSI. One line added to the .rc, and the
object goes to 5,672,482.

The table in kickstart_resources() gained the entry:

elenco_risorse.push_back(risorse(12, "WINFSP", "winfsp-2.1.25156.msi", 2191360,
    "073A70E00F77423E34BED98B86E600DEF93393BA5822204FAC57A29324DB9F7A"));

and mysqldump.exe had its size corrected from 4,875,064 to 4,785,064 (the hash
was right, the digits were transposed, and it only fed the progress bar).

estrairisorsa_in(number, destfile, size) extracts a resource anywhere,
where estrairisorsa() always writes next to the .exe because that is where a
DLL has to be. It returns false without a word when the resource is not there:
that is a plain zpaqfranz.exe, not an error.

kickstart_mount() now looks for the MSI in this order:

Size and SHA-256 are checked whatever it came from, then the captcha and
msiexec as before. The WINFSP package is left out of the
kickstart_resources("") sweep on purpose: the MSI is not something zpaqfranz
needs to run, and on a build without resources that sweep would mean 2 MB of
download that nobody asked for.

6. Wildcards and patterns

expandwildcards(filename, recursive) is the *nix half of what the Windows
shell does: opendir() on the folder part, ispath() on each name (case
sensitive here), no recursion into a matching folder, and never expanding into
something that is expandable again. myscandir() calls it when the argument
holds a wildcard and the file does not exist as it is written.

matchesfilepattern(pattern, filename) is for dir: the pattern is applied to
the name, not to the full path. It is deliberately not pushed into
onlyfiles, which would match the whole path and fail unless the pattern
started with *, and which the user may want for something else.

7. Franzen-only

franzenname(x) returns x when x is already a .franzen, and
x + ".franzen" otherwise. is_file_frenzen() does not refuse an archive whose
cleartext side does not exist, and OutputArchive never opens the cleartext
file when the archive itself is the franzen one.

8. Fixes

fseeko() on Windows: SetFilePointer() returns INVALID_SET_FILE_POINTER
both for a real error and for a perfectly good seek to offset 0xFFFFFFFF, so
the only way to tell them apart is to call SetLastError(NO_ERROR) first and
look at the error code only when the return value is that sentinel. Without it
fseeko() reported whatever some unrelated API had left behind, a
DeviceIoControl that failed long before being the usual suspect.

Jidac::mycopy(): edt is cumulative across scandir() calls, so a loop that
walked it once per source counted and copied the same file as many times as
there were sources. It is walked once now.

decodewinerror() takes an i_padded parameter: the 30 characters padding
keeps the columns of the error report aligned, and a message printed on its own
gets the whole text instead of a stump.

Note

Numbers in this document were measured on a single Windows 11 machine with
mingw-w64 builds. The deflate ratios depend entirely on the data, the mount ETA
depends on how many versions and how many files the archive holds.

Everything is built from one single .cpp, as usual:

g++ -O3 -std=gnu++14 -DZPAQMOUNT -I"C:/Program Files (x86)/WinFsp/inc" ^
    zpaqfranz.cpp -o zpaqfranz.exe -static -pthread ^
    -lws2_32 -ladvapi32 -luser32 -lurlmon -lole32

and zpaqfranz-full.exe is the same line plus c:\zpaqfranz\dll\resources.o.

Things to do

Testing on *nix
A better help for new commands