zpaqfranz 65.16

This is the release of repack. A zpaq archive is a journal: every
update is added at the end, nothing ever goes away. With years of backups in
it, the old versions cost space and nobody needs them all. Until now there
was no way out but extracting and starting again.

zpaqfranz repack z:\backup.zpaq 7d4w12my -to z:\new.zpaq

A new archive with only the versions you want: the last 7 days, 4 weeks, 12
months, every year. What they do not use is thrown away, and the new archive
is checked to the last bit before it gets its name.

• Which versions: a list (1,5,10-20), the last ones (47last, 7-last),
  or rules by date, grandfather-father-son (7d4w12my), with a reserve of the
  latest versions that are always kept.
• repack -test: which versions would be kept, and why. Nothing written.
• repack -quick: the blocks whose data are all kept are copied as they
  are. Not decompressed, not compressed again.
• repack -all: every version, so the same archive in another method.
• Another password, or none, or another kind of encryption: plain, AES,
  Franzen, AES + Franzen, from any of them to any of them.
• purge: the same command, by another name.

And:

• Files of a Linux or a NAS archive lost on Windows, with "all OK": read point
  9 of part one. Case.txt and case.txt, finale. and finale, .bashrc and
  .BASHRC, a file called nul: one of the two was lost, and x said all OK,
  exit code 0. No more: every file comes out, and what is renamed is said.
• w and tw on many archives: tw z:\test\*.zpaq, with a list at the
  end of what is not OK.
• h -no*: the help of a switch takes * and ?.
• -nodashboard works for every command.
• -tar on every build that can do it: an Oracle Linux 7.9 built by hand
  with -DANCIENT gave 79531! Cannot get metadata on every file. Now it
  works. Where it cannot (ESXi, the open build) it says so, before writing.
• After a control-C the console is as it was: no more all yellow, all
  green.

Part one, what is new, for the user

1. repack: only some versions
2. Which versions
3. repack -test: what would be kept
4. repack -quick: whole blocks copied
5. Another method, another password
6. What you see while it works
7. w and tw on many archives
8. -tar everywhere it can be
9. Names that are one file for Windows
10. h -no*, -nodashboard
11. Smaller things

1. repack: only some versions

zpaqfranz repack z:\1.zpaq 1,last -to z:\2.zpaq          the first and the last
zpaqfranz repack z:\1.zpaq 1,10,20-25 -to z:\2.zpaq      some of them
zpaqfranz repack z:\1.zpaq 7d4w12my -to z:\2.zpaq        by date
zpaqfranz repack z:\1.zpaq -all -m1 -to z:\2.zpaq        all of them, -m1

Version 3 of the new archive is the version it comes from: the same files,
with the same dates, attributes and hashes, the same date of the version,
its comment. -until with a date says the same in both archives.

The source is never touched: the output is another file (-to), one file only.
It is written as <name>.tmp, read again from the disk, checked, and only then
renamed. Anything wrong: exit code 2, no output. With -force an output that
already exists is replaced, at the end, if all is good.

By default the data are decompressed, checked and compressed again with -m
(default -m8, -m1 in a build without zstd). -t as usual.

2. Which versions

Rules by date: a number and a letter, h d w m y. 7d is the last version
of each of the last 7 days that have versions; 4w the same for weeks, 12m
for months; a letter without a number is every period (y: every year).

• The periods are counted from the last version of the archive, not from today,
  and only the ones with versions: a backup that stopped in March does not
  lose its history in April.
• A week starts on Monday. The time is local, as l shows it (-utc: UTC).
• -n X, the reserve: the last X versions are always kept, default 10.
  Only with rules by date: a list is exact.
• A list and rules go together: 1,7d4w12my keeps the first version too.

zpaqfranz repack z:\1.zpaq 24h30d -to z:\2.zpaq              hourly: a day of them, then 30 days
zpaqfranz repack z:\1.zpaq 1,7d4w12my -n 30 -to z:\2.zpaq    the first too, and a reserve of 30

3. repack -test: what would be kept

-test shows it, for every version, and why. No -to needed, nothing written.
60 versions in ten months, one every five days:

zpaqfranz repack demo.zpaq 7d4w12my -test

V00000043 2026-07-30 06:00:00  keep 4m
V00000044..V00000048 2026-08-04 06:00:00 .. 2026-08-24 06:00:00  - 5 versions
V00000049 2026-08-29 06:00:00  keep 3m
V00000050 2026-09-03 06:00:00  -
V00000051 2026-09-08 06:00:00  keep reserve
V00000052 2026-09-13 06:00:00  keep reserve
V00000053 2026-09-18 06:00:00  keep reserve
V00000054 2026-09-23 06:00:00  keep 7d
V00000055 2026-09-28 06:00:00  keep 6d 2m
V00000056 2026-10-03 06:00:00  keep 5d 4w
V00000057 2026-10-08 06:00:00  keep 4d 3w
V00000058 2026-10-13 06:00:00  keep 3d
V00000059 2026-10-18 06:00:00  keep 2d 2w
V00000060 2026-10-23 06:00:00  keep 1d 1w 1m 1y
-test: 18 versions of 60 would be kept (7d 7, 4w 4, 12m 10, y 1, reserve (-n 10) 3): nothing written

4. repack -quick: whole blocks copied

Compressing everything again takes time, and most of the time it is not needed:
when you prune an archive, you want it smaller, not different.

zpaqfranz repack z:\1.zpaq 7d4w12my -quick -to z:\2.zpaq

A block whose fragments are all kept is copied as it is. The others are
compressed again with -m. On the archive above, 200 MB in 60 blocks:

The copied blocks are checked by their bytes (size, number, SHA-1 of the bytes
written and read back). They are not decompressed: a block already
damaged in the source is copied as it is, exit code 0, and t on the new
archive finds it. If that matters, run t after, or do not use -quick.

-quick also says how many blocks it copied, and how many went the long way.

5. Another method, another password

-all and a method: the same archive, compressed in another way.

zpaqfranz repack z:\1.zpaq -to z:\2.zpaq -all -m5

The output is encrypted as the source, unless you say otherwise. -key and
-franzen are the passwords of the source; -key2 and -franzen2 the ones
of the output, . for none, and with no value -key2 is asked twice:

zpaqfranz repack z:\1.zpaq -to z:\2.zpaq -all -key old -key2 .                   AES => plain
zpaqfranz repack z:\1.zpaq -to z:\2.zpaq -all -key old -key2 new                 AES => AES, another password
zpaqfranz repack z:\1.zpaq -to z:\2.zpaq -all -key old -key2 . -franzen2 pw      AES => Franzen
zpaqfranz repack z:\1.zpaq -to z:\2.zpaq -all -key2 k -franzen2 pw               plain => AES + Franzen

6. What you see while it works

On a console a map of the archive, the one of cp and f. It fills while the
index is read, then every data block has its state: skipped (no version
taken uses it, it is never read), to do, read, decompressed, compressed,
written, error. While the new archive is checked, its own map. Under it
percentages, ETA, and new A=>B: the size the new archive is going to have,
estimated from what has been written so far.

-nodashboard: a line a second. Redirected: nothing.

Control-C during repack deletes the .tmp and leaves the source alone.

7. w and tw on many archives

zpaqfranz tw z:\test\*.zpaq
zpaqfranz tw "/copie/*.zpaq"                                       (*nix: in quotes)
zpaqfranz w z:\test\*.zpaq -ramdisk -test -checksum -ssd
zpaqfranz w z:\test\*.zpaq -to p:\out -ramdisk                     (a folder each)

As t and x with a *, every archive in turn. A real extraction goes into a
folder of its own for each archive, named after it. At the end one line, and
only what is not OK, because the lines of every run are far above:

zpaqfranz tw "/copie/*.zpaq"
...
03795! 5 archive(s): 3 OK, 1 WARNING, 1 ERROR
ERROR   <</copie/rovinato.zpaq>>
WARNING <</copie/tagliato.zpaq>>

The exit code is the worst of them: 2 here.

8. -tar everywhere it can be

-tar was off in the builds made with -DANCIENT, and an issue showed what
that means: Oracle Linux 7.9, gcc 4.8.5, built by hand with -DANCIENT (even
if the note in the source says not to), and 79531! Cannot get metadata on
every one of its 160,922 files, 0 good.

Now -tar is in every ANCIENT build but ESXi: what it needs is old POSIX.
Without utimensat (glibc before 2.6, an old Mac) the times are in seconds.
On ESXi and in the open build, -tar is refused at once, before anything is
written (79532!, exit code 2).

The right way on those systems stays:

g++ -O3 -Dunix -std=c++11 zpaqfranz.cpp -pthread -ldl -o zpaqfranz

-DANCIENT gives less: no -m8 and the other zstd methods, no Franzen, no
multithread a.

9. Names that are one file for Windows

An archive made on Linux or on a NAS can hold names that for Windows are one file:
Case.txt and case.txt, finale. and finale (Windows drops dots and spaces at
the end of a name), .bashrc and .BASHRC, Dir/a.txt and dir/a.txt. Or names
that for Windows are not a file at all: nul. Each step of x looked at its own form
of the name, none at the final one. The last file written won, the other one was
gone, and x said all OK with exit code 0.

Files lost, on an archive of 32 files made on purpose, each one holding its own name:

Now every file comes out. There is one last look at the names as Windows will see
them, and whoever comes second gets a number:

03807$ For Windows <<src/Case.txt>> is the name of another file: extracted as <<src/Case_00000001.txt>>
03807$ For Windows <<src/.bashrc>> is the name of another file: extracted as <<src/.bashrc_00000001>>
03807$ For Windows <<src/finale.>> is the name of another file: extracted as <<src/finale_00000001>>
03809$ 10 file(s) renamed, the same name for Windows: their data are all there, not their names (a WARNING)

It is a WARNING, exit code 1, not an OK: the data are all there, but on the disk
case.txt may hold what was Case.txt. The same for a file and a folder with the
same name (Mix and mix/a.txt), and for the case of all Unicode, not of a-z only.
w and tw do the same, so a test looks where x put the files.

nul is a device: a file with that name wrote its data to nothing. Now it is
_nul, as a:b.txt is a_b.txt. Which names depends on the Windows, and it is
Windows that is asked. Windows 11: only nul after a folder, and a con with no
folder before it; con.txt, aux, lpt1 are files. Windows 10: all of them, with an
extension too (con.txt), and conin$, conout$. A folder too.

With -longpath a nul is a real file, as before (Explorer will not open it, nor
delete it). With -nosanitize nothing is changed: a name that is a device says
03808!, exit code 2.

zip keeps every name as it is: inside a zip Case.txt and case.txt are two
files. A collision became Case_00000001.txt there too, for nothing.

10. h -no*, -nodashboard

zpaqfranz h -no*         every switch -no..., with what it does
zpaqfranz h a -no*       the same, of the a command

In the search of the help a switch takes * and ?.

-nodashboard was for -image. Now it is for every command with a map or bars:
f -test, -nuke, -zero, t -ssd, w, tw, cp, repack. The classic
lines come back. The defaults do not change.

11. Smaller things

• w -paranoid -verify with files that are not as stored said "with warnings",
  exit code 1: now an ERROR, exit code 2.
• A file x could not give its date or its attributes (00163!, 00165!) was all
  OK: now a WARNING.
• -flat, -fix255, -fixreserved: two names that came out equal lost a file, with
  00904: KOLLISION! and all OK. Now the second one gets a number, and a WARNING.
• w -ramdisk: a file that cannot be created said "no free RAM?". Now its name.
• Colors after a control-C: the bars of t -ssd, the maps of repack and cp
  went on coloring for a while, and the last color stayed. Now after a control-C
  only the handler colors, and the console goes back to how it was found. l
  shows its CONTROL-C and Goodbye (they ended in a buffer never written).
• w -ramdisk -checksum crashed on a file not decompressed whole (a damaged
  block). w -ramdisk -test said 0 there. Now an error, 03797!, as w
  without -ramdisk says.
• Two mount at once in two shells with no letter: both chose the same free
  one before their scan, the second one failed after a long wait. Now the
  first free letter is chosen again just before mounting (94035). A letter
  given and in use is an error, at once and after the scan too (94034!). The
  Explorer window opens after the mount, not before: after a scan of more than
  30 seconds it never opened.
• repack: the size of the source is in the title of the map.

Part two, the details, for power users

1. repack: the switches

As a source, what can be read as usual: one file, a multipart with ????, a
backup. A multipart or -chunk output: not yet.

2. repack: what is carried, what is not

• Carried: files, deletions, dates, attributes and the franz block of every
  file, byte by byte, the date of the version, its comment.
• Not carried: the VFILE-info of the versions, the -fast history and the
  state of -fastadd: they speak of the versions of the old archive. After the
  first a on the new archive they start again by themselves.
• The fragments are numbered again from 1.
• The output of a Franzen archive is <to>.franzen, never a .zpaq beside it.
  -to x.franzen on an output without Franzen is refused.

3. repack: the checks

Before anything is written, the index of the source is read by a reader of its
own, stricter than the usual one: c, h and i blocks decompressed with
their SHA-1, d blocks skipped with the jump of the c and equal to the h,
the numbers without holes, no byte outside a block. A transaction not finished
is ignored, as every command does, and said (03771$).

After writing, the .tmp is read from zero by that same reader: dates,
comments, the state of every version against the source (name, date, attributes
byte by byte, SHA-1 and size of the fragments), every fragment of a version used
by a file of it, every block decompressed and checked. With -quick, the copied
blocks are compared by their bytes.

4. What can change for a script

• -nodashboard now also changes f, t -ssd, w, tw, cp.
• w and tw with a * in the name are many archives, and the exit code is the
  worst of them (w: without a *, as before).
• w -ramdisk -test on a file not decompressed whole: an error, it said 0.
• a -tar on ESXi and in the open build: exit code 2 and no archive. It was an
  archive without the metadata and a 79531! for every file.
• x, w, tw on Windows, an archive from a *nix with names that are one file
  for Windows: exit code 1 (it was 0, with a file lost), and some files have another
  name. nul is _nul.
• w with errors: exit code 2 (it was 1). Only errors reading the index, every file
  as stored: 1 or 2, as tw.
• x with a date not set (00163!): exit code 1 (it was 0).
• The lines 00900 and 00903 (the check of the case) are gone: only what was found
  is said.

5. The messages

Part three, lo spiegone, for developers

1. repack: separate on purpose

All in one block (prefix rp_, C++98 for ESXi) after cp_copia, because it
uses franzdash. Only a handful of hooks outside: the parser, the dispatch, the
table of the commands, the control-C handler. The index is read by rp_leggi,
not by read_archive: what does not check is an error, nothing is skipped, and
nothing of the state of read_archive is needed. Writing uses the CompressJob
of add, and the analysis of the method by block is the one of add
(rp_misura).

2. Where a fragment goes

The versions are replayed; for every version taken, the differences from the one
taken before. Each fragment goes into the first version taken whose files use
it. Not "per block": with that rule a file deleted and then back gave 72414
in x -until. So a source block can be read more than once, if its data
serve versions far apart (03784). Blocks no version uses are never read.

3. -quick: a block that can move

A d block has 0 as the ID of its first fragment ("omit first frag ID to make
block movable", as zpaq writes it), so it can move. It is taken whole only if
all its fragments are kept and all of them have their first use in the same
version (chi.size()==1): a fragment of a copied block that no file uses would be
a 72414 for ever, and the check of repack would refuse it.

rp_rinomina works on the bytes: the tag of 16 bytes, zPQ, level 1 or 2, type
1, hsize at 18..19, then the segment at 20 + hsize, 0x01 and a name of 28 bytes:
jDC + 14 digits of date + d + 10 digits of first fragment. Date and number are
written in place. If that does not fit, the block is compressed again. AES and
Franzen go through the streams as for every byte. For every version the copied
blocks come first, written before comp->inizia(), then the h of the copied ones,
then the ones of the compressed, the same order of the d: read_archive
computes the offsets from the sizes in the h.

4. The map and the sizes

rp_compressore never waits still: it queues only with room in the queue, and
closes only with everything written, waiting 50 ms at a time and drawing (with
-m3 the map stood still for 35 s inside join). "Compressed" and "written" are
one state: the CompressJob says only when it writes. numeri() counts the
copies: done = (copied + fraction of the others x their bytes) / all. The size
in new A=>B: compressed written / raw data so far x raw data to do, plus what
is copied. rp_indice goes through indice(fatti, totale) ("writing its index
X/Y files"): on 3.48 million files it was 28 s of silence at 100%.

5. Control-C

g_rp_temporaneo, g_rp_fp, g_rp_franzenlettore, rp_controlc() in the
handler. On Windows the handler is a thread of its own, and a file open cannot be
deleted: g_rp_fermo stops the thread of repack for ever in rp_fermati()
(at the start of rp_cruscotto::aggiorna), waited at most 5 s by the clock
(Sleep(10) is 15.6 ms: 500 turns were 7.8 s), then the files are closed and
deleted. On *nix, unlink at once. rp_guardia (RAII) from the creation of the
output.

6. The colors after control-C

Three causes. After the handler the other threads go on for a while, and
color. resetconsolecolor on Windows puts back the color there was. And
color_restore ended in the buffer of l (outbuf), never written, with the
messages of the handler.

g_filo_controlc and colore_fermo(): after a control-C only that thread
colors. g_console_iniziali, taken in color_save. colori_ripristina(finale)
writes straight on the console (Windows: ESC[0m if VT, then
SetConsoleTextAttribute, the mode, the cursor; *nix: ESC[0m ESC[?25h if it is
a tty), at the start of the handler (after fflush(stdout) and g_outbuf_on= false), before its exit, and with atexit. Measured through a ConPTY with a true
control-C: t -ssd dirty 2 times of 20 before, 0 of 60 after.

An ANCIENT build (not NAS) and Solaris have no SIGINT handler: as before.

7. TAR_SECONDS

-tar was off with ANCIENT && !NAS. Now savefilemetadata is off only with
ESX || OPEN; validate_metadata, tar_restoreuid and tar_settimes only with
ESX. New TAR_SECONDS = ANCIENT && !NAS && !UTIME_NOW: no utimensat on
that system (UTIME_NOW comes with it), so utime(), nanoseconds 0, and a
symlink keeps the time of the extraction. On glibc 2.17 UTIME_NOW is there: the
nanoseconds are all there.

8. The names on Windows

Three passes changed a name, each one with its own idea of it. casekollision()
used rename() as a key of dt (so the -to went in twice, out\out\src\), hashed
the new name without stringtolower(), and with .bashrc it got an empty name and
gave up (00418! ERROR, and rc 0). sanitizzawindows() ran after it, with a map where
the case counts: A:b.txt and a_b.txt were lost there. Nobody looked at dots and
spaces at the end. extractw() had no casekollision() at all. And with
i_fix= true the list of the collisions stayed empty: "No unfixed case collision",
always.

Now collisioniwindows() runs once, after all of them, in extract() and extractw(),
on rename(key), the name that will be opened. chiavewindows() makes the key:
LCMapStringW(LOCALE_INVARIANT, LCMAP_UPPERCASE) for the case of all Unicode, and
without //?/ no dots and spaces at the end of every piece. A hash of 8 bytes for each
file and each folder: sorted, and if nothing repeats the pass is over. Only then a
std::set (visti, the folders in it from the start) and a free name for the second
one (nomecollisione(): .bashrc_00000001; the dots and spaces at the end are left
out, they would go away). changeKey(), not changedtmapkey(): its "deleted" key
stays in dt, and extractw() counts every key of chunkfiles. Two names with the
same hash (one in 2^64) cost a file renamed for nothing. It runs only if
searchunixfile(): from Windows no two such names can come.

The devices: nomesanato() is franzsanitizepath() plus every piece of the path
looked at by nomeforsedispositivo() (a few comparisons: the length first), and
only the few that can be go to nomedispositivo(): GetFullPathNameW() on the path
that will be opened gives \\.\NUL for a device. No table of which Windows does what.
The pieces in the middle too: makepath() asks for a folder with no / at the end,
and o\nul is the device.

close() gave back false on 00163, and nobody looked: two counters
(g_closefalliti, under the mutex of the writer) and GetFileType() on what is open
(a call for each file): what is not FILE_TYPE_DISK is 03808!, rc 2.

9. Known limits and open points

• -longpath: a real nul, finale., spazio : there, but few programs open them.
• repack: no multipart or -chunk output, no replacement of the source
  in place.
• repack -quick copies a damaged block as it is (see above).
• All the records of the source stay in memory, as l -all: on archives with
  many millions of records, watch the RAM.
• repack on a build without a SIGINT handler (ANCIENT but not NAS, Solaris):
  control-C leaves the .tmp. The next run refuses it (03766!), -force
  deletes it.
• -tar: what stays different from the tree is a socket, and a hard link whose
  first name is over 215 characters.

The 65 branch of zpaqfranz: what changed, and why the source doubled

In short:

• 56% of the growth is two bundled libraries, not code written here: zstd
  (1.9 MB) and mbedTLS (1.1 MB). zpaqfranz is a single .cpp file, so every
  library ends up in the count.
• The help in three languages weighs almost half a MB.
• The rest is about 2 MB of new functions, none over 210 KB. There is no
  monster: there are many things of medium size.
• The format did not change. The archives are still zpaq archives: zpaq 7.15
  reads them. This constraint is the thread that ties together the most
  interesting ideas of the branch (see part three).

Part one: the growth, version by version

The bytes are the ones of the final sources (65_1.cpp ... 65_15.cpp, and
zpaqfranz.cpp for 65.16h). A KB is 1,024 bytes; the sizes of the blocks are
measured on the differences between two consecutive versions.

Part two: what the growth is made of

1. The overall count

The zones of the libraries are the ones marked in the source (LICENSE_START /
LICENSE_END).

2. Why the libraries weigh so much

• zstd: 47,600 lines. The official amalgamation (combine.py) of compression
  and decompression, without multithreading and without the dictBuilder, compiled
  in namespace zzstd. Not a line of zstd was changed; around it, 759 macros are
  saved and restored (push_macro / pop_macro) so that they do not leak into the
  rest of the source. It serves -m7, -m8 (the default) and -m9.
• mbedTLS: almost 35,000 lines with postami. It is there for one thing only:
  sending a report by e-mail over TLS, with no external program, on every
  platform. It is a subset of the library, reduced to a TLS 1.2 client and brought
  to C++98.
• They do not weigh on every build. ESXi and the -DANCIENT builds (not the
  NAS ones) do not compile zstd; -DNOEMAIL leaves out the whole mail client. The
  source is big for everybody, the executable is not.

3. The help: 465 KB

In 65.14 the help stops being code and becomes a table of data (g_helprighe,
6,100 lines). The table weighs 555 KB: 364 KB are translations (Russian 212,
Italian 152), 175 KB the English. Russian costs more because in UTF-8 every letter
takes two bytes.

4. Our own code, by function

Every byte is assigned to the definition that holds it, and the definitions to
the functions by their name. The measure is taken on the whole branch, so it
also counts what grew after it first came in. It is an approximation: the small
numbers are good to a few KB.

Grouped: disks and images 522 KB, compression and archive 359 KB, new commands
299 KB, interfaces 185 KB, tests 153 KB.

The comments count. The comments of our own code go from 417 to 968 KB: more
than half a MB, about a fifth of the growth of our own code (and already included
in the numbers above). The reason for the choices is written in the source, next
to the code, and that is a choice: it is paid in bytes.

Part three: excursus, the truly innovative functions

A zpaq archive is a journal: every a adds a transaction, nothing is written
again. Matt Mahoney's format has a rare property: every block carries the
program that decodes it, in ZPAQL. The constraint of the whole branch was not
to change the format: any zpaq, 7.15 too, must go on reading the archives. The
best ideas came precisely from there: how to get more without touching
anything of what the others read.

(Franzen is a separate matter: it is an encryption wrapper around the archive,
and does not concern this discussion.)

1. -turbo: making parallel what looks serial

The problem. zpaq deduplicates by fragments, cut according to the content: a
rolling hash runs over the bytes and decides where to cut. The state passes from
one byte to the next, so it looks like serial work by nature. And the algorithm
cannot be changed: different cuts mean different fragments, and goodbye
deduplication against every existing archive. Anything faster must produce the
same cuts.

The idea. The state starts from zero at every cut. So:

1. the file is read in batches of 64 MB, and each batch is split into one piece per
   thread;
2. each thread cuts its piece as if a fragment began right there: it is a
   guess, probably a wrong one;
3. then the true chain is followed. As soon as a true cut falls exactly on a cut
   of the chain of the next piece, from that byte on the two chains start again
   equal on the same bytes: they are the same chain, and the guess is proven.

piece 0   |--a--|---b---|--c--|-d-|....
piece 1             |-x-|---y--|--c--|-d-|--e--|...   (starts from a guess)
                                ^ c starts at the same byte in both:
                                  from here on, piece 1 IS the true chain

The chains of a rolling hash synchronize by themselves, usually within one or two
fragments. If they do not meet, the work goes on with one thread until they do:
nothing is taken as good without proof. On 12.8 GB, in 3,362 pieces, the serial
part was 12 MB.

Around it: one thread reads the next batch while the current one is cut, the SHA-1
is computed while the data are still in the cache, and the cut has no conditional
jumps (the first version, with an if, was slower than the original: on
compressed data that jump is a coin toss).

The result. An archive identical to the byte, not just of the same size:
the deduplication with the old archives does not notice anything. A matrix of 25
combinations (every hash, every method, -t1, -verify...) gives identical
archives, and the autotest checks it again every time.

The jump from 34 to 25 seconds, without -turbo, comes from two other findings
made with the profiler: a table allocated with calloc wasted 25% of the CPU in
the kernel (copy-on-write of pages never touched), and a prefetch 32 bytes ahead
removes a good part of the cache misses. There too, the same archive to the byte.
Since 65.6 -turbo is the default.

2. -m8: zstd inside zpaq, readable by anyone

The problem. Adding a compressor to a format usually means breaking
compatibility: whoever has the old program cannot read the archives any more.

The idea. A zpaq block can have no model at all (the bytes are just copied)
and a post-processor, a ZPAQL program that every zpaq decoder must run on the
data. So:

• the bytes stored in the block are a zstd frame, made by the real library;
• the post-processor is a complete zstd decoder written in ZPAQL: 910 lines
  generated by a script, stored inside every block;
• zpaq 7.15 runs that program and extracts, without knowing what zstd is;
• zpaqfranz recognizes that program, comparing its bytecode, and calls native
  zstd. The output is the same: it is the ZPAQL program that defines what is right.

The format did not change by one bit. The idea had first been tried on a small
scale with LZ4 and LZAV (65.3); zstd beat both, and they were taken out.

Faster and smaller. With the high levels it gets even more interesting,
because zstd compresses slowly but always extracts fast:

The price. The decoder is for ever: the archives hold it, a defect in it would
stay there. That is why its SHA-1 is frozen and checked by the autotest, and why it
was tried on about 30,000 frames of every kind and 20,000 damaged streams. A
different decoder, one day, will be a new program next to this one, never a change
of it. And in ZPAQL it is slow (80-95 MB/s with the JIT): it works, but its purpose
is to guarantee that the data can always be read.

3. -fast and -fastadd: the index inside the archive, with no new format

The problem. Version N of an archive holds only what changed. To know what
is in the last version, all the indexes must be read again, from the first:
thousands of seeks and small decompressions.

Where to put the answer. A file next to the archive is lost as soon as it is
copied. A new kind of block breaks zpaq 7.15. A normal file in the archive needs
the index to be found: exactly what is to be avoided.

The idea. The history lives in normal data blocks, which every zpaq reads and
no file uses. The pointer to those blocks lives in the last block of the
archive, disguised as the deletion of a file that never existed, with the
information written in its name. zpaq 7.15 sees some unused fragments and a
harmless deletion; zpaqfranz looks at the end of the file and knows everything.

And the history is deduplicated with the same algorithm as the data: every a
pays only for its own tail, not for the whole history.

-fastadd applies the same trick to the update: it stores in the archive the state
a would get by reading everything again, and next time it starts from there. On
that same archive the initial read goes from 11 seconds to less than one.

4. mount: the archive as a disk

zpaqfranz mount backup.zpaq

The archive appears as a read-only disk, on Windows (WinFsp), Linux, FreeBSD,
OpenBSD and macOS (FUSE): every file can be opened, copied, searched, with nothing
extracted. By default you see the last version; if you want, all of them, each in
its own folder, or only what changed between two versions (-range).

The hard part is making something built of fragments scattered inside compressed
blocks look like a disk. An open file becomes the list of its fragments, with a
binary search; blocks are decompressed on demand, outside the lock; a single cache
knows when a block is "loading", so two reads do not decompress it twice; read-ahead
starts only for a sequential reader, with a tolerance of 8 MB because kernels ask
for data out of order from several threads.

5. ac and xc: the disk of Windows in a deduplicated archive

An archiver that saves a Windows while it is running, and puts it back on a
disk it boots from. ac looks at the disk as a list of zones: C: by its used
clusters with a shadow copy, everything needed to boot byte by byte (hidden
partitions, the table, the spaces between partitions), and a text map with the
hashes of every piece. xc reads the map again, checks every piece and writes to
a physical disk or to a .vmdk / .vhdx.

Why it starts with no repair: on MBR the boot configuration finds its partition by
the signature of the disk (4 bytes of sector 0) and an offset. The same sector 0,
the same offsets: it starts. On GPT, the same GUIDs.

The interesting part is that it is all inside a normal zpaq archive: every backup
is a version, deduplicated against the ones before, and every command that knows
images knows these too.

It is the youngest part of the branch. A real boot was tried on a restored
Windows 10, MBR and BIOS, in a virtual machine; a restored GPT and UEFI disk was
compared byte by byte with the original, not started. That is why since 65.9 the
notes carry the words DO NOT USE IN PRODUCTION.

6. -ntfs: the list of the files from the $MFT

Reading one folder at a time is slow; reading the file table of NTFS directly is
much faster, but Windows gives answers that are not on the disk (filtered
attributes, files compressed by WOF that look "normal", links).

The idea is the same list by construction: where only Windows knows what it
would answer, Windows is asked. A folder that cannot be read, a long path, a cloud
folder go back to the normal scan for that subtree. So the speed is gained without
having to reproduce all the rules: 7 times on a C:, 12 with a shadow copy.

7. Testing the tests: work faultgen and work probe

Usually one proves that a program works on good data. In 65.15 the tests
themselves were tested: faultgen prepares copies of archives damaged on purpose
(every byte, one at a time, in six ways), probe gives them to t, tw, pp, p
and x, and compares their verdict with what really comes out of the extraction,
trusting nothing of zpaqfranz.

The rule: OK only if it is really OK; what can be recovered is a WARNING;
everything else is an ERROR.

40,198 more copies of real archives, damaged where the structure of the blocks says
it matters, went through FreeBSD, also under AddressSanitizer. All of this without
making t and x slower by more than 1%.

8. repack: taking versions out of a journal

A journal cannot be pruned: to take a version out you had to extract and compress
again by hand. repack (or purge) writes a new archive with only the chosen
versions.

repack z:\1.zpaq 7d4w12my -to z:\2.zpaq     (7 days, 4 weeks, 12 months, every year)
repack z:\1.zpaq 7d4w12my -test             (what it would keep and why: writes nothing)
repack z:\1.zpaq 7d4w12my -quick -to z:\2.zpaq
repack z:\1.zpaq -all -m1 -to z:\2.zpaq     (every version, another method)
repack z:\1.zpaq -all -key old -key2 . -franzen2 new -to z:\2.zpaq

The things that are not obvious:

• every fragment goes into the first kept version that uses it, otherwise a
  file deleted and then back would break;
• -quick copies whole the blocks with all their fragments kept, without
  decompressing them: it renumbers them in place and checks their bytes. So pruning
  hundreds of GB is a matter of copy time;
• the output is a .tmp, read again from zero (dates, comments, every file of every
  version, every block), and it gets its name only if everything adds up.

With the same operation you change the method of a whole archive, or its encryption
in any direction.

9. Other ideas worth remembering

• a -append -stdout: the new version goes out on a pipe as a delta, never
  going back. It can be attached with cat to the end of the archive, even an
  encrypted one. To know beforehand the sizes to write in the header, add() runs
  twice: the first time it writes nothing.
• -rescue: on a dying disk you do not insist. At the first failed read it jumps
  ahead, doubling the jump, then comes back towards the trouble: a dead zone of 20 MB
  costs about fifteen failed reads, not two thousand.
• f -test, the triplet: the data of every block come from its number and from
  a key of the run, so they never repeat. A block read back with the data of another
  block unmasks a "32 GB" stick that is really 8.
• Help with a fingerprint: every translation carries the hash of the English text
  it came from. If the English changes, the old translation is no longer shown and
  the English comes back: a translation can never stay behind without it being known.
• t with a folder: it compares the files with the archive through the SHA-1 of
  the fragments, decompressing nothing. With -ssd in parallel: 116,725 files in
  2 seconds instead of 12.
• zip: the archive in one ZIP64 file, with a DEFLATE encoder written from
  scratch. The Linux names Windows cannot create stay legal inside the zip.