zpaqfranz 65.11

DO NOT USE IN PRODUCTION: IT IS STILL TO BE STABILIZED

What was young in 65.9 and 65.10 (ac, xc, mc) is as young as it was.
What is new here
touches the encrypted archives, the backups in parts, and a command that
wipes USB sticks: it was tried by scripts on every platform that was at
hand, not yet by many hands. If you use -franzen, read part one,
points 3 and 4, before anything else.

This is the release of the archive that holds on. What happens when
something goes wrong, to an archive or while it is written:

• x -recover gets out more from a damaged archive, and says what is
  damaged: file by file, where, and how many bytes.
• Franzen on archives in parts: backup and testbackup with
  -franzen (and with -key -franzen), multipart archives too.
• Franzen, four defects fixed. One of them lost the end of an archive,
  once in 65,536 runs, with an "all OK".
• A Franzen archive whose last update did not finish (the process
  killed, the power gone, the disk full) is read up to the update before,
  and the next a puts it right. It could not be opened at all.
• backup -chunk: no part bigger than a given size.
• -freeze made again: an archive that grows over a size is moved
  away (to a NAS), with all its files, and a new one starts.
  -findzpaq for backups and for reading.
• f -nuke: an NTFS USB stick with the work of a customer on it,
  made empty in a short time so that an undelete tool finds nothing.
  f -gui: a guide to the many cases of f.
• grep: a small one, inside zpaqfranz.
• -m7 and -m9: two more zstd methods, by name.
• pakka (the list for the GUIs) reads Franzen archives.

This document has three parts:

• Part one: what is new, in short, for everybody
• Part two: the same things in more detail, for power users
• Part three: "lo spiegone", how it works inside, and why, for developers

Part one, what is new, for the user

1. A damaged archive: x -recover
2. Franzen: archives in parts, backup, testbackup
3. Franzen: four defects fixed
4. Franzen: an update that did not finish
5. backup -chunk
6. -freeze and -findzpaq
7. f: -nuke, -gui, -n
8. grep
9. -m7 and -m9
10. pakka and the Franzen archives
11. Smaller things

1. A damaged archive: x -recover

zpaqfranz x z:\1.zpaq -to u:\restore -recover
zpaqfranz xx z:\1.zpaq -to u:\restore

(xx is x -recover.) An archive with a damaged block: -recover was
already the way to get out what is good. It lost a whole block for one
wrong byte, and it could write a file with a piece missing in the
middle, the rest moved up, without a word.

Now:

• every fragment that comes out is checked (its SHA-1);
• the good fragments of a damaged block are written;
• a fragment that did not come out at all becomes zeros, of its length:
  what comes after stays in its place;
• at the end, the list of the files that are not as they were:

66402: 00000001 DAMAGED 4.096 bytes wrong SHA-1 + 0 zero-filled <<u:/restore/doc/big.bin>>
66405! DAMAGED files: 1 (bad blocks in the archive: 1)

and the exit code is 2. With -verbose: where, in each file.

The same for the images (image, a volume put back on a file or on a
partition): a damaged block gave an image shifted from there to its end,
with "COMPLETED SUCCESSFULLY". Now the damage stays where it is, and the
run ends with an error.

Measured, on an archive with 4,096 bytes of zeros put inside a block:
30.8 MB of files not good (four of them shifted) before, 9.6 MB after,
as holes in their place. An image of 1.2 GB: 348 MB different before,
12 MB after.

2. Franzen: archives in parts, backup, testbackup

-franzen was for an archive in one file. Now:

zpaqfranz backup z:\prova.zpaq c:\nz -franzen pluto
zpaqfranz backup z:\prova.zpaq c:\nz -key pippo -franzen pluto
zpaqfranz testbackup z:\prova.zpaq -verify
zpaqfranz testbackup z:\prova.zpaq -paranoid -franzen pluto
zpaqfranz a "z:\multi_???.zpaq" c:\nz -franzen pluto

Every part is a Franzen file of its own (prova_00000003.zpaq.franzen),
the index of the backup too (prova_00000000_backup.index.franzen).
Nothing in clear is written. The quick testbackup, and -verify, do
not want the password: they check the files as they are on the disk.
-paranoid reads inside, and wants it.

One password, one wait: the key (about a second, and 256 MB) is made
once for the whole set, not for every part and every thread. A t of a
Franzen archive: 2.3 seconds before, 0.4 after.

A block that cannot be decrypted (a damaged part) was taken for the end
of the archive: everything after it disappeared in silence, and t said
"all OK". Now it is an error (43810, exit code 2).

3. Franzen: four defects fixed

Found by a review of the Franzen code (65.11r), and all four in the way
zpaqfranz writes the file, not in the encryption. Each of them ended
with "all OK".

The last 104 bytes of an archive not written. About one run in
65,536 (when the last piece of the archive to be written was exactly 104
bytes long, the size of the header of a version). a said "all OK"; the
archive could not be read any more (t: "cannot read_archive"). With
-franzen alone and with -key -franzen. A t after the a tells if
an archive was hit: test your Franzen archives.
Too quick, and too dirty

A password longer than 255 bytes was not used. zpaqfranz said
"42100! Error: password not good or too long", and went on with an
EMPTY password, to an "all OK". Such an archive opens with any other
long password. Now a new archive is refused:

42101! The -franzen password is longer than 255 bytes: not supported. Nothing is written

and an archive already made that way can still be read, as before, with
a warning that says what it is: make it again, with a password of 255
bytes at most.

A disk that gets full. With -franzen the writes that failed were
not counted: "all OK", exit code 0, and an archive cut short. Now:

43814! 889 write(s) on a franzen file failed (media full?): that file is NOT good

and exit code 2.

x -repack with -franzen made a Franzen file that could not be
read. Now it is good.

4. Franzen: an update that did not finish

The process killed, the computer off, the disk full while a adds to a
Franzen archive. The archive could not be opened any more ("Footer is
corrupted"): not the new version, and not the old ones either, all
still there in the first bytes of the file. (A plain zpaq archive in the
same state ignores its last, incomplete, version.)

Now t, l, x read it up to its last complete update:

41750$ <<z:/copy.zpaq.franzen>>: no valid footer at its end (an update that did not finish?)
41751$ Looking for the last complete update...
41752$ Found: it ends at 3.005.691. The 90.871.704 bytes after it are ignored (the next add removes them)

with exit code 1 (a warning), and the file is not touched. The next a
removes what was left and goes on from there: the archive is good again.

A control-C was already taken care of (the file goes back as it was).

A file with nothing complete in it (a first run that never finished) is
not opened, and not touched: delete it, and make it again.

5. backup -chunk

zpaqfranz backup z:\prova.zpaq c:\nz -chunk 1GB
zpaqfranz backup z:\prova.zpaq c:\nz -chunk 1GB -key pippo

No part bigger than 1 GB: a run makes as many parts as it takes (for a
cloud that does not want big files, for a disk with a limit). backup
refused -chunk.

The parts of a run get their names all together, at its end: a run that
dies leaves .tmp files, and the backup is the one of before. In clear
and with -key. Not with -franzen, not with -backupzeta.

6. -freeze and -findzpaq

zpaqfranz a z:\h.zpaq c:\nz\ -freeze y:\archived -freezesize 10GB
zpaqfranz backup z:\prova.zpaq c:\nz -freeze \\nas\old -freezesize 100GB

Before the run: if the archive is over -freezesize, it is moved into
a new folder of -freeze, named by date and time
(y:\archived\2026-10-12_18-15-07\), and a new archive starts. All of
it: every part of a multipart, every chunk, the index and the list of a
backup, the files with the checksums.

First everything is copied, and every copy is verified; only then the
originals are deleted. A copy that fails: nothing is touched, the run
goes on with the archive where it was, and ends with a warning.

-freeze was there before, with -maxsize for the size: that line
still works.

-findzpaq (Windows): the archive is on a USB disk that changes its
letter. The name is searched on every drive. It was for a only: now
for backup, testbackup, l, x, t, i, v too, for multipart
archives and for Franzen ones.

zpaqfranz backup k:\copie\prova.zpaq c:\nz -findzpaq
zpaqfranz l k:\copie\prova.zpaq -findzpaq

Not found: an error, never a new archive somewhere else. Found on more
than one drive: an error, with the list.

7. f: -nuke, -gui, -n

-nuke: a USB stick, NTFS, that carried the files of a customer and
must be used again. A format leaves everything there for an undelete
tool; writing over the whole stick takes hours.

zpaqfranz f E: -nuke 30 -force
zpaqfranz f E: -nuke 1 -force
zpaqfranz f E: -nuke 100 -force -verify

1. what NTFS itself points at is written over: the table of the files
   (names, small files), every file it knows, the deleted ones too, the
   folders;
2. the volume is formatted: an empty NTFS, same label;
3. 30% (the number given) of the free space is written over.

Minutes, not hours. With -nuke 100 everything is written over, the
old data first. A control-C in step 3: the stick is good and empty.

On a test stick (2 GB, 592 office files and 295 deleted before), the
files that PhotoRec finds and that open: 736 after a quick format, none
after -nuke 1, -nuke 30, -nuke 100. What can be left below 100:
pieces of the body of files over 16 MB.

Below 100 it is NOT a secure erase. Only a removable USB NTFS
volume, never the system one. Do not format the stick before: its table
of files is the map of what to wipe. Windows only.

-gui (Windows): f has many cases by now. A guide asks one thing
at a time: what (read a disk, write in the free space, wipe), how, where
(the list of the disks and of their volumes). Then the command is shown,
and run. Its captcha is always asked.

zpaqfranz f -gui

-n NN: read or write only NN% of the blocks, equispaced. A quick
look at a big disk.

zpaqfranz f 3 -test -n 30

8. grep

zpaqfranz grep c:\*.cpp d:\ something
zpaqfranz grep c:\src something -verbose
zpaqfranz grep c:\dropbox password -verbose -n 60

The last parameter is the text, the others are folders, files, masks.
Case-insensitive (ASCII), line by line, no regular expressions. Without
-verbose: the first match of each file. Files with a binary extension
are skipped (-all: not skipped).

For a Windows without a grep, and for names and paths that the other
tools do not open.
And a lot of binary files (to be automagically excluded).

9. -m7 and -m9

zstd was -m8, with its level by hand (-m8h19). Two more by name:

As every -m8: any zpaq (7.15 too) extracts them. h main has a line
for every method, from -m1 to -m9.

10. pakka and the Franzen archives

pakka makes the list that the GUIs read (catpaq, and others). On a
Franzen archive it said "archive contains no data".

zpaqfranz pakka h:\zarc\1.zpaq.franzen -all -catpaqmode -franzen pippo

Never a question on the console: a password that is missing or wrong is
one line (zpaqlist:...) and exit code 1. And the file of -out holds
the list, all of it and nothing else (the times of the progress lines
ended on top of its first records).

11. Smaller things

• The faster add everywhere: backup, a -append, -home, cloud
  add as a does (files over 4 MB cut and hashed by several threads).
  The archives are the same, to the byte. -noturbo: as before.
• a -append -chunk on a new archive ended with "Archive not
  found".
• backup: a run that died (a wrong password) left its .pid, and
  every backup after it said "Quit because backup seems running".
• backup, a list of hashes born MD5 and then a run with
  -backupxxh3 (or the other way round): testbackup -verify failed
  on that part for ever. The hash of the list is used.
• a -chunk: the sizes printed at the end did not count the first
  chunk.
• Franzen on macOS: the library was looked for as libsodium.so
  only. On a Mac it is libsodium.dylib (brew install libsodium). Yep. Nothing is easy on Mac.
• franzen autotest: five more tests, on what is in points 3 and 4.
• grep: folders, names that are not ASCII, a text with a \,
  very long lines, UTF-16 files.

Part two, the details, for power users

1. x -recover: what is written, what is said
2. The Franzen files of an archive in parts
3. The Franzen fixes: who, what, how to check
4. An update that did not finish: the rules
5. backup -chunk: the rules
6. -freeze and -findzpaq: the rules
7. f -nuke, f -gui
8. grep: the rules
9. The methods, measured
10. What can change for a script

1. x -recover: what is written, what is said

A fragment with a wrong SHA-1 is written as it is, not zeroed: in the
blocks without a model (-m0, -m1, zstd) one wrong byte in the
archive is often one wrong byte in the file, and 60 KB of zeros in its
place would throw away the rest.

• 66401: one line for each bad block (where in the archive, how many
  fragments).
• 66402: one line for each damaged file; 66404 (-verbose): each
  stretch, its offset in the file, its length.
• 66405, 66408, 66409, 66410: the totals.
• 66407: the archive has bad blocks, but none of the files extracted
  uses them: exit code 1.
• Damaged files: exit code 2.

An archive of zpaq 7.15 (no hash of the files inside) with one byte
turned over: no message at all before, exit code 0. Now the fragment is
found, and the file is in the list.

Images: 66411 "The image is NOT the one that was saved", exit code 2.
xc is as it was: it already stopped, with "RESTORE NOT DONE".

A good archive: the output is the same as before, byte by byte, and so
are the times.

2. The Franzen files of an archive in parts

All the files of a set are made with the salt of the first one: one
key. The nonce of every block is random.

With -key too: AES inside (as zpaq does), Franzen outside. Two
passwords, two keys that have nothing in common.

Refused:

a x_???.zpaq -franzen (multipart, not a backup) works: it was refused.

consolidatebackup and rebuildbackupindex do not know the Franzen
parts.

3. The Franzen fixes: who, what, how to check

The 104 bytes. Who: every archive written with -franzen (with or
without -key) by a zpaqfranz up to 65.11q. When: the bytes of a run
still to be written, when zpaqfranz goes back to write the header of the
version, are exactly 104. That is: the bytes written by that run, modulo
65,536, are 104. One run in 65,536, a little more or a little less.
What: those 104 bytes (the end of the index of that version) are not in
the file. How to check: t of the archive. A version hit by it is not
read.

The long password. Who: who gave a -franzen password over 255
bytes (a key of 128 bytes written in hexadecimal is 256). The archive is
encrypted with an empty password. With 65.11s:

• a new archive, a new part of a multipart: 42101, exit code 2, no file;
• an archive that is there: it opens with any password over 255 bytes,
  as before, and says 42102-42104. Extract it, and make it again.

The disk full. 31681 at the first write that fails, 43814 at the
end with their number, exit code 2. Also: the footer that cannot be
written (38415), the file that cannot be closed (38416). The archive on
the disk is then an update that did not finish: see the next point.

x -repack ... -franzen writes the repacked archive and, next to
it, its Franzen file. The second one kept the header of an incomplete
version.

4. An update that did not finish: the rules

• Who reads (t, l, x, i, pakka, the franzen command):
  the file is read up to its last complete update. 41750-41752 once for
  each file, 43815 at the end, exit code 1 if there is nothing worse.
  The file is not changed.
• Who writes (a): 41754, the bytes after the last complete update
  are removed, the new version goes there. Exit code 0.
• Nothing complete in the file: 41748 and 41755, as before. The
  file is not touched.
• The time: the bytes left by the unfinished update are read once,
  backward. A file cut by a copy that stopped is the same case: it opens
  at the last update that is all there.
• Not for the two files of -debug (x.zpaq and x.zpaq.franzen
  side by side): the check "Franzen is BEHIND .zpaq" stops it, as before.
• A footer damaged, with all its data good: the archive opens one
  update back, and the next a removes the data of the last one (and
  adds them again, if the files are still there). Before, it did not
  open at all.

5. backup -chunk: the rules

• A run makes N parts, numbered one after the other in the same
  sequence as the runs without -chunk; the list (_backup.txt) has a
  line for each. Runs with and without -chunk can follow one another.
• The parts are written as .tmp and named at the end of the run, from
  the last to the first. A run that dies while it writes: only .tmp
  files are left, the next run puts them aside (.spaz) and goes on.
• A run that dies while it names: a hole in the numbers. The next
  backup refuses to go on (43812, 43813: rename the .tmp of the hole,
  do not fill it with empty files).
• The index of the backup is one file, never cut.
• A chunk is closed after the buffer that goes over the size: it can be
  a little bigger (up to 128 KB).
• -copy copies all the chunks of the run.
• Refused: -franzen (43806), -backupzeta (43811).

6. -freeze and -findzpaq: the rules

-freeze X -freezesize N (a, backup):

• What is counted and moved: the archive (and its .franzen); all
  the parts of a multipart that match the name, after a hole too; for
  each of them _crc32.txt, _md5.txt, .md5; the index of -index;
  index and list of a backup; the file of -checktxt. Not the .pid,
  not the alternate data stream.
• Where: X\2026-10-12_18-15-07\; if it is there, ..._00001. The
  names of the files do not change. The free space is checked first.
• How: copy with verify (XXH3), then the originals renamed
  .frozen from the last to the first (one that cannot be renamed is in
  use: every name goes back, the copies are removed), then deleted.
• Exit code: 1 if it could not be done (68875: the archive is where
  it was), 1 if something is left to delete by hand (68876).
• -freeze without a size: 68861. -freeze with -maxsize and no
  -freezesize (the line of before): the size is that one, and it is
  not a filter on the files any more (68862).
• A backup whose first part alone is over the size is moved at every
  run.

-findzpaq (Windows; a, backup, testbackup, l, x, t, i,
v):

• searched: the archive or its .franzen, the first part of a
  multipart, name_00000001.zpaq (or .franzen) for a backup;
• on the letters of the fixed and removable drives, not CD and network;
  the letter given is always tried;
• none: 01882; more than one: 01883 and where; a * in the name:
  ignored (68871).

7. f -nuke, f -gui

f X: -nuke NN -force (NN from 1 to 100), in order:

1. Three passes over what NTFS points at, the most useful first (a
   control-C can come): the $MFT and its mirror; the first block of
   every file and the indexes of the folders; then the first 16 MB and
   the last MB of every file, the other system files, the first and the
   last 64 MB of the volume. Every record: in use or not (a deleted
   file still has its record), and the records of an older $MFT left
   by a format, as long as they are found. With -nuke 100 a fourth
   pass: the rest of the big files, and all that the volume says is in
   use.
2. A quick format (same label).
3. NN% of the free space; 100: all of it.

Not zeros: every sector says where it is and which pass wrote it (some
sticks do not write zeros at all). -verify reads back and checks place
and pass of every sector: a stick that drops or misplaces a write is
seen.

A disk that is not USB: only with -debug. The system disk: never. The
captcha is nuke.

f -gui: three screens (what, how, where), ENTER goes on, ESC goes
back, r reads the disks again (a stick plugged in after). Green: nothing
is lost; yellow: everything is lost; red: it cannot be done, and why.
Without the rights of an administrator only the cases by file are left.
Then 68850: f -gui: zpaqfranz f E: -nuke 30 -force and the command
runs, with its warnings and its captcha.

8. grep: the rules

• The first match of each file; -verbose: every matching line, the
  spaces in front removed; of a line over 200 bytes, the part around the
  match.
• Skipped: the binary extensions (zip, jpg, exe...; -all: not
  skipped), and always the files with NUL bytes in their first 64 KB,
  but the UTF-16 text with a BOM, which is read.
• -n X: the width of the column of the path (40).
• -debug: the files not read, and the ones skipped.
• A text with spaces: in quotes.

9. The methods, measured

One run each, 32 threads, three sets: mixed files (1.7 GB, 30,000
files), a tree of sources and tools (6.6 GB, 116,000 files), virtual
disks (11.8 GB). The archive in MB / the seconds to add / the seconds to
test.

-m9 is smaller than -m2 and as fast as -m1. -m7 is -m3 (from
4.9% smaller to 0.9% bigger) with an extraction of seconds instead of
minutes. -m4 and -m5 are still the smallest, and the slowest to
extract.

-m78 is -m88h19, -m96 is -m86h9, -m9h12 is -m8h12: block and
level can still be given. In the archive, and in i -verbose, the
method reads as 86h19 and 84h9. A build without zstd: -m7 and
-m9 are -m5, with a warning, as -m8.

10. What can change for a script

• Exit code 2 where it was 0: a ... -franzen on a disk that gets
  full; x -recover (and image) with damaged files; -franzen with a
  password over 255 bytes on a new archive; a Franzen archive with a
  block that is not decrypted.
• Exit code 1 where it was 2: t, l, x of a Franzen archive
  whose last update did not finish (it is read now).
• Exit code 1: -freeze that could not move the archive; x -recover on an archive with bad blocks that the files extracted do
  not use.
• backup -chunk is accepted. backup with -franzen is
  accepted. a x_???.zpaq -franzen is accepted.
• -freeze puts the archive in a folder named by date and time
  (it was name_00001.zpaq in the folder given). -freeze with
  -maxsize: the size is not a filter on the files any more.
• -findzpaq: not found is an error for every command (01882).
• -m7, -m9 are zstd (they were taken as -m5).
• pakka on a Franzen archive: the list, or zpaqlist:... and
  exit code 1.
• New messages: 31681, 38416, 41750 - 41755, 42101 - 42104, 43806 -
  43815, 66401 - 66411, 68840 - 68876.

Part three, lo spiegone, for developers

1. The recover: a state for every fragment
2. Franzen inside the archive: who owns what
3. The Franzen file, and what was looked at
4. The 104 bytes
5. writeat and its nonce
6. The last valid footer
7. add() is add2()
8. backup -chunk: an index that is not a chunk
9. congela and findzpaq
10. franznuke
11. How it was tested
12. Found on the way
13. Known limits and open points

1. The recover: a state for every fragment

x -recover is the engine of the extraction to stdout
(extractstdout, worker_routine): the archive is read block by block,
the files are written in order from a cache.

ThreadContext::frag_stato: one byte for every fragment (not looked at,
good, not good, no SHA-1 to check). The worker keeps what a block gave
before it failed (it was thrown away), and checks the SHA-1 of every
fragment: the unit of the loss is the fragment, not the block.

Two things were worse than in the plain x: no SHA-1 at all (a block
that decodes "well" with wrong bytes went into the files), and a
fragment that was missing was skipped, not written as zeros. Both gone.

Also: a block that threw an exception left the lock of the output
taken, and every other worker waited for ever.

recover_elenco and recover_totali make the list and the totals, for
the files and for the special modes (images). xc is left out
(xcmuto): it has its own check of every piece.

2. Franzen inside the archive: who owns what

• InputArchive owns its franzcri (franzen_nomi,
  franzen_parti, franzen_parte(i)): one for each part, opened at the
  first read that falls there. Before: one for each thread, tied to the
  first file it had opened, and used for any archive after that.
• The size of a part is read from its footer without the key
  (leggi_dimensione) when the parts are counted, and checked for real
  when the part is opened.
• The keys: g_franzen_chiavi, by password, salt and limits. The
  lock is kept while Argon2id runs: the other threads wait for that key,
  they do not make their own.
• OutputArchive: eindice (this is the index), scrivi() in the
  place of myfwrite (it says whose bytes they are:
  g_franzen_scriveindice), g_p_franzenindex for the Franzen file of
  the index. Before, the bytes of the index went into the Franzen file
  of the archive too, and closing the index closed that one.
• nowritebase: with no .zpaq behind, off is the position
  inside the part; AES encrypts by the offset in the whole archive.
• pakka_Archive: the reader of pakka comes from zpaqlist and
  reads a byte at a time; for a Franzen archive it goes through
  InputArchive, with a buffer in front.

3. The Franzen file, and what was looked at

[header 132 + nonce 12][blocks...][footer 84]            a new file
[header][blocks 1][footer 1][blocks 2][footer 2]...      after every update

• The key: Argon2id (3 passes, 256 MB), a random salt of 16 bytes.
• A block: [nonce 12, random][up to 64 KB, encrypted][tag 16],
  ChaCha20-Poly1305 (IETF) of libsodium. One tag for each block.
• The header is written once. A footer says how many bytes of data the
  file holds, where its segment (the blocks of that update) starts, and
  how many bytes it holds; it has its nonce and its tag. A new update
  writes after the old footer, which stays there: the file only grows
  (rsync --append).

The review (October 2026) read all of it. The choices of the encryption
are sound. What it found is in the way the file is written (points 4,
5, 6 here), and two things that are as they were:

• The header has one nonce for two tags (the check of the key, and
  the tag of its fields). Nothing follows from it in practice: the
  fields it covers do not say where the data are. To change it is a new
  format.
• A block is not tied to its place: no associated data. A byte
  changed anywhere is seen; a block moved to the place of another one
  (of the same file, or of a file with the same key) is not seen by
  Franzen itself: the checks of zpaq inside (the SHA-1 of every
  fragment) see it. To change it is a new format.

Everything changed by the review is marked in the source:
grep FIX-FRANZEN (36 places).

4. The 104 bytes

zpaq writes the header of a version (104 bytes) at the start, with no
size in it; at the end it goes back and writes it again, with the size
of the compressed data. With Franzen "going back" is writeat(): read
the block, decrypt, change, encrypt, write.

myfwrite knew that write by its size: 104 bytes. With
g_franzen_jidacheader not set yet, a write of 104 bytes was skipped.
So when what was left in the buffer of OutputArchive, flushed by the
seek back, was 104 bytes long, it was skipped: the archive lost them.

Now add() says it: g_franzen_jidacheader is set just before the
header is written, and cleared just after. Everything else is a
write().

One thing changed with it: a version with nothing in it. Its first
header was skipped, and the second one added at the end. Now the first
one is written and the second one goes over it. The bytes are the same.

repack writes its C block again in the same way, and did not say it.

5. writeat and its nonce

The block that writeat() put back was encrypted with the nonce it
already had. That is two contents under one key and one nonce: the
block as it was while the run went on, and as it is at the end. Who has
both (a snapshot, a sync, a copy taken during the run) sees the XOR of
the two headers, and the tag of that one block does not stand for it
any more. Not the key, not the data of the files.

Now: a new random nonce, as for every other block. The nonce is in
front of its block: the format is the same, every version reads the
files of the others (tried both ways with 65.11q).

The note in the source says how to have it as before: one line
(randombytes_buf) to remove.

6. The last valid footer

read_footer reads the last 84 bytes. If their tag is not good:
recupera_footer, cerca_ultimo_footer.

The file is read backward, 4 MB at a time. Every place is first asked
what costs nothing: a footer says where its segment starts and how many
bytes it holds; the start must be after the header and before the footer
itself, the data must fit in between. The bytes of an encrypted block
pass this once in billions of billions. What passes is checked with the
key (validate_footer_auth_tag: it was there, written for this, and
never called).

fine_valida is where that footer ends. get_actual_file_size() gives
it in the place of the size of the file: everything that walks the
chain of the footers starts from there, with no other change.

taglia_resti() cuts the file there. Called where a Franzen file is
opened to be written (OutputArchive, twice), and by write() before a
new segment. The size "at the start" (g_starting_franzenfile: where a
control-C, or a version with nothing in it, brings the file back) is
made that one.

g_franzen_fini: where the good part of a file ends, found once for a
run (every thread opens the file on its own). Checked again each time
it is used.

Also: fwrite on a FILE fills a buffer. The disk is touched by
fflush and fclose, and their result was not looked at.
g_franzen_scrittirotti counts every write that fails; with no .zpaq
behind, the "written bytes are not the expected ones" of the end of
every run could not see it.

7. add() is add2()

There were two: add(), and add2() (the fast one of a: the
fragments cut and hashed by several threads). backup, -append,
-home, cloud, q, g used the first one.

Now there is one, the second. The body of the old add() is under
#ifdef ANCIENT (old compilers: it stays). -noturbo and -715 are
one thread inside add2(). -turbo is accepted and ignored.

add2() did not have -append (the two passes): its five pieces were
brought over.

8. backup -chunk: an index that is not a chunk

OutputArchive::apezzi: this output is cut in chunks. It was "the size
of the chunk is not zero", for every OutputArchive: the index too
believed it was an archive in chunks. That is why -chunk and -index
could not live together.

The chunks of a backup are born .tmp (nomepezzo), as its parts are.
finalemultiparttmp names them at the end, from the last to the first:
if it dies half way there is a hole in the numbers, and the next backup
stops, in the place of writing after a version cut short.

gestiscicalcolifinali: a pass of calcolifinali(file) for every
chunk of the run (its hash, its line in the list).

Not with Franzen: at the end of a run the header is written again in
the first chunk, which with Franzen is a file already closed.

9. congela and findzpaq

Jidac::congela(): at the end of the check of the parameters of a,
and inside backup() after the check of the parts and before the
.pid. Once for a run.

Three phases, and the order is the point: filecopy with the verify of
every file; the originals renamed .frozen, from the last to the first;
the .frozen deleted. A move can stop half way, with the archive in
two places. Here, until the last copy is verified, nothing has changed.

multipart::enumerate used -maxsize and -minsize as filters on the
parts too: a part over -maxsize was not seen ("HOLE DETECTED").

Jidac::findzpaq(): in doCommand(), before everything. It was a piece
of the check of the parameters of a.

10. franznuke

The first idea (65.11g) wrote zeros on the first 64 KB of the files in
use, then on NN% of the volume. It left the deleted files whole (their
records were skipped), and with a small NN almost everything else.

franznuke reads every FILE record of the $MFT, in use or not, base
or extension, every attribute that is not resident, and makes a list of
runs of clusters. The three passes are an order on that list, by how
much an undelete tool gets from each piece. older_records goes on
after the end of the $MFT, as long as it finds records.

FSCTL_ALLOW_EXTENDED_DASD_IO after the lock: without it the last block
(the copy of the boot sector, outside the volume) always gave an error.

The format is FormatEx of fmifs.dll. The free space is
franzzerontfs, the one of f -zero -ntfs.

f -gui is Jidac::fillgui, on franzscelta (the screen of ac -gui
and xc -gui), with a few more things in it: ENTER that takes the row
under the cursor, a new screen in the same area. At the end it sets the
flags and calls fillami(): warnings and captcha are the ones of the
command line.

11. How it was tested

A lot :-D

12. Found on the way

• Franzen was never used on a Mac, an OpenBSD, a FreeBSD of the test
  bench: no libsodium there. On macOS zpaqfranz did not even look for
  the right file.
• A static binary cannot load libsodium (FreeBSD -static): no
  Franzen there, whatever is installed.
• The index of an archive is compressed: the size of an archive
  changes by a few bytes with the dates of the files. A test that wants
  a size to the byte must fix them.
• chkdsk in read only makes a shadow copy: Windows copies blocks
  of 16 KB into free space. Who measures a surface after it sees data
  "in the wrong place" that no program wrote.

13. Known limits and open points

• A real USB stick for -nuke and for f -gui: everything was
  done on virtual disks. Sectors of 4 KB, clusters that are not 4 KB, a
  fragmented $MFT, very big volumes: not tried.
• -nuke below 100: the body of the files over 16 MB that does not
  fall in the NN%; what was deleted long ago (its record used again).
• -chunk with -franzen: refused.
• The two files of -debug after an update that did not finish:
  not recovered (see part two).
• A Franzen archive whose first run did not finish: to be deleted
  by hand.
• consolidatebackup, rebuildbackupindex: no Franzen parts (yet).
• -freeze: the place with too little space was read in the code,
  not tried; a process killed between the rename and the delete; a real
  NAS. With -append and -home it is ignored.
• The recover: the stretches of a damaged image are offsets in the
  stream of the used clusters, not in the disk, and not names of files.
  xc stops at the first piece that is not good.
• A Franzen password is 255 bytes at most.
• Not tried here: Solaris, Arch, openSUSE.
  The ARM binaries for the NAS, and the arm64 half of the macOS binary,
  were built, not run.
  ... BUT OPENBSD IS DONE :-D