zpaqfranz 65.15

DO NOT USE IN PRODUCTION: IT IS STILL TO BE STABILIZED

65.15 has two "things":

1. Less RAM: to handle bigger archives (millions of files), and to work
   better where the RAM is little (a NAS, a VM, a small machine).
2. Tests that can be trusted: a general strengthening of the integrated
   test functions (t, t with a folder, tw, pp, p, and x that
   checks while it extracts). OK only when it is really OK.

1. Millions of files, little RAM

65.15 is about the extreme case: a folder of 3 million files, almost all
duplicated (hundreds of versions of the sources of Wine, 60 GB), on a
Windows 11 VM with 8 GB of RAM. Measured there, before and after:

And when the RAM is not enough:

zpaqfranz x z:\1.zpaq -to z:\out
03664$ RAM: 297.24 MB free for the blocks, 270.94 MB taken, this one 87.19 MB: fewer threads at a time (slower, but it goes)
RAM: at most 2 blocks decompressed at the same time (16 threads)
98.937s (00:01:38,1.27GB) (all OK)

For the index that does not fit at all there is x -lowram: the archive
is read once per pass, every pass takes a part of the names.

65.15b adds a command, tw: the test of w -ramdisk -test -checksum -ssd
in two letters, with a dashboard; a folder after the archive and it is
t archive folder, but decompressing every file.

2. Tests that can be trusted

65.15 tests the tests. Damaged copies
of archives, made on purpose (every byte of them, one at a time, in six
ways), given to t, t with a folder, tw, pp, p and x, and what
they say compared with what really comes out of x. The rule:
OK only if it is really OK; something that can be recovered is a
WARNING; everything else is an ERROR. The same campaign, before and after:

Then 40,198 copies of real archives (-m1 ... -m5, -m8, -nofast, -key, three
versions with files changed and deleted), damaged where the structure of
the blocks says it matters, on FreeBSD: t, tw and pp say the same on
every copy, p is never OK with a file lost, and the same copies under
AddressSanitizer gave no memory error. None of this makes the normal t and
x slower: measured, within 1%.

Part one, what is new, for the user

1. d: the duplicates of millions of files

After the hashes, d looked at every file of every group once more, by
its name, to see the hard links: 2.9 million more opens, one thread, no
output. Now who a file is on the disk (volume, file, how many names) is
taken while it is hashed, from the very file that is read. The scan keeps
only what d uses (size and date) and a small file is read with a small
buffer (it was 512 KB made and zeroed for every file).

The output is the same, to the line: the groups, the [hard link], the
sizes, the files deleted with -force.

2. a of millions of files

Measured on the VM: a of G:\test (3,480,314 files, 64.85 GB) on 8 GB of
RAM, 65.14n: the data written (26 min), then "zpaqfranz error:
std::bad_alloc" at the end, 7.02 GB of commit, no archive. Two things take
the RAM there: the files of the disk (every one with a franz block of 551
bytes that a never uses) and the history of -fast, written at the end
(about 1.3 KB of RAM for every file while it is written).

Now the files of the disk have no franz block: on 100,601 files the peak of
a goes from 398 MB to 343 MB (272 to 217 MB with -nofast), the same
archive. On the VM a of G:\test is done: 23.8 min, 5.17 GB of commit,
3,117,410 files added, the history in too ("stored 1 versions in 390.22
MB"). With -nofast: 24.7 min, 5.19 GB.

And the history turns itself off when the RAM is not enough for it: a
backup that fails is much worse than an l that reads the whole archive
(as zpaq did, and as zpaqfranz did before -fast).

03686$ -fast: the history with this version (11.53 MB) would take ~126.81 MB of RAM, 87.04 MB free: it stops here (l reads the whole archive, as zpaq)

It is not for ever: the next a with RAM enough makes it again from the
index of the archive (the same l as one that reads all, every version);
with little RAM that making again is watched as it grows, and stopped at
once. On a big machine nothing changes; on a small NAS it goes away only
when it would not fit. Measured on 13 versions of 248,058 files (a folder
that changes, files added and deleted every time): the history takes 216 MB
at the first version, 133 - 148 MB at the next ones, and almost no time
(4.1 s against 4.0 s for an update). The same versions with the RAM limited
to 700 MB:

3. Little RAM: slower, but it goes

x and t decompress a block only when the RAM can take it, with the
others: the memory of its model (written in the header of the block) and
its data. Otherwise the thread waits: fewer threads at a time, down to one.
Before, with little RAM, every thread got std::bad_alloc ("Job N
killed", one after the other) and the files were not extracted.

A std::bad_alloc that gets to the end now says what there was and what
to try:

3.480.314 +added, 0 -removed
03667! OUT OF MEMORY (std::bad_alloc): 0 files in the index, 3.480.314 files of the disk, free RAM now 1.11 GB
what to try: fewer threads (-t1), a part at a time (-only), more RAM or pagefile

(it was "zpaqfranz error: std::bad_alloc" and nothing more). Out of RAM
while reading the index is no more taken for a damaged block ("Skipping
block", and the index read on without RAM).

4. x -lowram: the index that does not fit

zpaqfranz x z:\1.zpaq -to z:\out -lowram
zpaqfranz x z:\1.zpaq -to z:\out -lowram -n 8

Ten million names of 100 bytes are 1 GB of RAM before anything else. With
-lowram the file on disk that has them is the archive itself: a first
look takes a sample of the names, then every pass extracts the files of an
interval of names (sorted), each one by a zpaqfranz of its own (all its
RAM given back at its end), then a last pass gives the folders their dates
(the files of the other passes went into them). -n X: X passes;
otherwise from the RAM free (half of it for the index).

-lowram: 100.601 records of names, ~90.18 MB of index (940.00  B each), free RAM 15.65 GB: 4 passes (-n)
-lowram: pass 1 of 4
-lowram, pass 1 of 4: names from <<(the first)>> to <<w/versione-049/sorgenti/sotto/file_0439.c>>
...
-lowram: the folders, their dates (after the files of 4 passes)
-lowram: 4 passes + the folders, all OK

What comes out is what a plain x gives: on 100,000 files, the same
100,602 names, sizes, dates (of the folders too) and contents. When x
sees little RAM and many files (the index already more than the RAM left)
it says so, with a box of stars, and suggests -lowram.

5. w -test -verify: where the files are looked for

An issue: w ... -to d:\temp -test -verify gave 22 files MISSING, because
-to is not "where to test" (with -test nothing is written) but where
the files on disk are looked for, as x -to would put them. Now it is said
at once:

-test -verify: the files at their stored paths (with -to: where x -to would put them)
-test -verify: the files under -to <<d:/temp/>> (where x -to would put them), not at their stored paths

The help of w says that -test needs no -to, the title of the bars
that they count the RAM and the disk (411 MB in 44 parts for 22 files of
205 MB).

6. a -verify wrote the CRC-32 as 00000000

Since 64.x: with -verify (and -collision) a wrote the CRC-32 of every
file from the CRC-32 of its fragments, and those were left at zero. Every
file had "GURU-C ... CRC-32 from fragments 00000000" on the screen, the run
ended "(all OK)", and the archive had CRC32: 00000000: the switch for a
safer backup took the CRC-32 away. 62.3a was right, 64.1e already not. Now
the CRC-32 is the one of a plain a.

7. -frugal with mixed hashes

a -frugal made the records of the archive (not only the files being
added) 50 or 76 bytes: a record of a version with SHA-256 was cut.
a -frugal -verify on such an archive: "FAIL SHA-256 : 3 of 6", rc 2. Now
every record of the index is as big as it needs (see Part three, 2): the
help no more says "DO NOT MIX HASHES".

8. tw: the test in RAM

zpaqfranz tw z:\1.zpaq
zpaqfranz tw z:\1.zpaq c:\data
zpaqfranz tw z:\1.zpaq -to d:\restored

tw is w -ramdisk -test -checksum -ssd: every file decompressed in RAM
(in chunks as big as the RAM free, -frugal as big as the biggest file),
the hash of the whole file against the one stored, nothing written. The
decompression has a dashboard as the one of t -ssd: a bar for every
thread, the blocks given by their size (the biggest first, to the thread
with less to do), red when a block cannot be decompressed. Then the bars of
the check in RAM.

zpaqfranz tw ../ex.zpaq
tw: every file decompressed in RAM, its whole hash against the stored one, nothing written
1 versions, 34 files, 34.449.407 bytes (32.85 MB)
Chunks 0001 x                     34.425.000 (total decompressed size 34.425.000)
============================================================================
Stage VEF 0001 : errors  0 (0=good)
============================================================================
tw: 31 files (32.83 MB) decompressed in RAM, all as stored
0.422s (00:00:00,32.87MB) (all OK)

With a folder after the archive it is t archive folder, decompressing:
every file in RAM, then against the file on disk (at its stored path; with
-to, where x -to would put it), and as t the files on disk that are
not in the archive (+external) and the files of the archive outside the
folder (-internal). The exit code is the one of t.

zpaqfranz tw ../ex.zpaq d
tw: every file decompressed in RAM, its whole hash against the stored one, nothing written; then against the files on disk
-test -verify: the files at their stored paths (with -to: where x -to would put them)
1 versions, 34 files, 34.449.407 bytes (32.85 MB)
Chunks 0001 x                     34.425.000 (total decompressed size 34.425.000)
============================================================================
02575! DIFFERENT   <<d/s/f3.bin>>
VEF 0001 : errors  1 (0=good) *** NOT GOOD ***
============================================================================
02579! Verify against the filesystem: 32 files (32.83 MB), the same 31, DIFFERENT 1
Scanned          1 00:00:00          2 file/s (                    0)
03692$ EXTERNAL    <<d/s/nuovo.txt>> (on disk, not in the archive)
03693$ tw: 1 file(s) on disk not in the archive (+external, as t), 5 bytes
0.422s (00:00:00,32.87MB) (with errors)

t archive folder compares the hashes stored with the files on disk: it
does not decompress. On an archive with 16 bytes changed in the middle of
a block t z:\r.zpaq c:\data says "same", rc 0; tw z:\r.zpaq c:\data
"NOT AS STORED 1", rc 2.

9. A damaged index does not hang any more

16 bytes changed 200 bytes before the end of an archive, and l, t, x
ran for ever, a core at 100%. 63.1 and 54.8i too: an old defect of zpaq.
The blocks of the index (i) are written with method 1, whose LZ77 decoder,
a ZPAQL program, is in clear in the archive; a damaged program can loop,
and ZPAQL has no limit of steps. One byte changed at random in that
program: 6 hangs in 200 tries; in the data under it: none in 160.

Now the program of a block of the index is looked at before it runs. The
one that zpaq (6.49, 6.60, 7.07, 7.10, 7.15) and zpaqfranz (54.8i, 62.3a,
64.1e, 64.5i, 65.15b) write there, measured, runs as always. Any other is
not refused (the format is made so that an archive carries its own
decoder: a "cousin" may write another one): it runs in the interpreter
with a limit of steps, very generous for a real decoder. Over it the block
is damaged, as with a bad checksum:

Skipping block at 34449231: ZPAQL program does not end (too many steps): damaged block?

400 more tries (the index and the pointer): no hang, the 7 programs that
did not end say so; the 6 bytes that hung before: rc 2, "does not end".
On normal archives nothing changes (l of 60,000 files: 0.27 s before and
after). With a program not known (all the 608 index blocks of that
archive run counted, as a test): 0.97 s, the same output. The blocks of
data are as before: a damaged program there can still hang (but not the
one of -m8, since 65.15z: see 12).

10. Smaller things

• w of an archive with only empty files (or folders) crashed, 65.14n
  too: now "no file with data in the archive: nothing to decompress", and
  the folders and the empty files are made (or checked) as before.
• The speed on the last line of the bars of t -ssd and w -test -verify
  was taken at the look after the end (up to 200 ms later): a short run
  showed 160 MB/s for 1 GB/s. With -t1 the clock of t -ssd started
  after the work.
• w -test -verify with nothing to compare: "no file to compare", not "0
  files (0.00 B) all the same".
• w -test -verify: the bar of a file not as stored in RAM (the archive)
  is red at once, as the one of a file different on disk; it was green,
  "different 0", and the verdict after.
• franztime 1.4: -mem N (512m, 3g...), the memory of the program
  limited as on a machine with little RAM (a job object): over it an
  allocation fails. The peaks count the request that was refused too (it
  says so).
• t: the "RAM ~" line counts the blocks really at the same time.
• The free RAM was wrong on some systems, and w -ramdisk did not work
  there ("Chunk size must be greater than zero"). Linux before 3.14 (no
  MemAvailable, as the machine that builds the ESXi binary): "invalid
  /proc/meminfo format" and 0; now MemFree + Buffers + Cached, 1.53 GB as
  expected. The NAS where long is 32 bit (i686, arm): "negative" over 2 GB;
  now 2.68 GB as expected. FreeBSD: the same overflow (int * int), fixed,
  not tried yet. Now that free RAM decides the threads of x and the
  history of a, it matters more.

11. The tests say OK only when it is really OK

Every test (t, t archive folder, tw, pp, p) now ends with one of
three answers, and its exit code:

Before, a test could see the damage, write it, and end with 0. Found by the
campaign, all fixed:

• t archive folder compared the files with the folder and ignored the
  errors reading the archive: rc 0 on 46 damaged copies (03734, now rc 2).
• pp printed "72415! DAMAGED ARCHIVE" and then "Hash OK for 3 (this is
  GOOD)", rc 0, with two files of five lost (03735).
• p returned 0 always, also after "WITH ERRORS"; "Total 0" (nothing
  tested: an empty file, an encrypted archive) was "All OK" (03738).
• The history of -fast damaged: all the tests OK, only p saw it. Now a
  WARNING (03739, 03746): the files can be good, l reads the whole archive.
• x with one file not extracted was rc 1 ("with warnings"), with two rc 2:
  now one file is enough for an ERROR.

And a damage that the tests saw only in part, now said by all of them, as a
WARNING when the files are good: bytes that are not a block before a block
(03747), after the last one (03744), the last byte of a c block (03748),
the end of a block of data (its name, its comment, its SHA-1: 03745), the
tag of a block not whole (03751), the comment of a block of the index
(03752).

When the archive is damaged but every file tested is good, the answer
depends on what is known of the rest: with VFILE-info (the record of every
version, written since 65.8t) a WARNING if every version has all its files
(03749), otherwise an ERROR (03750): a file that is not there is not tested,
and nobody can say it is not missing.

72415! DAMAGED ARCHIVE: blocks are missing in 1 version(s): 3
None of the 3 files of version 3 (the one read) uses the missing data: they can be extracted
03749$ The archive is damaged, but its index is whole (VFILE-info: every version with all its files) and the files tested are good: a WARNING

12. No more hangs, no more prompts nobody can answer

• A damaged archive whose first bytes looked like the start of an encrypted
  one asked "Enter AES password:" and waited for ever, also in a script with
  no console (most of the hangs of the first campaign).
  Now: 42267! Password required, but stdin is not a console: nobody can type it. Use -key, rc 2.
• The ZPAQL program of the -fast history, damaged, looped for ever (t,
  tw, pp, p): now it runs counted, as the index (65.15b).
• The program of a -m8 block (the zstd decoder in ZPAQL) with one byte
  changed looped for ever in t, tw, pp, x. Now it is recognized as
  the -m8 one damaged and its data go through zstd as always (03756, a
  WARNING): x extracts every file, the SHA-1 of the block says they
  are right. It costs nothing on a normal archive.

13. x recovers more, and says ERROR when a file is wrong

• After a block that cannot be read, the archive is read again from just
  after its tag. It was read from where its decoder stopped: a damaged LZ77
  ran over the blocks after it, up to the end, and the versions after it
  were lost for every command (3 files of 5).
• The length of a block written wrong (a stored block, a byte changed in
  its size): the input is now limited to the block. It was an overflow and
  two files lost; now all the files come out.
• zstd refuses a frame whose header is damaged in a byte that the decoder
  of the archive never reads: now the decoder of the archive (the ZPAQL in
  it) takes the block again, and x extracts everything (03755, a
  WARNING). Only when zstd fails: no cost otherwise.
• An h block with a wrong number (or written twice) moved the IDs of the
  fragments after it: x made a file of the wrong fragments, every one
  with the right SHA-1, and said WARNING. Now an ERROR (03757).
• An i block of another version in the place of one: x extracted the
  other list of files, with a WARNING. Now x reads VFILE-info as t
  does (in RAM, no I/O): an ERROR.
• A version that began (its c block) and of which nothing can be read
  (-key: a byte taken away in the c block, the rest decrypted wrong): it
  was a WARNING, two versions lost. Now an ERROR (03754).

14. pp and tw as t

• pp decompresses as x (to the end of the block, then its end looked
  at). A fragment not good used by no file of this version but by a file of
  an older one (the one before a change) is data lost for x -all: it was
  a WARNING (66407), now an ERROR (03753).
• tw now looks at the end of every block it decompresses, as t and
  pp. On an archive of 10 versions with 108 files rewritten: the same
  time (1,684 - 1,749 ms against 1,694 - 1,741 ms, one thread).
• Measured on the last 3,873 copies: t, tw and pp give the same
  answer on every one.

15. p is paranoid for real

p rebuilt the hash of every file from its fragments: a loop used the
index of the files instead of its own, and after a file of N fragments the
next N files were not hashed, nor counted:

31 files in the archive (one of 2 MB, 30 small)
before:  Total : 6   GOOD : 6 of 6   All OK (paranoid test)
65.15z:  Total : 31  GOOD : 31 of 31 All OK (paranoid test)

It was there since the first p. The autotest expected the wrong number
(42,727 files instead of 42,735): fixed too. And p sees more now:

• the comment of every block exactly as libzpaq writes it (<size> jDC),
  the space too;
• every block of a version with the date of its c block, the i blocks of a
  version numbered 1, 2...: a c block or an i block taken away (files lost)
  was "All OK";
• the 16 bytes before a block (tag and zPQ) exactly;
• bytes in no block, at the start, between blocks, after the last one;
• the program of a -m8 block, if damaged.

On the 40,198 copies p is never OK when a file is lost. It stays OK when
the copy decodes to the same bytes (a byte of a program that never runs
for that data, a field of a header that a block without a model does not
use): there the content is the same, and only a hash of the whole file (a -checktxt, backup) can say that a byte changed.

16. work faultgen and work probe (Windows)

The battery that found all this is in zpaqfranz itself, for Windows:

zpaqfranz work faultgen -to g:\set -kill -force
zpaqfranz work probe g:\set -t8

faultgen writes 5 small files, 4 archives of them (3 versions, -fast and
-nofast, with and without -whirlpool) and a damaged copy for every byte
of each one, six ways: the byte XOR FF, the archive cut there, 512 or 4096
bytes of zeros (a sector, a cluster), a byte put in, a byte taken away.
About 84,000 copies, 300 MB, 12 s. probe runs t, t with the folder,
tw, tw with the folder, pp, p, x on every copy (-t at a time,
each one in a job object: 1 s and 512 MB at most), compares what x wrote
with the originals byte by byte (and date, and attributes), and writes
report.txt: every check, the ones not OK with what they wrote, and the
verdicts. Run again on the same folder, it does only what is not OK.

03713! work probe g:/corrotti2/: 844420 checks, 220409 OK, 624011 NOT OK (ERROR 466961, WARNING 155634, INCONSISTENT 88, FALSEALARM 425, MISSED 903)
591094 checks run, 16 at a time (-t), in 17.628.813 ms: 33.53 checks/s (timeout 1.000 ms)

17. Smaller things

• pp crashed (0xC0000005) on a -nofast archive whose first block was
  lost: a binary search with unsigned numbers went to -1 (4 billion). Fixed
  there and in the two same searches of mount.
• Two zpaqfranz started in the same moment had the same temporary folder,
  and two x in folders with the same parent the same test file: one took
  away the file of the other ("cannot write", rc 2). Now the process ID is
  in both names.
• p -key hashed the key once more, and could not read an archive with
  the right password.
• -tar in the NAS builds (static, musl). An issue: Oracle Linux 7.9
  (glibc 2.17: a static binary is what runs there), a / -tar, and
  "79531! Cannot get metadata" for every one of 161,245 files, "0 good
  metadata": the NAS builds are ANCIENT for the rest, and there -tar
  did nothing. Now it works as on the other *nix (tried on the NAS VM:
  x86_64, and aarch64 and armv7 under qemu; the round trip of
  bench/tar/tartest_nas.sh gives the differences of GNU tar, the socket,
  plus the hard link group with a first name over 215 bytes). It runs only
  with -tar: on a NAS whose system lacks something of it, -tar fails,
  and it is not given. The ESXi build stays without.
• Oracle Linux, CentOS, RHEL 7 (gcc 4.8.5, glibc 2.17): zpaqfranz did not
  build there (67 errors), and the way out taken was -DANCIENT (the same
  issue: there -tar does nothing). Four things: C++98 by default (now a
  clear #error says "add -std=c++11", and the Makefile adds it when the
  compiler, asked, is older than C++11), statx() (glibc 2.28: without it,
  no birth time, as on the systems that do not keep one), the SHA
  instructions of the CPU (gcc 4.9: without them, the SHA in C++), and
  dlopen() in libdl before glibc 2.34 (the Makefile now gives -ldl on
  Linux always: libsodium and libcurl are loaded with it, not only with
  SFTP; make static keeps it too). Tried on Oracle Linux 7.9: make
  (SFTP too), make static, autotest -all, autotest -m3 (61 checks),
  the -tar round trip. By hand:
  g++ -O3 -Dunix -std=c++11 zpaqfranz.cpp -pthread -ldl -o zpaqfranz.

Part two, the details, for power users

1. The RAM of the blocks

• The budget: 75% of the RAM free when the threads start (the index is
  already in). On Windows no more than the commit still possible, and than
  the limit of a job object if zpaqfranz runs in one (franztime -mem, a
  service).
• A block takes its model and twice its data (the buffer doubles). One
  block always goes, when no other is being decompressed: it may still be
  too much ("03665$ RAM: a block takes ..., more than the ... free").
• When it waited once, every thread frees the data of its block before it
  waits again (a thread waiting holds nothing).
• x, t (and x -ramdisk); not w, that sizes its chunks by the RAM.

2. -lowram: the rules

• x only; not with -stdout, zip, -recover, -extras, -ramdisk
  (03681).
• Every other switch goes to the passes as it is (-to, -only, -not,
  -until, -checksum...): a pass is a plain x of the names of its
  interval. The exit code is the worst of them.
• The passes: -n X; or (records x 900 bytes + the names) / (half the RAM
  free) + 1; at most 1000, and no more than the names sampled.
• The cuts come from a sample of 65,536 names (reservoir): the passes are
  about the same size, the files of a folder stay together (fewer blocks
  decompressed again by another pass).
• The price: the index is read once per pass, and a block with files of
  two passes is decompressed twice. A deduplicated archive (versions of the
  same files) has many of them.
• The child gets its pass from the environment (ZPAQFRANZ_LOWRAMPASS)
  and a file in the temporary folder (the cuts, in hex): the command line
  is the same.

3. -fast that turns itself off: the rules

• The history takes about 11 bytes of RAM for every byte of it while it is
  made (the worst measured: 11 on 100,601 short names; 6.8 on 248,058 files
  at the first version, about 4 at the next ones, about 5 on 3,480,314).
• An archive with a history: its size is in the pointer at the end, the
  decision is taken before the history is made again in RAM: more than a
  third of the RAM free, and it stops (as -nofast: the "stop" record).
• A new archive, or a history that grows too much with this version: the
  decision is taken after the data, before the history is written: more
  than half of the RAM free, and it is not written (its RAM given back).
• The RAM free not known (a system where it cannot be read): kept, as
  before. -verbose says the estimate when it is kept (03687).

4. tw: the rules

• tw = w with -ramdisk -test -checksum -ssd: the other switches of
  w are the same (-maxsize, -frugal, -key, -franzen, -only,
  -not, -t N, -noeta...).
• A folder (or more) after the archive, or -to: the files on disk too
  (-verify). A folder: the whole archive is read, the files outside it
  are -internal and not decompressed. -to: no +external, no -internal.
• Exit code 2: a file not as stored (the archive), an error reading the
  archive (a block skipped, the index damaged; it was 1 for w), a file on
  disk not the same, missing, not readable, bytes in +external or
  -internal. 1: the only differences are empty files in +external or
  -internal ("maybe folder mismatch", as t).
• Stricter than t: an empty file missing on disk is MISSING, rc 2 (as
  w -test -verify); t counts it -internal, rc 1.
• The dashboard: rows = threads (no more than the blocks); a thread does
  its blocks first, then helps with those not started, their bytes on
  their rows (every bar gets to the end). Not on a console: a line a
  second (03688), none with -noeta. What the threads say meanwhile (a
  block skipped, a fragment with a bad checksum, out of RAM, the RAM
  waits 03664/03665) comes after the dashboard: on it, it would break the
  bars.
• The RAM of the blocks as x and t (part two, 1): what is free less the
  files of the chunk, already in RAM; "RAM X in N" on the info line.

5. What can change for a script

• tw: a new command.
• w of an archive with no file with data: 03694 (03700 if the index is
  damaged) instead of a crash. w -test -verify with nothing to compare:
  03697 instead of 02578.
• A block of the index whose program does not end: "Skipping block ...:
  ZPAQL program does not end" (an error, rc 2 for t) instead of a hang.
  autotest: one more check (03701 with -verbose, 03702 if it fails).
• w -test -verify: one more line at the start (03663).
• x with many files and little RAM: a box of stars (03683, 03684) that was
  not there.
• x/t with little RAM: 03664, 03666 instead of "Job N killed"; 03685
  for a thread out of RAM while it writes.
• Out of RAM: 03667 and 03668 instead of "zpaqfranz error: std::bad_alloc";
  while the index is read, instead of "Skipping block" (that is: damaged).
• a -verify: no more "GURU-C"; the archive has the real CRC-32.
• a with little RAM: 03686, and a version with no history (l of that
  archive reads all, until an a with RAM enough makes it again).

6. The answers of the tests: the messages

A code with $ is a WARNING (rc 1), with ! an ERROR (rc 2).

p has no WARNING: the least thing not as written is an ERROR (its errors
are "Error at offset N: ..."). "all" means every command that reads the
archive (l too says them); x says the WARNING with rc 1.

7. work faultgen and work probe: the rules

• Windows only (job objects, CreateProcess); not in the help of *nix.
• faultgen -to FOLDER [-n N] [-kill] [-force]: in FOLDER originali/
  (the 5 files), archivi/ (the 4 archives, full paths in them: the tests
  with a folder want them), preparati/ (the copies, named
  family_OFFSET_archive, no family for XOR), versioni.txt (the size of
  every archive after each version). -n N: only the first and the last N
  bytes; -kill: the copies damaged (without, identical: a dry run);
  -force: a FOLDER already there is made again.
• probe FOLDER [-tN]: the tests are a table (faultgen_prove[]), $
  placeholders for the archive, the folder, the passwords, the -to of x.
  States: OK, WARNING, ERROR, HANG (over the timeout, killed), CRASH, OOM,
  NOTSTARTED; then, for every copy, coherence (INCONSISTENT: the tests do
  not say the same, OK against not OK; t with a folder is not counted: it
  compares, it does not decompress) and verdict against the truth (the
  files of x, byte by byte): SILENT (OK with a file not as it was),
  UNDERRATED (WARNING with a file not as it was), FALSEALARM (ERROR with
  everything as it was), MISSED (p not ERROR on a copy not the same as
  its archive).
• A cut copy: the files expected are the ones of the versions begun (their
  c block whole).
• At the end three tables: the checks not OK by why, the verdicts by why,
  the INCONSISTENT by why.

8. What can change for a script (the tests)

• Exit codes of tests that were 0 on a damaged archive: t archive folder,
  pp, p are now 1 or 2 (see 6).
• p returns 1 or 2 (it returned 0 always); 2 with nothing tested.
• x with one file not extracted: 2 (it was 1).
• A WARNING (rc 1) where there was 0: the history of -fast damaged,
  bytes that are not a block, a comment, the end of a block of data.
• A password asked with no console: 42267, rc 2 (it waited).
• New lines in the output (the codes of 6).
• The NAS builds with -tar: the metadata are stored and given back
  (before: 79531 for every file, nothing stored).

Part three, lo spiegone, for developers

1. d: franzidentita

franz_do_hash::filehash() with i_identita takes volume, index and links
of the open FILE* (GetFileInformationByHandle on its handle, fstat
on *nix: franz_identitafp). The threads of summa() (tparametrihash,
now with a constructor: its six other users get identita= false) keep
them with their files; the index field of s_stringpair (unused there)
points to its identity, so vec is sorted as before. dedup_elenco()
takes them, franz_qualefile() only for a file not hashed from a
FILE* (-wyhash, -nilsimsa). The scan runs with g_dtlight and
g_dtnoblock, as l from 65.12e. Tried: tortura/d15/prova.sh (16 cases,
the old exe as reference, hard links, -force).

2. The franz block of a record

dt_inserisci() made a block of dt_franzdimensione() for every record:
551 bytes (FRANZOFFSETV3 + 1), 50/76 with -frugal, while without -tar a
record stores about 190. Now dt_franzmisura(): what was stored, with
zeros up to the farthest offset decode_franz_block() reads (79, 191 for a
'3', the rule of fl_record of 65.12e), the whole 550 of a '4' (its posix
is read by zip and -tar, without -tar too), record by record: an
archive can have every kind in its versions. Tried: tortura/franzblk
(11 versions of every kind, -tar on *nix; l, t, i, x -checksum the same
of 65.14n).

3. franzpermesso

Made before the try of a block in decompressThread and
decompressthreadramdisk, taken (prendi) after findBlock(&mem), given
back at the end of the loop (after the files of the block are written: its
data are in RAM while they are). ExtractJob has ram_libera,
ram_budget, ram_inuso, ram_attivi; budget 0 = no limit (w, an
unknown RAM). franz_ramlibera(): the free RAM, on Windows the commit and
the job too (K32GetProcessMemoryInfo taken at run time).

4. pocaram()

franzpocaram (mode 1 the first look, 2 an interval, 3 the folders) and
g_pocaram, looked at in read_archive() after g_rangefrom (not in
fl_readarchive, the -fast history of l). The parent makes the first
look in the very read_archive() of extract() (what comes before, as
-until and -comment, unchanged), then Jidac::pocaram(); a child:
CreateProcessW of GetCommandLineW() on Windows, fork + execvp of
the argv of main on *nix. In the pass of the folders a folder that is
there is not skipped (without -force every one would be, with the date
of the extraction). Tried: tortura/lowram/prova.sh (10 cases, on Windows
and Fedora).

5. a -verify

In add2(), three copies (-turbo, the batches, the plain loop) of
uint32_t crc= 0; just before ht[htptr].crc32= crc: a new variable that
hid the CRC-32 computed a few lines above, so every fragment got 0.

6. Out of RAM in the threads, and while the index is read

• A decompression thread out of RAM while it writes the files of its block
  (g_crc32.push_back, rename(), outside the try of the block) ended
  in std::terminate. Now a try around that part in both threads: the
  thread gives back write_mutex, says it once (03685) and ends, as "Job N
  killed"; its files are counted as not complete at the end (rc 2).
• read_archive() took a std::bad_alloc for a damaged block ("Skipping
  block") and read on: now it goes to the end (03667).
• dt_franzalloca() out of RAM: throw std::bad_alloc (it was
  seppuku(), with no word of why).

7. The scan of a, the buffer of filehash()

• add() and add2() scan with g_dtnoblock: the files of the disk have
  no franz block (it holds the block read from an archive; add hashes with
  the objects of the DT and writes from a buffer of its own in
  writefranzattr()). Tried: tortura/addnb/prova.sh (13 kinds of switch,
  two versions each, l -all -checksum with and without the history the
  same of 65.14n; on Windows and Fedora, -tar there).
• franz_do_hash::filehash(): capienza, the size of the buffer (16 KB at
  least, BUFSIZE at most) instead of BUFSIZE made and zeroed for every
  file. Tried: sum with 17 hashes on 17 sizes around the edges, the same
  of 65.14n.

8. fl_troppogrande()

fl_dimensionepuntatore() reads the bytes of the history from the pointer
(ZPFL1|versions|bytes|hash|...); fl_troppogrande() compares them, by
FL_RAMPERBYTE (11), with a share of franz_ramlibera(). In
read_archive_add(), after fl_findpointer(): too big, and
flagnofilelist goes on, the very way of -nofast (the "stop" record
written by filelist_store(), no second read). In filelist_store(),
before fl_beginversion(): fl_blob + the new i blocks; too big, and
fl_blob, fl_names, fl_vnames, fl_vrecs are given back, the stop
written if the archive had a history. The next a with RAM enough makes
the history again (fl_capture while the archive is read):
fl_controllacattura(), after every i block, looks at it every 4 MB and
stops it (flagnofilelist, its RAM given back) when it would not fit.
Without it a NAS that turned the history off once made it again at the next
a, and that a died in the compression. Tried: on
100,601 files with franztime -mem (new archive: not written, l reads
all; an archive with the history: stopped, then made again, l the same
of l -nofast on all the versions).

9. tw

• flagtw: the parser makes it w with the four flags. In extractw():
  flagverify with a folder or -to; without, soloram: g_wconti and
  wconfronta_prepara(..., false), the RAM only, the summary
  wconfronta_riassuntoram(). With a folder, wconfronta_riassunto()
  then tw_esterni(): scandir of files with g_dtlight and
  g_dtnoblock, dt.find of every file.
• -internal: files is moved away before read_archive() (the whole
  archive is read), then back, and the records not isselected() go to
  g_twinterni and out of dt, as if read_archive() had filtered them.
• The dashboard: ExtractJob has cruscotto, proprietario (the row of
  every block, LPT on the usize of its fragments), blocco_fatti,
  blocco_errore, finiti, fine_ms. decompressthreadramdisk() with
  cruscotto takes its blocks first, writes the bytes of its block as they
  come, gives it back to 0 when out of RAM; decompressthreadramdisk_tw()
  counts the threads that ended; tw_cruscotto() starts and joins them,
  franzbarre every 200 ms. With cruscotto the lines 00706-00709 of the
  thread and 03664/03665 of franzpermesso go to ExtractJob::messaggi
  (code, text), said after finale(); without, the very code of before.
• wconfronta_ramdiverso(): the checks of wconfronta_esito() on a whole
  file, in the thread after wconfronta_ram(): the bar red at once.
• barre_adesso(): the time of the last line, when the last thread ended
  (t -ssd, w -test -verify, tw).
• Tried: tortura/tw/prova.sh (64 cases: the RAM only, -t1, -t5,
  -noeta, -frugal, -maxsize, the folder, the rc of t on the same
  folder for: same, +external with and without bytes, different, missing,
  UTF-8, a subfolder, a wrong folder; -to; damaged archives in three
  places; a deleted file; -nochecksum; only empty files; -key, a wrong
  key; -franzen; the help; the old exe on w and t), tortura/tw/dash.sh
  in a pseudo terminal.

10. The PCOMP of the index

• libzpaq::Decompresser keeps the name of the first segment of a block
  (nomeblocco); in decompress(), at the first segment, nomeindice()
  (jDC, 14 digits, c h or i, 10 digits: the d blocks, the history of
  -fast too, are not) sets PostProcessor::indice. When the PCOMP is
  loaded, pcomp_indice_noto() (its SHA-1 among g_pcompindice); not
  known: z.passimax, and ZPAQL::run() goes to run0limitato()
  (ZPAQL_PASSIINIZIO 1,000,000 steps, ZPAQL_PASSIPERBYTE 10,000 more
  for every byte in). A model in a block of the index (zpaq writes none
  there): its HCOMP is counted too (it runs through ZPAQL::run(), JIT or
  not). The same in libzpaq2::list_Decompresser (the reader of 6.60).
  unzZPAQL (the reader of unz) is its own code: not changed.
• g_pcompindice: the 12 PCOMP of method "1" (x<k>,1,5,0,3,<19+k+(k<=6)>,
  k the size class of compressBlock, 0..11: 0..4 are the same, the one of
  every archive measured). pcomp_indice_check() makes them again with
  makeConfig() + Compiler in the autotest: a change of compressBlock
  would make the new archives slower to list, so it says "Do NOT release".
• Tried: tortura/pcomp/ (one byte changed at random, the index and the
  pointer, 460 tries: no hang); a test build with no PCOMP known and every
  jDC block counted, the data too: autotest -all -m1 ... -m5 (CM, BWT,
  LZ77: no false "does not end").

11. read_archive(): what it says, and how heavy

• libzpaq::Decompresser::findBlock() counts the bytes it read: saltati
  (before the tag of the block found: 03747), coda (to the end with no
  block: 03744), corto (less than 16: the tag not whole, 03751; the
  rolling hashes see only the low 2 bits of its first byte, so a byte taken
  away there still finds the block). The last byte of a c block (its d
  blocks jumped over) is read once (03748).
• The comment of a c, h or i block: commentobuono (the digits, then
  exactly " jDC\x01", what compressBlock() writes), looked at after its
  SHA-1 (03752). All of these add to g_fuoriblocchi: a WARNING in
  conlastoria() and extract().
• After a block that cannot be read: in.seek(bloccorotto + 16), the next
  one looked for from just after its tag (bloccorotto = block_offset +
  saltati). From where the decoder stopped (zpaq 7.15) the versions after
  it were lost; from bloccorotto + 1 the hash found the same block again,
  for ever.
• ra_finebuona: where the last block read well ends. After it, more than
  16 bytes with 64 zeros in a row, or more than 1 MB: a version can be lost
  there (g_codaperduta, 03750). Not with -key: zeros, decrypted, are not
  zeros. A c block skipped (ra_nomeblocco: jDC, the date, c, enough when
  the rest of the name is damaged), nothing read well after it, more than
  128 bytes after it: g_cperduto (03754). A c block cut at the end (an a
  stopped) has nothing after it: not this.
• "Unordered fragment tables" (an h block with a number lower than the
  fragments there): g_tabelledisordinate (03757).
• gravita_danno(), said once: g_fileperduti (72423), then
  g_codaperduta, g_cperduto, g_tabelledisordinate an ERROR; else
  vinfo_indicecompleto() (for every version iu, id, v of VFILE-info
  as found): WARNING, or ERROR without it.
• g_frammentifile: the fragments used by a file (not VFILE-) of any
  version, filled only for pp (command == 'P'): the others pay nothing.
• All of this is a compare or a counter where the data are already read:
  t and x on -m1 and -m8 archives, one thread, before and after these
  checks: within 1%.

12. The end of a test

• conlastoria() (t, t archive folder, tw, pp): VFILE-info
  (vinfo_controlla(), if not done yet), g_codaarchivio /
  g_fuoriblocchi (WARNING), the zeros at the end (gravita_danno()),
  g_anomalieblocchi (03745, counted by the threads), the zstd flags
  (03755, 03756), then the history of -fast (fl_controllastoria():
  its blocks and hash; an older one 03746).
• extract() (x): moltomale (a file not whole) an ERROR; read errors
  with every file whole to gravita_danno(); VFILE-info as t;
  the same flags. The history of -fast is not read (it is not data).
• testverify (t archive folder) and extractstdout (pp): the errors
  of read_archive() were ignored (03734, 03735).

13. Blocks of data

• lettorelimitato (the input of a block limited to its bsize) and
  limiteblocco() (the output: usize + bsize + 64 KB) in x, t,
  tw, pp: a damaged length cannot read the next blocks.
• blocco_controllafine() / blocco_controllasegmento(): after the
  fragments of a block are good, the rest of its segment, its name (jDC,
  date, d, first fragment), its comment, the SHA-1 of all of it, the end of
  the block (03745). t only with b.size >= b.frags (its fragments
  decompressed reach the end: it does not decompress a whole block for a
  small file); pp and tw always.
• -m8: zstd_is_canonical() (the program byte for byte) and
  zstd_quasi_canonico() (as long, at most 1,024 bytes different: the -m8
  one damaged, zstdprogramma): both to zstd. zstd_native_decode()
  returns false (and the bytes already given out) where it threw; then
  zstd_ripiego() runs the ZPAQL of the archive (JIT) from the start,
  zstd_scartaprimi dropping what zstd already wrote (zstdripiego).
  Without an error, the very path of before.
• Tried, and left: a table of the programs compressBlock() writes, the
  others in the interpreter with a limit. The interpreter is 8 - 10 times
  slower (FreeBSD, 120 MB, t -t1: -m1 1.1 s -> 11.8 s, -m2 1.4 s -> 10.5 s;
  build -DNOJIT: on amd64 -nojit is ignored), and a -method by hand
  would pay it.

14. p (unz)

• The second stage: for (uint32_t k= 0; i < f.ptr.size(); ++i), the i
  of the files; now k.
• unzerror() exits with 2 (it was 1); paranoid() returns unz().
• findBlock() wants the 16 bytes exactly (saltati = the bytes in no
  block, 03740), at the start, between blocks, after the last one.
• The comment: the digits, then exactly " jDC\x01".
• A block of a version: the date of its c block (datac); i blocks
  numbered from 1 (attesoi); zpaq and zpaqfranz write them so (the
  VFILE i block too, indexchunks.size() + 1).
• zstd_quasi_canonico(): an ERROR.
• The programs run counted (UNZ_PASSIPERUSCITA, 1,000 steps for every
  byte out), the c, h, i blocks over 1.5 GB of memory refused (as 98003),
  the end of a version (unzversione(): a d block with no h, data with no
  i).

15. The batteries

• work faultgen / work probe (Windows): ventaglio() starts the
  children (CreateProcessW of zpaqfranz itself, the output through a
  pipe), each in a job object (JOB_OBJECT_LIMIT_PROCESS_MEMORY, 512 MB,
  peak taken from it), -t at a time, killed over 1 s. raffica() the
  archives, riempi() the copies (faultgen_famiglie[]), faultgen_confronta()
  and faultgen_metadati() the truth (fread + memcmp, nothing of
  zpaqfranz trusted), faultgen_perche() the why (the codes ! and $, the
  "Error at offset" of p), faultgen_report() written to .tmp and moved.
• tortura/mirate/mirate.py (python3, *nix and Windows): genera real
  archives in three versions (text changed in the middle, a copy for dedup,
  a file deleted, a folder more; -fragment 0, relative paths), the blocks
  parsed (tag, zPQ, hsize, header, name, comment, stored length, program,
  content, FD, SHA-1, FF) and a copy for every byte that controls something
  (the data sampled): XOR, one bit, a byte taken away or put in, plausible
  values (the size in a comment +1, a digit of the date or number of a
  name, csize, the sizes of a fragment), a block taken away, doubled,
  swapped, taken from another version, 64 zeros, a cut, bytes at the end, two
  XOR at once (seeded). 40,198 copies of 9 archives. corri: t tw pp p x -t1, a timeout, RLIMIT_AS, the archives intact first (all OK, or it
  stops), the truth x -all of the intact one, and on with what is done.
  --asan for a build with -fsanitize=address (with asan_zzstd.cpp: the
  zstd in zpaqfranz declares the poisoning functions in its namespace, the
  link fails). analizza: the verdicts as probe, by format, family, zone.
• Numbers, FreeBSD 14.2, 16 CPU: the 40,198 copies in 42 min (14 at a
  time), with ASan 51 min at 12 (13 copies/s): no memory error.

16. Known limits and open points

• A damaged ZPAQL program in a block of data of -m1 ... -m5 (LZ77, BWT) can
  still loop for ever (x, t, tw, pp; p stops, "too many steps"):
  26 copies of 40,198. A cure at no cost (a fixed table of the ~23 programs
  of compressBlock(), the one nearly equal used in place of the damaged)
  was proposed; not done. -m8 is done (03756).

• -key: a byte put in or taken away in the last block of a version makes
  the rest decrypt wrong; the versions after it are lost and the tests say
  WARNING (the end has no zeros to be seen, 03744 + 72414). Left.

• Two damages at once that lose only data of an older version: t, tw,
  x WARNING, pp ERROR (3 copies). Left.

• p OK on a copy that decodes to the same bytes (a program, a header
  field), or cut exactly where a version ends (an archive with a version
  less): only a hash of the whole file sees them. Left (the "cousins").

• -lowram: the fragment table (ht, 48 bytes a fragment) is whole in
  RAM in every pass: 10 million fragments are about 480 MB.

• -lowram is tried on Windows and Fedora; the other *nix are to be built
  and tried.

• a of millions of files: the history (-fast) takes about 1.3 KB of RAM
  for every file while it is written (fl_names, a map of all the names;
  fl_blob, all the history in a string): 100,601 files 217 MB with
  -nofast, 343 MB with it. 3,480,314 files on 8 GB now fit (5.17 GB of
  commit); some millions more, or less RAM, and the history turns itself off
  (the estimate is the worst measured: it can go away when it would fit).

• The RAM of the history read again by an a is taken while the archive is
  read: the decision is taken on its size before, not on the RAM it takes.

• A hard limit of the commit (a job object, franztime -mem): a thread can
  die because its stack cannot grow (0xC00000FD), or Windows can end the
  process (0xC0000409) with no handler called: 65.14n too (13 of 46 limits
  from 20 to 110 MB), 65.15a 10 of 46.