zpaqfranz 65.20: improvements everywhere

No big new feature this time (just a small one: -thunderbird).
65.20 is a release of many improvements, spread all over the program: the
result of a bug hunt.

In short:

• A full disk no longer ruins an archive (bugfix, with data loss): the
  version written half way stayed in the archive, and every backup after it
  was lost. Now the archive is put back as it was.
• The archive is never added to itself (when it is inside the folder you back up).
• An empty name is never the root of the disk: d "" -force (an empty
  variable in a script) could delete the duplicates of the whole disk.
• Exit codes that tell the truth: a dozen cases said 0 ("all OK") on a damaged
  archive, a wrong path, a backup that could not be read, nothing done at all.
• No more crashes on the arguments tried (s with no folder, -until -1, -ramsize 0...).
• No more waiting for a key that cannot come: ls, pause, tui, gui in
  a script, with no console.
• xc with big blocks (issue 357): a restored disk could come out short,
  without a word, when the RAM was not enough.
• Dates after 2038 (Linux, BSD, macOS): x restored them wrong.
• Long paths on Windows: no more silent skips, and -longpath stores relative names.
• testbackup says OK only if the backup can really be read.
• New: -thunderbird: the mail of every user, from one PC to another.
• For packagers: -DBUILD_FEDORA (no cryptography), -DNOAES, -DNOUPDATE;
  -DNOFRANZEN builds again. The Makefile knows them (make BUILD_FEDORA=yes),
  and no longer gets wrong the byte order (ppc64le, s390x, sparc) or the target of a
  cross compilation. Every -D was built with every other: four did not go, now they do.

Part one, what is new, for the user

1. A full disk during a
2. The archive is never added to itself
3. An empty name is not the root of the disk
4. Exit codes that tell the truth
5. Crashes
6. No console, no waiting
7. xc and big blocks (issue 357)
8. Dates after 2038
9. Long paths on Windows
10. testbackup, as strict as t
11. -thunderbird
12. Builds for packagers (Fedora)

1. A full disk during a

Bugfix, with data loss. When the disk filled up during a, zpaqfranz said so
(01295 Something STRANGE happened... Media full?) and gave exit code 2. Right.
But the version written half way stayed in the archive: data without the
tables that describe them. From then on:

• t and i said DAMAGED, for ever (and trim said "not necessary");
• the next a, with the space back, wrote a new version that x could not
  extract (0 files).

In short: after the first full disk, every following backup into that archive
was lost. A kill -9 in the middle of a does not do this (the next run
overwrites the interrupted one, as zpaq always did: tried 20 times, the archive
always good).

Now. zpaq only appends: the bytes written before the run are untouched. If
the writes of a do not all go, the archive is cut back to the size it had
before the run, or removed if the run created it:

66858! <<z:/backup.zpaq>> put back as it was before this run (...bytes): nothing of this run is stored, the next run goes on from there

Exit code 2, as before; but the archive is the one of yesterday, good, and the
next backup works. If it cannot be put back, 66859 says so (then crop it).

This is done only where it is sure: a plain or -key archive, or a new part of a
multipart one. With -chunk, -index and Franzen the archive is not touched,
and 66860 says to check it with t.

2. The archive is never added to itself

zpaqfranz a z:\nas\docs.zpaq z:\nas\

The archive is inside the folder being archived. The second run stored the
archive of the first one, inside itself; and so did every run after it.

Now the archive being written, its .franzen, its -index file and the parts of
a multipart archive (x_???.zpaq) are never added. A line says so:

66862: <<z:/nas/docs.zpaq>> is the archive being written: not added to itself

3. An empty name is not the root of the disk

"" + / is /: the root. A script with an empty variable:

zpaqfranz d "$FOLDER" -force          (FOLDER empty)

was d / -force: the duplicates of the whole disk, deleted (read in the
code; not tried, for obvious reasons). sum "" read the whole disk.

Now an empty argument, as a file or as a folder, is refused by every command,
with exit code 2:

66841! An empty argument ("", an empty variable?) is not a file nor a folder: nothing done

The one exception is the archive to write: zpaqfranz a "" *.cpp
compresses without writing anything (to see how much it would take), as
always. And -to with no value (as in zpaq 7.15).

In the same family:

• a x.zpaq with nothing to add (or a ""): it printed the help, exit code 0.
  A backup script with an empty variable "succeeded" doing nothing. Now 66840,
  exit code 2.
• i "" (an empty table) and pakka "": exit code 2 (66842, 66843).

4. Exit codes that tell the truth

The rule is the one of t: 0 only if everything is really OK, 1 a warning
(something you can recover from), 2 an error. Several cases did not follow it:

A script that checks the exit code can now trust it in these cases too.

5. Crashes

On the arguments tried (several thousand command lines, see
part three), zpaqfranz no longer crashes. Fixed:

• s with no folder: segmentation fault.
• -until -1 (count back from the last version) on an archive that does not
  exist, and a wrong date in -until: an unhandled exception, abort, core dump.
  Now 66846 and 66845, exit code 2.
• -ramsize 0 (or -1, or with no value; -ram too): a division by zero in
  x (FreeBSD: "Floating point exception"). Now refused, 66844.
• -until 99999999999999999999: an overflow, a version at random. Now
  refused, 66847.
• A damaged block header asking for an absurd amount of RAM (2^77 bytes): an
  undefined conversion, which dirtied the count of the RAM. Now capped.
• hash: some fields were not initialized (undefined behaviour: work it did
  not need, for each file).
• MD5 and SHA-3 read their input as 32 or 64 bit words even when it was not
  aligned. On x86, nothing. On real ARM hardware it can be a crash (SIGBUS) or,
  on ARMv5, a wrong hash with no word. Under emulation the hashes were right, so
  I cannot say it ever happened on a NAS; anyway, now the data is copied first.

6. No console, no waiting

With the input that is not a console (a script, ssh, a pipe, a scheduled task):

• ls (Windows) waited for a command at its prompt for ever. Now it reads
  the commands from the input, and ends with it, as on Linux.
• tui and gui (Windows): the same. Now they end.
• pause waited for a key that could never come. Now it is skipped (a line
  says so) and exit code 0. pause -n 10 still waits 10 seconds.

From a console, everything is as before.

7. xc and big blocks (issue 357)

xc of an ac -m86h20 backup (64 MB blocks) restored a disk image of 49 GB that
was 247,720 bytes short, while tw said all OK. A good backup, a bad restore.

The cause: for xc (and for the restore of image), the cache of the
decompressed blocks could take all the free RAM. With 64 MB blocks the
threads that decompress found no memory left (bad_alloc), the block came out
empty, and its fragments were skipped without a word.

Now:

• a thread with no memory halves the cache and tries again, up to 4 times
  (09190, yellow: what comes out is good);
• if it really cannot: 09189 and 03877, which says that this is not a
  damaged archive (try -ramsize 4GB or -t2);
• the fragments that did not come out are counted, never skipped: 03876, exit
  code 2. Never a short file with no word;
• the automatic cache is the free RAM, less four of the biggest blocks for each
  thread, less 1 GB; never below a quarter of the free RAM.

On a 4 GB virtual machine, xc of a 5 GB disk (a 2 GB -m86h1 backup): all OK,
in about 12 minutes. For an older zpaqfranz: xc ... -ramsize 8GB.

8. Dates after 2038

x on Linux, the BSDs and macOS restored a file dated after 2038-01-19 with a
wrong date: 2040-06-01 became 1904-04-26, 2100 became 1964. The days were
multiplied by 86,400 in a 32-bit integer (code from zpaq 7.15). Now 64 bit.

9. Long paths on Windows

• Without -longpath, a file whose path is over 260 characters was skipped
  with a yellow line, and exit code 0: a script believed every file saved. Now
  each folder is listed (66861) and the exit code is 1.
• With -longpath, the archive stored absolute paths (W:/work/...) even
  for a folder given as relative (lunghi). Then t x.zpaq lunghi -longpath
  compared relative with absolute: every file "missing", exit code 2 on a good
  archive; x -to out put the files under out/W/work/.... Now:
  • relative stays relative, as without -longpath;
  • t ... -longpath scans the folder in the extended form (\\?\);
  • x -longpath without -to writes in the extended form (the long files
    failed before).

10. testbackup, as strict as t

testbackup said OK on a backup set that t and x cannot read:

• a part on the disk that is not in the list of the backup (a backup interrupted
  after writing the part, before the list): t said "cannot read_archive...
  THIS IS VERY BAD", x ERROR;
• the index (_backup.index) missing: l, t, x said "Archive not found".

Now, as t, it says OK only if everything is right, always (not only with
-paranoid): 66853 the index is missing, 66854 the index cannot be read whole,
66855 a part not in the list. Exit code 2.

An encrypted index is read only if its password is given (-key, -franzen);
without it, only its presence is checked (66857). And the message 02781 showed,
as "expected", the MD5 of the list instead of the hash it was compared with.

11. -thunderbird

Windows (not in the open build, as before). The mail of Thunderbird, from a
PC to another one, with nothing to set up again.

zpaqfranz a z:\mail.zpaq -thunderbird                   the user running zpaqfranz
zpaqfranz a z:\mail.zpaq c:\users\franco -thunderbird   the user of that profile folder (as before)
zpaqfranz a z:\mail.zpaq -thunderbird -all              every user of the PC (an elevated administrator)
          -kill      close Thunderbird first (open, its files change while they are read)
          -frugal    leave out what Thunderbird makes again (caches, crash reports, locks, *.tmp)

zpaqfranz x z:\mail.zpaq -thunderbird                   back to the users with the same name
zpaqfranz x z:\mail.zpaq -thunderbird -to caio          into another user (one user in the archive)

The typical use: on the old PC a -thunderbird; on the new one, a clean install
of Thunderbird, then x -thunderbird.

a takes AppData\Roaming\Thunderbird (the profiles, the mail) and
AppData\Local\Thunderbird, and the profiles of profiles.ini with an
absolute path (anywhere: without them, the backup is not the whole mail).

• -all: the users come from the registry (the real accounts), compared with
  the folders of C:\Users (a yellow line if they differ).
• A user whose mail cannot be read is skipped and said (66886), with a summary
  at the end (66876) and exit code 1.
• -frugal leaves out only the entries of one table, and only at the top of
  the Thunderbird folders and of the profiles: a mail folder called crashes
  or x.tmp is mail, and stays.
• Thunderbird open, without -kill: a yellow line (66891).

x -thunderbird never destroys what is there:

1. the users are taken from the paths in the archive;
2. each one goes to the user with the same name on this PC (or to -to);
   a user that is not on this PC is skipped and said, exit code 1;
3. what is there (the folders of the clean install) is renamed
   Thunderbird.001 (the first number free) and checked. If one rename fails,
   the ones done are put back, nothing is extracted, exit code 2;
4. the mail is extracted;
5. into another user: the paths of the old one, in prefs.js and
   profiles.ini, are corrected (a .bak is kept).

An archive with errors is not restored (66898: t to see them).

The mail of the other users. With the permissions Windows gives by default,
the Administrators group can read C:\Users\x:

• an elevated administrator reads the mail of the others: -all works;
• a non-elevated administrator, or a standard user, cannot: those users are
  skipped, and said;
• a user who denied access to the administrators: closed, elevated or not.

zpaqfranz never takes the ownership of a file and never changes a permission.

The old -thunderbird (which took the folder of one user) had some bugs:

• with no Thunderbird folder it said "quit!" and then archived the folder given:
  the whole c:\users\x;
• it wanted both Local and Roaming: with Roaming only, the mail was not
  taken;
• the profiles with an absolute path were left out;
• -kill wrote a .bat in the temporary folder (a user name with an accent:
  it failed), did not wait for Thunderbird to close and did not check;
• with -longpath the long paths of ImapMail were skipped;
• the paths were stored in lowercase.

12. Builds for packagers (Fedora)

zpaqfranz is packaged by Fedora. For that package (and for whoever wants the
same) there is now a build with no cryptography of its own in the binary:
-DBUILD_FEDORA, the sum of four switches, each one usable alone.

What stays, on purpose: the hashes (SHA-1 is the zpaq format; the others check
the integrity of the files, they do not encrypt anything) and the random numbers
of the system. The archives in clear are the same as in any other build. Its
banner says FEDORA, its usage "Archiver w/dedup, no encryption".

What is left out says so, and stops (exit code 2): an encrypted archive (03870),
-key (03871), update (03875). It never asks for a password a script would
wait for.

Careful: a BUILD_FEDORA zpaqfranz cannot read an archive encrypted with
-key. Read it with any other build.

The Makefile (build 1.8)

The Fedora package does not use my Makefile: "It does not recognize s390x, strips
executables". They were right, and it was worse than that:

• The byte order was guessed from the name of the machine (uname -m):
  ppc64le and mipsel got -DBIG, but they are little endian; s390x and sparc did
  not, and they are big endian. -DBIG decides the byte order of SHA-1 (the
  deduplication), SHA-2, SHA-3, xxHash, CRC-32C: wrong means wrong hashes. Now the
  compiler says it (__BYTE_ORDER__).
• CROSS_COMPILE took the flags of the machine that compiles, not of the
  target: an aarch64 build got the SHA instructions of x86 (-DHWSHA2), and with
  clang++ installed it looked for an aarch64-linux-gnu-clang++ that almost never
  exists. Now the target is the one of the compiler (-dumpmachine), and the cross
  compiler is g++.
• -DANCIENT only for the old big endian PowerPC (Macs G3, G4, G5), no longer for
  ppc64le (POWER8 and later).

New options:

make BUILD_FEDORA=yes         no AES, Franzen, e-mail, update, SFTP
make ENABLE_AES=no            or one at a time: ENABLE_FRANZEN=no ENABLE_EMAIL=no ENABLE_UPDATE=no
make ENABLE_IPV6=yes          do not force IPv4
make install ENABLE_STRIP=no ENABLE_DIRLINK=no PREFIX=/usr DESTDIR=...
                              for a package: the symbols stay (debuginfo), and no "dir"
                              link (on Linux /usr/bin/dir belongs to coreutils)

make check shows the target, the byte order and every option.

Every -D with every other

A switch that nobody compiles rots (-DNOFRANZEN did not build in 65.19). So the
-D switches were built together: not all the 2^n combinations, but enough builds
that every three of them are together in at least one (a covering array). On
Windows (the full source, the open one, 32 bit and XP), FreeBSD (clang) and Linux
(gcc, with the profiles ANCIENT, NAS, ESX too). Four did not go, all fixed:

• -DALIGNMALLOC crashed at once on every 64 bit system: an alignment of 4
  bytes, smaller than a pointer, and every allocation was refused. It is the one
  the Makefile gives to sparc, which today is sparc64. Now 8.
• -DIPV6 did not build on Windows (it is for the http client of *nix: Windows
  ignores it now).
• -DOPEN did not build on Windows without -DNOFRANZEN (libsodium stayed in,
  and called a piece of the full build).
• -DANCIENT -DZPAQMOUNT without Mbed TLS (-DNOEMAIL, BUILD_FEDORA) did not
  build on FreeBSD (signal.h).

The comment at the top of the source now lists every -D, the five above
included; -DSERVER and -DGUI, which no longer did anything, are gone.

Also: the Twofish implementation (for a client-server version never released) is
no longer in the source, and gcc 16 in C++20 builds it with no warnings (++ on
a volatile).

Part two, the details, for power users

1. What can change for a script

• Exit codes: the table of point 4.
  Where a script got 0, it can now get 1 or 2.
• "" as a file or a folder: refused, exit code 2, for every command
  (66841). The archive to write can still be "".
• a x.zpaq alone: exit code 2.
• -ramsize 0, -ram 0, negative, or with no value: exit code 2 (66844).
• -until with more than 19 digits: exit code 2 (66847).
• pause with no console: it does not wait (pause -n X does).
• ls, tui, gui on Windows with a pipe or a file as input: they read it,
  and end with it.
• a on a full disk: the archive is put back as it was (exit code 2, as
  before).
• a into a folder that contains the archive: the archive is not added.
• testbackup: exit code 2 if the index is missing or damaged, or if a part
  is not in the list.
• a -thunderbird: with no folder, the user running zpaqfranz (before: help
  and nothing done); with -all, every user; a user skipped = exit code 1.
• The Makefile on ppc64le and mipsel: no more -DBIG (and on ppc64le no
  more -DANCIENT): another binary, the right one. On s390x and sparc: -DBIG.
  make install strips and makes the dir link as before, unless
  ENABLE_STRIP=no / ENABLE_DIRLINK=no.
• -DALIGNMALLOC: the banner says .8 (was .4).

2. The switches

3. The messages

Part three, for developers

1. How it was tested

On virtual machines (Windows 11, Fedora, FreeBSD, the Debian that builds the NAS
binaries), with builds that watch every access to memory: gcc and clang with
AddressSanitizer + UndefinedBehaviorSanitizer + assertions, MinGW with
assertions on Windows.

• Degenerate arguments. Every command with: nothing, "", a path that does
  not exist, a text file, a folder, an empty archive, a text file called
  .zpaq, an archive cut in half, a good archive, a good archive and a path that
  does not exist: 660 command lines on Linux and FreeBSD, 336 on Windows (plus
  the "dangerous" commands by hand). Then every switch that takes a value, with
  no value, "", 0, -1, 20 digits, a path that does not exist: about 2,100
  command lines on Linux and FreeBSD, 972 on Windows. Then paths up to 411
  characters and names in Japanese, Greek, emoji.
• ARM. The NAS binaries (ARMv5, ARMv7 Cortex-A9, AArch64 Cortex-A53), under
  qemu, 660 command lines each.
• Hostile archives. 44,047 damaged copies of 10 kinds of archive (Franzen
  too), one in ten tried with t tw pp p x, then with 10 more commands. An
  archive of 4,347 bytes cut at every offset, and each byte XOR FF and zeroed:
  12,640 copies, each one with l t x i.
• Resources. ThreadSanitizer on 18 commands: no race on the data (only on
  counters for the statistics and the bars, estimates on purpose). Little RAM
  (ulimit -v from 64 to 512 MB), a full disk (a RAM disk of 64 and 128 MB), 20
  kill -9 during a: after each one the archive was good, and the next a worked.
• Against 65.19, on the same command lines: 65.19 had 5 crashes, 3 hangs, 5
  "error printed, exit code 0"; 65.20 has none (the only "hang" left is b, the
  benchmark, which is long on purpose). Every verdict that changed went towards
  OK; every exit code that changed is one of the fixes above. No regression.
• Against zpaq 7.15: 7.15 lists and extracts archives of 65.20 made with
  -m0 to -m9, -key, -nochecksum, byte for byte; 65.20 reads the archives
  of 7.15. 154 checks.
• Regressions. The batteries of the earlier releases (278 checks on Windows,
  232 on Fedora, 225 on FreeBSD, and 17 torture batteries); autotest -all with
  the sanitizers: nothing.
• Builds. Every -D with every other, as a covering array (every three
  switches together in at least one build): on Windows 26 builds of the full
  source, 9 of the open one, 7 at 32 bit and for XP; on FreeBSD (clang) 24 and 9;
  on Linux (gcc) 38 syntax checks, with ANCIENT, NAS, ESX. Every one that
  runs here: autotest, a/e/cmp, -key and -franzen refused or not as
  they should, update with NOUPDATE, the banner. After the four fixes: no KO.
  -DALIGNMALLOC: autotest -all on FreeBSD and Linux.
• The Makefile, for real, on Oracle Linux 7.9 (make 3.82, gcc 4.8.5), Arch,
  openSUSE, macOS (make 3.81, Apple clang), Solaris 11, OpenBSD, Alpine (musl),
  Fedora and FreeBSD: the default build, BUILD_FEDORA with the mount, install
  for a package and as always, static, a cross compilation.
• Then the release binaries on every system (see the end).

2. Left as they are, on purpose

I decided to keep these as they are:

• An unknown switch (a typo, or a switch with no value) is ignored, with
  00562! Unknown option ignored, and the command goes on: exit code 0.
  Careful with -excldue instead of -exclude.
• A wrong -datefrom / -dateto (a year out of 1970-2070, an odd length):
  the filter is dropped, with a message, and the command works on everything.
• mount x.vhdx -test of a disk restored by xc: the partitions that were
  not in the backup are there, empty (RAW), and Windows gives them a letter that
  cannot be read: exit code 2.
• pakka on a damaged archive: exit code 0. It is for the GUI: it lists
  every version of every file, in sequence.
• password ... -key2 . (an encrypted archive made plain): t of the new
  archive gives a WARNING (03739, the -lfast history is 32 bytes off: the salt
  is gone). The data are all there (l lists them all). A remote case.

3. Known limits

• -thunderbird was tested on Windows 11 with six users (accents, a space, ü,
  one who denied access to the administrators) and profiles with the structure
  of real ones, made by a script: 94 checks. Not with Thunderbird itself
  installed on that machine. No Win Server etc.
• The restore of an -image archive (not xc): a block that really cannot be
  decompressed (a damaged archive, not the lack of memory) is still skipped with
  no word.
• A few damaged archives (bytes changed inside the post-processor of -m1 or
  -m2) can make t and x loop.
• l shows 2100-05-31 for a file of 2100-06-01 (the leap year rule of zpaq: from
  2100 on).
• The counters of the RAM and of the bars are not atomic: with many threads they
  are an estimate.
• -DALIGNMALLOC was tried on x86-64 only: I have no sparc. Or, better, I do have a PPC Mac but too lazy to reconnect. With gcc -O3 it
  gives 15 warnings -Wfree-nonheap-object, all on the same branch of
  aligned_malloc(), one that cannot run.
• make static needs the static C library of the system (on openSUSE
  glibc-devel-static, not installed by default).

4. The binaries

Built from the same source and tried on each system (autotest, then quick
checks of repack, w, tw, -tar and, where there is a handler, Ctrl+C in
the middle of a):

Alpine (musl) too, as a check only: no binary. The warnings of the compilers
are the known ones (most of them in code by others: Mbed TLS, libzpaq), none
new. As always, the pairs A53 = A55 and A57 = A72 are the same file: gcc makes
the same code for them (see NAS.md).