zpaqfranz 65.6

From 65.5g to 65.6i.

BEWARE: A LOT OF NEWS == A LOT OF NEW BUGS TOO!!! BE CAREFUL, NOT VERY TESTED!

Seven headlines:

• New defaults: a is now -m8 -turbo -fast (zstd, the parallel
  fragmenter, the history for a fast l: -fast is the -filelist of
  65.4 and 65.5, renamed). On Windows x writes big files as sparse files.
  -715, -noturbo, -nofast, -nosparse bring the old behaviour back.
• f -test: a surface test of a whole device, read only, with a live
  map and a verdict in words. With -force -paranoid the "triello"
  (Windows): it writes the whole device, reads it back and compares, and
  finds fake capacity, lost writes and wrong data.
• A dashboard in the console (franzdash): the live map of f -test,
  and of a -image while an image is taken (by default; -nodashboard or
  -noeta for the classic line).
• Images that go on: a sector that cannot be read becomes zeros, the
  image goes on, and at the end a giant READ ERROR with the exit code 2.
• mount -range A:B: only what versions A..B added or updated, one
  folder each.
• work email: an SMTP client with TLS inside zpaqfranz (for the
  e-mail reports of the next versions; today a command of its own), and
  work zip.
• Damaged archives are said, not read wrong: missing blocks, a broken
  block name, an index block lost at the end of a version (72401-72406,
  72414); and a -chunk no longer loses the end of the archive on Linux.

This document has three parts:

• Part one: what is new, in short, for everybody
• Part two: the same things in more detail, for power users
• Part three: "lo spiegone", how it works inside, and why, for developers

Part one, what is new, for the user

1. New defaults
2. f -test: is this disk (or stick) good?
3. The triello: write, read back, compare
4. a -image: the dashboard
5. Images of a damaged source
6. mount -range: only the changes
7. work email and work zip
8. Exit codes you can trust
9. Smaller things

1. New defaults

zpaqfranz a z:\backup.zpaq c:\data              (now: -m8 -turbo -fast)
zpaqfranz a z:\backup.zpaq c:\data -715         (as zpaq 7.15: -m1, no -turbo, no -fast)
zpaqfranz a z:\backup.zpaq c:\data -m1 -noturbo -nofast   (as before 65.6)

The archive made by -turbo is the same, to the byte, of the one made
without it: only faster. -m8 is smaller than -m1 and, with -turbo,
faster; any zpaq extracts it (zpaq 7.15 too, through the decoder stored in
the archive, at 80-95 MB/s). The builds for old machines (ESX, NAS)
keep the old defaults.

-filelist and -nofilelist are now -fast and -nofast. The old names
still work (the scripts written for 65.4 and 65.5 do not change), and the
history stored inside the archives is the same: an archive made by 65.5 is
listed fast by 65.6, and the other way round.

2. f -test: is this disk (or stick) good?

zpaqfranz f E: -test
zpaqfranz f 3 -test -buffer 4194304
zpaqfranz f b -test                   (Linux: /dev/sdb)

Reads the whole device, raw, block by block (1 MB by default), as HD Tune
does: nothing is written. While it runs a map shows every stretch of the
device (green ok, yellow slow, red stall or error, grey still to do); at the
end the speed band by band, the median, the slowest and fastest 5%, the
slow blocks (read again: slow again means the device, fast means a hiccup),
the stalls, the I/O errors, and a conclusion in words:

VERDICT: GOOD: the device is practically perfect           exit code 0
VERDICT: SUSPICIOUS: it works, with mild anomalies         exit code 1
VERDICT: BAD: serious failure, copy the data elsewhere now exit code 2

Ctrl+C stops it and shows the report of what was done (a second Ctrl+C
quits).

Virtual disks: the speed of a virtual disk depends on the host and its
cache. On the test VM the same 8 GB disk was "GOOD" read with 4 MB blocks
and "BAD, it may be faulty" read with 1 MB blocks (the speed jumped by 44%
between near zones). The verdict is meant for real devices.

3. The triello: write, read back, compare

zpaqfranz f E: -test -force -paranoid           (DESTROYS EVERYTHING on E:)
zpaqfranz f 3 -test -force -paranoid -quick     (write only)

Windows only. Every block of the device is written with data never
repeated, then read back and compared. It finds what a read test cannot:
fake capacity (the addresses wrap around), writes that went nowhere, wrong
data, a write cache that hides a slow device. It asks the captcha
nomercy (even with -nocaptcha), works only on USB devices, never on the
disk that holds Windows, and leaves the device without partitions:
initialize it again (Disk Management, or diskpart).

On the test VM, the 8 GB virtual disk: written at 1.3 GB/s, read back at
1.8 GB/s, every byte right, 11.5 seconds.

4. a -image: the dashboard

zpaqfranz a z:\c.zpaq c: -image -ntfs -vss
zpaqfranz a /tmp/sdb.zpaq /dev/sdb -image
zpaqfranz a /tmp/sdb.zpaq /dev/sdb -image -nodashboard     (the classic line)

The map of the source while the image is taken, by default (it was asked
with -dashboard, which is still accepted). With -ntfs it is known
from the start which blocks are free (not read), excluded (the pagefile,
-not) and to read. A block read is green when its data were new
(compressed into the archive) and blue when they were already there
(deduplicated): on the second image of the same disk, blue is what did not
change. Slow and stalled reads as in f -test, E where the source could
not be read.

With the output redirected there is no live map: the map is written as text
at the end (in a log, some 20-30 lines). Not with -verbose or -debug
(their lines would break it). -nodashboard or -noeta give the classic
progress line (and no map in the log).

5. Images of a damaged source

A sector that cannot be read no longer stops or blocks a -image: it could
loop forever on the same error (on Linux adding the block read before,
again and again). Now:

• what cannot be read becomes zeros, and the image goes on;
• the first error is said at once, on a line of its own;
• at the end a giant READ ERROR, how many sectors, where, and the exit
  code 2 (also with -ignore): the image is complete, but it is not a copy
  of the source;
• a source that goes away (unplugged) gives zeros up to the end, without
  trying every sector;
• an internal error of the reader stops everything and the archive goes
  back to the version before.

6. mount -range: only the changes

zpaqfranz mount z:\backup.zpaq -range 2:2
zpaqfranz mount z:\backup.zpaq -range 5:9 z:\m

Mounts the versions A..B (numbered as l numbers them), one folder each
(VER00000001...), and each folder holds only the files that version
added or updated, the same thing l -range and x -all -range show.
-all alone mounts every version as a full snapshot, as before. A range
with no version in it is refused (exit code 2).

7. work email and work zip

zpaqfranz work email -mailconfig c:\zpaqfranz\email.conf -mailto you@example.com -mailsubject "backup OK" -mailzip z:\backup.log
zpaqfranz work zip z:\backup.log z:\backup.zip

zpaqfranz can send an e-mail by itself: SMTP with TLS (STARTTLS on port 587
or implicit TLS on 465), login, attachments as they are or zipped. No
external program, no DLL: the TLS client is inside. Today it is a command
of its own; the e-mail report at the end of a backup comes in a next
version. The exit code is 0 when the server accepted the message, 2 when
not (and the reason is written).

work zip makes a .zip of one file (deflate), the kind that opens with a
double click: handy to mail a log.

8. Exit codes you can trust

• A fatal error (an archive that does not exist, a wrong parameter, a
  device that cannot be opened...) used to end with exit code 0: a
  script took it for a success. Now it is 2. The help, dir, q in
  the pager are still 0.
• Commands that counted errors (x, t on many archives, a -home) could
  return 256 errors as exit code 0. Now the exit code is 0, 1 or 2: the
  worst of the parts, not their sum (two warnings are a warning).
• A damaged archive gives an exit code other than 0 (see the next section
  and part three): missing blocks are an error, the last index block of a
  version lost is a warning (72414, exit code 1).

9. Smaller things

• *a -chunk on Linux, BSD, macOS: when the archive ended a few KB past
  a multiple of 128 KB, the last part was deleted as empty and the archive
  could not be read, with exit code 0. On Windows an empty last part stayed
  on disk. Both fixed.
• x -stdout of a damaged archive: its messages went to stdout, into the
  data; now they go to stderr, and the data are clean.
• A bit flipped in the name of an index block: x extracted a junk file
  named jDC...i0000000001 and lost the last version, with exit code 0;
  now the block is skipped as damaged, with a message and exit code 1.
• x without -force: a file already on disk with a different size (an
  extraction killed halfway) is no longer skipped in silence: a warning for
  each one and exit code 1.
• Windows: names with characters outside the BMP (emoji, rare ideographs)
  are now stored as real UTF-8, so they come back right on Linux and macOS.
• Windows, -longpath with a relative -to works (it produced unusable
  paths); the open build also suggests -longpath when a name is too long.
• Windows, restoring a raw image of a partition: it failed (exit code 2) on
  the last block, while the data were already right.
• Windows, a -vss with -not or -only: they were ignored (and -only
  gave an empty backup).
• Windows, a symbolic link to a file with the same name as its target made
  the link, and everything after it in the folder, vanish from the backup.
• utf -kill renames the files even when no folder has a UTF-8 name.
• h something > file no longer waits for a key that never comes.
• l -csv and -csvhf with a % in them crashed.
• x -ramdisk: a file that could not be written stopped the writing of all
  the others (they were only in RAM); now it goes on, exit code 2.
• The help of zip -range says where to find the "only the changes"
  (x -all -range, mount -range): zip -all -range keeps making full
  snapshots.
• A batch of fixes of issues reported on GitHub, and of what a review of
  a, x, l, f and the images found (see part three).

Part two, the details, for power users

1. The defaults, and how to go back
2. f -test in practice
3. The triello in practice
4. The dashboard
5. READ ERROR
6. mount -range
7. work email
8. What can change for a script
9. Backup and restore of a whole disk, measured

1. The defaults, and how to go back

• -append never stores the history (its first pass writes nothing): the
  next plain a stores it again.
• x on Windows: a file is made sparse when it was sparse at the origin
  (as zpaq 7.15 does) or when its size, known before writing it, is 16 MB
  or more. Its all-zeros fragments are not written and become holes. The
  smaller files are written as before (tiny files stay in the MFT). On a
  filesystem without sparse files (FAT32, exFAT) the file is a normal one,
  not an error. The archives themselves are no longer created sparse.
• Measured on the VM: a 24 MB file with 23 MB of zeros came out sparse with
  the zeros not allocated; with -nosparse a normal file; the same SHA-256.

2. f -test in practice

The device: a letter or a disk number on Windows (see drive), b for
/dev/sdb or a full /dev/... path on *nix. Reading a device is allowed
on the system disk too.

How a block is judged (the first 3 seconds, the warm-up, are on the map but
not in the statistics):

Then the surface is split into 32 zones, and near zones are compared (a
jump of speed between them), as the spread of the blocks inside a zone.
The report lists the anomalies, the worst first.

3. The triello in practice

• Writing is judged with wider limits (the stops of the flash to clean up
  are normal): slow when 50 ms are lost, stall at 0.5 s.
• The write cache: a moving median under 60% of the first plateau means
  the cache is over; the report says how much slower the device is after
  it.
• Reading back, a block can be: right; wrong in some bits; all zeros; all
  0xFF (never written flash); old data (the write went nowhere); the data
  of another block of this run (the addresses wrap: fake capacity). A
  block wrong once and right when read again is "unstable".
• Refused: a device that is not USB (-debug lifts this, for tests on
  virtual machines: never the system disk), the disk that holds Windows
  (always), a wrong captcha (exit code 1, nothing written).

4. The dashboard

Only the cells that change are drawn again (4 times a second at most). With
-nocolor or the output redirected, letters instead of colors; redirected,
the map comes at the end as text. For a -image the map is the default:
-nodashboard and -noeta turn it off (also when -dashboard is given),
-verbose and -debug too.

5. READ ERROR

69100! READ ERROR at 20.971.520 of the source: zeros in the image, going on
69110! READ ERROR: 1.024 sectors of the source could not be read
69111! The image has 524.288 bytes (512.00 KB) of ZEROS instead of the data, in 1 stretch
69112! Unreadable from offset 20.971.520 for 524.288 bytes
69115! THIS IMAGE MAY BE CORRUPTED: do not trust this version of the archive
69116! Check the source with a surface test: zpaqfranz f /dev/mapper/bad -test

(the first 16 stretches are listed, then how many more). A read that fails
is tried again 64 KB at a time, then sector by sector for the 64 KB that
fail; 1 MB in a row with nothing readable is a dead zone, not tried sector
by sector until a read works again.

Measured on Fedora with device-mapper: a 64 MB device with 200 unreadable
sectors (100 KB) at 20 MB: exit code 2, the image complete, identical to the
source outside the zeros. On Linux the zeros are 512 KB, not 100 KB (see
part three: the reads go through the cache of the kernel).

6. mount -range

• -range A:B implies one folder per version; -until is refused with it
  (as for l and x).
• The folders keep the numbering of the mount: version 2 is VER00000001.
• In each folder, only the files whose last change is that version (added
  or updated, not deleted). A path typed by hand of a file not changed in
  that version does not exist, even if it exists in the version.
• A range beyond the last version is cut to it; a range with no version is
  refused: 94033! -range 4:4: no such version, the archive has 3.
• mount -test knows -range (it checks the files of the last version of
  the range). Tested on Linux (FUSE) and on Windows (WinFsp).

7. work email

The password is better in the -mailconfig file than on the command line.
-verbose shows the log, -debug the SMTP dialogue.

8. What can change for a script

9. Backup and restore of a whole disk, measured

On the Windows 11 VM, the second virtual disk (8 GB, NTFS, E:):

Part three, lo spiegone, for developers

1. Two branches, one source
2. The image reader that goes on
3. franzdash
4. f -test: numbers, not impressions
5. mount -range
6. Names outside the BMP
7. The e-mail client
8. Fixes
9. Known limits and open points

1. Two branches, one source

65.6 joins two lines of work that started from the same 65.6a (the one
with the e-mail client): the fixes of the GitHub issues and of a review of
a, x, l, f (on one side), the new f -test, franzdash, the image
reader and the new defaults (on the other). The three-way merge had 10
conflicts, all small (the two sides had fixed the same printf formats and
the same pager in the same way). Built and tested on Windows (full, open,
mount), Fedora (plain, -DSFTP, -DZPAQMOUNT) and FreeBSD (clang):
autotest 16/16 everywhere.

The source: 188,125 lines in 65.5g, 227,767 now (8.2 MB). 33,500 of them are
the TLS library of the e-mail client.

2. The image reader that goes on

• img_leggi (Windows) and img_leggiunix (*nix, pread at the offset
  every time: after an error a handle does not know where it is): the whole
  buffer at once; if it fails, 64 KB at a time; for the 64 KB that fail,
  sector by sector (the sector size from the device: IOCTL_DISK_GET_DRIVE_GEOMETRY_EX,
  BLKSSZGET). What cannot be read is zeros, and img_errore counts the
  sectors and the stretches (the first 16 remembered).
• A dead zone: 16 pieces of 64 KB in a row with nothing are not tried
  sector by sector (a dead area of a disk would take hours) until a read
  works again. A source gone (ERROR_DEVICE_NOT_CONNECTED, ENODEV...):
  zeros to the end, no more reads.
• The loop of add() and add2(): a reader that returns an error code
  (not data) would be called again forever on the same block. Now
  imagefatale() stops, through the same housekeeping of Ctrl+C (the
  archive back to the last complete version, a new one removed, the VSS
  deleted), exit code 2 (g_fermatafatale).
• imagebanner() at the end of add(): the giant READ ERROR; errors= 2
  even with -ignore.
• The Linux DD reader also got its progress (it stayed at 0).

3. franzdash

A class that knows nothing of disks: a title, two lines of information, a
map of cells, a legend. The client declares its "looks" (a letter, a color,
a square or a UTF-8 glyph, a name for the legend) and sets the look of each
cell; the dashboard draws once, then only the cells that changed:
relative moves of the cursor on *nix, the rows of the console buffer on
Windows. Redirected, or with -nocolor: letters, and the map as text at
the end. The cursor is hidden while it draws and shown again by
my_handler too (Ctrl+C). Two clients: franzusbmappa (f -test: a cell
shows the worst state of its blocks) and franzimgdash (a -image: the
blocks of the source; new or deduplicated is decided by the fragments of
the block, imgdash.frammento() called by the fragmenter with htptr == 0).
In the ANCIENT builds a stub: the classic progress line.
flagdashboard is on by default (nuovodefault, as -turbo and -fast);
after the switches are read -nodashboard and -noeta turn it off.

4. f -test: numbers, not impressions

• Every block timed; the statistics after a warm-up of 3 s; median, p5,
  p95, coefficient of variation.
• 32 zones: the biggest jump of speed between near zones, how far a block
  typically is from the speed of its zone; the first and the last zone.
• Slow blocks are read again: slow again is the device, fast is a hiccup
  (of the host, of the bus).
• The triello: the data of each block are generated from its number and
  from a key of the run, so they are never repeated: a block read back
  with the data of another block of the same run means that the addresses
  wrap (a "32 GB" stick that is really 8 GB); data of an older run means
  the write went nowhere.
• Five levels, from "practically perfect" to "serious failure"; the exit
  code is 0 for the first two, 1 for "suspicious" (or a test stopped by
  Ctrl+C), 2 for "may be faulty" and "serious failure".

5. mount -range

The mount read the version range (-range sets all, the "one folder per
version" mode) but never used g_rangefrom/g_rangeto: it built the full
snapshot of every version. Now st.firstversion and st.nversions delimit
the versions exposed, and the jidac backend, with g_mountchangesonly,
builds the tree of a version from jidac_changed_in(): the last change of
the file is that version, and it is not a deletion. entry() uses the same
test, so a path typed directly is consistent with the directory listing.

6. Names outside the BMP

wtou() (UTF-16 to UTF-8, from zpaq 7.15) converted one wchar_t at a
time: a surrogate pair became two 3-byte sequences (CESU-8), which is not
UTF-8. Windows did not notice (utow() reads both forms back), Linux wrote
6 bytes of garbage, and utf8toansi() (MB_ERR_INVALID_CHARS) returned an
empty string. Now a pair becomes the real 4 bytes; a lone surrogate keeps
the 3-byte form; utf8toansi() turns every byte of a name that is not
UTF-8 into _ instead of returning nothing. A name with such characters
already stored in CESU-8 looks renamed at the next a (the old entry
deleted, the new one added, the data deduplicated).

7. The e-mail client

• postami 0.3.0, the SMTP client, and a subset of Mbed TLS 3.6.4 configured
  as a TLS 1.2 client (ECDHE-RSA/ECDSA, AES-GCM, ChaCha20-Poly1305, AES-CBC,
  X25519/P-256/P-384, X.509 with RSA and ECDSA), mechanically reduced
  (unused code removed, C99 turned into C++98).
• Windows: no link library added (ws2_32, crypt32, bcrypt are loaded
  at run time); the certificates from the system store unless -mailcafile.
• The command line interface of postami (its own parser, --help) is gone:
  work email takes the -mail... switches from loadparameters(), as
  every other switch. zpaqfranz_sendmail() is the entry point for the
  reports of the backups of the next versions (not called yet).
• -DNOEMAIL (set by -DOPEN) leaves it all out. Note: the "open"
  source made by rella.bat (the NOSFTPSTART/NOSFTPEND purge) still
  contains the e-mail client and the TLS library, because they are not
  inside those markers.

8. Fixes

GitHub issues:

• 300 utf -kill: the files were renamed only when some folder had a
  UTF-8 name (the crash of the same issue was fixed in 65.5).
• 314-319: a damaged archive is described instead of being read
  silently wrong: data without fragment tables, tables without index, holes
  in the numbering of the blocks (messages 72401-72406); a multipart with a
  missing part in the middle (72407), and a that would fill it refuses
  (72408); the last block damaged at its very end but read: a appends
  after it instead of destroying it (72410); *nix, files not archived
  because the path is too long (ENAMETOOLONG) counted as a warning (72411).
• 318 the -longpath hint was only in the full build.
• 320 a bit flipped in the jDC\x01 comment of a block sent it down the
  streaming path: a junk file, the last version lost, exit code 0. A full
  jDC block name there is now a damaged block, as a bad block name is.
• 323 missing blocks, the rest of it: an index block lost at the end of
  a version leaves no hole in the numbering. read_archive() now marks
  every fragment used by a file (all the records, before any selection) and
  the fragments of the -fast history (the ranges in the last field of its
  pointer); on a clean read (nothing skipped, no hole, no incomplete
  version, no streaming block) the fragments that nobody uses are the data
  of the files that were in the lost block: 72414, a warning, exit code 1
  (g_frammentiorfani, read by test(), extract() and the final
  summary). No false alarm on 86 real archives (up to 3,497 versions) and on
  archives made with every method and option; the reading time does not
  change. On x -stdout it goes to stderr.
• 321, 325 -longpath with a relative -to made //?/relative paths;
  the summary of long names suggests -longpath.
• 324 x over a partial file: warning and exit code 1.
• 326 mount -range.

From the review of a, x, l, f and of the images:

• a -external ran the external command twice when it failed.
• l -csv, -csvhf and sftp run used a string of the user (or of the
  server) as the format of printf: a % crashed.
• x -ramdisk: one file that could not be opened stopped the writing of
  all the others, exit code 0.
• Images with exclusions (NTFS): an excluded cluster beyond the end of a
  short block overflowed a memset; an exclusion on another drive letter
  was applied to the drive being imaged (now ignored, with a warning); a
  physical disk numbered 10 or more is refused (its image name has one
  character only).
• a -vss: -not, -only, -always were not moved onto the snapshot.
• Windows scan: a symbolic link to a file with the name of its target moved
  the whole rest of the folder under the target (out of the backup), and
  its size was read as 0 (a HOUSTON error).
• Restore of a raw partition image on Windows: the volume handle stops at
  the end of the filesystem, the last block was refused;
  FSCTL_ALLOW_EXTENDED_DASD_IO now.
• Exit codes: x/t on many archives and a -home summed the codes;
  every command is capped at 2.
• -chunk: an a with nothing new left an empty 104-byte chunk.
• -chunk: at the end of add() the header of the version is written again
  on the first part; that path of OutputArchive::flush() closed the first
  part and set the current one to none without closing it. On *nix its last
  bytes stayed in the stdio buffer, the part looked empty and was deleted
  (an unreadable archive, exit code 0, also in 65.5); on Windows the handle
  stayed open and an empty part could not be deleted (error 32). Now the
  current part is closed there.
• x -stdout: the archive was read before the silent mode started, so the
  messages of a damaged archive (72401-72406, "Skipping block") went to
  stdout, before the data. The silent mode now starts before the reading,
  and in it, with -stdout, errors and warnings go to stderr.
• a over a damaged tail says that it overwrites it (72413); -index and
  * names say that holes are not checked (72412).
• *nix: a folder named like a part of a multipart was taken for an archive;
  folders that cannot be read are reported as on Windows; the error of the
  wrong call was printed.
• l: a file named VCOMMENT plus a few characters crashed it.
• printf formats: %ld and %zu read with the wrong size on Windows;
  several messages with missing or wrong arguments.
• The captcha passed with stdin closed; h waited for a key with the output
  redirected; dir /os and the tree view sorted the other way round; the
  autotest -n of a small size wrote out of its buffer.
• The Makefile really prefers clang++ (CXX ?= did nothing with GNU make).

9. Known limits and open points

• Linux, READ ERROR: the retries sector by sector go through the page
  cache of the kernel, which reads in big pieces (large folios): around
  bad sectors good data can become zeros too (512 KB for 100 KB of bad
  sectors in the test), and the report counts them. Reading with O_DIRECT
  would fix it; not done yet.
• f -test on virtual disks: the verdict is not reliable (see part one).
• Export of an NTFS image to a VHD (image x.zpaq E: -to d:\3.vhd -ntfs):
  -to is a folder, the file inside is image_E.vhd (the help suggests a
  file name).
• An image of a physical disk made by number is extracted as a file,
  but image cannot write it back to a disk ("you need a drive letter").
• The imager writes with FILE_FLAG_NO_BUFFERING from buffers that are not
  explicitly aligned to the sector (it works on the tested disks).
• The Ctrl+C handler does its housekeeping inside the signal (not
  async-signal-safe): rarely, a Ctrl+C in the middle of an a can abort
  with a heap error (the archive stays at the last complete version).
• A version made only of deletions that loses its only index block is
  not seen without -fast: it is identical, byte by byte, to the empty
  version that a -force writes when nothing changed (zpaq 7.15 drops that
  one). With -fast (the default) the numbering of the blocks shows it.
• l with the -fast history does not read the archive: a damage is found
  by t, by x or by l -nofast.