zpaqfranz 65.8b

This is the release of the backup that looks after itself. It tells by
e-mail how it went, it goes to the cloud and says OK or ERROR, it takes
what can still be read of a dying disk. And the images got thinner, and
found one more way out:

• The report by e-mail, sent by zpaqfranz: -mailfull and
  -mailprivacy at the end of any command, on every platform. Two logs:
  the one as it is for the owner of the data, the one without the names of
  the files for who looks after the backup. No external program any more.
• cloud: every phase is OK or ERROR, nothing in between; the upload
  goes on from where the remote archive ends; cloud ... c: -image sends
  the image of a drive, and asks for the administrator by itself.
• -rescue: the image of a dying disk. A read that fails is not
  insisted on: zpaqfranz jumps ahead and comes back, tens of failed reads
  instead of thousands.
• Thin images: the free clusters that came along with the used ones
  are zeros in the image. A volume with its free space full of old data:
  53 MB instead of 4,978. The files are the same, the source is not
  touched.
• image -to x.vmdk: a virtual disk for VMware Workstation,
  VirtualBox, QEMU/Proxmox. Sparse (about as big as the used data), or
  flat with -raw.
• f: the live map for the fill of a folder too, zeros instead of
  random data, and f X: -test -force -zero -ntfs: zeros in the free
  clusters of an NTFS volume, the files untouched.
• Every version says what it is: who wrote it, on which system, how
  long it took, how much it read. i -verbose shows it, t checks that
  the versions are where they were written.
• Smaller: image out of Windows (the image of a /dev/sdX back to a
  raw file), kickstart, a restore of an image full of zeros eight times
  faster, the window that waits for an elevated run tells how it went.
• Fixes that matter: clusters over 64 KB, a raw image through VSS that
  came out short, password that wrote garbage with a wrong key, -to
  silently ignored with a wildcard, a help that showed a switch cut short.

This document has three parts:

• Part one: what is new, in short, for everybody
• Part two: the same things in more detail, for power users
• Part three: "lo spiegone", how it works inside, and why, for developers

Part one, what is new, for the user

1. The report by e-mail
2. cloud: OK, or ERROR
3. -rescue: the image of a dying disk
4. Thin images
5. image -to x.vmdk
6. f: the map, the zeros, the free clusters
7. Every version says what it is
8. The window that waits
9. image out of Windows, and kickstart
10. Smaller things

1. The report by e-mail

zpaqfranz a z:\1.zpaq c:\data -stat -mailconfig c:\zpaqfranz\mail.conf
          -mailfull owner@x.com -mailprivacy monitor@provider.com -customer smith

At the end of any command (a, backup, cloud, t...) the log goes
by e-mail. zpaqfranz sends it by itself (SMTP with TLS): -maila and
mailsend.exe are gone, and it works on Windows, Linux, the BSDs, macOS,
ESX and the NAS builds.

The typical use: one mailbox sends, two addresses get it. The customer
reads everything; the provider sees that the backup ran, how much it took,
how it ended, and not one name of a file.

The e-mail has a summary and the last lines in its body, and the whole log
attached, zipped. An e-mail that cannot be sent is a WARNING (exit code 1)
when the command itself was OK: the backup is good, but nobody knows.

2. cloud: OK, or ERROR

zpaqfranz cloud u:\prod.zpaq c:\server\*.* -host secure.example.com -user produser
          -ssh ~/.ssh/prod_key -remote /secure/backups/ -test -verify -key
zpaqfranz cloud u:\c.zpaq c: -image -host ... -remote /home/p/image/ -key

• If it ends OK, it is OK. Anything else is an error. The upload, the
  upload of the checksum, the final check and the deep check are OK or
  ERROR: a result that was a warning is now an error to be told.
• The upload goes on. A zpaq archive only grows at its end: zpaqfranz
  checks that what is on the server is the beginning of the local archive,
  and sends only what is missing. An upload that broke off is taken up
  again from there.
• A local folder is a remote folder, as an rsync would leave it: the
  final check compares the whole folder of the archive with -remote. A
  file here and not there is an error.
• cloud ... c: -image is a archive c: -image as it is (-raw,
  -novss, -rescue... too), then all the rest. From a normal prompt it
  asks for the administrator (UAC), the work goes on in a new elevated
  window, the first one waits and ends with its exit code.
• The password is asked twice for a new archive, here and in a and
  backup. An empty answer to -key is refused: the archive would not be
  encrypted.
• It does not wait forever any more. On ERROR, a wrong password, a
  refused captcha: what is on the screen stays for 60 seconds (a key ends
  the wait), then cloud ends. Before, it waited for a key: with nobody at
  the keyboard the scheduler never started the next run.
• cloud always lists the names of the changed files (as -stat does):
  for a log without them, -mailprivacy.

3. -rescue: the image of a dying disk

zpaqfranz a z:\dying.zpaq d: -rescue
zpaqfranz a /mnt/safe/sdb.zpaq /dev/sdb -rescue 64 -rescuetime 0

A read that fails costs seconds, because the disk insists by itself (7 to
60 seconds on a bad sector), and a dying disk may not have many reads left.
The normal image loses nothing that can be read, but it pays one failed
read for every bad sector: on a dead zone of 20 MB, more than 2,000 failed
reads. Hours.

-rescue does it once, and takes all that reads at once:

1. forward, 1 MB at a time;
2. at the first read that fails, a jump ahead: 64 KB, then twice as far
   each time, up to N MB (-rescue N, default 16), until a read works;
3. from there back towards the trouble, up to the first read that fails.

Both edges of a bad zone are taken to the last good sector. What is in
between is not tried: zeros in the image. About 15 failed reads for
that zone of 20 MB.

The price: where the bad sectors are close to each other (less than
64 KB apart) the good ones between them are lost too.

-rescuetime S (default 2): a read that works but takes more than S
seconds is a sick zone too. Its data is kept, then the jump out of the
slow zone. 0: only the errors count.

-rescue implies -image. On Windows no shadow copy is made (nothing
must be written on that disk).

4. Thin images

Nothing to ask for. An image of the used clusters is made of blocks of
2 MB, and a block is read whole when even one of its clusters is used: the
free ones came along, with what is in them (deleted files, old data). On a
volume where the free space is in small pieces that is every block, and
what does not compress in the free clusters went into the archive.

Now the free clusters inside a block become zeros in the image. The
source is not touched, the files in the image are the same.

It happens when the volume cannot change while it is read: with a shadow
copy (the default on NTFS), or with the volume locked. A volume read while
in use (no VSS, not locked) is left as it is, with the warning of always.
NTFS, FAT and exFAT. Volumes over 2 TB are imaged as before, without the
zeroing: not supported (yet) there, and it says so.

The restore of such an image, full of zeros, was slow (the same was true
of the image of a volume zeroed by hand): 66 seconds became 8 for a
partition of 8 GB.

5. image -to x.vmdk

zpaqfranz image z:\c.zpaq c: -to d:\c.vmdk           (one sparse file)
zpaqfranz image z:\c.zpaq c: -to d:\c.vmdk -raw      (flat: c.vmdk and c-flat.vmdk)
zpaqfranz image z:\disk.zpaq 0: -to d:\disk.vmdk     (a whole disk)

A .vhd is what Windows mounts by itself. A .vmdk is what the
hypervisors open: VMware Workstation, VirtualBox, QEMU and so Proxmox.
7-Zip opens it too, down to the files. Windows by itself does not.

From the images of the used clusters (NTFS, FAT, exFAT), from the raw
image of a partition and from the image of a whole disk. No administrator
needed.

A thin image shows here: the tortured test volume (8 GB, 3,056 MB used) is
a .vmdk of 3,386 MB, written in 4.6 seconds. The .vhd of the same
volume is about 8.1 GB.

To read the files. To start a virtual machine from it the image of the
whole disk is needed, with its hidden partitions: that is for the next
release.

6. f: the map, the zeros, the free clusters

zpaqfranz f z:\                               (fill the free space: the live map)
zpaqfranz f z:\ -zero                         (zeros instead of random data)
zpaqfranz f z:\ -nodashboard                  (the classic lines)
zpaqfranz f 3 -test -force -paranoid -zero    (zeros on ALL of disk 3: DESTROYS IT)
zpaqfranz f E: -test -force -zero -ntfs       (zeros in the free clusters of E:)

• f folder\ has the live map and the report of f -test: the free
  space is written and read back block by block, without the cache of the
  system. -nodashboard gives the lines of before.
• -zero with the triplet (-test -force -paranoid): zeros on the
  whole device, and zeros expected back.
• f X: -test -force -zero -ntfs: zeros in every free cluster of
  an NTFS volume, written straight on the volume. The files are not
  touched. It is what a thin virtual disk wants before being shrunk, and
  five seconds on the 8 GB test volume. The volume is locked meanwhile, so
  not the one of Windows and not one with open files: there, f X:\ -zero
  (files of zeros). -verify reads the free clusters back. Not NTFS:
  refused. Over 2 TB: refused, not supported (yet).
• The classic fill ended with exit code 0 whatever happened. Now a write
  that fails is an error (2).

7. Every version says what it is

zpaqfranz i z:\c.zpaq -verbose

VFILE-info: 2 of 2 version(s) say what they are
------------------------------------------------------------------------------------------
<  Ver  > zpaqfranz os        -m         time           bytes read    begins at byte  type
------------------------------------------------------------------------------------------
V00000001 65.8b     win64     84     00:04:50      221.620.522.452                 0  image of c:
V00000002 65.8b     win64     84     00:04:26      220.819.213.276    94.838.370.045  image of c:
------------------------------------------------------------------------------------------
                                     00:09:16      442.439.735.728

Together with the history of -fast (the default) every new version gets
a small record: which zpaqfranz wrote it, on which system, the method, how
long it took, files added, changed and removed, bytes read, and what it
is (multipart, franzen, the image of a drive, tar, stdin). It costs
nothing: no read more of the archive, a few bytes.

• i -verbose shows the table. When every version is an image the three
  columns of the files are left out (they would count the pieces of the
  image).
• t checks, for each version that has a record, that it is the version
  it was written as, that it begins where it began and with the fragment
  it began with. A version taken away from the middle, a piece of a
  multipart archive that is not the right one: an error, exit code 2.

8. The window that waits

C:\> zpaqfranz a z:\c.zpaq c: -image

Admin rights required => getting the power!
136.891s (00:02:16,536) (all OK)

a with -image or -vss, and q, from a normal prompt start
themselves again in an elevated window. The first window waited, said
nothing, and ended with exit code 2, always. Now it prints the last line
every run has, the time of the whole thing and how the elevated run went,
and it ends with its exit code: a script sees 0, 1 or 2.

9. image out of Windows, and kickstart

zpaqfranz image /backup/sda.zpaq /dev/sda -to /mnt/big/sda.raw
zpaqfranz kickstart -to z:\tools

• On Linux, the BSDs and macOS a -image is the device byte by byte.
  image now gives it back as a raw file (to be put back with dd, or
  looked into with losetup), there and on Windows too. Before, on *nix
  it printed the help and ended with exit code 0.
• kickstart (Windows, 64 bit) gets every external file zpaqfranz can
  use: the DLLs of ssh, curl and sodium, mysql.exe, mysqldump.exe, the
  WinFsp installer. The full build takes them out of itself, the others
  download them. The SHA-256 of every file is checked. Not in the open
  build.

10. Smaller things

• NTFS with clusters over 64 KB: the image of the used clusters did
  not understand them ("record MFT $Bitmap not good") and took the whole
  partition instead. A volume with clusters of 2 MB: a .vhd of 4.1 GB
  instead of 8.1.
• A raw image through a shadow copy (-raw -vss, and what -image
  falls back to) ended one cluster before the end of the partition: the
  .vhd made from it was not mounted by Windows. Now it is as long as the
  partition. The images of that kind already made stay short: they go
  back on a partition, not to a .vhd.
• -image -novss on a volume just written said "read while in use"
  and ended with exit code 1 with nobody using it: the lock was tried
  once. Now for three seconds.
• password with a wrong -key wrote an archive of garbage, and with
  -force it wrote it over a good one. Now the key of the source is
  checked first: nothing written. -key2 . takes the password away
  without a terminal (it encrypted with the key ".").
• a "folder\*" -to x: -to was not applied and nothing was said. It
  is refused now, as x does since 65.6.
• x over a file left half way by a killed extraction: when it had
  already its final size it was skipped as good. Now, when its date is not
  the archived one, its content is compared: a warning, exit code 1, and
  the file is not touched (-force to write it again).
• The help cut its headers: twelve lines came out short or glued to
  the text, -test -force -paranoid as -test -force -paran. A help that
  shows a wrong switch is a bug.
• An upload taken up again that sent only a part, when what was
  missing on the server was more than what was already there (an archive
  that more than doubles, a first upload broken before its half). The
  check found it, and a second run fixed it: now the first one is enough.
• mount x.vhd from a normal prompt: Ctrl+C in the window that waits
  did not detach the disk, the elevated window stayed there. Now the
  elevated one follows the first: gone the first, detached the disk.
• mount archive -test said FAILED with paths over 260 characters.
• -always took the folders too: a tried to open them as files.
• The first line of _crc32.txt has the name of the archive only, without
  its path: that file goes to the cloud as it is.
• "Destination too small" suggested -space, that does nothing there.
• On Linux, after a read error of a device, good data around the bad
  sector was lost (see part three).

Part two, the details, for power users

1. The e-mail: who sends, what is taken away
2. cloud, phase by phase
3. -rescue: what is lost, and what is not
4. The thin image: when, and what it says
5. image: every combination, again
6. f: every way
7. t and the records of the versions
8. What can change for a script

1. The e-mail: who sends, what is taken away

mail.conf:   server = mail.provider.com
             port = 587
             tls = starttls
             user = log@provider.com
             password = pw

What -mailprivacy takes away: the names of files and folders become
***, the |STAT| lines and the listings are left out. Totals, times,
versions, results and errors stay. The purge is best effort, and never
with -debug (those lines can hold names).

The progress lines are not in the report: what is redrawn in place on the
screen (percentages, the lines of the upload) does not go in the e-mail.

The log attached is zpaqfranz_log.zip (or zpaqfranz_log_privacy.zip);
over 15 MB of zip it is not attached. Who starts an elevated window does
not send: the elevated run does.

Sent, and accepted by a real server, from: Windows, Fedora, FreeBSD 14.2,
OpenBSD 7.9, macOS 12.7, ESX (a static build for a gcc 3.4.6 system), the
static NAS builds for x86_64 and, under qemu, ARMv5, ARMv7, ARMv8.
-DNOEMAIL builds without it.

2. cloud, phase by phase

• The deep check runs once, at the end: it ran inside both uploads, and
  the final one never ran.
• -onlyupload: the summary shows only what was done.
• -force overwrites the remote archive: it asks the captcha.
• A new archive: the password twice (51852 if they differ, and no e-mail:
  somebody is at the keyboard). -key with an empty answer: 51853.
• Still a warning, yellow: a folder to save that is not there, an e-mail
  not sent.
• Not an administrator after asking for it (UAC off, no desktop: ssh, a
  scheduled task): 91553, said from inside cloud, so that the e-mail of
  the error goes. Run it elevated (a scheduled task: with the highest
  privileges).
• The shadow copy of a cloud -image is released right after the image,
  not at the end of the program: the upload of a C: takes hours.

3. -rescue: what is lost, and what is not

Measured on Fedora with real failures (device-mapper over 256 MB: one bad
sector alone; 40 stretches of 4 KB every 256 KB; one bad sector every
16 KB for 10 MB; a dead zone of 40 MB; a slow zone of 10 MB, 3 seconds a
read; the last 64 KB), sector by sector against the source:

Mind what is counted: device-mapper fails at once, a real disk does not.
At 7 seconds each, 3,987 failed reads are almost 8 hours, 195 are 23
minutes. No sector was ever wrong: lost means zeros.

There is no map of the bad sectors in the archive, and no second pass:
once, what reads at once. At the end the image says how many reads
failed, how many jumps, how much is zeros.

4. The thin image: when, and what it says

With -verbose:

free clusters inside the blocks read: 4.81 GB as zeros in the image

• The list of the used clusters is the one of the shadow copy, not of the
  volume as it is while it is read: files deleted (2.6 GB) and written
  (240 MB) during the image are in the image as they were.
• A block with no used cluster was never read and never stored: nothing
  changes there. What changes is inside the blocks that are read.
• With clusters of 2 MB a block is a cluster: nothing to zero.
• An archive begun by 65.7 and continued by 65.8 is fine: the old versions
  are what they were.
• The other way to the same result, for a volume that is already a
  virtual disk: f X: -test -force -zero -ntfs, that writes the zeros on
  the volume itself.

The restore, the same image of 8 GB (53 MB of archive), on the test VM:

The same bytes, faster.

5. image: every combination, again

• A .vmdk has sectors of 512 bytes (62309 for a source that has not).
• A sparse .vmdk over 2040 GB: not supported (yet), 62310. The flat one
  is the way.
• The flat .vmdk of a whole disk is the disk: the same bytes, the
  same hash.
• A file already there is not overwritten without -force (62311):
  x.vmdk, and x-flat.vmdk.
• A disk full while writing: an error (2); what was written is left there.
• A .vmdk is a disk, not a partition: the image of a letter gets the
  same MBR of ours that the .vhd has.
• On a volume with its free space in small pieces a sparse .vmdk is
  much smaller than a .vhd: its grains are of 64 KB, the blocks of a
  .vhd of 2 MB, and one used cluster keeps a whole block.

6. f: every way

About -zero -ntfs:

• It asks a captcha. It warns that the shadow copies of that volume
  (restore points, previous versions) can be lost: seen, one was gone
  after the zeroing.
• The volume cannot be locked (68779): nothing written, exit code 2. No
  fall back, no forced dismount.
• A cluster is written only when it is free twice: for NTFS before the
  lock, and in the $Bitmap on the disk after it.
• No read back unless -verify.

In the fill of a folder the speed along the space says little (the files
are where there is room): jumps between zones are shown, and do not make
the verdict. Stalls, slow blocks confirmed, errors and wrong data count
as in -test.

7. t and the records of the versions

The record is written with the history of -fast: not with -nofast,
-index, -chunk, -append, -715, nor when the archive goes to
stdout. A version that adds nothing gets no record (and is not written at
all, as before).

What it cannot say: a version without a record (an older zpaqfranz, zpaq
7.15), a tail that is not there any more: an archive cut after a version
is an older, valid archive. For that, something outside is needed
(_crc32.txt, the size in the e-mail).

zpaq 7.15 and the older zpaqfranz read these archives as before. The
older zpaqfranz show the record as one more deleted entry in l -all.

8. What can change for a script

Part three, lo spiegone, for developers

1. The capture, and the purge
2. The upload in append
3. -rescue inside, and the cache of Linux
4. VFILE-info: a protocol
5. franzusb on files, and a locked NTFS that does not answer
6. The thin image
7. The restore that was slow
8. franzvmdk
9. Asking for the administrator
10. How it was tested
11. Fixes
12. Tried and thrown away
13. Known limits and open points

1. The capture, and the purge

Everything printed goes through one place, and from the very first line
(mailreport_avvia, before the parameters are read) it is kept in memory
twice: as it is, and purged. The purge is made at the source: a name
of a file is printed with %Z (the format of zpaqfranz for a path), and
in the purged copy a %Z is ***. printUTF8, list_out and the
|STAT| lines know they are names too. A last net looks for what still
seems a path.

So the rule for new code: a message that prints a name uses %Z, never
%s, or the name ends up in the log of who must not see it.

The progress: a print that ends with \r takes back the line it began,
"as on the screen" (in about a hundred places the text and its \r are
two prints: the texts stayed, all on one line). A display redrawn in
place by a thread (the lines of the upload) is between
mailcattura_live(true) and (false): what that thread prints does not
go in the report, the errors of the others do.

mailreport() sends: called by main, by seppuku() (a command that
dies must tell too) and by cloud() before its banner. The library that
talks SMTP and TLS and the one that makes the zip are in the source,
between their own markers (they are generated: not to be touched by hand).

2. The upload in append

A zpaq archive is written forward: a new version is appended. So the
remote file, when it is good, is a prefix of the local one. The check
is "quick": the size, and the hash of samples of both; then only the tail
goes up, and a second quick check (size, three samples of 16 KB: start,
middle, end) says how it went. The deep check (-sha256deep and the
others) asks the server for the hash of the whole file.

The bug of the resume: the local file was already positioned at the point
to start from, and CURLOPT_INFILESIZE_LARGE was the size of the tail,
with CURLOPT_RESUME_FROM_LARGE the point. But libcurl takes the resume
point away from the file size by itself: it sent (tail minus resume)
bytes. When the tail was smaller than what was already there the count
went below zero, which for libcurl is "size unknown, up to the end": that
is why it usually worked. When the archive more than doubled in one run,
or a first upload was broken before its half, the remote stayed short. A
valid prefix, so the next run finished it. The size given is the whole
local file now.

The quick hash of a file shorter than 64 KB read all of it instead of
its first N bytes: the append of a tiny archive was always refused.

3. -rescue inside, and the cache of Linux

img_rescue*, after img_errore(). A state machine over the reads of an
image, the same for the three readers (the used clusters, the raw
partition, a *nix device): forward at 1 MB; a failed read is found down to
64 KB and then to the sector; the jump doubles from 64 KB to the limit;
the landing is halved until it stands on good data; then back by 64 KB
and by sectors to the last good one. A slow read inside a jump born of a
real error counts as a good one, or good islands would be jumped over.

On *nix the device is opened without the cache (O_DIRECT,
F_NOCACHE), every read is of whole sectors, and a descriptor that
answers EINVAL (a file on a filesystem of 4 KB sectors) is opened again
with the cache.

The cache of Linux loses good data. Found while testing, and it is of
the normal image too, not of -rescue: after a sequential read, one bad
sector makes the kernel fail 512 KB on that descriptor (the read-ahead);
with a new descriptor, the page of 4 KB; with O_DIRECT, the sector. And
after an EIO it reads one page at a time. On a device with many
failures: 5,503 good sectors lost (2.75 MB), 9,497 failed reads. Now what
is read again after an error goes through a second descriptor without the
cache: 0 good sectors lost, 3,987 failed reads.

Failures made up by a hidden switch (-rescuefake) are above the kernel
and do not see any of this: every change to the reads of an image on *nix
must be tried on a device that really fails (device-mapper, error and
delay).

4. VFILE-info: a protocol

One more entry with date 0 in the last index block of the version, next
to the pointer of the -fast history, in clear:

VFILE-info:1|v=2|ps=94838370045|fr=1234567|iu=4|id=1|zv=65.8b|os=win64|m=84|im=c:|du=266000|ba=220819213276|fa=4|fu=0|fd=0

v the version, ps where it begins, fr its first fragment, iu and
id the records of its index (with a date, and deletions); then what it
is and what it did. Keys only when true for mp (multipart), fz
(franzen), im (image, and of what), tar, si (stdin).

It is a protocol, to be extended: key=value, a key that is not known is
ignored, a key that is not there means that check is not made, a record
of a higher generation (the number after the colon) is ignored whole.
Keys are only added, a meaning never changes.

What it knows before writing is taken before writing (vinfo_prima), the
rest when the index is made (vinfo_riga): nothing is read from the
disk for it. A first version of this (-112, see section 12) kept the
CRC-32 of the archive too, and to know the one "before" it read the
archive again. Thrown away: no metadata must cost one more read.

The pointer line of -fast (ZPFL1|...) was not made longer: its reader
wants exactly its eight fields, and every zpaqfranz from 65.4 would fall
back to the slow listing.

5. franzusb on files, and a locked NTFS that does not answer

The fill of a folder is franzusbtest on a "device made of files":
franzusb::apricartella, files zchunk_NNNNN of 512 MB written without
the cache (FILE_FLAG_NO_BUFFERING; on *nix O_DIRECT when the
filesystem has it, else fsync and posix_fadvise). The same passes,
map, report and verdict of f -test. -zero is in the generator
(zeri()): zeros written and zeros expected, for the files and for the
triplet.

franzzerontfs writes the free clusters of a volume raw. The order is
what took the time:

• FlushFileBuffers on the volume first: NTFS keeps for a while the
  clusters of what was just deleted, and they would look used.
• The map of the used clusters as NTFS says (FSCTL_GET_VOLUME_BITMAP),
  before the lock: a locked NTFS volume does not answer any more
  (error 21: the lock flushes everything and lets the volume go).
• FSCTL_LOCK_VOLUME, a few tries.
• The $Bitmap as it is on the disk, after the lock: boot sector, record
  6 of the MFT, its fixups, the runs of its unnamed $DATA. What was
  allocated between the first map and the lock is there.
• A cluster is written only if it is free in both. Anything not
  understood on the disk (a $Bitmap spread over more MFT records):
  nothing is written.

6. The thin image

franzimager::azzeraliberi(): after a block is read, and after the
clusters of the files left out are cleared, the runs of free clusters
inside it are set to zero in the buffer. Only when m_usingvss || m_bloccato: the list of the used clusters must be the one of the moment
the data is read. With VSS both come from the same device, the shadow
copy.

The last cluster. segnaultima() says the last cluster of the volume is
used, always: its block must be in the image for what comes after it (the
copy of the boot sector of NTFS, in the last sector of the partition).
When that cluster was free it stayed in the image with its old data, the
only free cluster not zeroed. It was found by the comparison byte by
byte, on a volume extended after it was filled. m_ultimolibero
remembers how it was.

Clusters over 64 KB: in the boot sector of NTFS the byte "sectors per
cluster" above 0x80 is an exponent (2 to the power of 256 minus the
byte). It was read as a number: 2 MB clusters were 244 sectors.

The raw image through VSS: the device of a shadow copy ends where the
volume ends, one cluster (or less) before the end of the partition. The
tail is now read from the physical disk (42287).

The lock: FSCTL_LOCK_VOLUME is refused for a moment on a volume just
written, or looked at by the antivirus or the indexer. Ten tries, 300 ms
apart, only while the answer is "access denied".

7. The restore that was slow

Jidac::extractstdout is the engine that hands the fragments over in
order, from a cache of decompressed blocks, with worker threads that
decompress ahead. For every fragment it looked ahead for the next blocks
to ask for, starting again from the current fragment, and when
everything ahead was already cached or asked for it walked to the end of
the file without finding anything to do. An image full of zeros is a long
row of the same fragment (about 50 KB each, all in the same block): the
time grew with the square of their number.

Now a cursor (scan_voce, scan_it, scan_idx) remembers where the
last scan stopped, and the next one goes on from there; it is cleared
when blocks are thrown out of the cache (the eviction, the reset after a
stall), because then what is behind it can be needed again. The same
output, byte by byte, on the other roads through that engine too
(x -recover, pp, -stdout of a file not stored in order, zip -deflate).

8. franzvmdk

A writer of the two kinds of hosted .vmdk that everybody reads.

Sparse (monolithicSparse): a header of one sector (KDMV, version
1), the text that describes the disk inside the file (20 sectors), the
grain directory and the grain tables twice (the "redundant" copy
first: VMware and QEMU write both, and so does zpaqfranz), then the grains
of 64 KB one after the other as they come. A grain table says where each of its 512 grains
is in the file, in sectors, in 32 bits: so the file cannot be longer than
2 TB. A grain of zeros is not written: its entry is 0, and who reads gets
zeros. No checksums anywhere. The capacity is known before the first
byte, so the tables have their place from the start and are written at
the end.

Flat (monolithicFlat): the disk byte by byte in x-flat.vmdk, and
x.vmdk with the text:

# Disk DescriptorFile
version=1
CID=f539468c
parentCID=ffffffff
createType="monolithicFlat"

# Extent description
RW 16744448 FLAT "x-flat.vmdk" 0

# The Disk Data Base
#DDB

ddb.encoding = "UTF-8"
ddb.virtualHWVersion = "4"
ddb.geometry.cylinders = "1042"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

ddb.encoding: without it VMware reads the name of the flat file in the
code page of the PC, and does not find a name with an accent.

The bytes come from where the .vhd takes them. The image of the used
clusters is a row of records, a sector of bitmap and a 2 MB block of the
virtual disk (our MBR and the partition at 1 MiB are already in it), and
the blockmap says which block each one is: preparavmdkthin and its
handler put each block in its place. A raw image goes down the road of
the raw image to a .vhd (preparavhdraw: the MBR for a partition, a
whole disk as it is), told to hand its blocks to franzvmdk instead
(setvmdk).

One thing to know: franzimager is copied with the Jidac that holds it
(Jidac jidac(*this)), so the writer is a pointer there, not a member.

9. Asking for the administrator

Three places ask, in two ways.

cloud -image and mount x.vhd: ShellExecuteExW with runas and the
command line as it was typed (GetCommandLineW: the quotes of a path
with spaces, or of a password, are still there), a hidden -elevated
added (never twice), the handle waited for, its exit code taken. For
cloud it is done before anything is read from the keyboard, or the
password would be asked twice.

a -image, a -vss, q: the older way, through the shell. It now hands
the exit code of the elevated run over as 100 plus the code (a code of
the shell itself is not taken for it), 222 when nothing started, 223 when
the code cannot be read; and -elevated is added there too.
ultimariga_tempo() is the first half of the last line of every run,
taken out of main to be printed by the window that waits too.

mount x.vhd: the elevated window gets -elevatedpid, the process that
asked for it. It opens it (SYNCHRONIZE) and looks at it in its loop:
gone the first window, it detaches and ends.

-elevatefake (hidden) starts again without runas, in a hidden window:
on a test bench there is no UAC question to answer.

10. How it was tested

The images, on a virtual machine with a second disk of 8 GB to destroy at
will. tortura.ps1 makes NTFS as complicated as it can be: every kind of
file (sparse, compressed, encrypted, 4,000 extents, 1,023 hard links, 300
streams, reparse points, a path of 25,000 characters...), and holes:
the volume is filled, two files out of three are deleted, so that the
free space is in about 17,000 pieces. Then, with tools that are not
zpaqfranz:

• the manifest (manifesto.ps1): one line for each entry of the
  volume, with the raw Win32 calls: attributes, sizes, dates, links, file
  id, short name, security descriptor, reparse data, every stream with its
  SHA-256. After a restore it must be the same in everything;
• chkdsk, and the image mounted by Windows (a .vhd) or by OSFMount
  (a .vmdk);
• the imprint (t_esatto.ps1): for every cluster, used or free (as
  Windows says) and a hash of its bytes. The disk is made read only
  first, so NTFS mounts it read only and not a byte changes between the
  imprint of the source and the image. Then: every used cluster the same,
  every free cluster zeros, the same bitmap, the same bytes after the last
  cluster.

The rest: cloud against a real SFTP server (archives that double, tails
of a few bytes, uploads cut and taken up again, a remote file changed);
the e-mails to a real server from every platform, and to a fake one that
keeps what it gets, to read what really goes out; -rescue on devices
that really fail (device-mapper) and with made up failures on Windows and
on every *nix; the password, the wildcards, the half written files, with
scripts that compare the old build and the new one.

Built on Windows (g++ 14.2 ucrt64: plain and with the mount; the SFTP,
open, 32 bit and old compiler variants checked for syntax) and Fedora 44
(gcc 16.2); the part of the series up to the e-mail and -rescue also on
FreeBSD, OpenBSD, macOS, ESX and the NAS targets, and there with the full
and the open build of Windows too. The autotest is all OK on Windows and
Fedora. The source is 234,596 lines (8.5 MB).

11. Fixes

• password: -key2 . went through the hashing of every key, so the
  branch "enter . for no password" was never reached. Compared as a hash
  now. The key of the source is checked as checkpassword does before
  anything is written (and before -force deletes); the copy and the
  final check are in a try: on any failure the output is deleted.
• a with a wildcard and -to: rename() replaces a prefix, and
  dir/* is a prefix of nothing. Refused in testparametriadd() (71410),
  the same choice made for x in 65.6: rename() is used by dozens of
  commands. A file that is really called q?.txt (on *nix it can be) is
  not a wildcard.
• x, "exists, skip": with more threads the blocks of a file are
  written out of order, and a killed x can leave a file with its final
  size and holes inside. The date is the hint (the right one is set only
  when a file is complete), the content is the verdict (equal(), the
  SHA-1 of the fragments).
• *image on nix: the parser knew the command on every platform, the
  dispatch was under #ifdef _WIN32: the help, and exit code 0.
  restoreimagedevice() is outside the ifdefs.
• kickstart: the list of the resources (name, size, SHA-256) is one
  function now, used by the command and by who extracts them at need.
• The help: scrivi() cut a header longer than its column. A long
  one is printed whole on its own line, the description below.
• -always marked the folders too: skipped.
• mount -test on Windows did not use \\?\ over 248 characters.
• f, the classic fill: writes and closes are checked, exit(0)
  taken away, the 99% is really written (the last chunk can be partial),
  the cache is flushed and dropped before the verify, the hash of the
  zeros is computed once.
• The message of image "destination too small" suggested -space,
  which is not read there.
• cloud: the flags of the deep check were "saved" and the copies
  cleared, not the flags; with -turbo the files went through the plain
  add(); the summary of -onlyupload showed as OK phases never run.

12. Tried and thrown away

-112: the CRC-32 of the archive inside the archive ("the European
number for emergencies"). A record with the size and the CRC-32 of the
archive before and after each version, and a t that read the whole file
in pages and checked them. It worked, on multipart and encrypted archives
too. But to know the CRC "before" on an archive that had no record yet it
read the archive again, and a metadata that costs a read of a 200 GB file
is not a metadata. What is left is VFILE-info: only what is known without
reading anything.

A map of the bad sectors in the archive of a -rescue, and a second
pass to try them again: no. The copy of a dying disk is made once.

Two SMTP accounts as the normal case, one of the customer and one of
the provider: the normal case is one mailbox and two addresses.
-mailprovider is still there for who wants the second account.

Hiding the password of the mailbox in the configuration file: it
would be hidden from nobody who wants to read it. In clear, and said.

13. Known limits and open points

• Not tested: the elevated window with a real UAC question. I do not have UAC at all 😄 . Volumes over 2 TB: there the free clusters are not zeroed and f -zero -ntfs refuses. Disks with sectors of 4,096 bytes,
  clusters under 4 KB. A flat .vmdk over 2 TB. The changes after
  -rescue (the thin image, f, the .vmdk) on the platforms that are
  not Windows and Fedora: they are Windows code, the others compile them
  away.
• A .vmdk was read, not run: by the disk tools of VMware and QEMU,
  by OSFMount, by 7-Zip. Not attached to a virtual machine that was then
  started. ESXi wants its own kinds of .vmdk: not tried there.
• To start a virtual machine from an image the whole disk is needed,
  with the EFI partition and the others: a archive 0: -image. The image
  of C: alone does not boot, whatever the format.
• VHDX is still not written. It would serve for more than 2040 GB,
  for sectors of 4,096 bytes, and to boot in a second generation Hyper-V
  machine. The flat .vmdk has no limit of size. I do not use Hyper-Microsoft VM, therefore not very interested in VHDX.
• The image of a *nix device goes to a raw file only: not to a .vhd,
  nor to a .vmdk.
• -zero -ntfs: the bitmap is in memory (a bit a cluster); a $Bitmap
  spread over more MFT records is refused.
• t exits with 1, and says "VERDICT: OK", on an archive with one byte
  changed in the middle of a block (the files that use it are told
  corrupted). A fast l does not tell a missing piece in the middle of a
  multipart archive (l -nofast and t do).
• x -recover with a cache smaller than three blocks (-ramsize 40MB)
  throws blocks away and decompresses them again without end. With the
  default cache (1 GB) it does not happen.
• An image -image -novss taken a few seconds after a chkdsk or a resize
  of the partition can find the volume busy for more than the three
  seconds of the lock: the warning, and exit code 1.
• A shadow copy already on a volume can be deleted by Windows during a
  -zero -ntfs (its storage cannot grow while the volume is locked).
• backup asks the password twice at every run (it looks for an archive
  that, in a multipart, has another name).
• The limits of 65.7 that are still there: parallel reads of an image, a
  .vhd whose extraction is killed is left sparse, the files left out
  with -not are in the .vhd with zeros inside.

And finally... the ... allin!

This is an example of how to create a local, encrypted backup, send it to a remote SFTP server, check it, and send two log messages: one that has been purged (for privacy) and one that is complete (with details of the files added, modified, and deleted).

zpaqfranz.exe cloud franco_whatever.zpaq c:\zpaqfranz c:\wallpaper -not musicall -key lapasswordona -remote /home/franco/repository -host sftp.somewhere.com -user thegoduser -port 23 -ssh keyfile_toload -stat -test -verify -ignore -mailserver mail.yourisp.com -mailport 587 -mailuser provona@yourisp.com -mailpassword "thepassword" -mailfrom provona@yourisp.com -mailfull iamtheclient@whoknows.com -mailprivacy iamtheprovider@power.com -customer franco_backuppone