zpaqfranz 65.7

This is the release of the images. Almost everything new is about
taking the image of a drive, getting it back, and using it without
restoring it:

BEWARE: MORE TEST IS NEEDED!

• One switch: a archive C: -image chooses by itself. NTFS: the used
  clusters, with a shadow copy (VSS) when it can be made, without the swap
  files. FAT, FAT32, exFAT: the used clusters, the drive locked. Anything
  else, or when it cannot: the whole partition. -raw is the whole
  partition (or disk) byte by byte, no questions.
• Faster: the image goes through the parallel fragmenter (-turbo)
  too, as files do since 65.6. On the author's workstation the image of
  C: runs at about 800 MB/s.
• A .vhd that Windows mounts: image archive C: -to d:\c.vhd, from
  NTFS, FAT, exFAT, raw images and whole disks. Double click, Disk
  Management, Mount-DiskImage, or...
• mount d:\c.vhd: zpaqfranz asks Windows to attach a .vhd, .vhdx
  or .iso read-only. No WinFsp, in every Windows build.
• image does not need to be told what the image is: the archive says
  it, -to says what to do with it (a .vhd, a raw file, a partition).
  -raw there means "write the unused part too, as zeros".
• x and virtual disks: a .vhd or .vhdx extracted from an archive
  is never left sparse (Windows refuses it), and it is not written twice
  any more to get there. -huge reviewed.
• Fixes that matter: a raw restore that wrote on the source drive, an
  export of a damaged archive that said "all OK", the free space measured
  on the wrong drive.

This document has three parts:

• Part one: what is new, in short, for everybody
• Part two: the same things in more detail, for power users
• Part three: "lo spiegone", how it works inside, and why, for developers

Part one, what is new, for the user

1. a -image: one switch
2. Faster images
3. A .vhd that Windows mounts
4. mount x.vhd
5. image: back on a drive, or to a file
6. x: virtual disks, and -huge
7. Smaller things

1. a -image: one switch

zpaqfranz a z:\c.zpaq c: -image            (it chooses by itself)
zpaqfranz a z:\usb.zpaq f: -image          (a FAT32 or exFAT stick: the same)
zpaqfranz a z:\e.zpaq e: -raw              (the whole partition, byte by byte)
zpaqfranz a z:\disk.zpaq 3: -image         (the whole physical disk 3)

Before, the image of C: wanted -image -ntfs -vss, and you had to know
what was on the drive. Now -image looks at the drive:

Two switches for the image, and two to say no:

When the shadow copy cannot be made (a USB disk, no space for it...) the
image goes on without, the drive locked, and the reason is told. When the
drive can be neither snapshotted nor locked (the drive of Windows without
VSS, files open on it) the image is still taken, with a warning at the
start and at the end and the exit code 1: what was written meanwhile can
be inconsistent.

2. Faster images

Nothing to ask for: it is -turbo, the default since 65.6, that now works
on the stream of an image too. Until 65.6 an image went through one thread
that cut the fragments and computed their hashes, and that thread was the
limit of every image of a fast drive.

The archive is the same, fragment by fragment: only faster. -turbo 1,
-t1 and -noturbo are the old way.

3. A .vhd that Windows mounts

zpaqfranz image z:\c.zpaq c: -to d:\c.vhd
zpaqfranz image z:\c.zpaq c: -to d:\folder          (image_C.vhd in it)
zpaqfranz image z:\c.zpaq c: -to d:\old.vhd -until 3

An image inside an archive can now become a virtual disk that Windows
opens by itself: you look into yesterday's C: without restoring it, copy
out what you need, and close it. It works from the images of NTFS, FAT,
FAT32, exFAT, from raw images of a partition and from the image of a whole
disk. No administrator needed to make the .vhd.

In 65.6 the .vhd made from an NTFS image was refused by Windows ("the
file or directory is corrupted", then "virtual disk system limitation").
Two reasons, both fixed: see part three.

A .vhd cannot hold more than 2040 GB, nor a file system with sectors
that are not of 512 bytes: for those, a raw file or a partition.

4. mount x.vhd

zpaqfranz mount d:\c.vhd
zpaqfranz mount d:\c.vhd Y:
zpaqfranz mount d:\c.vhd -test

Windows attaches the virtual disk read-only, zpaqfranz shows its
volumes (letter, file system, size, label) and opens Explorer on the first
one. A key, or Ctrl+C, detaches it; so does closing the window, or a crash:
the attach is never permanent. .vhd, .vhdx, .avhdx and .iso.

It needs an administrator: from a normal prompt zpaqfranz starts itself
again in an elevated window (the UAC question), and waits for it. It does
not need WinFsp, and it is in every Windows build, the open one too.
mount archive.zpaq is the mount of an archive, as before.

5. image: back on a drive, or to a file

Restore to file
zpaqfranz image i5.zpaq f: -to d:\3.vhd              (a .vhd)
zpaqfranz image i5.zpaq 3: -to d:\folderone          (the disk 3: image_3.vhd in the folder)
zpaqfranz image i2.zpaq f: -to d:\1.raw              (the partition, byte by byte)
zpaqfranz image i5.zpaq f: -to d:\old.vhd -until 3   (an older version)

Restore to drive
zpaqfranz image i1.zpaq f: -to i: -image             (on the partition i:)
zpaqfranz image i1.zpaq f: -to i: -image -raw        (the same, every sector)

zpaqfranz drives                                      (letters, disks, partitions)

The archive says what the image is (used clusters, or raw); -to says
what to do with it:

-raw here means the unused part too. An image of the used clusters,
written on a partition, touches only those clusters: what was in the free
space of the destination stays there. With -raw the whole partition is
written, zeros where the image has nothing. It changes nothing for a
.vhd, for a raw file, and for a raw image (they have everything already).

A destination file that exists is never overwritten without -force.

6. x: virtual disks, and -huge

A .vhd, .vhdx, .avhd or .avhdx inside a normal backup (a folder of
Hyper-V, for example) is extracted so that Windows can mount it: never
left sparse. Since 65.6 x makes big files sparse, and Windows refuses a
sparse virtual disk.

On NTFS it is also fast: the file is sparse only while it is written, and
when it is complete the holes are filled with zeros and the attribute is
taken away. Where there are no sparse files (exFAT, FAT, some network
shares) the virtual disk is written the -huge way, by itself.

-huge ("files without gaps: the zeros are written, never a seek past the
end") was reviewed: it now works for every file and not only for the first
one, it turns -sparse off, and it shows one progress with one ETA.

7. Smaller things

• Restore of a raw image on a partition wrote on the SOURCE. image a.zpaq F: -to G: -image -raw ignored -to and overwrote F:. It was
  already so in 65.6. Fixed.
• An archive with the images of more drives: a restore took all of them,
  one over the other, on the same destination. Now only the one asked.
• A destination bigger than the image: Windows could find, at its end, the
  backup boot sector of the file system that was there before, and show a
  broken NTFS instead of the one restored. The last MiB is zeroed now. An
  exFAT is adapted to its new partition (Windows mounts an exFAT only if it
  is as long as its partition).
• The export of a .vhd from a damaged archive ended with "all OK" and
  exit code 0, with a broken .vhd. Now: the list of what is missing, exit
  code 2.
• A disk full while extracting is said at the first write that fails
  (56483, exit code 2). With -image it was not said at all.
• -to f:\x.vhd (a file in the root of a drive) run from another drive:
  the free space checked was the one of the current drive. An export to
  an empty 600 GB disk was refused because C: had 16 GB free.
• Windows, a standard user without WMI (an ssh session): every command
  crashed at startup.
• mount relaunched as administrator where the elevation is not possible
  (no desktop, UAC off): it started itself again forever. Now it stops
  (77122, exit code 2).
• The help of image and of a -image rewritten around the new switches.
  The old ones (-image -ntfs, -vss, -ntfs -raw, -vhd) still work.

Part two, the details, for power users

1. What -image decides, and what it says
2. The image of the drive of Windows
3. image: every combination
4. mount x.vhd in practice
5. Virtual disks, sparse files and -huge
6. How fast, and where the time goes
7. What can change for a script

1. What -image decides, and what it says

The first line tells the choice:

Image of C: (NTFS): the used clusters, with VSS
Image of F: (exFAT): the used clusters
Image of S: (not formatted, or unknown to Windows): the whole partition (raw)

The reasons of a missing shadow copy are the ones Windows gives: access
denied, volume not supported (a removable disk), not enough space for the
shadow copy, another one being made, a veto of the provider...

The old switches are not in the help any more, and still do what they did:
-ntfs forces the used clusters (an error if they cannot be read, instead
of the fall back to raw), -vss wants the shadow copy (without it, stop).
-frugal leaves out about 220 folders of temporary files and caches too
(look at the list before using it on a drive you want back as it was);
-frugal with -nofrugal is refused.

On Linux, *BSD and macOS -image and -raw are the same thing: the whole
device.

2. The image of the drive of Windows

zpaqfranz a z:\c.zpaq c: -image

is -image -ntfs -vss of before, without the swap files. The shadow copy
is what makes it consistent: C: cannot be locked. If the shadow copy
cannot be made the image is taken anyway, with the red warning and exit
code 1: better than nothing, and you are told.

Keep the archive on another drive: with the archive on the drive of the
image the drive is never locked (the archive could not be written), and
the image grows while it is read.

3. image: every combination

• Both kinds of image of the same drive in the archive: the latest one
  (22332); -until for the other.
• -to G: without -image: refused (09412). It is the consent.
• G: not empty: the RISKY question; -space skips it.
• A destination bigger than the image: the image, then the rest as it was
  (but the last MiB, zeroed). -raw writes zeros up to the size of the
  partition that was imaged, not beyond.
• Files left out with -not when the image was taken: on a partition they
  are deleted after the restore; in a .vhd they are there, with zeros
  inside.
• The raw file from an image of the used clusters is a sparse file on NTFS
  (-nosparse to write its zeros).

4. mount x.vhd in practice

Every image has its own disk signature, so the .vhd of two different
images can be mounted together; read-only, two copies of the same image
too. A .vhd mounted read-only cannot be changed, so chkdsk on it only
looks.

5. Virtual disks, sparse files and -huge

The data of a file come out of an archive in the order they are stored,
not in the order of the file. For a big file stored over many versions
(an image, a virtual machine) the end can come before the start.

A virtual disk must not be sparse when Windows mounts it. So:

Measured: one file of 8 GB (1.3 GB of zeros in it) from an archive that
gives its end first, to a SATA SSD, 16 threads:

From an archive in order there is no difference (17-19 s in every case).
-huge does not write less: it makes the wait visible, and on drives
where the silent filling is slow (USB sticks, spinning disks, shares) it
is the better way. If the extraction is killed, the .vhd is left sparse
(and incomplete).

While the zeros are written the progress line starts with zeroing; the
normal line keeps the only ETA of the job. With -verbose, at the end:

-huge: zeros written by us 8.589.586.168 (8.00 GB) in 1 gaps
virtual disks: 1.429.733.376 (1.33 GB) of zeros written in 3 holes at the end, not sparse any more

6. How fast, and where the time goes

With -verbose the image tells how long it spent reading:

image stream: 4.526.758.912 read, 4.38 s inside the reads (986.49 MB/s), 6.44 s in all (670.45 MB/s), -turbo 4 threads

On the test VM (4 cores, a virtual NVMe disk):

The first image is limited by the CPU (cutting, hashing, compressing): more
cores, more speed, up to the speed of the drive. The second one is limited
by the drive: everything must be read to find what changed. On the
author's workstation (16 cores, NVMe) the image of C: runs at about
800 MB/s.

A whole partition (-raw) is read 1 MB at a time: -buffer 4194304 reads
4 MB at a time, 20% more in the reads on the VM (947 against 773 MB/s).
The whole a did not change there, because there the CPU is the limit:
it is worth a try where the cores are many.

7. What can change for a script

Part three, lo spiegone, for developers

1. Why Windows refused the .vhd
2. The .vhd that is written
3. FAT and exFAT
4. VSS, lock, and nothing
5. extract() and the three pieces
6. -turbo on a stream that is not a file
7. Sparse while written
8. -huge
9. image -raw: the zeros
10. franzmonta
11. Fixes
12. Tried and thrown away
13. Known limits and open points

1. Why Windows refused the .vhd

Two things, one after the other.

A short last block. The image of the used clusters is read from the
handle of the volume (\\.\E:), and that handle stops at the length of
the NTFS volume, which is one cluster shorter than the partition: after it
there is the copy of the boot sector. The last block was read short and
written short, and the block allocation table of the .vhd made it end
inside the footer: "the file or directory is corrupted and unreadable".
The bit of the last cluster is always on, so it happened always. Now the
handle is opened with FSCTL_ALLOW_EXTENDED_DASD_IO (the read goes up to
the end of the partition, backup boot sector included) and a block is
always whole: what is not read is zeros.

A sparse file. 65.6 made x write big files as sparse files. Windows
does not attach a sparse virtual disk: "virtual disk system limitation...
must not be sparse". isdiscovirtuale() (the four extensions) now decides.

2. The .vhd that is written

A dynamic VHD, in pieces inside the archive: image_X.fhd (the used
blocks: 512 bytes of sector bitmap and 2 MB of data each), image_X.header
(footer copy, dynamic header, block allocation table), image_X.footer,
image_X.meta, and .exclud / .zeroed when something was left out. The
.vhd is header + blocks + footer.

• An MBR of ours, the partition at 1 MiB, in sectors of 512 bytes (on a 4Kn
  source it was put at 2048 sectors of the source, 8 MiB, past the first
  block: an overflow). The MBR type is the one Windows writes: 0x07 NTFS and
  exFAT, 0x0C FAT32, 0x0E FAT16, 0x01 FAT12.
• The sector bitmap of a block is full: one bit for every 512-byte sector
  of the virtual disk, whatever the sector of the source.
• The disk signature in the MBR comes from the unique id of the VHD (it was
  a constant: two copies attached together collided). Geometry (CHS) by the
  algorithm of the specification (above 127 GB the old one gave 255 heads).
• Refused, reading the metadata: sectors that are not of 512 bytes (62302),
  more than 2040 GB (62303). Warned when the image is taken (45482, 45483):
  the image is still good for a partition or a raw file.
• A raw image becomes a .vhd through its own writer
  (vhdraw_export_handler, franzimager::preparavhdraw): the blocks that
  are all zeros are left out of the allocation table, the table is written
  at the end. A whole disk goes in as it is, with its own partition table.

3. FAT and exFAT

readntfsbootsector now knows NTFS, exFAT and FAT (the type from the
number of clusters, as the Microsoft specification says). The map of the
used clusters is FSCTL_GET_VOLUME_BITMAP, the one the defragmenters use,
on the same handle of the reads; FSCTL_GET_RETRIEVAL_POINTER_BASE must
agree with the boot sector on where the clusters begin, or there is no
thin image (43601). The map of franzimager stays what it was for NTFS,
"units from the start of the partition": a unit is a cluster (64 KB at
most), what is before the first cluster (boot region, FATs, the root of a
FAT16) is always taken, a used cluster turns on every unit it touches. The
file system type is in the .meta (in a byte that was reserved: 0 is NTFS,
so the images of before are still read).

Windows mounts FAT and exFAT moved to another offset with their boot
sector untouched (the "hidden sectors" are not checked): the boot sector is
copied as it is. But an exFAT is mounted only if its VolumeLength is the
partition: restored on a bigger one it is adapted (adattaexfat: the
length, the checksum of the boot region, the backup region).

4. VSS, lock, and nothing

sceglimodoimage() asks GetVolumeInformation and sets what add() then
does. franzimager::createvss now reads the return value of
Win32_ShadowCopy.Create and keeps the reason. Without a shadow copy:
FSCTL_LOCK_VOLUME on the volume, after the files to leave out have
been looked up (they are opened by name: impossible on a locked volume),
and then the map of the used clusters is read again, and the blocks
counted again, because the volume could have changed in between.
franzraw no longer dismounts a volume it could not lock (it took the
files away from who had them open). Neither VSS nor lock:
g_imgincoerente, said twice, exit code 1.

The fall back: aprivhd() was rewritten so that a failure closes handle,
lock and shadow copy, and gestisciflagimage() can go on with the raw
reader.

5. extract() and the three pieces

extract() joined header + image + footer by joining their lists of
fragments, but after the blocks to decompress had been planned. The
image was planned with its own fragments only: header and footer ended up
in the .vhd only because they were in the same block as the tail of the
image, and because their own entries (not to be written) were planned too.
Those entries were treated as real files: a file with their name in the
current folder was "existing, skipped" (a shorter block: a broken .vhd),
and with -force it was deleted. The join is now done before the
planning, with the size of the whole file, and the planning skips the
entries that are not to be written.

That is also why the check of the files not complete was turned off for a
.vhd (header and footer always looked incomplete), and why a damaged
archive gave a broken .vhd with exit code 0. They are not planned any
more: the check is back on, and one missing fragment of a .vhd is an
error (2), not a warning.

rename() applies .fhd => .vhd before files[0] => tofiles[0]:
with -to x.vhd the second one did not match any more and the file went
to the current folder as image_E.vhd, "all OK". The replacement is now
set only when the name still ends in .fhd.

6. -turbo on a stream that is not a file

add2() (the -turbo of 65.6) read a file in batches of 64 MB, one being
read while the other is cut by several threads; images were excluded
(!flagimage) and went through the loop of add(). Measured on plain
files, this machine: 390 MB/s through the loop, 1.1 GB/s through -turbo.

add2_imgthread is the reader for an image: it fills the batch with the
reads of the source, one after the other exactly as the loop takes them
(handle_vhd_read for the used clusters, prendiraw, elaboradump), and
updates the hash of the file in the same order. A raw read still goes into
the buffer it always went into (those handles are opened unbuffered), and
is copied. The cuts are found by the same speculative chain of 65.6, so
the fragments are the same.

The live map was the delicate part: it assumed that the block just read
is the one whose fragments are going by. Every read is noted (where its
data are in the batch, the block of the source, how long it took, the
unreadable sectors counted right after it), and the main thread puts them
on the map when their data go through the fragments
(franzimgdash::dopocon: dopo() with a time and an error count that
were measured before). A reader that gives an error code stops the image
at the end of the batch (imagefatale).

Comparing two archives of the same image: the first block of an image of
the used clusters holds the signature of the disk, new at every run, so
its cuts can differ, and the dates differ. Compare the fragments as a set
(dump -verbose), not by position: 10 fragments in 3,760 differ, the ones
with dates and signature.

7. Sparse while written

trysparse() takes a virtual disk too when extract() asks
(job.vhdsparse): whatever its size, unless -nosparse, -huge, -715.
After the threads, toglisparse() for every virtual disk extracted:
FSCTL_QUERY_ALLOCATED_RANGES gives what is allocated, what is in between
is filled with zeros, then FSCTL_SET_SPARSE with SetSparse = FALSE,
then date and attributes again (the zeros changed the date), before the
read-only attributes. Windows 11 would allocate the holes by itself when
the attribute is cleared; filling them first works on every Windows and
has a progress. If it fails (disk full: a sparse file does not reserve its
space): 56487-56491, exit code 2, and the file is said to be still sparse.
A file not complete (a damaged archive) is still open by the job: closed
first.

8. -huge

It kept the position of the last write in the job, not in the file: after
the first file the filling started from a stale place, and usually not at
all. Content always right, purpose lost. Now job.hugefile and
job.last_write are set when a file is opened (0 for a new one, its size
for one opened again: the same file is closed and opened dozens of times
when fragments are shared). Every gap is written, not only the ones above
1 GB. hugezeri() writes 1 MB at a time under the write mutex.

The write of a fragment now checks its result in both ways (plain and
-huge): the final "written against expected" check of zpaqfranz is
skipped with -image on Windows, so a full disk went unnoticed there.

9. image -raw: the zeros

restoreimage() sends to the old branches only with -ntfs.
restoreimageauto() remembers -raw and turns flagraw off: for the
engine (extractstdout) that flag means "this is a raw image" and picks
another writer. In the thin restore m_estraithin.finoa is how far the
destination is written; before each block zeriestrai() writes zeros from
there to the block (4 MB at a time, an aligned buffer: the destination is
opened unbuffered), and at the end up to the size of the partition that
was imaged.

10. franzmonta

virtdisk.dll loaded at run time from System32 (no import library, its
types written in the source): OpenVirtualDisk with read access,
AttachVirtualDisk read-only and not permanent, so Windows detaches
when the handle is closed, however the program ends. The volumes of the
disk are found by device number (IOCTL_STORAGE_GET_DEVICE_NUMBER); a
letter is given with SetVolumeMountPoint. Ctrl+C is caught to detach
instead of dying. The elevation is ShellExecuteExW with runas, the
same arguments, -pause and a hidden -elevated: without a desktop, or
with UAC off, runas starts the program not elevated, which started
itself again, one process every five seconds (85 before they were stopped
by hand). The dispatch is outside the markers of the open build.

11. Fixes

• Raw restore on the source (65.6 too): extractstdout writes on
  lettera, which was the source asked to restore, not -to.
  restore_raw_to_disk sets it to the destination.
• The restore handlers took every .fhd (or every file) of the archive:
  with images of more drives, all on one destination. Only the chosen one
  now (g_immaginescelta).
• The end of a bigger destination: the last MiB after the image is zeroed
  (azzeracoda), and the last unit of the partition is always in the image
  (the backup boot sector of NTFS goes in the .vhd and on the disk).
• The count of the used blocks after the exclusions counted a short last
  block (blocks are whole now): the expected size could be wrong.
• The static counters of elaboravhd were cleared only with -verbose.
• getfreespace(): the folder of f:/x.vhd was cut to f:, which is not
  a folder, then to nothing, then to .: the free space of the current
  folder. A drive is now its root, and nothing above it. The hint "-to is
  a single file, not a folder" is not shown for a .vhd.
• franzVSS is a global: initWMI() failing released an interface and
  did not clear the pointer, the cleanup released it again. A crash at the
  start of every command for a user without WMI.
• image to a raw file: one path overwrote an existing file without
  asking (CREATE_ALWAYS), the other left it there and said nothing. Both
  ask for -force now.
• The warning "without VSS the pagefile cannot be read" was for the letter
  C: it is for the drive of Windows.

Built and tested on Windows (g++ 14.2 ucrt64: full, open, with the mount)
and Fedora 44 (gcc 16.2); autotest 16/16 on all. The source is 230,351
lines (8.3 MB). The batteries on the VM, with every .vhd really mounted
by Windows and its files compared with the source: images and restores of
NTFS, FAT16, FAT32, exFAT, raw partitions and a whole disk (51 checks),
swap files, -raw, a tree of virtual disks extracted on NTFS and on exFAT
(21 checks, and 51 more on each tree), the same fragments with and
without -turbo (12), image -raw on a destination with old data in its
free space (20).

12. Tried and thrown away

A fixed-size fragmenter for the images (-fix N, a branch of its own
for a few hours): the 512 bytes of sector bitmap as a fragment, then the
2 MB of the block cut in pieces of N bytes, so that a piece is always the
same clusters. On the test VM, with synthetic data, 64 KB looked better
than the cut on the content. On a real C: (the author's, first image):

Worse at every size (and 4 KB costs 4 bytes of index for every fragment,
at every version). Removed. Random data with a few changes say nothing
about the deduplication of a real disk.

SetFileValidData to write a not sparse file in any order without the
filling: it works (NTFS and exFAT), but it needs an administrator, and an
extraction that stops halfway leaves in the file whatever was on the disk
before. Not done.

13. Known limits and open points

• VHDX is read (mounted), not written: a source over 2040 GB or with
  sectors that are not of 512 bytes cannot become a virtual disk. To a
  partition, or to a raw file.
• Not tested: 4Kn sources, volumes over 2 TB, FAT on 4Kn disks, exFAT with
  32 MB clusters; the elevated window of mount with a real UAC question,
  and its key and Ctrl+C (the tests run over ssh, where there is no
  desktop); .iso.
• Reading in parallel (-ssd for an image) is not done. It would help
  the second image, where the drive is the limit, and only as a read-ahead
  that hands the blocks over in order: an image stored out of order is the
  very problem of section 7.
• A .vhd whose extraction is killed is left sparse and incomplete.
• Out of Windows -huge is only on request: there a hole costs nothing
  and a .vhd with holes is fine.
• Files left out with -not are in the .vhd, with zeros inside (they are
  deleted only by the restore on a partition).
• The three writers of add2() (plain loop, -turbo on files, -turbo on
  images) share the code that stores a fragment but not the one that reads:
  to be joined.
• The limits of 65.6 that are still there: the Linux READ ERROR through
  the page cache, f -test on virtual disks, the Ctrl+C handler.