zpaqfranz 65.12
DO NOT USE IN PRODUCTION: IT IS STILL TO BE STABILIZED
Two of the new things touch the core: the way every command reads an archive (less memory, and a report when blocks are missing), and a -fastadd, an update that
does not read the archive again. Aka: much faster add on archives with many versions
This is the release of the big archive. One archive of more than
1,300 versions, 5.6 million files and 7.6 GB, damaged in 2022, was the
bench of most of it:
- A damaged archive says what is missing: which version, which
blocks, how many files can no longer come out. Before: two lines that
only an expert could read. - Reading an archive takes much less memory: 8 GB down to 260 MB on
that archive. On a Linux with 4 GB,awas killed; now it works. a -fastadd: the update starts from a state stored in the archive
by the update before, instead of reading all the index again. On that
archive: 11 seconds down to less than one.ton a damaged archive said "VERDICT: OK" (exit code 1, a
warning), with every file corrupted. Now: CORRUPTED, exit code 2.-commentrevised: a comment is no longer counted as a deleted
file,ialways shows the comments,t -commenttests that version,
and finding a version by its comment no longer reads the whole archive
twice (12 seconds down to less than one).dumpmade again: the "virtual files" of zpaqfranz, the blocks of
every version, a verdict on what can read the archive.zpaqfranz archive.zpaqon a console istui archive.zpaq.work fat32: FAT32 up to 2 TB, and a USB stick that boots a WinPE
on an old PC (BIOS).autotest -all -toin five levels, from-m1(a few MB, a minute)
to-m5(everything). No-m: the suite of before, the same script.work poke: change bytes of a file, on purpose (for tests). Yep, poke like on C64 😄
This document has three parts:
- Part one: what is new, in short, for everybody
- Part two: the same things in more detail, for power users
- Part three: "lo spiegone", how it works inside, and why, for developers
Part one, what is new, for the user
- A damaged archive says what is missing
- Less memory to read an archive
a -fastaddton a damaged archive-commentdumpzpaqfranz archive.zpaqistuiwork fat32autotestin levelswork poke- Smaller things
1. A damaged archive says what is missing
An archive of 1,378 versions gave, at every l, t, x:
72403! Version 1314: index block 131, 1 expected (missing i block?)
72401! Version 1314: data without fragment tables (missing h/i blocks?)
What it means was not written anywhere. That version, made in April
2022, is broken on the disk: its data block stops in the middle, and
130 of its 131 index blocks are not there. Now, after the reading, the
same command says it in words:
72415! DAMAGED ARCHIVE: blocks are missing in 1 version(s): 1314 (2022-04-22 16:30:18)
72416! 1314: index blocks 1-130 not found: the list of its files is not complete (189 records read)
72418! 1314: 206.698 bytes of data have no fragment table: what is stored only there cannot be extracted
and how many files of the version read (or of all of them, with
-all) use the missing data:
None of the 3.432 files of version 1378 (the one read) uses the missing data: they can be extracted
The other versions are whole, the last one too. The exit codes are the
ones of before.
2. Less memory to read an archive
Every command reads the index of the archive first. For every file of
every version it kept two things it never gave back: an object for a
hash (only a needs it, and only for the files it writes) and 551
bytes of zpaqfranz attributes, also for the files that the command did
not want at all.
On that archive, on Windows:
| command | before | now |
|---|---|---|
i
| 10.7 s, 7.97 GB | 7.0 s, 247 MB |
t
| 16.9 s, 8.02 GB | 14.4 s, 286 MB |
a
| 15 s, 7.98 GB | 11.7 s, 258 MB |
l -nofast
| 13.5 s, 6.07 GB | 12.9 s, 284 MB |
On a Fedora with 4 GB a was killed (out of memory); now 10.3 s and
258 MB. On an OpenBSD it stopped with "cannot franz_malloc"; now 28 s.
Nothing to do: it is the way every command reads.
3. a -fastadd
zpaqfranz a z:\big.zpaq c:\data -fastadd
An a reads all the index of the archive before it adds anything:
with many versions and millions of files, that is most of the time of a
daily update. With -fastadd the update stores, inside the archive, the
state that reading gave; the next a -fastadd loads it and reads only
the last version.
-fastadd: from the stored state, the last version read (0.003 s)
-fastadd: state of 47.98 KB stored (47.98 KB new)
On that archive: 11.2 seconds and 258 MB for a normal a; with
-fastadd, 13.9 s the first time (it reads everything and writes the
state), then 0.83 s and 17 MB. On Fedora 13.5 s then 1.0 s, on OpenBSD
34.6 s then 1.7 s. What it costs: some tens of KB for every version (on
that archive about 29 KB), deduplicated.
It is not the default. It works on plain archives, -key, -franzen,
multipart. Anything strange (no state, a state of another version, of
a newer zpaqfranz, damaged): it says so, and reads the archive as
always. The archive stays a zpaq archive: zpaq 7.15 and older
zpaqfranz read it, and skip the state (zpaqfranz up to 65.12e says that
some fragments are used by no file, 72414: a warning).
What it does not do: a damage in the versions before the last one is not
seen any more by -fastadd, as it is not seen by the quick l of the
-fast history. t sees it.
4. t on a damaged archive
An archive with bytes changed in the middle of its data:
Checked 0 of 256 files OK (256 errors) using 16.845 MB x 16 threads
...
VERDICT : OK (CRC-32 stored vs decompressed)
0.031s (00:00:00,171.46KB) (with warnings)
and exit code 1. It was so since 63 at least: the final check (CRC-32)
does not see the files that could not be decompressed, and said OK on
nothing. Now:
Checked 0 of 256 files OK (256 errors) using 16.845 MB x 16 threads
01448! VERDICT : 00000256 file(s) CORRUPTED (not all their fragments decompressed)
WITH ERRORS
0.031s (00:00:00,171.46KB) (with errors)
and exit code 2. A good archive: as before.
5. -comment
zpaqfranz a z:\1.zpaq c:\data -comment "daily %date"
zpaqfranz l z:\1.zpaq -comment
zpaqfranz x z:\1.zpaq -comment "first copy" -to z:\restore
zpaqfranz t z:\1.zpaq -comment "first copy"
A version comment is stored as a record of a deleted file (zpaq 7.15
skips it). It was counted as one: i, and l -all, said "removed 1"
for every version with a comment, and listed a deleted
VCOMMENT 00000001 first copy:$DATA. Now:
V00000001 2026-10-18 18:00:22 +00000004 -00000000 -> 118.420 <<first copy>>
ishows the comments, also without-comment.t -comment texttests that version (it tested the last one, in
silence).l,x,twith-comment text: the version comes from the-fast
history, without reading the archive (it was read twice).l -comment
on that big archive: 12.5 s down to 0.7 s.- A comment that is not there: exit code 2 for every command.
a -commentwith no text: a warning (it was silence).a -715 -comment: an archive as zpaq 7.15 makes it has no comments;
the comment is not written, and it is said (it was written).- A comment longer than 255 characters no longer gives a false "long
filenames" warning; a text with:$DATAinside is kept as it is. - The help says how
%dateand the other ones are replaced.
6. dump
zpaqfranz dump z:\1.zpaq
dump shows the blocks of an archive. Now, after the blocks:
Virtual files (records of zpaqfranz that are not files)
Kind Records Bytes First ver Last ver
-fast history: pointer 2 168 1 2
VCOMMENT (version comment) 1 43 1 1
VFILE-info 1 100 1 1
VFILE-info, with -fastadd state 1 142 2 2
-fast history: 2 fragments (984 bytes, deduplicated) in 2 d blocks, 1.301 bytes in the archive (0.867%)
Blocks of the versions
All the 2 versions have their blocks in sequence (c, d and h in pairs, i from 1)
...
VERDICT: zpaq 7.15 archive with extensions of zpaqfranz 60+ (Jul 2024 and later)
Extensions: hashes and CRC-32 of the files (6, versions 1-2), -fast history: pointer (2, versions 1-2), ...
zpaq 7.15 lists and extracts it: the extensions are attributes it skips, and records it takes for deleted files
The verdict knows the plain zpaq 7.15 archive, the one with zpaqfranz
extensions (and from which zpaqfranz, by the kind of hash), Franzen and
Franzen with -key, multipart. A version with blocks missing is named,
and the exit code is 2. A block that cannot be read no longer ends the
dump: it goes on from the next good block.
7. zpaqfranz archive.zpaq is tui
zpaqfranz z:\1.zpaq
zpaqfranz z:\1.zpaq -all
are zpaqfranz tui z:\1.zpaq (and tui z:\1.zpaq -all): the archive
name first, and it opens. .zpaq and .franzen. Only on a console: in
a script, or with a pipe, nothing changes (tui would wait there for a
key forever).
8. work fat32
zpaqfranz work fat32 e: DIAG
zpaqfranz work fat32 3 WINPE -boot
Windows formats FAT32 only up to 32 GB; a WinPE on an old PC (BIOS), and
many other things, want FAT32. work fat32 makes it up to 2 TB:
X:a volume, orNa whole disk (a new MBR, one partition).- The label is optional; the cluster size is the one Windows would use.
- Only a USB or removable disk (
-debug: any). Never C:, the volume of
Windows, the disk that holds them, the disk zpaqfranz runs from. - Always a captcha, also with
-nocaptcha. -boot: a boot code that startsBOOTMGRon a BIOS PC. On a disk
with GPT,X: -bootmakes the whole disk MBR, if X: is its only
partition (a Microsoft reserved one is allowed); if not, it says which
partitions are there, and the command to use (work fat32 N -boot).
Made after a look at fat32format 1.07 (the program that everybody
uses): its FAT is one entry short in 0.8% of the sizes (Linux fsck.fat
refuses those volumes).
9. autotest in levels
zpaqfranz autotest -all -to z:\test -m1
zpaqfranz autotest -all -to z:\test -m5 -franzen pippo
autotest -all -to writes a script (dotest.bat, dotest.sh) that
runs many commands and checks their output at the end. Now in five
levels:
| level | what | data | free space |
|---|---|---|---|
-m1
| default values, x, the -fast history, VFILE-info, -comment, -fastadd, dump, difficult names
| a few MB | 0.2 GB |
-m2
| + 31,080 small files: hashes, -key, multipart, -chunk, backup -chunk, healing, -m6, -turbo, -tar (*nix)
| 31 MB | 1 GB |
-m3
| + 1.75 GB of folders: the hashes, encrypted -chunk, -frugal, -m8, -fastadd
| 1.75 GB | 10 GB |
-m4
| + 6 GB of big files: Franzen, mount, -turbo, backup -chunk
| 7.4 GB | 28 GB |
-m5
| everything: every method (t, and p -verify), -all, -715, damaged archives, -fastadd for 20 versions
| 7.4 GB | 32 GB |
No -m is -m4: the script of before, the same to the byte, plus some
new checks at its end. -quick is as it was. -checktxt checks the
level the folder was made with.
How long. Only as an order of magnitude, since it depends on the
hardware: on a modern machine -m1 takes about a minute, -m3 some
minutes, -m5 about an hour. On slower hardware, much longer.
How much memory. In -m5, a -m5 with 16 threads takes about 6 GB
of RAM. Do not run two of them at the same time on one machine (with
virtual machines on it too): that is how a test of 65.12 ended with
std::bad_alloc.
10. work poke
zpaqfranz work poke file.bin 3000 25 -force -kill
zpaqfranz work poke file.bin 3000:4000 27 -force -kill
As the POKE of the Commodore 64: the byte at offset 3000 (from 0)
becomes 25; from 3000 to 4000, both included, they become 27. Not past
the end of the file, which keeps its size; what was written is read
back. Without both -force and -kill it says what it would do, and
does nothing:
03486$ work poke <<n.zpaq>> offset 1.000: 7 => 0: nothing done (-force -kill to do it)
It is for tests: autotest -m5 damages an archive with it.
11. Smaller things
cloud -ignore: a warning of the update of the archive (a file
that could not be read) no longer makes the whole run fail.f -test -force -paranoidwith a file open on the volume: the lock
failed every time ("cannot be locked, error 5"). It works now.- The output of
autotest -checktxtis in columns. - Binaries: Windows, ESXi, Solaris, openSUSE, macOS (x86_64 and
arm64 in one file), and eleven for NAS (x86_64, i686, ARM v5, v7, v8,
Cortex A9, A15, A53, A55, A57, A72).
Part two, the details, for power users
- The damage report: line by line
-fastadd: when it is used, when it is not-comment: the rulesdump: what it sayswork fat32: the rulesautotest: the levels, step by step- What can change for a script
1. The damage report: line by line
A version is written as blocks: c (where it ends), d (the data),
h (the fragment table of each d), i (the index: the names, from 1
up). The report is made while the archive is read, version by version,
and written once at the end (on stderr with -stdout, as the other
warnings).
| message | what was seen | what it means |
|---|---|---|
| 72415 | the versions with damage (ten at most, then "...and N more") | |
| 72416 | index blocks not found (one, a range, or how many) | the list of the files of that version is not complete |
| 72417 | index blocks out of place | the list can be wrong |
| 72418 | data with no fragment table | what is stored only there cannot be extracted |
| 72419 | no index at all | its files are lost, their data is there |
| 72420 | fragments with no table, before or among the tables of the version | the files that use them cannot be extracted |
| 72421 | a block that cannot be read | what is after it, up to the next good block, is lost |
| 72422 | no file of the version read (or of all, -all) uses what is missing
| they can be extracted |
| 72423 | N files of M use what is missing | they cannot come out as they were; x -recover: what is left of them, shorter
|
The exit codes did not change. x -recover on a file whose fragments
have no table writes it shorter, without zeros: without the table the
size of the missing piece is not known.
2. -fastadd: when it is used, when it is not
The state is used only if:
- the last block of the archive is the pointer of the
-fasthistory,
and the block before it is the VFILE-info of the last version with a
state in it; - the state is of this archive as it is now: the version, where it
begins, how many blocks, a hash of the state itself; - the archive is read with the same
-key/-franzen.
If not, one line says why, and the archive is read all (65450 - 65456).
After a run without -fastadd, the next one with it reads everything
once more and stores the state again.
The state goes in the archive with every a -fastadd, deduplicated in
pieces of about 8 KB: a version that changes little costs little. The
files selected by the a (its paths, -only, -not) are chosen after
the state is loaded, as the normal reading does.
-fastaddtest (not in the help): the same, then the archive is read
again the usual way and the two states are compared byte by byte;
different is exit code 2. autotest uses it at every level.
3. -comment: the rules
- One text for one version: unique, case does not matter.
%date,
$dateand the other ones of-commentare replaced (see the help
ofa). - Stored as a record of a deleted file named
VCOMMENT 00000002 text:$DATA: zpaq 7.15 lists the archive without
it. zpaqfranz counts it as a comment (VER.comments), not as a
deletion:iandl -allshow it on the line of its version. l -comment(no text): the list of the comments.l,x,twith
-comment text: that version, as-until N; found in the-fast
history (the last block of the archive), else by reading the archive
as before.- Not found, or more than one: 00750, 00751 (
l), 03495, 03496 (t),
02956, 02957 (fzf);xstops with an error. Exit code 2 for all. - 03493:
-commentwithout a text. 03494:-715 -comment, not stored.
4. dump: what it says
In this order:
- the blocks (as before), and their count by kind;
- the virtual files: the records of zpaqfranz that are not files of
the user (the pointer of the-fasthistory and its stop with
-nofast, the pointer of the older-filelist, VFILE-info, VFILE-info
with the state of-fastadd, the version comments), how many, how
many bytes, from which version to which; - the
-fasthistory: of how many versions, its fragments, its
blocks, how much of the archive it takes; - the blocks of the versions: every version must have its
c, its
dandhin pairs, itsifrom 1; the ones that do not are named; - the zpaqfranz attributes (always counted now: before, only with
-all, and every archive looked like one of zpaq 7.15); - the verdict: encrypted or not (Franzen, Franzen + AES, AES only:
who can open it), multipart, plain 7.15 or with extensions (from which
zpaqfranz, by the kind of hash), the files without a zpaqfranz hash,
the list of the extensions, what zpaq 7.15 does with it.
A block that cannot be read: dump looks for the next block and goes on
(before it stopped there, and the versions after were not shown).
5. work fat32: the rules
work fat32 X: [label]: the volume X: becomes FAT32; on an MBR disk
the type of the partition becomes 0x0C, nothing else of the disk is
touched.work fat32 N [label]: the whole disk N (as inGet-Disk): a new MBR,
one primary partition, active, aligned to 1 MB, FAT32.-boot: the MBR and the boot sector of zpaqfranz, that load
BOOTMGRfrom the root (a WinPE: copybootmgr,boot,efi,
sources\boot.wimon it). Not on a logical partition, not with
sectors other than 512 bytes. On a GPT disk by letter: the whole disk
becomes MBR, only if X: is its only partition apart from a Microsoft
reserved one; else 69318, with the partitions there and the command
for the whole disk.- More than 2 TB (the volume, or the disk): refused.
- Refused: C:, the volume of Windows, the disk that holds them, the disk
zpaqfranz runs from, a disk that is not USB or removable (without
-debug). A volume in use is locked and dismounted first, asformat
does. - Messages 69300 - 69337.
6. autotest: the levels, step by step
autotest -all -to X writes the data (the folders 00-09, the big
files, a few small folders), the script, and the file autotestlevel.txt.
The script runs the commands, each with -out outNN.txt, then
autotest -checktxt X, which looks for one text in each output.
-m1:xandsum -renameof the 256 files ofsha256.zpaq, an
archive of two versions with the default values (a,t,v,xand
the files checked by their names),lfrom the-fasthistory, the
VFILE-info of the versions,l -comment,-fastaddtest, the verdict
ofdump, difficult names (UTF-8, emoji,%, spaces).-m2: five hashes over five folders (t -verify,v -all,
p -verify),-key(t,v,lof-fast,-fastadd), multipart
(t -all,lof-fast,-fastadd),-chunkplain and-key,
backup -chunkplain and-keywithtestbackup -paranoid -verify,
the backup in 16 parts of before, the hash healing (attributes,
-touch),-m6,-turboagainst-noturbo(the same file),-tar
(*nix).-m3:testme.zpaqand the encrypted-chunkof before (the ten
folders),-frugal,r,c,-m8,-fastaddover 1 GB.-m4: the big files, Franzen, the mount (if it works there),-m6,
-m8h19,-turbo;backup -chunkof big files, plain and-key;
-fastaddwith Franzen.-m5:-m0to-m5,-m7,-m9on the same data (a,t, and
p -verify, the other decoder),t -allof the archives with many
versions, an archive made by-715then by zpaqfranz, an archive
damaged bywork poke(tmust say CORRUPTED,x -recovermust say
DAMAGED), twenty versions of-fastaddtest, the mount with-all.
Measured on a Windows PC of 16 threads: -m1 34 s to prepare + 14 s,
-m2 46 s + 72 s, -m3 72 s + 5 minutes, -m5 (with -franzen) 1
minute + 58 minutes, 153 checks. As said in part one: an order of
magnitude, the hardware decides. The free space asked is the most the
folder takes during the run, plus a little.
The 34 seconds of every level are the check of the hashes in memory,
over 1.75 GB: the autotest -all of before, the same at every level.
7. What can change for a script
- Exit code 2 where it was 1:
tof an archive with files that
cannot be decompressed. - Exit code 2:
l,x,twith-comment textnot found or found
twice;dumpof an archive with blocks missing;autotest -all -to
with-mother than 1 to 5 (03491). - Exit code 0 where it was 1:
cloud -ignore, when the update of the
archive gave a warning (and nothing else went wrong). autotest -all -towithout-spacewants 28 GB free (it was 18,
too few: 27.6 GB were measured at the peak, with-franzen).
-checktxtprints the checks in the order of the script.iandl -alldo not count a comment as a deleted file, and
l -allno longer lists it.t -comment texttests that version (it tested the last one).a -715 -commentwrites no comment.zpaqfranz x.zpaqopenstui, on a console.- New messages: 01448, 03100 - 03121 (
dump), 03480 - 03497, 65450 -
65466, 69300 - 69337, 72415 - 72423. Gone: 03050, 03052, 03053 (the
dumpof before).
Part three, lo spiegone, for developers
read_archiveand the damage- The memory of
read_archive -fastadd: the state- The comments
work fat32: the boot codeautotest: one table- How it was tested
- Found on the way
- Known limits and open points
1. read_archive and the damage
read_archive already noticed the holes: the number of an i block
that is not the one expected (72403), a d without its h (72401). It
said it in the middle of the reading, block by block. Now every event
goes into seq_danno of its version (index blocks missing, a range of
them, out of place, data without a table, tables without an index,
fragments skipped before or among the tables, a block that cannot be
read), and seq_scrivi writes the summary once, at the end
(72415 - 72421). A jump of the numbers at the start of the version after
a "data without table" is the same hole, not another version.
Then, for the files of the version read (-all: of every version),
the fragments are compared with the ones that have no table: 72422 or
72423. seq_versionebucata gives 0, 1, 2 (no damage, damage that the
files do not use, damage they use).
The archive of the bench: version 1314, of 2022-04-22. Its c says
206,698 bytes, its d (200,240 bytes) breaks at the fourth piece of its
chain (a length of 6,342,921,596), then come i129, i130, i131. The
h and i1-i128 are not there. read_archive jumps (by the c)
into i130 and reads only i131: 189 records of about 25,000. It was
so already on 2022-04-24: version 1315 deletes exactly those 189.
2. The memory of read_archive
For every record of every version, read_archive built a DT: its
constructor made an object for the hash (pfile_*, by new) and asked
551 bytes for the zpaqfranz attributes (franz_malloc). Neither was
ever given back, also for the records that isselected throws away
(for a with a path: almost all of them).
Now:
- the record's
DTis built light (g_dtlight,g_dtnoblock): no
hash object, no block; - the attributes stay where they are, in the
iblock being read;
dt_inserisci()copies them only for a record that goes intodt,
and into the block of the name if it is already there (that block was
lost before); - the copy is as long as the block (with
-frugalthe block is 51
bytes, and up to 550 were copied: an overflow); dt_franzdimensione()is the one rule of the size, the one of the
constructor.
The hash objects of a DT are needed only by a, for the files it
writes (edt): checked. With -stat the old way is kept.
3. -fastadd: the state
At the start of its version, a -fastadd serializes what read_archive
gave (sc_serializza): "ZPSC" 1, then LEB128 numbers (zigzag where
they can be negative):
x files dcsize dhsize nver nblock nht ndt
ver[1..] date lastdate offset data_offset csize updates deletes firstFragment usize
block[0..] offset usize bsize start frags
ht[1..] usize sha1[20]
dt namelen name date version attr[8] franzlen franz[] ni ptr[] (delta)
dt has the files alive and the version comments (they are deletion
records, but a -comment must see them: unique). The blob goes into
d blocks of the archive, deduplicated with fragments of about 8 KB
(SC_FRAGMENT 3; the -fast history uses 64 KB), and the place is a
key sc in the VFILE-info record of the version
(|sc=1;size;xxh64;blocks;intervals): an unknown key, skipped by the
others.
The next a -fastadd: the pointer of the -fast history is the last
block, the VFILE-info with sc the block before. sc_caricablob reads
the fragments of the state, checks size and hash, sc_deserializza
rebuilds ver, block, ht, dt; read_archive goes on from the
start of the last version only (sc_riprendida). The selection of the
a is applied after (sc_nofiltro, then sc_filtra). The -fast
history is taken from the archive as it is.
zpaqfranz up to 65.12e does not know sc: for it the fragments of the
state are fragments used by no file (72414, a warning).
4. The comments
vcomment_leggi(name, &version, &text): a record named
VCOMMENT NNNNNNNN text:$DATA (only the last :$DATA goes). In
read_archive such a record makes --deletes, ++comments on its
version, and goes into commentiversioni; the long name check is
skipped for it. The VFILE-info check compares its id with
deletes + comments.
versionedicommento(text): the version of a comment before the archive
is read for the command. From fl_commenti (the comments in the -fast
history, the last block of the archive) if there is a history; else by
reading the archive, as before. l -comment without a text uses the
same history.
5. work fat32: the boot code
Three arrays, with their nasm source in a comment next to them:
g_fat32_mbr(440 bytes): the MBR code. It finds the active
partition, loads its first sector, jumps.g_fat32_boot(1,024 bytes: sector 0 and sector 2 of the partition):
the boot sector reads the FAT32 root, findsBOOTMGR(8.3 name), follows
its cluster chain, loads it at 2000:0000 and jumps. Messages "BOOTMGR is
missing", "broken FAT chain", "too big".g_fat32_noboot(512 bytes): without-boot, a boot sector that only
says "Not a bootable disk".
The FAT: fat32_forma (cluster size by the table of Microsoft KB140365,
FAT size computed so that every cluster has its entry), fat32_disponi,
fat32_scrivi. The volume is locked and dismounted with
franzusb_bloccauno, which opens the handle again after a forced
dismount (the old bloccavolumi used the same handle, and failed with a
file open: that was the defect of f -test -force -paranoid).
6. autotest: one table
The script and the checks were two lists, written by hand, kept in
step by hand. Now one table, g_autotestpassi: for every step its
level, its conditions (Franzen, mount, zstd, Windows, *nix), how many
times, the command, the output to check, the text to find, a name.
%E (zpaqfranz), %D (the folder), %S (the separator), %F (the
Franzen password), %N (the number of a step repeated) and a few more
are replaced in one pass (autotest_espandi: a folder with a % in its
name is not expanded again). A step without a command is one more check
on the output before. -checktxt reads the level from
autotestlevel.txt (none: 4, a folder of before) and walks the same
table.
The rows of -m4 were made from the script of 65.12g, line by line
(trailing spaces too): the script of 65.12i without -m is that one,
byte for byte, plus 61 lines before the final check.
7. How it was tested
In tortura/dump, tortura/memoria, tortura/fastadd,
tortura/comment, tortura/poke, tortura/autotest, tortura/fat32.
- The damage report and
dump(t12d.sh): archives of every kind
with blocks cut out (taglia.py), 46 checks. - The memory (
eq12e.sh): 65.12d against 65.12e on the same
archives, 64 checks (l -nofast,-all,i -verbose,t,xfile
by file,a,-touchon 7.15,-key,-franzen, multipart,
-frugal); measured on Windows, Fedora, OpenBSD. -fastadd(t_fa.sh): the same updates with and without, on
plain,-key,-franzen, multipart archives, 12 checks;
-fastaddteston the big archive on Windows, Fedora, OpenBSD.-comment(t_comment.sh): 32 checks; 16 of them fail on 65.12f.work poke(t_poke.sh): 25 checks.work fat32: on virtual disks of a Windows 10 (MBR, GPT, logical,
from 33 MB to almost 2 TB, 2.05 TB refused, a file open),fsck.fatand
Linux on the result, qemu for-boot.autotest: on Windows-m1(11 checks),-m2(36),-m3(57),
-m5with-franzen(153); on Fedora-m1to-m3(11, 38, 59); on
OpenBSD-m1,-m2(11, 38). The mount steps: on a Windows with
WinFsp.- Every build: Windows (ucrt64
-O1and-O3 -Wall, and the syntax
of open,-DANCIENT,-DNOEMAIL,-DSFTP, 32 bit, DLL), ESXi (gcc
3.4.6), Solaris 11.4, openSUSE, FreeBSD 14.2 (plain and mount), OpenBSD
7.9, macOS 12.7 (x86_64, arm64, macFUSE, FUSE-T), Fedora (g++ 16,
-O1,-O3,-std=c++11,-DANCIENT,-DNAS), the eleven NAS
binaries;autotestandautotest -allwhere the binary runs. Only
the warnings already known.
8. Found on the way
- *65.12g did not compile on nix: after
-comment, anelsewith
nothing after it (on Windows its statement was inside#ifdef _WIN32). Before 65.12g thatelsetook, on *nix, theif (all)
that came after it:l -all -statgave no version numbers there. Now
the same on every platform. dumphad a buffer of 256 bytes for the zpaqfranz attributes,
which are 550 with-tar: an overflow.- Message codes: five codes of 65.12g (
-comment) were already used
by other messages. Now 03493 - 03497. - A test can fail because of another one: two
-m5at once on one
PC, with virtual machines, ran out of memory.
9. Known limits and open points
work fat32: disks of 4 KB sectors, superfloppy
(no partition table): not tried (but who cares)-fastadddoes not see damage in the versions before the last
one; the quicklof the-fasthistory says "all OK" on a version
with blocks missing (known, still open).tandl -nofastsee it.x -recover: the files with fragments without a table come out
shorter, not with zeros.autotest: the mount steps run only where the mount works (on
this Windows only with WinFsp);-m5wants about 6 GB of RAM with 16
threads, and 32 GB of disk.-chunkwith-franzen: still refused.