zpaqfranz 65.13

STILL TO BE STABILIZED, BUT BETTER

This is a fixing release.

I'm removing a few bugs and, above all, removing limitations, that is, parts of the code that were never really finished.
It's starting to look like something usable.

By the way, I don't expect any major bugs; in fact, I think I've reduced their number in key areas. Clearly, user feedback is essential for improving the program.

• k could delete the files that ARE in the archive: with one /
  more (-to z:\knb\, as the TAB of Windows writes it) or another case
  on Windows, every file of the -to folder was "to be killed", the
  folder itself too.
• trim -kill on a -chunk archive with an interrupted update
  emptied the part that held the good versions too.
• cp of a file on itself emptied it, and password, m, trim,
  crop, x, -out could write their output on their own input. Now
  never: a command refuses when its output is its input (the same file on
  the disk, whatever the spelling).
• crop of a Franzen archive copied the first bytes of the encrypted
  file and said "DONE": not an archive.
• password of a Franzen archive wrote, beside the .franzen, a
  .zpaq with AES only (or all in clear with -key2 .).

And the new things:

• -chunk: no part bigger than X, and -chunk with -franzen.
• cp -verify: written, read back from the media (not the cache) and
  checked piece by piece; one file on a console shows a map.
• password -franzen2: a new Franzen password.
• -dateto 2017 is up to the end of 2017 (it was its first second),
  and the dates are typed in local time, as they are shown.
• monitor reads the NTFS journal from where it stopped, and says
  what it does not know.
• autotest -m5 checks the multipart, -chunk and backup archives
  byte by byte against the archive in one file (work quadra).

This document has three parts:

• Part one: what is new, in short, for everybody
• Part two: the same things in more detail, for power users
• Part three: "lo spiegone", how it works inside, and why, for developers

Part one, what is new, for the user

1. Never an output on its own input
2. k
3. trim
4. crop
5. -chunk: no part bigger than X, and with -franzen
6. cp -verify
7. password, and password -franzen2
8. The dates: -datefrom, -dateto
9. find and dirsize
10. sync, 1on1, n
11. m and consolidate
12. monitor
13. s -home /os
14. autotest -m5 and work quadra
15. Smaller things

1. Never an output on its own input

zpaqfranz cp z:\data\a.txt -to z:\data

copied a.txt on itself: the file was emptied. The same family:

"The same file" is the same file on the disk: another case on Windows, a
relative path, ./, a hard link, the .franzen of the archive: all seen.

2. k

zpaqfranz k z:\1.zpaq c:\z -to z:\knb

deletes, in z:\knb, what is not in the archive. With -to z:\knb\, or
c:\z\, or Z:\Z on Windows, nothing matched:

Dir  to be killed z:/c5/knb/
File to be killed z:/c5/knb/a.txt
File to be killed z:/c5/knb/sub/b.txt
...

all the folder, the files of the archive too, and the folder itself. The
captcha shows numbers, not names: typed, everything was gone (seen on a
Fedora). Now the names are compared from the two folders down, and on
Windows without case: only the files that are not in the archive, never
the -to folder, never a folder with files of the archive in it.

3. trim

• -chunk archives: an update interrupted in the middle left its
  transaction (the c block says -1) over many pieces; trim -kill
  renamed the piece with that c to .spaz and left it empty. That
  piece held the versions before it too: "not a zpaq archive". Now the cut
  is where the reading stops: that piece is written again up to there,
  the pieces after it are renamed .spaz, nothing is deleted.
• On *nix trim "x_????.zpaq" said "COMPLETE" for every part: the
  check was Windows only. Now on every system.
• After a good trim -kill or trim -to the run ended with the
  "incomplete" banner, "use the trim command ASAP" and exit code 2. Now 0.
• trim -to copies only the good part (it copied all the archive,
  then cut it).
• A good archive ("TRIM seems not necessary"): exit code 0 (it was 1).
• A multipart with -to: refused (-to was ignored).

4. crop

• A Franzen archive: refused. crop x.zpaq.franzen -until 2 -to y -kill
  copied the first bytes of the encrypted file ("DONE").
• Free space: it wanted the space of all the archive (and refused,
  with that number); now the space of what it copies.
• -force with -to: it cropped in place all the same. Now in place
  only without -to.
• A crop of an archive with an interrupted update: exit code 0 (it was 2).

5. -chunk: no part bigger than X, and with -franzen

zpaqfranz a "z:\x_????.zpaq" c:\data -chunk 100m
zpaqfranz backup z:\x.zpaq c:\data -chunk 100m -franzen pluto

The parts could be bigger than X (a buffer of 64 KB more, and the 32
bytes of the salt). Now no part is bigger than X, and every part of a run
but the last one is X exactly. -chunk under 4 KB: refused. -chunk with
-franzen (with or without -key) works: every part is a .franzen file,
no part bigger than X on the disk.

On FreeBSD and macOS, a -append -key on a new archive left a file of 0
bytes ("cannot read salt"), and every run after it stopped. Fixed.

6. cp -verify

zpaqfranz cp k:\*.mp4 -to z:\copy -verify

-verify read the copy back from the cache of the computer, not from the
disk; with -append it said "does not match" always; a close that failed
(disk full) was not seen. Now the copy goes by pieces (the smaller of 1 GB
and 10% of the file; -n X: X% exactly): each piece is written, sent to
the media, dropped from the cache, read back and compared. A piece that
differs is written again, up to three times; then an error, and the copy
is cut at the last good piece. One file on a console shows a map of the
file (written, verified, wrong, to do).

cp -append: a destination bigger than the source, or whose first bytes
are not the ones of the source, is an error (it was deleted).

7. password, and password -franzen2

zpaqfranz password z:\f.zpaq z:\g.zpaq -franzen old -franzen2 new -verify

A new Franzen password. The franzen file is read block by block with
the old password and written again with the new one (a new salt, new
nonces). The archive inside is not touched (AES inside stays AES: an
AES+Franzen archive stays AES+Franzen), nothing goes to the disk in
clear. -verify reads all the output back with the new password and
compares it. One password at a time: -key2 and -franzen2 together are
refused.

The rest of password:

• -franzen (-key2 only): the output was the .franzen AND a
  .zpaq beside it (AES only, or all in clear with -key2 .). Now the
  .franzen only, as a does.
• After a password added or removed, t said "VFILE-info: version 1
  begins at byte 32, it was written at 0 ... NOT as they were written",
  exit code 2, on a good archive (the 32 bytes of the salt). Fixed.
• The new password typed (no -key2) is asked twice; an Enter with
  nothing is refused (it was an output with no password).
• The help said "By default DRY RUN": there is no dry run.

8. The dates: -datefrom, -dateto

zpaqfranz find . -datefrom 2017 -dateto 2017
zpaqfranz l z:\1.zpaq -datefrom 2022-10-01 -dateto 2022-10-01

found nothing: -dateto 2017 was 2017-01-01 00:00:00. Now -dateto is
the END of its period (2017 = up to 2017-12-31 23:59:59, 202202 = up to
2022-02-28, 2024-02 up to the 29th), and both ends are included. For
a, l, find, dir.

The dates are typed in local time, as zpaqfranz shows them. A file
shown at 13:00 (12:00 UTC) was out of -datefrom 2017-01-01_12:30. With
-utc, the dates are UTC as before.

9. find and dirsize

• The totals were wrong: "Files 7 for 63.507 bytes" of 63,500 bytes.
  Two numbers in the same line used one buffer: 22 lines had it, in many
  commands (now fixed for all).
• find c:\uno z:\kb -only *franco* (the example of the help)
  searched only in c:\uno: the second folder was taken as a mask.
• The folders were counted as files: "Files 14 for ..., folders 10".
• find -n 10: "shown 10 of 250".
• dirsize marcus missed /tank/marcus/x/marcus/ and took
  /xmarcus/; it showed folders deleted in a later version; a folder not
  found was silence and "all OK" (now a warning and exit code 1, as with
  -force).

10. sync, 1on1, n

• sync: a file deleted in a later version was "only in the
  archive"; an empty file, or a folder, on one side only was listed and
  then "No delta (this is VERY GOOD)", exit code 0. Now a difference is a
  difference. An archive without hashes: "N file(s) checked by size only".
• 1on1: two copies of a file in the source, one in -deleteinto:
  on *nix the second delete failed ("CANNOT DELETE", exit code 2 after a
  good run). The exit code was the number of errors (256 errors = 0 on
  *nix): now 0 or 2. The dry run shows the space to be freed.
  1on1 -home: two folders with the same contents under other names
  were EQUAL: now the names count.
• n: a folder matching the * took one of the -n places (-n 4
  kept 3 files); n -n 3 with no files crashed; the dry run lists the
  files it would delete.

11. m and consolidate

• m of Franzen parts made a file that is not an archive ("Done"):
  refused. A hole among the parts: the parts after it were left out,
  "Done": refused.
• m: a write error, a different -verify: exit code 2 (it was 1), and
  the half output is deleted. -verify reads the media, not the cache.
• consolidate -to twice on the same name: the old backup lost its
  .txt (moved to .bak), then "guru". Now refused before anything.
• consolidate -destination on *nix copied instead of renaming;
  parts already there with the new name: refused before anything (the
  .txt could point to files that were not the copies).
• consolidate x.zpaq -recover: "Archive not found:
  x.zpaq_00000001.zpaq". Now it works with or without .zpaq, and
  without -force it makes only what is missing.

12. monitor

monitor (Windows, audit of the NTFS journal) read all the journal at
every scan (up to 1 GB every 10 seconds), lost the first record after
it started, said "Backup completed ... archived" (it archives nothing),
and its list was erased by the screen that came after it. Now: from where
it stopped, the changes grouped by file, the paths of now, the renamed
folders said (what is inside them is not listed), a journal that went on
without it said "NOT RELIABLE". -not and -only with wildcards.

13. s -home /os

zpaqfranz s c:\users -home /os

The folders by size, the smallest first (/od, /on too). The number of
files no longer counts the folders, the files of the folder itself have
their row, -verbose shows the names inside the folder.

14. autotest -m5 and work quadra

-m5 makes five versions of four folders in clear, AES, Franzen,
AES+Franzen, as one file, multipart, -chunk and backup, and checks
that the parts glued together are the archive in one file, byte by byte
(work quadra A B), and that one byte changed is seen. -debug4 is a
deterministic encryption (tests only, NOT secure): the same input, the
same bytes.

pp on a -chunk archive with a block across two pieces: "Incomplete
read". Fixed.

15. Smaller things

• The "incomplete" banner is shown once (it was at the reading and at the
  end).
• "founded" is "found" in 41 messages; some message codes used twice are
  new; no TAB in a line, no "inf files/s".
• password of -fastadd archives: the next a -fastadd says the
  stored state does not match, as a warning, and reads all (it was an
  error message).
• trim: a part that cannot be opened stops it (it was taken as
  incomplete, and renamed with -kill).

Part two, the details, for power users

1. Exit codes that change
2. Dates: the rules
3. Outputs and inputs: where it is checked
4. -chunk: the rules
5. cp -verify: the rules
6. password -franzen2: the rules
7. trim of a multipart: the rules
8. What can change for a script

1. Exit codes that change

2. Dates: the rules

• -datefrom X: the files from X on, X included. -dateto Y: up to Y, Y
  included.
• A date with fewer digits is a period: -datefrom takes its first
  second, -dateto its last (2022 = 2022-12-31 23:59:59, 202209 = up to
  2022-09-30, 2022-10-01 = all of that day, 2022-10-01_14 = up to
  14:59:59).
• Local time, with its summer time of that day; -utc (and -715): UTC.
• -timestamp and -touch are as before.

3. Outputs and inputs: where it is checked

One function, stessofile: the two names exist and are the same file
(on Windows the volume and the index of the file, on *nix the device and
the inode). An output that is not there yet is never its input.

• password: source and output (also as .franzen).
• m: the output and every part.
• trim -to, crop -to, x -repack: the output and the archive.
• x -force: every file it writes whose name is the name of a file of
  the archive (a part, its .franzen): checked on the disk, not written.
• -out, -errorlog: the archive.
• cp and every copy (filecopy): the source and the destination.
• consolidate -to and -destination: they already refused an output
  that is there.

4. -chunk: the rules

• No part bigger than X; every part of a run, but its last one, is X.
• Every run (every a, every backup) starts a new part.
• The first part keeps the header of the archive and the salt: X at least
  4 KB (03548).
• With -franzen: on the disk, a part is X at most (the header of 144
  bytes, the footer of 84, 28 bytes every 64 KB block are counted).
• An update interrupted on a -chunk archive: a refuses to go on
  (69068); trim "x_????.zpaq" -kill cuts it.

5. cp -verify: the rules

• Pieces: the smaller of 1 GB and 10% of the file; -n X (1-100): X% of
  the file, no limit. Files under 100 MB (or under -chunk X): one piece.
• A piece that differs: written again (three times at most), then an
  error and the destination cut at the end of the last good piece.
• -append: the part already there must be the start of the source (a
  quick hash), else an error and the file is not touched; with -verify
  the last piece already there is read back too.
• Not cached on *nix (posix_fadvise) and Windows (a flush and a new
  handle without buffering); on macOS the read back can come from the
  cache. A cache of a file server: what can be done.

6. password -franzen2: the rules

• -franzen OLD is needed; -key2 together: refused (one at a time: the
  AES key with -key -key2, then the Franzen password with -franzen2).
• The names: f.zpaq (its f.zpaq.franzen is read) or f.zpaq.franzen.
• The output: refused if it is there (-force to overwrite), refused if it
  is the source, deleted if anything goes wrong.
• At the end the output must open with the new password, with the same
  size and the same kind; -verify reads it all and compares an XXH3.
• Multipart and backups: not done (a part alone would not be the
  backup).

7. trim of a multipart: the rules

• The parts are read as one stream, that stops at the incomplete
  transaction.
• Parts of one transaction each (the multipart of a "x_????.zpaq"): as
  before (the incomplete part renamed .spaz, the parts after it left
  there: "extract the data ASAP").
• Otherwise (-chunk): the piece with the cut saved whole as .spaz and
  written again up to the cut, the pieces after it renamed .spaz.

8. What can change for a script

• -datefrom / -dateto are local time: a script that gave UTC must
  add -utc. -dateto is the end of its period.
• The exit codes of the table above.
• "found" instead of "founded" in 41 messages; the codes 80829 (Only
  INT), 80831 (No difference), 80843, 81541, 03291, 03293, 02832 have
  new numbers where they were used twice (03589 - 03597).
• find counts files and folders apart; find -n says "shown N of M".
• n in dry run writes "to be deleted <>" for every file.
• dirsize: a parameter matches a folder name (marcus is not xmarcus).

Part three, lo spiegone, for developers

1. Two numbers, one buffer
2. stessofile
3. trim and the stream
4. VFILE-info and the salt
5. -chunk: scrivipezzi
6. franzcopia
7. -debug4
8. setpassword_franzen
9. How it was tested
10. Found on the way
11. Known limits and open points

1. Two numbers, one buffer

migliaia() writes in one static buffer, from the right. Two calls in
the same printf give two pointers into it: the second number is written
over the last digits of the first ("63.507" for 63,500 with 7 files).
Now migliaia() and tohuman() turn on 8 buffers. migliaia2() ...
keep their own buffer, as before.

2. stessofile

Near exists(). Windows: CreateFileW with access 0 and
FILE_FLAG_BACKUP_SEMANTICS, GetFileInformationByHandle, the volume
serial and the 64 bits of the file index. *nix: stat, st_dev and
st_ino. stessoarchivio tries the four pairs with the .franzen. In
x the name is compared first (a set of the names of the files of the
archive): the disk is asked only when a file has that name.

3. trim and the stream

fix() (the trim of a multipart) took every part as one transaction.
trim_pezzi() reads the parts with read_archive: it stops at the c
with -1 ("roll back to here") and gives that offset of the stream. The
parts are "of one transaction each" if the starts of the transactions
(the ver[] offsets and the cut) are the starts of the parts, and no part
after the cut is without a c block (is_incomplete_trans counts them in
g_trim_cvisti: with -chunk every a starts a new part, so a first
version of one part gives the same starts as a multipart). Else the cut.
is_incomplete_trans was all inside #ifdef _WIN32; a part that cannot
be opened gave return 2 from a bool (= incomplete).

4. VFILE-info and the salt

The record of a version says where it begins (ps), with the 32 bytes of
the salt of -key in it, and it does not say if the archive was
encrypted. password adds or takes away those 32 bytes in front of every
version. vinfo_controlla takes a difference of exactly +32 (now with
-key) or -32 (now without) as the salt, not as another archive before
it. A version 1 longer by 10 or by 32 bytes (plain), with the version 2
after it, is still said. Bytes put between two versions do not move
ver[].offset (it is where the last block ended): that check never saw
them.

5. -chunk: scrivipezzi

OutputArchive::flush() with parts: the buffer (already encrypted at its
global offset: AES does not change) is cut where the part is full;
nuovopezzo() opens the next part only when there are bytes for it (no
empty last part). The first part of the run stays open for the header of
104 bytes written at the end. With -franzen: franzen_capienza(X) is
the room for data in a part of X on the disk, nuovopezzofranzen()
closes the full part (not the first one) and opens the next with the salt
of the first, g_p_franzenprimo keeps the first one open for the header.

On BSD, "ab+" creates the file: a new archive with -key read its salt
from an empty file. Now ab+ only if the file is there, and a seek to 0
before reading the salt (on BSD ab+ is at the end).

6. franzcopia

cp -verify and the map: class franzcopia + cp_copia(). A piece:
written (XXH3 of the source on the fly), fill_sumedia (flush, fsync /
FlushFileBuffers), fill_scordacache (out of the cache), read back,
compared. The map is the one of f (franzdash, not in ANCIENT: there,
the line). The hidden -cpfake "piece[:times],..." spoils a byte of the
destination after that piece: for the tests.

7. -debug4

It wrote a copy of every buffer on z:\ (author's debug: now commented,
//fika). Now it is a deterministic encryption: the AES salt is zeros,
the Franzen salt fixed, every Franzen nonce from a counter (in the
threads of encode_parallela, from the number of the block). The same
input, the same bytes: the parts of a -chunk can be compared with the
archive in one file. NOT secure: tests only.

8. setpassword_franzen

Inside #ifndef NOFRANZEN / ZPAQFULL. A franzcri opened on the source
with the old password, one created with the new one and the aes_too of
the first (FRENZEN stays FRENZEN); read_at_to by 1 MB, write;
g_franzen_scrittirotti before and after the close; a third franzcri
opens the output with the new password and checks size and kind. The
archive inside is never read: the AES key is not needed.

9. How it was tested

• Batteries (bash, in tortura\): dirsize 47, trimcrop 84 (with
  an archive of zpaq 7.15 killed in the middle of an update, the only way
  to get a real c = -1), cinque (i, sync, 1on1, k, n) 85,
  consolida 57, password 35, approvati 51, franzen2 34 (the
  archive decrypted before and after: the same bytes), all of them on
  65.13n; stretto 426, identici 374, quadra 60, cp_verifica 17 on
  their builds (65.13d - 65.13g). Every battery against the build before
  too: the defects seen, then gone.
• Windows here, Windows 11 (a PowerShell twin: 25), Fedora
  (every battery), FreeBSD, OpenBSD, macOS, Solaris, ESXi, openSUSE (a
  POSIX sh smoke test: 26, or 18 where Franzen is not there).
• Every build: Windows (ucrt64 -O1, -O3 -Wall, open), ESXi (gcc
  3.4.6), Solaris 11.4, openSUSE (plain and mount, no warnings), FreeBSD
  (plain and mount), OpenBSD 7.9, macOS 12.7
  (x86_64, arm64, macFUSE, FUSE-T), Fedora (-O1, -O3, -std=c++11,
  -DANCIENT, -DNAS), the eleven NAS binaries; autotest and
  autotest -all where the binary runs; autotest -m2 38/38. Only the
  warnings already known.

10. Found on the way

• migliaia() twice in a line: 22 lines (section 1 of part three).
• tohuman() the same, in the threads of the hash.
• truncate() on Windows left its handle open (and the file locked)
  when SetEndOfFile failed.
• is_same_path on *nix compared the two files, not their folders:
  never the same (consolidate -destination copied).
• rebuildbackupindex: the name with its .zpaq before
  _????????.zpaq.
• a of zpaqfranz, on 1.2 GB with -m3, writes nothing on the disk
  for the first 25 seconds of 48 (5.5 GB of RAM): an interrupted update
  leaves nothing to trim. Seen, not touched (decided: not worth the risk).

11. Known limits and open points

• trim of a multipart where the incomplete part is not the last one
  and parts of one transaction each: as before, "extract the data ASAP".
• FreeBSD: a static build does not load libsodium.so (dlopen):
  there, Franzen only with a dynamic build. As before.
• cp -verify on macOS: the read back can come from the cache.
• monitor: what is inside a renamed folder is not listed.