lifthrasiir/aqueous v0.0.20261110

Aqueous archiver 2026-11-10

on GitHub

Download and execute aqr-20261110-win-install.exe if you have no other reason.
Download aqr-20261110-win7.exe if you use Windows 8 or earlier.

Newly Supported Formats

• Disk images --- files are taken out of the file systems and recompressed individually
  • Partition tables: GPT, MBR (with logical partitions), APM
  • File systems: FAT12/16/32, exFAT, NTFS, ext2/3/4, APFS, HFS+/HFSX (in addition to ISO 9660 and UDF)
  • Virtual disks: VHD, VHDX, VMDK (hosted sparse), qcow2, VDI
  • Not yet: differencing disks, encryption, decmpfs (APFS/HFS+ compressed files)
• Compressed disk images --- chunks are rebuilt from the disk inside
  • Apple disk images (DMG/UDIF): raw, zlib (UDZO), bzip2 (UDBZ) and LZFSE (ULFO) chunks
  • qcow2 with compressed clusters, streamOptimized VMDK, OVA (through its tar member), EnCase E01
• NTFS compressed files (LZNT1) and Windows Overlay Filter (WOF) files (XPRESS4K/8K/16K, LZX)
• Zstandard (.zst) --- Portions only: frames are kept as is, but files contained in them (including tar members) are deduplicated

Changes to Existing Formats

• tar, ar and cpio
  • Text members now share one dictionary
• xz, LZMA and 7z
  • Streams of large dictionaries (up to 64 MiB) are recompressed within the memory bound
• Large text, CSV/TSV and raster images (PNM, BMP and the like) are coded in segments, so that their memory doesn't grow with the input
• Inputs too large for their recompressors are kept as they are instead of exhausting memory (see below)

Breaking Changes

• Archives made by this release cannot be decoded by earlier releases; archives of earlier releases are still readable
• -B and --batch are now one option taking a SIZE (512M by default); --batch previously took MiB
• Options are now parsed like getopt: short options can be bundled (-cf, -oOUT) and long options take --long=VALUE
  • A value given to an option that takes none is now an error
• Exit codes now tell what failed: 3 invalid archive, 4 not decrypted, 5 over the supported or allowed resources, 6 unknown critical extension, 7 done but something was left out (1 for other failures as before)

User-facing Changes

• Archives can be edited in place:
  • --delete ARCHIVE PATH... removes paths (by appending, like -a)
  • --compact ARCHIVE rewrites an archive without what is no longer needed, repacking solid blocks where files were removed
  • --move ARCHIVE PATH --rename TO... moves paths within an archive
  • Not for encrypted archives, archives of a single file and (--compact, --move) archives with mirror blocks
• - reads from standard input and -o - writes to standard output, so aqr can be used in a pipe
• --passphrase-from SOURCE reads the passphrase from stdin, a file, an environment variable (env:NAME) or a file descriptor (fd:N)
• -x|--exclude NAME leaves inputs out (no wildcards yet); --files-from FILE (with --null) reads the inputs from a list
• -t --json lists archives as JSON lines
• A manual page (aqr.1) has been added (not yet installed by the bundles)
• --setup-autocomplete[=SHELL] sets up shell completion for zsh, bash, fish and PowerShell
• Long names were added for options that had none: --output, --threads, --level, --batch, --block-size, --segment-size, --dictionaries
• Extracted files are now written sparse where the file system supports it
• Extracting or listing a few paths reads only the blocks and metadata needed, and decrypts only what it reads (several times faster on large archives)
• -t -v shows each file's segment correctly (it was always 1)
• -I -v shows each module application's memory limits
• --application-bound SIZE|unlimited limits the memory of a single module application
  • 256 MiB by default when compressing (a recompression over it is not used), 512 MiB when decompressing
• -M now also limits how many tasks of a module run at once when decoding in WebAssembly
• --stats improvements
  • The JSON and HTML files are written every 10 seconds and when interrupted, with state (live, interrupted, finished)
  • The HTML page reloads itself while the run is live, keeping the view
  • The JSON records the version of aqr and its arguments
  • Fallbacks (recompressions given up for memory) are reported, also with -v
  • --stats=FILE.html alone writes an empty page to open JSON files with

Specification Changes

• A single Apply instruction (opcode 16) with a memory limit for the instance and each class of tasks
  • Task classes, gang classes and channels owned by gangs; aqr_spawn takes regions to copy into the task
  • The decoder chooses the concurrency of each class, and MAY fail up front when the limits don't fit
  • The earlier Apply (opcode 11) is reserved, accepted only for modules of the earlier releases
  • Decoders MUST support applications of up to 512 MiB (the "minimal bound"); encoders SHOULD NOT go over
• Part modules may have windows smaller than their data, so that their memory doesn't grow with the data
• Module blocks have a split form (the streams of wasm-recons), about 13% smaller
• Recovery sets: Reed-Solomon parity over arbitrary byte ranges of the archive (parity blocks C1h--C3h, index blocks C4h--C7h; not implemented yet)
• Patch of an empty source with dropped pieces is clarified
• References to the design documents have been removed

Compression Optimizations & Changes

• Modules are smaller (an allocator of their own, and the split form of module blocks), which matters most for small archives
• zlib chunks of disk images (hdiutil, qemu-img, ewfacquire) are rebuilt at aqr-flate's exact zlib emulation where it predicts every token
• Disk image contents are found in linear time (quadratic in the number of files before)

Performance Optimizations

• Decoding memory no longer grows with the input
  • Formula blocks are evaluated as streams
  • Most modules of large streams (DEFLATE, ZIP, LZMA/xz, bzip2, LZ4, Snappy, BCJ/BCJ2, MSZIP, git packs, x86 and ARM64 executables, Mach-O code signatures, PNG, FLAC, PCM, MP3, FASTA, tar) are part modules that stream their inputs and outputs
  • Every module's memory is planned exactly instead of estimated, and native decoding is held to the same limits
• Compression memory has been reduced
  • Verification uses composable content hashes instead of keeping the modules' outputs
  • The spill is written in order and mapped read only
  • DEFLATE streams are written to the spill as they are parsed, and adaptive profiles no longer copy the tokens
  • Zstandard contexts are no longer kept per content
  • Each recompression runs under a heap allowance (-M, or half the system memory), the input kept as it is when over it
• Windows: the spill's freed pages are no longer written back to disk
• The verification of disk images and other formulas of many applications runs on threads
• The release aqr binary is about 5% smaller

Bug Fixes

• Crafted or damaged inputs no longer make the compressor or decoder abort by huge allocations (DEFLATE, ZIP, PNG, JPEG, executables, DEX, ARSC, DWARF, PSD, SQLite, RAR, LZFSE)
• The -c summary now counts hard links' bytes (the ratio was overstated)
• --stats no longer reports all memory as mapped files on Windows before 10 1809
• The PowerShell completion script no longer fails to parse
• Fix overflows in debug builds
• Fix clippy warnings and other code quality issues