Serious Bug in All Previous Releases
In #350, cipriancraciun started a discussion that got me thinking about the file scanner class. At some point, I realized that there might be a bug, and, after looking at the code, it turned out that there was one: when collecting hard-linked files, the class did not take the device these files were on into account.
When --file-hash is set to anything except none, the issue is triggered if and only if all of the following are true:
- The input to
mkdwarfsspans more than one device (i.e. mount point). - There are regular files with the same inode number on more than one of these devices, and these files are part of the input to
mkdwarfs(e.g. not filtered out). - At least two files in such a set of files with identical inode numbers also have a hard link count greater than 1 on their respective devices.
For any set of files with identical inode numbers and hard link counts greater than 1 for which all of the above conditions are true, only one inode will be chosen to represent all files. That means the data for all other inodes in the set is lost and will not be present in the resulting DwarFS image.
When --file-hash is set to none, the issue is triggered regardless of condition (3) above. In this case, however, mkdwarfs is guaranteed to crash with an assertion if it runs into the issue:
$ mkdir data && echo "hello" >data/x
$ mkdwarfs -i data -o data.dwarfs
$ mkdir -p mnt/a mnt/b
$ dwarfs data.dwarfs mnt/a && dwarfs data.dwarfs mnt/b
$ mkdwarfs --file-hash=none -i mnt -o /dev/null --force
[...]
Assertion `!files_.empty()` failed in /home/mhx/dwarfs/src/writer/internal/inode_manager.cpp(212): inode has no file (any)
So the default, hash-based deduplication mode is much more dangerous because it fails silently, but I hope that condition (3) is rarely true in practice.
The fix is 3c15ab2, along with a dedicated test. I strongly recommend upgrading to this new release if your input to mkdwarfs spans multiple devices.
Bug fixes
-
mkdwarfsdid not correctly handle inputs where hardlinks had the same inode number on different devices. To run into this issue, you would have to makemkdwarfsscan files from multiple devices (e.g. the root of a directory tree with multiple mounted filesystems) and have files with the same inode number on different devices and have at least two of those files also have a link count greater than 1. While this is hopefully rare in practice, it is a serious bug that can lead to crashes (in the best case) or even data loss (in the worst case), as only the data of one of these files would be stored in the image. This has been fixed and a test has been added to cover this case. -
A missing dependency was causing linker errors with shared library builds on macOS. This has been fixed.
Build
-
The static release binaries are now all built using Clang and link-time optimization. This was previously not the case for some architectures due to bugs in the toolchain. As a result, the binaries are now significantly smaller.
-
There is now a new set of binaries (
dwarfs-universal-small) that are built without brotli support and without support for the performance monitor. The performance monitor is rarely used and brotli compression comes with a huge dictionary that bloats the binary size without offering much benefit over lzma or zstd in most cases. If you care about binary size, these new binaries are a good default choice.
Full Changelog: v0.15.0...v0.15.1
SHA-256 Checksums
ff5ef1716dec13082356a23ca8b9a349d00e8af71712cd659d95195202838e5d dwarfs-0.15.1-Linux-aarch64.tar.xz
1d017f5da0a92f61d8620c45670cd799e0bd452a8c1f31080cea554bce880dda dwarfs-0.15.1-Linux-arm.tar.xz
c91bfe1eb348a8a34581d6377a49c937f18a96a8d1460241fde7f79fe7d3cd47 dwarfs-0.15.1-Linux-i386.tar.xz
eaead2ac3c61c6765b80fe57e6c870c7e9f14b83c4947a533aeda0d3720aac7a dwarfs-0.15.1-Linux-loongarch64.tar.xz
c70c58a47a81b51bb61fff6f878d9b9e984ac944b704c713b126aaabae9429f6 dwarfs-0.15.1-Linux-ppc64le.tar.xz
0a11aff785ae7ebed0aa2f9a0f12e85cb14336901937387e45c4ecc94399b7b6 dwarfs-0.15.1-Linux-ppc64.tar.xz
7d7af8685ac2527760fa4ca848ded3780a6fc10760242ac789094cd9ee612153 dwarfs-0.15.1-Linux-riscv64.tar.xz
f5b9c87e4471fe658690951861949b8fe9bf8ddae0fde36a04dc672cc4926568 dwarfs-0.15.1-Linux-s390x.tar.xz
24453ca3f18e08cde0e323fae4447d2e9f47c65508d8da223d22553e73cafc36 dwarfs-0.15.1-Linux-x86_64.tar.xz
a180086f9a898b4b52a5217e336c0134d63a10b395b493f2e19f231d575a87ec dwarfs-0.15.1.tar.xz
b3d3ff5608766f05a37b9a9d1f3cc3cac7ca01959ac388d43140136c858d54b7 dwarfs-0.15.1-Windows-AMD64.7z
1258d788d9950de4db55e22b5ec510665eb3049f8c199386fce16a132ab6f846 dwarfs-fuse-extract-0.15.1-Linux-aarch64
8f8f958217180f1d49fd775a84fb3f2df4a6c7d42b9537f7258c155c805f5ece dwarfs-fuse-extract-0.15.1-Linux-aarch64.upx
814204960bdd739da73c965288f4d86dab91620241a05cc99ec84be8c0377ffd dwarfs-fuse-extract-0.15.1-Linux-arm
5ea3eb07bc38d5a9cad9a296b83a6049df80c9508888341fb74b9cbb1695a750 dwarfs-fuse-extract-0.15.1-Linux-arm.upx
b7974ad380b1dac65516cd06fa1f0918708e212cb89fb6e79a79ca133e227097 dwarfs-fuse-extract-0.15.1-Linux-i386
60e448f27084f30727ae64346d5ae2c2e13cff28445e0c5b5f85a7666cfcc325 dwarfs-fuse-extract-0.15.1-Linux-i386.upx
cfd53bd6d61474cf4adf32cfbdf937717c9c329f5fd833ee997a8cd50afff74d dwarfs-fuse-extract-0.15.1-Linux-loongarch64
925433bac0999babff000767e1a3bc5f33cfc93bc4d4f5344e9c160e0fe04e0a dwarfs-fuse-extract-0.15.1-Linux-ppc64
7574dd2f519cc76c39a67d7a62fdb700a264be56f379df893872f7c864c7b5da dwarfs-fuse-extract-0.15.1-Linux-ppc64le
ccec93d4277e82ac44b78116e27c3df68b376e1ed3588c60d57a94ba5234d8ff dwarfs-fuse-extract-0.15.1-Linux-riscv64
7cae9c8f7b0ea283f7ac1f70fa8fdcd6fd50f34597536211ca854c78c6088d5a dwarfs-fuse-extract-0.15.1-Linux-riscv64.upx
afe1cfcc82314af7593054b81cee5191aa35a40a0b06bf6aab036e9d97e3da79 dwarfs-fuse-extract-0.15.1-Linux-s390x
0d0a771c7849f6f6e56cfba879ad03cf4194b57cb4fa94b4d34b797ebdd34a99 dwarfs-fuse-extract-0.15.1-Linux-x86_64
dce7fb26462f7ef0ba4c5aef5563e5e74ba23b880195ce37b4a0bd7ca46ff431 dwarfs-fuse-extract-0.15.1-Linux-x86_64.upx
36fd1012426446d85ab37503b1f8780b9c2e94e6368dac5cd3de1fd58c224cca dwarfs-universal-0.15.1-Linux-aarch64
7927c4ba101efe64fc9015de0f4b6b0c4898b3e854e037670b9998556dc6406f dwarfs-universal-0.15.1-Linux-aarch64.upx
29fcfc2bd09abe023ae8f17072b1a5967fc4536867b19fcbe79818f9f3dd9920 dwarfs-universal-0.15.1-Linux-arm
00f84a9a3e93102a880e641fcf0bda77216c8ef65b84e6239bbcdc4aa70271fa dwarfs-universal-0.15.1-Linux-arm.upx
f59359c67e5e43e67a086ad020c66f08ceb87ded0ddd641e21d35fa3ffb323b5 dwarfs-universal-0.15.1-Linux-i386
dd8116704ea22628699a0500c1d4b1fb7ff81acd3f4fe2c836604c6f654ff1ef dwarfs-universal-0.15.1-Linux-i386.upx
48624d95428286e8cac2801f23b62c000fcabedbc01bc91db734341fa621facb dwarfs-universal-0.15.1-Linux-loongarch64
428565e0033be5fe47b97c4cb1ec60da4550ec76e388326250148ecefb04ae21 dwarfs-universal-0.15.1-Linux-ppc64
a9671a135f7362d42dd8479e02593a63ac7e34430c94b9257e1fb6f28bf506a2 dwarfs-universal-0.15.1-Linux-ppc64le
c8c2ef012b80e554b0052bf85fd2271b374abd4a82337777d75200ebf40ffc0d dwarfs-universal-0.15.1-Linux-riscv64
459aa0cac669f9794f114dc868a628f8bebb8ca538d4213b7d25448e424db3ae dwarfs-universal-0.15.1-Linux-riscv64.upx
ff7b663e44f1f6d5e5b1b022fce85a6c993991f9c99e3d0e1d89c45a42d25284 dwarfs-universal-0.15.1-Linux-s390x
69750c543ea2272d96397a14627f5bb98a42385026df118cb3a4363312892fa4 dwarfs-universal-0.15.1-Linux-x86_64
83dd7dad048a86347a53cfa62d1b2dc3765a0803681a3c701a32029b464f4873 dwarfs-universal-0.15.1-Linux-x86_64.upx
5a4da367840829874f89a80112e1f49b25393969121130dc489ed6b6e8e6b782 dwarfs-universal-0.15.1-Windows-AMD64.exe
fb9baa894d32d7182d07ce54c4623d27b2b0ce6a0c6b4f1f0101462877d4ab0e dwarfs-universal-small-0.15.1-Linux-aarch64
accef8fb32026084953e9e2e0174950f9fa789e9d1ece3263bccc1db58dde6a6 dwarfs-universal-small-0.15.1-Linux-aarch64.upx
0499ffc9b7093bee499dd56845659590fdcbd32aaf3e0a5f4bb9a5db92dac60c dwarfs-universal-small-0.15.1-Linux-arm
fcd1422cd2315a6dbab0526ebe0275db9713623bd64f8f5b7bf11437f2e74f19 dwarfs-universal-small-0.15.1-Linux-arm.upx
1f3faf769fefe6517e0086288e17a0e60780cfab63b527c8e9c0cc6f695ea1df dwarfs-universal-small-0.15.1-Linux-i386
a0bad3f10125a08ac51b765371f1d82c14ff0738ddda76aa40597e64e8faf9c9 dwarfs-universal-small-0.15.1-Linux-i386.upx
eac51d831d84a89df62718ba7c3c55f4994d0c5d9387199e060d9c2242572eed dwarfs-universal-small-0.15.1-Linux-loongarch64
588de2941e5353daa164afbb55f5b4245ef27d7b0a19a2302a76093ba0c68005 dwarfs-universal-small-0.15.1-Linux-ppc64
4db6b7fec01bbaef8d51c8850eaccccbb72723dea02d5966fc8cd5198cc9f316 dwarfs-universal-small-0.15.1-Linux-ppc64le
58cda2c96c82bae3fed2d461be37786093e8f68bba7573c57ac7a1fa45cedd33 dwarfs-universal-small-0.15.1-Linux-riscv64
18b2d1c203b07a5bee39453f200d79d2069ca607e8913e060befdd5456b9aa7a dwarfs-universal-small-0.15.1-Linux-riscv64.upx
391e3e75743899a23fefe3b816cd2dda86f2048deedbc1490870a55e49cb6309 dwarfs-universal-small-0.15.1-Linux-s390x
0d84291b3e7e26f3791d50169097a7b0901542440ce934fb1ec263ae0c4a256e dwarfs-universal-small-0.15.1-Linux-x86_64
fc607ba7af3485feb8b90fd4f4678bd586c587ce8dd982e57f4fd9afb1faf391 dwarfs-universal-small-0.15.1-Linux-x86_64.upx