github mhx/dwarfs v0.15.8
dwarfs-0.15.8
on GitHub

3 hours ago

Bug fixes

  • Correctly handle sparse files when querying entries in "data order". This is primarily used by dwarfsextract to speed up extraction and avoid repeatedly decompressing the same blocks. All regular files are sorted by the block referenced by the first chunk. While subsequent chunks may also reference earlier blocks, these blocks are typically still in the cache. However, a sparse file starting with a hole will reference a special "hole marker" block in its first chunk, causing all sparse files that start with a hole to be clustered at the end of the list, without considering any of the subsequent data chunks. This is likely an academic edge case, but in an image containing tens of thousands of such sparse files, this can quite dramatically slow down extraction. Furthermore, this also triggers other bugs that can quite easily cause dwarfsextract to run out of memory. This has been fixed by ignoring the holes and only considering the first data chunk when sorting.

  • Limit size of worker group queues in filesystem extractor. During extraction, the filesystem extractor used by dwarfsextract uses a worker pool for writing regular files. This pool has a job queue that by default can grow unbounded. Each job, before being added to the queue, will already trigger an asynchronous read of the file data. This is actually bounded based on the size of the queued regular files, but not considering the amount of memory used by the decompressed blocks in order to actually read the queued file data. This usually isn't a problem, since the files are added in "data order", but in combination with the bug described earlier where sparse files starting with holes end up not actually being sorted, this can cause requests for pretty much all blocks in the image to be decompressed simultaneously. Even worse: if the decompressed blocks are larger than the configured cache size, blocks that are still referenced by eariler requests can be evicted from the cache, causing multiple simultaneous decompression requests for the same block. So even though the decompressed image might actually fit in memory, the actual memory consumption can very easily reach tens of gigabytes. Limiting the queue size is a workaround for this issue, but it's a good idea to bound the queue size anyway. There are two real fixes: one is the fix for the sorting issue described earlier, the other one is for the block cache to keep track of blocks that are evicted, but still in use, and re-insert these blocks into the cache rather than decompressing them again. The latter is a more involved change, so it is not included in this release, and it's really not necessary for the dwarfsextract use case with the former fix in place.

  • Correctly compute total hardlink size. The algorithm used to compute the total hardlink size (i.e. the total size of all hardlinks beyond the first one) was not correctly accounting for non-hardlinked duplicates. The total hardlink size is currently really only used for progress reporting in dwarfsextract. However, v0.15.0 introduced a strict check that would abort the process if there was a mismatch between the total computed size and the total written size if --stdout-progress was used. This strict check would fail if the total hardlink size value was actually used, which is only the case for extracting to formats that do not support hardlinks, e.g. ZIP or 7z. A workaround would be to simply drop --stdout-progress, or to make the strict check just report an error rather than aborting (this is also done in this release). The real fix is to correctly compute the total hardlink size, and fix a wrong value when re-writing an image and using --rebuild-metadata.

  • Don't abort extraction in release builds when discovering a progress mismatch with --stdout-progress. The mismatch is still reported as an error, but the process will continue in a release build and abort in a debug build.

  • Preserve preferred path separator when rebuilding metadata. When rebuilding metadata, the preferred path separator should never be changed, but it was erroneously always set to the platform default of the binary used for the rebuild. So an image created on Windows with \ as the preferred path separator and later rebuilt on Linux would end up with / as the preferred path separator (and vice versa). This would only cause problems if the image contained symbolic links with target paths that used a path separator. Those symbolic links would now be broken. A workaround would be to rebuild the image on the same platform it was created on, with a version of DwarFS that still contains the bug (i.e. v0.15.7). This is now fixed and the preferred path separator is preserved when rebuilding metadata.

  • Call archive_write_finish_entry() to ensure that warnings are reported for all entries in dwarfsextract. Before this fix, all warnings except for the last were silently ignored.

  • Handle sparse files when using readahead. The readahead implementation did not handle hole chunks in sparse files and would actually crash in debug builds. This has been fixed.

  • Default max_eager_map_size to 1 TiB on 64-bit architectures. This defaulted to unlimited on 64-bit architectures before, which could cause errors when attempting to map extremely large (sparse) files.

Build fixes

  • Add several missing includes to avoid compilation errors on newer compiler versions.

Full Changelog: v0.15.7...v0.15.8

SHA-256 Checksums

 
ee100574a1340ad054bdf7543bedde3bd040639114ede65f1c20e08567eff2b1  dwarfs-0.15.8-Linux-aarch64.tar.xz
484727f2efdf44f597abe4950d780b9fcfc0a499672338dc1ba99099d92f252e  dwarfs-0.15.8-Linux-arm.tar.xz
71298018cdd73f5de267b4af33c015df840adf18c6131d8ca7b162dc8e20226b  dwarfs-0.15.8-Linux-i386.tar.xz
974056bafd72731f0b18f324211e903031cb3809e2c50ff505f22efd2f4af6c6  dwarfs-0.15.8-Linux-loongarch64.tar.xz
70e83eaff3dd28f966502c890971a79f23d69a7a0f19ca4d0f37791a888cba7e  dwarfs-0.15.8-Linux-ppc64le.tar.xz
2c681ce53a9f29c0259f22700259fca2dcd4ff5ef8bd2715a01e210232e558b6  dwarfs-0.15.8-Linux-ppc64.tar.xz
78e1a66e1f3bd3da6967ba7590ae8ac30915c419c8d5fda75b19b08d570b51ac  dwarfs-0.15.8-Linux-riscv64.tar.xz
da0b5e6995b2a7679cb950610fc00dd3bad663623e0a55b49b5f342e8a20d5fd  dwarfs-0.15.8-Linux-s390x.tar.xz
f5eb34c3e2ac1bfcbe66e15b8a66da20897049b22867d2c193bffedf25cc08ae  dwarfs-0.15.8-Linux-x86_64.tar.xz
a2382a2d06f4539b1c53b8b4f800776945e2f13f71c0a3226d3bab3e1b25fe04  dwarfs-0.15.8.tar.xz
49ffae19533b0c5b3dfc5516b19e97700f1197416062a5517f65bfb631631566  dwarfs-0.15.8-Windows-AMD64.7z
e6babb706254fd98b7012325c78997983a7824c097f87706efc0fbd365c04fd8  dwarfs-fuse-extract-0.15.8-Linux-aarch64
3b233d30051ecfe4cd4d9bb33d0b9001d99c96f0e4067a3d5d5d7b1ea71733e0  dwarfs-fuse-extract-0.15.8-Linux-aarch64.upx
789f07f790e28ee198b5854364759b579b3744407d22f17bb2170e7a2c006bb6  dwarfs-fuse-extract-0.15.8-Linux-arm
caf35bda41970c426e1fa162bf7340eef80001a7ec2cb59643e04b3ab0d89a00  dwarfs-fuse-extract-0.15.8-Linux-arm.upx
ec796f13b5757ce516d4d4b7a172aebe5aa6cded7a4388e8e9eb873a8291bf98  dwarfs-fuse-extract-0.15.8-Linux-i386
594b838742d9310448adaa21fafd084c46e7e0bf319254045d31374fe257466d  dwarfs-fuse-extract-0.15.8-Linux-i386.upx
6fccfa6437467eaa6ee59051bd7148989b3b5eaeefbbeedbb092b72d9808721a  dwarfs-fuse-extract-0.15.8-Linux-loongarch64
5709f9fe9f5d492969e860af226777ece0bc88a4d61bcf2fd5f919d1c433e786  dwarfs-fuse-extract-0.15.8-Linux-ppc64
cd6832bccb37f58312621f4c6c1b72f623d667a366867811cca8451a2524f18a  dwarfs-fuse-extract-0.15.8-Linux-ppc64le
94fc718d54b7b1164df5405e9e319043d263cfc62bad7f56b4c9031d75748779  dwarfs-fuse-extract-0.15.8-Linux-riscv64
13bbfc0fa36d3527b56726d480119a4b920c372d4f0a47b8cc01d492a951de40  dwarfs-fuse-extract-0.15.8-Linux-riscv64.upx
3724678c157108c10f1c6560923ef1319a24011a94744c6a6ccae3ac03d0298b  dwarfs-fuse-extract-0.15.8-Linux-s390x
494030845b6b61efe74bd00449069faede1720ef5b8e0f59ccc5eea7d74a244e  dwarfs-fuse-extract-0.15.8-Linux-x86_64
0cbe87b4b6961a1003d3a4dc704a0271148551a4584405857d6bff1b6aa921d7  dwarfs-fuse-extract-0.15.8-Linux-x86_64.upx
f21cbc33840bbffe410c7043fab85e0d1b4eee86d77bcc0fbf3a51b94fc4b222  dwarfs-universal-0.15.8-Linux-aarch64
84330541b1d989637ddcc4b521bc33e2654e7bf962f7e488884cc87ff86081b4  dwarfs-universal-0.15.8-Linux-aarch64.upx
c098d0a222e737a71764343e37c700f502e75c3de7fb1e4802410976f7053a23  dwarfs-universal-0.15.8-Linux-arm
9d1fdc9ad57166efdb52a900b6a3b8f7acae649e3de33c3b3f0b62d1a056b379  dwarfs-universal-0.15.8-Linux-arm.upx
5c93f65020ad20c0d0be8271d30d0a8da4bbfdec0a1078e6f0606658e7e110f6  dwarfs-universal-0.15.8-Linux-i386
a5f47ae2bd0a35340a5db9ce4b081a8fbcf5facff187bed869420d1a7ca73d01  dwarfs-universal-0.15.8-Linux-i386.upx
e966c33a66b0ee00e4bf0705426c7261c4798018b9c67d719c833bdd48f1beea  dwarfs-universal-0.15.8-Linux-loongarch64
0b45d8f8c09e5de71f9ea3d290871cfd3aef4aaac1aa92d8363b84e6c8f96c3a  dwarfs-universal-0.15.8-Linux-ppc64
257fcab374891dadf5b5f9947997b9ad34951942f4f724dc18917411add9957c  dwarfs-universal-0.15.8-Linux-ppc64le
2370e78a69f9255cd9ae0ca54c987f3dd706608e6d142972379f1a924de9a9e5  dwarfs-universal-0.15.8-Linux-riscv64
1cc6182f040cd7037563f57827cc0fdacf21df318e41d1e8f105568e95fa10b8  dwarfs-universal-0.15.8-Linux-riscv64.upx
baa138983599c8a8ecd3266c77c4c8794b96e587d1bef667ecef53bd4dea3a3a  dwarfs-universal-0.15.8-Linux-s390x
0ed66df82f9bb6671c1f898404ff4d57c263460078d46dbac8efb9d8eec271b7  dwarfs-universal-0.15.8-Linux-x86_64
f149494612fa4d6b2a3b49892af25f629ed0e241da1686924a0f4f121f1ff521  dwarfs-universal-0.15.8-Linux-x86_64.upx
7bce40f0652c52c9444d8029af5821d77e38a9360d9dfab52c4dbecd6180506d  dwarfs-universal-0.15.8-Windows-AMD64.exe
e78d20df86b1382b44e0e15b3b7fc69402878ed8552eb218544f0336267a69ec  dwarfs-universal-small-0.15.8-Linux-aarch64
da540fb3961bb62f9e676ada3570c024501b87af63d5ac98e1074ce918cec53f  dwarfs-universal-small-0.15.8-Linux-aarch64.upx
69846272903ff28ea3878146b9a323b37bc8d6addaa75372631eee1668c805d1  dwarfs-universal-small-0.15.8-Linux-arm
1e2648736308d66dec871f1ee970cb5c2c46419345e7da9b182df5d8166cf9f2  dwarfs-universal-small-0.15.8-Linux-arm.upx
28a8027220c084e82ae2d6f78789edbbc23d585f6a9ade86b326799719c06592  dwarfs-universal-small-0.15.8-Linux-i386
87f585381507387f9bfa230a7ff08acee7e5ef0eff52d3de920043ac98c04bdf  dwarfs-universal-small-0.15.8-Linux-i386.upx
2316584997ee6441b483d859b184a986db92639efc5de899ad48c4e69c86a8c4  dwarfs-universal-small-0.15.8-Linux-loongarch64
fff8c7671a4cba109c48cf40aa919e0809bc59feda140e3685afd3a46b673f53  dwarfs-universal-small-0.15.8-Linux-ppc64
22e61cb0021f1bfba953f86b520f4f192e9b33163453a7074ba6dbf6d0f9bf81  dwarfs-universal-small-0.15.8-Linux-ppc64le
4ce7177f04f1e2ce149cadf461e098489f7e4fa0559ed0466e60b7c9320cd2e7  dwarfs-universal-small-0.15.8-Linux-riscv64
f58d0991f0c4ba5475ed4457df2ab08a28f261e0ed23b26277017dab8a48e059  dwarfs-universal-small-0.15.8-Linux-riscv64.upx
72489cfaf6e2a66992590a6075fc6681122c367b55c311d2bbed15dad590375a  dwarfs-universal-small-0.15.8-Linux-s390x
d29734ed51547f5a5e1c92ede3ea0f2e003902f5c58df10d5e060c87b4e07e69  dwarfs-universal-small-0.15.8-Linux-x86_64
0217b8bf803b934aa1fd097b1c131cff0ffc112a03f575996d93a7882caeb8d4  dwarfs-universal-small-0.15.8-Linux-x86_64.upx
Check out latest releases or
releases around mhx/dwarfs v0.15.8

Don't miss a new dwarfs release

NewReleases is sending notifications on new releases.

Get notifications