ComunidadAylas/PackSquash v0.3.0-rc.1

on GitHub

Additions

• Initial support for data packs was added, by handling .nbt and .mcfunction files. Currently, these data pack specific files are just copied to the generated ZIP file, but improved file-specific optimization techniques will be introduced in the future.
• JSON files (.json, .mcmeta and, if OptiFine is allowed, .jem and .jpm) can now contain Unix shell style line comments (# line comment), C style block comments (/* block comment */), and C style line comments (// line comment). Comments are ignored and removed from the generated ZIP file. JSON with comments files (.jsonc) are now supported, and their extension is canonicalized to .json in the generated ZIP file, as Minecraft expects.
• Pack file names can now at least contain every character in the Unicode Basic Multilingual Plane. This is a substantial improvement from the previous limitation of only ASCII characters.

Compression improvements

• The PackSquash ZIP compressor was greatly revamped to allow for identical file deduplication and a greater range of ZIP specification conformance levels, so you can now choose the right balance between optimization and interoperability for your needs in a more precise way.
• Key-value pairs in JSON files that are known to be ignored by Minecraft, but usually present anyway, are now removed. This behavior can be disabled.
• The number of Zopfli compression iterations applied to pack files to store them in the generated ZIP file is now variable and configurable. This is useful to customize the desired tradeoff between compression and performance. The precise number of iterations done to a file depends on its size, with smaller files being compressed more, and larger files being compressed less, so performance is more predictable and consistent. As packs used with PackSquash tend to contain lots of small files but only a handful of big ones that are not already compressed, this new behavior can yield more space savings.
• PNG files can now be quantized to 2, 4 and 16 colors, in addition to the previously hardcoded, but still default, 256 colors.
• The generated ZIP files can now make use of ZIP64 extensions when needed to overcome certain limits of the original ZIP file format, like a maximum of 65535 files, or a total of less than 4 GiB of compressed data. Note that this ZIP64 support does not improve the per-file 4 GiB limit, because such big files do not make sense to use with Minecraft.

Performance improvements

• ZIP files that PackSquash has generated in previous runs can now be automatically used to vastly speed up next runs, as unchanged files can just be copied from that ZIP file instead of being processed again. PackSquash relies on the file modification timestamps provided by the filesystem to quickly deduce if a file may have been changed.
• Gigantic textures (> 4096x4096) are now rejected by default, because they usually make little sense to use with Minecraft, unless they are atlases. Limiting this ensures that PackSquash does not take too long processing them and helps authoring resource packs that work well accross a wide range of maximum GPU texture sizes.
• Buffering is now used by default in a way that is better adapted to the device PackSquash runs on and can be configured by the user. This substantially improves performance when dealing with big (> 64 MiB) packs if there is plenty of available memory, while also improving stability in environments with low available memory. When a buffer gets rolled over to disk because the available memory is exhausted, now it will be replaced by a much smaller buffer, so performance does not drop as much as it did before.
• PackSquash is now designed around a multithreaded asynchronous programming model, as opposed to the previously used multithreaded synchronous programming model. This new model leverages cooperative task scheduling to improve thread utilization. Some internal tests showed that the total execution time was reduced by ≈5%, but this figure may change depending on the contents of your pack and the device you run PackSquash on.
• Lots of micro-optimizations in regards to both memory and CPU usage.

Extraction protection improvements

• Now PackSquash uses much more sophisticated extraction protection techniques, with additional configuration knobs that allow tuning the generated ZIP file for increased compressibility or obfuscation. These extraction protection techniques were possible to implement thanks to original research on Minecraft internals, in addition to suggestions and ideas from some Discord users.

Fixes

• Allowing the OptiFine mod now properly deals with the Custom Entity Models feature, which adds .jem and .jpm files to resource packs. Previously these files were treated as unknown junk files and skipped.
• Fixed issue #12, which made grayscale-looking color images look wrong on older versions of Minecraft, by working around the underlying Minecraft quirk. This workaround must be explicitly enabled in the options file because it may hurt compression.
• Input PNG files with embedded gamma information are now corrected to a display gamma value of 2.2, which closely matches that of the sRGB color space, because this is the gamma that Minecraft and most other end-user applications and devices assume and use. Previously, input images with an unconventional gamma would have that gamma information stripped without their colors being corrected, which led to wrong-looking results down the line.
• Fixed issue #13 as much as possible.
• JSON, shader and OptiFine-added text files with a UTF-8 BOM now work with PackSquash. The BOM is routinely added to UTF-8 text files by some editors, like Windows Notepad, but it serves no purpose within a pack, so it is now ignored and removed.
• Audio files are no longer downmixed to mono by default, because the different 3D effects that downmixing may imply can be inappropriate, and doing so did not yield very significant space savings anyway, due to Ogg Vorbis joint encoding.
• Shader files with Windows (CR + LF) line endings now work fine with PackSquash.
• The OxiPNG library is now told to stop trying further optimizations on images if the process has been running for more than ten minutes. This deadline is just a best-effort, and does not fully address the fact that some images still take too long to optimize in a way that is not affected by this change.

Stability improvements

• PackSquash' software architecture was reworked to support dealing with files larger than the available memory. Nevertheless, the usefulness of this change is limited in practice by libraries that require the entire file to be copied to memory to be processed. Currently, only audio and copied files can be processed without storing their entire contents in memory.
• Now PackSquash aborts its execution if an error occurs while processing the pack, instead of continuing and risking generating corrupt or incomplete ZIP files. This behavior also allows using PackSquash as a pack validator more efficiently.
• A configurable limit of simultaneously open files was introduced. This avoids errors caused by the open file limit imposed by the operating system being reached, which can happen in big packs if lots of threads are used.

Codebase changes

• The entire codebase was refactored to be much more modular, testable, and maintainable. In particular, the application and logic code were separated in two binary and library crates, respectively, promoting separation of concerns and paving the way for a future API.
• Automated unit tests were added for the most critical components of PackSquash. This increases the confidence that those components work okay, and continue to work fine after changes. These tests are run automatically for each build.
• Improved build reproducibility by checking in to the repository the Cargo.lock file.
• Updated build environment software. This included updating Linux build runners from Ubuntu 18.04 to 20.04.
• Improved the GitHub Actions workflow, fixing semantic version calculations, build artifacts caching, and other changes.
• Visual Studio Code configuration files were checked in to the repository. This should help new contributors configure their development environment more easily.
• An .editorconfig file was added to improve code readability in the GitHub website.
• Updated libraries and toolchain components for the latest fixes and improvements.

Miscellaneous

• JSON files can now be prettified instead of minified before storing them in the generated ZIP file in a per-file basis. This may be useful for packs under a permissive license that wants to encourage contributions from its users. Similarly, minification of shader and .properties files can now be disabled.
• Opus audio files are now supported as input files in resource packs. They are transcoded to Ogg so that Minecraft can read them.
• Several tidy-ups and improvements to the text that PackSquash outputs. The error message shown when the output path file was a folder was improved to be much more user-friendly.
• Detailed version, author and license information is now shown on startup. More detailed information can be seen with the -v command line switch.
• The hardcoded list of system files that are ignored when ignore_system_and_hidden_files is set to true was expanded and improved.
• Progress messages are now printed to the standard error stream, following the trend and conventions established by other command line applications.
• Official builds for the Linux AArch64/ARM64 platform are now available. Even though these builds should work okay, support for this platform is limited to a best-effort, because PackSquash has not actually been run on this platform on the device of a repository maintainer.
• Information for sponsoring PackSquash developers was added to the README.md file. A FUNDING.yml file was also added so that GitHub displays a relevant "Sponsor" button in its web interface.

Options files changes

• Settings files have been renamed to options files.
• The following options are new (i.e. they were introduced in v0.3.0 and do not have an exact equivalent in v0.2.1):
  • zip_compression_iterations
  • work_around_minecraft_quirks
  • size_increasing_zip_obfuscation
  • percentage_of_zip_structures_tuned_for_obfuscation_discretion
  • never_store_squash_times
  • spooling_buffers_size
  • open_files_limit
  • zip_spec_conformance_level
  • color_quantization_target
  • minify_json
  • delete_bloat_keys
  • maximum_width_and_height
  • minify_shader
  • minify_properties
• The following options were replaced:
  • strict_zip_spec_compliance, by zip_spec_conformance_level
  • quantize_image, by color_quantization_target
• The following options were renamed, with no change in functionality or value types:
  • resource_pack_directory → pack_directory
  • compress_already_compressed_files → recompress_compressed_files
  • allowed_mods → allow_mods