uutils/coreutils 0.12.0

on GitHub

Rust Coreutils 0.12.0 Release:

We are happy to announce the release of Rust Coreutils 0.12.0 - a release focused on compatibility fixes for our Ubuntu friends.
Ubuntu ships uutils coreutils as its default coreutils implementation, and this cycle was driven by what their users and maintainers reported: the bugs they asked us to prioritize are all fixed here, along with a long tail of smaller divergences from GNU. Few of these are glamorous, but they are exactly what breaks a script that has worked for twenty years.

Alongside that, we removed the last C dependency of expr (onig is replaced by the pure-Rust fancy-regex), continued the Windows port (nice, nohup and hostid now build there, and uucore's filesystem layer was rewritten with safe Win32 wrappers) and landed another round of security hardening on ls, sort and install.
Also, some contributions had carried comments and code over from GNU coreutils, against our own rule
that uutils is an original implementation. We reviewed the tree and rewrote what we found, in comments and in the few functions that followed the C too closely; no behaviour change. The GNU coreutils maintainers have been notified.

GNU coreutils 9.12 was released during our cycle (kudos to the team!), and some of the work went that way too. Chasing compatibility tends to turn up things worth fixing or improving on both sides.

We also made it official: a utility being significantly slower than GNU is now treated as a bug, not a wishlist item. If you hit one, please report it as a bug.

We are now at 653 passing, 21 failing, 0 errors on the GNU test suite.

GNU Test Suite Compatibility:

Note: the suite grew by 7 tests (backport of GNU tests) and 8 more tests are skipped on our reference runner, so the raw pass count is flat while the failures went down again. Counting only the tests that actually run, we are at 653/674 (96.9%), our best ratio so far, and the second release in a row with no erroring test. We are comparing against Coreutils 9.11.

Highlights:

• Ubuntu Compatibility

  • The issues Ubuntu asked us to fix first are all fixed: data loss in a parallel install -D (#12355), du on 32-bit architectures (#11848), ls ACL dereferencing (#13234), cp -R name collisions (LP#2167118) and chmod -R through symlinks (LP#2167122)
  • A whole family of options now accept a hyphen-leading value, like GNU does: csplit --suffix-format, paste -d, join -t, stat --format/--printf/-c and all of numfmt's options. A - in a delimiter or a format string is data, not a flag
  • rm no longer segfaults on very deeply nested directories
  • sort -n applies to the key instead of the whole line, echoes a disorder line verbatim instead of reformatting it as a number, and no longer keeps every input file open while sorting
  • basename trailing-separator handling for dot components, realpath exit status in quiet mode, ln keeping the destination when a forced link fails, csplit always creating the final split even when empty, fmt -x/-X, printf treating a -- past the format as an argument, test rejecting an -a/-o that ends the expression, echo -e no longer expanding \u, \U and \"
  • ls: dired fixes, a tie in the time sort broken by the name, name sort for -u/-c in long format, capability markers that respect the dereference config; dir and vdir match the GNU default quoting
  • cp: --no-preserve is retained over a later --preserve, and --attributes-only gives the destination the source's type
  • Error messages and exit codes aligned with GNU in uniq, chgrp, factor, date, expr, od, tty, tsort and pathchk
  • truncate finally implements --io-blocks, and date accepts multiple --reference

• Security Hardening

  • ls: report stat() failures on directory entries instead of silently showing wrong metadata
  • sort: temporary files are created with mode 0600 instead of the umask, the -o output copy is kept private too, and a failure to truncate the output file is now reported
  • install: the fd-based destination is created exclusively and at 0600
  • rm, chmod, chown, chgrp: --preserve-root compares (st_dev, st_ino) rather than the path
  • chcon walks directories with DirFd instead of fts
  • SECURITY.md now requires a boundary crossing, not just a GNU divergence, for a report to be treated as a vulnerability

• Robustness

  • Panics and overflows fixed in stat (out-of-range octal escape), ls (a color style with no attributes), split (a huge -n l/K/N chunk count), truncate (a size above i64::MAX no longer creates the file), cksum (an over-length BLAKE2b digest in a check file) and uucore (invalid input lines)
  • tsort handles invalid UTF-8 input and flushes stdout properly; date reports invalid UTF-8 lines in -f input; truncate supports non-UTF-8 paths for --reference

• Dependency Diet

  • expr replaces onig with fancy-regex, removing a C dependency and the CI hacks that came with it. As a side effect, expr switched to leftmost-longest regex matching, matching GNU
  • mv replaces fs_extra with std directory sizing; fts-sys is gone; cksum uses aws-lc instead of vendored OpenSSL on MSVC, so a Perl install is no longer needed to build on Windows
  • bstr default features disabled, uucore's hand-rolled process functions dropped in favor of rustix

• Cross-platform Reach

  • Windows: nice (with a Priority Class mapping), nohup and hostid now build, dirname handles backslashes, user locale detection landed, checksum paths use forward slashes, and mv, sync, hostname, touch and rm moved to safe std APIs
  • uucore on Windows: fsext rewritten with safe Win32 wrappers, real free space and mount paths; df resolves paths by mount prefix
  • New and improved targets: aarch64-unknown-linux-ohos, AIX (fsext), Haiku (fs), Hurd (signals), NetBSD (safe-traversal), plus wider process, utmpx and entries support
  • WASI: cp attribute preservation and reflink handling, mktemp temporary directories, tail integration coverage
  • macOS: tail stdin-redirect, pid and follow-name tests re-enabled

• Performance

  • sort gained benchmarks for spilling, checking and shared prefixes; du benchmarks now warm the OS page cache before timing
  • cp avoids resolving the cwd for absolute recursive copies; tsort gives the OS a sequential access hint
  • A significant performance gap with GNU is now stated in the README as a bug, and we handle it as one

• Contributions: This release landed 191 pull requests from 16 new contributors

Call to Action:

Try it in your browser - Online Playground powered by WebAssembly
Help us translate - Contribute to Rust Coreutils on Weblate
Sponsor us on GitHub to accelerate development: github.com/sponsors/uutils

What's Changed

basename

• basename: fix trailing-separator handling for dot components by @wangyusheng1985 in #14365

chcon

• chcon: walk directories with DirFd instead of fts by @sylvestre in #14413

chgrp

• rm, chmod, chown, chgrp: compare (st_dev, st_ino) for --preserve-root by @sylvestre in #14411
• chgrp: strip error when changing group of file fails by @Devel08 in #14403

chmod

• rm, chmod, chown, chgrp: compare (st_dev, st_ino) for --preserve-root by @sylvestre in #14411
• chmod: fix clippy lints by @xtqqczze in #14447
• chmod: do not check umask on skipped symlinks during recursion by @okaythi in #14512
• chmod: gate by target support by @xtqqczze in #14585

chown

• rm, chmod, chown, chgrp: compare (st_dev, st_ino) for --preserve-root by @sylvestre in #14411

cksum

• cksum: use aws-lc instead of vendored openssl on MSVC to avoid perl by @oech3 in #14559
• cksum: do not abort on an over-length BLAKE2b digest in a check file by @Dijaa in #14508

cp

• cp: avoid resolving cwd for absolute recursive copies by @varranvar in #14336
• fix(wasi): cp attribute preservation and reflink handling by @eduardomourar in #13685
• cp: fix copying entries sharing parent directory name with -R . by @okaythi in #14530
• cp: retain --no-preserve over later --preserve by @wtcpython in #14555
• cp: --attributes-only must give the destination the source's type by @sylvestre in #14622

csplit

• csplit: accept a hyphen-leading --suffix-format value by @arbelonson-source in #14320
• csplit: always create the final split, even when empty by @yuefdev in #14565

date

• date: report invalid UTF-8 lines in -f file/stdin input by @alexchen-sys in #14280
• date: wrap errors with thiserror by @Devel08 in #14333
• date: exit with 1 for expected file, got directory by @Devel08 in #14435
• date: rewrite test to remove thread::sleep by @cakebaker in #14535
• date: allow multiple --reference by @cakebaker in #14546

dd

• dd: simplify windows stdin in match guard by @hlsxx in #14347
• dd: fix broken man page formatting by @hlsxx in #14348

dir

• dir, vdir: match GNU default quoting by @ChihweiLHBird in #14538

dirname

• dirname: Fix backslash support on Windows by @lhecker in #14479

du

• du: warm OS page cache before timing in benchmarks by @sylvestre in #13958
• du: fix EOVERFLOW on 32-bit Linux by @sylvestre in #14548

echo

• echo: don't expand \u, \U and " in -e by @SichenLiang in #14475

env

• env: trace the environment changes under -v by @Socialpranker in #14401
• Revert "env: trace the environment changes under -v" by @cakebaker in #14493

expr

• expr: replace onig with fancy-regex by @wtcpython in #14329
• expr: re-enable test_regex_caret on Windows arm64 by @wtcpython in #14429
• expr: exit with on a write error by @Devel08 in #14500
• expr: switch to leftmost-longest regex matching behavior by @keith-hall in #14527

factor

• factor: strip error code in stdin error message by @Devel08 in #14511

fmt

• fmt: honor -x and -X instead of always matching prefixes exactly by @MaxFreedomPollard in #14417

head

• head, tail: reject multiplier suffixes that GNU rejects by @kennywillbe in #14228

hostid

• hostid: remove unuseful lock() by @oech3 in #14426
• hostid: build for Windows & remove unsafe by @oech3 in #14427

hostname

• mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445
• hostname: gate by target support by @xtqqczze in #14612

install

• install: fix EEXIST race losing files in parallel install -D by @eyupcanakman in #12429
• test_install.rs: add test case for eCryptfs by @oech3 in #14545
• install: create the fd-based destination exclusively and at 0600 by @sylvestre in #14625

join

• join: accept a hyphen-leading -t separator, quote it in the error by @arbelonson-source in #14318

link

• link: add tests for zero args and existing destination by @anihal in #11119

ln

• ln: keep the destination when a forced link fails by @nagendramohan in #14574

logname

• logname: use no_flush by @oech3 in #14364

ls

• ls: fix panic on a color style with no attributes by @harshasiddartha in #14373
• ls: avoid accessing by index by @oech3 in #14380
• ls: apply dir and vdir format defaults in Config::from by @ChihweiLHBird in #14450
• ls: break a tie in the time sort with the name by @luantaraschi in #14049
• ls: fix two dired bugs by @sylvestre in #14499
• fix(ls): ensure ACL and capability markers respect dereference config by @sachinskss in #13368
• ls: use name sort for -u/-c in long format by @ChihweiLHBird in #14573
• ls: report stat() failures on directory entries (GHSA-fhfr-qj69-pq5m) by @sylvestre in #14553

mknod

• mknod: use dev_t type to avoid casting by @xtqqczze in #14569

mktemp

• fix(mktemp): add wasi platform support for temporary directory handling by @eduardomourar in #13786

mv

• mv: avoid accessing by index by @oech3 in #14379
• mv: test broken symlink file/dir as target by @hlsxx in #10739
• mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445
• mv: replace fs_extra with std directory sizing by @wtcpython in #14432
• generalize some mv code for ln, cp by @sylvestre in #14623

nice

• nice: prepare supporting Windows by @oech3 in #14452
• nice: map Windows Priority Class by @oech3 in #14457

nohup

• nohup: simplify by .into() by @oech3 in #14476
• nohup: build for Windows by @oech3 in #14461
• nohup: move platform code into platform/{unix,windows}.rs by @sylvestre in #14525

numfmt

• numfmt: accept hyphen-leading -d/--suffix/--padding values by @arbelonson-source in #14322
• numfmt: accept hyphen-leading values for the remaining options by @sylvestre in #14362
• numfmt: zero-pad the number, not the unit suffix by @Socialpranker in #14393

od

• od: say the skip ran past the end of the combined input by @Socialpranker in #14399

paste

• paste: accept a hyphen-leading -d delimiter list by @arbelonson-source in #14321

pathchk

• pathchk: reject empty path in portability mode by @cakebaker in #14506
• pathchk: remove #![allow(unused_must_use)] by @cakebaker in #14507

pinky

• pinky: make it more rust idiomatic by @sylvestre in #14424

pr

• pr: three small refactorings by @cakebaker in #14534

printf

• printf: honor POSIXLY_CORRECT for the character-constant warning by @SichenLiang in #14416
• printf: treat a -- past the format as an argument by @Socialpranker in #14394

ptx

• ptx: make the code more rust idiomatic + fix some comments by @sylvestre in #14595

realpath

• realpath: fix exit status in quiet mode by @ChihweiLHBird in #14442

rm

• rm, chmod, chown, chgrp: compare (st_dev, st_ino) for --preserve-root by @sylvestre in #14411
• mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445
• rm: fix segfault on very deeply nested directories by @sylvestre in #14554

sort

• sort: apply -n to the key, not the whole line by @Socialpranker in #14331
• sort: add benchmarks for spilling, checking and shared prefixes by @sylvestre in #14367
• sort: echo the disorder line verbatim, not reformatted as a number by @Socialpranker in #14389
• sort: do not hold all inputs open while sorting by @Hekzory in #14423
• sort: simplify get_unit by @oech3 in #14505
• sort: gate tests correctly for BSD by @xtqqczze in #14533
• sort: report a failure to truncate the -o output file by @sylvestre in #14617
• sort: create temporary files with mode 0600 instead of the umask by @sylvestre in #10647
• sort: keep the output-file copy private too by @sylvestre in #14624

split

• split: fix overflow on a huge -n l/K/N chunk count by @harshasiddartha in #14374

stat

• stat: accept a hyphen-leading --format/--printf/-c value by @arbelonson-source in #14323
• stat: fix overflow on out-of-range octal escape by @harshasiddartha in #14371

stty

• test(stty): gate by target support by @xtqqczze in #14571

sync

• mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445

tail

• head, tail: reject multiplier suffixes that GNU rejects by @kennywillbe in #14228
• test(tail): fix follow name truncate failing on macos by @eduardomourar in #14438
• tail: enable working stdin-redirect and pid tests on macOS by @wtcpython in #14433

test

• test: reject a -a/-o that ends the expression by @Socialpranker in #14395
• test: Honor PATHEXT on Windows by @lhecker in #14481

touch

• mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445

truncate

• truncate: reject a size above i64::MAX instead of creating the file by @AlejandroCoronadoN in #13902
• fix(truncate): implement --io-blocks by @ThomasHabets in #13432
• truncate: group fifo-related tests in mod by @cakebaker in #14385
• truncate: support non-UTF8 paths for --reference by @cakebaker in #14386
• truncate: remove unreachable code by @cakebaker in #14422

tsort

• tsort stdout flushing fix and several error messages handling ameliorations by @anastygnome in #14384
• tsort: handle invalid utf8 in input by @anastygnome in #14464
• tsort: add sequential file access hint on windows and where available on unix by @anastygnome in #14463

tty

• tty: add missing newline to output by @ChrisX101010 in #10252
• tty: avoid exit & show write error by @oech3 in #14366

uniq

• uniq: drop the "Could not open" prefix from open errors by @abhijeetvichare76 in #14262

uptime

• uptime: accept hour or minute in pretty-print tests by @ChihweiLHBird in #14547

vdir

• dir, vdir: match GNU default quoting by @ChihweiLHBird in #14538

who

• who: make the code more rust idiomatic by @sylvestre in #14410

uucore

• uucore: update fsext to build on aix by @xtqqczze in #14430
• uucore: fix a fsext test by @xtqqczze in #14446
• uucore: Avoid panic on invalid input lines by @lhecker in #14470
• uucore: use libc implementations of major, minor, and makedev by @xtqqczze in #14471
• uucore: Implement user locale detection for Windows by @lhecker in #14477
• uucore: Use forward slashes for checksum paths on Windows by @lhecker in #14469
• uucore: Fix strip_errno doctest for Windows by @lhecker in #14482
• uucore: remove process functions by @xtqqczze in #11158
• uucore: os_str_from_bytes does not need to alloc by @lhecker in #14466
• uucore(windows): rewrite fsext with safe Win32 wrappers, real free space and mount paths; df(windows): resolve paths by mount prefix by @nikolalukovic in #14455
• uucore: support fs feature for haiku by @xtqqczze in #14570
• uucore: support process feature for more platforms by @xtqqczze in #14579
• chore(uucore): fix clippy lints for fsext feature by @xtqqczze in #14581
• uucore: fix test_crlf_across_blocks not compiling on Windows by @abhijeetvichare76 in #14254
• uucore: support signals feature for hurd by @xtqqczze in #14454
• uucore: fix entries feature by @xtqqczze in #14601
• uucore: support utmpx feature for more platforms by @xtqqczze in #14584
• uucore: escape embedded locale content instead of using a raw string by @sylvestre in #14616
• uucore: support safe-traversal feature for netbsd by @xtqqczze in #14606

Testing

• tests: don't pipe input into commands that exit before reading it by @Socialpranker in #14431

Code Quality & Cleanup

• fix: redundant_homepage cargo lint by @xtqqczze in #14440
• fix: missing_lints_inheritance cargo lint by @xtqqczze in #14441
• fix outdated comments about onig by @oech3 in #14448
• fix: needless_range_loop clippy lint by @xtqqczze in #14444
• fix: unused_dependencies cargo lint by @xtqqczze in #14443
• simplify matches! and inequalities in match by @oech3 in #14502
• remove some matches! & use .get() by @oech3 in #14503
• use .is_root() of rustix::process by @oech3 in #14519
• fix: missing_const_for_thread_local clippy lint by @xtqqczze in #14462
• chore: add missing newlines to source header by @xtqqczze in #14572
• chore: fix clippy missing_const_for_thread_local lint by @xtqqczze in #14613

CI & Build

• Revert "remove rust-version from fuzz" by @xtqqczze in #14105
• fetch-gnu.sh: fix hardcoded pathes to GNU test binary by @oech3 in #14332
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #14335
• Cargo.toml: remove unused fts-sys by @cakebaker in #14425
• CICD.yml: remove a hack for onig by @oech3 in #14428
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #14439
• openbsd.yml: remove C toolchain by @oech3 in #14449
• ci: fix style_empty_assertions job by @cakebaker in #14451
• Cargo.toml: remove timeout & kill from feat_require_unix_core by @oech3 in #14453
• freebsd.yml: fix warning by @oech3 in #14488
• Fix the GNU reference + send a notification to coreutils-tracking for data refresh by @sylvestre in #14492
• github copilot: Create code review guidelines for uutils/coreutils by @sylvestre in #14510
• cargo: use workspace inheritance for dev-dependencies by @xtqqczze in #14517
• CICD.yml, test_openssl: drop non-*sum & avoid release build by @oech3 in #14473
• CI: fix artifact name for Cygwin by @oech3 in #14494
• Cargo.toml: reconsider feat_{wasm, common_core,Tier1} for wasi by @oech3 in #14543
• CICD.yml: publish binaries with static openssl by @oech3 in #14539
• CI: remove brew install by @oech3 in #14557
• fix(build): adjust check for cross-platform building by @xtqqczze in #14568
• CICD.yml: drop unnecessay setup, simplify by @oech3 in #14528
• fix cspell by @oech3 in #14580
• GnuTests: autoreconf for portability by @oech3 in #14575
• chore(cargo): use workspace inheritance for dependencies by @xtqqczze in #14596
• cargo: avoid safe_traversal feature on unsupported platforms by @xtqqczze in #14588
• deny.toml: remove windows-sys 0.59 and related crates from skip list by @cakebaker in #14611

Documentation

• README.md: clear statement wrt performances by @sylvestre in #14345
• uudoc: improve loop by @oech3 in #14361
• CONTRIBUTING.md: update link to DL GNU coreutils by @oech3 in #14608
• SECURITY.md: require a boundary crossing, not just a GNU divergence by @sylvestre in #14615

Platform Support

• tests: enable tail integration coverage on WASI by @DePasqualeOrg in #13805
• coreutils: replace /proc/ by /dev/ by @oech3 in #12553
• ohos: add aarch64-unknown-linux-ohos support (fsext/hostid/date) by @YodonTan in #14288

Dependency Updates

• build(deps): update rust crates by @xtqqczze in #14338
• fix(deps): refactor rustix/nix package configuration by @xtqqczze in #14337
• chore(deps): update rust crate rstest to 0.27.0 by @renovate[bot] in #14418
• chore(deps): update vmactions/freebsd-vm action to v1.5.6 by @renovate[bot] in #12977
• chore(deps): update rust crate console to v0.16.6 by @renovate[bot] in #14491
• Revert "chore(deps): update vmactions/freebsd-vm action to v1.5.6 (#12977)" by @xtqqczze in #14513
• chore(deps): update vmactions/freebsd-vm action to v1.5.5 by @xtqqczze in #14516
• chore(deps): update rust crate jiff to v0.2.37 by @renovate[bot] in #14521
• chore(deps): update rust crate textwrap to v0.16.4 by @renovate[bot] in #14486
• chore(deps): update rust crate clap to v4.6.7 by @renovate[bot] in #14566
• chore(deps): update rust crate clap_complete to v4.6.10 by @renovate[bot] in #14567
• chore(deps): update rust crate clap_complete to v4.6.11 by @renovate[bot] in #14582
• chore(deps): update ICU derive crates by @xtqqczze in #14587
• chore(deps): update vmactions/freebsd-vm action to v1.5.6 by @xtqqczze in #14589
• chore(deps): update rust crate platform-info to v2.2.0 by @renovate[bot] in #14597
• deps: disable default features of bstr by @cakebaker in #14605

Version Management

• Prepare version 0.12.0 by @sylvestre in #14578

New Contributors

• @kennywillbe made their first contribution in #14228
• @ChrisX101010 made their first contribution in #10252
• @wangyusheng1985 made their first contribution in #14365
• @harshasiddartha made their first contribution in #14371
• @varranvar made their first contribution in #14336
• @SichenLiang made their first contribution in #14416
• @YodonTan made their first contribution in #14288
• @Hekzory made their first contribution in #14423
• @MaxFreedomPollard made their first contribution in #14417
• @ChihweiLHBird made their first contribution in #14450
• @lhecker made their first contribution in #14470
• @okaythi made their first contribution in #14530
• @keith-hall made their first contribution in #14527
• @sachinskss made their first contribution in #13368
• @yuefdev made their first contribution in #14565
• @Dijaa made their first contribution in #14508

Full Changelog: 0.11.0...0.12.0