huggingface-hub 2.4.0

[v2.4.0] Copy files into Buckets with `HfFileSystem`, outdated CLI extension hints

on Python PyPI

🪣 Copy files into Buckets with HfFileSystem

HfFileSystem raised NotImplementedError on any copy involving a Bucket. Copying into a Bucket now works from the same Bucket, from another Bucket, or from a model, dataset or Space repo. Files stored with Xet are copied on the Hub by hash, so nothing is downloaded. Other repo files are downloaded and then uploaded to the Bucket. fsspec moves a file by copying it and deleting the source, so mv and rename now work inside a Bucket too.

>>> from huggingface_hub import hffs
>>> hffs.cp_file("buckets/my-username/my-bucket/data/train.csv", "buckets/my-username/my-bucket/backup/train.csv")
>>> hffs.mv("buckets/my-username/my-bucket/data/tmp.csv", "buckets/my-username/my-bucket/data/final.csv")
>>> hffs.copy("buckets/my-username/my-bucket/data", "buckets/my-username/other-bucket/data", recursive=True)
>>> hffs.cp_file("datasets/my-username/my-dataset/train.parquet", "buckets/my-username/my-bucket/train.parquet")

Copying from a Bucket to a repo is not supported yet.

• [HfFileSystem] Support same-bucket file copies by @ankitkumartiwari-web in #5080

🧩 See when a CLI extension is outdated

hf extensions list only showed when each extension was installed, so there was no way to know that an update was available. It now has a VERSION column (the commit the extension was installed from) and a LATEST column, plus a hint naming the outdated extensions. Running an outdated extension also prints a hint, at most once a day. To stay within GitHub's rate limits, hf looks up the latest commit of each extension at most once a day. Set HF_HUB_DISABLE_UPDATE_CHECK=1 to turn these checks off.

hf extensions update also repairs Python extensions that no longer start. After a Homebrew upgrade removed the Python an extension was installed with, running it failed with dyld: Library not loaded, and hf extensions update said it was already up to date. It now recreates the extension's virtual environment when its Python doesn't start, even if no newer commit exists.

$ hf extensions update speedtest
Creating virtualenv at: ~/.local/share/hf/extensions/hf-speedtest/venv
✓ Extension updated

• [CLI] Show extension versions and hint when one is outdated by @hanouticelina in #5269
• [CLI] Make hf extensions update repair broken Python extensions by @hanouticelina in #5270

🔧 Other QoL improvements

• Add auth hint to HfFileSystem "repository not found" error by @davanstrien in #5258. A private repo accessed without a token gave the same (repository not found) error as a typo in the repo name. The error now also tells you to check that you are authenticated.

🐛 Bug and typo fixes

• [Webhooks] Use "discussion" domain and default to all domains by @hanouticelina in #5263. hf webhooks create failed unless you passed --domain repo, and watching discussions was not possible because the accepted value was discussions instead of discussion. create_webhook now watches both domains when domains is omitted.
• Validate filename against path traversal in try_to_load_from_cache by @bodapatisaikrishna in #5249. Filenames with ../ or absolute paths could resolve outside the cache folder. They are now rejected, as in hf_hub_download.
• [HfFileSystem] Return file info from find(detail=True) on a file path by @ontdepart in #5246. It returned an empty dict, which broke datatrove readers pointing at a single hf:// file.
• [Cache] Skip local-dir metadata when verifying checksums by @alphasafal in #5257. hf cache verify --local-dir reported the .cache/huggingface/ metadata as extra files, so --fail-on-extra-files failed on every clean download.
• [CLI] Don't crash hf cache verify on Windows redirected stdout by @Sonica-B in #5255. When output was piped or redirected, hf cache verify crashed with UnicodeEncodeError instead of listing the files that failed verification.

📖 Documentation

• [Docs] Fix docstring examples that fail when copy-pasted by @tkr109 in #5238
• [Docs] Await endpoint batch inference before pausing by @emrehangorgec in #5247
• Docs: build English only, other languages are machine-translated by @mishig25 in #5276. The docs in other languages are now machine-translated from the English docs, in 11 languages.

🏗️ Internal

• Post-release: bump version to 2.4.0.dev0 by @huggingface-hub-bot[bot] in #5240
• [Tests] Don't pin the avatar hash in OAuth test by @hanouticelina in #5241
• [CI] Drop urllib3<2.0 test pin broken by vcrpy 9 by @hanouticelina in #5262
• Bump the actions group with 7 updates by @dependabot[bot] in #5268