huggingface/huggingface_hub v2.3.0.rc0

[v2.3.0] Bucket live-follow, faster `HfFileSystem` listings and more

on GitHub

🔄 Keep Bucket listings fresh with live_follow=True

HfFileSystem caches directory listings, so files added to a Bucket by another process did not show up until you called invalidate_cache(). With live_follow=True, HfFileSystem opens one background connection per listed Bucket to receive its file changes from the Hub. When a change arrives, it removes the cached listings it affects.

>>> from huggingface_hub import HfFileSystem
>>> hffs = HfFileSystem(live_follow=True)
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # starts following that bucket
>>> # ... another process uploads a file to the bucket ...
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # refreshed as changes arrive
['buckets/my-username/my-bucket/data/train.csv', 'buckets/my-username/my-bucket/data/new-file.csv']

It also works through fsspec with fsspec.filesystem("hf", live_follow=True), and through the storage_options argument of libraries such as pandas and pyarrow. The option is off by default because it keeps a connection open for each followed Bucket. It only affects listings and metadata (ls, info, exists, glob, ...), because reading a file always fetches its content from the Hub. It is not available for model, dataset and Space repos: use refresh=True to see a new commit.

• [HfFileSystem] Add live-follow for Buckets by @lhoestq in #5096

🚀 HfFileSystem lists large repos in parallel

Recursive listings such as find and glob used to be slow on large repos. The Hub returns a repo's file tree 1000 entries at a time, and each page can only be requested after the previous one has arrived. Listing datasets/allenai/objaverse (~800k files) took ~800 requests in a row, about 13 minutes.

HfFileSystem now splits recursive listings by sub-directory and lists them in parallel, with almost the same number of requests. A sub-tree that fits in a single page is never split. maxdepth is now applied during the listing, so the Hub only returns the levels you ask for.

>>> from huggingface_hub import hffs
>>> files = hffs.glob("datasets/allenai/objaverse/**/*.glb", maxdepth=3)

The new max_workers argument of HfFileSystem sets the number of concurrent requests. It defaults to 16. Set it to 1 to list sequentially like before.

• [HfFileSystem] List large repos in parallel by @lhoestq in #5153

📎 Attach images and videos to discussions and PRs

The new HfApi.upload_attachment(path) uploads an image, video or audio file and returns its URL, which you can embed in a discussion, PR description or comment. Before, this required calling an undocumented Hub endpoint by hand.

>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> url = api.upload_attachment("screenshot.png")
>>> api.comment_discussion("username/repo-name", 5, f"Here is the fix:\n\n![screenshot]({url})")

On the CLI, hf discussions create (including --pr), comment and edit take a repeatable --attach FILE[#alt text] option. If the body already references the file as ![alt](./file), that reference is replaced with the uploaded URL. Other files are added at the end of the body, the same way the Hub web editor does it.

$ hf discussions create celinah/hfh-attach-test --title "Fix rendering" --pr \
    --body "Before: ![Before the fix](./before.png)" --attach ./before.png --attach "./after.png#After the fix"
num=1 url=https://huggingface.co/celinah/hfh-attach-test/discussions/1 ref=refs/pr/1

$ hf discussions info celinah/hfh-attach-test 1 --format json | jq -r '.events[0].content'
Before: ![Before the fix](https://cdn-uploads.huggingface.co/production/uploads/6192895f3b8aa351a46fadfd/QdFSiYU0Plm7GM3kGSipw.png)

![After the fix](https://cdn-uploads.huggingface.co/production/uploads/6192895f3b8aa351a46fadfd/efw80nbXnJzUL5-rJXLo0.png)

• [Discussions] Add upload_attachment and --attach for media files by @hanouticelina in #5161

🧩 Read every safetensors shard header in one request

get_safetensors_metadata reads the header of each shard of a sharded model separately. The new get_sharded_safetensors_metadata gets all of them from the Hub in a single request and returns the same SafetensorsRepoMetadata. It only reads the main revision. get_safetensors_metadata is unchanged.

>>> from huggingface_hub import get_sharded_safetensors_metadata
>>> metadata = get_sharded_safetensors_metadata("HuggingFaceH4/zephyr-7b-beta")
>>> len(metadata.files_metadata)
8

• [Safetensors] Add get_sharded_safetensors_metadata to get every shard header in one request by @mishig25 in #5159

🔒 Access requests on gated repos that hide requester identity

Owners of gated repos can now choose not to share the username and/or email of the people requesting access. On those repos, list_pending_access_requests (and the accepted and rejected variants) failed with a KeyError. Each AccessRequest now has a request_id, and accept_access_request, reject_access_request and cancel_access_request accept request_id= instead of user.

>>> from huggingface_hub import list_pending_access_requests, accept_access_request
>>> for request in list_pending_access_requests("my-org/gated-model"):
...     accept_access_request("my-org/gated-model", request_id=request.request_id)

• [HfApi] Support gated repos hiding requester username/email by @Wauplin in #5047

🛡️ create_commit no longer deletes files you re-add in the same commit

create_commit skips adds and copies whose content is already on the Hub. When the same commit also deleted that path or its parent folder, only the delete was sent, and the file was removed from the repo without any error. This affected deleting a folder and pushing the same files back in one commit, and had been happening since v0.24. Those adds are now kept.

api.create_commit(repo_id, operations=[
    CommitOperationDelete("x/", is_folder=True),
    CommitOperationAdd("x/a.txt", b"keep me\n"),  # unchanged on the Hub
], commit_message="Reset x/")
# before: x/a.txt is deleted
# after:  x/a.txt is kept

• [HfApi] Keep unchanged files re-added after a delete in create_commit by @hanouticelina in #5196

💼 hf jobs run keeps streaming logs and shows queue status

hf jobs run stopped printing logs when the log stream ended before the Job did, and crashed after a few dropped connections. It now reconnects until the Job reaches a final stage. Ctrl+C used to stop the logs while the CLI kept waiting silently for the Job to end. It now exits right away and tells you that the Job is still running and how to follow or cancel it.

While a Job waits for hardware, hf jobs run, hf jobs uv run and hf jobs rerun printed nothing, so a queued Job looked like a stuck CLI. They now show the Hub's status message and how long the Job has been waiting, in interactive terminals only:

• [Jobs] Fix hf jobs run losing logs and ignoring Ctrl+C by @hanouticelina in #5162
• [CLI] Show the Job's scheduling status in hf jobs run by @hanouticelina in #5163

🐍 Python 3.15 support

huggingface_hub now officially supports Python 3.15 and is tested on it in CI. No code changes were needed. Python 3.10 is still the minimum version.

• [CI] Add CI + official support for Python 3.15 by @hanouticelina in #5155

🤖 Inference

• [Inference] Don't keep non-streaming responses alive in InferenceClient by @hanouticelina in #5197. A reused InferenceClient kept every non-streaming response in memory until close(). With 30 calls and a 1 MB prompt, that was 30 MB.
• [Inference] Run provider get_response in a thread in AsyncInferenceClient by @hanouticelina in #5200. Queue-based providers (fal-ai, wavespeed, together video, zai-org) wait for results with blocking calls, which froze the event loop for the whole job.
• [Inference] Raise InferenceTimeoutError on httpx timeouts by @hanouticelina in #5150. timeout now raises InferenceTimeoutError as documented, instead of the raw httpx timeout error.

🖥️ CLI

• [Cache] Show unreferenced shared blobs in hf cache ls by @hanouticelina in #5148. Files left behind when an older huggingface_hub version deletes a repo now show up as a hint pointing to hf cache prune.
• [CLI] Fix hf skills add when .claude/skills links to .agents/skills by @hanouticelina in #5204

🔧 Other QoL improvements

• [Jobs] Export list_scheduled_jobs from the top-level package by @guptaparin in #5140
• [Bot] Update hardware flavor enums and docs by @huggingface-hub-bot[bot] in #5175

🐛 Bug and typo fixes

• fix(kernels): allow uploads and commits to kernel repos by @tengomucho in #5233
• [HfApi] Respect custom endpoint in webhooks, access requests and user APIs by @hanouticelina in #5149. With HfApi(endpoint=...), 20 methods (webhooks, access requests, user and org info, followers, members) were sent to huggingface.co with the instance's token. Setups using HF_ENDPOINT were not affected.
• [HfFileSystem] Fix find/glob skipping folders that share a name prefix by @hanouticelina in #5147. Once a listing was cached, find() and glob() could silently skip folders such as images-ar next to images.
• [HfFileSystem] Fix glob crash when the parent listing is outdated by @hanouticelina in #5237. Since v2.2.0, Dataset.push_to_hub("buckets/...") into a new bucket failed with FileNotFoundError after uploading the data files.
• [Download] Skip the cached tree listing in offline mode by @hanouticelina in #5199. With HF_HUB_OFFLINE=1, hf_hub_download at a pinned commit could delete the existing file in local_dir instead of returning it.
• [Download] Report outdated local files as downloads in dry runs by @hanouticelina in #5219
• [Repocard] Read templates and metadata as UTF-8 by @hanouticelina in #5198. Fixes a UnicodeDecodeError when the process runs with the C locale, which broke checkpoint saving in TRL.
• [Repocard] Ignore a UTF-8 BOM when reading README metadata by @hanouticelina in #5220. metadata_update on a README saved with a BOM added a second metadata block, and the Hub lost the original metadata.

📖 Documentation

• [Docs] Fix indentation in ZipScheduler examples by @JimmyWang0417 in #5121
• [Docs] Clarify sequential writes since hf-xet 1.3.0 by @wakamex in #5146
• [Docs] Fix broken docstring examples in utils by @hanouticelina in #5151
• [Docs] Fix CommitScheduler error handling description by @hanouticelina in #5152
• [Docs] Fix Python syntax in Space management examples by @Ra1nhy in #5156
• [Docs] Fix Zarr example for zarr 3 by @hanouticelina in #5201
• [Docs] Pass keyword-only arguments by name in Jobs and create_branch examples by @Blacksujit in #5212

🏗️ Internal

• Post-release: bump version to 2.3.0.dev0 by @huggingface-hub-bot[bot] in #5122
• [CLI] Fix ty invalid-mutable-override errors by @hanouticelina in #5164
• Bump the actions group across 1 directory with 10 updates by @dependabot[bot] in #5166
• [Tests] Allow missing source in live dataset leaderboard test by @moon-bot-app[bot] in #5221
• Bump the actions group with 3 updates by @dependabot[bot] in #5228