huggingface/datasets 5.1.0 on GitHub

New Formats

Harbor ⚓: RL environments:
- Add harbor benchmarks by @lhoestq in #8729
Vortex 🌀: High performance columnar data
- Add support for datasets backed by vortex file format by @robert3005 in #8502
Bio formats 🧬: store biological data like DNA sequences, protein chains, and 3D molecular structures
- feat(fasta): add lightweight FASTA file format support by @behroozazarkhalili in #7923
- Decode BioSequence and BioStructure records with universal newlines by @behroozazarkhalili in #8600
- Add lightweight FASTQ file format support by @behroozazarkhalili in #7924
- feat: Add GenBank file format support for biological sequence data by @behroozazarkhalili in #7951
- Add lightweight PDB (Protein Data Bank) file support by @behroozazarkhalili in #7926
- feat: Add mmCIF file support for macromolecular structures by @behroozazarkhalili in #7925
- feat(features): add BioSequence and BioStructure decodable features by @behroozazarkhalili in #8542

Dataset Features

Add a strategy argument to split_dataset_by_node for iterable datasets by @behroozazarkhalili in #8565
Allow IterableDataset.push_to_hub num_shards above dataset.num_shards by @behroozazarkhalili in #8716

Other improvements and bug fixes

Fix fixed size binary by @Alex-PLACET in #8281
Fix offline simulation with huggingface_hub 1.27 by @hanouticelina in #8414
Fix info from data_dir/data_files or malformed yaml by @lhoestq in #8462
Forward iter_arrow, is_typed and features from RepeatExamplesIterable by @ebarkhordar in #8395
Fix archive path check allowing sibling-directory escapes by @icn5381 in #8490
disallow hdf5 external links by @lhoestq in #8523
Fix deepcopy bottleneck in state_dict() by @yuxin00j in #8387
Optimize PyArrow block shuffling with dynamic rebatched chunk interleaving by @yuxin00j in #8407
Fix Pdf.decode_example raising TypeError on remote non-Hub paths by @behroozazarkhalili in #8566
Fix dead PDF processing link in README by @aarushkandukoori in #8512
Forward max_buffer_input_shards from IterableDatasetDict.shuffle by @vineethsaivs in #8594
Preserve storage options across URL hops by @tandede in #8496
Better error when a column written as null gets values in a later batch by @ebarkhordar in #8591
Advance the user-supplied generator in Dataset.shuffle even on cache hits by @BenYang12 in #8589
Bump fsspec upper bound to 2026.7.0 by @roshanjrajan in #8561
Shard the sources when skipping a distributed IterableDataset by @ebarkhordar in #8569
Fix infinite repetition of empty iterable datasets by @Jokasa7 in #8577
Reject a non-callable generator in GeneratorConfig by @LeSingh1 in #8585
Fix async map checkpoint accounting across repeated resumes by @SBrouck in #8575
[httpx migration] Import httpx from huggingface_hub, require huggingface_hub>=1.31.0 by @Wauplin in #8598
chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in #8221
[docs] Fix legacy hf CLI references (datasets) by @Wauplin in #8657
Scope GITHUB_TOKEN permissions per job by @hf-security-analysis[bot] in #8672
Pin GitHub Actions to commit SHAs by @hf-security-analysis[bot] in #8673
Fix push_to_hub for IterableDataset with unknown features by @davanstrien in #8658
Vectorize interleave_datasets index generation (probabilities + first/all_exhausted) by @saforem2 in #8318
Fix torchcodec docs links by @pratikgx in #8700
[docs] Fix HF_HOME example in cache guide by @ZainnQureshii in #8694
Fix broken internal anchor link in share.mdx by @aarushkandukoori in #8666
Keep Json fields inside a struct from being double-encoded when some struct rows are null by @breken-ai in #8712
Don't import aiohttp when importing datasets by @kuldeeepy in #8680
Fix inverted span for minus-strand multi-exon GenBank locations by @Linxiushen in #8677
Key parquet column options by leaf path so nested columns get compressed by @jayzuccarelli in #8674
Implement arrow_array on PandasArrayExtensionArray so ArrayXD columns can go back from pandas to Arrow by @JoeyTan21 in #8701
Batch in arrow when the source is arrow-backed, and let _consolidate type numpy scalars by @Yonghui-Lee in #8665
Fix Json.cast_storage double-encoding large_string and string_view columns by @SulimanAbdulrazzaq in #8682
Fix audio array handling in preprocess_function by @causius0 in #8731
Decode remote NIfTI files through the bytes path by @KennyMcSimpson in #8737
Keep take/skip counts when there are more DataLoader workers than shards by @yupengtang in #8695
Fix epoch iteration after skipping or taking a streaming subset by @galleonli in #8707
Fix IterableDataset.filter with arrow, pandas and polars formats by @felipebridge in #8725
Bump huggingface_hub upper bound to <3.0 (datasets) by @Wauplin in #8678
Preserve source shards when shuffling iterable datasets by @yuchenwang3 in #8670
Speed up JSON export of indexed datasets by @aprylewu in #8662
Fix repeated Arrow rebatch checkpoint restoration by @tryu-ai in #8727
fix default iterable dataset dict and polars exports by @cwklurks in #8632
harbor: fix paths on windows by @lhoestq in #8739

New Contributors

@Alex-PLACET made their first contribution in #8281
@robert3005 made their first contribution in #8502
@icn5381 made their first contribution in #8490
@yuxin00j made their first contribution in #8387
@behroozazarkhalili made their first contribution in #8565
@aarushkandukoori made their first contribution in #8512
@tandede made their first contribution in #8496
@BenYang12 made their first contribution in #8589
@roshanjrajan made their first contribution in #8561
@Jokasa7 made their first contribution in #8577
@LeSingh1 made their first contribution in #8585
@SBrouck made their first contribution in #8575
@hf-dependantbot-rollout[bot] made their first contribution in #8221
@saforem2 made their first contribution in #8318
@pratikgx made their first contribution in #8700
@ZainnQureshii made their first contribution in #8694
@breken-ai made their first contribution in #8712
@kuldeeepy made their first contribution in #8680
@Linxiushen made their first contribution in #8677
@JoeyTan21 made their first contribution in #8701
@Yonghui-Lee made their first contribution in #8665
@SulimanAbdulrazzaq made their first contribution in #8682
@causius0 made their first contribution in #8731
@KennyMcSimpson made their first contribution in #8737
@yupengtang made their first contribution in #8695
@galleonli made their first contribution in #8707
@felipebridge made their first contribution in #8725
@yuchenwang3 made their first contribution in #8670
@aprylewu made their first contribution in #8662
@tryu-ai made their first contribution in #8727
@cwklurks made their first contribution in #8632

Full Changelog: 5.0.1...5.1.0