github huggingface/datasets 3.4.0

19 hours ago

Dataset Features

  • Faster folder based builder + parquet support + allow repeated media + use torchvideo by @lhoestq in #7424

    • /!\ Breaking change: we replaced decord with torchvision to read videos, since decord is not maintained anymore and isn't available for recent python versions, see the video dataset loading documentation here for more details. The Video type is still marked as experimental is this version
    from datasets import load_dataset, Video
    
    dataset = load_dataset("path/to/video/folder", split="train")
    dataset[0]["video"]  # <torchvision.io.video_reader.VideoReader at 0x1652284c0>
    • faster streaming for image/audio/video folder from Hugging Face
    • support for metadata.parquet in addition to metadata.csv or metadata.jsonl for the metadata of the image/audio/video files
  • Add IterableDataset.decode with multithreading by @lhoestq in #7450

    • even faster streaming for image/audio/video folder from Hugging Face if you enable multithreading to decode image/audio/video data:
    dataset = dataset.decode(num_threads=num_threads)
  • Add with_split to DatasetDict.map by @jp1924 in #7368

General improvements and bug fixes

New Contributors

Full Changelog: 3.3.2...3.4.0

Don't miss a new datasets release

NewReleases is sending notifications on new releases.