github apache/opendal v0.46.0
on GitHub

latest releases: v0.50.2, v0.50.2-rc.1, draft-v0.50.2...
6 months ago

Upgrade to v0.46

Public API

MSRV Changed to 1.75

Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like RPITIT and AFIT.

Services Feature Flag

Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.

Additionally, we have removed all reqwest-related feature flags:

  • Users must now directly use reqwest's feature flags for options like rustls, native-tls, etc.
  • The rustls feature is no longer enabled by default; it must be activated manually.
  • OpenDAL no longer offers the trust-dns option; users should configure the client builder directly.

Range Based Read

Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.

opendal::Reader now have APIs like: 

let r = op.reader("test.txt").await?;
let buf = r.read(1024..2048).await?;

Buffer Based IO

Since version 0.46, OpenDAL features a native Buffer struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.

OpenDAL's Reader will return Buffer and Writer will accept Buffer as input. Users who have implemented their own IO traits should update their code to use the new Buffer struct. 

let r = op.reader("test.txt").await?;
// read returns `Buffer`
let buf: Buffer = r.read(1024..2048).await?;

let w = op.writer("test.txt").await?;

// Buffer can be created from continues bytes.
w.write("hello, world").await?;
// Buffer can also be created from non-continues bytes.
w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;

// Make sure file has been written completely.
w.close().await?;

To enhance usability, we've integrated bridges into bytes::Buf and bytes::BufMut, allowing users to directly interact with the bytes API. 

let r = op.reader("test.txt").await?;
let mut bs = vec![];
// read_into accepts bytes::BufMut
let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;

let w = op.writer("test.txt").await?;

// write_from accepts bytes::Buf
w.write_from("hello, world".as_bytes()).await?;

// Make sure file has been written completely.
w.close().await?;

Bridge API

OpenDAL's Reader and Writer previously implemented APIs such as AsyncRead and AsyncWrite directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.

Since v0.46, OpenDAL provides bridge APIs for Reader and Writer instead. 

let r = op.reader("test.txt").await?;

// Convert into futures AsyncRead + AsyncSeek.
let reader = r.into_futures_async_read(1024..2048);
// Convert into futures bytes stream.
let stream = r.into_bytes_stream(1024..2048);

let w = op.writer("test.txt").await?;

// Convert into futures AsyncWrite
let writer = w.into_futures_async_write();
// Convert into futures bytes sink;
let sink = w.into_bytes_sink();

Raw API

Async in IO trait

Since version 0.46, OpenDAL has adopted Rust's native async_in_trait for our core IO traits, including oio::Read, oio::Write, and oio::List.

This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.

Users who have implemented their own IO traits should update their code to use the new async trait syntax.

What's Changed

Added

  • feat(services/github): add github contents support by @hoslo in #4281
  • feat: Allow selecting webpki for reqwest by @arlyon in #4303
  • feat(services/swift): add support for storage_url configuration in swift service by @zjregee in #4302
  • feat(services/swift): add ceph test setup for swift by @zjregee in #4307
  • docs(website): add local content search based on lunr plugin by @m1911star in #4348
  • feat(services/sled): add SledConfig by @yufan022 in #4351
  • feat : Implementing config for part of services by @AnuRage-git in #4344
  • feat(bindings/java): explicit async runtime by @tisonkun in #4376
  • feat(services/surrealdb): support surrealdb service by @yufan022 in #4375
  • feat(bindings/java): avoid double dispose NativeObject by @tisonkun in #4377
  • feat : Implement config for services/foundationdb by @AnuRage-git in #4355
  • feat: add ofs ctrl-c exit unmount hook by @oowl in #4393
  • feat: Implement RFC-4382 Range Based Read by @Xuanwo in #4381
  • feat(ofs): rename2 lseek copy_file_range getattr API support by @oowl in #4395
  • feat(services/github): make access_token optional by @hoslo in #4404
  • feat(core/oio): Add readable buf by @Xuanwo in #4407
  • feat(ofs): add freebsd OS support by @oowl in #4403
  • feat(core/raw/oio): Add Writable Buf by @Xuanwo in #4410
  • feat(bin/ofs): Add behavior test for ofs by @ho-229 in #4373
  • feat(core/raw/buf): Reduce one allocation by Arc::from_iter by @Xuanwo in #4440
  • feat: ?Send async trait for HttpBackend when the target is wasm32 by @waynexia in #4444
  • feat: add HttpClient::with() constructor by @waynexia in #4447
  • feat: Move Buffer as public API by @Xuanwo in #4450
  • feat: Optimize buffer implementation and add stream support by @Xuanwo in #4458
  • feat(core): Implement FromIterator for Buffer by @Xuanwo in #4459
  • feat(services/ftp): Support multiple write by @xxxuuu in #4425
  • feat(raw/oio): block write change to buffer by @hoslo in #4466
  • feat(core): Implement read and read_into for Reader by @Xuanwo in #4467
  • feat(core): Implement into_stream for Reader by @Xuanwo in #4473
  • feat(core): Tune buffer operations based on benchmark results by @Xuanwo in #4468
  • feat(raw/oio): Use Buffer as cache in RangeWrite by @reswqa in #4476
  • feat(raw/oio): Use Buffer as cache in OneshotWrite by @reswqa in #4477
  • feat: add list/copy/rename for dropbox by @zjregee in #4424
  • feat(core): Implement write and write_from for Writer by @zjregee in #4482
  • feat(core): Add auto ranged read and concurrent read support by @Xuanwo in #4486
  • feat(core): Implement fetch for Reader by @Xuanwo in #4488
  • feat(core): Allow concurrent reading on bytes stream by @Xuanwo in #4499
  • feat: provide send-wrapper to contidionally implement Send for operators by @waynexia in #4443
  • feat(bin/ofs): privileged mount by @ho-229 in #4507
  • feat(services/compfs): init compfs by @George-Miao in #4519
  • feat(raw/oio): Add PooledBuf to allow reuse buffer by @Xuanwo in #4522
  • feat(services/fs): Add PooledBuf in fs to allow reusing memory by @Xuanwo in #4525
  • feat(core): add access methods for Buffer by @George-Miao in #4530
  • feat(core): implement IoBuf for Buffer by @George-Miao in #4532
  • feat(services/compfs): compio runtime and compfs structure by @George-Miao in #4534
  • feat(core): change Result to default error by @George-Miao in #4535
  • feat(services/github): support list recursive by @hoslo in #4423
  • feat: Add crc32c checksums to S3 Service by @JWackerbauer in #4533
  • feat: Add into_bytes_sink for Writer by @Xuanwo in #4541

Changed

  • refactor(core/raw): Migrate oio::Read to async in trait by @Xuanwo in #4336
  • refactor(core/raw): Align oio::BlockingRead API with oio::Read by @Xuanwo in #4349
  • refactor(core/oio): Migrate oio::List to async fn in trait by @Xuanwo in #4352
  • refactor(core/raw): Migrate oio::Write from WriteBuf to Bytes by @Xuanwo in #4356
  • refactor(core/raw): Migrate oio::Write to async in trait by @Xuanwo in #4358
  • refactor(bindings/python): Change the return type of File::read to PyResult<&PyBytes> by @reswqa in #4360
  • refactor(core/raw): Cleanup not used oio::WriteBuf and oio::ChunkedBytes after refactor by @Xuanwo in #4361
  • refactor: Remove reqwest related features by @Xuanwo in #4365
  • refactor(bin/ofs): make ofs API public by @ho-229 in #4387
  • refactor: Impl into_futures_io_async_write for Writer by @Xuanwo in #4406
  • refactor(core/oio): Use ReadableBuf to remove extra clone during write by @Xuanwo in #4408
  • refactor(core/raw/oio): Mark oio::Write::write as an unsafe function by @Xuanwo in #4413
  • refactor(core/raw/oio): Use oio buffer in write by @Xuanwo in #4436
  • refactor(core/raw): oio::Write is safe operation now by @Xuanwo in #4438
  • refactor(core): Use Buffer in http request by @Xuanwo in #4460
  • refactor(core): Polish FuturesBytesStream by avoiding extra copy by @Xuanwo in #4474
  • refactor: Use Buffer as cache in MultipartWrite by @tisonkun in #4493
  • refactor: kv::adapter should use Buffer (read part) by @tisonkun in #4494
  • refactor: typed_kv::adapter should use Buffer by @tisonkun in #4497
  • refactor: kv::adapter should use Buffer (write part) by @tisonkun in #4496
  • refactor: Migrate FuturesAsyncReader to stream based by @Xuanwo in #4508
  • refactor(services/fs): Extract FsCore from FsBackend by @Xuanwo in #4523
  • refactor(core): Migrate Accessor to async fn in trait by @George-Miao in #4562
  • refactor(core): Align naming for Accessor by @George-Miao in #4564

Fixed

  • fix(bin/ofs): crashes when fh=None by @ho-229 in #4297
  • fix(services/hdfs): fix poll_close when retry by @hoslo in #4309
  • fix: fix main CI by @xxchan in #4319
  • fix: fix ghac CI by @xxchan in #4320
  • fix(services/dropbox): fix dropbox batch test panic in ci by @zjregee in #4329
  • fix(services/hdfs-native): remove unsupported capabilities by @jihuayu in #4333
  • fix(bin/ofs): build failed when target_os != linux by @ho-229 in #4334
  • fix(bin/ofs): only memory backend available by @ho-229 in #4353
  • fix(bindings/python): Fix the semantic of size argument for python File::read by @reswqa in #4359
  • fix(bindings/python): File::write should return the written bytes by @reswqa in #4367
  • fix(services/s3): omit default ports by @yufan022 in #4379
  • fix(core/services/gdrive): Fix gdrive test failed for refresh token by @Xuanwo in #4435
  • fix(core/services/cos): Don't allow empty credential by @Xuanwo in #4457
  • fix(oay): support WebdavFile continuous reading and writing by @G-XD in #4374
  • fix(integrations/webdav): Fix read file API changes by @Xuanwo in #4462
  • fix(s3): don't delete bucket by @sameer in #4430
  • fix(core): Buffer offset misuse by @George-Miao in #4481
  • fix(core): Read chunk should respect users input by @Xuanwo in #4487
  • fix(services/cos): fix mdx by @hoslo in #4510
  • fix: minor doc issue by @George-Miao in #4517
  • fix(website): community sync calendar iframe by @suyanhanx in #4549
  • fix: community sync calendar iframe load failed by @suyanhanx in #4550

Docs

CI

Chore

  • chore(deps): bump tempfile from 3.9.0 to 3.10.1 in /bin/oli by @dependabot in #4298
  • chore(deps): bump wasm-bindgen-test from 0.3.40 to 0.3.41 in /core by @dependabot in #4299
  • chore(deps): bump log from 0.4.20 to 0.4.21 in /bin/ofs by @dependabot in #4301
  • chore(services/redis): Fix docs & comments typos by @AJIOB in #4306
  • chore(editorconfig): add yaml file type by @jbampton in #4339
  • chore(editorconfig): add rdf file type as indent_size = 2 by @jbampton in #4341
  • chore(editorconfig): order entries and add indent_style = tab for Go by @jbampton in #4342
  • chore(labeler): order the GitHub labeler labels by @jbampton in #4343
  • chore: Cleanup of oio::Read, docs, comments, naming by @Xuanwo in #4345
  • chore: Remove not exist read operations by @Xuanwo in #4412
  • chore(deps): bump toml from 0.8.10 to 0.8.12 in /bin/oay by @dependabot in #4418
  • chore(deps): bump toml from 0.8.10 to 0.8.12 in /bin/oli by @dependabot in #4420
  • chore(deps): bump tokio from 1.36.0 to 1.37.0 in /bin/ofs by @dependabot in #4419
  • chore(deps): bump prometheus-client from 0.22.1 to 0.22.2 in /core by @dependabot in #4417
  • chore: update copyright year to 2024 in NOTICE by @shoothzj in #4433
  • chore: Bump bytes to 1.6 to avoid compileing error by @Xuanwo in #4472
  • chore: Add output types in OperatorFutures by @Xuanwo in #4475
  • chore(core): Use reader's chunk size instead by @Xuanwo in #4489
  • chore: sort tomls by taplo by @tisonkun in #4491
  • chore(core): Align Reader and Writer's API design by @Xuanwo in #4498
  • chore: Add docs and tests for reader related types by @Xuanwo in #4513
  • chore: Reorganize the blocking reader layout by @Xuanwo in #4514
  • chore: Align with chunk instead of confusing buffer by @Xuanwo in #4528
  • chore: Refactor Write and BlockingWrite API by @Xuanwo in #4540
  • chore(core): Change std result to opendal result in core by @tannal in #4542
  • chore: fixup download links by @tisonkun in #4547
  • chore(deps): bump clap from 4.5.1 to 4.5.4 in /bin/oay by @dependabot in #4557
  • chore(deps): bump anyhow from 1.0.80 to 1.0.82 in /bin/oli by @dependabot in #4559
  • chore(deps): bump libc from 0.2.153 to 0.2.154 in /bin/ofs by @dependabot in #4558
  • chore: bump version to 0.46.0 by @tisonkun in #4567

New Contributors

Full Changelog: v0.45.1...v0.46.0

Check out latest releases or
releases around apache/opendal v0.46.0

Don't miss a new opendal release

NewReleases is sending notifications on new releases.

Get notifications