Upgrade

Rust Core Public API

New VISION: One Layer, All Storage

OpenDAL has refined its vision to One Layer, All Storage, driven by the following core principles: Open Community, Solid Foundation, Fast Access, Object Storage First, and Extensible Architecture.

Explore the detailed vision at OpenDAL Vision.

RFC-5313: Remove Metakey

OpenDAL v0.51 implements RFC-5313, which removes the concept of metakey.

The following structs have been removed:

• Metakey

The following APIs have been removed:

• list_with(path).metakey()

Users no longer need to pass the metakey into the list. Instead, services will make their best effort to return as much metadata as possible. Users can check items like Capability::list_has_etag before making a call.

Remove not used capability: write_multi_align_size

The capability write_multi_align_size is not utilized by any services, and we have no plans to support it in the future; therefore, we have removed it.

CapabilityCheckLayer and CorrectnessCheckLayer

OpenDAL used to perform capability checks for all services, but since v0.51, it only conducts checks that impact data correctness like write_with_if_not_exists or delete_with_version by default in the CorrectnessCheckLayer. If users wish to verify other non-critical capabilities like write_with_content_type or write_with_cache_control, they should manually enable the CapabilityCheckLayer.

RFC-3911: Deleter API

OpenDAL v0.51 implements RFC-3911, which adds Deleter in OpenDAL to replace batch operation.

The following new APIs have been added:

• [Operator::delete_iter]
• [Operator::delete_try_iter]
• [Operator::delete_stream]
• [Operator::delete_try_stream]
• [Operator::deleter]
• [Deleter::delete]
• [Deleter::delete_iter]
• [Deleter::delete_try_iter]
• [Deleter::delete_stream]
• [Deleter::delete_try_stream]
• [Deleter::flush]
• [Deleter::close]
• [Deleter::into_sink]
• [DeleteInput]
• [IntoDeleteInput]
• [FuturesDeleteSink]

The following APIs have been deprecated and will be removed in the future releases:

• Operator::remove (replace with [Operator::delete_iter])
• Operator::remove_via (replace with [Operator::delete_stream])

As a result of this change, the limit and with_limit APIs on Operator have also been deprecated; they are currently no-ops.

Rust Core Raw API

adapter::kv now returns Scanner instead of Vec<String>

To support returning key-value entries in a streaming manner instead of loading them all into memory, OpenDAL updated its adapter API to return a Scanner instead of a Vec<String>.

- async fn scan(&self, path: &str) -> Result<Vec<String>>
+ async fn scan(&self, path: &str) -> Result<Self::Scanner>

All services intending to implement kv::Adapter should adhere to this API change.

Align metadata API to info

OpenDAL changes it's old metadata API to info to align with the new AccessorInfo struct.

- fn metadata(&self) -> Arc<AccessorInfo>
+ fn info(&self) -> Arc<AccessorInfo>

Remove not used struct: RangeWriter

The struct RangeWriter is not utilized by any services, and we have no plans to support it in the future; therefore, we have removed it.

What's Changed

Added

• feat(adapter/kv): support async iterating on scan results by @PragmaTwice in #5208
• feat(bindings/ruby): Add simple operators to Ruby binding by @erickguan in #5246
• feat(core/services-gcs): support user defined metadata by @jorgehermo9 in #5276
• feat(core): add if_not_exist in OpWrite by @kemingy in #5305
• feat: Add {stat,list}has* to carry the metadata that backend returns by @Xuanwo in #5318
• feat(core): Implement write if not exists for azblob,azdls,gcs,oss,cos by @Xuanwo in #5321
• feat(core): add new cap shared by @TennyZhuang in #5328
• feat(bindings/python): support pickle [de]serialization for Operator by @TennyZhuang in #5324
• feat(bindings/cpp): init the async support of C++ binding by @PragmaTwice in #5195
• feat(bindings/go): support darwin by @yuchanns in #5334
• feat(services/gdrive): List shows modified timestamp gdrive by @erickguan in #5226
• feat(service/s3): support delete with version by @Frank-III in #5349
• feat: upgrade pyo3 to 0.23 by @XmchxUp in #5368
• feat: publish python3.13t free-threaded wheel by @XmchxUp in #5387
• feat: add progress bar for oli cp command by @waynexia in #5369
• feat(types/buffer): skip copying in to_bytes when NonContiguous contains a single Bytes by @ever0de in #5388
• feat(bin/oli): support command mv by @meteorgan in #5370
• feat(core): add if-match to OpWrite by @Frank-III in #5360
• feat(core/layers): add correctness_check and capability_check layer to verify whether the operation and arguments is supported by @meteorgan in #5352
• feat(bindings/ruby): Add I/O class for Ruby by @erickguan in #5354
• feat(core): Add content_encoding to MetaData by @Frank-III in #5400
• feat:(core): add content encoding to Opwrite by @Frank-III in #5390
• feat(services/obs): support user defined metadata by @Frank-III in #5405
• feat: impl configurable OperatorOutputStream maxBytes by @tisonkun in #5422

Changed

• refactor (bindings/zig): Improvements by @kassane in #5247
• refactor: Remove metakey concept by @Xuanwo in #5319
• refactor(core)!: Remove not used cap write_multi_align_size by @Xuanwo in #5322
• refactor(core)!: Remove the range writer that has never been used by @Xuanwo in #5323
• refactor(core): MaybeSend does not need to be unsafe by @drmingdrmer in #5338
• refactor: Implement RFC-3911 Deleter API by @Xuanwo in #5392
• refactor: Remove batch concept from opendal by @Xuanwo in #5393

Fixed

• fix(services/webdav): Fix lister failing when root contains spaces by @skrimix in #5298
• fix(bindings/c): Bump min CMake version to support CMP0135 by @palash25 in #5308
• fix(services/webhdfs): rename auth value by @notauserx in #5342
• fix(bindings/cpp): remove the warning of CMP0135 by @PragmaTwice in #5346
• build(python): fix pyproject meta file by @trim21 in #5348
• fix(services/unftp): add / when not presented by @Frank-III in #5382
• fix: update document against target format check and add hints by @waynexia in #5361
• fix: oli clippy and CI file by @waynexia in #5389
• fix(services/obs): support huawei.com by @FayeSpica in #5399
• fix(integrations/cloud_filter): use explicit stat instead of Entry::metadata in fetch_placeholders by @ho-229 in #5416
• fix(core): S3 multipart uploads does not set file metadata by @catcatmu in #5430
• fix: always contains path label if configured by @waynexia in #5433

Docs

• docs: Enable force_orphan to reduce clone size by @Xuanwo in #5289
• docs: Establish VISION for "One Layer, All Storage" by @Xuanwo in #5309
• docs: Polish docs for write with if not exists by @Xuanwo in #5320
• docs(core): add the description of version parameter for operator by @meteorgan in #5144
• docs(core): Add upgrade to v0.51 by @Xuanwo in #5406
• docs: Update release.md by @tisonkun in #5431

CI

• ci: Remove the token of codspeed by @Xuanwo in #5283
• ci: Allow force push for gh-pages by @Xuanwo in #5290
• build(bindings/java): fix lombok process by @tisonkun in #5297
• build(bindings/python): add python 3.10 back and remove pypy by @trim21 in #5347

Chore

• chore(core/layers): align info method of trait Access and trait LayeredAccess by @koushiro in #5258
• chore(core/raw): align info method of kv::Adapter and typed_kv::Adapter by @koushiro in #5259
• chore(layers/oteltrace): adjust tracer span name of info method by @koushiro in #5285
• chore(services/s3): remove versioning check for s3 by @meteorgan in #5300
• chore: Polish the debug output of capability by @Xuanwo in #5315
• chore: Update maturity.md by @tisonkun in #5340
• chore: remove flagset in cargo.lock by @meteorgan in #5355
• chore: add setup action for hadoop to avoid build failures by @meteorgan in #5353
• chore: fix cargo clippy by @meteorgan in #5379
• chore: fix cargo clippy by @meteorgan in #5384
• chore: fix Bindings OCaml CI by @meteorgan in #5386
• chore: Add default vscode config for more friendly developer experience by @Zheaoli in #5331
• chore(website): remove outdated description by @meteorgan in #5411
• chore(deps): bump clap from 4.5.20 to 4.5.21 in /bin/ofs by @dependabot in #5372
• chore(deps): bump anyhow from 1.0.90 to 1.0.93 in /bin/oay by @dependabot in #5375
• chore(deps): bump serde from 1.0.210 to 1.0.215 in /bin/oli by @dependabot in #5376
• chore(deps): bump openssh-sftp-client from 0.15.1 to 0.15.2 in /core by @dependabot in #5377
• chore(ci): fix invalid Behavior Test Integration Cloud Filter trigger by @Zheaoli in #5414

New Contributors

• @skrimix made their first contribution in #5298
• @palash25 made their first contribution in #5308
• @kemingy made their first contribution in #5305
• @drmingdrmer made their first contribution in #5338
• @notauserx made their first contribution in #5342
• @trim21 made their first contribution in #5348
• @Frank-III made their first contribution in #5349
• @ever0de made their first contribution in #5388
• @FayeSpica made their first contribution in #5399
• @catcatmu made their first contribution in #5430

Full Changelog: v0.50.2...v0.51.0