github apache/opendal v0.40.0
on GitHub

latest releases: v0.50.2-rc.1, draft-v0.50.2, v0.50.1...
13 months ago

Checkout our OwO #1 to know more about this release!

Upgrade Note

Public API

RFC-2578 Merge Append Into Write

RFC-2578 merges append into write and removes append API.

  • For writing a file at once, please use op.write() for convenience.
  • For appending a file, please use op.write_with().append(true) instead of op.append().

The same rule applies to writer() and writer_with().

RFC-2774 Lister API

RFC-2774 proposes a new lister API to replace current list and scan. And we add a new API list to return entries directly.

  • For listing a directory at once, please use list() for convenience.
  • For listing a directory recursively, please use list_with().delimiter("") or lister_with().delimiter("") instead of scan().
  • For listing in streaming, please use lister() or lister_with() instead.

RFC-2779 List With Metakey

RFC-2779 proposes a new op.list_with().metakey() API to allow list with metakey and removes op.metadata(&entry) API.

Please use op.list_with().metakey() instead of op.metadata(&entry), for example: 

// Before
let entries: Vec<Entry> = op.list("dir/").await?;
for entry in entris {
  let meta = op.metadata(&entry, Metakey::ContentLength | Metakey::ContentType).await?;
  println!("{} {}", entry.name(), entry.metadata().content_length());
}

// After
let entries: Vec<Entry> = op
  .list_with("dir/")
  .metakey(Metakey::ContentLength | Metakey::ContentType).await?;
for entry in entris {
  println!("{} {}", entry.name(), entry.metadata().content_length());
}

RFC-2852: Native Capability

RFC-2852 proposes new native_capability and full_capability API to allow users to check if the underlying service supports a capability natively.

  • native_capability returns true if the capability is supported natively.
  • full_capability returns true if the capability is supported, maybe via a layer.

Most of time, you can use full_capability to replace capability call. But if to check if the capability is supported natively for better performance design, please use native_capability instead.

Buffered Writer

OpenDAL v0.40 added buffered writer support!

Users don't need to specify the content_length() for writer anymore! 

- let mut w = op.writer_with("path/to/file").content_length(1024).await?;
+ let mut w = op.writer_with("path/to/file").await?;

Users can specify the buffer() to control the size we call underlying storage: 

let mut w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?;

If buffer is not specified, we will call underlying storage everytime we call write. Otherwise, we will make sure to call underlying storage when buffer is full or close is called.

Raw API

RFC-3017 Remove Write Copy From

RFC-3017 removes copy_from API from the oio::Write trait. Users who implements services and layers by hand should remove this API.

What's Changed

Added

  • feat(service/etcd): support list by @G-XD in #2755
  • feat: setup the integrate with PHP binding by @godruoyi in #2726
  • feat(oay): Add read_dir by @Young-Flash in #2736
  • feat(obs): support loading credential from env by @everpcpc in #2767
  • feat: add async backtrace layer by @dqhl76 in #2765
  • feat: Add OCaml Binding by @Ranxy in #2757
  • feat(bindings/haskell): support logging layer by @silver-ymz in #2705
  • feat: Add FoundationDB Support for OpenDAL by @ArmandoZ in #2751
  • feat(oay): add write for oay webdav by @Young-Flash in #2769
  • feat: Implement RFC-2774 Lister API by @Xuanwo in #2787
  • feat(bindings/haskell): enhance original OpMonad to support custom IO monad by @silver-ymz in #2789
  • feat: Add into_seekable_read_by_range support for blocking read by @Xuanwo in #2799
  • feat(layers/blocking): add blocking layer by @yah01 in #2780
  • feat: Add async list with metakey support by @Xuanwo in #2803
  • feat(binding/php): Add basic io by @godruoyi in #2782
  • feat: fuzz test support read from .env by different services by @dqhl76 in #2824
  • feat(services/rocksdb): Add scan support by @JLerxky in #2827
  • feat: Add postgresql support for OpenDAL by @Xuanwo in #2815
  • feat: ci for php binding by @godruoyi in #2830
  • feat: Add create_dir, remove, copy and rename API for oay-webdav by @Young-Flash in #2832
  • feat(oli): oli stat should show path as specified by users by @sarutak in #2842
  • feat(services/moka, services/mini-moka): Add scan support by @JLerxky in #2850
  • feat(oay): impl some method for WebdavMetaData by @Young-Flash in #2857
  • feat: Implement list with metakey for blocking by @Xuanwo in #2861
  • feat(services/redis): add redis cluster support by @G-XD in #2858
  • feat(services/dropbox): read support range by @suyanhanx in #2848
  • feat(layers/logging): Allow users to control print backtrace or not by @Xuanwo in #2872
  • feat: add native & full capability by @yah01 in #2874
  • feat: Implement RFC-2758 Merge Append Into Write by @Xuanwo in #2880
  • feat(binding/ocaml): Add support for operator reader and metadata by @Ranxy in #2881
  • feat(core): replace field _pin with !Unpin as argument by @morristai in #2886
  • feat: Add retry for Writer::sink operation by @Xuanwo in #2896
  • feat: remove operator range_read and range_reader API by @oowl in #2898
  • feat(core): Add unit test for ChunkedCursor by @Xuanwo in #2907
  • feat(types): remove blocking operation range_read and range_reader API by @oowl in #2912
  • feat(types): add stat_with API for blocking operator by @oowl in #2915
  • feat(services/gdrive): credential manage by @suyanhanx in #2914
  • feat(core): Implement Exact Buf Writer by @Xuanwo in #2917
  • feat: Add benchmark for buf write by @Xuanwo in #2922
  • feat(core/raw): Add stream support for multipart by @Xuanwo in #2923
  • feat(types): synchronous blocking operator and operator's API by @oowl in #2924
  • feat(bindings/java): bundled services by @tisonkun in #2934
  • feat(core/raw): support stream body for mixedpart by @silver-ymz in #2936
  • feat(bindings/python): expose presign api by @silver-ymz in #2950
  • feat(bindings/nodejs): Implement presign test by @suyanhanx in #2969
  • docs(services/gdrive): update service doc by @suyanhanx in #2973
  • feat(bindings/cpp): init cpp binding by @silver-ymz in #2980
  • feat: gcs insert object support cache control by @fatelei in #2974
  • feat(bindings/cpp): expose all api returned by value by @silver-ymz in #3001
  • feat(services/gdrive): implement rename by @suyanhanx in #3007
  • feat(bindings/cpp): expose reader by @silver-ymz in #3004
  • feat(bindings/cpp): expose lister by @silver-ymz in #3011
  • feat(core): Avoid copy if input is larger than buffer_size by @Xuanwo in #3016
  • feat(service/gdrive): add gdrive list support by @Young-Flash in #3025
  • feat(services/etcd): Enable etcd connection pool by @Xuanwo in #3041
  • feat: Add buffer support for all services by @Xuanwo in #3045
  • feat(bindings/java): auto enable blocking layer by @tisonkun in #3049
  • feat(bindings/java): support presign ops by @tisonkun in #3069
  • feat(services/azblob): Rewrite the method signatures using OpWrite by @acehinnnqru in #3068
  • feat(services/cos): Rewrite the method signatures using OpWrite by @acehinnnqru in #3070
  • feat(services/obs): Rewrite method signatures using OpWrite by @hanxuanliang in #3075
  • feat(services/cos): Rewrite the methods signature using OpStat/OpRead by @acehinnnqru in #3073
  • feat: Add AtomicServer Support for OpenDAL by @ArmandoZ in #2878
  • feat(services/onedrive): Rewrite the method signatures using OpWrite by @acehinnnqru in #3091
  • feat(services/azblob): Rewrite azblob methods signature using OpRead/OpStat by @acehinnnqru in #3072
  • feat(services/obs): Rewrite methods signature in obs using OpRead/OpStat by @hanxuanliang in #3094
  • feat(service/gdrive): add gdrive copy by @Young-Flash in #3098
  • feat(services/wasabi): Rewrite the method signatures using OpRead,OpW… by @acehinnnqru in #3099

Changed

  • refactor(bindings/haskell): unify ffi of creating operator by @silver-ymz in #2778
  • refactor: Remove optimize in into_seekable_read_by_range by @Xuanwo in #2796
  • refactor(bindings/ocaml): Refactor module to support documentation by @Ranxy in #2794
  • refactor: Implement backtrace for Error correctly by @Xuanwo in #2871
  • refactor: Move object_store_opendal to integrations by @Xuanwo in #2888
  • refactor(services/gdrive): prepare for CI by @suyanhanx in #2892
  • refactor(core): Split buffer logic from underlying storage operations by @Xuanwo in #2903
  • refactor(service/webdav): Add docker-compose file to simplify the CI by @dqhl76 in #2873
  • refactor(raw): Return written bytes in oio::Write by @Xuanwo in #3005
  • refactor: Refactor oio::Write by accepting oio::Reader instead by @Xuanwo in #3008
  • refactor(core): Rename confusing pipe into copy_from by @Xuanwo in #3015
  • refactor: Remove oio::Write::copy_from by @Xuanwo in #3018
  • refactor: Make oio::Write accept Buf instead by @Xuanwo in #3021
  • refactor: Relax bounds on Writer::{sink, copy} by @huonw in #3027
  • refactor: Refactor oio::Write into poll-based to create more room for optimization by @Xuanwo in #3029
  • refactor: Polish multipart writer to allow oneshot optimization by @Xuanwo in #3031
  • refactor: Polish implementation details of WriteBuf and add vector chunks support by @Xuanwo in #3034
  • refactor: Add ChunkedBytes to improve the exact buf write by @Xuanwo in #3035
  • refactor: Polish RangeWrite implementation to remove the extra buffer logic by @Xuanwo in #3038
  • refactor: Remove the requirement of passing content_length to writer by @Xuanwo in #3044
  • refactor(services/azblob): instead parse_batch_delete_response with Multipart::parse by @G-XD in #3071
  • refactor(services/webdav): Refactor webdav_put signatures by using OpWrite. by @laipz8200 in #3076
  • refactor(services/azdls): Use OpWrite instead of passing all args as parameters by @liul85 in #3077
  • refactor(services/webdav): Use OpRead in webdav_get. by @laipz8200 in #3081
  • refactor(services/oss): Refactor oss_put_object signatures by using OpWrite by @sysu-yunz in #3080
  • refactor(services/http): Rewrite http methods signature by using OpRead/OpStat by @miroim in #3083
  • refactor(services/gcs): Rewrite gcs methods signature by using OpXxxx by @wavty in #3087
  • refactor: move all fixtures from core/src/services/{service} to top-level fixtures/{service} by @G-XD in #3088
  • refactor(services/webhdfs): Rewrite webhdfs methods signature by using OpXxxx by @cxorm in #3109

Fixed

  • fix(docs): KEYS broken link by @suyanhanx in #2749
  • fix: scheme from_str missing redb and tikv by @Ranxy in #2766
  • fix(ci): pin zig version to 0.11.0 by @oowl in #2772
  • fix: fix compile error by low version of backon in old project by @silver-ymz in #2781
  • fix: Bump openssh-sftp-client from 0.13.5 to 0.13.7 by @yah01 in #2797
  • fix: add redis for nextcloud to solve file locking problem by @dqhl76 in #2805
  • fix: Fix behaivor tests for blocking layer by @Xuanwo in #2809
  • fix(services/s3): remove default region us-east-1 for non-aws s3 by @G-XD in #2812
  • fix(oli): Fix a test name in ls.rs by @sarutak in #2817
  • fix(oli, doc): Fix examples of config.toml for oli by @sarutak in #2819
  • fix: Cleanup temporary files generated in tests automatically by @sarutak in #2823
  • fix(services/rocksdb): Make sure return key starts with input path by @Xuanwo in #2828
  • fix(services/sftp): bump openssh-sftp-client to 0.13.9 by @silver-ymz in #2831
  • fix(oli): oli commands don't work properly for files in CWD by @sarutak in #2833
  • fix(oli): oli commands should not accept invalid URI format by @sarutak in #2845
  • fix(bindings/c): Fix an example of the C binding by @sarutak in #2854
  • fix(doc): Update instructions for building the C binding in README.md by @sarutak in #2856
  • fix(oay): add some error handle by @Young-Flash in #2879
  • fix: Set default timeouts for HttpClient by @sarutak in #2895
  • fix(website): broken edit link by @suyanhanx in #2913
  • fix(binding/java): Overwrite default NOTICE file with correct years by @tisonkun in #2918
  • fix(services/gcs): migrate to new multipart impl for gcs_insert_object_request by @silver-ymz in #2838
  • fix(core): Invalid lister should not panic nor endless loop by @Xuanwo in #2931
  • fix: Enable exact_buf_write for R2 by @Xuanwo in #2935
  • fix(services/s3): allow 404 resp when deleting a non-existing object by @gongyisheng in #2941
  • fix(doc): use crate::docs::rfc to replace relative path in doc by @gongyisheng in #2942
  • fix: S3 copy error on non-ascii file path by @BoWuGit in #2909
  • fix: copy error on non-ascii file path for cos/obs/wasabi services by @BoWuGit in #2948
  • fix(doc): add GCS api reference and known issues to service/s3 doc by @gongyisheng in #2949
  • fix(oay): pass litmus copymove test by @Young-Flash in #2944
  • fix(core): Make sure OpenDAL works with http2 on GCS by @Xuanwo in #2956
  • fix(nodejs|java): Add place holder for BDD test by @Xuanwo in #2962
  • fix(core): Fix capability of services is not set correctly by @Xuanwo in #2968
  • fix(core): Fix capability of services is not set correctly by @JLerxky in #2982
  • fix(services/gcs): Fix handling of media and multipart insert by @Xuanwo in #2997
  • fix(services/webdav): decode path before set Entry by @G-XD in #3020
  • fix(services/oss): set content_md5 in lister by @G-XD in #3043
  • fix: Correct the name of azdfs to azdls by @Xuanwo in #3046
  • fix: Don't apply blocking layer when service support blocking by @Xuanwo in #3050
  • fix: call flush before sync_all by @WenyXu in #3053
  • fix: Metakeys are not propagated with the blocking operators by @Xuanwo in #3116

Docs

CI

  • build(deps): bump serde_json from 1.0.99 to 1.0.104 by @dependabot in #2746
  • build(deps): bump tracing-opentelemetry from 0.17.4 to 0.19.0 by @dependabot in #2744
  • build(deps): bump paste from 1.0.13 to 1.0.14 by @dependabot in #2742
  • build(deps): bump opentelemetry from 0.19.0 to 0.20.0 by @dependabot in #2743
  • build(deps): bump object_store from 0.5.6 to 0.6.1 by @dependabot in #2745
  • ci: use cache to speed up haskell ci by @silver-ymz in #2792
  • ci: Add setup for php and ocaml in dev container by @Xuanwo in #2825
  • ci: Trying to fix rocksdb build by @Xuanwo in #2867
  • ci: add reproducibility check by @tisonkun in #2863
  • ci(services/postgresql): add docker-compose to simplify the CI by @G-XD in #2877
  • ci(service/s3): Add docker-compose-minio file to simplify the CI by @gongyisheng in #2887
  • ci(services/hdfs): Load native lib instead by @Xuanwo in #2900
  • ci(services/rocksdb): Make sure rocksdb lib is loaded by @Xuanwo in #2902
  • build(bindings/java): bundle bare binaries in JARs with classifier by @tisonkun in #2910
  • ci(bindings/java): enable auto staging JARs on Apache Nexus repository by @tisonkun in #2939
  • ci(fix): Add PORTABLE to make sure rocksdb compiled with the same CPU feature set by @gongyisheng in #2976
  • ci(oay): Polish oay webdav test by @Young-Flash in #2971
  • build(deps): bump cbindgen from 0.24.5 to 0.25.0 by @dependabot in #2992
  • build(deps): bump actions/checkout from 2 to 3 by @dependabot in #2995
  • build(deps): bump pin-project from 1.1.2 to 1.1.3 by @dependabot in #2993
  • build(deps): bump chrono from 0.4.26 to 0.4.28 by @dependabot in #2989
  • build(deps): bump redb from 1.0.4 to 1.1.0 by @dependabot in #2991
  • build(deps): bump lazy-regex from 2.5.0 to 3.0.1 by @dependabot in #2990
  • build(deps): bump korandoru/hawkeye from 3.1.0 to 3.3.0 by @dependabot in #2994
  • ci(bindings/cpp): add ci for test and doc by @silver-ymz in #2998
  • ci(services/tikv): add tikv integration test with tls by @G-XD in #3026
  • ci: restrict workflow that need password by @dqhl76 in #3039
  • ci: Don't release while tag contains rc by @Xuanwo in #3048
  • ci(bindings/java): skip RedisServiceTest on macos and windows by @tisonkun in #3054
  • ci: Disable PHP build temporarily by @Xuanwo in #3058
  • ci(bindings/java): release workflow always uses bash by @tisonkun in #3056
  • ci(binding/java): Enable release build only when releasing by @Xuanwo in #3057
  • ci(binding/java): Use cargo profile instead of --release by @Xuanwo in #3059
  • ci: Move platform build checks from java binding to rust core by @Xuanwo in #3060
  • ci(bindings/haskell): add release workflow by @silver-ymz in #3082
  • ci: Build rc but don't publish by @Xuanwo in #3089
  • ci: Don't verify content for dry run by @Xuanwo in #3115

Chore

  • chore(core): bump cargo.toml http version to 0.2.9 by @oowl in #2740
  • chore: do not export example directory by @oowl in #2750
  • chore: Fix build after merging of ocaml by @Xuanwo in #2776
  • chore: Bump bytes to 1.4 to allow the usage of spare_capacity_mut by @Xuanwo in #2784
  • chore: disable oldtime feature of chrono by @paolobarbolini in #2793
  • chore: Disable blocking layer unitl we make all services passed by @Xuanwo in #2806
  • chore(bindings/haskell): post release 0.1.0 by @silver-ymz in #2814
  • chore(bindings/ocaml): Add contributing document to readme by @Ranxy in #2829
  • chore: Make clippy happy by @Xuanwo in #2851
  • chore: add health check for docker-compose minio by @oowl in #2899
  • chore(ci): offload healthcheck logic to docker-compose config by @oowl in #2901
  • chore: Make clippy happy by @Xuanwo in #2927
  • chore: Make C Binding clippy happy by @Xuanwo in #2928
  • chore: Fix failed ci by @silver-ymz in #2938
  • chore(ci): remove unreviewable test file and add generate test file step before testing by @gongyisheng in #3003
  • chore(bindings/cpp): update CMakeLists.txt to prepare release by @silver-ymz in #3030
  • chore: fix typo of SftpWriter error message by @silver-ymz in #3032
  • chore: Polish some details of layers implementation by @Xuanwo in #3061
  • chore(bindings/haskell): make cargo build type same with cabal by @silver-ymz in #3067
  • chore(bindings/haskell): add PVP-compliant version bounds by @silver-ymz in #3093
  • chore(bindings/java): align ErrorKind with exception code by @tisonkun in #3095
  • chore: Bump version to v0.40 to start release process by @Xuanwo in #3101
  • chore(bindings/haskell): rename library name from opendal-hs to opendal by @silver-ymz in #3112
  • chore: Bump to v0.40.0 round 2 by @Xuanwo in #3118

New Contributors

Full Changelog: v0.39.0...v0.40.0

Check out latest releases or
releases around apache/opendal v0.40.0

Don't miss a new opendal release

NewReleases is sending notifications on new releases.

Get notifications