github apache/opendal v0.42.0
on GitHub

latest releases: v0.50.0, bindings/go/v0.1.3, bindings/go/v0.1.3-rc.2...
10 months ago

Upgrade Note

Rust Core

MSRV Changed

OpenDAL bumps it's MSRV to 1.67.0.

S3 Service Configuration

  • The enable_exact_buf_write option has been deprecated and is superseded by BufferedWriter, introduced in version 0.40.

Oss Service Configuration

  • The write_min_size option has been deprecated and replaced by BufferedWriter, also introduced in version 0.40.
  • A new setting, allow_anonymous, has been added. Since v0.41, OSS will now return an error if credential loading fails. Enabling allow_anonymous to fallback to request without credentials.

Ghac Service Configuration

  • The enable_create_simulation option has been removed. We add this option to allow ghac simulate create empty file, but it's could result in unexpected behavior when users create a file with content length 1. So we remove it.

Wasabi Service Removed

wasabi service native support has been removed. Users who want to access wasabi can use our s3 service instead.

Java binding

There are no API changes.

Node.js binding

There are no API changes.

Python binding

Breaking change for layers

Operator and BlockingOperator won't accept layers anymore. Instead, we provide a layer API: 

op = opendal.Operator("memory").layer(opendal.layers.RetryLayer())

We removed not used layers ConcurrentLimitLayer and ImmutableIndexLayer along with this change.

File and AsyncFile

OpenDAL removes Reader and AsyncReader classes, instead, we provide file-like object File and AsyncFile.

Open a file for reading in blocking way: 

with op.open(filename, "rb") as r:
    content = r.read()

Open a file for reading in async way: 

async with await op.open(filename, "rb") as r:
    content = await r.read()

Breaking change for Errors

We remove the old error classes and provide a couple of Exception based class for the error handling.

  1. opendal.Error is based class for all the exceptions now.
  2. opendal.exceptions.Unexpected is added.
  3. opendal.exceptions.Unsupported is added.
  4. opendal.exceptions.ConfigInvalid is added.
  5. opendal.exceptions.NotFound is added.
  6. opendal.exceptions.PermissionDenied is added.
  7. opendal.exceptions.IsADirectory is added.
  8. opendal.exceptions.NotADirectory is added.
  9. opendal.exceptions.AlreadyExists is added.
  10. opendal.exceptions.IsSameFile is added.
  11. opendal.exceptions.ConditionNotMatch is added.
  12. opendal.exceptions.ContentTruncated is added.
  13. opendal.exceptions.ContentIncomplete is added.
  14. opendal.exceptions.InvalidInput is added.

C binding

The naming convention for C binding has been altered.

Struct Naming

Renaming certain struct names for consistency.

  • opendal_operator_ptr => opendal_operator
  • opendal_blocking_lister => opendal_lister
  • opendal_list_entry => opendal_entry

API Naming

We've eliminated the blocking_ prefix from our API because C binding doesn't currently support async. In the future, we plan to introduce APIs such as opendal_operator_async_write.

  • opendal_operator_blocking_write => opendal_operator_write
  • opendal_operator_blocking_read => opendal_operator_read

What's Changed

Added

  • feat(binding/java): add rename support by @G-XD in #3238
  • feat(prometheus): add bytes metrics as counter by @flaneur2020 in #3246
  • feat(binding/python): new behavior testing for python by @laipz8200 in #3245
  • feat(binding/python): Support AsyncOperator tests. by @laipz8200 in #3254
  • feat(service/libsql): support libsql by @G-XD in #3233
  • feat(binding/python): allow setting append/buffer/more in write() call by @jokester in #3256
  • feat(services/persy): change blocking_x in async_x call to tokio::task::blocking_spawn by @Zheaoli in #3221
  • feat: Add edge test cases for OpenDAL Core by @Xuanwo in #3274
  • feat(service/d1): Support d1 for opendal by @realtaobo in #3248
  • feat(services/redb): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in #3276
  • feat: Add blocking layer for C bindings by @jiaoew1991 in #3278
  • feat(binding/c): Add blocking_reader for C binding by @jiaoew1991 in #3259
  • feat(services/sled): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in #3280
  • feat(services/rocksdb): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in #3279
  • feat(binding/java): make Metadata a POJO by @G-XD in #3277
  • feat(bindings/java): convey backtrace on exception by @tisonkun in #3286
  • feat(layer/prometheus): Support custom metric bucket for Histogram by @Zheaoli in #3275
  • feat(bindings/python): read APIs return memoryview instead of bytes to avoid copy by @messense in #3310
  • feat(service/azfile): add azure file service support by @dqhl76 in #3312
  • feat(services/oss): Add allow anonymous support by @Xuanwo in #3321
  • feat(bindings/python): build and publish aarch64 and armv7l wheels by @messense in #3325
  • feat(bindings/java): support duplicate operator by @tisonkun in #3330
  • feat(core): Add enabled for Scheme by @Xuanwo in #3331
  • feat(bindings/java): support get enabled services by @tisonkun in #3336
  • feat(bindings/java): Migrate behavior tests to new Workflow Planner by @Xuanwo in #3341
  • feat(layer/prometheus): Support output path as a metric label by @Zheaoli in #3335
  • feat(service/mongodb): Support mongodb service by @Zheaoli in #3355
  • feat: Make PrometheusClientLayer Clonable by @flaneur2020 in #3352
  • feat(service/cloudflare_kv): support cloudflare KV by @my-vegetable-has-exploded in #3348
  • feat(core): exposing Metadata::metakey() api by @G-XD in #3373
  • feat(binding/java): add list & remove_all support by @G-XD in #3333
  • feat: Add write_total_max_size in Capability by @realtaobo in #3309
  • feat(core): service add DBFS API 2.0 support by @morristai in #3334
  • feat(bindings/java): use random root for behavior tests by @tisonkun in #3408
  • feat(services/oss): Add start-after support for oss list by @wcy-fdu in #3410
  • feat(binding/python): Export full_capability API for Python binding by @Zheaoli in #3402
  • feat(test): Enable new test workflow planner for python binding by @Zheaoli in #3397
  • feat: Implement Lazy Reader by @Xuanwo in #3395
  • feat(binding/nodejs): upgrade test behavior and infra by @eryue0220 in #3297
  • feat(binding/python): Support Copy operation for Python binding by @Zheaoli in #3454
  • feat(bindings/python): Add layer API for operator by @Xuanwo in #3464
  • feat(bindings/java): add layers onto ops by @tisonkun in #3392
  • feat(binding/python): Support rename API for Python binding by @Zheaoli in #3467
  • feat(binding/python): Support remove_all API for Python binding by @Zheaoli in #3469
  • feat(core): fix token leak in OneDrive by @morristai in #3470
  • feat(core): service add OpenStack Swift support by @morristai in #3461
  • feat(bindings/python)!: Implement File and AsyncFile to replace Reader by @Xuanwo in #3474
  • feat(services): Implement ConfigDeserializer and add S3Config as example by @Xuanwo in #3490
  • feat(core): add OpenStack Swift e2e test by @morristai in #3493
  • feat(doc): add OpenStack Swift document for the website by @morristai in #3494
  • feat(services/sqlite): add SqliteConfig by @hoslo in #3497
  • feat(bindings/C): implement capability by @Ji-Xinyou in #3479
  • feat: add mongodb gridfs service support by @realtaobo in #3491
  • feat(services): add RedisConfig by @hoslo in #3498
  • feat: Add opendal_metadata_last_modified and opendal_operator_create_dir by @jiaoew1991 in #3515

Changed

  • refactor(services/sqlite): Polish sqlite via adding connection pool by @Xuanwo in #3249
  • refactor: Remove cucumber based test in python by @laipz8200 in #3253
  • refactor: Introduce OpenDAL Workflow Planner by @Xuanwo in #3258
  • refactor(bindings/C): Implement error with error message by @Ji-Xinyou in #3250
  • refactor(oay): import dav-server-opendalfs by @Young-Flash in #3285
  • refactor(bindings/java): explicit error handling by @tisonkun in #3288
  • refactor(services/gdrive): Extract folder search logic by @Xuanwo in #3234
  • refactor(core): use list_with in Operator::list by @G-XD in #3305
  • refactor(!): Bump and update MSRV to 1.67 by @Xuanwo in #3316
  • refactor(tests): Apply OPENDAL_TEST for behavior test by @Xuanwo in #3322
  • refactor(bindings/java): align test idiom with OPENDAL_TEST by @tisonkun in #3328
  • refactor(bindings/java): split behavior tests by @tisonkun in #3332
  • refactor(ci/behavior_test): Migrate to 1password instead by @Xuanwo in #3338
  • refactor(core/{fuzz,benches}): Migrate to OPENDANL_TEST by @Xuanwo in #3343
  • refactor(bindings/C): Alter naming convention for consistency by @Ji-Xinyou in #3282
  • refactor(service/mysql): Migrate to new task planner by @Zheaoli in #3357
  • refactor(service/postgresql): Migrate task to new task planner by @Zheaoli in #3358
  • refactor(services/etcd): Migrate etcd task to new behavior test planner by @Zheaoli in #3360
  • refactor(services/http): Migrate http task to new behavior test planner by @Zheaoli in #3362
  • refactor(services/sqlite): Migrate sqlite task to new behavior test planner by @Zheaoli in #3365
  • refactor(services/gdrive): migrate to test planner by @suyanhanx in #3368
  • refactor(services/redis): migrate to test planner for kvrocks,dragonfly by @suyanhanx in #3369
  • refactor(services/azblob): migrate to test planner by @suyanhanx in #3370
  • refactor(services/cos,obs): migrate to test planner by @suyanhanx in #3371
  • refactor(services/oss): migrate to test planner by @suyanhanx in #3375
  • refactor(services/memcached): migrate to test planner by @suyanhanx in #3377
  • refactor(services/gcs): migrate tot test planner by @suyanhanx in #3391
  • refactor(services/moka): migrate to test planner by @G-XD in #3394
  • refactor(services/dashmap): migrate to test planner by @G-XD in #3396
  • refactor(services/memory): migrate to test planner by @suyanhanx in #3390
  • refactor(services/azdls): migrate to test planner by @G-XD in #3405
  • refactor(services/mini_moka): migrate to test planner by @dqhl76 in #3416
  • refactor(core/fuzz): Fix some bugs inside fuzzer by @Xuanwo in #3418
  • refactor(tests): Extract tests related logic into raw::tests for reuse by @Xuanwo in #3420
  • refactor(service/dropbox): migrate to test planner by @suyanhanx in #3381
  • refactor(services/supabase): migrate to test planner by @G-XD in #3406
  • refactor(services/sftp): migrate to test planner by @suyanhanx in #3412
  • refactor(services/wasabi)!: Remove native support for wasabi services by @Xuanwo in #3455
  • refactor(ci): Polish the test planner code by @Zheaoli in #3457
  • refactor(services/webdav): migrate to test planner for webdav by @shauvet in #3379
  • refactor(services/redis): Enable rustls support by default for redis by @Xuanwo in #3471
  • refactor(bindings/python): Refactor layout for python bindings by @Xuanwo in #3473
  • refactor(services/libsql): Migrate libsql task to new behavior test planner by @Zheaoli in #3363
  • refactor(service/postgresql): Add PostgresqlConfig to implement ConfigDeserializer by @sd44 in #3495
  • refactor(binding/python): Add multiple custom exception for each of error code in Rust Core by @Zheaoli in #3492
  • refactor(service/libsql): Add LibsqlConfig to implement ConfigDeserializer by @sd44 in #3501
  • refactor(service/http): Add HttpConfig to implement ConfigDeserializer by @sd44 in #3507
  • refactor(service/ftp): Add FtpConfig to implement ConfigDeserializer by @sd44 in #3510
  • refactor(service/sftp): Add SftpConfig to implement ConfigDeserializer by @sd44 in #3511
  • refactor(service/tikv): Add TikvConfig to implement ConfigDeserializer by @caicancai in #3512

Fixed

  • fix: Fix read result not full by @jiaoew1991 in #3350
  • fix(services/cos): fix prefix param by @G-XD in #3384
  • fix(services/ghac)!: Remove enable_create_simulation support for ghac by @Xuanwo in #3423
  • fix: ASF event URL by @tisonkun in #3431
  • fix(binding/java): fix return value of presign-related method by @G-XD in #3433
  • fix(mongo/backend): remove redundant code by @bestgopher in #3439
  • fix: nodejs test adapt OPENDAL_DISABLE_RANDOM_ROOT by @suyanhanx in #3456
  • fix(services/s3): Accept List responses without ETag by @amunra in #3478
  • fix(bindings/python): fix type annotations and improve docs by @messense in #3483
  • fix(services/dropbox): Check if folder exists before calling create dir by @leenstx in #3513

Docs

CI

  • ci: Add bindings_go workflow by @jiaoew1991 in #3260
  • ci: Only fetch origin while in pull request by @Xuanwo in #3268
  • ci: add a new test case for the disk is full by @sunheyi6 in #3079
  • ci: Passing GITHUB_TOKEN to avoid rate limit by @Xuanwo in #3272
  • ci(services/hdfs): Use dlcdn.apache.org instead by @Xuanwo in #3308
  • ci: Fix HDFS test by @Xuanwo in #3320
  • ci: Fix plan not generated correctly for PR from forked repo by @Xuanwo in #3327
  • ci(services/azfile): add azfile integration test by @dqhl76 in #3409
  • ci: Fix behavior tests been ignored by @Xuanwo in #3422
  • ci(binding/java): remove testWriteFileWithNonAsciiName behavior test by @G-XD in #3424
  • ci(bindings/python): Remove not passing test cases until we addressed by @Xuanwo in #3432
  • ci(services/sftp): Move setup logic into docker-compose by @Xuanwo in #3430
  • ci(test): Add health check for WebDAV docker compose config by @Zheaoli in #3448
  • ci: Switch to 1password connect to avoid rate limit by @Xuanwo in #3447
  • ci: Use cargo test instead of carge nextest by @Xuanwo in #3505
  • build(bindings/java): Allow building on linux-aarch_64 by @amunra in #3527
  • ci: support behavior test for gridfs by @realtaobo in #3520

Chore

New Contributors

Full Changelog: v0.41.0...v0.42.0

Check out latest releases or
releases around apache/opendal v0.42.0

Don't miss a new opendal release

NewReleases is sending notifications on new releases.

Get notifications