Apache Celeborn community is glad to announce the
new release candidate of Apache Celeborn 0.7.0-rc0
Highlight
- Engine Support: Spark 4.1/4.2 and Flink 2.1/2.2/2.3 support; Spark 4.0 and Flink 2.0 client maturity. Removed Spark 2.4, Flink 1.16/1.17, and Hadoop 2 client support.
- C++ Native Client (CIP-14): Full push/merge data path, revive, LZ4/ZSTD compression, retry logic, heartbeat decoding, native transport, and C++-write/Java-read hybrid integration tests — a major push toward feature parity with the Java client.
- Encryption at Rest (CIP-22): Spark-side implementation landed.
- Shuffle Integrity: End-to-end integrity check for Flink, iterator fully-consumed validation, and extended E2E checked zone.
- Remote Storage (S3/OSS/HDFS): S3 client caching, credentials provider support, DFS replication factor configuration, MapPartitionData on DFS, and multiple robustness fixes.
- Observability: Flush latency per storage tier, eviction counts, read buffer availability, metadata operation metrics, and more — all reflected in the Grafana dashboards.
- Performance: Zero-copy sendfile in Netty native transports, parallelized open stream and commit paths, and reduced lock contention on hot paths.
- Durability & Reliability: fsync on commit, automatic RocksDB restoration, disk-full rejection, Ratis stepdown and outbox retry, and numerous race-condition and resource-leak fixes.
- Security: RPC authorization, Java deserialization filter, and configurable HTTP auth bypass paths.
- Standalone LifecycleManager and Rust SDK, plus Helm chart improvements.
Download
https://dist.apache.org/repos/dist/dev/celeborn/v0.7.0-rc0/
Maven Staging Repository
https://repository.apache.org/content/repositories/orgapacheceleborn-1135