juicedata/juicefs v1.4.0-beta1

on GitHub

This is the first beta release for JuiceFS v1.4. Since v1.3 it has 373 commits from 49 contributors, thanks to @jiefenghuang @zhijian-pro @Xuyuchao-juice @YunhuiChen @zxh326 @polyrabbit @CaitinChen @chenjie4255 @yuhr123 @solracsf @201341 @cxljs @davies @Light-City @Looka149 @wzhallright @zhoucheng361 @mdolr @eakman-datadog @JoyLiuc @mt5225 @prOOrc @sw-cho @tangyoupeng @anthonyeleven @asyqm @CodeRusher @CodingPoeta @cuiweixie @hegdeadithyak @Joffref @juchaosong @krmeda-msft @kungf @kyungwan-nam @lesterMing @lynus @mcassaniti @mwkang @mzygQAQ @nautaa @ndeepak-baseten @qingzhongli @rifeplight @simpleqin @VerTiGoEtrex @vyalamar @zexee @zhitaoli-6

Highlights

• Support storage tiers with configurable storage classes (#6761,#6769,#6809,#6805,#6803,#6854,#6820,#6824,#6816,#6807,#6786,#6817,#6799,#6831,#6787,#6785,#6795,#6782,#6930)
• Support user and group quota management (#6330,#6369,#6386,#6396,#6361,#6766,#6789,#6813,#6511,#6381,#6771,#6776,#6743,#6751,#6737,#6826,#6788,#6698,#6808,#6480,#6933)
• Support checkpoint for resumable sync (#6755,#6852,#6905,#6877,#6892,#6872,#6870,#6839,#6950,#6958,#6959)
• Support metadata operation changelog (#6777,#6942,#6951,#6956,#6979)

New

• Support hadoop kerberos (#6445, #6644, #6970)
• windows: improved stability and compatibility (#6662,#6651,#6645,#6643,#6642,#6641,#6634,#6547,#6907)
• cmd/rmr: support batch unlink (#6553,#6581,#6618,#6537,#6457,#6564,#6603,#6609,#6610,#6647)
• cmd/clone: support batch clone (#6656,#6695,#6828,#6779,#6650,#6922,#6715,#6938)
• meta/redis: support client-side caching (#6495,#6857)
• meta: support trash realtime statistics (#6818,#6838,#6899,#6901)
• cache: support LRU cache (#6271)
• cmd/sync: support encryption/decryption (#6806,#6834,#6955,#6947)
• cmd/sync: support global traffic control (#6626)
• Support SM encryption for format and data at rest (#6419, #6414)
• Support end-to-end context cancellation (#6312, #6516, #6382, #6383, #6377, #6379, #6567, #6565, #6598, #6582, #6397, #6499, #6524, #6590, #6410, #6477, #6927)
• object: support cifs/smb (#6368, #6712, #6702, #6949)
• object: support native Storj object storage via storj.io/uplink (#6763,#6929)
• vfs: add support to skip trash if s attribute is set (#6359)
• cmd/fsck: support check data by path (#6475)
• cmd/fsck: add repair-dir-mode option to specify directories mode (#6561)
• cmd/mount: add max-downloads option to limit concurrent get for object storage (#6472)
• cmd/mount: add writeback-threshold-size option to filter blocks for writeback (#6447)
• cmd/mount: add fuse option ReadDirPlusAuto for automatic readdir optimization (#6299)
• cmd/mount: add network-interfaces option for IP discovery configuration (#6392)
• cmd/mount: add hide-internal option (#6307,#6911)
• cmd/mount: add META_PASSWORD_FILE env to read metadata engine password from file (#6438)
• cmd/gateway: add --bucket-name option for using custom bucket names (#6741)
• cmd: add mountpoint option to improve symlink resolution for absolute paths (#6228)
• sdk/java: support multiple subdir for hadoop-sdk (#6608)
• sdk/java: add max-downloads configuration option (#6679)
• sdk/java: add startTime option for nnbench (#6757)
• sdk/java: add support for sending metrics via Prometheus remote_write protocol (#6296)
• object: allow unordered list in fsck/backup/gc/destroy commands (#6389)
• object: add support for enabling/disabling default CRC checksum via query parameters (#6239,#6232)
• object/azure: add managed identity authentication support (#6424)
• metrics: add metadata background jobs metrics (#6544, #6660)
• metrics: add subdir mount configuration metric (#6592)

Changed

• meta: speed up background trash cleanup job (#6602)
• meta/badger: improve reliability and resource management (#6725)
• meta: replace bulk doReaddir with batched DirHandler in cloneEntry to reduce memory pressure on large directories (#6469)
• meta/backup: delete backups older than two years (#6484)
• fs: return sorted entries when sort-dir is true (#6442, #6468)
• cache: speed up scanning (#6549, #6528, #6557)
• cache/stage: reduce the priority of the stage cache by introducing a cooldown period (#6435)
• cache: use partial reads when cache is disabled to halve read amplification (#6364)
• cmd/mount: warn when both readdir-cache and skip-dir-mtime are enabled as they may conflict (#6209)
• cmd/mount: warn when negative-entry-cache is used on kernels below 5.11 to avoid concurrent mkdir failures (#6302)
• cmd/sync: limit cached directory entries number per level to reduce memory usage (#6652)
• cmd/sync: reduce the number of concurrent large files (#6341,#6372)
• cmd/sync: set umask in sync cluster mode to preserve correct file permissions (#6400)
• cmd/sync: only retry temporary failures (#6132)
• cmd/sync: skip non-existent paths in files-from list instead of failing (#6339)
• cmd/sync: skip empty keys to prevent errors from blank entries (#6193)
• cmd/sync: rename sync temporary files to ".jfs.xxx.tmp.xxx" (#6665)
• cmd/sync: warn and skip non-regular files during entry listing (#6954)
• cmd/gateway: improve performance for batch deletion of objects (#6605)
• cmd/gateway: reduce memory for list objects (#6476)
• cmd/gateway: add ETag verification in multi-part upload completion for data integrity (#6485)
• cmd/gateway: add descriptive error messages for bucket name validation failures (#6773)
• cmd/restore: improve restore progress labels and conflict description (#6888)
• cmd: use %q for CLI-derived strings in logs (#6804)
• cmd: add adaptive sleep time to read response (#6526)
• object/oss: support oss private link endpoint (#6686)
• object/oss: support switching signature version v1 (#6631)
• object/s3: limit the maximum number for list operations to 1000 (#6367)
• object/bos: remove redundant data validation from Put interface, already covered by SDK (#6904)
• object/wasb: support SAS token authentication via --session-token (#6926)
• sdk/java: exclude gspt from libjfs.so to prevent Java process argv truncation on Linux (#6726)
• sdk: support fully closing filesystem on delete (#6939)
• build: support make debug mode in makefile (#6573)
• monitor: merge grafana dashboards with node_label variable (#6492)

Bug Fixes

• fuse: disable readdir-cache for special inodes like .trash to avoid showing stale entries (#6690)
• vfs: remove O_TMPFILE file directly when closed (#6464)
• meta/redis: fix potential deadlock when rename triggers checkTrash with nested transactions (#6558)
• meta/redis: close standalone client after cluster mode detection (#6948)
• meta/tkv: fix potential data corruption issues caused by clock diff and TiKV gc (#6217)
• meta/tkv: avoid repeatedly setting the same key which can lead to excessive memory usage in CopyFileRange (#6449)
• meta/sql: fix tx_isolation setting error for MySQL <= 5.7 (#6768)
• meta: fix format decrypt error when uuid changed (#6542)
• meta: fix stale cache of the overwritten file in rename (#6893)
• meta: fix GetDirStat returning stale inodes count before flush (#6886)
• meta: fix group check when setting the setgid bit (#6943)
• cache: fix disk cache full checking logic (#6675)
• cache/stage: fix duplicate upload of staged blocks (#6697)
• cache/stage: fix potential EIO when reading stage blocks (#6388)
• cache: fix cache-items option description and log (#6252)
• cmd/mount: add nofail to fstab mount options to improve mount reliability (#6280)
• cmd/mount: fix fstab option parsing that dropped parameter values like attr-cache=1 to attr-cache (#6420)
• cmd/mount: disable shared metrics reporting when no-bgjob is set (#6530)
• cmd/sync: fix failed object count (#6533)
• cmd/sync: fix metrics for excluded and extra objects (#6863)
• cmd/sync: fix lost count in summary log (#6928)
• cmd/config: fix bug where --bucket is treated as local path when --storage is not explicitly set (#6710)
• cmd/fsck: fix repair to set directory permissions to 0755
• cmd/info: fix tail empty slice in output (#6832)
• cmd/destroy: allow destroy with --force to ignore missing bucket (#6767)
• cmd/gateway: fix incorrect object name assignment in getObjectInfoNoFSLock (#6493)
• cmd/objbench: fix incorrect mtime check log message in list results (#6490)
• sdk/python: load platform-specific libjfs extension (#6967)
• sdk: close file handle in jfs_chown to prevent fd leak (#6965)
• sdk: fix close session handling (#6977)
• sdk/java: fix resource leaks, ancestor lookup, and null checks (#6964,#6961,#6960)
• object: fix client dial delay and prefer ipv6 (#6753)
• object/bos: fix endpoint format for bucket (#6746)
• object/s3: use default credential chain in autoS3Region when accessKey is empty (#6754)
• object/azure: handle nil ArchiveStatus to prevent panic (#6843)
• object/s3: pass session token to region auto-detection for STS support (#6946)
• object/minio: fix MinIO object storage info printing to remove duplicated bucket name in path (#6310)
• object: decode S3 object listing responses based on the encoding type returned by server (#6532)

Deps

• deps: upgrade pgx/v5 from 5.7.3 to 5.9.2 (#6963)
• deps: upgrade ibm-cos-sdk-go from v1.12.1 to v1.13.0 (#6851)
• deps: upgrade qingstor sdk from v4.4.0 to v4.4.1 (#6855)
• deps: upgrade bunny sdk from v0.0.10 to v1.0.0 (#6848)
• deps: upgrade bce sdk from v0.9.221 to v0.9.263 (#6847)
• deps: upgrade azure sdk from v1.3.1 to v1.6.4 (#6842)
• deps: upgrade tos sdk from v2.7.8 to v2.9.0 (#6840)
• deps: upgrade cos sdk from v0.7.63 to v0.7.73 (#6837)
• deps: upgrade oss sdk from v1.2.1 to v1.4.1 (#6836)
• deps: update go-fuse to fix flock (#6778)
• deps: bump gmsm to v0.41.1 (#6775)
• deps: update go version to 1.25 (#6303, #6680, #6862)
• deps: upgrade aws-sdk-go-v2 from v1.36.1 to v1.41.5 (#6883)
• deps: upgrade dgraph-io/badger to v4.9.1 (#6879)
• deps: upgrade go-sql-driver/mysql to v1.9.3 (#6878)
• deps: upgrade redis sdk from v9.16.0 to v9.18.0 (#6871)
• deps: upgrade sftp sdk from v1.13.5 to v1.13.10 (#6868)
• deps: upgrade qiniu sdk from v7.25.2 to v7.26.8 (#6861)