Automatic Defaults for High Specification Machines
Cloud Storage FUSE automatically optimizes default configuration settings when running on specific high-performance Google Cloud machine types to maximize performance for demanding, high-throughput workloads. Values that are manually set at the time of mount will override these defaults.
Configurations are automated for the following high-performance machine types:
a2-megagpu-16g, a2-ultragpu-8g, a3-edgegpu-8g, a3-highgpu-8g, a3-megagpu-8g, a3-ultragpu-8g, a4-highgpu-8g-lowmem, ct5l-hightpu-8t, ct5lp-hightpu-8t, ct5p-hightpu-4t, ct5p-hightpu-4t-tpu, ct6e-standard-4t, ct6e-standard-4t-tpu, ct6e-standard-8t, ct6e-standard-8t-tpu.
When a supported machine type is detected, Cloud Storage FUSE automatically applies the following configuration values:
implicit-dirs: true
metadata-cache.negative-ttl-secs: 0
metadata-cache.ttl-secs: -1
metadata-cache.stat-cache-max-size-mb: 1024
metadata-cache.type-cache-max-size-mb: 128
file-system.rename-dir-limit: 200000
Streaming Writes is the Default Write Algorithm
Streaming Writes was introduced in v2.9.1. After the upgrade to v3.0.0, the Streaming Writes algorithm becomes the default path for New file Writes superseding the previous algorithm that temporarily stages the entire write in a local directory. The previous algorithm continues to be the fallback in case Streaming Writes cannot be used, such as for Edit file scenarios. Additional improvements to streaming writes in this release include:
In addition to Streaming writes becoming the default write path, the following improvements are also included:
Concurrent Reads During Writes: Streaming Writes now supports reading files even while writes are in progress. Upon a read request, the object will be finalized, and reads will be served directly from GCS. Subsequent write operations to the same file will revert to legacy staged writes.
Stalled Write Retries: Streaming Writes now includes support for retrying stalled write operations.
Memory Control for Streaming Writes: A new flag, --write-global-max-blocks (or the write:global-max-blocks config), has been added to control the memory usage of streaming writes. By default, each file actively being written via streaming writes is allocated one 32 MiB block. This flag allows you to control the total number of blocks utilized by streaming writes across the entire mount to limit the memory if needed. Once the memory limit is reached, any new file write will fallback to the previous algorithm.
Fix for Inconsistent File Deletions: Previously, an edge case involving out-of-order writes during streaming and subsequent file deletion from the mount could prevent files from being removed from GCS. This has now been fixed.
Truncation Support: Truncating files to a size smaller than their current size is now supported during streaming writes.
If needed, users can switch back the previous method of fully staged writes using by passing:
Command-line flag: --enable-streaming-writes=false
Configuration file: write:enable-streaming-writes:false