DNS Caching
- GCSFuse will not do a DNS resolution for every new connection, instead it will cache the DNS lookup results. By caching DNS results, it will avoid making an additional call for DNS resolution for JSON API (using HTTP/1) shielding the GCSFuse performance from DNS latencies.
- This will improve both read and write multi-stream performance of GCSFuse significantly benefitting scenarios where HTTP/1 connections need to be created at high frequencies or multiple parallel requests to read/write data such as Orbax checkpoint restores.
- Impact:
-
- 3x reduction in accelerator block % (14.5% to 4.9%) in Llama 3.1-70B training.
-
- vLLM safetensors serving latency reduction : We observed an average 3-7% latency reduction and a peak latency reduction of 21%.
-
- Improves stability by caching DNS results and preventing call failures due to DNS resolution failures and delays
Metrics Fixes
- Fixed double counting for gcs/reader_count and gcs/download_bytes_count metrics.
- Fixed issue with metric updates via gcp-metrics-exporter in multi-mount scenarios.
-
- The metric updates in multi-mount cases on the same VM would incorrectly update the same time-series. This would result in errors saying that a metric is being updated more frequently than allowed. It is addressed in this release.
- Fixed error message during unmount due to double-shutdown of gcp-metrics-exporter.
-
- Whenever trace was enabled, during shutdown it'd raise an error: "Error while shutting down trace exporter: …" even if the error was not due to double-shutdown of gcp-metrics-exporter. With this release, this error message won’t appear during the unmount.
Full Changelog: v3.4.0 ... v3.5.1