google-cloud-storage-fuse

Use a PV/PVC on the pre-installed gcsfusecsi-training StorageClass on GKE 1.35.1-gke.1616000 or later, with a plain Kubernetes ServiceAccount authorized via a principal:// IAM binding on the bucket. Add the gke-gcsfuse/volumes annotation to the pod for sidecar injection.

Use native gs:// reads when your library supports them, such as pandas via gcsfs or TensorFlow tf.io.gfile. Use gcsfuse only when code is hardcoded to POSIX paths with read-heavy or new-file-write patterns; use Filestore or Managed Lustre for shared mutable writes needing file locking.

File renames are atomic since gcsfuse v3.2.0, but directory renames on flat buckets fail by default and become non-atomic if forced. The pattern is fully safe only on hierarchical namespace buckets, where directory renames are atomic metadata-only operations.

The default mount is coherency-tuned: file cache off, 60-second metadata TTL, and a 34 MiB stat cache, so every epoch re-downloads data and re-stats files. Enable the file cache sized to the dataset and set infinite metadata TTLs for immutable data, then verify with gcsfuse metrics.

No. Under streaming writes, the default since v3.0, fsync does not finalize the object; data is durable on Cloud Storage only after close(). Rely on close-time finalization rather than fsync for checkpoint durability.

Without a file cache, every epoch re-reads all objects as Class B operations, and expiring metadata plus disabled list caching add repeated objects.get and objects.list calls. Class A list operations cost roughly 12.5 times Class B per operation on standard buckets.