zarr-python

Use zarr.create_array with a store path, shape, chunks tuple, and dtype, for example zarr.create_array(store="data.zarr", shape=(10000, 10000), chunks=(1000, 1000), dtype="f4"). Convenience functions like zarr.zeros, zarr.ones, and zarr.array also accept chunk and store arguments.

Pass an fsspec URI such as s3://bucket/path.zarr directly to zarr.open_group or zarr.create_array with storage_options, or build an explicit FsspecStore.from_url. Install zarr[remote] plus pinned s3fs or gcsfs, and prefer IAM roles or workload identity for credentials.

Yes. Dask loads Zarr arrays lazily with da.from_zarr for parallel out-of-core computation, and Xarray opens Zarr stores with xr.open_zarr for labeled multidimensional datasets. Both can write results back with to_zarr.

Zarr 3 defaults to the Zarr format 3 on-disk spec, requires Python 3.12+, moves stores to zarr.storage (LocalStore, FsspecStore), replaces create_dataset with create_array, and uses zarr.codecs instead of numcodecs for v3 arrays. Synchronizers and several legacy stores were removed.

Slow reads usually come from chunk shapes misaligned with the access pattern or chunks that are too small. Aim for 1-10 MB chunks aligned with how you slice the data, use consolidated metadata on cloud stores, and consider sharding when arrays have millions of small chunks.

Use sharding when an array would produce millions of small chunk files, which causes filesystem overhead and excessive cloud object requests. Shards group many chunks into larger storage objects, though entire shards must fit in memory before writing.