Zarr

Zarr stores n-dimensional arrays as independently addressable chunks, each a separate object, with metadata as JSON alongside. It covers the same ground as netCDF and makes a different assumption about where the data lives.

The problem it solves

netCDF assumes a filesystem. Reading a slice means seeking within a file, which is efficient locally and impossible over HTTP — object stores have no seek, only whole-object GET. Reading one time step from a 500 GB netCDF file in S3 means downloading 500 GB.

Zarr makes each chunk its own object. Reading a slice means fetching the chunks covering it, in parallel, over ordinary HTTP. That single change is what makes cloud-hosted climate data workable rather than theoretical, and it is why Pangeo is built around it.

The parallelism is the second benefit and nearly as important. Independent chunks mean many writers can produce different regions of one array simultaneously with no coordination, which netCDF’s single-file model makes awkward.

Chunking is the decision that matters

Chunk shape determines performance far more than the format choice does, and it is fixed at write time.

Data chunked as full spatial fields per time step serves map-making well and time-series extraction terribly — pulling one grid point’s full series means reading every chunk in the dataset. Chunked the other way, the reverse. There is no shape that is good for both, and the honest answer is either choosing for the dominant access pattern or storing the data twice.

This applies equally to netCDF-4, which also chunks; Zarr just makes the consequences more visible because each chunk is a separate request with its own latency. Too-small chunks mean thousands of requests dominated by overhead; too-large means fetching far more than needed. Somewhere in the low tens of megabytes is usually reasonable.

Where it stands

Zarr does not replace netCDF so much as extend it to a different deployment. Locally, netCDF remains excellent and has decades of tooling. In object storage, Zarr is the practical option.

The formats are converging: the netCDF library can read Zarr, and Zarr datasets can carry CF conventions, which matters because the conventions are what make data interpretable and they were never tied to the container. Kerchunk goes further, indexing existing netCDF files so they can be read with Zarr-style access without duplication — a good option for archives too large to convert.

Format versioning is the practical caveat: Zarr v2 and v3 differ, and library support has been uneven.

See also: netCDF, Pangeo, CF conventions, and climate data.