Pangeo

Pangeo is a community effort around a stack for big geoscience data analysis, rather than a single piece of software. The components are Xarray for labelled n-dimensional arrays, Dask for parallel and out-of-core computation, Zarr for cloud-friendly storage, and Jupyter as the interface.

The idea underneath it

Traditional workflow: download data, analyse locally. That stopped working when datasets outgrew local storage. A CMIP6 analysis across many models and variables is tens of terabytes, and nobody downloads that.

Pangeo’s answer is move the computation to the data. Data sits in object storage as Zarr; computation runs on machines in the same facility; only results come back. The download step disappears, and with it the storage requirement that was the binding constraint.

Xarray is what makes it usable. Labelled dimensions mean selecting a region reads as ds.sel(lat=slice(-10, 10)) rather than index arithmetic, which is both clearer and far less error-prone — off-by-one errors in index-based slicing are silent and produce subtly wrong answers. It understands CF conventions, so netCDF and Zarr datasets open with coordinates and units intact.

Dask handles the size. An Xarray dataset backed by Dask builds a task graph instead of computing immediately, then executes in parallel across chunks when a result is requested. Analysis code looks like it operates on an in-memory array while working on something far larger.

Where it gets difficult

Lazy evaluation is a leaky abstraction. Code appears to work until something triggers computation, at which point a memory error arrives with a stack trace pointing at the wrong line. Understanding when Dask computes — and that some operations force it — is necessary rather than optional, and it is the main learning curve.

Chunking again. Dask chunks should align with the Zarr chunks underneath, or every operation reads and reshuffles. Misaligned chunking is the single most common performance problem, and it is invisible until you look.

Cost moves rather than disappearing. Cloud compute next to the data is not free, and a poorly written analysis that repeatedly triggers computation can be expensive in a way local analysis never was.

The community aspect is genuinely part of the value: shared conventions and public cloud-hosted datasets mean an analysis can be reproduced by someone else without transferring anything.

See also: Zarr, netCDF, Python, climate data, and Binder for the reproducibility angle.