HPC

High-performance computing means running work on a shared cluster: many compute nodes, a fast interconnect, a parallel filesystem, and a scheduler allocating time between users. Regional climate modelling with WRF does not happen anywhere else — the Galapagos downscaling is not a workstation job.

What is actually scarce

The name emphasises computation, and in my experience the binding constraints are elsewhere.

I/O is usually the real limit. A climate model writing output frequently spends more time in the filesystem than in the dynamics. Hundreds of processes writing simultaneously to a parallel filesystem can saturate it, and the symptom is a run that scales well up to a point and then stops improving — which looks like a compute scaling problem and is not. Reducing output frequency or aggregating writes usually buys more than more cores.

Queue time is the other. A job requesting 512 nodes may wait days; one requesting 64 may start in an hour. Total time to result is queue time plus run time, and the configuration minimising run time frequently does not minimise the sum. Requesting less and starting sooner is often the faster path, and it took me longer to internalise than it should have.

Memory per node constrains domain size in ways that are easy to discover late, at the point where the model fails during initialisation rather than at submission.

Working on a shared system

You cannot install anything system-wide. Environment modules or a container are how software arrives, and module versions change under you between sessions — which is why recording the exact module set alongside a run matters for reproducibility, and why I would use a container where the site permits it.

Login nodes are shared and not for computation. Running anything substantial there affects everyone, and it is the fastest way to receive email from an administrator.

Jobs are submitted to a scheduler, usually Slurm, with a resource request and a wall-clock limit. Checkpointing matters more than it seems: a job exceeding its limit is killed with no output, and a multi-day run that cannot resume from a checkpoint is a multi-day loss.

See also: BeeGFS, Lustre, WRF, the cluster context, and Linux administration in datacenters.