Lustre

Lustre is the parallel filesystem used by a large share of the world’s biggest supercomputers. Structurally it resembles BeeGFS: metadata servers holding the namespace, object storage servers holding file data, and clients accessing both directly.

The difference is scale and what it costs to reach it. Lustre scales to thousands of servers and terabytes per second, and it has been proven at the top of the field for two decades.

The trade

It expects dedicated expertise. Lustre components have historically been tied closely to specific kernel versions, so upgrading the operating system on storage servers is a coordinated project rather than routine maintenance. Tuning is deep and consequential, and diagnosing performance problems requires understanding the interaction between clients, network, and object servers.

Sites running Lustre well generally have someone whose job includes it. Sites without that tend to run it badly, and a badly tuned Lustre performs worse than a well-run alternative — which is the practical argument for BeeGFS on a mid-sized cluster rather than any claim that Lustre is inferior.

What to know as a user

Striping is per-file and set at creation. A file’s stripe count and size determine how many object servers it spreads across, and it cannot be changed afterwards. Large files that will be read in parallel want wide striping; small files want a single stripe, because striping a small file across many servers adds coordination overhead for no bandwidth gain. lfs setstripe sets this on a directory so files inherit it.

Getting this wrong is a common and invisible cause of poor performance — the file works, it is just slower than it should be, and nothing indicates why.

ls -l is expensive. It requires contacting object servers for each file’s size, so listing a large directory with attributes generates substantial load. On a busy filesystem this is antisocial as well as slow, and plain ls avoids it.

Small files remain the pathological case, as with any parallel filesystem. The design optimises for large parallel I/O, and a workload of many small files uses none of that and stresses metadata instead.

See also: BeeGFS, Ceph, GlusterFS, HPC, and filesystem performance in smaller clusters.