Ceph
Ceph provides object, block, and file storage from one cluster. That breadth is unusual: the same storage serves S3-compatible object access, block devices for virtual machines, and a POSIX filesystem through CephFS.
CRUSH, and why it matters
The distinguishing piece is CRUSH, the algorithm determining where data lives. Rather than a lookup table mapping objects to servers, clients compute the location from the object name and a map of the cluster.
This removes the central metadata bottleneck that most distributed storage has. Clients contact storage nodes directly with no lookup, and adding or removing nodes changes the map, causing only the affected data to move rather than a full rebalance.
CRUSH is also topology-aware. The map encodes racks, rooms, and power domains, and placement rules can require replicas in separate failure domains — so losing a rack does not lose data. That is a genuinely strong property and it is configured rather than automatic.
Its natural home, and where it is wrong
Ceph’s natural home is virtualisation and cloud infrastructure, where block storage for VMs and object storage for applications come from the same pool. Under Proxmox it is the standard way to get shared storage enabling live migration.
For HPC scratch it is generally the wrong choice. Lustre and BeeGFS are built for large sequential parallel I/O and beat CephFS at it. Ceph’s strength is durability and flexibility, not peak throughput for a single large file.
The operational reality
Ceph is a distributed system with a distributed system’s failure modes. It self-heals, rebalances, and reports health continuously, and understanding what it is doing requires real familiarity. A cluster in a degraded state recovering from a failed disk generates substantial background I/O, and a cluster that has lost enough replicas to block writes needs someone who knows why.
The requirements are unforgiving. It wants a fast dedicated network, plenty of memory per node, and — most importantly — enough nodes. A three-node cluster is the practical minimum and has little margin; small clusters spend a lot of time degraded.
Erasure coding saves capacity over replication at the cost of much more expensive recovery, and choosing between them is a real decision rather than an optimisation.