Sizing
Management
- No special requirements
Metadata
- Needs RAM - the more the better, used for caching
- RAID protection on 1 or 10
- Disks best SSD or NVMe
- Chunk size best at 64KB as small files need to be written
- ext4 is fastet for writing small files
- 0.5% capacity of total storage per metadata server
Storage
-
Needs CPU when software RAID
-
the more the better
-
RAID-6 most common
-
RAID chunk size 512 KB
-
Avoid linux md-raid
-
raidz2 with up to 6 disks
-
Hyper-converged solutions are seldomly used
Built/in Benchmark
- storagebench → cleanup
- beegfs-ctl —storagebench —cleanup —alltargets
- netbench
- use to check network performance
- especially for RDMA
- IOZone (not part of teh repos)
- use for underlying file system
- fio
- use for underlying fs
- IOR ğet from GitHub)
- use for overall system (beegfs perf)
- usually mpirun —hosts (on a number of clients)
- mdtest
- use for metadata throughput
- mpirun
- elbencho
- metadata or storage
- many small files for metadata
- fewer larger for storage
- Easy-to-read output
- qperf
- RDMA and TCP benchmark
Tuning XFS
Tune Block devices
- nr_requests 2048 (more is not allowed by kernel - 1023 for NVMe)
Tuning Daemons
- connUseRDM (true)
- tune per user (false)
- meta
- tuneNumWorkers ruleOfThumb 4-8xCPU cores
- Client
Data availability and data loss
Buddy mirroring
- Performance loss: 25%-50%
See also: BeeGFS course overview, BeeGFS course day 1, MOC Computing DevOps and Software