13 521
правка
Изменения
м
→Why is it so slow
Ceph is a Software-Defined Storage system, and its «software» is a significant overhead. The general rule currently is: with Ceph it’s hard to achieve random read latencies less than 0.5ms and random write latencies less than 1ms, '''no matter what drives or network you use'''. This stands for only 2000 iops random read and 1000 iops random write with one thread, and even if you manage to achieve this result you’re already in a good shape. With best-in-slot hardware and some tuning you may be able to improve it further, but only twice or so.
But does latency matter? Yes, it does, when it comes to single-threaded (synchronous) random reads or writes. Basically, everything all software that wants the data to be durable does fsync()'s calls which serializes serialize writes. For example, all DBMS DBMSs do. So to understand the performance limit of these apps you should benchmark your cluster with iodepth=1.
The latency doesn’t scale with the number of servers or OSDs-per-SSD or with two-RBD-in-RAID0. When you’re benchmarking your cluster with iodepth=1 you’re benchmarking only ONE placement group at a time (PG is a triplet or a pair of OSDs). The result is only affected by how fast 1 a signle OSD is responding to 1 processes a single request. In fact, with iodepth=1 IOPS=1/latency. There is Nick Fisk’s presentation titled «Low-latency Ceph». By «low-latency» he means 0.7ms, which is only ~1500 iops.
=== Micron setup example ===