13 521
правка
Изменения
Нет описания правки
Ceph is a Software-Defined Storage system, and its «software» is a significant overhead. The general rule currently is: with Ceph it’s hard to achieve random read latencies less than 0.5ms and random write latencies less than 1ms, '''no matter what drives or network you use'''. This stands for only 2000 iops random read and 1000 iops random write with one thread, and even if you manage to achieve this result you’re already in a good shape. With best-in-slot hardware and some tuning you may be able to improve it further, but only twice or so.
But does latency matter? Yes, it does, when it comes to single-threaded (synchronous) random reads or writes. Basically, everything that wants the data to be durable does fsync()'s which serializes writes. For example, all DBMS’s DBMS do. So to understand the performance limit of these apps you should benchmark your cluster with iodepth=1.
The latency doesn’t scale with the number of servers or OSDs-per-SSD or with two-RBD-in-RAID0. When you’re benchmarking your cluster with iodepth=1 you’re benchmarking only ONE placement group at a time (PG is a triplet or a pair of OSDs). The result is only affected by how fast 1 OSD is responding to 1 request. In fact, with iodepth=1 IOPS=1/latency. There is Nick Fisk’s presentation titled «Low-latency Ceph». By «low-latency» he means 0.7ms, which is only ~1500 iops.
