13 521
правка
Изменения
Нет описания правки
=== Test your disks ===
Run `fio` on your drivesbefore deploying Ceph:
{{Box|[[File:Warning icon.svg|32px|link=]] {{red|WARNING!}} For those under a rock — fio write test is DESTRUCTIVE. Don’t dare to run it on disks which have important data… for example, OSD journals (I’ve seen such cases).}}
[[File:Warning icon.svg|32px|link=]] A useful habit is to leave an empty partition for later benchmarking on each SSD you deploy Ceph OSDs on, because some SSDs tend to slow down when filled.
=== Test your Ceph cluster ===
Recommended benchmarking tools:
* The first recommended tool is again `fio` with `-ioengine=rbd -pool=<your pool> -rbdname=<your image>`. All of the above tests valid for raw drives can be repeated for RBD and they mean the same things, just sync, direct and invalidate flags don't matter, because RBD has no concept of "sync" - all operations are always "sync". And there's no page cache involved either, so "direct" also doesn't mean anything.
* The second recommended tool, especially useful for hunting performance problems, comes in the several improved varieties of "Mark's bench" from russian Ceph chat: https://github.com/rumanzo/ceph-gobench or https://github.com/vitalif/ceph-bench. Both use a non-replicated Ceph pool (size=1), create several 4MB objects (16 by default) in each separate OSD and do random single-thread 4kb writes. This allows to determine the problematic OSDs by benchmarking them separately.
*: To create the non-replicated benchmark pool use {{Cmd|ceph osd pool create bench 128 replicated; ceph osd pool set bench size 1; ceph osd pool set bench min_size 1}}. Just not that 128 (PG count) should be enough for all OSDs to get at least one PG each.
* Do not use `rados bench`. It creates a small number of objects (1-2 for a thread) so all of them always reside in cache and improve the results far beyond they should be.
* You can also use the simple `fio -ioengine=libaio` with a kernel-mounted RBD. However, that requires to disable some features of that RBD, because kernel client still lacks their support. Note that regardless of the overhead of moving data in and out the kernel, the kernel client is actually faster.
* And you can also use it from inside your VMs, the results are usually similar to the above. Just note that the result also depends on the storage driver being used. Virtio is the fastest, virtio-scsi is slightly slower and everything else (like LSI emulation) is terribly slow.
== Bluestore vs Filestore ==