13 494
правки
Изменения
Нет описания правки
* RBD cache helps a lot on HDDs, but on all-flash clusters it slows everything down. Something is implemented with locks, something is single-threaded, somebody tries to optimize it all, but the work isn’t done yet.
* There are the following drive emulation options: lsi (slowest), virtio-scsi (fast), virtio (fastest, but can’t do TRIM until QEMU 4.0). virtio-scsi can use multiple queues and thus should be the fastest with fast underlying storage (with a local NVMe?) — but it seems it doesn’t matter with Ceph.
* The filesystem also slows thing things down! Specifically it updates inode mtime on each small write if you don’t have lazytime enabled. mtime is part of the metadata, so this change is journaled, which makes the <tt>fio -sync=1 -iodepth=1 -direct=1</tt> test result 3-4 times worse when you run it over a file in FS.
* If you’re so unlucky that you run Oracle in your Ceph VMs then it’s crucial to set FILESYSTEMIO_OPTIONS=SETALL. I/O will be terribly slow if you don’t set it.
