13 521
правка
Изменения
Нет описания правки
[[File:Ceph-funnel-en.svg|500px|right]] [[ru:Производительность Ceph]]
Ceph is a Software-Defined Storage system. It’s very feature-rich: it provides object storage, VM disk storage, shared cluster filesystem and a lot of additional features. In some ways, it’s even unique.
It could be an excellent solution which you could take for free, immediately solve all your problems, become a cloud provider and earn piles of money. However there is a subtle problem: PERFORMANCE. Rational people rarely want to lower the performance by 95 % in production. It seems cloud providers like AWS, GCP, Yandex don’t care — all of them run their clouds on top of their own crafted SDS-es (not even Ceph) and all these SDS-es are just as slow. :-) we don’t judge them of course, that’s their own business.
This article describes which performance numbers you can achieve with Ceph and how. But I warn you: you won’t catch up with local SSDs. Local SSDs (especially NVMe) are REALLY fast right now, their latency is about 0.05ms. It’s very hard for an SDS to achieve the same result, and beating it is almost impossible. The network alone eats those 0.05ms...
'''UPDATE: It’s possible to achieve good latency with an SDS. I did it in my own project — Vitastor: https://vitastor.io :-) it's a block SDS architecturally similar to Ceph, but FAST. It achieved 0.14 ms latency (both read and write) in a cluster with SATA SSDs. Ceph only achieved 1 ms for writes and 0.57 ms for reads on the same hardware. See [https://yourcmc.ru/git/vitalif/vitastor/src/branch/master/README.md README] for details.'''
== General benchmarking principles ==
Main test cases for benchmarking are:
* Linear read and write (big blocks, big long queue) in MB/s
* Highly parallel random read and write of small blocks (4-8kb, iodepth=32-128) in IOPS (Input/Output ops per second)
* Single-threaded transactional random write (4-8kb, iodepth=1) and read (though single-threaded reads are more rare) in IOPS
=== Test your disks ===
[https://docs.google.com/spreadsheets/d/1E9-eXjzsKboiCCX-0u0r5fAjjufLKayaut_FOPxYZjc SSD Bench Google Docs]
Run `fio` on your drives before deploying Ceph:
{{Box|[[File:Warning icon.svg|32px|link=]] {{red|WARNING!}} For those under a rock — fio write test is DESTRUCTIVE. Don’t dare to run it on disks which have containing important data… for example, OSD journals (I’ve seen such cases).}}
* Try to disable drive cache before testing: {{Cmd|hdparm -W 0 /dev/sdX}} (SATA drives), {{Cmd|1=sdparm --set WCE=0 /dev/sdX}} (SAS drives). This is usually ABSOLUTELY required for server SSDs like Micron 5100 or Seagate Nytro (see [[#Drive cache is slowing you down]]) as it increases random write iops ''more than by two magnitudes'' (from 288 iops to 18000 iops!). In some cases it may not improve anything, so try both options -W0 and -W1.
[[File:Warning icon.svg|32px|link=]] A useful habit is to leave an empty partition for later benchmarking on each SSD you deploy Ceph OSDs on, because some SSDs tend to slow down when filled.
==== Lyrical digression ====
Why use this approach in benchmarking? After all, disk performance depends on many parameters, such as:
* Block size;
* Mode — read, write, or various mixed read/write modes;
* Parallelism — queue depth and the number of threads, in other words, the number of parallel I/O requests;
* Test duration;
* Initial disk state — empty, filled linearly, filled randomly, randomly written over a specific period of time;
* Data distribution — for example, 10% of hot data and 90% of cold data or hot data located in a certain place (e.g., at the beginning of the disk);
* Other mixed test modes, e.g, benchmarking using different block sizes at the same time.
The results can also be presented with varying levels of detail — you can provide graphs, histograms, percentiles, and so on in addition to mere average operation count or megabytes per second. This, of course, can reveal more information about the behavior of the disk under test.
Benchmarking also contains a bit of philosophy. For example, some manufacturers of server SSDs argue that you must do preconditioning by randomly overwriting the disk at least twice to fill translation tables before testing. I rather believe that it puts the SSD in unrealistically bad conditions rarely seen in real life.
Others say you should plot a graph of latency against the number of operations per second, but my opinion is that it’s also a bit strange because it implies that you plot a graph of F1(q) against F2(q) instead of «q» itself.
In short, benchmarking can be a never-ending process. It can take quite a few days to get a complete view. This is usually what resources like 3dnews do in their SSD reviews. But we don’t want to waste several days. We need a test that allows us to estimate performance quickly.
Therefore we isolate a few «extreme» modes, check the disk in them and pretend that other results are somewhere between these «extreme points», forming some kind of a smooth function depending on the parameters. It’s also handy that each of these modes also corresponds to a valid use case:
* Applications that mainly use linear or large-block access. For such applications, the crucial characteristic is the linear I/O speed in megabytes per second. Therefore, the first test mode is linear read/write with 4 MB blocks and medium queue depth — 16-32 operations. Test results should be in MB/s.
* Applications that use random small-block access and support parallelism. This leads us to 4 KB random I/O modes with large queue depth — at least 128 operations. 4 KB is the standard block size for most filesystems and DBMS. Multiple (2-4-8) CPU threads should be used if a single thread can’t saturate the drive during test. Test results should include iops (I/O operations per second), but not latency. Latency is meaningless in this test because it can be arbitrarily increased just by increasing queue depth — latency is directly related to iops with a formula latency=queue/iops.
* Applications that use random small-block access and DO NOT support parallelism. There are more such applications than you might think; regarding writes, all transactional DBMSs are a notable example. This leads us to 4 KB random I/O test with queue depth of 1 and, for writes, with an fsync after each operation to prevent the disk or storage system from «cheating» by writing the data into a volatile cache. Results should include either iops or latency, but not both because, as already said, they directly relate to each other.
=== Test your Ceph cluster ===
Recommended benchmarking tools:
* The first recommended tool is again `fio` with `-ioengine=rbd . Run the following:*# fio -ioengine=rbd -direct=1 -name=test -bs=4M -iodepth=16 -rw=write -pool=<your rpool_hdd -runtime=60 -rbdname=testimg*# fio -ioengine=rbd -direct=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -pool> =rpool_hdd -runtime=60 -rbdname=<your image>`testimg*# fio -ioengine=rbd -direct=1 -name=test -bs=4k -iodepth=128 -rw=randwrite -pool=rpool_hdd -runtime=60 -rbdname=testimg*: Then repeat for rw=read/randread. All of *: The idea is to test a) the above best possible latency b) linear bandwidth c) random access iops.*: Reading from an empty RBD image is very fast :) so pre-fill it before testing.*: Run tests valid for raw drives can be repeated for from node(s) where your actual RBD and they mean the users will reside. The results are usually slightly better when you run tests from a separate physical server.* The same thingsfrom inside a VM or through the kernel RBD driver (krbd):*# fio -ioengine=libaio -direct=1 -name=test -bs=4M -iodepth=16 -rw=write -runtime=60 -filename=/dev/rbdX*# fio -ioengine=libaio -direct=1 -sync=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -filename=/dev/rbdX*# fio -ioengine=libaio -direct=1 -name=test -bs=4k -iodepth=128 -rw=randwrite -runtime=60 -filename=/dev/rbdX*: Don't miss the added -sync=1 option. SyncIt is added on purpose, direct and invalidate flags can be omitted, because RBD to match the ioengine=rbd test. ioengine=rbd has no concept of «sync» — all operations are sync, everything is always «sync»with it. And there’s no page cache involved either, so «direct» also doesn’t mean anythingOverall this write pattern — transactional single-threaded write — corresponds to a DBMS.* The second recommended tool, especially useful for hunting performance problems, comes in several improved varieties : Note that regardless of «Mark’s bench» from russian Ceph chatthe supposed overhead of moving data in and out the kernel, the kernel client is actually faster.* ceph-gobench*: Or https://github.com/rumanzovitalif/ceph-gobench or bench. The original idea comes from the «Mark’s bench» from russian Ceph chat ([https://github.com/vitalifsocketpair/ceph-benchoriginal outdated tool was here]). Both use a non-replicated Ceph pool (size=1), create several 4MB objects (16 by default) in each separate OSD and do random single-thread 4kb writes in randomly selected objects within one OSD. This mimics random writes to RBD and allows to determine the problematic OSDs by benchmarking them separately. Original Mark’s bench (outdated) was here: https://github.com/socketpair/ceph-bench
*: To create the non-replicated benchmark pool use {{Cmd|ceph osd pool create bench 128 replicated; ceph osd pool set bench size 1; ceph osd pool set bench min_size 1}}. Just note that 128 (PG count) should be enough for all OSDs to get at least one PG each.
* Do not S3 (rgw):** [https://github.com/intel-cloud/cosbench cosbench]** [https://github.com/markhpc/hsbench hsbench]** [https://github.com/minio/warp minio warp] Notes:* Never use dd to test disk performance.* Don't use `rados bench`. It creates a small number of objects (1-2 for a thread) so all of them always reside in cache and improve the results far beyond they should be.* You can also use the simple `rbd bench`, but fio -ioengineis better. =libaio` with a kernel== Test your network === ping -mounted RBDf (flood ping). However, that requires to disable some features of that RBD, because kernel client still lacks their support sockperf. Note that regardless of On the overhead of moving data in and out the kernelfirst node, run <tt>sockperf sr -i IP --tcp</tt>. On the kernel client second, run <tt>sockperf pp -i SERVER_IP --tcp -m 4096</tt>. Decent average number is actually fasteraround 0.05-0.07ms.* And you can also use it from inside your VMs, the results are usually similar to the above<s>qperf. Just note that On the result also depends on the storage driver being usedfirst node, just run <tt>qperf</tt>. Virtio is On the fastestsecond, virtio<tt>qperf -scsi vvs SERVER_IP tcp_lat -m 4096</tt>.</s> Don’t use qperf. It is slightly slower and everything else super-stupid: it doesn’t disable Nagle (like LSI emulationno TCP_NODELAY) and it doesn’t honor the <tt>-m 4096</tt> parameter — message size is terribly slowalways set to 1 BYTE in latency tests. Results are also considerably affected [[File:Warning icon.svg|32px|link=]] Warning: Ubuntu has AppArmor enabled by whether the RBD cache default and it affects network latency adversely. Disable it if you want good performance. The effect of AppArmor is enabled or not like the following (cache=writeback qemu option turns on RBD cache automaticallyIntel X520-DA2): * centos 3. For random reads or writes, disabling RBD cache is faster10: rtt min/avg/max/mdev = 0.039/0.053/0.132/0.012 ms* ubuntu 4.x + apparmor: rtt min/avg/max/mdev = 0.068/0.163/0.230/0.029 ms* ubuntu 4.x: rtt min/avg/max/mdev = 0.037/0.071/0.157/0.018 ms
== Why is it so slow ==
'''However''', the naive expectation is that as you replace your HDDs with SSDs and use a fast network — Ceph should become almost as faster. Everyone is used to the idea that I/O is slow and software is fast. And this is generally NOT true with Ceph.
Ceph is a Software-Defined Storage system, and its «software» is a significant overhead. The general rule currently is: with Ceph it’s hard to achieve random read latencies less than below 0.5ms and random write latencies less than below 1ms, '''no matter what drives or network you use'''. With one thread, this stands for only 2000 random read iops and 1000 random write iops, and even if you manage to achieve this result you’re already in a good shape. With best-in-slot hardware and some tuning you may be able to improve it further, but only twice or so.
But does latency matter? Yes, it does, when it comes to single-threaded (synchronous) random reads or writes. Basically, all software that wants the data to be durable does fsync() calls which serialize writes. For example, all DBMSs do. So to understand the performance limit of these apps you should benchmark your cluster with iodepth=1.
The latency doesn’t scale with the number of servers or OSDs-per-SSD or two-RBD-in-RAID0. When you’re benchmarking your cluster with iodepth=1 you’re benchmarking only ONE placement group at a time (PG is a triplet or a pair of OSDs). The result is only affected by how fast a single OSD processes a single request. In fact, with iodepth=1 IOPS=1/latency. There is Nick Fisk’s presentation titled «Low-latency Ceph». By «low-latency» he means 0.7ms, which is only ~1500 iops.
=== Expected performance ===
Estimating the cluster performance based on disks' performance is absolutely wrong.
The real expected performance for Bluestore is like the following (iops applies to random 4KB reads/writes):
1 HDD (usual SATA, 7200 rpm, non SMR, without SSD cache) is:
* ~100-120 iops with QD=128
* ~66 iops with QD=1
* ~40 MB/s with linear read/write
* The numbers will be worse if you're short on available RAM, because you'll get a lot of metadata cache misses
1 fast SSD or NVMe SSD with capacitors (see below) and write iops >= 25000:
* ~1000 write iops with QD=1. May vary between 300 and, in the best possible case, ~2500 iops depending on CPU frequency and settings.
* Up to ~10000-20000 write iops with QD=128 per 1 OSD.
* Read iops are around 2-2.5 times better: QD=1 ~2000 iops (up to ~4000), QD=128 ~20000 (up to ~50000 depending on the CPU).
* Of course, the QD=128 iops number is limited by the performance of the disk itself :). However, as good SSDs usually perform great in parallel mode, they're usually not a bottleneck.
* By running multiple OSDs on a single drive, you can multiply your parallel (QD=128) iops number by the number of OSDs, as long as the drive allows it. Of course, you get the same increase in CPU load. HUGE increase.
* Linear reads and writes are almost as fast as raw disk reads and writes.
* Difference between SATA SSDs and NVMes in terms of random I/O in Ceph is negligible as long as they both have capacitors. Of course, server NVMes are still the best and you should try to get them instead of SATA and SAS, but it's hard to notice the difference with Ceph and random I/O.
* Modern SSDs often have slower QD=1 random reads than writes, just because they write into a fast capacitor-protected cache, but they can't serve all random reads from it. The difference is usually like 8000 QD=1 read iops compared to 40000 QD=1 write iops.
Aggregate performance:
* Linear read from the cluster = OSD number * MB/s of one OSD
* Linear write to a replicated pool = OSD number / Replica number * MB/s of one OSD
* Linear write to a EC pool = OSD number / (K+M) * K * MB/s of one OSD
* Random QD=1 performance is the average for all OSDs (treat it like latency); iops with QD=128 is the sum
* Random IOPS are limited by the client, too. 1 RBD client can squeeze out up to ~30000 read iops and up to ~15000 write iops
* Linear I/O is of course limited by the network bandwidth, too
=== Micron setup example ===
Here’s an example setup from Micron. They used 2x replication, very costly CPUs (2x 28-core Xeon Gold Platinum per server), very fast network (100G2x100G, in fact 2x2x100G — 2 cards with 2 ports each) and 10x their best NVMes in each of 4 nodes: https://www.micron.com/resource-details/30c00464-e089-479c-8469-5ecb02cfe06f
They only got 350000 peak write iops with high parallelism with 100 % CPU load. It may seem a lot, but if you divide it by the number of NVMes — 350000/40 NVMe — it’s only 8750 iops per a an NVMe. If we account for 2 replicas and WAL we get 8750*2*2 = 35000 iops per drive. So… Ceph only squeezed 35000 iops out of a an NVMe '''that can deliver 260000 iops alone'''. That’s what Ceph’s overhead is.
Also there are no single-thread latency tests in that PDF. Such tests could be very interesting.
=== Update ===
https://www.micron.com/-/media/client/global/documents/products/other-documents/micron_9300_and_red_hat_ceph_reference_architecture.pdf
New NVMes are Micron 9300 of maximum possible capacity — 12.8 TB. Each of these delivers even more write iops than 9200’s: 310k instead of 260k. Everything else remains the same.
The new write performance result for 100 RBD clients is 477029 iops (36 % more than in the previous test). Remember that it’s still only 4770 iops per client, though. For 10 RBD clients the result is better: 294000 iops, which stands for 29400 iops per client.
What helped the performance? I guess the configuration did. In comparison to the previous test they changed the following:
* disabled messenger checksums (ms_crc_data=false) and bluestore checksums (bluestore_csum_type=none)
* tuned rocksdb: <tt>bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,<br />write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,<br />level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB</tt> — this divides into:
** The main part is probably 64x32x4 MB memtables setting (number x merge x size) instead of default 4x1x256 MB. The effect of this change isn’t really clear for me. It may slightly reduce CPU load because sorting a big memtable is slower than sorting a small one. However, 32x4 compactions aren’t probably that much faster than 1x256.
** max_bytes_for_level_base is changed dramatically — it’s raised to 6 GB from 256 MB!
** added compaction threads
* allocated 14 GB RAM per each OSD
* osd_max_pg_log_entries=osd_min_pg_log_entries=osd_pg_log_dups_tracked=osd_pg_log_trim_min = 10. However I’m not sure about this — it did nothing in my tests.
Other remarks:
* cephx was already disabled in the previous version of the test. This time they also disabled signatures. It seems pointless though — disabled cephx doesn’t sign anything.
* they already had debug objecter = 0/0 and the rest of debug levels set to zero.
* it seems they haven’t tried changing prefer_deferred_size and min_alloc_size.
* new NVMes definitely didn’t change anything. 260000 iops is over the top for Ceph anyway.
* in the new PDF there is a 70/30 R/W test with QD=1. It was done for 100 RBD clients, but for their cluster it was a «low-load» condition (19.38 % CPU load on the hosts). They report 0.37ms/0.72ms random read/write latencies. In fact they report it reversed :) but let’s assume that 0.37ms is actually for reads because reads are always faster in Ceph. This again corresponds only to 2700/1388 single-thread read/write iops.
== CAPACITORS! ==
And this increases '''transactional write IOPS, making it equal to non-transactional'''.
Supercaps are usually called «enhanced/advanced power loss protection» in the datasheets. This is a feature almost exclusively present only in «server-grade» SSDs (not even all of them). For example, Intel DC S4600 has supercaps and Intel DC S3100 doesn’t.
{{Note}} This is the main difference between server and desktop SSDs. An average user doesn’t need transactions, but servers run DBMS’es, and DBMS’es want them really, really bad.
== Bluestore vs Filestore ==
TODO: This section lacks random read performance comparisons. Bluestore is the «new» storage layer of Ceph. All presentations and documents say it’s better in all ways, which in fact indeed seems reasonable for something «new».
Bluestore is really 2x faster than Filestore for linear write workloads, because it has no double-writes — big blocks are written only once, not twice as in Filestore. Filestore journals everything, so all writes first go to the journal and then get copied to the main device.
Bluestore is also more feature-rich: it has checksums, compression, erasure-coded overwrites and virtual clones. Checksums allow 2x-replicated pools self-heal better, erasure-coded overwrites make EC usable for RBD and CephFS, and virtual clones make VMs run faster after taking a snapshot.
In HDD-only (or bad-SSD-only) setups Bluestore uses a lot more RAM, though, because it uses RocksDB for all metadata, additionally caches some of them by itself and is also tries to cache some data blocks to compensate 2x faster than Filestore for the lack of page cache usagerandom writes. The general rule of thumb This is 1GB again because it can do 1 commit per 1TB of storagewrite, but not less than 2GB per an OSDat least if you apply this patch: https://github.com/ceph/ceph/pull/26909 and turn bluefs_preextend_wal_files on. In fact it’s OK to say that Bluestore’s deferred write implementation is really optimal for transactional writes to slow drives.
=== HDD for data + SSD for journal ===
Filestore writes everything to the journal and only starts to flush it to the data device when the journal fills up to the configured percent. This is very convenient because it makes journal act as a «temporary buffer» that absorbs random write burtsbursts.
Bluestore can’t do the same even when you put its WAL+DB on SSD. It also has sort of a «journal» which is called «deferred write queue», but it’s very small (only 64 requests) and it lacks any kind of background flush threads. So you actually can increase the maximum number of deferred requests, but after the queue fills up the performance will drop until OSD restartrestarts.
So, Bluestore’s performance is very consistent, but it’s worse than peak performance of Filestore for the same hardware. In other words, Bluestore OSD refuses to do random writes faster than the HDD can do them on average.
With Filestore you easily get 1000—2000 iops while the journal is not full. With Bluestore you only get 100—300 iops regardless of the SSD journal, but these are absolutely stable over time and never drop.
But it’s still a shame that the increase is only 5-10 % for that amount of architectural effort.
=== HDD-only (or bad-SSD-only) RAM usage === Another thing to note is that Bluestore uses a lot more RAM. This is because it uses RocksDB for all metadata, additionally caches some of them by itself and also tries to cache some data blocks to compensate for the lack of page cache usage. The general rule of thumb is 1GB RAM per 1TB of storage, but not less than 2GB per an OSD.
Official documents say that you should allocate 4 % of the slow device space for block.db (Bluestore’s metadata partition). This is a lot, Bluestore rarely needs that amount of space.
But the main problem is that Bluestore uses RocksDB and RocksDB puts a file on the fast device only if it thinks that the whole layer will fit there (RocksDB is organized in files). So, default Default RocksDB settings in Ceph are:
* 1 GB WAL = 4x256 Mb
* max_bytes_for_level_base and max_bytes_for_level_multiplier are default, thus 256 Мб Mb and 10, respectively
* so L1 = 256 Mb
* L2 = 2560 Mb
…so…
RocksDB puts L2 files to block.db only if it’s at least 2560+256+1024 Mb (almost 4 GB). And it will put puts L3 there only if it’s at least 25600+2560+256+1024 Mb (almost 30 GB). And L4 — only if it’s at least 256000+25600+2560+256+1024 Mb (roughly 286 GB).
In other words, all block.db sizes except 4, 30, 286 GB are pointless, Bluestore won’t use everything anything above the previous «round» size. At least if you don’t change RocksDB settings. And of these 4 is too small, 286 is too big.
So just stick with 30 GB for all Bluestore OSDs :)
* If you still can’t throw it away — disable all caches and pray :) the problem is that RAID controllers sometimes ignore fsync requests so Ceph can become corrupted on a sudden power loss. Even some HBAs may do that (namely some Adaptecs).
* In theory, you may try to leverage the battery- (or supercap-) backed controller cache in RAID0 mode to improve write latency. However, that can easily become shooting yourself. At least conduct some power-unplug tests if you do that.
* A good post in the mailing list about RAID0: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/036237.html (TLDR: never use it!)
* IOPS difference between RAID and HBA/SATA may be very noticeable. A bad or old RAID controller can easily become a bottleneck.
* HBAs also have IOPS limits. For example, it’s ~280000 iops for all disks for the LSI 9211-8i.
* CPU is the main bottleneck for Ceph running on good SSDs.
* As Nick Fisk said in his presentation — Ceph is a Software-Defined Storage '''and every piece of Ceph «Software»''' will run faster with every GHz of CPU clock speed.
* Server CPUs often have NUMA (Non-Uniform Memory Access). Which means that some CPU cores don’t have direct access to the RAM and/or to the part of the hardware, but . They must forward requests to other coresin order to access this hardware.
* You should try to use CPUs with more clock speed and without NUMA to maximize the performance. That is, a slower CPU with more cores is probably worse than a faster CPU with a smaller number of cores…
* …but within reason as one Bluestore OSD can eat up to ~6 cores under full load.
* «Clock speed» means nominal, not Turbo Boost speed, because Turbo Boost is only beneficial for single-threaded workloads.
* CPU pinning recommendations (taskettaskset) are almost outdated, because Ceph OSDs are multi-threaded. At least 4 threads are active during writeswrite, so you’ll only slow your OSDs down if you allocate less than 4-6 cores for each of them.
* There are 2 parameters responsible for OSD worker thread count — osd_op_num_shards and osd_op_num_threads_per_shard…
* …But trying to tune them is pointless, default configuration (1x5 for HDDs and 2x8 for SSDs) is optimal. The problem is that all worker threads still serialize writes into a single kv_sync_thread, and the whole scheme only scales up to ~6 worker threads.
* There is one thing that decreases latency 2-3 times at once. It’s disabling all power-save functions of CPUs:** <tt>cpupower idle-set -D 10</tt> — this disables C-States (or you can pass <tt>processor.max_cstate=1 intel_idle.max_cstate=0</tt> to the kernel command-line)** <tt>cpupower frequency-set -g performance</tt> or (for older versions) <tt>for i in $(seq 0 $((`nproc`-1))); do cpufreq-set -c $i -g performance; done</tt> — this disables frequency scaling.
* When power-save is disabled CPU heats up as a GTX, but you get 2-3 times more iops.
* High CPU requirement is one of the cases NOT arguments to NOT use Ceph in a «hyperconverged setup», the setup in which storage and compute nodes are combined.
* You can also disable all hardware vulnerability mitigations: <tt>noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier</tt> (or just <tt>mitigations=off</tt> for newer kernels)
== VM setup and filesystem options ==
* Default qemu options for RBD are bad.
* Bad means that a) it uses the slow emulated LSI controller b) it uses the mode with cached reads, but without cached writes.
* Drive cache in qemu is controlled by the `cache` option (surprise). It can be <missing>, writethrough, writeback, none, unsafe, directsync. With RBD this option also affects rbd cache, which is the cache on the Ceph’s client library (librbd) side.
* But cache=unsafe doesn’t work with RBD, it still waits for write confirmations. And writethrough, <missing> and directsync are basically equivalent.
* RBD cache helps a lot on HDDs, but it slows everything down in all-flash clusters. Something is implemented with locks, something is single-threaded, somebody tries to optimize it all, but the work isn’t done yet.
* There are the following drive emulation options: lsi (slowest), virtio-scsi (fast), virtio (fastest, but can’t do TRIM until QEMU 4.0). virtio-scsi can use multiple queues and thus should be the fastest with fast underlying storage (with a local NVMe?) — but it seems it doesn’t matter with Ceph.
* The filesystem also slows things down! Specifically it updates inode mtime on each small write if you don’t have lazytime enabled. mtime is part of the metadata, so this change is journaled, which makes the <tt>fio -sync=1 -iodepth=1 -direct=1</tt> test result 3-4 times worse when you run it over a file in FS.
* If you’re so unlucky that you run Oracle in your Ceph VMs then it’s crucial to set FILESYSTEMIO_OPTIONS=SETALL. I/O will be terribly slow if you don’t set it.
So…
* For HDD / SSD+HDD clusters it is recommended to use qemu cache=writeback. This mode is safe, because guest fsyncs make qemu flush RBD cache. That is, guests don’t lose the journaled data.
* For SSD-only clusters it’s best to disable the cache at all (cache=none). It usually increases maximum parallel iops 2 times or so.
* The best emulation driver is virtio. Now go find the way to set it in your VM GUI (Proxmox, Opennebula) :). Opennebula, for example, has quite a perverted way of changing the emulation driver.
* Try to use lazytime everywhere. It requires a decently recent kernel: at least 4.0 for ext4 and at least 4.17 for XFS. For XFS, it also seems that a recent version of util-linux is required.
== Network, DPDK and SPDK ==
** SPDK is enabled for NVMes by passing <tt>spdk:PCI_serial_number</tt> as the device name and deploying OSDs using the Manual Deployment documentation
** But…
* DPDK support is broken — build scripts are broken and even if you fix them by hand like me there are some bugs that make OSD OSDs crash after processing ~50 packets.
* SPDK build scripts are OK and Ceph is even built with it by default. There are even some reports that it works, however, my OSDs have just hung when I tried to start them with SPDK.
* Both are pointless to use because Ceph itself isn’t that fast. It doesn’t matter if your network latency is 0.05ms or 0.005ms — Ceph software takes 0.5-1ms. There was an experiment report in the mailing list — one guy tried to isolate AsyncMessenger from all other Ceph code and benchmark it alone — https://www.spinics.net/lists/ceph-devel/msg43555.html — and he only got ~80000 iops.
In addition to that, SATA/SAS drives also have a cache disable command. When you disable the cache Linux stops sending flushes at all. It may seem that this should also result in the same performance as fsync/O_SYNC, but that’s not the case either! SSDs with supercaps give '''much''' better performance with disabled cache. For example, Seagate Nytro 1351 gives you 288 iops with cache and 18000 iops without it (!).
Why? It seems that’s because FLUSH CACHE is interpreted by the drive as a «please flush all caches, including non-volatile cache» command, and «disable cache» is interpreted as «please disable the volatile cache, but you may leave the non-volatile one on if you want to». This makes writes with a flush after every write slower than writes with the cache disabled.
What about NVMe? NVMe has slightly less variability — there is no «disable cache» command in the NVMe spec at all, but just as in the SATA spec there is the FLUSH CACHE command and FUA bit. But again, based on the personal experience I can say that it seems that FUA is often ignored with NVMe either by Linux or by the drive itself, thus '''fio -sync=1''' gives the same results as '''fio -direct=1''' without any sync flags. '''-fsync=1''' performs correctly and lands the performance down to where it must belong (1000—2000 iops for desktop NVMes).
== Quick insight into SSD and flash memory organization ==
But that’s not the case with modern SSDs — even cheap models are very fast and usually very durable. But why? The credit goes to SSD controllers: SSDs contain very smart and powerful controllers, usually with at least 4 cores and 1-2 GHz clock frequency, which means they’re as powerful as mobile phones' processors. All that power is required to make FTL firmware run smoothly. FTL stands for «Flash Translation Layer» and it is the firmware responsible for translating addresses of small blocks into physical addresses on flash memory chips. Every write request is always put into a space freed in advance, and FTL just remembers the new physical location of the data. This makes writes very fast. FTL also defragments free space and moves blocks around to achieve uniform wear across all memory cells. This feature is called Wear Leveling. SSDs also usually have some extra physical space reserved to add even more endurance and to make wear leveling easier; this is called overprovisioning. Pricier server SSDs have a lot of space overprovisioned, for example, Micron 5100 Max has 37,5 % of physical memory reserved (extra 60 % is added to the user-visible capacity).
When I tried to lecture someone in the mailing list about «all SSDs doing fsyncs correctly» I got this as the reply: https://www.usenix.org/system/files/conference/fast13/fast13-final80.pdf. Long story short, it says that in 2013 a common scenario was SSDs not syncing metadata on fsync calls at all which led to all kinds of funny things on a power loss, up to (!!!) total failures of some SSDs.
There also exist some very old SSDs without capacitors (OCZ Vector/Vertex) which are capable of very large sync iops numbers. How do they work? Nobody knows, but I suspect that they just don’t do safe writes :). The core principle of flash memory overwrites hasn't hasn’t changed in recent years. 5 years ago SSDs were also FTL-based, just as now.
So it seems there are two kinds of «power loss protection»: simple PLP means «we do fsyncs and don’t die or lose your data when a power loss occurs», and advanced PLP means that fsync’ed writes are just as fast as non-fsynced. It also seems that in the current years (2018—2019) simple PLP is already a standard and most SSDs don’t lose data on power failure.
This limit is always sufficient to copy big files to a flash drive formatted in any of common filesystems. One opened block receives metadata and another receives data, then it just moves on. But if you start doing random writes you stop hitting the opened blocks and this is where lags come in.
== Bonus: Micron vSAN reference architecture ==
[https://media-www.micron.com/-/media/client/global/documents/products/other-documents/micron_vsan_6,-d-,7_on_x86_smc_reference_architecture.pdf Micron Accelerated All-Flash SATA vSAN 6.7 Solution]
Node configuration:
* 384 GB RAM 2667 MHz
* 2X Micron 5100 MAX 960 GB (randread: 93k iops, randwrite: 74k iops)
* 8X Micron 5200 ECO 3.84TB (randread: 95k iops, randwrite: 17k iops)
* 2x Xeon Gold 6142 (16c 2.6GHz)
* Mellanox ConnectX-4 Lx
* Connected to 2x Mellanox SN2410 25GbE switches
«Aligns with VMWare AF-6, aims up to 50K read iops per node»
* 2 replicas (like Ceph size=2)
* 4 nodes
* 4 VMs on each node
* 8 vmdk per VM
* 4 threads per vmdk
Total I/O parallelism: 512
100%/70%/50%/30%/0% write
* «Baseline» (fits in cache): 121k/178k/249k/314k/486k iops
* «Capacity» (doesn’t): 51k/66k/90k/134k/363k
* Latency is 1000*512/IOPS ms in all tests (1000ms * parallelism / iops)
* '''No latency tests with low parallelism'''
* '''No linear read/write tests'''
Conclusion:
* ~3800 write iops per drive
* ~11343 read iops per drive
* ~1600 write iops per drive when not in cache
* Parallel workload doesn’t look better than Ceph. vSAN is hyperconverged, though.
== Good SSD models ==
* Micron 5100/5200, 9300. Maybe 5300, 7300 too
* Seagate Nytro 1351/1551
* HGST SN260
* At least until Nautilus: <tt>[global] debug objecter = 0/0</tt> (there is a big client-side slowdown)
* Try to disable rbd cache in the userspace driver (QEMU options cache=none)
* <s>For HDD-only or Bad-SSD-Only and at least until it’s backported — backported (it is) — remove the handbrake https://github.com/ceph/ceph/pull/26909</s>