Ceph-funnel-en.svg

Ceph is a Software-Defined Storage system. It’s very feature-rich: it provides object storage, VM disk storage, shared cluster filesystem and a lot of additional features. In some ways, it’s even unique.

It could be an excellent solution which you could take for free, immediately solve all your problems, become a cloud provider and earn piles of money. However there is a subtle problem: PERFORMANCE. Rational people rarely want to lower the performance by 95 % in production. It seems cloud providers like AWS, GCP, Yandex don’t care — all of them run their clouds on top of their own crafted SDS-es (not even Ceph) and all these SDS-es are just as slow. :-) we don’t judge them of course, that’s their own business.

This article describes which performance numbers you can achieve with Ceph and how. But I warn you: you won’t catch up with local SSDs. Local SSDs (especially NVMe) are REALLY fast right now, their latency is about 0.05ms. It’s very hard for an SDS to achieve the same result, and beating it is almost impossible. The network alone eats those 0.05ms...

Содержание

General benchmarking principles

Main test cases for benchmarking are:

Test your disks

SSD Bench Google Docs

Run `fio` on your drives before deploying Ceph:

Warning icon.svg WARNING! For those under a rock — fio write test is DESTRUCTIVE. Don’t dare to run it on disks containing important data… for example, OSD journals (I’ve seen such cases).

You wanna ask why it’s so slow? See below.

Warning icon.svg A useful habit is to leave an empty partition for later benchmarking on each SSD you deploy Ceph OSDs on, because some SSDs tend to slow down when filled.

Test your Ceph cluster

Recommended benchmarking tools:

Notes:

Test your network

ping -f (flood ping).

sockperf. On the first node, run sockperf sr -i IP --tcp. On the second, run sockperf pp -i SERVER_IP --tcp -m 4096. Decent average number is around 0.05-0.07ms.

qperf. On the first node, just run qperf. On the second, qperf -vvs SERVER_IP tcp_lat -m 4096.

Don’t use qperf. It is super-stupid: it doesn’t disable Nagle (no TCP_NODELAY) and it doesn’t honor the -m 4096 parameter — message size is always set to 1 BYTE in latency tests.

Warning icon.svg Warning: Ubuntu has AppArmor enabled by default and it affects network latency adversely. Disable it if you want good performance. The effect of AppArmor is like the following (Intel X520-DA2):

Why is it so slow

First of all:

  1. Ceph isn’t slow for linear reads and writes.
  2. Ceph isn’t slow on HDDs: theoretical single-thread random write performance of Bluestore is 66 % (2/3) of your drive’s IOPS (currently it’s 33 % in practice, but if you push this handbrake down: https://github.com/ceph/ceph/pull/26909 it goes back to 66 %), and multi-threaded read/write performance is almost 100 % of the raw drive speed.

However, the naive expectation is that as you replace your HDDs with SSDs and use a fast network — Ceph should become almost as faster. Everyone is used to the idea that I/O is slow and software is fast. And this is generally NOT true with Ceph.

Ceph is a Software-Defined Storage system, and its «software» is a significant overhead. The general rule currently is: with Ceph it’s hard to achieve random read latencies below 0.5ms and random write latencies below 1ms, no matter what drives or network you use. With one thread, this stands for only 2000 random read iops and 1000 random write iops, and even if you manage to achieve this result you’re already in a good shape. With best-in-slot hardware and some tuning you may be able to improve it further, but only twice or so.

But does latency matter? Yes, it does, when it comes to single-threaded (synchronous) random reads or writes. Basically, all software that wants the data to be durable does fsync() calls which serialize writes. For example, all DBMSs do. So to understand the performance limit of these apps you should benchmark your cluster with iodepth=1.

The latency doesn’t scale with the number of servers or OSDs-per-SSD or two-RBD-in-RAID0. When you’re benchmarking your cluster with iodepth=1 you’re benchmarking only ONE placement group at a time (PG is a triplet or a pair of OSDs). The result is only affected by how fast a single OSD processes a single request. In fact, with iodepth=1 IOPS=1/latency. There is Nick Fisk’s presentation titled «Low-latency Ceph». By «low-latency» he means 0.7ms, which is only ~1500 iops.

Expected performance

Estimating the cluster performance based on disks' performance is absolutely wrong.

The real expected performance for Bluestore is like the following (iops applies to random 4KB reads/writes):

1 HDD (usual SATA, 7200 rpm, non SMR, without SSD cache) is:

1 fast SSD or NVMe SSD with capacitors (see below) and write iops >= 25000:

Aggregate performance:

Micron setup example

Here’s an example setup from Micron. They used 2x replication, very costly CPUs (2x 28-core Xeon Platinum per server), very fast network (2x100G, in fact 2x2x100G — 2 cards with 2 ports each) and 10x their best NVMes in each of 4 nodes: https://www.micron.com/resource-details/30c00464-e089-479c-8469-5ecb02cfe06f

They only got 350000 peak write iops with high parallelism with 100 % CPU load. It may seem a lot, but if you divide it by the number of NVMes — 350000/40 NVMe — it’s only 8750 iops per an NVMe. If we account for 2 replicas and WAL we get 8750*2*2 = 35000 iops per drive. So… Ceph only squeezed 35000 iops out of an NVMe that can deliver 260000 iops alone. That’s what Ceph’s overhead is.

Also there are no single-thread latency tests in that PDF. Such tests could be very interesting.

Update

https://www.micron.com/-/media/client/global/documents/products/other-documents/micron_9300_and_red_hat_ceph_reference_architecture.pdf

New NVMes are Micron 9300 of maximum possible capacity — 12.8 TB. Each of these delivers even more write iops than 9200’s: 310k instead of 260k. Everything else remains the same.

The new write performance result for 100 RBD clients is 477029 iops (36 % more than in the previous test). Remember that it’s still only 4770 iops per client, though. For 10 RBD clients the result is better: 294000 iops, which stands for 29400 iops per client.

What helped the performance? I guess the configuration did. In comparison to the previous test they changed the following:

Other remarks:

CAPACITORS!

One important thing to note is that all writes in Ceph are transactional, even ones that aren’t specifically requested to be. It means that write operations do not complete until they are written into all OSD journals and fsync()'ed to disks. This is to prevent #RAID WRITE HOLE-like situations.

To make it more clear this means that Ceph does not use any drive write buffers. It does quite the opposite — it clears all buffers after each write. It doesn’t mean that there’s no write buffering at all — there is some on the client side (RBD cache, Linux page cache inside VMs). But internal disk write buffers aren’t used.

This makes typical desktop SSDs perform absolutely terribly for Ceph journal in terms of write IOPS. The numbers you can expect are something between 100 and 1000 (or 500—2000) iops, while you’d probably like to see at least 10000 (even a noname Chinese SSD can do 10000 iops without fsync).

So your disks should also be benchmarked with -iodepth=1 -fsync=1 (or -sync=1, see #O_SYNC vs fsync vs hdparm -W 0).

The thing that will really help us build our lightning-fast Ceph cluster is an SSD with (super)capacitors, which are perfectly visible to the naked eye on M.2 SSDs:

Micron 5100 sata m2.jpg

Supercaps work like a built-in UPS for an SSD and allow it to flush DRAM cache into the persistent (flash) memory when a power loss occurs. Thus the cache becomes «non-volatile» — and thus an SSD safely ignores fsync (FLUSH CACHE) requests, because it’s confident that cache contents will always make their way to the persistent memory.

And this increases transactional write IOPS, making it equal to non-transactional.

Supercaps are usually called «enhanced/advanced power loss protection» in datasheets. This is a feature almost exclusively present only in «server-grade» SSDs (not even all of them). For example, Intel DC S4600 has supercaps and Intel DC S3100 doesn’t.

Note.svg This is the main difference between server and desktop SSDs. An average user doesn’t need transactions, but servers run DBMS’es, and DBMS’es want them really, really bad.

And… Ceph also does :) you should only buy SSDs with supercaps for Ceph clusters. Even if you consider NVMe — NVMe without capacitors is WORSE than SATA with them. Desktop NVMes do 150000+ write iops without syncs, but only 600—1000 iops with them.

Another option is Intel Optane. Intel Optane is also an SSD, but based on the different physics — Phase-Change Memory instead of Flash memory. Specs say these drives are capable of 550000 iops without the need to erase blocks and thus no need for write cache and supercaps. But even if Optane’s latency is 0.005ms (it is), Ceph’s latency is still 0.5ms, so it’s pointless to use them with Ceph — you get the same performance for a lot more money compared to usual server SSDs/NVMes.

Bluestore vs Filestore

TODO: This section lacks random read performance comparisons.

Bluestore is the «new» storage layer of Ceph. All presentations and documents say it’s better in all ways, which indeed seems reasonable for something «new».

Bluestore is really 2x faster than Filestore for linear write workloads, because it has no double-writes — big blocks are written only once, not twice as in Filestore. Filestore journals everything, so all writes first go to the journal and then get copied to the main device.

Bluestore is also more feature-rich: it has checksums, compression, erasure-coded overwrites and virtual clones. Checksums allow 2x-replicated pools self-heal better, erasure-coded overwrites make EC usable for RBD and CephFS, and virtual clones make VMs run faster after taking a snapshot.

In HDD-only (or bad-SSD-only) setups Bluestore is also 2x faster than Filestore for random writes. This is again because it can do 1 commit per write, at least if you apply this patch: https://github.com/ceph/ceph/pull/26909 and turn bluefs_preextend_wal_files on. In fact it’s OK to say that Bluestore’s deferred write implementation is really optimal for transactional writes to slow drives.

However, if you switch to faster drives, Bluestore's random writes don't appear to be much better than Filestore's. This shows up differently in HDD+SSD and All-Flash setups both of which are certainly very popular.

HDD for data + SSD for journal

Filestore writes everything to the journal and only starts to flush it to the data device when the journal fills up to the configured percent. This is very convenient because it makes journal act as a «temporary buffer» that absorbs random write bursts.

Bluestore can’t do the same even when you put its WAL+DB on SSD. It also has sort of a «journal» which is called «deferred write queue», but it’s very small (only 64 requests) and it lacks any kind of background flush threads. So you actually can increase the maximum number of deferred requests, but after the queue fills up the performance will drop until OSD restarts.

So, Bluestore’s performance is very consistent, but it’s worse than peak performance of Filestore for the same hardware. In other words, Bluestore OSD refuses to do random writes faster than the HDD can do them on average.

With Filestore you easily get 1000—2000 iops while the journal is not full. With Bluestore you only get 100—300 iops regardless of the SSD journal, but these are absolutely stable over time and never drop.

SSD-only (All-Flash)

In All-Flash clusters Bluestore’s own latency is usually 30-50 % greater than Filestore’s. However, this only refers to the latency of Bluestore itself, so the absolute number for these 30-50 % is something around 0.1ms which is hard to notice in front of the total Ceph’s latency. And even though latency is greater, peak parallel throughput is usually slightly better (+5..10 %) and peak CPU usage is slightly lower (-5..10 %).

But it’s still a shame that the increase is only 5-10 % for that amount of architectural effort.

RAM usage

Another thing to note is that Bluestore uses a lot more RAM. This is because it uses RocksDB for all metadata, additionally caches some of them by itself and also tries to cache some data blocks to compensate for the lack of page cache usage. The general rule of thumb is 1GB RAM per 1TB of storage, but not less than 2GB per an OSD.

About block.db sizing

As usual, there’s something wrong :)

Official documents say that you should allocate 4 % of the slow device space for block.db (Bluestore’s metadata partition). This is a lot, Bluestore rarely needs that amount of space.

But the main problem is that Bluestore uses RocksDB and RocksDB puts a file on the fast device only if it thinks that the whole layer will fit there (RocksDB is organized in files). Default RocksDB settings in Ceph are:

…so…

RocksDB puts L2 files to block.db only if it’s at least 2560+256+1024 Mb (almost 4 GB). And it puts L3 there only if it’s at least 25600+2560+256+1024 Mb (almost 30 GB). And L4 — only if it’s at least 256000+25600+2560+256+1024 Mb (roughly 286 GB).

In other words, all block.db sizes except 4, 30, 286 GB are pointless, Bluestore won’t use anything above the previous «round» size. At least if you don’t change RocksDB settings. And of these 4 is too small, 286 is too big.

So just stick with 30 GB for all Bluestore OSDs :)

Controllers

CPUs

VM setup and filesystem options

So…

Network, DPDK and SPDK

Drive cache is slowing you down

O_SYNC vs fsync vs hdparm -W 0

SATA and SCSI drives have two ways of flushing cache: FLUSH CACHE command and FUA (Force Unit Access) flag for writes. The first is an explicit flush, the second is an instruction to write the data directly to media. To be more precise, there is FUA flag in SCSI/SAS, but the situation is unclear with SATA: it’s there in the NCQ spec, but in practice most drives don’t support it.

It seems that fsync() sends the FLUSH CACHE command and opening a file with O_SYNC sets the FUA bit.

Does it make any difference? Usually no. But sometimes, depending on the exact controller and/or its configuration, there may be a difference. In this case fio -sync=1 and fio -fsync=1 start to give different results. In some cases drives just ignore one of the flush methods.

In addition to that, SATA/SAS drives also have a cache disable command. When you disable the cache Linux stops sending flushes at all. It may seem that this should also result in the same performance as fsync/O_SYNC, but that’s not the case either! SSDs with supercaps give much better performance with disabled cache. For example, Seagate Nytro 1351 gives you 288 iops with cache and 18000 iops without it (!).

Why? It seems that’s because FLUSH CACHE is interpreted by the drive as a «please flush all caches, including non-volatile cache» command, and «disable cache» is interpreted as «please disable the volatile cache, but you may leave the non-volatile one on if you want to». This makes writes with a flush after every write slower than writes with the cache disabled.

What about NVMe? NVMe has slightly less variability — there is no «disable cache» command in the NVMe spec at all, but just as in the SATA spec there is the FLUSH CACHE command and FUA bit. But again, based on the personal experience I can say that it seems that FUA is often ignored with NVMe either by Linux or by the drive itself, thus fio -sync=1 gives the same results as fio -direct=1 without any sync flags. -fsync=1 performs correctly and lands the performance down to where it must belong (1000—2000 iops for desktop NVMes).

P.S: Bluestore uses fsync. Filestore uses O_SYNC.

Server SSDs

Disabling cache is not a joke!

fio -ioengine=libaio -name=test -filename=/dev/sdb -(sync|fsync)=1 -direct=1 -bs=(4k|4M) -iodepth=(1|32|128) -rw=(write|randwrite)

Micron 5100 Eco 960GB

Write
sync or fsync bs iodepth rw hdparm -W 1 hdparm -W 0
sync 4k 1 write 612 iops 22200 iops
sync 4k 1 randwrite 612 iops 22200 iops
sync 4k 32 randwrite 6430 iops 59100 iops
sync 4k 128 randwrite 6503 iops 59100 iops
sync 4M 32 write 469 MB/s 485 MB/s
fsync 4k 1 write 659 iops 25100 iops
fsync 4k 1 randwrite 671 iops 25100 iops
fsync 4k 32 randwrite 695 iops 59100 iops
fsync 4k 128 randwrite 701 iops 59100 iops
fsync 4M 32 write 384 MB/s 486 MB/s
Read
bs iodepth rw result
4k 1 randread 6000 iops
4k 4 randread 15900 iops
4k 8 randread 18500 iops
4k 16 randread 24800 iops
4k 32 randread 37400 iops
4M 1 read 460 MB/s
4M 16 read 514 MB/s

Reads don’t differ for hdparm -W 0 and 1.

Seagate Nytro 1351 XA3840LE10063

Disk was filled 90-100 % before the test.

Write
sync or fsync bs iodepth rw hdparm -W 1 hdparm -W 0
sync 4k 1 randwrite 18700 iops 18700 iops
sync 4k 4 randwrite 49400 iops 54700 iops
sync 4k 32 randwrite 51800 iops 65700 iops
sync 4M 32 write 516 MB/s 516 MB/s
fsync=1 4k 1 randwrite 288 iops 18100 iops
fsync=1 4k 4 randwrite 288 iops 52800 iops
fsync=4 4k 4 randwrite 1124 iops 53500 iops
fsync=1 4k 32 randwrite 288 iops 65700 iops
fsync=32 4k 32 randwrite 7802 iops 65700 iops
fsync=1 4M 32 write 336 MB/s 516 MB/s
Read
bs iodepth rw result
4k 1 randread 8600 iops
4k 4 randread 21900 iops
4k 8 randread 30500 iops
4k 16 randread 39200 iops
4k 32 randread 50000 iops
4M 1 read 508 MB/s
4M 16 read 536 MB/s

Disable the cache if you want more than 288 iops.

RAID WRITE HOLE

Even if we’re talking about RAID, the thing that is much simpler than distributed software-defined storage like Ceph, we’re still talking about a distributed storage system — every system that has multiple physical drives is distributed, because each drive behaves and commits the data (or doesn’t commit it) independently of others.

Write Hole is the name for several situations in RAID arrays when drives go out of sync. Suppose you have a simple RAID1 array of two disks. You write a sector. You send a write command to both drives. And then a power failure occurs before commands finish. Now, after the system boots again, you don’t know if your replicas contain same data, because you don’t know which drive had succeeded in writing it and which hadn’t.

You say OK, I don’t care. I’ll just read from both drives and if I encounter different data I’ll just pick one of the copies, and I’ll either get the old data or the new.

But then imagine that you have RAID 5. Now you have three drives: two for data and one for parity. Now suppose that you overwrite a sector again. Before the write, your disks contain: (A1), (B1) and (A1 XOR B1). You want to overwrite (B1) with (B2). To do so you write (B2) to the second disk and (A1 XOR B2) to the third. A power failure occurs again… At the same time disk 1 (one that you didn’t write anything to) fails. You might think that you can still reconstruct your data because you have RAID 5 and 2 disks out of 3 are still alive.

But imagine that disk 2 succeeded to write new data and disk 3 failed (or vice versa). Now you have: (lost disk), (B2) and (A1 XOR B1). If you try to reconstruct A from these copies you’ll get (A1 XOR B1 XOR B2) which is obviously not equal to A1. Bang! Your RAID5 corrupted the data that you haven’t even been writing at the time of the power loss.

Because of this problem, Linux `mdadm` refuses to start an incomplete array after unclean shutdown at all. There’s no solution to this problem except full data journaling at the level of each disk drive. And this is… exactly what Ceph does! So, Ceph is actually safer than RAID. Slower — but safer :)

Quick insight into SSD and flash memory organization

The distinctive feature of NAND flash memory is that you can write it in small blocks, but erase only big block groups at once, and you must erase any block before overwriting it. Write unit is called «page», erase unit is called «block». Actual NAND chips have 16 KB pages and 16-24 MB blocks (1024 pages for Micron MLC and 1536 pages for Micron TLC). This is probably because erasing is slow compared to writing, but it can be done for a lot of blocks at once (common sense suggests that erasing is ~1000 times slower than writing). Another distinctive feature is that the total number of erase/program cycles is physically limited — after several thousands cycles (a usual number for MLC memory) the block becomes faulty and stops accepting new writes or even loses the data previously written to it. Denser and cheaper (MLC/TLC/QLC, 2/3/4 bits per cell) memory chips have smaller erase limits, while sparser and more expensive ones (SLC, 1 bit per cell) have bigger limits (up to 100000 rewrites). However, all limits are still finite, so stupidly overwriting the same block would be very slow and would break SSD very rapidly.

But that’s not the case with modern SSDs — even cheap models are very fast and usually very durable. But why? The credit goes to SSD controllers: SSDs contain very smart and powerful controllers, usually with at least 4 cores and 1-2 GHz clock frequency, which means they’re as powerful as mobile phones' processors. All that power is required to make FTL firmware run smoothly. FTL stands for «Flash Translation Layer» and it is the firmware responsible for translating addresses of small blocks into physical addresses on flash memory chips. Every write request is always put into a space freed in advance, and FTL just remembers the new physical location of the data. This makes writes very fast. FTL also defragments free space and moves blocks around to achieve uniform wear across all memory cells. This feature is called Wear Leveling. SSDs also usually have some extra physical space reserved to add even more endurance and to make wear leveling easier; this is called overprovisioning. Pricier server SSDs have a lot of space overprovisioned, for example, Micron 5100 Max has 37,5 % of physical memory reserved (extra 60 % is added to the user-visible capacity).

And this is also the FTL which makes power loss protection a problem. Mapping tables are metadata which must also be forced into non-volatile memory when you flush the cache, and it’s what makes desktop SSDs slow with fsync… In fact, as I wrote it I thought that they could use RocksDB or similar LSM-tree based system to store mapping tables and that could make fsyncs fast even without the capacitors. It would lead to some waste of journal space and some extra write amplification (as every journal block would only contain 1 write), but still it would make writes fast. So… either they don’t know about LSM trees or the FTL metadata is not the only problem for fsync.

When I tried to lecture someone in the mailing list about «all SSDs doing fsyncs correctly» I got this as the reply: https://www.usenix.org/system/files/conference/fast13/fast13-final80.pdf. Long story short, it says that in 2013 a common scenario was SSDs not syncing metadata on fsync calls at all which led to all kinds of funny things on a power loss, up to (!!!) total failures of some SSDs.

There also exist some very old SSDs without capacitors (OCZ Vector/Vertex) which are capable of very large sync iops numbers. How do they work? Nobody knows, but I suspect that they just don’t do safe writes :). The core principle of flash memory overwrites hasn’t changed in recent years. 5 years ago SSDs were also FTL-based, just as now.

So it seems there are two kinds of «power loss protection»: simple PLP means «we do fsyncs and don’t die or lose your data when a power loss occurs», and advanced PLP means that fsync’ed writes are just as fast as non-fsynced. It also seems that in the current years (2018—2019) simple PLP is already a standard and most SSDs don’t lose data on power failure.

A bonus: USB thumb drives

Why are USB flash drives so slow then? In terms of small random writes they usually only deliver 2-3 operations per second, while being powered by similar flash memory chips — maybe slightly cheaper and worse ones, but obviously not 1000 times worse.

The answer also lies in the FTL. Thumb drives also have FTL and they even have some Wear Leveling, but it’s very small and dumb compared to SSD FTLs. It has a slow CPU and only a little memory. Thus it doesn’t have place to store a full mapping table for small blocks and thus it translates the positions of big blocks (1-2 megabytes or even bigger) instead. Writes are buffered and then flushed one block at a time; there is a small limit on the number of blocks that can be buffered at once. The limit is usually only between 3 and 6 blocks.

This limit is always sufficient to copy big files to a flash drive formatted in any of common filesystems. One opened block receives metadata and another receives data, then it just moves on. But if you start doing random writes you stop hitting the opened blocks and this is where lags come in.

Bonus: Micron vSAN reference architecture

Micron Accelerated All-Flash SATA vSAN 6.7 Solution

Node configuration:

«Aligns with VMWare AF-6, aims up to 50K read iops per node»

Total I/O parallelism: 512

100%/70%/50%/30%/0% write

Conclusion:

Good SSD models

https://docs.google.com/spreadsheets/d/1E9-eXjzsKboiCCX-0u0r5fAjjufLKayaut_FOPxYZjc

Conclusion

Quick guide for optimizing Ceph for random reads/writes: