Ceph performance
Quick insight into SSD and flash memory organization
Although flash memory allows fast random writes in small (usually 512 byte) blocks, its distinctive feature is that every block must be erased before being written to. But erasing is slow compared to reading and writing, so manufacturers design memory chips so that they always erase a large group of blocks at once (which takes the same time as erasing one block could take). This group of blocks is called «erase unit» is typically 2-4 megabytes in size. Another distinctive features is that the total number of erase/program cycles is physically limited — after several thousands cycles (a usual number for MLC memory) the block becomes faulty and either just stops accepting new writes or loses the data previously written to it. Denser and cheaper (MLC/TLC/QLC, 2/3/4 bits per cell) memory chips have smaller erase limits, while sparser and more expensive ones (SLC, 1 bit per cell) limits can be as large as 100000 rewrites. However, all limits are still finite, so stupidly overwriting the same block would be very slow and would break SSD very rapidly.
But that’s not the case with modern SSDs, they are very fast and last very long. Even cheap models are usually rather strong. But why? The credit goes to SSD controllers: SSDs contain very smart and powerful controllers, usually with at least 4 cores and 1-2 GHz clock frequency, which means they’re as powerful as mobile phones' processors. All that power is required to make FTL firmware run smoothly. FTL stands for «Flash Translation Layer» and it is the firmware responsible for translating addresses of small blocks into physical addresses on flash memory chips. Every write request is always put into the space freed in advance, and FTL just remembers the new physical location of the data. This makes writes very fast. FTL also defragments free space and moves blocks around to achieve uniform wear across all memory cells. This feature is called Wear Leveling. SSDs also usually have some extra physical space reserved to add even more endurance and to make wear leveling easier; this is called overprovisioning. Pricier server SSDs have a lot of space overprovisioned, for example, Micron 5100 Max has extra 60 % of the user capacity on top of it.
And this is also the FTL which makes power loss protection a problem. The mapping tables are the metadata which must also be forced into the non-volatile memory when you flush the cache, and it’s what makes desktop SSDs slow with fsync. As I wrote it I thought that they could use RocksDB or similar LSM-tree based system for storing mapping tables and that could make fsyncs fast even without the capacitors. It would lead to some waste of journal space, but still it would make writes fast. So… either they don’t know about LSM trees or the fsync problem is not caused only by the FTL metadata.
When I tried to lecture someone in the mailing list about «all SSDs doing fsyncs correctly» I got this as the reply: https://www.usenix.org/system/files/conference/fast13/fast13-final80.pdf. Long story short, it says that in 2013 a common scenario was SSDs not syncing metadata on fsync calls at all which lead to all kinds of funny things on a power loss, up to (!!!) total failures of some SSDs.
There also exist some very old SSDs without capacitors (OCZ Vector/Vertex) which are capable of very large sync iops numbers. How do they work? Nobody knows, but I suspect that they just don’t do safe writes :). The core principle of flash memory overwrites didn’t change in the last years, and SSDs were also based on FTLs just as they do now.
So it seems there are two kinds of «power loss protection»: simple PLP means «we do fsyncs and don’t die or lose your data when a power loss occurs», and advanced PLP means fsync’ed writes are just as fast as non-fsynced. It also seems that in the current years (2018—2019) simple PLP is already a standard and most SSDs don’t lose data on power failure.
A bonus: USB thumb drives
А почему тогда USB-флешки такие медленные? Случайная запись на флешку 512-байтными (или 4 Кб) блоками обычно идёт со скоростью 2-3 iops. А флеш-память там ровно та же, что в SSD — ну, может, более дешёвые вариации, но разница же не на порядки. Ответ кроется в том, что на флешках тоже есть FTL (и даже Wear Leveling), но по сравнению с SSD-шным он маленький и тупой. У него слабый процессор и мало памяти. Из-за малого объёма RAM контроллеру флешки, в отличие от контроллера SSD, негде хранить полную таблицу сопоставления виртуальных и реальных секторов — поэтому отображаются не сектора, а крупные блоки где-то по мегабайту или больше, а при записи есть лимит на количество «открытых» блоков. Как это происходит:
- Допустим, вы пишете в сектор X.
- Контроллер отображает блок, которому принадлежит этот сектор, на реальный блок, и «открывает» его — выделяет пустой блок, запоминает, что он «дочерний» для открытого и записывает туда один изменённый вами сектор.
- Таким макаром можно открыть максимум N разных блоков; число N обычно очень маленькое — от 3 до 6.
- Дальше если вы пишете следующий сектор из уже открытого блока — он просто записывается в его дочерний блок (что быстро).
- Если же следующий записываемый сектор принадлежит другому блоку — какой-то из открытых блоков приходится закрывать и «сливать» содержимое дочернего блока с оригинальным.
Для копирования больших файлов на флешку, отформатированную в любую из стандартных файловых систем, двух блоков достаточно: в один открытый блок пишутся данные, во второй — метаданные записываемого файла. Запись последовательная, всё быстро. А вот при случайной записи вы перестаёте попадать в уже «открытые» блоки и каждая операция записи превращается в полное стирание. Тут-то и начинаются тормоза…
