Изменения

Перейти к: навигация, поиск

Ceph performance

1229 байтов добавлено, 13:07, 18 июля 2020
Нет описания правки
[[File:Ceph-funnel-en.svg|500px|right]] [[ru:Производительность Ceph]]
Ceph is a Software-Defined Storage system. It's It’s very feature-rich: it provides object storage, VM disk storage, shared cluster filesystem and a lot of additional features. In some ways, it's it’s even unique.
It could be an excellent solution which you could take for free, immediately solve all your problems, become a cloud provider and earn piles of money. However there is a subtle problem: PERFORMANCE. Rational people rarely want to lower the performance down by 95 95 % in production. It seems cloud providers like AWS, GCP, Yandex don't care - don’t care — all of them run their clouds on top of their own crafted SDS-es (not even Ceph) and all these SDS-es are just as slow. :-) we don't don’t judge them of course, that's that’s their own business.
This article describes which performance numbers you can achieve with Ceph and how. But I warn you: you won't won’t catch up with local SSDs. Local SSDs (especially NVMe) are REALLY fast right now, their latency is about 0.05ms. It's It’s very hard for an SDS to achieve the same result, and beating it is almost impossible. It's just the The network that will eat alone eats those 0.05ms...
== General benchmarking principles ==
This limit is always sufficient to copy big files to a flash drive formatted in any of common filesystems. One opened block receives metadata and another receives data, then it just moves on. But if you start doing random writes you stop hitting the opened blocks and this is where lags come in.
 
== Bonus: Micron vSAN reference architecture ==
 
[https://media-www.micron.com/-/media/client/global/documents/products/other-documents/micron_vsan_6,-d-,7_on_x86_smc_reference_architecture.pdf Micron Accelerated All-Flash SATA vSAN 6.7 Solution]
 
Node configuration:
 
* 384 GB RAM 2667 MHz
* 2X Micron 5100 MAX 960 GB (randread: 93k iops, randwrite: 74k iops)
* 8X Micron 5200 ECO 3.84TB (randread: 95k iops, randwrite: 17k iops)
* 2x Xeon Gold 6142 (16c 2.6GHz)
* Mellanox ConnectX-4 Lx
* Connected to 2x Mellanox SN2410 25GbE switches
 
«Aligns with VMWare AF-6, aims up to 50K read iops per node»
 
* 2 replicas (like Ceph size=2)
* 4 nodes
* 4 VMs on each node
* 8 vmdk per VM
* 4 threads per vmdk
Total I/O parallelism: 512
 
100%/70%/50%/30%/0% write
* «Baseline» (fits in cache): 121k/178k/249k/314k/486k iops
* «Capacity» (doesn’t): 51k/66k/90k/134k/363k
* Latency is 1000*512/IOPS ms in all tests (1000ms * parallelism / iops)
* '''No latency tests with low parallelism'''
* '''No linear read/write tests'''
 
Conclusion:
* ~3800 write iops per drive
* ~11343 read iops per drive
* ~1600 write iops per drive when not in cache
* Parallel workload doesn’t look better than Ceph. vSAN is hyperconverged, though.
== Good SSD models ==

Навигация