Compare commits

...

552 Commits

Author SHA1 Message Date
Vitaliy Filippov 989675a780 s/etcd_ws_keepalive_timeout/etcd_ws_keepalive_interval/ in docs 2024-03-26 01:56:08 +03:00
Vitaliy Filippov f8c403ec9e Add newer benchmark results 2024-03-23 18:28:48 +03:00
Vitaliy Filippov bfbb85e653 Replace -Oanything with -O3, not just -O/-O1/-O2
Test / test_move_reappear (push) Successful in 22s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m11s Details
Test / test_snapshot_chain_ec (push) Successful in 3m14s Details
Test / test_rebalance_verify_imm (push) Successful in 3m29s Details
Test / test_rebalance_verify (push) Successful in 4m4s Details
Test / test_switch_primary (push) Successful in 40s Details
Test / test_write (push) Successful in 42s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_write_xor (push) Successful in 1m3s Details
Test / test_rebalance_verify_ec (push) Successful in 4m36s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m43s Details
Test / test_heal_pg_size_2 (push) Successful in 3m36s Details
Test / test_heal_ec (push) Successful in 6m7s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m39s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m33s Details
Test / test_heal_csum_32k (push) Successful in 6m38s Details
Test / test_scrub (push) Successful in 1m52s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m32s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m30s Details
Test / test_heal_csum_4k (push) Successful in 6m25s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m32s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s Details
Test / test_scrub_pg_size_3 (push) Successful in 43s Details
Test / test_nfs (push) Successful in 13s Details
Test / test_scrub_ec (push) Successful in 19s Details
2024-03-18 02:03:44 +03:00
Vitaliy Filippov 9ad6822353 Release 1.5.0
Test / test_rm (push) Successful in 14s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m36s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m20s Details
Test / test_snapshot_chain_ec (push) Successful in 3m5s Details
Test / test_rebalance_verify_imm (push) Successful in 5m11s Details
Test / test_rebalance_verify (push) Successful in 5m55s Details
Test / test_switch_primary (push) Successful in 33s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m26s Details
Test / test_write (push) Successful in 54s Details
Test / test_write_xor (push) Successful in 57s Details
Test / test_write_no_same (push) Successful in 19s Details
Test / test_rebalance_verify_ec (push) Successful in 7m21s Details
Test / test_heal_pg_size_2 (push) Successful in 4m36s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m33s Details
Test / test_heal_ec (push) Successful in 6m15s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m31s Details
Test / test_heal_csum_32k (push) Successful in 6m29s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m15s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m16s Details
Test / test_scrub (push) Successful in 1m18s Details
Test / test_scrub_xor (push) Successful in 1m13s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m10s Details
Test / test_scrub_ec (push) Successful in 56s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 59s Details
Test / test_heal_csum_4k (push) Successful in 6m2s Details
Test / test_scrub_pg_size_3 (push) Successful in 2m11s Details
Test / test_nfs (push) Successful in 11s Details
After half a year of hard work, VitastorFS is finally here ! :-)

New features:
- VitastorFS, a full-featured clustered (read-write-many) file system.
  Documentation: [VitastorFS](docs/usage/nfs.en.md)
- Embedded key-value database implementation based on Parallel Optimistic B-Tree
  algorithm and used for the metadata of VitastorFS
- Pool management commands in vitastor-cli (create-pool, list-pools, rm-pool, modify-pool).
  Thanks MIND Software (https://mindsw.io) for their contribution!
  [Documentation](docs/usage/cli.en.md#create-pool)

Bug fixes:
- Fix a very rare "infinite loop" in the client library
- Fix a rare OSD hang on during start when zeroing out bad metadata entries left from the previous run
2024-03-16 15:35:10 +03:00
Vitaliy Filippov 2043b4e374 Fix build errors for gcc 8 2024-03-16 15:35:10 +03:00
Vitaliy Filippov de840e6fe3 Reduce kv-cli loadjson load parallelism to 16 2024-03-16 15:35:10 +03:00
Vitaliy Filippov b5e04bf809 Fix build warning 2024-03-16 15:35:10 +03:00
Vitaliy Filippov 8807a1623b Fix markdown tables 2024-03-16 15:35:10 +03:00
Vitaliy Filippov f12855c31b Add vitastor-kv to packages 2024-03-16 15:35:10 +03:00
Vitaliy Filippov e75dcc9a71 Add documentation for VitastorFS 2024-03-16 15:16:43 +03:00
Vitaliy Filippov 88516ab4bd Remove extra log 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 6221126b4f Allow to print simple-offsets just given the device size 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 6783d4a13c Implement fool protection for FS pools 2024-03-16 13:24:36 +03:00
Vitaliy Filippov dcbe1afac3 Store pool ID in inode metadata 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 0bde28c24a Make nfs_do_rmw a library function 2024-03-16 13:24:36 +03:00
Vitaliy Filippov bb8ca6184e Support setattr guard 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 87310ef7bb Support ctime 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 4f4b2dab80 Log NFS liveness checks 2024-03-16 13:24:36 +03:00
Vitaliy Filippov f70da82317 Add loadjson command to vitastor-kv 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e42148f347 Allow to specify KV commands on command line 2024-03-16 13:24:36 +03:00
Vitaliy Filippov c289584469 Add JSON dump format 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 018e89f867 Erase verf key left from creation from ientries on every modification 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 603dc68f11 Implement async mtime change 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 7b12342933 Allow to specify additional NFS mount options 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 44bf0f16ee Fix malloc/free in nfs_kv_read/write 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 8840c84572 Fix "bad key in etcd" in mon for FS pools 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 5b747c12ec Check if already mounted before mounting 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 05f5f46162 Fix zero used space, update mtime when moving/changing inode 2024-03-16 13:24:36 +03:00
Vitaliy Filippov b5604191c8 Ignore ECANCELED in nfs-proxy (happens in io_uring on fork) 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e871de27de Support unaligned shared_offsets, align shared file data instead of header 2024-03-16 13:24:36 +03:00
Vitaliy Filippov f600ce98e2 Implement auto-unmount local NFS server mode for vitastor-nfs 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 57605a5c13 Return error on failed shrink 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 29bd4561bb Implement rename over an existing file/directory 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 7142460ec8 Support --logfile in nfs-proxy 2024-03-16 13:24:36 +03:00
Vitaliy Filippov d03f19ebe5 Fix shared file overlap, add FIXMEs 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 88f9d18be3 Create inode, then direntry, not direntry, then inode; retry ID collisions 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 6213fbd8c6 Fix NFS shared/aligned write FIXMEs 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 3aee37eadd Allow to disable per-inode stats for VitastorFS pools 2024-03-16 13:24:36 +03:00
Vitaliy Filippov ecfc753e93 Add basic NFS tests, fix bugs 2024-03-16 13:24:36 +03:00
Vitaliy Filippov a574f9ad71 Return block NFS implementation back as an option too 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 7c235c9103 Move KV FS header into a separate file 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e5bb986164 Implement packing small files into shared inodes 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 181795d748 Split new NFS proxy implementation into multiple files 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 8cdc38805b WIP VitastorFS with metadata storage in VitastorKV 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 0cd455d17f First just recheck version without actually re-reading block in vitastor-kv 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 32ba653ba6 Fix vitastor-kv hang on reopen & unfinished closed listing 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 231d4b15fc Add loadable dump format to vitastor-kv (dump) 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 9dc4d5fd7b Fix freeing r/w buffers on errors in kv_db 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e58538fa47 Fix eviction when random_pos selects the end 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 11ac9e7024 Implement min/max list_count to make listings during performance test reasonable 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 511bc3df1c Fix and improve parallel allocation
- Do not try to allocate more DB blocks in an inode block until it's "confirmed" and "locked" by the first write
- Do not recheck for new zero DB blocks on first write into an inode block - a CAS failure means someone else is already writing into it
- Throw new allocation blocks away regardless of whether the known_version is 0 on a CAS failure
2024-03-16 13:24:36 +03:00
Vitaliy Filippov a64f0d1f73 Implement key_prefix for K/V stress test 2024-03-16 13:24:36 +03:00
Vitaliy Filippov ec5f7c6b87 More fixes
- do not overwrite a block with older version if known version is newer
  (read may start before update and end after update)
- invalidated block versions can't be remembered and trusted
- right boundary for split blocks is right_half when diving down, not key_lt
- restart update also when block is "invalidated", not just on version mismatch
- copy callback in listings to avoid closure destruction bugs too
2024-03-16 13:24:36 +03:00
Vitaliy Filippov 3ebed9a749 Add logging and one more assert 2024-03-16 13:24:36 +03:00
Vitaliy Filippov eab67a6e8f Make get_block() wait for updating when unrelated block is found along the path 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 20993d9b7a Fix a race condition where changed blocks were parsed over existing cached blocks and getting a mix of data 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 5cf9b343c0 Simplify code by removing an unneeded "optimisation" 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 79ae0aadcd Add kv_log_level, print warnings on level 1, trace ops on level 10 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 605afc3583 Fix duplicate keys in listings on parallel updates -- do not rewind key "iterator position" 2024-03-16 13:24:36 +03:00
Vitaliy Filippov c0681d8242 Implement key suffix to avoid collisions of multiple test workers 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 763e77b4f4 Do not complain on empty first block 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 19426aa4c5 Add JSON output for stress-tester 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 08f586bcec Print total stats 2024-03-16 13:24:36 +03:00
Vitaliy Filippov f1cd87473a Do not send more than op_count operations (fix segfault on finish) 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 1bd8d2da56 Add some more resiliency to serialize() 2024-03-16 13:24:36 +03:00
Vitaliy Filippov a7396d2baf Invalidate blocks being updated too 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e98a38810d Change new block allocation method: make each writer choose multiple empty PG blocks and place blocks in them 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 28c4324c36 Remove blocks from cache on unsuccessful updates 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 31ec3fa8f5 Allow to track multiple updates per block (it should never happen though) 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e4fa26f60a Do not call stop_updating after failed write_new_block and after clear_block (both delete the item) 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 59ae27f9e5 Track versions of parent blocks and recheck if changed during update 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 2c6a301d9b Fix resume_split condition (key_lt can also be "") 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 01558349f8 Experiment: transform offsets for better sharding 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 36f4717d0d More post-stress-test fixes
- Prevent _split types of new blocks
- Stop updating new blocks only after the whole update, otherwise pointers
  may become invalid
- Use recheck_none for updates initially
- Use UINT64_MAX as initial block version when postponing ops, otherwise the
  check fails when the block is initially empty. This for example leads to
  writing both leaf items & block pointers (which is incorrect) into the root
  block when starting stress-test with --parallelism 32
- Fix -EINTR comparison
2024-03-16 13:24:36 +03:00
Vitaliy Filippov babaf2a0ce Print operation statistics 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 5773f1a375 K/V fixes after stress-test :-)
- track block versions correctly - per inode block (128kb) instead of tree block (4kb)
- prevent multiple parallel CAS writes of the same inode block
- add logging for EILSEQ which means invalid data in the tree
- fix get_block updated flag which was true for blocks already in cache and was leading to infinite loops on "unrelated block" errors
- apply changes to blocks in cache only after successful writes (using "virtual changes")
- do not replace cached block with an older version from disk
- recheck "unrelated blocks" (read/update collisions) until data stops changing
- track tree path correctly - do not treat split block as parent of its right half
- correctly move blocks when finding new empty place on disk
- restart updates from the beginning when one of blocks is changed by a parallel update
- fix delete using SET opcode and setting key to the empty value instead
- prevent changing the same key more than 1 time in parallel
- fix listing verification
- resume continue_updates in update_find (required because it uses continue_update itself)
- add allow_old_cached parameter to get()
2024-03-16 13:24:36 +03:00
Vitaliy Filippov 57222a9f79 Implement K/V DB stress tester 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 61ef000c6e Evict blocks based on memory limit & block usage 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 7d5e1cc393 Track blocks per level 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 5e7f27a02d Track block level 2024-03-16 13:24:36 +03:00
Vitaliy Filippov fd1d8a8520 Experimental B-Tree Vitastor embedded K/V database implementation! 2024-03-16 13:24:36 +03:00
Vitaliy Filippov c364e14c40 Stop then retry, not retry then stop 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 3ebbfa0428 Fix another rare OSD hang on zeroing out entries on start 2024-03-16 13:24:36 +03:00
Vitaliy Filippov aa79d1db1c Fix incorrect "changing scheme" message in modify-pool
Test / test_rm (push) Successful in 14s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s Details
Test / test_move_reappear (push) Successful in 20s Details
Test / test_snapshot_down (push) Successful in 29s Details
Test / test_snapshot_down_ec (push) Successful in 29s Details
Test / test_splitbrain (push) Successful in 28s Details
Test / test_snapshot_chain (push) Successful in 2m5s Details
Test / test_snapshot_chain_ec (push) Successful in 3m3s Details
Test / test_rebalance_verify_imm (push) Successful in 4m0s Details
Test / test_rebalance_verify (push) Successful in 4m40s Details
Test / test_switch_primary (push) Successful in 38s Details
Test / test_write (push) Successful in 41s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_write_xor (push) Successful in 1m2s Details
Test / test_rebalance_verify_ec (push) Successful in 5m34s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m34s Details
Test / test_heal_pg_size_2 (push) Successful in 3m22s Details
Test / test_heal_ec (push) Successful in 4m58s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m37s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m21s Details
Test / test_heal_csum_32k (push) Successful in 7m1s Details
Test / test_scrub (push) Successful in 1m37s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m59s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m26s Details
Test / test_scrub_xor (push) Successful in 1m3s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m20s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m7s Details
Test / test_scrub_ec (push) Successful in 36s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m37s Details
Test / test_heal_csum_4k (push) Successful in 6m23s Details
2024-03-06 00:41:35 +03:00
Vitaliy Filippov a1fecb7eff Move callback away when calling it in cluster_client 2024-03-06 00:41:35 +03:00
Vitaliy Filippov ff74b19423 Fix rare OSD hang on zeroing out bad entries on start 2024-03-06 00:41:35 +03:00
Vitaliy Filippov 4cf6dceed7 Merge branch 'rel-1.4'
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot_chain (push) Has been cancelled Details
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_switch_primary (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
2024-02-29 09:59:01 +03:00
Vitaliy Filippov 38b8963330 Release 1.4.8
Test / test_rm (push) Successful in 19s Details
Test / test_move_reappear (push) Successful in 26s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m40s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 34s Details
Test / test_splitbrain (push) Successful in 27s Details
Test / test_snapshot_chain (push) Successful in 2m18s Details
Test / test_snapshot_chain_ec (push) Successful in 2m59s Details
Test / test_rebalance_verify_imm (push) Successful in 5m32s Details
Test / test_rebalance_verify (push) Successful in 6m11s Details
Test / test_switch_primary (push) Successful in 41s Details
Test / test_write (push) Successful in 45s Details
Test / test_write_no_same (push) Successful in 23s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m2s Details
Test / test_write_xor (push) Successful in 55s Details
Test / test_rebalance_verify_ec (push) Successful in 6m22s Details
Test / test_heal_pg_size_2 (push) Successful in 5m41s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m59s Details
Test / test_heal_csum_32k_dj (push) Successful in 7m19s Details
Test / test_heal_csum_32k (push) Successful in 7m17s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m14s Details
Test / test_scrub (push) Successful in 1m12s Details
Test / test_heal_ec (push) Successful in 9m2s Details
Test / test_scrub_xor (push) Successful in 56s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m8s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 2m1s Details
Test / test_heal_csum_4k_dj (push) Successful in 4m45s Details
Test / test_scrub_pg_size_3 (push) Successful in 2m31s Details
Test / test_heal_csum_4k (push) Successful in 4m54s Details
Test / test_scrub_ec (push) Successful in 46s Details
- Do not use \r if output is not a terminal (should fix unexpected job output in proxmox)
- Fix rm/rm-data error return code, add --down-ok option to bypass the error
- Add EIO retry timeout and allow to disable these retries, rename up_wait_retry_interval to client_retry_interval
- Add ubuntu jammy build
- Wait for blockstore initialisation before starting OSD (prevent timeouts when init takes time)
- Fix a rare use-after-free in automatic sync after delete in blockstore
2024-02-29 09:58:34 +03:00
Vitaliy Filippov 77167e2920 Do not use \r if output is not a terminal 2024-02-29 00:21:17 +03:00
Vitaliy Filippov 5af23672d0 Fix rm/rm-data error return code, add --down-ok option to bypass the error 2024-02-29 00:20:10 +03:00
Vitaliy Filippov 6bf1f539a6 Add EIO retry timeout and allow to disable these retries, rename up_wait_retry_interval to client_retry_interval 2024-02-28 13:10:02 +03:00
Vitaliy Filippov 4eab26f968 Add documentation and a very basic test for pool management commands
Test / test_snapshot_ec (push) Successful in 31s Details
Test / test_rm (push) Successful in 17s Details
Test / test_move_reappear (push) Successful in 24s Details
Test / test_snapshot_down (push) Successful in 27s Details
Test / test_snapshot_down_ec (push) Successful in 33s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_snapshot_chain (push) Successful in 2m15s Details
Test / test_snapshot_chain_ec (push) Successful in 2m58s Details
Test / test_rebalance_verify_imm (push) Successful in 5m3s Details
Test / test_rebalance_verify (push) Successful in 5m36s Details
Test / test_switch_primary (push) Successful in 37s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m3s Details
Test / test_write_no_same (push) Successful in 21s Details
Test / test_write (push) Successful in 58s Details
Test / test_write_xor (push) Successful in 1m31s Details
Test / test_rebalance_verify_ec (push) Successful in 6m20s Details
Test / test_heal_pg_size_2 (push) Successful in 4m7s Details
Test / test_heal_ec (push) Successful in 4m33s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m53s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m17s Details
Test / test_heal_csum_32k (push) Successful in 7m23s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m56s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m26s Details
Test / test_scrub (push) Successful in 1m29s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m1s Details
Test / test_scrub_xor (push) Successful in 1m1s Details
Test / test_heal_csum_4k (push) Successful in 6m34s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 32s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m19s Details
Test / test_scrub_ec (push) Successful in 24s Details
2024-02-28 13:08:04 +03:00
Vitaliy Filippov 86243b7101 Rework & fix pool-create / pool-modify / pool-ls 2024-02-28 13:08:04 +03:00
idelson dc92851322 vitastor-cli: add commands to control pools: pool-create, pool-ls, pool-modify, pool-rm
PR #59 - https://github.com/vitalif/vitastor/pull/58/commits

By MIND Software LLC

By submitting this pull request, I accept Vitastor CLA
2024-02-28 13:08:04 +03:00
Zibort Cloud 02d1f16bbd Add ubuntu jammy build
PR #62 #62

I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
2024-02-28 11:43:54 +03:00
Vitaliy Filippov fc413038d1 Wait for blockstore initialisation before starting OSD
Test / test_cas (push) Has been cancelled Details
Test / test_change_pg_count (push) Has been cancelled Details
Test / test_change_pg_count_ec (push) Has been cancelled Details
Test / test_change_pg_size (push) Has been cancelled Details
Test / test_create_nomaxid (push) Has been cancelled Details
Test / test_etcd_fail (push) Has been cancelled Details
Test / test_interrupted_rebalance (push) Has been cancelled Details
Test / test_interrupted_rebalance_imm (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec_imm (push) Has been cancelled Details
Test / test_failure_domain (push) Has been cancelled Details
Test / test_snapshot (push) Has been cancelled Details
Test / test_snapshot_ec (push) Has been cancelled Details
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot_chain (push) Has been cancelled Details
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_switch_primary (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
2024-02-27 02:20:04 +03:00
Vitaliy Filippov 1bc0b5aab3 Fix a rare use-after-free in automatic sync after delete in blockstore
Test / test_interrupted_rebalance_ec (push) Successful in 2m49s Details
Test / test_rm (push) Successful in 14s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m29s Details
Test / test_snapshot_chain_ec (push) Successful in 2m48s Details
Test / test_rebalance_verify_imm (push) Successful in 4m9s Details
Test / test_rebalance_verify (push) Successful in 4m42s Details
Test / test_switch_primary (push) Successful in 41s Details
Test / test_write (push) Successful in 43s Details
Test / test_write_no_same (push) Successful in 21s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m37s Details
Test / test_write_xor (push) Successful in 1m11s Details
Test / test_rebalance_verify_ec (push) Successful in 7m14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m3s Details
Test / test_heal_ec (push) Successful in 4m18s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m5s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m52s Details
Test / test_heal_csum_32k (push) Successful in 6m23s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m23s Details
Test / test_scrub (push) Successful in 1m30s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m18s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m9s Details
Test / test_scrub_xor (push) Successful in 57s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m5s Details
Test / test_scrub_ec (push) Successful in 1m6s Details
Test / test_scrub_pg_size_3 (push) Successful in 2m3s Details
Test / test_heal_csum_4k (push) Successful in 4m54s Details
ASan report: [0] READ of size 16 at operator() /root/vitastor/src/blockstore_write.cpp:100
...[5] blockstore_impl_t::ack_sync(blockstore_op_t*) /root/vitastor/src/blockstore_sync.cpp:232
2024-02-24 00:06:36 +03:00
Vitaliy Filippov 5e934264cf Release 1.4.7
- Fix another old "BUG: Attempt to overwrite used offset" in a very simple
  case: bs=4k rw=write iodepth=16 from OSD start; add this case to tests
- Fix a rare crash with "unexpected state during flush: 0x51" possible with
  EC since 1.4.2 during rebalance and OSD outages
- Fix a rare write stall with EC & immediate_commit=none caused by sync
  operations reserving unneeded space in the journal
- Fix 32-bit build warnings, most in printf/scanf format strings
2024-02-22 12:45:52 +03:00
Vitaliy Filippov f20564b44b Fix 32-bit build warnings (99.9% in printf) 2024-02-22 12:22:16 +03:00
Vitaliy Filippov b3c15db331 32M journal by default in simple-offsets
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_rm (push) Successful in 18s Details
Test / test_move_reappear (push) Successful in 24s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m17s Details
Test / test_snapshot_chain_ec (push) Successful in 2m55s Details
Test / test_rebalance_verify_imm (push) Successful in 2m46s Details
Test / test_rebalance_verify (push) Successful in 3m9s Details
Test / test_switch_primary (push) Successful in 39s Details
Test / test_write (push) Successful in 43s Details
Test / test_write_no_same (push) Successful in 19s Details
Test / test_write_xor (push) Successful in 55s Details
Test / test_rebalance_verify_ec (push) Successful in 3m35s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m37s Details
Test / test_heal_pg_size_2 (push) Successful in 3m36s Details
Test / test_heal_ec (push) Successful in 5m47s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m21s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m16s Details
Test / test_heal_csum_32k (push) Successful in 6m45s Details
Test / test_scrub (push) Successful in 1m56s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m39s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m42s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m16s Details
Test / test_scrub_xor (push) Successful in 47s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m26s Details
Test / test_heal_csum_4k (push) Successful in 6m32s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 48s Details
Test / test_scrub_ec (push) Successful in 49s Details
2024-02-21 15:25:02 +03:00
Vitaliy Filippov 685bcd6ef9 Do not reserve extra space for big_writes during sync - sync itself is needed to commit and clear them 2024-02-21 13:00:14 +03:00
Vitaliy Filippov 3eb389b321 Supposed fix for "unexpected state during flush: 0x51" with EC
Test / test_move_reappear (push) Successful in 22s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_snapshot_chain (push) Successful in 2m4s Details
Test / test_snapshot_chain_ec (push) Successful in 2m51s Details
Test / test_rebalance_verify_imm (push) Successful in 2m47s Details
Test / test_rebalance_verify (push) Successful in 3m30s Details
Test / test_switch_primary (push) Successful in 38s Details
Test / test_write (push) Successful in 51s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_write_xor (push) Successful in 52s Details
Test / test_rebalance_verify_ec (push) Successful in 3m32s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m7s Details
Test / test_scrub_zero_osd_2 (push) Successful in 59s Details
Test / test_scrub (push) Successful in 1m2s Details
Test / test_scrub_xor (push) Successful in 36s Details
Test / test_scrub_ec (push) Successful in 38s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 40s Details
Test / test_scrub_pg_size_3 (push) Successful in 49s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m12s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m8s Details
Test / test_heal_csum_32k (push) Successful in 4m55s Details
Test / test_heal_ec (push) Failing after 10m14s Details
Test / test_heal_csum_4k_dmj (push) Successful in 4m59s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m5s Details
Test / test_heal_pg_size_2 (push) Successful in 3m54s Details
Test / test_heal_csum_4k (push) Successful in 3m49s Details
2024-02-21 01:32:06 +03:00
Vitaliy Filippov 3d16cde23c Fix assertions, add small sequential write test
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m8s Details
Test / test_snapshot_chain_ec (push) Successful in 2m48s Details
Test / test_rebalance_verify_imm (push) Successful in 2m57s Details
Test / test_rebalance_verify (push) Successful in 3m29s Details
Test / test_switch_primary (push) Successful in 36s Details
Test / test_write (push) Successful in 54s Details
Test / test_write_xor (push) Successful in 51s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_rebalance_verify_ec (push) Successful in 3m40s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m20s Details
Test / test_scrub (push) Successful in 1m1s Details
Test / test_scrub_zero_osd_2 (push) Successful in 46s Details
Test / test_scrub_xor (push) Successful in 41s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m0s Details
Test / test_scrub_ec (push) Successful in 58s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m45s Details
Test / test_heal_pg_size_2 (push) Failing after 4m52s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m36s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m33s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m35s Details
Test / test_interrupted_rebalance (push) Successful in 2m28s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m30s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m41s Details
Test / test_heal_ec (push) Failing after 10m20s Details
Test / test_heal_csum_4k_dmj (push) Successful in 4m21s Details
Test / test_heal_csum_32k (push) Successful in 5m15s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m48s Details
Test / test_heal_csum_4k (push) Successful in 5m32s Details
2024-02-20 19:41:48 +03:00
Vitaliy Filippov c6406d67fc Fix journal space_check incorrectly checking for space at the beginning 2024-02-20 19:40:56 +03:00
Vitaliy Filippov f87964861d Release 1.4.6
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_rm (push) Successful in 18s Details
Test / test_move_reappear (push) Successful in 26s Details
Test / test_snapshot_down (push) Successful in 28s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m3s Details
Test / test_snapshot_chain_ec (push) Successful in 2m46s Details
Test / test_rebalance_verify_imm (push) Successful in 3m1s Details
Test / test_rebalance_verify (push) Successful in 3m30s Details
Test / test_switch_primary (push) Successful in 38s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_rebalance_verify_ec (push) Successful in 4m38s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m57s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m14s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m21s Details
Test / test_heal_csum_32k (push) Successful in 5m45s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m27s Details
Test / test_scrub (push) Successful in 1m30s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m26s Details
Test / test_scrub_zero_osd_2 (push) Successful in 38s Details
Test / test_scrub_xor (push) Successful in 40s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m8s Details
Test / test_scrub_ec (push) Successful in 1m5s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m49s Details
Test / test_heal_csum_4k (push) Successful in 5m41s Details
Test / test_heal_ec (push) Successful in 4m11s Details
Test / test_heal_pg_size_2 (push) Successful in 4m22s Details
Unwavering stabilization of 1.4.x, continued :-)

- Include the accidentally lost part of 1.4.5 journal trimming fix
- Fix a possible OSD crash with "BUG: Attempt to overwrite used offset"
  which was probably present for long time, but became apparent after
  fixing flapping tests in CI
- Fix remaining flapping tests in CI. It was the first time when tests
  actually passed without retries :-)
2024-02-20 17:01:26 +03:00
Vitaliy Filippov 62a4f45160 Raise test_scrub waiting timeout
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_rm (push) Successful in 19s Details
Test / test_move_reappear (push) Successful in 25s Details
Test / test_snapshot_down (push) Successful in 28s Details
Test / test_snapshot_down_ec (push) Successful in 33s Details
Test / test_splitbrain (push) Successful in 28s Details
Test / test_snapshot_chain (push) Successful in 2m17s Details
Test / test_snapshot_chain_ec (push) Successful in 3m5s Details
Test / test_rebalance_verify_imm (push) Successful in 3m0s Details
Test / test_rebalance_verify (push) Successful in 3m43s Details
Test / test_switch_primary (push) Successful in 40s Details
Test / test_write (push) Successful in 41s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 42s Details
Test / test_rebalance_verify_ec (push) Successful in 3m55s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m6s Details
Test / test_heal_pg_size_2 (push) Successful in 4m51s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m47s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m50s Details
Test / test_heal_csum_32k (push) Successful in 5m42s Details
Test / test_heal_ec (push) Failing after 10m30s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m22s Details
Test / test_scrub (push) Successful in 1m21s Details
Test / test_scrub_xor (push) Successful in 46s Details
Test / test_scrub_zero_osd_2 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m21s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m56s Details
Test / test_scrub_ec (push) Successful in 55s Details
Test / test_heal_csum_4k (push) Successful in 4m28s Details
Test / test_heal_csum_4k_dj (push) Failing after 10m15s Details
2024-02-20 16:26:09 +03:00
Vitaliy Filippov 7048228678 Supposed fix for "BUG: Attempt to overwrite used offset" 2024-02-20 15:56:48 +03:00
Vitaliy Filippov ea73857450 Add asserts to catch "BUG: Attempt to overwrite used offset" 2024-02-20 15:56:48 +03:00
Vitaliy Filippov 6cfe38ec04 Followup to empty cur.oid as stop condition for forced trim fix 2024-02-20 15:56:38 +03:00
Vitaliy Filippov 7ae5766fdb Wait to clear has_degraded in test_heal - should fix flaps of test_heal_* in CI 2024-02-20 15:56:27 +03:00
Vitaliy Filippov f882c7dd87 Release 1.4.5
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m23s Details
Test / test_rm (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 29s Details
Test / test_snapshot_chain (push) Successful in 2m17s Details
Test / test_snapshot_chain_ec (push) Successful in 3m14s Details
Test / test_rebalance_verify_imm (push) Successful in 3m24s Details
Test / test_rebalance_verify (push) Successful in 3m59s Details
Test / test_switch_primary (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 32s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec (push) Successful in 3m46s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m52s Details
Test / test_heal_ec (push) Successful in 5m25s Details
Test / test_heal_csum_32k_dj (push) Successful in 4m24s Details
Test / test_heal_csum_4k_dmj (push) Successful in 4m23s Details
Test / test_heal_csum_4k_dj (push) Successful in 4m17s Details
Test / test_scrub (push) Successful in 38s Details
Test / test_scrub_zero_osd_2 (push) Successful in 29s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 43s Details
Test / test_scrub_ec (push) Successful in 32s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m46s Details
Test / test_heal_csum_4k (push) Successful in 4m4s Details
Test / test_write (push) Successful in 1m38s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m5s Details
Test / test_heal_csum_32k (push) Successful in 4m15s Details
- Fix a write stall caused by incorrect journal trimming introduced in 1.4.4 :)
- Fix PGs sometimes hanging in "starting" state on mass OSD restarts
- Fix a rare crash with "map::at" during OSD pings
- Use new defaults for non-capacitor (desktop) SSDs - improves T1Q256 random write from ~6k iops to ~45k iops
- Make journal_trim_interval configurable
2024-02-16 10:13:33 +03:00
Vitaliy Filippov 26dd863c8d Fix sometimes possible crash on clients.at() during pings 2024-02-16 10:13:33 +03:00
Vitaliy Filippov 2ae859fbc6 Use min/max_flusher_count=32/256, 128M journal and autosync_writes=512 for non-capacitor SSDs by default 2024-02-16 10:13:33 +03:00
Vitaliy Filippov f6cd9f9153 Add a note about pg_minsize 2024-02-15 23:38:52 +03:00
Vitaliy Filippov 8389c0f33b Fix PGs sometimes hanging in "starting" state on mass OSD restarts 2024-02-15 23:38:52 +03:00
Vitaliy Filippov 9db2196aef Make journal_trim_interval configurable 2024-02-15 23:38:51 +03:00
Vitaliy Filippov 8d6ae662fe Use empty cur.oid as stop condition for forced trim, not journal_trim_counter 2024-02-15 23:27:17 +03:00
Vitaliy Filippov c777a0041a Release 1.4.4
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m23s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_snapshot_chain (push) Successful in 2m18s Details
Test / test_snapshot_chain_ec (push) Successful in 3m13s Details
Test / test_rebalance_verify_imm (push) Successful in 3m8s Details
Test / test_rebalance_verify (push) Successful in 3m41s Details
Test / test_switch_primary (push) Successful in 36s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_write_xor (push) Successful in 39s Details
Test / test_rebalance_verify_ec (push) Successful in 4m56s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m21s Details
Test / test_heal_pg_size_2 (push) Successful in 4m15s Details
Test / test_heal_ec (push) Successful in 5m1s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m32s Details
Test / test_heal_csum_32k (push) Successful in 5m38s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m43s Details
Test / test_scrub (push) Successful in 1m31s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m17s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m57s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m7s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 41s Details
Test / test_scrub_ec (push) Successful in 24s Details
Test / test_heal_csum_32k_dmj (push) Successful in 3m56s Details
Test / test_heal_csum_4k (push) Successful in 3m16s Details
A couple of fixes for EC pools

- Fix a segfault possible on partial EC overwrite in 1234 -> 5030 rebalance scenario
- Fix two problems leading to EC pools stalling on rebalance & parallel sudden stops
  of OSDs, for example during a sudden poweroff of a host:
  - Recovery auto-tuning (1.4.0 feature) could apply too large delays and stall
    the EC journal - fixed by limiting delays with a new recovery_tune_sleep_cutoff_us
    parameter (10 seconds by default) and applying recovery pauses before write
    operations, not after them, to not occupy space in the journal for long time
  - Dynamic journal space reservation (1.3.0 feature) wasn't accounting new writes
    when checking the limit so OSDs could still fill the journal fully and stall -
    fixed by including new writes into the limit
- Print etcd dbSize instead of dbSizeInUse in status
2024-02-11 16:23:08 +03:00
Vitaliy Filippov 2947ea93e8 Raise test_snapshot_chain_ec timeout to 6 minutes 2024-02-11 16:13:52 +03:00
Vitaliy Filippov 978bdc128a Apply recovery pause before writes, after commits, and do not apply it to syncs to not block EC pools from functioning 2024-02-11 16:13:52 +03:00
Vitaliy Filippov bb2f395f1e Add cutoff threshold for recovery auto-tuning 2024-02-11 16:13:52 +03:00
Vitaliy Filippov b127da40f7 Add a FIXME about incomplete PGs 2024-02-11 13:42:51 +03:00
Vitaliy Filippov ca34a6047a Fix dynamic journal space reservation: include the new write itself, too 2024-02-11 13:42:51 +03:00
Vitaliy Filippov 38ba76e893 Fix flusher sometimes being unable to trim journal when the flush queue is empty 2024-02-11 13:42:51 +03:00
Vitaliy Filippov 1e3c4edea0 Print etcd dbSize instead of dbSizeInUse in status 2024-02-11 13:42:51 +03:00
Vitaliy Filippov e7ac855b07 Fix that EC segfault (1234 -> 5030 partial overwrite) 2024-02-11 13:42:51 +03:00
Vitaliy Filippov c53357ac45 Add a test for EC segfault with partial overwrite in 1234 -> 5030 rebalance scenario 2024-02-11 13:42:51 +03:00
Vitaliy Filippov 27e9f244ec Release 1.4.3
Test / test_move_reappear (push) Successful in 22s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Successful in 36s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_interrupted_rebalance (push) Successful in 5m3s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_snapshot_chain (push) Successful in 3m1s Details
Test / test_snapshot_chain_ec (push) Successful in 3m13s Details
Test / test_rebalance_verify_imm (push) Successful in 3m0s Details
Test / test_rebalance_verify (push) Successful in 3m29s Details
Test / test_switch_primary (push) Successful in 37s Details
Test / test_write (push) Successful in 44s Details
Test / test_write_xor (push) Successful in 39s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m13s Details
Test / test_rebalance_verify_ec (push) Successful in 5m31s Details
Test / test_heal_ec (push) Successful in 4m54s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m25s Details
Test / test_heal_csum_32k (push) Successful in 6m8s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m17s Details
Test / test_scrub (push) Successful in 1m8s Details
Test / test_scrub_zero_osd_2 (push) Successful in 55s Details
Test / test_scrub_xor (push) Successful in 45s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m22s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m11s Details
Test / test_scrub_ec (push) Successful in 46s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m39s Details
Test / test_heal_csum_4k (push) Successful in 6m8s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m41s Details
Hotfix for hotfix O:-)

- "Write stall fix" was incomplete and EC write stalls could
  continue even on 1.4.2. Now they're finally fixed O:-)
- Make monitor ignore statistics of stopped OSDs. Previously if you stopped all
  OSDs the last total I/O numbers would remain the same indefinitely
2024-02-09 00:29:31 +03:00
Vitaliy Filippov 8e25a28a08 Ignore down OSDs in monitor statistics aggregation
Test / test_move_reappear (push) Successful in 20s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m50s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 18s Details
Test / test_snapshot_chain (push) Successful in 2m10s Details
Test / test_snapshot_chain_ec (push) Failing after 3m8s Details
Test / test_rebalance_verify (push) Successful in 3m6s Details
Test / test_interrupted_rebalance (push) Failing after 10m52s Details
Test / test_rebalance_verify_imm (push) Successful in 5m28s Details
Test / test_switch_primary (push) Successful in 37s Details
Test / test_write (push) Successful in 42s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_rebalance_verify_ec (push) Successful in 6m7s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m3s Details
Test / test_heal_pg_size_2 (push) Successful in 4m12s Details
Test / test_heal_ec (push) Successful in 5m20s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m53s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m23s Details
Test / test_heal_csum_32k (push) Successful in 5m59s Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
2024-02-09 00:22:36 +03:00
Vitaliy Filippov 5d3317e4f2 Followup to 1.4.2 write stall fix - sadly, the previous version was not working correctly :)
Test / test_move_reappear (push) Successful in 19s Details
Test / test_snapshot_chain (push) Successful in 1m21s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_chain_ec (push) Successful in 1m50s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_etcd_fail (push) Successful in 6m42s Details
Test / test_rebalance_verify_imm (push) Successful in 2m19s Details
Test / test_rebalance_verify (push) Successful in 4m7s Details
Test / test_switch_primary (push) Successful in 36s Details
Test / test_write (push) Successful in 35s Details
Test / test_rebalance_verify_ec (push) Successful in 4m6s Details
Test / test_write_no_same (push) Successful in 22s Details
Test / test_write_xor (push) Successful in 1m34s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m7s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m7s Details
Test / test_heal_csum_32k_dj (push) Successful in 4m59s Details
Test / test_heal_csum_32k (push) Successful in 5m4s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m59s Details
Test / test_scrub (push) Successful in 1m9s Details
Test / test_scrub_zero_osd_2 (push) Successful in 37s Details
Test / test_scrub_xor (push) Successful in 52s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m5s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m12s Details
Test / test_heal_csum_4k (push) Successful in 5m1s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m48s Details
Test / test_scrub_ec (push) Successful in 19s Details
Test / test_interrupted_rebalance (push) Successful in 1m38s Details
Test / test_heal_pg_size_2 (push) Successful in 3m20s Details
Test / test_heal_ec (push) Successful in 3m3s Details
2024-02-08 19:34:29 +03:00
Vitaliy Filippov 016115c0d4 Release 1.4.2
Test / test_rm (push) Successful in 16s Details
Test / test_move_reappear (push) Successful in 20s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_snapshot_down_ec (push) Successful in 39s Details
Test / test_interrupted_rebalance (push) Successful in 4m52s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_snapshot_chain (push) Successful in 3m11s Details
Test / test_rebalance_verify_imm (push) Successful in 3m16s Details
Test / test_rebalance_verify (push) Successful in 3m45s Details
Test / test_switch_primary (push) Successful in 36s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_xor (push) Successful in 40s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m8s Details
Test / test_rebalance_verify_ec (push) Successful in 5m57s Details
Test / test_heal_pg_size_2 (push) Successful in 4m22s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m20s Details
Test / test_heal_ec (push) Successful in 5m54s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m24s Details
Test / test_heal_csum_32k (push) Successful in 6m3s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m54s Details
Test / test_scrub_zero_osd_2 (push) Successful in 53s Details
Test / test_scrub (push) Successful in 55s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m14s Details
Test / test_scrub_xor (push) Successful in 1m1s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m50s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 57s Details
Test / test_scrub_ec (push) Successful in 52s Details
Test / test_heal_csum_4k (push) Successful in 5m47s Details
Test / test_snapshot_chain_ec (push) Successful in 1m24s Details
- Log to systemd by default
- Fix excessive autosyncs after every operation with disabled immediate_commit (introduced in 1.1.0)
- Fix a possible write stall with EC due to the lack of OSD wakeup after stabilizing previous writes
- Change sync operation semantics as a final fix to possible write stalls with EC and disabled immediate_commit
- Sync after deleting data in CLI rm / rm-data if immediate_commit is disabled
- Fix OSDs ignoring syncs & autosyncs for delete operations
- Fix OSD space reporting sometimes adding garbage zeros for deleted inodes (causing extra pool/stats etcd keys for deleted pools)
- Speed up monitor failover - change default etcd_mon_ttl from 30 to 5 seconds
- Speed up operation retries - change default up_wait_retry_interval to 50 ms
- Add patch for libvirt 9.10
2024-02-04 02:23:49 +03:00
Vitaliy Filippov e026de95d5 Log to systemd by default
Test / test_move_reappear (push) Successful in 20s Details
Test / test_etcd_fail (push) Successful in 5m19s Details
Test / test_snapshot_chain (push) Successful in 1m26s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_snapshot_down_ec (push) Successful in 28s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_snapshot_chain_ec (push) Failing after 3m8s Details
Test / test_interrupted_rebalance (push) Successful in 7m44s Details
Test / test_rebalance_verify_imm (push) Successful in 3m11s Details
Test / test_switch_primary (push) Successful in 34s Details
Test / test_write (push) Successful in 34s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m41s Details
Test / test_rebalance_verify_ec (push) Successful in 3m18s Details
Test / test_write_no_same (push) Successful in 22s Details
Test / test_write_xor (push) Successful in 1m41s Details
Test / test_heal_pg_size_2 (push) Failing after 3m54s Details
Test / test_rebalance_verify (push) Successful in 9m38s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m4s Details
Test / test_heal_csum_32k_dj (push) Successful in 4m23s Details
Test / test_heal_csum_32k (push) Successful in 5m24s Details
Test / test_heal_ec (push) Failing after 10m18s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m31s Details
Test / test_scrub (push) Successful in 1m18s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m0s Details
Test / test_scrub_xor (push) Successful in 51s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m10s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 48s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m59s Details
Test / test_scrub_ec (push) Successful in 48s Details
Test / test_heal_csum_4k (push) Successful in 4m39s Details
2024-02-04 01:21:31 +03:00
Vitaliy Filippov 77c10fd1f8 In fact, do not autosync blockstore when autosync_writes=0
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 17s Details
Test / test_snapshot_chain (push) Successful in 1m57s Details
Test / test_snapshot_chain_ec (push) Successful in 2m41s Details
Test / test_rebalance_verify (push) Successful in 3m5s Details
Test / test_rebalance_verify_imm (push) Successful in 2m26s Details
Test / test_switch_primary (push) Successful in 45s Details
Test / test_write (push) Successful in 33s Details
Test / test_write_xor (push) Successful in 33s Details
Test / test_rebalance_verify_ec (push) Successful in 3m42s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m57s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m24s Details
Test / test_heal_csum_32k_dj (push) Successful in 4m29s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m10s Details
Test / test_heal_csum_32k (push) Successful in 5m13s Details
Test / test_scrub (push) Successful in 1m5s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m1s Details
Test / test_scrub_xor (push) Successful in 1m2s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m2s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 57s Details
Test / test_scrub_ec (push) Successful in 50s Details
Test / test_scrub_pg_size_3 (push) Successful in 2m1s Details
Test / test_heal_csum_4k (push) Successful in 4m40s Details
Test / test_interrupted_rebalance (push) Successful in 1m38s Details
Test / test_heal_pg_size_2 (push) Successful in 4m2s Details
Test / test_heal_ec (push) Successful in 5m17s Details
2024-02-03 20:37:36 +03:00
Vitaliy Filippov 581d02e581 Mark secondary OSDs with deletions as dirty to not forget to sync & autosync them
Test / test_change_pg_count (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot_chain (push) Has been cancelled Details
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_switch_primary (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_cas (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
2024-02-03 20:31:08 +03:00
Vitaliy Filippov f03a9db4d9 Fix OSD space reporting sometimes adding garbage zeros for deleted inodes (causing extra pool/stats etcd keys for deleted pools) 2024-02-03 20:31:08 +03:00
Vitaliy Filippov cb9c30bc31 Sync after sending all deletes to each PG in cli rm-data 2024-02-03 20:31:08 +03:00
Vitaliy Filippov a86a380d20 Fix invalid parsing of autosync_writes in blockstore leading to autosyncs after every operation with disabled immediate_commit :D 2024-02-03 20:31:08 +03:00
Vitaliy Filippov d2b43cb118 Change default etcd_mon_ttl
Test / test_move_reappear (push) Successful in 35s Details
Test / test_interrupted_rebalance_ec (push) Successful in 3m29s Details
Test / test_interrupted_rebalance (push) Successful in 4m47s Details
Test / test_snapshot_down (push) Successful in 29s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m46s Details
Test / test_snapshot_chain_ec (push) Failing after 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 4m24s Details
Test / test_rebalance_verify (push) Successful in 4m54s Details
Test / test_switch_primary (push) Successful in 35s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m38s Details
Test / test_write (push) Successful in 46s Details
Test / test_write_xor (push) Successful in 49s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_rebalance_verify_ec (push) Successful in 7m14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m10s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m10s Details
Test / test_heal_csum_32k_dj (push) Successful in 4m52s Details
Test / test_heal_csum_32k (push) Successful in 5m20s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m8s Details
Test / test_heal_ec (push) Failing after 10m21s Details
Test / test_scrub (push) Successful in 1m2s Details
Test / test_scrub_xor (push) Successful in 54s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m4s Details
Test / test_heal_csum_4k_dj (push) Successful in 4m48s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m26s Details
Test / test_scrub_ec (push) Successful in 50s Details
Test / test_scrub_pg_size_3 (push) Failing after 2m5s Details
Test / test_heal_csum_4k (push) Successful in 4m33s Details
2024-01-29 23:45:19 +03:00
Vitaliy Filippov cc76e6876b Fix flapping "scrub" test
Test / test_rm (push) Successful in 16s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m53s Details
Test / test_snapshot_down (push) Successful in 29s Details
Test / test_snapshot_down_ec (push) Successful in 38s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_interrupted_rebalance (push) Successful in 5m46s Details
Test / test_snapshot_chain (push) Successful in 2m59s Details
Test / test_rebalance_verify_imm (push) Successful in 2m36s Details
Test / test_rebalance_verify (push) Successful in 3m22s Details
Test / test_switch_primary (push) Successful in 36s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m46s Details
Test / test_write_xor (push) Successful in 40s Details
Test / test_rebalance_verify_ec (push) Successful in 4m46s Details
Test / test_heal_pg_size_2 (push) Successful in 4m23s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m40s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m45s Details
Test / test_scrub (push) Successful in 1m1s Details
Test / test_scrub_zero_osd_2 (push) Successful in 43s Details
Test / test_scrub_xor (push) Successful in 35s Details
Test / test_heal_csum_4k (push) Successful in 4m14s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m19s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s Details
Test / test_scrub_ec (push) Successful in 21s Details
Test / test_heal_csum_32k (push) Successful in 4m48s Details
Test / test_heal_csum_4k_dmj (push) Successful in 4m48s Details
Test / test_heal_csum_4k_dj (push) Successful in 4m27s Details
Test / test_snapshot_chain_ec (push) Successful in 2m29s Details
Test / test_heal_ec (push) Successful in 3m7s Details
2024-01-28 14:59:33 +03:00
Vitaliy Filippov 1cec62d25d Sync only completed writes
Test / test_move_reappear (push) Successful in 21s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_snapshot_down_ec (push) Successful in 35s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_interrupted_rebalance (push) Successful in 5m14s Details
Test / test_snapshot_chain (push) Successful in 2m50s Details
Test / test_rebalance_verify_imm (push) Successful in 2m47s Details
Test / test_rebalance_verify (push) Successful in 3m42s Details
Test / test_switch_primary (push) Successful in 33s Details
Test / test_write (push) Successful in 42s Details
Test / test_write_xor (push) Successful in 44s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m52s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify_ec (push) Successful in 4m19s Details
Test / test_heal_ec (push) Successful in 6m20s Details
Test / test_heal_csum_32k (push) Successful in 3m29s Details
Test / test_scrub (push) Successful in 1m24s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m11s Details
Test / test_heal_csum_4k_dmj (push) Successful in 4m23s Details
Test / test_scrub_xor (push) Successful in 1m9s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m29s Details
Test / test_heal_csum_4k (push) Successful in 5m36s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m53s Details
Test / test_scrub_ec (push) Successful in 29s Details
Test / test_heal_pg_size_2 (push) Successful in 3m9s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m13s Details
Test / test_heal_csum_32k_dj (push) Successful in 4m17s Details
Test / test_snapshot_chain_ec (push) Successful in 1m25s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 24s Details
Should be a final remaining fix to EC + non-capacitor (non-immediate-commit) write hangs :).

First it was breaking non-EC ("instantly stable") writes because they sometimes
complete out of order which was leading to the following error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  BUG: Unexpected dirty_entry 1000000000001:29480000 v65540 unstable state during flush: 0x151

But it is easily fixed by scanning previous and next dirty_entries in mark_stable.
2024-01-27 15:17:22 +03:00
Vitaliy Filippov 1c322b33ed Change default up_wait_retry_interval to 50 ms
Test / test_rm (push) Successful in 14s Details
Test / test_interrupted_rebalance_ec (push) Successful in 3m59s Details
Test / test_snapshot_chain (push) Successful in 1m34s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_snapshot_down_ec (push) Successful in 29s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_snapshot_chain_ec (push) Successful in 2m35s Details
Test / test_interrupted_rebalance (push) Successful in 8m15s Details
Test / test_rebalance_verify_imm (push) Successful in 3m54s Details
Test / test_switch_primary (push) Successful in 36s Details
Test / test_write (push) Successful in 35s Details
Test / test_rebalance_verify_ec (push) Successful in 4m48s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m51s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Failing after 3m9s Details
Test / test_heal_pg_size_2 (push) Successful in 3m55s Details
Test / test_heal_ec (push) Successful in 3m50s Details
Test / test_rebalance_verify (push) Failing after 9m30s Details
Test / test_heal_csum_32k_dmj (push) Failing after 5m40s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m12s Details
Test / test_heal_csum_32k (push) Successful in 6m25s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m56s Details
Test / test_scrub (push) Successful in 1m4s Details
Test / test_scrub_zero_osd_2 (push) Successful in 55s Details
Test / test_scrub_xor (push) Successful in 56s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m19s Details
Test / test_scrub_pg_size_3 (push) Failing after 2m14s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m53s Details
Test / test_scrub_ec (push) Successful in 1m1s Details
Test / test_heal_csum_4k (push) Successful in 5m17s Details
2024-01-26 01:51:08 +03:00
Vitaliy Filippov d27524f441 Add patch for libvirt 9.10 2024-01-25 01:09:12 +03:00
Vitaliy Filippov ba55f91409 Release 1.4.1
Test / test_move_reappear (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 1m27s Details
Test / test_interrupted_rebalance_ec (push) Successful in 4m41s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_snapshot_chain_ec (push) Successful in 2m0s Details
Test / test_splitbrain (push) Successful in 18s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_rebalance_verify_ec (push) Failing after 2m21s Details
Test / test_rebalance_verify_imm (push) Successful in 2m30s Details
Test / test_switch_primary (push) Successful in 39s Details
Test / test_write (push) Successful in 35s Details
Test / test_interrupted_rebalance (push) Failing after 10m8s Details
Test / test_write_xor (push) Successful in 36s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m4s Details
Test / test_heal_pg_size_2 (push) Successful in 3m55s Details
Test / test_rebalance_verify (push) Successful in 8m31s Details
Test / test_heal_ec (push) Successful in 5m9s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m27s Details
Test / test_heal_csum_32k (push) Successful in 5m42s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m1s Details
Test / test_scrub (push) Successful in 59s Details
Test / test_scrub_zero_osd_2 (push) Successful in 38s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m5s Details
Test / test_scrub_xor (push) Successful in 58s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m25s Details
Test / test_scrub_ec (push) Failing after 42s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m32s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m38s Details
Test / test_heal_csum_4k (push) Successful in 5m38s Details
- Fix a monitor crash on primary OSD switching introduced in 1.4.0
- Fix "partly outside array bounds" warnings for GCC 12 in cpp-btree
- Fix a realloc memory leak in theory possible with too large listings (OSD_OP_LIST)
2024-01-18 02:31:42 +03:00
Vitaliy Filippov 80aac39513 Add detailed formula for theoretical EC N+K random write performance 2024-01-18 00:36:32 +03:00
Vitaliy Filippov 2aa5aa7ab6 Add a test for simple master switching without PG reconfiguration
Test / test_move_reappear (push) Successful in 20s Details
Test / test_snapshot_chain (push) Successful in 1m27s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_chain_ec (push) Successful in 1m56s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 17s Details
Test / test_interrupted_rebalance_ec (push) Successful in 6m40s Details
Test / test_interrupted_rebalance (push) Successful in 8m12s Details
Test / test_rebalance_verify_imm (push) Successful in 3m12s Details
Test / test_switch_primary (push) Successful in 34s Details
Test / test_write (push) Successful in 46s Details
Test / test_rebalance_verify_ec (push) Successful in 3m18s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m42s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 6m36s Details
Test / test_heal_ec (push) Successful in 5m2s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m33s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m58s Details
Test / test_heal_csum_32k (push) Successful in 6m6s Details
Test / test_scrub (push) Successful in 47s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m17s Details
Test / test_scrub_zero_osd_2 (push) Successful in 43s Details
Test / test_scrub_xor (push) Successful in 47s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m44s Details
Test / test_scrub_ec (push) Successful in 41s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m18s Details
Test / test_scrub_pg_size_3 (push) Successful in 2m11s Details
Test / test_heal_csum_4k (push) Successful in 6m12s Details
Test / test_heal_pg_size_2 (push) Successful in 3m16s Details
Test / test_write_xor (push) Successful in 34s Details
Also use osd_out_time:1 only in select tests and restart mon in tests only on connection errors
2024-01-17 00:19:01 +03:00
Vitaliy Filippov 3ca3b8a8d8 Fix recheck_pgs bug introduced in 1.4.0
Test / test_rm (push) Successful in 14s Details
Test / test_interrupted_rebalance_ec (push) Successful in 3m27s Details
Test / test_snapshot_chain (push) Successful in 1m24s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_snapshot_chain_ec (push) Successful in 1m54s Details
Test / test_snapshot_down_ec (push) Successful in 20s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify_imm (push) Successful in 2m42s Details
Test / test_etcd_fail (push) Failing after 10m8s Details
Test / test_interrupted_rebalance (push) Failing after 10m9s Details
Test / test_write (push) Successful in 1m22s Details
Test / test_rebalance_verify_ec (push) Failing after 1m51s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m27s Details
Test / test_write_xor (push) Failing after 3m13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m22s Details
Test / test_rebalance_verify (push) Failing after 10m9s Details
Test / test_heal_ec (push) Successful in 4m41s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m42s Details
Test / test_heal_csum_32k_dj (push) Successful in 4m58s Details
Test / test_heal_csum_32k (push) Successful in 6m34s Details
Test / test_scrub (push) Successful in 54s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m56s Details
Test / test_scrub_zero_osd_2 (push) Successful in 49s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m1s Details
Test / test_scrub_ec (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
2024-01-16 23:49:21 +03:00
Vitaliy Filippov 2cf649eba6 Fix "partly outside array bounds" warnings for GCC 12 in cpp-btree
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_snapshot_chain (push) Successful in 2m13s Details
Test / test_interrupted_rebalance (push) Successful in 7m36s Details
Test / test_rebalance_verify (push) Successful in 3m35s Details
Test / test_write (push) Successful in 1m0s Details
Test / test_rebalance_verify_imm (push) Successful in 4m4s Details
Test / test_rebalance_verify_ec (push) Successful in 3m13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m35s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m32s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m29s Details
Test / test_heal_ec (push) Successful in 5m47s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m47s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m4s Details
Test / test_heal_csum_32k (push) Successful in 6m19s Details
Test / test_scrub (push) Successful in 56s Details
Test / test_scrub_zero_osd_2 (push) Successful in 43s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m14s Details
Test / test_scrub_xor (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 57s Details
Test / test_scrub_ec (push) Successful in 47s Details
Test / test_heal_csum_4k (push) Successful in 5m56s Details
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_scrub_pg_size_3 (push) Successful in 46s Details
Test / test_snapshot_chain_ec (push) Successful in 1m40s Details
Test / test_write_xor (push) Failing after 3m6s Details
2024-01-15 03:04:33 +03:00
Vitaliy Filippov 5935640a4a Add CLA PR form 2024-01-14 16:48:24 +03:00
Vitaliy Filippov d00d4dbac0 Initialize mod_revision field in etcd_state_client
Test / test_interrupted_rebalance_ec (push) Successful in 2m28s Details
Test / test_rm (push) Successful in 17s Details
Test / test_move_reappear (push) Successful in 29s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_snapshot_down_ec (push) Successful in 26s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_snapshot_chain (push) Successful in 2m0s Details
Test / test_rebalance_verify_imm (push) Successful in 2m28s Details
Test / test_rebalance_verify (push) Successful in 3m0s Details
Test / test_rebalance_verify_ec (push) Successful in 3m14s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m7s Details
Test / test_heal_pg_size_2 (push) Successful in 3m33s Details
Test / test_heal_ec (push) Successful in 4m40s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m40s Details
Test / test_heal_csum_32k (push) Successful in 6m8s Details
Test / test_scrub (push) Successful in 1m4s Details
Test / test_scrub_zero_osd_2 (push) Successful in 47s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m33s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m28s Details
Test / test_scrub_xor (push) Successful in 44s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m2s Details
Test / test_scrub_ec (push) Successful in 42s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m38s Details
Test / test_heal_csum_4k (push) Successful in 5m56s Details
Test / test_interrupted_rebalance (push) Successful in 1m53s Details
Test / test_snapshot_chain_ec (push) Failing after 3m17s Details
Test / test_write (push) Failing after 3m15s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m6s Details
Test / test_write_xor (push) Failing after 3m11s Details
2024-01-13 01:30:28 +03:00
Vitaliy Filippov 5d9d6f32a0 Fix common realloc memory leak mistakes found by cppcheck 2024-01-13 01:30:28 +03:00
Vitaliy Filippov 5280d1d561 Release 1.4.0
Test / test_snapshot (push) Successful in 26s Details
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_rm (push) Successful in 16s Details
Test / test_move_reappear (push) Successful in 24s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 28s Details
Test / test_snapshot_chain (push) Successful in 2m41s Details
Test / test_rebalance_verify_imm (push) Successful in 2m48s Details
Test / test_rebalance_verify (push) Successful in 3m28s Details
Test / test_write (push) Successful in 47s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m5s Details
Test / test_rebalance_verify_ec (push) Successful in 3m41s Details
Test / test_heal_pg_size_2 (push) Successful in 3m45s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m52s Details
Test / test_heal_ec (push) Successful in 5m11s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m42s Details
Test / test_heal_csum_32k (push) Successful in 5m56s Details
Test / test_scrub (push) Successful in 1m25s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m18s Details
Test / test_scrub_xor (push) Successful in 42s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m49s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m32s Details
Test / test_heal_csum_4k (push) Successful in 5m31s Details
Test / test_scrub_ec (push) Successful in 50s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m2s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m5s Details
Test / test_snapshot_chain_ec (push) Successful in 1m21s Details
Test / test_write_xor (push) Successful in 36s Details
New features:
- Intelligent recovery/rebalance speed auto-tuning to reduce its impact on clients (see README -> Features)
- Auto-restoration of dead VDUSE daemons in CSI plugin
- Add vitastor-disk update-sb command
- Update QEMU for Debian Bookworm to 8.1 and use it for CSI plugin

Bug fixes:
- Fix pools SOMETIMES staying inactive after stopping a node due to OSDs not reacting
  to PG state changes caused by incorrect full reload of state from etcd on reconnection
- Make monitors retry pool configuration changes quickier which fixes them being unable
  to apply changes when an ongoing rebalance is quickly making a lot of PGs clean
- Fix CSI plugin not accepting array of strings as etcd address in /etc/vitastor/vitastor.conf
- Allow multiple interfaces with the same IP address, for "simple routed" full mesh network
- Do not ignore loopback addresses for OSD network (to make ECMP setups with frr possible)
- Fix a rare client crash during OSD reconnections
- Only treat data partitions as existing OSDs in vitastor-disk prepare
- Remove etcd parameter from default command examples
- Fix reported free space sometimes changing non-immediately after deletion of data from OSDs
- Fix a possible OSD crash on print_slow when bs_op is NULL
- Use the same etcd_ws_keepalive_interval in mon as in OSD
- Fix mon not using values from config when /config/global is not present
- Remove pve-storage-portal-dns-list format for vitastor_etcd_address
- Parse log_level in cluster_client
- Fix vitastor-nbd image existence check not working because of non-zeroed inode_watch fields
- Do not warn on EPIPE in client unless log_level is raised explicitly
- Fix incorrect error in CSI when searching for the device in /sys
- Remove 2 last prints to stdout in etcd_state_client
- Fix a possible OSD crash when checking corrupted journal entries
2024-01-12 01:28:33 +03:00
Vitaliy Filippov 317b0feb0a Add a note about VDUSE daemon auto-restart 2024-01-12 01:27:36 +03:00
Vitaliy Filippov 247f0552db Fix debug log "killing..." in CSI 2024-01-10 01:19:34 +03:00
Vitaliy Filippov 2f228fa96a Only treat data partitions as existing OSDs in vitastor-disk prepare
Test / test_interrupted_rebalance_ec (push) Successful in 2m40s Details
Test / test_rm (push) Successful in 31s Details
Test / test_move_reappear (push) Successful in 39s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_interrupted_rebalance (push) Successful in 4m42s Details
Test / test_snapshot_down_ec (push) Successful in 26s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Failing after 3m17s Details
Test / test_snapshot_chain_ec (push) Failing after 3m13s Details
Test / test_rebalance_verify_imm (push) Successful in 2m51s Details
Test / test_write (push) Successful in 37s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m37s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_rebalance_verify_ec (push) Successful in 3m20s Details
Test / test_write_xor (push) Failing after 3m8s Details
Test / test_rebalance_verify (push) Successful in 8m20s Details
Test / test_heal_pg_size_2 (push) Successful in 4m17s Details
Test / test_heal_ec (push) Successful in 4m59s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m15s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m35s Details
Test / test_heal_csum_32k (push) Successful in 6m47s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m49s Details
Test / test_scrub (push) Successful in 1m2s Details
Test / test_scrub_zero_osd_2 (push) Successful in 45s Details
Test / test_scrub_xor (push) Successful in 40s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m16s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m9s Details
Test / test_scrub_ec (push) Successful in 45s Details
Test / test_heal_csum_4k (push) Successful in 5m26s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m38s Details
2023-12-31 11:46:47 +03:00
Vitaliy Filippov 2f6b9c0306 Remove etcd parameter from default command examples 2023-12-31 02:50:41 +03:00
Vitaliy Filippov 48b5f871e0 Add Contributor License Aggrement in Russian and English 2023-12-31 01:23:52 +03:00
Vitaliy Filippov c17f76a3e4 Add documentation for recovery auto-tuning
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_snapshot_down_ec (push) Successful in 26s Details
Test / test_snapshot_chain (push) Successful in 1m50s Details
Test / test_splitbrain (push) Successful in 52s Details
Test / test_snapshot_chain_ec (push) Successful in 2m31s Details
Test / test_rebalance_verify_imm (push) Successful in 2m28s Details
Test / test_rebalance_verify (push) Successful in 3m25s Details
Test / test_rebalance_verify_ec (push) Successful in 3m31s Details
Test / test_write (push) Successful in 1m17s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m36s Details
Test / test_heal_pg_size_2 (push) Successful in 4m12s Details
Test / test_heal_ec (push) Successful in 5m20s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m36s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m11s Details
Test / test_heal_csum_32k (push) Successful in 6m13s Details
Test / test_scrub (push) Successful in 56s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m6s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m31s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m58s Details
Test / test_scrub_xor (push) Successful in 43s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m10s Details
Test / test_scrub_ec (push) Successful in 49s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m40s Details
Test / test_heal_csum_4k (push) Successful in 5m59s Details
Test / test_write_xor (push) Successful in 34s Details
Test / test_interrupted_rebalance (push) Successful in 1m19s Details
2023-12-31 01:23:17 +03:00
Vitaliy Filippov a6ab54b1ba Do not allow negative util_low/high 2023-12-31 01:23:17 +03:00
Vitaliy Filippov 99ee8596ea Rename min/max_util to util_low/high 2023-12-31 01:23:17 +03:00
Vitaliy Filippov c4928e6ecd Protect from try_send completing the operation immediately
Fixes a possible use-after-free in case of continue_ops() calling try_send(),
then connect_peer() -> set_timer() -> trigger_nearest() -> handle_op_part() -> continue_ops() again
2023-12-31 01:23:17 +03:00
Vitaliy Filippov ec7dcd1be5 Do not apply very large recovery pauses during tests 2023-12-31 01:23:17 +03:00
Vitaliy Filippov e600bbc151 Fix flapping move_reappear test by adding an fsync before stopping PG 2023-12-31 01:23:17 +03:00
Vitaliy Filippov 8b8c1179a7 Use a separate used_blocks counter for free space stats to hide possibly delayed on-flush deallocation 2023-12-31 01:23:17 +03:00
Vitaliy Filippov d5a6fa6dd7 Fix possible crash on print_slow when bs_op is NULL 2023-12-31 01:23:17 +03:00
Vitaliy Filippov f757a35a8d Retry PG changes without re-running lpsolve when pool configuration and OSD tree don't change
OSDs often change their /pg/history keys during rebalance, so monitor receives additional
transaction failures from etcd if it re-runs lpsolve which sometimes may even lead to monitor
being unable to apply PG changes at all until rebalance completes
2023-12-31 01:23:17 +03:00
Vitaliy Filippov 1edf86ed26 Aggregate recovery delay using simple mean over last 10 observations (EWMA is shit) 2023-12-31 01:23:17 +03:00
Vitaliy Filippov 5ca7cde612 Experiment/WIP: Try to track "secondary" recovery ops separately 2023-12-31 01:23:17 +03:00
Vitaliy Filippov 751935ddd8 WIP Auto-tune recovery speed 2023-12-31 01:23:17 +03:00
Vitaliy Filippov d84dee7098 Track recovery op latencies + refactor into a structure 2023-12-31 01:23:17 +03:00
Vitaliy Filippov dcc76eee15 Add a parity chunk count change test script 2023-12-26 23:48:41 +03:00
Vitaliy Filippov 2f38adeb3d Restart dead VDUSE daemons at regular intervals 2023-12-24 12:58:50 +03:00
Vitaliy Filippov f72f14e6a7 Clear old PG states, history, and OSD states on etcd state reload
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m24s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m24s Details
Test / test_snapshot_chain_ec (push) Successful in 3m5s Details
Test / test_rebalance_verify_imm (push) Successful in 3m21s Details
Test / test_write (push) Successful in 36s Details
Test / test_rebalance_verify (push) Successful in 4m12s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_write_xor (push) Successful in 52s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m29s Details
Test / test_rebalance_verify_ec (push) Successful in 5m25s Details
Test / test_heal_pg_size_2 (push) Successful in 4m10s Details
Test / test_heal_ec (push) Successful in 4m46s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m31s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m41s Details
Test / test_heal_csum_32k (push) Successful in 6m41s Details
Test / test_scrub (push) Successful in 1m13s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m53s Details
Test / test_scrub_xor (push) Successful in 54s Details
Test / test_scrub_zero_osd_2 (push) Successful in 58s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m27s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m15s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m27s Details
Test / test_heal_csum_4k (push) Successful in 6m20s Details
Test / test_scrub_ec (push) Successful in 29s Details
Test / test_move_reappear (push) Successful in 17s Details
Also add protection from etcd watcher messages being split into multiple websocket
messages - I'm not sure if etcd actually does that, but it's better to have extra
protection anyway.

Also check that all etcd watchers are started in the keepalive routine, otherwise
it sometimes tries to revive etcd watchers starting with revision=1 which obviously
always fails because this revision is nearly always compacted.

All these changes should fix an old rarely reproduced bug where SOMETIMES OSDs
didn't react to PG config changes which was leading to offline pools on node reboot.
It happened on the full reload of state from etcd.
2023-12-24 02:02:13 +03:00
Vitaliy Filippov 1299373988 Use the same etcd_ws_keepalive_interval in OSD and mon
Test / test_snapshot_ec (push) Successful in 33s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m58s Details
Test / test_move_reappear (push) Successful in 22s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 32s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_snapshot_chain (push) Successful in 2m36s Details
Test / test_snapshot_chain_ec (push) Failing after 3m8s Details
Test / test_rebalance_verify_imm (push) Successful in 2m58s Details
Test / test_rebalance_verify (push) Successful in 3m55s Details
Test / test_write (push) Successful in 39s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m18s Details
Test / test_rebalance_verify_ec (push) Successful in 4m8s Details
Test / test_write_xor (push) Failing after 3m11s Details
Test / test_heal_pg_size_2 (push) Successful in 3m47s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m58s Details
Test / test_heal_ec (push) Successful in 6m21s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m11s Details
Test / test_heal_csum_32k (push) Successful in 6m22s Details
Test / test_scrub (push) Successful in 1m17s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m17s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m35s Details
Test / test_scrub_xor (push) Successful in 57s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m27s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m3s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m33s Details
Test / test_scrub_ec (push) Successful in 44s Details
Test / test_heal_csum_4k (push) Successful in 6m9s Details
2023-12-23 20:07:29 +03:00
Vitaliy Filippov 178bb0e701 Prevent re-entry into timerfd set_nearest
Test / test_interrupted_rebalance_ec (push) Successful in 2m0s Details
Test / test_rm (push) Successful in 19s Details
Test / test_move_reappear (push) Successful in 23s Details
Test / test_snapshot_ec (push) Successful in 40s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 26s Details
Test / test_snapshot_chain (push) Successful in 2m32s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_rebalance_verify (push) Successful in 4m2s Details
Test / test_write (push) Successful in 39s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m25s Details
Test / test_rebalance_verify_ec (push) Successful in 4m19s Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m8s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m26s Details
Test / test_heal_csum_32k (push) Successful in 6m12s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m41s Details
Test / test_scrub (push) Successful in 1m17s Details
Test / test_scrub_zero_osd_2 (push) Successful in 57s Details
Test / test_scrub_xor (push) Successful in 53s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m36s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 59s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m36s Details
Test / test_scrub_ec (push) Successful in 31s Details
Test / test_heal_csum_4k (push) Successful in 5m23s Details
Test / test_write_xor (push) Successful in 1m4s Details
Test / test_heal_ec (push) Successful in 3m29s Details
Test / test_snapshot_chain_ec (push) Successful in 1m20s Details
2023-12-22 02:32:40 +03:00
Vitaliy Filippov 4ece4dfdd0 Fix mon not using values from config when /config/global is not present
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_rm (push) Successful in 17s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m24s Details
Test / test_move_reappear (push) Successful in 22s Details
Test / test_snapshot_down (push) Successful in 32s Details
Test / test_snapshot_down_ec (push) Successful in 33s Details
Test / test_splitbrain (push) Successful in 26s Details
Test / test_snapshot_chain (push) Successful in 2m13s Details
Test / test_snapshot_chain_ec (push) Successful in 3m0s Details
Test / test_rebalance_verify_imm (push) Successful in 2m57s Details
Test / test_rebalance_verify (push) Successful in 3m47s Details
Test / test_write (push) Successful in 44s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 56s Details
Test / test_rebalance_verify_ec (push) Successful in 5m4s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m29s Details
Test / test_heal_pg_size_2 (push) Successful in 3m42s Details
Test / test_heal_ec (push) Successful in 5m1s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m10s Details
Test / test_heal_csum_32k_dmj (push) Successful in 6m57s Details
Test / test_heal_csum_32k (push) Successful in 6m21s Details
Test / test_scrub (push) Successful in 1m16s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m8s Details
Test / test_scrub_xor (push) Successful in 1m13s Details
Test / test_heal_csum_4k_dmj (push) Successful in 8m15s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m1s Details
Test / test_heal_csum_4k (push) Successful in 5m47s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m14s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m24s Details
Test / test_scrub_ec (push) Successful in 27s Details
2023-12-22 02:25:09 +03:00
Vitaliy Filippov 95631773b6 Remove pve-storage-portal-dns-list format for vitastor_etcd_address 2023-12-20 02:22:06 +03:00
Vitaliy Filippov 7239cfb91a Parse log_level in cluster_client
Test / test_snapshot_ec (push) Successful in 32s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m52s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_rm (push) Successful in 20s Details
Test / test_snapshot_down (push) Successful in 27s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 28s Details
Test / test_snapshot_chain (push) Successful in 2m18s Details
Test / test_snapshot_chain_ec (push) Failing after 3m6s Details
Test / test_rebalance_verify_imm (push) Successful in 3m0s Details
Test / test_rebalance_verify (push) Successful in 3m43s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m17s Details
Test / test_rebalance_verify_ec (push) Successful in 4m9s Details
Test / test_write_xor (push) Failing after 3m16s Details
Test / test_heal_pg_size_2 (push) Successful in 3m49s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m6s Details
Test / test_heal_ec (push) Successful in 6m46s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m25s Details
Test / test_heal_csum_32k (push) Successful in 6m41s Details
Test / test_scrub (push) Successful in 1m15s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m25s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m33s Details
Test / test_scrub_xor (push) Successful in 1m7s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m16s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m56s Details
Test / test_scrub_ec (push) Successful in 52s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m7s Details
Test / test_heal_csum_4k (push) Successful in 6m14s Details
2023-12-20 02:21:23 +03:00
Vitaliy Filippov 7cea642f4a Fix vitastor-nbd image existence check not working because of non-zeroed inode_watch fields
Test / test_interrupted_rebalance_ec (push) Successful in 1m55s Details
Test / test_snapshot_ec (push) Successful in 38s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_move_reappear (push) Failing after 50s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m14s Details
Test / test_snapshot_chain_ec (push) Successful in 2m53s Details
Test / test_rebalance_verify_imm (push) Successful in 2m49s Details
Test / test_write (push) Successful in 35s Details
Test / test_rebalance_verify (push) Successful in 3m34s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 53s Details
Test / test_rebalance_verify_ec (push) Successful in 4m48s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m16s Details
Test / test_heal_pg_size_2 (push) Successful in 4m3s Details
Test / test_heal_ec (push) Successful in 4m37s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m49s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m0s Details
Test / test_heal_csum_32k (push) Successful in 6m59s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m6s Details
Test / test_scrub (push) Successful in 1m13s Details
Test / test_scrub_xor (push) Successful in 51s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m2s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m11s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m44s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m44s Details
Test / test_scrub_ec (push) Successful in 45s Details
Test / test_heal_csum_4k (push) Successful in 6m56s Details
2023-12-19 01:11:37 +03:00
Vitaliy Filippov dc615403d9 Do not warn on EPIPE in client unless log_level is raised explicitly
Test / test_snapshot_ec (push) Successful in 36s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m58s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_move_reappear (push) Failing after 49s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m26s Details
Test / test_snapshot_chain_ec (push) Failing after 3m6s Details
Test / test_rebalance_verify_imm (push) Successful in 3m11s Details
Test / test_write (push) Successful in 35s Details
Test / test_rebalance_verify (push) Successful in 4m6s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_write_xor (push) Successful in 41s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m10s Details
Test / test_rebalance_verify_ec (push) Successful in 5m0s Details
Test / test_heal_ec (push) Successful in 4m32s Details
Test / test_heal_pg_size_2 (push) Successful in 4m50s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m44s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m50s Details
Test / test_heal_csum_32k (push) Successful in 7m6s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m1s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m2s Details
Test / test_scrub (push) Successful in 1m6s Details
Test / test_scrub_xor (push) Successful in 58s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m57s Details
Test / test_heal_csum_4k (push) Successful in 6m42s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m42s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 53s Details
Test / test_scrub_ec (push) Successful in 22s Details
2023-12-17 13:42:26 +03:00
Vitaliy Filippov 1a704e06ab Allow multiple interfaces with the same IP address, for "simple routed" full mesh network
Test / test_snapshot_ec (push) Successful in 32s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m59s Details
Test / test_rm (push) Successful in 20s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_move_reappear (push) Failing after 51s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m29s Details
Test / test_snapshot_chain_ec (push) Successful in 3m5s Details
Test / test_rebalance_verify_imm (push) Successful in 3m7s Details
Test / test_write (push) Successful in 34s Details
Test / test_rebalance_verify (push) Successful in 3m59s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_write_xor (push) Successful in 41s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m20s Details
Test / test_rebalance_verify_ec (push) Successful in 5m17s Details
Test / test_heal_pg_size_2 (push) Successful in 3m52s Details
Test / test_heal_ec (push) Successful in 5m7s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m38s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m7s Details
Test / test_heal_csum_32k (push) Successful in 6m56s Details
Test / test_scrub (push) Successful in 1m10s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m11s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m8s Details
Test / test_scrub_xor (push) Successful in 49s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m32s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m19s Details
Test / test_heal_csum_4k (push) Successful in 6m3s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m45s Details
Test / test_scrub_ec (push) Successful in 28s Details
2023-12-17 13:25:56 +03:00
Vitaliy Filippov 575475de71 Do not ignore loopback addresses for OSD network (to make ECMP setups with frr possible)
Test / test_interrupted_rebalance_ec (push) Successful in 1m55s Details
Test / test_snapshot_ec (push) Successful in 32s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_move_reappear (push) Failing after 50s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m21s Details
Test / test_snapshot_chain_ec (push) Successful in 2m55s Details
Test / test_rebalance_verify_imm (push) Successful in 2m42s Details
Test / test_write (push) Successful in 43s Details
Test / test_rebalance_verify (push) Successful in 3m42s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 39s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m9s Details
Test / test_rebalance_verify_ec (push) Successful in 4m54s Details
Test / test_heal_pg_size_2 (push) Successful in 4m20s Details
Test / test_heal_ec (push) Successful in 4m53s Details
Test / test_heal_csum_32k_dmj (push) Successful in 6m14s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m9s Details
Test / test_heal_csum_32k (push) Successful in 6m49s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m56s Details
Test / test_scrub (push) Successful in 1m11s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m10s Details
Test / test_scrub_xor (push) Successful in 55s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m31s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m19s Details
Test / test_heal_csum_4k (push) Successful in 6m16s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m44s Details
Test / test_scrub_ec (push) Successful in 21s Details
2023-12-17 11:55:13 +03:00
Vitaliy Filippov aca2bef15f Add vitastor-disk update-sb command
Test / test_snapshot_ec (push) Successful in 31s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m55s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_snapshot_chain (push) Successful in 2m8s Details
Test / test_snapshot_chain_ec (push) Successful in 2m52s Details
Test / test_rebalance_verify_imm (push) Successful in 2m56s Details
Test / test_write (push) Successful in 36s Details
Test / test_rebalance_verify (push) Successful in 3m38s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec (push) Successful in 4m0s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m14s Details
Test / test_heal_pg_size_2 (push) Successful in 3m22s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m9s Details
Test / test_heal_ec (push) Successful in 6m49s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m14s Details
Test / test_heal_csum_32k (push) Successful in 6m12s Details
Test / test_scrub (push) Successful in 1m21s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m11s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m21s Details
Test / test_scrub_xor (push) Successful in 1m13s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m9s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m3s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m48s Details
Test / test_scrub_ec (push) Successful in 52s Details
Test / test_heal_csum_4k (push) Successful in 6m2s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_write_xor (push) Failing after 3m5s Details
2023-12-14 01:11:42 +03:00
Vitaliy Filippov 4dd6e89263 Change qemu to qemu-system-x86 in docs 2023-12-14 01:01:00 +03:00
Vitaliy Filippov 9bac99ffb6 Fix incorrect error in CSI when searching for the device in /sys 2023-12-14 01:00:32 +03:00
Vitaliy Filippov 62ed130960 Support building qemu 8.1 from bookworm-backports 2023-12-10 00:34:13 +03:00
Vitaliy Filippov 9c7755b6e8 Use qemu-storage-daemon from QEMU 8.1.2 for CSI 2023-12-08 00:10:12 +03:00
Vitaliy Filippov 691ebd991a Move 2 last log printfs to stderr from stdout in etcd_state_client
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m46s Details
Test / test_move_reappear (push) Successful in 20s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 33s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_snapshot_chain (push) Successful in 2m12s Details
Test / test_snapshot_chain_ec (push) Successful in 2m57s Details
Test / test_rebalance_verify_ec_imm (push) Failing after 22s Details
Test / test_rebalance_verify_imm (push) Successful in 2m45s Details
Test / test_write (push) Successful in 31s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m32s Details
Test / test_write_xor (push) Successful in 1m15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m3s Details
Test / test_rebalance_verify_ec (push) Successful in 6m34s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m43s Details
Test / test_heal_ec (push) Successful in 5m33s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m45s Details
Test / test_heal_csum_32k (push) Successful in 6m37s Details
Test / test_scrub (push) Successful in 1m3s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m39s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m37s Details
Test / test_scrub_zero_osd_2 (push) Successful in 54s Details
Test / test_scrub_xor (push) Successful in 53s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m29s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 48s Details
Test / test_scrub_ec (push) Successful in 46s Details
Test / test_heal_csum_4k (push) Successful in 5m31s Details
2023-12-08 00:01:52 +03:00
Vitaliy Filippov 6d5df908a3 Fix possible out of bounds when checking invalid journal entries 2023-12-08 00:01:07 +03:00
Vitaliy Filippov fa87769ed8 Correct config options in vduse docs 2023-12-06 02:09:04 +03:00
Vitaliy Filippov 2ce8292803 Also log when killing process 2023-12-06 01:06:53 +03:00
Vitaliy Filippov 7f8f7ded52 Check for empty output of vitastor-nbd map (just in case) 2023-12-06 01:01:14 +03:00
Vitaliy Filippov 68553eabbb Log executed CLI commands 2023-12-06 00:48:12 +03:00
Vitaliy Filippov 3147c5c8d5 Remove internal error wrapping 2023-12-06 00:39:42 +03:00
Vitaliy Filippov 576e2ae608 Fix etcd_address check in CSI 2023-12-06 00:28:21 +03:00
Vitaliy Filippov a1c7cc3d8d Release 1.3.1
Test / test_interrupted_rebalance_ec (push) Successful in 1m46s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_ec (push) Successful in 35s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m22s Details
Test / test_snapshot_chain_ec (push) Successful in 2m59s Details
Test / test_rebalance_verify_imm (push) Successful in 3m3s Details
Test / test_rebalance_verify (push) Successful in 3m47s Details
Test / test_write (push) Successful in 44s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m36s Details
Test / test_rebalance_verify_ec (push) Successful in 4m20s Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m45s Details
Test / test_heal_ec (push) Successful in 6m22s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m51s Details
Test / test_heal_csum_32k (push) Successful in 6m2s Details
Test / test_scrub (push) Successful in 1m14s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m19s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m54s Details
Test / test_scrub_xor (push) Successful in 1m1s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m59s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m54s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m2s Details
Test / test_scrub_ec (push) Successful in 34s Details
Test / test_heal_csum_4k (push) Successful in 6m0s Details
Test / test_write_xor (push) Successful in 32s Details
Hotfix to 1.3.0 - new "journal space reservation" had a bug which
caused OSDs to crash with EC and without immediate_commit.
2023-12-04 18:35:09 +03:00
Vitaliy Filippov a5e3dfbc5a Oops, 1.3.0 needs a hotfix
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 27s Details
Test / test_snapshot_chain (push) Successful in 2m13s Details
Test / test_snapshot_chain_ec (push) Successful in 2m56s Details
Test / test_rebalance_verify_imm (push) Successful in 2m51s Details
Test / test_rebalance_verify (push) Successful in 3m38s Details
Test / test_write (push) Successful in 45s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec (push) Successful in 4m5s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m22s Details
Test / test_heal_pg_size_2 (push) Successful in 3m51s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m39s Details
Test / test_heal_ec (push) Successful in 6m39s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m55s Details
Test / test_heal_csum_32k (push) Successful in 6m5s Details
Test / test_scrub (push) Successful in 1m18s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m19s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m25s Details
Test / test_scrub_xor (push) Successful in 50s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m46s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m17s Details
Test / test_heal_csum_4k (push) Successful in 5m51s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m0s Details
Test / test_scrub_ec (push) Successful in 21s Details
Test / test_write_xor (push) Successful in 33s Details
2023-12-04 13:45:54 +03:00
Vitaliy Filippov 7972502eaf Release 1.3.0
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m1s Details
Test / test_snapshot_down (push) Successful in 19s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_snapshot_down_ec (push) Failing after 3m10s Details
Test / test_rebalance_verify (push) Successful in 2m45s Details
Test / test_rebalance_verify_imm (push) Successful in 2m17s Details
Test / test_write (push) Successful in 1m11s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m41s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_write_xor (push) Failing after 3m6s Details
Test / test_rebalance_verify_ec (push) Failing after 5m27s Details
Test / test_heal_pg_size_2 (push) Failing after 3m7s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m36s Details
Test / test_heal_csum_32k_dj (push) Failing after 4m53s Details
Test / test_heal_csum_32k (push) Failing after 5m27s Details
Test / test_heal_ec (push) Failing after 10m15s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m14s Details
Test / test_scrub (push) Successful in 1m11s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m15s Details
Test / test_scrub_zero_osd_2 (push) Successful in 56s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m4s Details
Test / test_heal_csum_4k (push) Failing after 5m31s Details
Test / test_scrub_xor (push) Failing after 3m17s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 3m6s Details
Test / test_change_pg_count_ec (push) Failing after 3m5s Details
Test / test_snapshot_ec (push) Failing after 3m5s Details
Test / test_scrub_ec (push) Failing after 3m5s Details
Test / test_snapshot_chain_ec (push) Failing after 3m5s Details
Test / test_interrupted_rebalance_ec (push) Failing after 10m5s Details
New features:
- RDMA without ODP - much faster and all cards are now supported, not just Mellanox
- VDUSE in CSI - faster, more stable and can even recover after CSI pod restart!
- Reserve journal space for stabilize requests dynamically to prevent stalls under load with EC
- Raise default NBD timeout from 30 to 300 seconds and allow to take it from /etc/vitastor/vitastor.conf
- Remove explicit etcdUrl/etcdPrefix K8S storage class parameter support to prevent
  etcd migration issues for volumes created with these parameters
- Support QEMU 8.1 and pve-qemu 8.1

Bug fixes:
- Fix RDMA connection (and thus memory) leak
- Fix rare crashes under load due to incorrect io_uring queue size tracking
- Fix monitor statistics aggregation in case of empty /osd/stats keys
- Fix crash on unknown long argument to vitastor-disk
- Allow trailing comma in JSONs again
- Fix crash on attempts to dump a long listing of objects "to stabilize" or "to rollback" in a slow op
2023-12-04 02:36:43 +03:00
Vitaliy Filippov e57b7203b8 Use cmake3 on RHEL 7 2023-12-04 02:36:29 +03:00
Vitaliy Filippov c8a179dcda Note that Proxmox 8.1 is supported 2023-12-04 02:20:33 +03:00
Vitaliy Filippov 845454742d Fix warning with QEMU 8.1 2023-12-04 01:59:07 +03:00
Vitaliy Filippov d65512bd80 Add patches for QEMU 8.1 2023-12-04 01:56:17 +03:00
Vitaliy Filippov 53de2bbd0f Support VDUSE in CSI
VDUSE has multiple advantages:
- Better performance
- Lack of timeout problems
- And even the ability to recover after restart of the vitastor-csi pod!
2023-12-04 00:41:24 +03:00
Vitaliy Filippov 628aa59574 Raise default NBD timeout from 30 to 300 seconds and allow to take it from /etc/vitastor/vitastor.conf
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m0s Details
Test / test_snapshot_down (push) Successful in 20s Details
Test / test_snapshot_ec (push) Failing after 3m6s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_snapshot_chain_ec (push) Failing after 3m7s Details
Test / test_snapshot_down_ec (push) Failing after 3m7s Details
Test / test_rebalance_verify (push) Successful in 2m42s Details
Test / test_rebalance_verify_imm (push) Successful in 2m14s Details
Test / test_write (push) Successful in 45s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m19s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_interrupted_rebalance_ec (push) Failing after 10m40s Details
Test / test_write_xor (push) Failing after 3m5s Details
Test / test_rebalance_verify_ec (push) Failing after 5m22s Details
Test / test_heal_pg_size_2 (push) Failing after 3m48s Details
Test / test_heal_csum_32k_dj (push) Successful in 4m27s Details
Test / test_heal_ec (push) Failing after 10m6s Details
Test / test_heal_csum_32k_dmj (push) Failing after 10m14s Details
Test / test_heal_csum_32k (push) Failing after 10m14s Details
Test / test_scrub (push) Successful in 22s Details
Test / test_scrub_zero_osd_2 (push) Successful in 19s Details
Test / test_heal_csum_4k_dmj (push) Failing after 10m10s Details
Test / test_scrub_pg_size_3 (push) Successful in 30s Details
Test / test_scrub_xor (push) Failing after 3m6s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 3m5s Details
Test / test_heal_csum_4k_dj (push) Failing after 10m13s Details
Test / test_scrub_ec (push) Failing after 3m5s Details
Test / test_heal_csum_4k (push) Failing after 10m8s Details
2023-12-02 14:11:14 +03:00
Vitaliy Filippov 037cf64a47 Remove explicit etcdUrl/etcdPrefix from volume parameters 2023-12-02 13:26:00 +03:00
Vitaliy Filippov 19e2d9d6fa Fix crash on unknown long argument to vitastor-disk
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 59s Details
Test / test_snapshot_down (push) Successful in 19s Details
Test / test_snapshot_ec (push) Failing after 3m5s Details
Test / test_splitbrain (push) Successful in 11s Details
Test / test_snapshot_chain_ec (push) Failing after 3m6s Details
Test / test_rebalance_verify_imm (push) Failing after 22s Details
Test / test_rebalance_verify_ec (push) Failing after 44s Details
Test / test_snapshot_down_ec (push) Failing after 3m5s Details
Test / test_rebalance_verify (push) Successful in 2m53s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m14s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_write (push) Failing after 3m19s Details
Test / test_interrupted_rebalance_ec (push) Failing after 10m7s Details
Test / test_write_xor (push) Failing after 3m8s Details
Test / test_heal_pg_size_2 (push) Failing after 3m9s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m45s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m1s Details
Test / test_heal_csum_4k_dmj (push) Successful in 4m12s Details
Test / test_heal_ec (push) Failing after 10m13s Details
Test / test_heal_csum_4k_dj (push) Failing after 4m13s Details
Test / test_scrub_zero_osd_2 (push) Successful in 48s Details
Test / test_scrub (push) Successful in 51s Details
Test / test_scrub_pg_size_3 (push) Successful in 55s Details
Test / test_heal_csum_32k (push) Failing after 10m12s Details
Test / test_heal_csum_4k (push) Failing after 3m55s Details
Test / test_scrub_xor (push) Failing after 3m7s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 3m14s Details
Test / test_scrub_ec (push) Failing after 3m11s Details
2023-12-01 00:55:51 +03:00
Vitaliy Filippov bfc7e61909 Add more notes + performance comparison about VDUSE 2023-11-25 02:25:56 +03:00
Vitaliy Filippov 7da4868b37 Fix monitor statistics aggregation in case of empty /osd/stats keys
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 1m1s Details
Test / test_snapshot_down (push) Successful in 20s Details
Test / test_snapshot_ec (push) Failing after 3m6s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_snapshot_chain_ec (push) Failing after 3m6s Details
Test / test_snapshot_down_ec (push) Failing after 3m7s Details
Test / test_rebalance_verify_ec (push) Failing after 48s Details
Test / test_rebalance_verify (push) Successful in 3m30s Details
Test / test_rebalance_verify_imm (push) Successful in 3m3s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m36s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_interrupted_rebalance_ec (push) Failing after 10m7s Details
Test / test_write (push) Failing after 3m7s Details
Test / test_write_xor (push) Failing after 3m6s Details
Test / test_heal_csum_32k_dmj (push) Failing after 4m5s Details
Test / test_heal_csum_32k (push) Failing after 3m16s Details
Test / test_heal_pg_size_2 (push) Failing after 10m7s Details
Test / test_heal_ec (push) Failing after 10m10s Details
Test / test_heal_csum_32k_dj (push) Failing after 10m14s Details
Test / test_scrub (push) Successful in 59s Details
Test / test_scrub_zero_osd_2 (push) Successful in 54s Details
Test / test_heal_csum_4k_dj (push) Successful in 4m27s Details
Test / test_scrub_pg_size_3 (push) Successful in 41s Details
Test / test_heal_csum_4k (push) Failing after 4m56s Details
Test / test_scrub_xor (push) Failing after 3m11s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 3m5s Details
Test / test_scrub_ec (push) Failing after 3m5s Details
Test / test_heal_csum_4k_dmj (push) Failing after 10m8s Details
2023-11-24 01:05:21 +03:00
Vitaliy Filippov b5c020ce0b Use io_uring SQ size for ringloop capacity - otherwise get_sqe could return NULL when space_left() was > 0 under load
Test / test_rm (push) Successful in 11s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_snapshot_chain (push) Successful in 1m0s Details
Test / test_snapshot_down (push) Successful in 19s Details
Test / test_snapshot_ec (push) Failing after 3m5s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_snapshot_chain_ec (push) Failing after 3m5s Details
Test / test_snapshot_down_ec (push) Failing after 3m6s Details
Test / test_rebalance_verify_ec (push) Failing after 42s Details
Test / test_rebalance_verify_imm (push) Successful in 2m47s Details
Test / test_rebalance_verify (push) Successful in 3m14s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m38s Details
Test / test_interrupted_rebalance_ec (push) Failing after 10m5s Details
Test / test_write_no_same (push) Successful in 11s Details
Test / test_write (push) Failing after 3m6s Details
Test / test_write_xor (push) Failing after 3m5s Details
Test / test_heal_pg_size_2 (push) Failing after 3m45s Details
Test / test_heal_csum_32k_dj (push) Failing after 4m41s Details
Test / test_heal_csum_32k (push) Successful in 4m28s Details
Test / test_heal_ec (push) Failing after 10m12s Details
Test / test_heal_csum_4k_dmj (push) Failing after 4m27s Details
Test / test_scrub (push) Successful in 53s Details
Test / test_heal_csum_32k_dmj (push) Failing after 10m16s Details
Test / test_scrub_zero_osd_2 (push) Successful in 59s Details
Test / test_heal_csum_4k_dj (push) Failing after 4m24s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m7s Details
Test / test_scrub_xor (push) Failing after 3m14s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 3m6s Details
Test / test_scrub_ec (push) Failing after 3m7s Details
Test / test_heal_csum_4k (push) Failing after 10m9s Details
Raise default io_uring size to 1024 for the same effective capacity as previously
2023-11-20 03:04:06 +03:00
Vitaliy Filippov 6b33ae973d %d -> %lu 2023-11-20 03:02:26 +03:00
Vitaliy Filippov cf36445359 Reserve journal space for stabilize requests dynamically to prevent stalls 2023-11-20 03:01:57 +03:00
Vitaliy Filippov 3fd873d263 Add -fno-omit-frame-pointer by default 2023-11-20 02:59:54 +03:00
Vitaliy Filippov a00e8ae9ed Fix mismatch journal pos format in vitastor-disk
Test / test_interrupted_rebalance_ec (push) Successful in 1m47s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_ec (push) Successful in 35s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_move_reappear (push) Failing after 50s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m12s Details
Test / test_snapshot_chain_ec (push) Successful in 3m0s Details
Test / test_rebalance_verify_imm (push) Successful in 3m42s Details
Test / test_write (push) Successful in 35s Details
Test / test_rebalance_verify (push) Successful in 4m23s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec (push) Successful in 4m45s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m49s Details
Test / test_write_xor (push) Failing after 3m9s Details
Test / test_heal_pg_size_2 (push) Successful in 3m20s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m41s Details
Test / test_heal_ec (push) Successful in 6m24s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m53s Details
Test / test_heal_csum_32k (push) Successful in 6m10s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m21s Details
Test / test_scrub (push) Failing after 3m13s Details
Test / test_scrub_zero_osd_2 (push) Successful in 56s Details
Test / test_scrub_xor (push) Successful in 44s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m48s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m0s Details
Test / test_scrub_ec (push) Successful in 52s Details
Test / test_heal_csum_4k (push) Successful in 5m43s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m8s Details
2023-11-19 15:19:54 +03:00
Vitaliy Filippov 75674545dc Limit the number of printed object versions in slow op dump (otherwise it may overflow the fixed buffer)
Test / test_interrupted_rebalance_ec (push) Successful in 1m55s Details
Test / test_snapshot_ec (push) Successful in 35s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_move_reappear (push) Failing after 49s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m17s Details
Test / test_snapshot_chain_ec (push) Successful in 2m58s Details
Test / test_rebalance_verify_imm (push) Successful in 3m12s Details
Test / test_write (push) Successful in 35s Details
Test / test_rebalance_verify (push) Successful in 3m52s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 57s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m18s Details
Test / test_rebalance_verify_ec (push) Successful in 5m21s Details
Test / test_heal_pg_size_2 (push) Successful in 4m0s Details
Test / test_heal_ec (push) Successful in 4m48s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m8s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m29s Details
Test / test_heal_csum_32k (push) Successful in 6m21s Details
Test / test_scrub (push) Successful in 1m15s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m42s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m1s Details
Test / test_scrub_xor (push) Successful in 49s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m22s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m6s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m37s Details
Test / test_scrub_ec (push) Successful in 35s Details
Test / test_heal_csum_4k (push) Successful in 6m16s Details
2023-11-13 01:10:28 +03:00
Vitaliy Filippov 225eb2fe3d Support RDMA without ODP by stupidly copying memory. Disable ODP by default
ODP is slower than regular RDMA even with memory copy overhead

Example numbers:
- 3950000 random read iops without ODP vs 240000 iops with ODP
- 1447000 random write iops without ODP vs 101000 iops with ODP

Reference: https://tkygtr6.github.io/pub/ISPASS21_slides.pdf
2023-11-12 15:03:47 +03:00
Vitaliy Filippov 7e82573ed0 Fix RDMA connection leak which was preventing stable functioning of RDMA :)
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_snapshot_ec (push) Successful in 38s Details
Test / test_rm (push) Successful in 16s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m19s Details
Test / test_snapshot_chain_ec (push) Successful in 3m0s Details
Test / test_rebalance_verify_imm (push) Successful in 2m45s Details
Test / test_rebalance_verify (push) Successful in 3m36s Details
Test / test_write (push) Successful in 37s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_write_xor (push) Successful in 53s Details
Test / test_rebalance_verify_ec (push) Successful in 4m54s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m8s Details
Test / test_heal_pg_size_2 (push) Successful in 4m5s Details
Test / test_heal_ec (push) Successful in 4m48s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m25s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m29s Details
Test / test_heal_csum_32k (push) Successful in 6m22s Details
Test / test_scrub (push) Successful in 1m7s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m39s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m9s Details
Test / test_scrub_xor (push) Successful in 47s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m3s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m12s Details
Test / test_heal_csum_4k (push) Successful in 6m11s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m29s Details
Test / test_scrub_ec (push) Successful in 33s Details
2023-11-11 23:40:47 +03:00
Vitaliy Filippov 12a6bed2d5 Return the new accidentally rolled back json11 commit ("allow trailing comma")
Test / test_snapshot_ec (push) Successful in 28s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m51s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m22s Details
Test / test_rebalance_verify_imm (push) Successful in 3m4s Details
Test / test_rebalance_verify (push) Successful in 3m48s Details
Test / test_write (push) Successful in 39s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m14s Details
Test / test_rebalance_verify_ec (push) Successful in 4m2s Details
Test / test_heal_pg_size_2 (push) Successful in 3m44s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m43s Details
Test / test_heal_ec (push) Successful in 6m47s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m57s Details
Test / test_heal_csum_32k (push) Successful in 6m11s Details
Test / test_scrub (push) Successful in 1m10s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m17s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m19s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m10s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m47s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m3s Details
Test / test_heal_csum_4k (push) Successful in 6m3s Details
Test / test_scrub_ec (push) Successful in 28s Details
Test / test_scrub_xor (push) Successful in 23s Details
Test / test_snapshot_chain_ec (push) Successful in 2m25s Details
Test / test_write_xor (push) Failing after 3m6s Details
2023-11-07 15:49:23 +03:00
Vitaliy Filippov 5524dbdab7 Release 1.2.0
Test / test_snapshot_ec (push) Successful in 25s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m18s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 18s Details
Test / test_snapshot_chain (push) Successful in 2m13s Details
Test / test_snapshot_chain_ec (push) Successful in 2m57s Details
Test / test_rebalance_verify_imm (push) Successful in 2m51s Details
Test / test_write (push) Successful in 38s Details
Test / test_rebalance_verify (push) Successful in 3m39s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_rebalance_verify_ec (push) Successful in 3m56s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m6s Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m35s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m44s Details
Test / test_heal_csum_32k (push) Successful in 5m50s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m44s Details
Test / test_scrub_zero_osd_2 (push) Successful in 57s Details
Test / test_scrub (push) Successful in 1m0s Details
Test / test_scrub_xor (push) Successful in 1m5s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m9s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m38s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 54s Details
Test / test_scrub_ec (push) Successful in 52s Details
Test / test_heal_csum_4k (push) Successful in 5m8s Details
Test / test_heal_ec (push) Successful in 3m17s Details
Test / test_write_xor (push) Successful in 35s Details
Test / test_move_reappear (push) Failing after 48s Details
New features:

- Implement CSI volume expansion
- Implement CSI volume snapshots
- CSI driver now requires Kubernetes >= 1.20

Bug fixes:

- Important bug fix for EC: fix EC n+k, k>=2 read recovery in ISA-L version returning
  incorrect data when reading at least the second chunk out of multiple missing chunks
  without reading the first one. All users of EC n+k, k>=2 should upgrade as soon as
  possible, and upgrade should be conducted with downtime: first stop all clients
  (VMs/containers), then all OSDs, then upgrade and restart everything.
- Fix unstable statistics aggregation in monitor (affecting vitastor-cli status and df)
- Make udev not wait for OSDs to start during boot
- Do not report negative numbers of offline PGs in vitastor-cli status when changing PG count
- Report both old and new PG counts in vitastor-cli df when changing it
- Fix OSDs sometimes not starting with "The code only supports journal versions 1 and 2,
  but it is 2 on disk" error after upgrading from pre-1.0 versions and letting OSDs run
  for some time
- Fix monitors sometimes returning old PG count back after OSD configuration changes
- Make monitor PG changes more stable and timeout errors less probable
2023-11-05 01:48:57 +03:00
Vitaliy Filippov cd3dec06ac Remove spaces from old->new PG count in df
Test / test_interrupted_rebalance_ec (push) Successful in 1m50s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot_ec (push) Successful in 33s Details
Test / test_snapshot_down (push) Successful in 29s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m34s Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s Details
Test / test_rebalance_verify (push) Successful in 4m9s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m15s Details
Test / test_rebalance_verify_ec (push) Successful in 4m29s Details
Test / test_heal_pg_size_2 (push) Successful in 3m21s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m38s Details
Test / test_heal_ec (push) Successful in 6m14s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m22s Details
Test / test_heal_csum_32k (push) Successful in 6m40s Details
Test / test_scrub (push) Successful in 1m11s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m12s Details
Test / test_scrub_xor (push) Successful in 1m16s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m4s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m34s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m4s Details
Test / test_heal_csum_4k (push) Successful in 5m37s Details
Test / test_scrub_ec (push) Successful in 43s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m14s Details
Test / test_write_xor (push) Successful in 1m11s Details
Test / test_snapshot_chain_ec (push) Successful in 2m43s Details
2023-11-05 01:45:45 +03:00
Vitaliy Filippov 371d79e059 Document vitastor-csi features 2023-11-05 01:05:26 +03:00
Vitaliy Filippov 0e888e6c60 Prevent spamming etcd with last_clean_pgs update requests
Test / test_snapshot_ec (push) Successful in 34s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m49s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m23s Details
Test / test_snapshot_chain_ec (push) Successful in 3m4s Details
Test / test_rebalance_verify_imm (push) Successful in 2m54s Details
Test / test_rebalance_verify (push) Successful in 3m48s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 55s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m15s Details
Test / test_rebalance_verify_ec (push) Successful in 5m3s Details
Test / test_heal_pg_size_2 (push) Successful in 3m59s Details
Test / test_heal_ec (push) Successful in 4m56s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m48s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m44s Details
Test / test_heal_csum_32k (push) Successful in 6m35s Details
Test / test_scrub (push) Successful in 1m14s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m54s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m2s Details
Test / test_scrub_xor (push) Successful in 49s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m9s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m54s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m17s Details
Test / test_heal_csum_4k (push) Successful in 6m18s Details
Test / test_scrub_ec (push) Successful in 37s Details
2023-11-05 00:12:00 +03:00
Vitaliy Filippov 408c21d8f0 Scale last_clean_pgs PG count even if current PGs already contain the new number of PGs
Test / test_interrupted_rebalance_ec (push) Successful in 1m49s Details
Test / test_snapshot_ec (push) Successful in 31s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_move_reappear (push) Failing after 50s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_snapshot_chain (push) Successful in 2m15s Details
Test / test_snapshot_chain_ec (push) Successful in 2m56s Details
Test / test_rebalance_verify_imm (push) Successful in 2m59s Details
Test / test_write (push) Successful in 34s Details
Test / test_rebalance_verify (push) Successful in 3m44s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_write_xor (push) Successful in 52s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m5s Details
Test / test_rebalance_verify_ec (push) Successful in 5m1s Details
Test / test_heal_pg_size_2 (push) Successful in 4m1s Details
Test / test_heal_ec (push) Successful in 5m3s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m13s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m37s Details
Test / test_heal_csum_32k (push) Successful in 6m19s Details
Test / test_scrub (push) Successful in 1m11s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m13s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m5s Details
Test / test_scrub_xor (push) Successful in 48s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m11s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m10s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m29s Details
Test / test_heal_csum_4k (push) Successful in 6m9s Details
Test / test_scrub_ec (push) Successful in 35s Details
2023-11-04 23:45:59 +03:00
Vitaliy Filippov 43cb9ae212 Prevent multiple parallel recheck_pgs in case of timeouts
Test / test_snapshot_ec (push) Successful in 37s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_rm (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m25s Details
Test / test_snapshot_chain_ec (push) Failing after 3m7s Details
Test / test_rebalance_verify_imm (push) Successful in 3m0s Details
Test / test_rebalance_verify (push) Successful in 3m54s Details
Test / test_write (push) Successful in 34s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 52s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m6s Details
Test / test_rebalance_verify_ec (push) Successful in 5m10s Details
Test / test_heal_pg_size_2 (push) Successful in 4m1s Details
Test / test_heal_ec (push) Successful in 4m21s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m10s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m51s Details
Test / test_heal_csum_32k (push) Successful in 6m54s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m38s Details
Test / test_scrub (push) Successful in 1m9s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m2s Details
Test / test_scrub_xor (push) Successful in 43s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m24s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m16s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m38s Details
Test / test_scrub_ec (push) Successful in 37s Details
Test / test_heal_csum_4k (push) Successful in 6m2s Details
2023-11-04 20:59:56 +03:00
Vitaliy Filippov e15b6e7805 Fix "cannot be narrowed" in clang
Test / test_snapshot_ec (push) Successful in 44s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m10s Details
Test / test_rm (push) Successful in 16s Details
Test / test_move_reappear (push) Failing after 51s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m32s Details
Test / test_snapshot_chain_ec (push) Successful in 3m2s Details
Test / test_rebalance_verify_imm (push) Successful in 3m0s Details
Test / test_write (push) Successful in 33s Details
Test / test_rebalance_verify (push) Successful in 3m53s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m23s Details
Test / test_rebalance_verify_ec (push) Successful in 4m11s Details
Test / test_write_xor (push) Failing after 3m12s Details
Test / test_heal_pg_size_2 (push) Successful in 3m47s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m17s Details
Test / test_heal_ec (push) Successful in 5m34s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m43s Details
Test / test_heal_csum_32k (push) Successful in 6m30s Details
Test / test_scrub (push) Successful in 1m18s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m11s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m24s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m23s Details
Test / test_scrub_xor (push) Successful in 54s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m1s Details
Test / test_scrub_ec (push) Successful in 54s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m25s Details
Test / test_heal_csum_4k (push) Successful in 6m10s Details
2023-11-04 18:14:44 +03:00
Vitaliy Filippov 31017d8412 Allow to start with V2 journal with header size from V1, as incorrectly updated by previous versions 2023-11-04 18:13:42 +03:00
Vitaliy Filippov 4819854064 Fix OSDs incorrectly updating journal superblock after upgrade to 1.x from pre-1.x and refusing to start after it
Test / test_interrupted_rebalance_imm (push) Successful in 3m38s Details
Test / test_snapshot_ec (push) Successful in 33s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_move_reappear (push) Failing after 47s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m31s Details
Test / test_snapshot_chain_ec (push) Successful in 3m7s Details
Test / test_rebalance_verify_imm (push) Successful in 2m54s Details
Test / test_write (push) Successful in 32s Details
Test / test_rebalance_verify (push) Successful in 3m46s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_write_xor (push) Successful in 37s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m56s Details
Test / test_rebalance_verify_ec (push) Successful in 5m0s Details
Test / test_heal_pg_size_2 (push) Failing after 4m18s Details
Test / test_heal_ec (push) Successful in 5m3s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m19s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m44s Details
Test / test_heal_csum_32k (push) Successful in 6m37s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m46s Details
Test / test_scrub (push) Successful in 1m5s Details
Test / test_scrub_zero_osd_2 (push) Successful in 48s Details
Test / test_scrub_xor (push) Successful in 45s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m37s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m17s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m40s Details
Test / test_scrub_ec (push) Successful in 34s Details
Test / test_heal_csum_4k (push) Successful in 7m13s Details
2023-11-04 15:02:24 +03:00
Vitaliy Filippov 1f509cca77 Fix unused capture warnings and void* arithmetic (clang)
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_snapshot_ec (push) Successful in 40s Details
Test / test_rm (push) Successful in 16s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 33s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m31s Details
Test / test_snapshot_chain_ec (push) Failing after 3m7s Details
Test / test_rebalance_verify_imm (push) Successful in 3m6s Details
Test / test_write (push) Successful in 39s Details
Test / test_rebalance_verify (push) Successful in 4m7s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m23s Details
Test / test_rebalance_verify_ec (push) Successful in 4m20s Details
Test / test_write_xor (push) Failing after 3m9s Details
Test / test_heal_pg_size_2 (push) Successful in 3m55s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m44s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m8s Details
Test / test_heal_csum_32k (push) Successful in 5m58s Details
Test / test_heal_ec (push) Failing after 10m16s Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m57s Details
Test / test_scrub (push) Successful in 1m8s Details
Test / test_scrub_zero_osd_2 (push) Successful in 59s Details
Test / test_scrub_xor (push) Successful in 47s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m30s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m7s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m34s Details
Test / test_heal_csum_4k (push) Successful in 5m21s Details
Test / test_scrub_ec (push) Successful in 43s Details
2023-11-04 14:55:12 +03:00
Vitaliy Filippov aa8e8e8271 Add version info to --help output
Test / test_minsize_1 (push) Successful in 16s Details
Test / test_snapshot_ec (push) Successful in 39s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 32s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m31s Details
Test / test_snapshot_chain_ec (push) Failing after 3m6s Details
Test / test_rebalance_verify_imm (push) Successful in 2m47s Details
Test / test_rebalance_verify (push) Successful in 3m38s Details
Test / test_write (push) Successful in 38s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m19s Details
Test / test_rebalance_verify_ec (push) Successful in 4m6s Details
Test / test_write_xor (push) Failing after 3m10s Details
Test / test_heal_pg_size_2 (push) Successful in 3m54s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m6s Details
Test / test_heal_ec (push) Successful in 5m48s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m22s Details
Test / test_heal_csum_32k (push) Successful in 6m30s Details
Test / test_scrub (push) Successful in 1m18s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m12s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m38s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m14s Details
Test / test_scrub_xor (push) Successful in 1m0s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 56s Details
Test / test_scrub_ec (push) Successful in 51s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m20s Details
Test / test_heal_csum_4k (push) Successful in 5m58s Details
2023-11-04 13:32:12 +03:00
Vitaliy Filippov 4d79e531c5 Do not print "-X offline" in status when changing pool PG count, print it in df instead
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m20s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_ec (push) Successful in 36s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_move_reappear (push) Failing after 48s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m21s Details
Test / test_snapshot_chain_ec (push) Successful in 3m1s Details
Test / test_rebalance_verify_imm (push) Successful in 3m6s Details
Test / test_write (push) Successful in 35s Details
Test / test_rebalance_verify (push) Successful in 3m46s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec (push) Successful in 4m7s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m23s Details
Test / test_write_xor (push) Failing after 3m8s Details
Test / test_heal_pg_size_2 (push) Successful in 4m15s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m39s Details
Test / test_heal_ec (push) Successful in 6m35s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m5s Details
Test / test_heal_csum_32k (push) Successful in 6m45s Details
Test / test_scrub (push) Successful in 1m8s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m24s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m6s Details
Test / test_scrub_xor (push) Successful in 41s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m18s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m29s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m34s Details
Test / test_heal_csum_4k (push) Successful in 6m7s Details
Test / test_scrub_ec (push) Successful in 30s Details
2023-11-04 13:12:13 +03:00
Vitaliy Filippov 30dff8893f Fix ISA-L version EC recovery with first missing data chunk not being read
Test / test_snapshot (push) Successful in 44s Details
Test / test_snapshot_ec (push) Successful in 28s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m20s Details
Test / test_snapshot_chain_ec (push) Successful in 3m1s Details
Test / test_rebalance_verify_imm (push) Successful in 2m49s Details
Test / test_rebalance_verify (push) Successful in 3m37s Details
Test / test_write (push) Successful in 42s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 54s Details
Test / test_rebalance_verify_ec (push) Successful in 4m55s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m13s Details
Test / test_heal_pg_size_2 (push) Successful in 4m4s Details
Test / test_heal_ec (push) Successful in 5m2s Details
Test / test_heal_csum_32k_dmj (push) Failing after 5m54s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m6s Details
Test / test_heal_csum_32k (push) Successful in 6m59s Details
Test / test_scrub (push) Successful in 1m16s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m56s Details
Test / test_scrub_xor (push) Successful in 51s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m1s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m25s Details
Test / test_heal_csum_4k (push) Successful in 6m9s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m33s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m37s Details
Test / test_scrub_ec (push) Successful in 26s Details
(Yes, all EC n + k with k >= 2 users should upgrade as soon as possible)
2023-11-04 01:34:18 +03:00
Vitaliy Filippov becf14a705 Add a test for EC with multiple missing data chunks, but without recovery of first of them 2023-11-04 01:34:18 +03:00
Vitaliy Filippov 64388788c1 Implement CSI volume expansion
Test / test_snapshot_ec (push) Successful in 35s Details
Test / test_minsize_1 (push) Successful in 16s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_move_reappear (push) Failing after 50s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_snapshot_chain (push) Successful in 2m24s Details
Test / test_snapshot_chain_ec (push) Failing after 3m6s Details
Test / test_rebalance_verify_imm (push) Successful in 3m15s Details
Test / test_write (push) Successful in 41s Details
Test / test_rebalance_verify (push) Successful in 4m13s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_write_xor (push) Successful in 50s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m28s Details
Test / test_rebalance_verify_ec (push) Successful in 5m30s Details
Test / test_heal_pg_size_2 (push) Successful in 4m5s Details
Test / test_heal_ec (push) Successful in 4m57s Details
Test / test_heal_csum_32k_dmj (push) Successful in 6m13s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m10s Details
Test / test_heal_csum_32k (push) Successful in 6m40s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m24s Details
Test / test_scrub (push) Successful in 1m7s Details
Test / test_scrub_xor (push) Successful in 47s Details
Test / test_scrub_zero_osd_2 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m25s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m59s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m10s Details
Test / test_heal_csum_4k (push) Successful in 6m2s Details
Test / test_scrub_ec (push) Successful in 40s Details
2023-11-01 12:46:20 +03:00
Vitaliy Filippov 37653abe4b Implement CSI volume snapshots 2023-11-01 12:46:20 +03:00
Vitaliy Filippov 7c054c6f10 Add "id" to df --json output 2023-11-01 12:46:16 +03:00
Vitaliy Filippov bb7709e824 Support listening on non-127.0.0.1 in tests 2023-11-01 12:45:27 +03:00
Vitaliy Filippov ebeace5a2d Add cmake and pkg-config to debian build depends 2023-11-01 12:45:27 +03:00
Vitaliy Filippov a378789f10 Rollback erroneous go.mod changes in 1.1.0 O:-)
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m52s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 32s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m23s Details
Test / test_snapshot_chain_ec (push) Successful in 3m2s Details
Test / test_rebalance_verify_imm (push) Successful in 3m1s Details
Test / test_rebalance_verify (push) Successful in 3m49s Details
Test / test_write (push) Successful in 41s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_write_xor (push) Successful in 43s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m15s Details
Test / test_rebalance_verify_ec (push) Successful in 5m9s Details
Test / test_heal_pg_size_2 (push) Successful in 4m13s Details
Test / test_heal_ec (push) Successful in 4m31s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m59s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m14s Details
Test / test_heal_csum_32k (push) Successful in 6m47s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m47s Details
Test / test_scrub (push) Successful in 1m11s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m0s Details
Test / test_scrub_xor (push) Successful in 52s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m19s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m52s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m2s Details
Test / test_heal_csum_4k (push) Successful in 5m46s Details
Test / test_scrub_ec (push) Successful in 25s Details
2023-10-30 18:47:48 +03:00
Vitaliy Filippov 1fe678e57b Add --no-block to udev rule
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m0s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Successful in 30s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m29s Details
Test / test_snapshot_chain_ec (push) Successful in 3m4s Details
Test / test_rebalance_verify_ec_imm (push) Failing after 18s Details
Test / test_write (push) Successful in 29s Details
Test / test_rebalance_verify_imm (push) Successful in 2m53s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 3m46s Details
Test / test_write_xor (push) Failing after 3m6s Details
Test / test_rebalance_verify_ec (push) Successful in 5m1s Details
Test / test_heal_pg_size_2 (push) Successful in 4m50s Details
Test / test_heal_ec (push) Successful in 4m34s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m5s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m57s Details
Test / test_heal_csum_32k (push) Successful in 6m56s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m28s Details
Test / test_scrub (push) Successful in 1m10s Details
Test / test_scrub_zero_osd_2 (push) Successful in 57s Details
Test / test_scrub_xor (push) Successful in 53s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m34s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m8s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m37s Details
Test / test_scrub_ec (push) Successful in 41s Details
Test / test_heal_csum_4k (push) Successful in 6m6s Details
2023-10-30 12:18:29 +03:00
Vitaliy Filippov 2e592a2f22 Fix undefined variable "timeout"
Test / test_snapshot_ec (push) Successful in 44s Details
Test / test_rm (push) Successful in 17s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m9s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_move_reappear (push) Failing after 51s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m26s Details
Test / test_snapshot_chain_ec (push) Failing after 3m6s Details
Test / test_rebalance_verify_imm (push) Successful in 3m2s Details
Test / test_write (push) Successful in 35s Details
Test / test_rebalance_verify (push) Successful in 3m56s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_rebalance_verify_ec (push) Successful in 5m2s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m13s Details
Test / test_heal_pg_size_2 (push) Successful in 4m17s Details
Test / test_heal_ec (push) Successful in 5m2s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m43s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m36s Details
Test / test_heal_csum_32k (push) Successful in 7m4s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m47s Details
Test / test_scrub (push) Successful in 1m11s Details
Test / test_scrub_zero_osd_2 (push) Successful in 59s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m26s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m32s Details
Test / test_heal_csum_4k (push) Successful in 6m31s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 27s Details
Test / test_scrub_ec (push) Successful in 25s Details
Test / test_scrub_xor (push) Failing after 3m7s Details
2023-10-29 01:30:55 +03:00
Vitaliy Filippov b92f644e3a Fix statistics aggregation, calculate inode stats by first deriving per-OSD stats, too 2023-10-29 01:30:55 +03:00
Vitaliy Filippov 890ea3dbc0 Forgot to add new parameter page to README 2023-10-28 13:39:53 +03:00
Vitaliy Filippov 06630369bf Plans++ 2023-10-28 13:38:04 +03:00
Vitaliy Filippov b4740acf62 Fix operations paused for 0.5-1 second when it happens that io_uring submit is not triggered
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 33s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_rm (push) Successful in 17s Details
Test / test_move_reappear (push) Failing after 48s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m30s Details
Test / test_snapshot_chain_ec (push) Successful in 3m4s Details
Test / test_rebalance_verify_imm (push) Successful in 2m41s Details
Test / test_write (push) Successful in 48s Details
Test / test_rebalance_verify (push) Successful in 3m42s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m17s Details
Test / test_rebalance_verify_ec (push) Successful in 4m11s Details
Test / test_write_xor (push) Failing after 3m8s Details
Test / test_heal_pg_size_2 (push) Successful in 3m40s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m9s Details
Test / test_heal_ec (push) Successful in 6m31s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m30s Details
Test / test_heal_csum_32k (push) Successful in 6m22s Details
Test / test_scrub (push) Successful in 1m14s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m20s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m23s Details
Test / test_scrub_xor (push) Successful in 1m4s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m2s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 59s Details
Test / test_scrub_ec (push) Successful in 50s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m35s Details
Test / test_heal_csum_4k (push) Successful in 6m1s Details
2023-10-28 13:18:21 +03:00
Vitaliy Filippov eae81bbda6 Fix typo 2023-10-28 01:09:20 +03:00
Vitaliy Filippov 8222e3c77d Release 1.1.0
Test / test_interrupted_rebalance_ec (push) Successful in 1m49s Details
Test / test_snapshot_ec (push) Successful in 38s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_move_reappear (push) Failing after 49s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m25s Details
Test / test_snapshot_chain_ec (push) Successful in 3m5s Details
Test / test_rebalance_verify_imm (push) Successful in 2m51s Details
Test / test_write (push) Successful in 34s Details
Test / test_rebalance_verify (push) Successful in 3m38s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_write_xor (push) Successful in 50s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m3s Details
Test / test_rebalance_verify_ec (push) Successful in 5m0s Details
Test / test_heal_pg_size_2 (push) Successful in 4m2s Details
Test / test_heal_ec (push) Successful in 4m49s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m27s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m44s Details
Test / test_heal_csum_32k (push) Successful in 6m57s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m50s Details
Test / test_scrub (push) Successful in 1m12s Details
Test / test_scrub_xor (push) Successful in 48s Details
Test / test_scrub_zero_osd_2 (push) Successful in 54s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m14s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m32s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m38s Details
Test / test_heal_csum_4k (push) Successful in 6m20s Details
Test / test_scrub_ec (push) Successful in 27s Details
New features:

- Implement [client writeback cache](docs/config/client.en.md#client_enable_writeback)
- Add the third I/O mode: [O_DIRECT|O_SYNC](docs/config/osd.en.md#data_io) (good for Optane)
- Reduce load on etcd by splitting OSD lease and statistics reporting intervals:
  [etcd_stats_interval](docs/config/osd.en.md#etcd_stats_interval) (default 30 sec)
- Make MON automatically filter OSDs by layout (block_size/immediate_commit/bitmap_granularity)
  to prevent "refusing to start PGs of this pool" errors on misconfiguration
- Support running fio benchmarks on systems without io_uring
- Make QEMU driver compatible with QEMU 8.1
- Document usage of [vhost-user-blk](docs/usage/qemu.en.md#vhost-user-blk)

Bug fixes:

- Fix resizing disks in QEMU driver (for example, in Proxmox)
- Fix "unexpected result" in Proxmox driver by making CLI flush output on exit
- Remove unneeded block_size mismatch warnings on pools without matching PGs
- Fix possible segfault in vitastor-cli ls -l (usually with deleted pools)
- Fix QEMU driver compatibility with systems without io_uring
- Fix monitor eating 100% CPU when etcd is down (caused by infinite retries)
- Fix potential incorrect write processing with snapshots (not caught in tests
  but could probably lead to client hangs)
- Fix buffer insertion in cluster_client (not caught in tests but could
  probably lead to incorrect writes in rare cases)
- Fix rare OSD crash during sync operation processing
- Fix a reenterability issue in cluster_client not reproducible in QEMU/fio,
  but reproducible with the currently developed K/V database implementation
- Fix deletion of the first modified object - OSDs could crash if you modified
  the same object a lot of times, then deleted it, and then modified it again
- Fix the fio_sec_osd test tool
2023-10-28 00:33:06 +03:00
Vitaliy Filippov 29cbe70e74 Bump qemu version to vitastor4 2023-10-28 00:33:06 +03:00
Vitaliy Filippov a883e79507 Make docs to add etcd_stats_interval 2023-10-27 14:09:26 +03:00
Vitaliy Filippov be7e76f849 Split etcd_stats_interval out of etcd_report_interval
Test / test_interrupted_rebalance_ec (push) Successful in 1m46s Details
Test / test_snapshot_ec (push) Successful in 36s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Successful in 29s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 26s Details
Test / test_snapshot_chain (push) Successful in 2m15s Details
Test / test_snapshot_chain_ec (push) Successful in 2m57s Details
Test / test_rebalance_verify_imm (push) Successful in 2m29s Details
Test / test_rebalance_verify (push) Successful in 3m40s Details
Test / test_write (push) Successful in 1m0s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_write_xor (push) Successful in 50s Details
Test / test_rebalance_verify_ec (push) Successful in 4m58s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m21s Details
Test / test_heal_ec (push) Successful in 4m5s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m36s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m28s Details
Test / test_heal_csum_32k (push) Successful in 6m38s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m46s Details
Test / test_scrub_zero_osd_2 (push) Successful in 59s Details
Test / test_scrub (push) Successful in 1m16s Details
Test / test_scrub_xor (push) Successful in 53s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m57s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m18s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m7s Details
Test / test_heal_csum_4k (push) Successful in 5m43s Details
Test / test_scrub_ec (push) Successful in 32s Details
2023-10-27 01:26:26 +03:00
Vitaliy Filippov 6fd2cf5df6 Add documentation for the write-back cache 2023-10-27 01:26:26 +03:00
Vitaliy Filippov 294a754c9e Allow write-back by default in NBD & NFS 2023-10-27 01:26:26 +03:00
Vitaliy Filippov 8bfea6e7de Support vitastor_c_create_epoll() in fio driver
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_rm (push) Successful in 21s Details
Test / test_snapshot_down (push) Successful in 27s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m29s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m35s Details
Test / test_rebalance_verify (push) Successful in 3m22s Details
Test / test_rebalance_verify_imm (push) Successful in 3m22s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_rebalance_verify_ec (push) Successful in 3m56s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m51s Details
Test / test_heal_pg_size_2 (push) Successful in 3m21s Details
Test / test_heal_ec (push) Successful in 5m45s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m43s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m10s Details
Test / test_heal_csum_32k (push) Successful in 6m29s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m22s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m51s Details
Test / test_scrub (push) Successful in 36s Details
Test / test_scrub_zero_osd_2 (push) Successful in 34s Details
Test / test_heal_csum_4k (push) Successful in 5m35s Details
Test / test_scrub_xor (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 1m31s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 42s Details
Test / test_scrub_ec (push) Successful in 20s Details
Test / test_snapshot_chain_ec (push) Successful in 1m23s Details
2023-10-26 22:57:36 +03:00
Vitaliy Filippov bac9e34836 Allow to create vitastor_c with plain epoll without uring :-) 2023-10-26 22:57:36 +03:00
Vitaliy Filippov 8aa4d492c1 Allow to use epoll_manager without ringloop 2023-10-26 22:57:36 +03:00
Vitaliy Filippov 9336ee5476 Correctly free manual "small vector" in cluster_client %-) 2023-10-26 22:57:36 +03:00
Vitaliy Filippov ad30b11519 Add the missing ringloop creation check to vitastor_c_create_uring_json()
Test / test_minsize_1 (push) Failing after 1s Details
Test / test_move_reappear (push) Failing after 1s Details
Test / test_rm (push) Failing after 0s Details
Test / test_snapshot_chain (push) Failing after 0s Details
Test / test_snapshot_chain_ec (push) Failing after 1s Details
Test / test_snapshot_down (push) Failing after 1s Details
Test / test_snapshot_down_ec (push) Failing after 0s Details
Test / test_splitbrain (push) Failing after 1s Details
Test / test_rebalance_verify (push) Failing after 1s Details
Test / test_rebalance_verify_imm (push) Failing after 1s Details
Test / test_rebalance_verify_ec (push) Failing after 1s Details
Test / test_rebalance_verify_ec_imm (push) Failing after 1s Details
Test / test_write (push) Failing after 1s Details
Test / test_write_xor (push) Failing after 1s Details
Test / test_write_no_same (push) Failing after 1s Details
Test / test_heal_pg_size_2 (push) Failing after 1s Details
Test / test_heal_ec (push) Failing after 1s Details
Test / test_heal_csum_32k_dmj (push) Failing after 1s Details
Test / test_heal_csum_32k_dj (push) Failing after 1s Details
Test / test_heal_csum_32k (push) Failing after 1s Details
Test / test_heal_csum_4k_dmj (push) Failing after 1s Details
Test / test_heal_csum_4k_dj (push) Failing after 1s Details
Test / test_heal_csum_4k (push) Failing after 1s Details
Test / test_scrub (push) Failing after 1s Details
Test / test_scrub_zero_osd_2 (push) Failing after 1s Details
Test / test_scrub_xor (push) Failing after 1s Details
Test / test_scrub_pg_size_3 (push) Failing after 1s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 0s Details
Test / test_scrub_ec (push) Failing after 1s Details
Test / build (push) Successful in 2m35s Details
2023-10-26 18:07:23 +03:00
Vitaliy Filippov a061246997 Do not attempt to initialize QEMU driver via vitastor_c_create_qemu_uring()
Test / test_snapshot_ec (push) Successful in 38s Details
Test / test_minsize_1 (push) Successful in 18s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot_down (push) Failing after 26s Details
Test / test_move_reappear (push) Failing after 52s Details
Test / test_snapshot_down_ec (push) Failing after 27s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_snapshot_chain (push) Successful in 2m28s Details
Test / test_snapshot_chain_ec (push) Successful in 3m6s Details
Test / test_rebalance_verify (push) Successful in 3m30s Details
Test / test_rebalance_verify_imm (push) Successful in 3m29s Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
It doesn't add any compatibility because vitastor_c_uring_register_eventfd()
is added in the same VITASTOR_C_API_VERSION 2.
2023-10-26 17:46:19 +03:00
Vitaliy Filippov 5066e35a49 Fix write-over-delete failing for the very first entry in dirty_db
Test / test_minsize_1 (push) Successful in 17s Details
Test / test_snapshot_ec (push) Successful in 38s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot_down (push) Failing after 27s Details
Test / test_move_reappear (push) Failing after 53s Details
Test / test_snapshot_down_ec (push) Failing after 27s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_snapshot_chain (push) Successful in 2m28s Details
Test / test_snapshot_chain_ec (push) Successful in 2m59s Details
Test / test_rebalance_verify (push) Successful in 3m22s Details
Test / test_rebalance_verify_imm (push) Successful in 3m21s Details
Test / test_write (push) Successful in 42s Details
Test / test_write_xor (push) Successful in 48s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_rebalance_verify_ec (push) Successful in 4m41s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m8s Details
Test / test_heal_pg_size_2 (push) Successful in 4m17s Details
Test / test_heal_ec (push) Successful in 4m33s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m23s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m18s Details
Test / test_heal_csum_32k (push) Successful in 6m44s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m53s Details
Test / test_scrub (push) Successful in 1m11s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m5s Details
Test / test_scrub_xor (push) Successful in 1m1s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m58s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m2s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m41s Details
Test / test_scrub_ec (push) Successful in 37s Details
Test / test_heal_csum_4k (push) Successful in 6m4s Details
2023-10-21 17:00:14 +03:00
Vitaliy Filippov 93dc31f3fc Fix possible segfault in vitastor-cli ls -l
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m21s Details
Test / test_snapshot_ec (push) Successful in 32s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot_down (push) Failing after 26s Details
Test / test_move_reappear (push) Failing after 52s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Failing after 26s Details
Test / test_snapshot_chain (push) Successful in 2m22s Details
Test / test_snapshot_chain_ec (push) Successful in 2m44s Details
Test / test_rebalance_verify (push) Successful in 3m20s Details
Test / test_rebalance_verify_imm (push) Successful in 3m20s Details
Test / test_write (push) Successful in 36s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_rebalance_verify_ec (push) Successful in 3m59s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m49s Details
Test / test_write_xor (push) Failing after 3m11s Details
Test / test_heal_pg_size_2 (push) Failing after 4m40s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m13s Details
Test / test_heal_ec (push) Successful in 5m32s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m1s Details
Test / test_heal_csum_32k (push) Successful in 6m54s Details
Test / test_scrub (push) Successful in 1m28s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m1s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m7s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m25s Details
Test / test_scrub_xor (push) Successful in 59s Details
Test / test_heal_csum_4k (push) Successful in 6m39s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 59s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m6s Details
Test / test_scrub_ec (push) Successful in 35s Details
2023-10-18 11:11:41 +03:00
Vitaliy Filippov f245b56176 Fix another possible reenterability issue in cluster_client
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m9s Details
Test / test_snapshot_ec (push) Successful in 42s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Failing after 27s Details
Test / test_move_reappear (push) Failing after 50s Details
Test / test_snapshot_down_ec (push) Failing after 26s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m26s Details
Test / test_snapshot_chain_ec (push) Successful in 2m58s Details
Test / test_rebalance_verify (push) Successful in 3m26s Details
Test / test_rebalance_verify_imm (push) Successful in 3m27s Details
Test / test_write (push) Successful in 42s Details
Test / test_write_xor (push) Successful in 51s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_rebalance_verify_ec (push) Successful in 4m56s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m11s Details
Test / test_heal_pg_size_2 (push) Successful in 4m18s Details
Test / test_heal_ec (push) Successful in 5m5s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m7s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m14s Details
Test / test_heal_csum_32k (push) Successful in 6m54s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m55s Details
Test / test_scrub (push) Successful in 1m23s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m8s Details
Test / test_scrub_xor (push) Successful in 1m0s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m15s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m7s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m40s Details
Test / test_scrub_ec (push) Successful in 42s Details
Test / test_heal_csum_4k (push) Successful in 6m17s Details
Non-reproducible in QEMU/FIO, only caught during K/V DB debugging
2023-10-08 11:02:53 +03:00
Vitaliy Filippov befca06f18 Support any OSD count in test_heal 2023-10-08 11:02:53 +03:00
Vitaliy Filippov fbf0263625 Add qemu-storage-daemon to documentation 2023-09-16 18:40:52 +03:00
Vitaliy Filippov 3bcf276d4d Run tests with writeback
Test / test_snapshot_ec (push) Successful in 35s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m56s Details
Test / test_rm (push) Successful in 21s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_chain (push) Successful in 2m24s Details
Test / test_snapshot_chain_ec (push) Successful in 2m54s Details
Test / test_rebalance_verify (push) Successful in 3m32s Details
Test / test_rebalance_verify_imm (push) Successful in 3m31s Details
Test / test_write (push) Successful in 46s Details
Test / test_write_xor (push) Successful in 52s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_rebalance_verify_ec (push) Successful in 4m52s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m3s Details
Test / test_heal_pg_size_2 (push) Successful in 3m52s Details
Test / test_heal_ec (push) Successful in 5m7s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m29s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m7s Details
Test / test_heal_csum_32k (push) Successful in 6m44s Details
Test / test_scrub (push) Successful in 1m20s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m57s Details
Test / test_scrub_xor (push) Successful in 55s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m11s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m42s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m12s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m33s Details
Test / test_scrub_ec (push) Successful in 42s Details
Test / test_heal_csum_4k (push) Successful in 6m14s Details
Test / test_move_reappear (push) Successful in 18s Details
2023-09-16 17:52:17 +03:00
Vitaliy Filippov 38db53f5ee Implement client writeback cache
- Disabled by default, enable with client_enable_writeback=true
- Even then only enabled in FIO when -direct is disabled and in QEMU when
  block device cache is enabled in settings
- Can also be enabled in other clients like vitastor-cli using parameter
  client_writeback_allowed=true, but not recommended
2023-09-16 17:52:17 +03:00
Vitaliy Filippov cd543a90bc Prevent stack overflows in cli_merge with CAS and writeback cache 2023-09-16 17:52:17 +03:00
Vitaliy Filippov f600cc07b0 Autosync in blockstore every autosync_writes, too 2023-09-16 17:52:17 +03:00
Vitaliy Filippov 6a8e530e6b Add FIXME to timerfd_manager 2023-09-16 17:52:17 +03:00
Vitaliy Filippov 5cadb170b9 Fix possible OSD crash during sync due to missing min_flushed_journal_sector reset 2023-09-16 17:52:17 +03:00
Vitaliy Filippov e72d4ed1d4 Remove unused bs_sync fields 2023-09-16 17:52:17 +03:00
Vitaliy Filippov ff479a102d Make MON filter OSDs by block layout to prevent "refusing to start PGs of this pool" errors on misconfiguration 2023-09-16 17:52:17 +03:00
Vitaliy Filippov 27d0d5b06a Reads do not have to wait for buffer flushes anymore 2023-09-16 17:52:17 +03:00
Vitaliy Filippov 33950c1ec8 Fix fio_sec_osd attr_len
Test / test_failure_domain (push) Has been cancelled Details
Test / test_snapshot (push) Has been cancelled Details
Test / test_snapshot_ec (push) Has been cancelled Details
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot_chain (push) Has been cancelled Details
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / build (push) Has been cancelled Details
2023-09-16 17:49:10 +03:00
Vitaliy Filippov eea7ef1f19 Remove debug osd_trace from test_write
Test / test_snapshot_ec (push) Successful in 41s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_rm (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_snapshot_down (push) Successful in 32s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_snapshot_chain (push) Successful in 2m18s Details
Test / test_snapshot_chain_ec (push) Successful in 3m1s Details
Test / test_rebalance_verify (push) Successful in 3m24s Details
Test / test_rebalance_verify_imm (push) Successful in 3m26s Details
Test / test_write (push) Successful in 45s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_write_xor (push) Successful in 49s Details
Test / test_rebalance_verify_ec (push) Successful in 4m59s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m56s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m39s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m9s Details
Test / test_heal_csum_32k (push) Successful in 6m8s Details
Test / test_scrub (push) Successful in 1m26s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m53s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m15s Details
Test / test_scrub_xor (push) Successful in 57s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m18s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 59s Details
Test / test_scrub_ec (push) Successful in 57s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m37s Details
Test / test_heal_csum_4k (push) Successful in 5m21s Details
Test / test_heal_ec (push) Successful in 7m21s Details
Test / test_heal_pg_size_2 (push) Successful in 3m59s Details
2023-09-12 01:35:36 +03:00
Vitaliy Filippov cc0fdc6253 Remove erroneous block_size mismatch warnings on pools without matching PGs
Test / test_snapshot_ec (push) Successful in 36s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_move_reappear (push) Failing after 52s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m23s Details
Test / test_snapshot_chain_ec (push) Successful in 2m58s Details
Test / test_rebalance_verify (push) Successful in 3m32s Details
Test / test_rebalance_verify_imm (push) Successful in 3m29s Details
Test / test_write (push) Successful in 52s Details
Test / test_write_xor (push) Successful in 56s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify_ec (push) Successful in 5m0s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m30s Details
Test / test_heal_ec (push) Successful in 4m6s Details
Test / test_heal_pg_size_2 (push) Failing after 4m19s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m2s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m12s Details
Test / test_heal_csum_32k (push) Successful in 6m24s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m19s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m8s Details
Test / test_scrub (push) Successful in 1m15s Details
Test / test_scrub_xor (push) Successful in 1m8s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m45s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m58s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m9s Details
Test / test_scrub_ec (push) Successful in 42s Details
Test / test_heal_csum_4k (push) Successful in 5m26s Details
2023-09-08 23:19:04 +03:00
Vitaliy Filippov 79ecd59b10 Flush STDOUT and STDERR before exiting from cli to fix Proxmox "Unexpected result"
Test / test_interrupted_rebalance_ec (push) Successful in 1m51s Details
Test / test_snapshot_ec (push) Successful in 33s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot_down (push) Successful in 29s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m18s Details
Test / test_snapshot_chain_ec (push) Successful in 2m55s Details
Test / test_rebalance_verify (push) Successful in 3m15s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_write (push) Successful in 47s Details
Test / test_write_xor (push) Successful in 57s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_rebalance_verify_ec (push) Successful in 5m3s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m24s Details
Test / test_heal_pg_size_2 (push) Failing after 3m44s Details
Test / test_heal_ec (push) Successful in 4m55s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m48s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m11s Details
Test / test_heal_csum_32k (push) Successful in 6m14s Details
Test / test_scrub (push) Successful in 1m32s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m33s Details
Test / test_scrub_xor (push) Successful in 59s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m7s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m25s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m21s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m46s Details
Test / test_scrub_ec (push) Successful in 53s Details
Test / test_heal_csum_4k (push) Successful in 5m20s Details
2023-09-07 17:30:26 +03:00
Vitaliy Filippov 51081c9b45 Put etcd into tmpfs for tests
Test / test_minsize_1 (push) Successful in 16s Details
Test / test_rm (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_snapshot_ec (push) Successful in 25s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 31s Details
Test / test_splitbrain (push) Successful in 26s Details
Test / test_snapshot_chain (push) Successful in 2m17s Details
Test / test_snapshot_chain_ec (push) Successful in 2m58s Details
Test / test_rebalance_verify (push) Successful in 3m18s Details
Test / test_rebalance_verify_imm (push) Successful in 3m11s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_rebalance_verify_ec (push) Successful in 4m0s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m16s Details
Test / test_heal_ec (push) Successful in 4m44s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m38s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m36s Details
Test / test_heal_csum_32k (push) Successful in 6m35s Details
Test / test_scrub (push) Successful in 1m38s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m39s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m17s Details
Test / test_scrub_xor (push) Successful in 57s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m4s Details
Test / test_heal_csum_4k (push) Successful in 6m8s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 48s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m9s Details
Test / test_scrub_ec (push) Successful in 29s Details
Test / test_write_xor (push) Successful in 1m13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m24s Details
2023-09-07 02:35:09 +03:00
Vitaliy Filippov b7d398be5b Fix sscanf validation usage (field count instead of null_byte == 0)
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot_chain (push) Has been cancelled Details
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / build (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
2023-09-07 02:34:35 +03:00
Vitaliy Filippov 85e9f67d9d Add supported_truncate_flags
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m19s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_snapshot_chain (push) Successful in 2m13s Details
Test / test_snapshot_chain_ec (push) Successful in 2m54s Details
Test / test_rebalance_verify (push) Successful in 3m25s Details
Test / test_rebalance_verify_imm (push) Successful in 3m22s Details
Test / test_write (push) Successful in 52s Details
Test / test_write_xor (push) Successful in 55s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify_ec (push) Successful in 4m50s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m8s Details
Test / test_heal_ec (push) Successful in 5m1s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m32s Details
Test / test_heal_csum_32k (push) Successful in 6m14s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m23s Details
Test / test_scrub (push) Successful in 1m27s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m24s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m15s Details
Test / test_scrub_xor (push) Successful in 58s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m1s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 55s Details
Test / test_scrub_ec (push) Successful in 53s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m24s Details
Test / test_heal_csum_4k (push) Successful in 4m44s Details
Test / test_move_reappear (push) Successful in 20s Details
Test / test_heal_pg_size_2 (push) Successful in 4m1s Details
2023-09-06 17:37:52 +03:00
Vitaliy Filippov 79c6d6f323 Make QEMU driver compatible with QEMU 8.1
Test / test_snapshot_ec (push) Successful in 31s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 2m2s Details
Test / test_snapshot_chain_ec (push) Successful in 2m44s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec (push) Successful in 3m39s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 7m1s Details
Test / test_write (push) Successful in 36s Details
Test / test_write_xor (push) Successful in 1m11s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m1s Details
Test / test_heal_ec (push) Successful in 6m7s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m19s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m7s Details
Test / test_heal_csum_32k (push) Successful in 6m21s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m22s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m40s Details
Test / test_heal_csum_4k (push) Successful in 5m34s Details
Test / test_scrub (push) Successful in 1m2s Details
Test / test_scrub_zero_osd_2 (push) Successful in 46s Details
Test / test_scrub_xor (push) Successful in 38s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m7s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 50s Details
Test / test_scrub_ec (push) Successful in 48s Details
2023-08-24 02:23:55 +03:00
Vitaliy Filippov ae760dbc1d Fix co_truncate size division by BDRV_SECTOR_SIZE 2023-08-24 01:55:35 +03:00
Vitaliy Filippov 65487da4b1 Do not include msgr_rdma.h into messenger.h 2023-08-24 01:55:35 +03:00
Vitaliy Filippov 7862282938 Extract validation to check_rw(), remove duplicate code with OP_SYNC
Test / test_snapshot_ec (push) Successful in 19s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 59s Details
Test / test_snapshot_chain_ec (push) Successful in 1m41s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_rebalance_verify (push) Successful in 3m23s Details
Test / test_rebalance_verify_imm (push) Successful in 3m19s Details
Test / test_rebalance_verify_ec (push) Successful in 5m33s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m28s Details
Test / test_write (push) Successful in 51s Details
Test / test_write_xor (push) Successful in 1m6s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_heal_pg_size_2 (push) Successful in 4m26s Details
Test / test_heal_ec (push) Successful in 5m21s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m56s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m2s Details
Test / test_heal_csum_32k (push) Successful in 6m41s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m53s Details
Test / test_heal_csum_4k_dj (push) Successful in 3m58s Details
Test / test_heal_csum_4k (push) Successful in 4m55s Details
Test / test_scrub (push) Successful in 48s Details
Test / test_scrub_zero_osd_2 (push) Successful in 54s Details
Test / test_scrub_xor (push) Successful in 50s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m4s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 46s Details
Test / test_scrub_ec (push) Successful in 47s Details
2023-08-13 23:49:52 +03:00
Vitaliy Filippov 30ce2bd951 Fix buffer insert in cluster_client 2023-08-12 11:08:50 +03:00
Vitaliy Filippov b1a0afd10a Aggregate buffer flushes
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot_chain (push) Successful in 2m21s Details
Test / test_snapshot_chain_ec (push) Successful in 2m56s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_rebalance_verify (push) Successful in 3m39s Details
Test / test_rebalance_verify_imm (push) Successful in 3m35s Details
Test / test_rebalance_verify_ec (push) Successful in 5m6s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m8s Details
Test / test_write (push) Successful in 50s Details
Test / test_write_xor (push) Successful in 55s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m14s Details
Test / test_heal_ec (push) Successful in 5m2s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m1s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m12s Details
Test / test_heal_csum_32k (push) Successful in 6m35s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m18s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m24s Details
Test / test_heal_csum_4k (push) Successful in 5m12s Details
Test / test_scrub (push) Successful in 1m39s Details
Test / test_scrub_zero_osd_2 (push) Successful in 49s Details
Test / test_scrub_xor (push) Successful in 45s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m19s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 57s Details
Test / test_scrub_ec (push) Successful in 59s Details
2023-08-11 11:26:13 +03:00
Vitaliy Filippov 85b6134910 Return dirty buffers on read in client
Test / test_snapshot_ec (push) Successful in 22s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 1m57s Details
Test / test_snapshot_chain_ec (push) Successful in 2m39s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m4s Details
Test / test_rebalance_verify_imm (push) Successful in 3m7s Details
Test / test_rebalance_verify_ec (push) Successful in 3m26s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 7m35s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_write_no_same (push) Successful in 21s Details
Test / test_heal_pg_size_2 (push) Successful in 4m58s Details
Test / test_heal_ec (push) Successful in 3m15s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m52s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m3s Details
Test / test_heal_csum_32k (push) Successful in 5m26s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m12s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m9s Details
Test / test_heal_csum_4k (push) Successful in 5m13s Details
Test / test_scrub (push) Successful in 1m24s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m15s Details
Test / test_scrub_xor (push) Successful in 1m5s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m45s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m11s Details
Test / test_scrub_ec (push) Successful in 56s Details
Required at least to return buffers when they need to be replayed, but until
they are actually replayed
2023-08-09 00:57:08 +03:00
Vitaliy Filippov b1b07a393d Fix incorrect marking op parts as done with snapshots (could probably lead to client hangs) 2023-08-09 00:57:08 +03:00
Vitaliy Filippov 7333022adf Add a third I/O mode: O_DIRECT|O_SYNC, change parameters to data_io/meta_io/journal_io 2023-08-09 00:57:08 +03:00
Vitaliy Filippov ab8627c9fa Fix monitor retrying failed etcd connection in an infinite loop without pauses 2023-08-09 00:57:08 +03:00
Vitaliy Filippov 6acf562e01 Release 1.0.0
New features:

- Data and metadata checksums!
  - Metadata checksums are always used with new disk format
  - Data checksums can be turned on with --data_csum_type crc32c for new OSDs
  - Checksum block size can be configured
  - inmemory_metadata now also affects keeping checksums in memory
- Linux page cache I/O caching support which can be enabled separately for
  data, metadata (including checksums) and journal (O_SYNC instead of O_DIRECT)
- Details [here](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/config/layout-osd.en.md#data_csum_type)
- Backwards compatibility is preserved, you can use new OSDs with old disks

Release also includes bug fixes from [0.9.6](https://git.yourcmc.ru/vitalif/vitastor/releases/tag/v0.9.6).

0.9.6 is moved to "-oldstable" repositories and will be available for some additional time.
2023-07-29 18:57:19 +03:00
Vitaliy Filippov 6f797f429e Add a note about -oldstable 2023-07-29 18:11:13 +03:00
Vitaliy Filippov b8a1734465 Reword checksum docs one more time 2023-07-29 14:42:56 +03:00
Vitaliy Filippov c752b68167 Remove "without checksums" from docs :) 2023-07-29 12:19:03 +03:00
Vitaliy Filippov 564df2eb5d Support using buffered I/O with O_SYNC instead of direct I/O 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 9a427dd70a Allow to override OSD devices in tests 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 1a4ceb420d Track used blocks, not object versions 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 21b5124a4b Document data_csum_type and csum_block_size parameters 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 4181add1f4 Remove creepy "metadata copying" during overwrite
Instead of it, just do not verify checksums of currently mutated objects.
When clean data modification during flush runs in parallel to a read request,
that request may read a mix of old and new data. It may even read a mix of
multiple flushed versions if it lasts too long... And attempts to verify it
using temporary copies of metadata make the algorithm too complex and creepy.
2023-07-29 12:17:18 +03:00
Vitaliy Filippov a8464c19af Support keeping checksums on disk (not in memory)
Definitely beneficial for SSD+HDD setups
2023-07-29 12:17:18 +03:00
Vitaliy Filippov 819cb70cdd Check for "Checksum mismatch" and "BUG" messages during test_heal 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 3c8e4c6b72 Use clean_dyn_size for space check 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 8ef4cf89dc Log more details about checksum mismatch in big_writes 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 7bfb1639ea Use find_holes() in flusher for unification 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 628e481c32 Fill journal header to know checksum type & size when dumping journal with --all 2023-07-29 12:17:18 +03:00
Vitaliy Filippov af6f2046fc Fix journal read checksum verification with inmemory_journal=false 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 9357e5293e Call fill_partial_checksum_blocks() correctly in regard to COPY_BUF_CSUM_FILL 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 12851dc07d Wait for journal reads before checking them in clear_incomplete_csum_block_bits 2023-07-29 12:17:18 +03:00
Vitaliy Filippov a5753e35a3 Check for checksum mismatch absence in test_heal 2023-07-29 12:17:18 +03:00
Vitaliy Filippov d6ee1ca17c Use zero checksum size for zero-length writes 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 71674d00cf Fix journal data checksum mangling on corrupted block overwrite 2023-07-29 12:17:18 +03:00
Vitaliy Filippov ddb078d5a7 Check journal entry size when checking block checksums 2023-07-29 12:17:18 +03:00
Vitaliy Filippov d22d56f90a Fix journal data checksum verification on start 2023-07-29 12:17:18 +03:00
Vitaliy Filippov eb1331a079 Add more details to "journal entry data is corrupt" messages 2023-07-29 12:17:18 +03:00
Vitaliy Filippov c5274f655b ...and partially remove the perversion with bitmap inlining 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 45e07d6294 Sadly we have to refcount dyn_data... 2023-07-29 12:17:18 +03:00
Vitaliy Filippov a8ee391e05 Fix clean block checksum read 2023-07-29 12:17:18 +03:00
Vitaliy Filippov de48fa3fd2 Allow to forcibly set meta_format 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 874a766b62 Rename meta_version to meta_format 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 384bd8e28f Support old metadata format in vitastor-disk dump-meta 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 430994f48a Fix journal big_write simple reads after checksum changes 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 3d7f838c59 Verify checksums in test_heal in different combinations 2023-07-29 12:17:18 +03:00
Vitaliy Filippov b909d81f41 Fix bitmap-granular checksums 2023-07-29 12:17:18 +03:00
Vitaliy Filippov e42975ffd1 Fix wait_journal_count not being zeroed 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 93778324e5 Rewrite and fix find_holes into a more obvious version 2023-07-29 12:17:18 +03:00
Vitaliy Filippov eeb6727170 Fix missing checksum read offset 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 7fe82c692e Add a test for checksums 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 92c6e16eba Fix checksum verification in big_write journal reads 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 213a9ccb4d Verify checksums during journal reads 2023-07-29 12:17:18 +03:00
Vitaliy Filippov a166147110 Add backwards compatibility with non-checksum metadata and journal formats 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 7d532880c3 Implement large csum_block_size support (more than 4k) + refactor blockstore_flush 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 0b0405d115 Implement bitmap-granular (4k) metadata & data checksums 2023-07-29 12:17:18 +03:00
Vitaliy Filippov e651c93a90 Release 0.9.6
Test / test_interrupted_rebalance (push) Successful in 1m59s Details
Test / test_interrupted_rebalance_imm (push) Successful in 3m41s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m29s Details
Test / test_failure_domain (push) Successful in 50s Details
Test / test_snapshot (push) Successful in 45s Details
Test / test_snapshot_ec (push) Successful in 23s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_chain (push) Successful in 2m23s Details
Test / test_snapshot_chain_ec (push) Successful in 3m1s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_rebalance_verify (push) Successful in 3m34s Details
Test / test_rebalance_verify_imm (push) Successful in 3m31s Details
Test / test_rebalance_verify_ec (push) Successful in 5m14s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m14s Details
Test / test_write (push) Successful in 44s Details
Test / test_write_xor (push) Successful in 54s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m38s Details
Test / test_heal_ec (push) Successful in 3m56s Details
Test / test_scrub (push) Successful in 36s Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 46s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 25s Details
Test / test_scrub_ec (push) Successful in 24s Details
- Fix vitastor-disk partition zeroing (sometimes it was writing garbage instead of zeroes)
- Fix incorrect EC space statistics in `vitastor-cli status`
- Several bug fixes for NFS:
  - Add . and .. in NFS directory listings
  - Return FILE_SYNC from NFS writes if immediate_commit is enabled
  - Return the same "verifier" in NFS COMMIT as in NFS WRITE
  - Make parallel NFS extending writes work correctly, without conflicts
  - Handle parallel NFS extending writes without imposing extra load on etcd
- Support UTF-8 in vitastor-cli table output
- Also allow "0" and "no" as false for inmemory_metadata and inmemory_journal
- Use HDD defaults for HDD-only in automatic `vitastor-disk prepare` mode
2023-07-29 10:54:00 +03:00
Vitaliy Filippov 988e90be69 Fix vitastor-disk partition zeroing (it was writing random garbage instead of zeroes :D)
Test / test_interrupted_rebalance (push) Successful in 1m59s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m38s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m12s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m18s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 20s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m3s Details
Test / test_snapshot_chain_ec (push) Successful in 1m25s Details
Test / test_snapshot_down (push) Successful in 20s Details
Test / test_snapshot_down_ec (push) Successful in 20s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 3m16s Details
Test / test_rebalance_verify_imm (push) Successful in 3m40s Details
Test / test_rebalance_verify_ec (push) Successful in 4m17s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m20s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_xor (push) Successful in 33s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m54s Details
Test / test_heal_ec (push) Successful in 4m7s Details
Test / test_scrub (push) Successful in 58s Details
Test / test_scrub_zero_osd_2 (push) Successful in 53s Details
Test / test_scrub_xor (push) Successful in 40s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m6s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 37s Details
Test / test_scrub_ec (push) Successful in 39s Details
2023-07-28 12:29:07 +03:00
Vitaliy Filippov 272a45ad63 Fix modprobe command in docs 2023-07-27 23:57:02 +03:00
Vitaliy Filippov 25a15d24cf Fix incorrect EC space statistics in `vitastor-cli status`
Test / test_etcd_fail (push) Successful in 1m21s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m9s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m52s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m25s Details
Test / test_failure_domain (push) Successful in 10s Details
Test / test_snapshot (push) Successful in 28s Details
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 2m1s Details
Test / test_snapshot_chain_ec (push) Successful in 2m41s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 17s Details
Test / test_rebalance_verify (push) Successful in 3m9s Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec (push) Successful in 3m23s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m38s Details
Test / test_write (push) Successful in 33s Details
Test / test_write_xor (push) Successful in 43s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m16s Details
Test / test_heal_ec (push) Successful in 5m0s Details
Test / test_scrub (push) Successful in 56s Details
Test / test_scrub_zero_osd_2 (push) Successful in 41s Details
Test / test_scrub_xor (push) Successful in 32s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 45s Details
Test / test_scrub_ec (push) Successful in 40s Details
2023-07-27 02:26:17 +00:00
Vitaliy Filippov 700e0e9bff Handle parallel NFS extending writes without imposing extra load on etcd 2023-07-27 02:26:17 +00:00
Vitaliy Filippov ab0ca7c00f Return FILE_SYNC from NFS writes if immediate_commit is enabled 2023-07-26 02:09:47 +03:00
Vitaliy Filippov f153bc950b Return the same "verifier" in NFS COMMIT as in NFS WRITE
This fixes buffered (not O_DIRECT) NFS writes in Linux - previously they were
hanging in an infinite loop because COMMIT didn't return the same verifier as
previous WRITEs, and NFS kernel client was infinitely retrying the same writes.

Also this probably allows for correct NFS failover, at least for the same
buffered writes, because NFS clients repeat all write requests until a COMMIT
confirms them.
2023-07-26 02:09:47 +03:00
Vitaliy Filippov 425ff8818d Add . and .. in NFS directory listings
MC, for example, hangs with infinite listing retries without them
2023-07-26 02:09:47 +03:00
Vitaliy Filippov 9e287a7778 Handle extending writes correctly in NFS proxy
Previously, multiple parallel writes extending file size through NFS were
racing with each other and triggering deletions of part of the written data

I.e. if you mounted vitastor-nfs and just copied a file into it in MC then
you could end up with only a part of the file actually written
2023-07-26 02:09:43 +03:00
Vitaliy Filippov f52f58b9e9 Support UTF-8 in vitastor-cli table output 2023-07-25 01:48:57 +00:00
Vitaliy Filippov 1fe6b0c0e2 Also allow "0" and "no" as false for inmemory_metadata and inmemory_journal 2023-07-25 01:48:57 +00:00
Vitaliy Filippov e4237e9ed8 Enable HDD defaults for HDD-only in automatic `vitastor-disk prepare` mode
Test / test_interrupted_rebalance (push) Successful in 2m20s Details
Test / test_interrupted_rebalance_imm (push) Failing after 10m5s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m4s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m19s Details
Test / test_failure_domain (push) Successful in 39s Details
Test / test_snapshot (push) Successful in 34s Details
Test / test_snapshot_ec (push) Successful in 23s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m59s Details
Test / test_snapshot_chain_ec (push) Successful in 2m40s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 3m11s Details
Test / test_rebalance_verify_ec (push) Successful in 3m29s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m54s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_xor (push) Successful in 35s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_heal_pg_size_2 (push) Successful in 3m59s Details
Test / test_heal_ec (push) Successful in 5m12s Details
Test / test_scrub (push) Successful in 59s Details
Test / test_scrub_zero_osd_2 (push) Successful in 47s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 38s Details
Test / test_scrub_ec (push) Successful in 22s Details
2023-07-23 02:33:22 +03:00
Vitaliy Filippov 10a5fd6abb Release 0.9.5
Test / test_etcd_fail (push) Successful in 50s Details
Test / test_interrupted_rebalance (push) Successful in 2m27s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m39s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m49s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m29s Details
Test / test_failure_domain (push) Successful in 19s Details
Test / test_snapshot (push) Successful in 31s Details
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot_chain (push) Successful in 1m49s Details
Test / test_snapshot_chain_ec (push) Successful in 2m51s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 18s Details
Test / test_rebalance_verify (push) Successful in 3m8s Details
Test / test_rebalance_verify_imm (push) Successful in 3m13s Details
Test / test_rebalance_verify_ec (push) Successful in 3m36s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m59s Details
Test / test_write (push) Successful in 48s Details
Test / test_write_xor (push) Successful in 37s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s Details
Test / test_heal_ec (push) Successful in 4m6s Details
Test / test_scrub (push) Successful in 35s Details
Test / test_scrub_zero_osd_2 (push) Successful in 34s Details
Test / test_scrub_xor (push) Successful in 42s Details
Test / test_scrub_pg_size_3 (push) Successful in 52s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 33s Details
Test / test_scrub_ec (push) Successful in 28s Details
A hotfix to 0.9.4 containing only one bugfix: 100% CPU usage in the new QEMU
driver caused by the lack of eventfd reset on io_uring event handling :)
2023-07-21 00:04:41 +03:00
Vitaliy Filippov 1c316ef350 Reset eventfd on every ringloop::loop() 2023-07-21 00:04:41 +03:00
Vitaliy Filippov 0b2d12eef1 Remove has_work, it was unnecessary 2023-07-21 00:04:37 +03:00
Vitaliy Filippov 1c10430ae1 Release 0.9.4
Test / test_interrupted_rebalance (push) Successful in 1m54s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m4s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m40s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m25s Details
Test / test_failure_domain (push) Successful in 15s Details
Test / test_snapshot (push) Successful in 25s Details
Test / test_snapshot_ec (push) Successful in 20s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 1m56s Details
Test / test_snapshot_chain_ec (push) Successful in 2m33s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m3s Details
Test / test_rebalance_verify_imm (push) Successful in 3m2s Details
Test / test_rebalance_verify_ec (push) Successful in 3m13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 8m35s Details
Test / test_write (push) Successful in 33s Details
Test / test_write_xor (push) Successful in 40s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m25s Details
Test / test_heal_ec (push) Successful in 3m9s Details
Test / test_scrub (push) Successful in 1m0s Details
Test / test_scrub_zero_osd_2 (push) Successful in 46s Details
Test / test_scrub_xor (push) Successful in 1m1s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m55s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m25s Details
Test / test_scrub_ec (push) Successful in 52s Details
- Improve QEMU driver performance by integrating io_uring in it (up to 1.5x total iops improvement)
- Fix QEMU driver deadlocks which started to reproduce in qemu-img after iothread fixes
- Fix `vitastor-cli status` reporting more etcds than actually exists (fix etcd address duplication in config on reload)
- Fix `vitastor-cli ls` crashing on inodes in non-existing pools
- Delete old garbage /pool/stats/ keys for non-existing (deleted) pools
- Reduce memory usage of etcds initialized by make-etcd script
- Fix OSDs almost always crashing on etcd restart due to "revisions were compacted" (support reloading state from etcd)
- Fix a crash and a stall possible mostly in HDD setups with small journal and big (512k, 900k) random writes
- Add notes about HDDs to documentation. You are officially allowed to use HDD-only Vitastor with HGST/Toshiba/EXOS :)
2023-07-19 02:50:30 +03:00
Vitaliy Filippov dfce91d168 Change git url in docs, correct block/vitastor.c path 2023-07-19 01:02:12 +03:00
Vitaliy Filippov 332a13ba30 Build patched QEMU against local packages 2023-07-19 00:05:02 +03:00
Vitaliy Filippov d0e257ee81 Fix non-existing pool handling in `vitastor-cli ls`
Test / test_interrupted_rebalance (push) Successful in 2m6s Details
Test / test_interrupted_rebalance_imm (push) Successful in 3m11s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m6s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m20s Details
Test / test_failure_domain (push) Successful in 22s Details
Test / test_snapshot (push) Successful in 50s Details
Test / test_snapshot_ec (push) Successful in 33s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 1m23s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 2m22s Details
Test / test_snapshot_chain_ec (push) Successful in 3m6s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_rebalance_verify (push) Successful in 3m28s Details
Test / test_rebalance_verify_imm (push) Successful in 3m27s Details
Test / test_rebalance_verify_ec (push) Successful in 9m10s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 9m29s Details
Test / test_write (push) Successful in 1m36s Details
Test / test_write_xor (push) Successful in 2m17s Details
Test / test_write_no_same (push) Successful in 36s Details
Test / test_heal_pg_size_2 (push) Successful in 6m27s Details
Test / test_heal_ec (push) Successful in 5m53s Details
Test / test_scrub (push) Successful in 44s Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s Details
Test / test_scrub_xor (push) Successful in 36s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m1s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 46s Details
Test / test_scrub_ec (push) Successful in 36s Details
2023-07-18 23:52:02 +03:00
Vitaliy Filippov 004912aac0 Add RPM spec patches for 6.2-el8 and 7.2-el9
Test / test_interrupted_rebalance (push) Successful in 1m57s Details
Test / test_interrupted_rebalance_imm (push) Successful in 3m16s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m52s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m25s Details
Test / test_failure_domain (push) Failing after 47s Details
Test / test_snapshot (push) Successful in 40s Details
Test / test_snapshot_ec (push) Successful in 24s Details
Test / test_minsize_1 (push) Successful in 16s Details
Test / test_move_reappear (push) Failing after 52s Details
Test / test_rm (push) Successful in 19s Details
Test / test_snapshot_chain (push) Successful in 2m27s Details
Test / test_snapshot_chain_ec (push) Failing after 3m9s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 21s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_rebalance_verify (push) Successful in 3m34s Details
Test / test_rebalance_verify_imm (push) Successful in 3m32s Details
Test / test_rebalance_verify_ec (push) Successful in 5m14s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m18s Details
Test / test_write (push) Successful in 49s Details
Test / test_write_xor (push) Successful in 58s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m55s Details
Test / test_heal_ec (push) Failing after 10m39s Details
Test / test_scrub (push) Successful in 33s Details
Test / test_scrub_zero_osd_2 (push) Successful in 29s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m1s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 44s Details
Test / test_scrub_ec (push) Successful in 25s Details
2023-07-18 23:38:14 +03:00
Vitaliy Filippov c18e92273e Copy qemu 5.1 -> 5.2 patch for convenience 2023-07-18 23:37:53 +03:00
Vitaliy Filippov 9815d70ffc It is impossible to use io_uring with older vitastor-client because it does not have vitastor_c_uring_has_work() 2023-07-18 23:37:53 +03:00
Vitaliy Filippov 4a4627dcab Do not use bool in C library 2023-07-18 23:37:53 +03:00
Vitaliy Filippov b963f2fd93 Add QEMU 2.12 patch (basically the same as 3.1) 2023-07-18 23:37:06 +03:00
Vitaliy Filippov ba7427020e Fix deadlocks possible in qemu-img after fixing iothread
Deadlock was caused by switching QEMU coroutines directly inside
vitastor_co_read_bitmap_cb() callback. The correct way is to schedule a BH
/BH is a QEMU term for setImmediate() :)/, same as in read and write callbacks.
2023-07-18 23:32:16 +03:00
Vitaliy Filippov a0aac7eb2a Update drives
Test / test_interrupted_rebalance (push) Failing after 1m42s Details
Test / test_interrupted_rebalance_imm (push) Failing after 1m37s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m48s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m25s Details
Test / test_failure_domain (push) Successful in 19s Details
Test / test_snapshot (push) Successful in 25s Details
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_move_reappear (push) Failing after 49s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_chain (push) Successful in 2m24s Details
Test / test_snapshot_chain_ec (push) Successful in 3m0s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_rebalance_verify (push) Successful in 3m39s Details
Test / test_rebalance_verify_imm (push) Successful in 3m38s Details
Test / test_rebalance_verify_ec (push) Successful in 5m29s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m32s Details
Test / test_write (push) Successful in 54s Details
Test / test_write_xor (push) Successful in 1m5s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 3m51s Details
Test / test_heal_ec (push) Successful in 4m16s Details
Test / test_scrub (push) Successful in 35s Details
Test / test_scrub_zero_osd_2 (push) Successful in 32s Details
Test / test_scrub_xor (push) Successful in 32s Details
Test / test_scrub_pg_size_3 (push) Successful in 55s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 48s Details
Test / test_scrub_ec (push) Successful in 27s Details
2023-07-13 01:49:45 +03:00
Vitaliy Filippov ac7b834af3 Disable journal_no_same_sector_overwrites by default for HDD-only
Test / test_interrupted_rebalance (push) Successful in 1m17s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m29s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m31s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_snapshot (push) Successful in 20s Details
Test / test_snapshot_ec (push) Successful in 21s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 39s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m10s Details
Test / test_snapshot_chain_ec (push) Successful in 2m9s Details
Test / test_snapshot_down (push) Successful in 21s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_rebalance_verify (push) Successful in 2m56s Details
Test / test_rebalance_verify_imm (push) Successful in 2m53s Details
Test / test_rebalance_verify_ec (push) Successful in 5m39s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m44s Details
Test / test_write (push) Successful in 49s Details
Test / test_write_xor (push) Successful in 59s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m41s Details
Test / test_heal_ec (push) Successful in 3m49s Details
Test / test_scrub (push) Successful in 33s Details
Test / test_scrub_zero_osd_2 (push) Successful in 30s Details
Test / test_scrub_xor (push) Successful in 25s Details
Test / test_scrub_pg_size_3 (push) Successful in 40s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s Details
Test / test_scrub_ec (push) Successful in 33s Details
2023-07-10 00:34:35 +03:00
Vitaliy Filippov ee0c78fd74 Fix default HDD block size in docs (actual size is 1 MB) 2023-07-09 13:12:30 +03:00
Vitaliy Filippov e6646a5b2f Bump QEMU version to vitastor3 2023-07-09 13:01:04 +03:00
Vitaliy Filippov ae69662b17 Add "Recommended drives" 2023-07-09 12:59:18 +03:00
Vitaliy Filippov 57ad4c3636 Add a note about HDD, enable throttling only for hybrid OSDs
Test / test_interrupted_rebalance (push) Successful in 2m4s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m30s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m45s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m23s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 19s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Failing after 1m32s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m8s Details
Test / test_snapshot_chain_ec (push) Successful in 2m2s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 21s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 2m52s Details
Test / test_rebalance_verify_imm (push) Successful in 2m46s Details
Test / test_rebalance_verify_ec (push) Successful in 5m9s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m5s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_xor (push) Successful in 49s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s Details
Test / test_heal_ec (push) Successful in 4m35s Details
Test / test_scrub (push) Successful in 37s Details
Test / test_scrub_zero_osd_2 (push) Successful in 40s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 40s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s Details
Test / test_scrub_ec (push) Successful in 22s Details
2023-07-09 12:45:11 +03:00
Vitaliy Filippov b7e4d0c9bf Fix journal dirty_start position tracking and some debug prints
Fixes two bugs found during HDD testing :-)
1) OSD crashed with "BUG: Attempt to overwrite used offset of the journal" during
   `fio -bs=900k -iodepth=128` test with 16 MB journal
2) OSD stalled during `fio -bs=512k -iodepth=128` test with 64 MB journal
2023-07-09 01:17:55 +03:00
Vitaliy Filippov 161a23c966 Support reloading state when etcd says "revisions were compacted"
Test / test_interrupted_rebalance (push) Successful in 3m9s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m38s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m54s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m36s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_snapshot (push) Successful in 23s Details
Test / test_snapshot_ec (push) Successful in 22s Details
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 2m2s Details
Test / test_snapshot_chain_ec (push) Successful in 2m38s Details
Test / test_snapshot_down (push) Successful in 21s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec (push) Successful in 3m27s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m2s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 45s Details
Test / test_write_no_same (push) Successful in 22s Details
Test / test_heal_pg_size_2 (push) Successful in 4m0s Details
Test / test_heal_ec (push) Successful in 3m52s Details
Test / test_scrub (push) Successful in 1m1s Details
Test / test_scrub_zero_osd_2 (push) Successful in 42s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 45s Details
Test / test_scrub_ec (push) Successful in 26s Details
Before this change, OSDs almost always died when one of the etcds was restarted,
even though the rest of them was still in quorum and the lease was still active
2023-07-07 01:33:48 +03:00
Vitaliy Filippov 2f999d8607 Reduce etcd memory usage
Test / test_etcd_fail (push) Successful in 1m34s Details
Test / test_interrupted_rebalance (push) Successful in 1m52s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m4s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m13s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 20s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m56s Details
Test / test_snapshot_chain_ec (push) Successful in 2m36s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m6s Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec (push) Successful in 3m27s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m30s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 41s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 3m48s Details
Test / test_heal_ec (push) Successful in 4m15s Details
Test / test_scrub (push) Successful in 33s Details
Test / test_scrub_zero_osd_2 (push) Successful in 31s Details
Test / test_scrub_xor (push) Successful in 28s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s Details
Test / test_scrub_ec (push) Successful in 28s Details
With default --snapshot-count 100000 and GOGC=100 it easily reaches 6.6 GB
even when we only store 1-2 MB of data in it
2023-07-06 00:46:26 +03:00
Vitaliy Filippov d007a374f2 Delete extra /pool/stats/ keys for non-existing pools
Test / test_interrupted_rebalance (push) Failing after 10m5s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m29s Details
Test / test_interrupted_rebalance_ec (push) Failing after 10m7s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 19s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m1s Details
Test / test_snapshot_chain_ec (push) Successful in 1m25s Details
Test / test_snapshot_down (push) Successful in 20s Details
Test / test_snapshot_down_ec (push) Successful in 19s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 3m1s Details
Test / test_rebalance_verify_imm (push) Successful in 4m11s Details
Test / test_rebalance_verify_ec (push) Successful in 4m19s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m51s Details
Test / test_write (push) Successful in 31s Details
Test / test_write_xor (push) Successful in 41s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_heal_pg_size_2 (push) Successful in 4m10s Details
Test / test_heal_ec (push) Failing after 10m11s Details
Test / test_scrub (push) Successful in 43s Details
Test / test_scrub_zero_osd_2 (push) Successful in 36s Details
Test / test_scrub_xor (push) Successful in 37s Details
Test / test_scrub_pg_size_3 (push) Successful in 48s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 23s Details
Test / test_scrub_ec (push) Successful in 22s Details
2023-07-06 00:40:13 +03:00
Vitaliy Filippov 45c0694853 Clear etcd_local addresses on reload and also skip duplicates 2023-07-06 00:39:39 +03:00
Vitaliy Filippov 57bcba2406 Add notes about VDUSE 2023-07-04 16:51:46 +03:00
Vitaliy Filippov 30ac899074 Make QEMU driver compatible with older vitastor_client and with systems without io_uring
Test / test_interrupted_rebalance (push) Successful in 1m45s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m25s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m23s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m37s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_snapshot (push) Successful in 20s Details
Test / test_snapshot_ec (push) Successful in 18s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m26s Details
Test / test_snapshot_chain_ec (push) Successful in 2m8s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 21s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 2m48s Details
Test / test_rebalance_verify_imm (push) Successful in 2m48s Details
Test / test_rebalance_verify_ec (push) Successful in 4m10s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m13s Details
Test / test_write (push) Successful in 1m16s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m10s Details
Test / test_heal_ec (push) Successful in 3m28s Details
Test / test_scrub (push) Successful in 43s Details
Test / test_scrub_zero_osd_2 (push) Successful in 41s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 44s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 38s Details
Test / test_scrub_ec (push) Successful in 36s Details
2023-07-04 15:51:43 +03:00
Vitaliy Filippov 2348d39cf4 Avoid repeated qemu_uring_handlers, add 2.0-2.7 compatibility 2023-07-04 00:28:23 +03:00
Vitaliy Filippov 3de7929fe5 Integrate v2 - direct epoll 2023-07-04 00:28:23 +03:00
Vitaliy Filippov 07b2196bc2 Integrate QEMU driver with io_uring 2023-07-04 00:28:23 +03:00
Vitaliy Filippov b8e30608d6 Bump QEMU version to vitastor2
Test / test_etcd_fail (push) Successful in 1m40s Details
Test / test_interrupted_rebalance (push) Successful in 2m45s Details
Test / test_interrupted_rebalance_imm (push) Successful in 7m42s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m59s Details
Test / test_failure_domain (push) Successful in 17s Details
Test / test_snapshot (push) Successful in 56s Details
Test / test_snapshot_ec (push) Successful in 51s Details
Test / test_minsize_1 (push) Successful in 11s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m14s Details
Test / test_snapshot_chain_ec (push) Successful in 1m50s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 20s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 3m18s Details
Test / test_rebalance_verify_imm (push) Successful in 3m15s Details
Test / test_rebalance_verify_ec (push) Successful in 3m38s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m45s Details
Test / test_write (push) Successful in 50s Details
Test / test_write_xor (push) Successful in 46s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 3m36s Details
Test / test_heal_ec (push) Successful in 4m37s Details
Test / test_scrub (push) Successful in 37s Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s Details
Test / test_scrub_xor (push) Successful in 29s Details
Test / test_scrub_pg_size_3 (push) Successful in 33s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 31s Details
Test / test_scrub_ec (push) Successful in 27s Details
2023-07-01 00:55:32 +03:00
Vitaliy Filippov a612cdca47 Release 0.9.3
- Add patch for libvirt 9.0
- Add support for Proxmox VE 8.0
- Fix compatibility of the QEMU driver with iothread (QEMU rebuilds are coming)
- Fix vitastor-cli rm-data/rm/merge hanging when some OSDs are down.
  Allow deletions in unclean cluster at the cost of some data possibly
  "reappearing" when those OSDs start back. In that case you can just repeat
  the deletion request using rm-data.
- A bunch of bug fixes for snapshots:
  - Fix snapshot reads often not working at all with snapshot chain size > 2
  - Fix optimized snapshot data merge (children to parent)
  - Fix updating of image name index key during optimized merge
  - Fix auto-selection preventing the use of optimized merge with only 1 snapshot
  - Fix incorrect CAS retries during snapshot merge
  - Fix snapshot merge progress reporting
- Fix primary_read bitmap buffers use-after-free which could lead to
  incorrect allocation map reads
- Remove /usr/local/bin path from make-etcd
- Some documentation fixes
2023-07-01 00:25:58 +03:00
Vitaliy Filippov c8d61568b5 Fix primary_read bitmap buffers being freed too early (use-after-free)
Test / test_etcd_fail (push) Successful in 1m43s Details
Test / test_interrupted_rebalance (push) Successful in 1m23s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m33s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m39s Details
Test / test_failure_domain (push) Successful in 10s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 17s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m7s Details
Test / test_snapshot_chain_ec (push) Successful in 2m1s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 20s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 2m51s Details
Test / test_rebalance_verify_imm (push) Successful in 2m49s Details
Test / test_rebalance_verify_ec (push) Successful in 4m6s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m3s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_xor (push) Successful in 34s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_heal_pg_size_2 (push) Successful in 3m11s Details
Test / test_heal_ec (push) Successful in 3m54s Details
Test / test_scrub (push) Successful in 44s Details
Test / test_scrub_zero_osd_2 (push) Successful in 30s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 37s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 32s Details
Test / test_scrub_ec (push) Successful in 1m31s Details
2023-06-30 12:47:45 +03:00
Vitaliy Filippov 84ed3c6395 Fix CAS retries during snapshot merge
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Failing after 10m10s Details
Test / test_interrupted_rebalance (push) Successful in 2m46s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m35s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m45s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m53s Details
Test / test_failure_domain (push) Successful in 7s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Failing after 49s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m35s Details
Test / test_snapshot_chain_ec (push) Successful in 2m9s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_rebalance_verify (push) Successful in 3m0s Details
Test / test_rebalance_verify_imm (push) Successful in 3m5s Details
Test / test_rebalance_verify_ec (push) Successful in 3m20s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m24s Details
Test / test_write (push) Successful in 34s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m0s Details
Test / test_heal_ec (push) Successful in 3m58s Details
Test / test_scrub (push) Successful in 58s Details
Test / test_scrub_zero_osd_2 (push) Successful in 38s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 35s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 30s Details
Test / test_scrub_ec (push) Successful in 23s Details
2023-06-30 02:30:23 +03:00
Vitaliy Filippov a7b57386c0 Do not print last subcommand result twice during "inverse" snapshot merge
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 1m28s Details
Test / test_interrupted_rebalance (push) Successful in 1m45s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m46s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m44s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m30s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_snapshot (push) Successful in 30s Details
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_minsize_1 (push) Failing after 19s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot_chain (push) Successful in 1m57s Details
Test / test_snapshot_chain_ec (push) Successful in 2m42s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m23s Details
Test / test_rebalance_verify_imm (push) Successful in 3m17s Details
Test / test_rebalance_verify_ec (push) Successful in 4m5s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m50s Details
Test / test_write (push) Successful in 41s Details
Test / test_write_xor (push) Failing after 3m17s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m12s Details
Test / test_heal_ec (push) Successful in 4m13s Details
Test / test_scrub (push) Successful in 38s Details
Test / test_scrub_zero_osd_2 (push) Successful in 36s Details
Test / test_scrub_xor (push) Successful in 33s Details
Test / test_scrub_pg_size_3 (push) Successful in 46s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s Details
Test / test_scrub_ec (push) Successful in 33s Details
2023-06-30 02:07:10 +03:00
Vitaliy Filippov 9d4ea5f764 Fix inverse parent selection which prevented the use of optimized merge in case of only 1 snapshot 2023-06-30 01:39:11 +03:00
Vitaliy Filippov 000e4944ec Remove "inverse parent" image name index key from etcd during snapshot merge 2023-06-30 01:23:30 +03:00
Vitaliy Filippov 8426616d89 Warn about unfinished deletions in rm-data
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 1m46s Details
Test / test_interrupted_rebalance (push) Successful in 2m34s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m50s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m35s Details
Test / test_interrupted_rebalance_ec_imm (push) Failing after 10m6s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_snapshot (push) Successful in 21s Details
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Failing after 48s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 1m35s Details
Test / test_snapshot_chain_ec (push) Successful in 2m11s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m4s Details
Test / test_rebalance_verify_imm (push) Successful in 3m0s Details
Test / test_rebalance_verify_ec (push) Successful in 3m25s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m58s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 34s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_heal_pg_size_2 (push) Successful in 4m3s Details
Test / test_heal_ec (push) Successful in 3m53s Details
Test / test_scrub (push) Successful in 26s Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s Details
Test / test_scrub_xor (push) Successful in 35s Details
Test / test_scrub_pg_size_3 (push) Successful in 45s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 32s Details
Test / test_scrub_ec (push) Successful in 25s Details
2023-06-30 01:18:25 +03:00
Vitaliy Filippov 1a841344ec Print progress of all operations during snapshot merge 2023-06-30 01:13:47 +03:00
Vitaliy Filippov 8603b5cb1d Do not hang on inactive OSDs during delete, report and skip them instead
Test / test_create_nomaxid (push) Successful in 8s Details
Test / test_etcd_fail (push) Successful in 46s Details
Test / test_interrupted_rebalance (push) Successful in 3m13s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m30s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m18s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m34s Details
Test / test_failure_domain (push) Successful in 7s Details
Test / test_snapshot (push) Successful in 29s Details
Test / test_snapshot_ec (push) Successful in 32s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m37s Details
Test / test_snapshot_chain_ec (push) Successful in 2m21s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 3m14s Details
Test / test_rebalance_verify_imm (push) Successful in 3m8s Details
Test / test_rebalance_verify_ec (push) Successful in 3m18s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m20s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_xor (push) Successful in 43s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m55s Details
Test / test_heal_ec (push) Successful in 4m2s Details
Test / test_scrub (push) Successful in 56s Details
Test / test_scrub_zero_osd_2 (push) Successful in 51s Details
Test / test_scrub_xor (push) Successful in 29s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m2s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 37s Details
Test / test_scrub_ec (push) Successful in 32s Details
2023-06-30 00:15:16 +03:00
Vitaliy Filippov f12b8e45a9 Remove /usr/local/bin path from make-etcd
Test / test_create_nomaxid (push) Successful in 6s Details
Test / test_etcd_fail (push) Successful in 51s Details
Test / test_interrupted_rebalance (push) Successful in 7m28s Details
Test / test_interrupted_rebalance_imm (push) Failing after 10m10s Details
Test / test_interrupted_rebalance_ec (push) Successful in 7m21s Details
Test / test_interrupted_rebalance_ec_imm (push) Failing after 10m7s Details
Test / test_failure_domain (push) Successful in 7s Details
Test / test_snapshot (push) Successful in 17s Details
Test / test_snapshot_ec (push) Successful in 18s Details
Test / test_minsize_1 (push) Successful in 11s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 10s Details
Test / test_snapshot_chain (push) Successful in 1m19s Details
Test / test_snapshot_chain_ec (push) Successful in 1m43s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 3m9s Details
Test / test_rebalance_verify_imm (push) Successful in 3m3s Details
Test / test_rebalance_verify_ec (push) Successful in 3m31s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m33s Details
Test / test_write (push) Successful in 49s Details
Test / test_write_xor (push) Successful in 58s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_heal_pg_size_2 (push) Successful in 3m47s Details
Test / test_heal_ec (push) Successful in 3m59s Details
Test / test_scrub (push) Successful in 58s Details
Test / test_scrub_zero_osd_2 (push) Successful in 42s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 39s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 31s Details
Test / test_scrub_ec (push) Successful in 31s Details
2023-06-29 23:49:31 +03:00
Vitaliy Filippov 878ccbb6ea Fix snapshot chain "down-merge" ("up-merge" worked well...)
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_interrupted_rebalance (push) Successful in 7m46s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m42s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m22s Details
Test / test_failure_domain (push) Successful in 7s Details
Test / test_snapshot (push) Successful in 18s Details
Test / test_snapshot_ec (push) Successful in 20s Details
Test / test_minsize_1 (push) Successful in 11s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m11s Details
Test / test_snapshot_chain_ec (push) Successful in 2m13s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 3m26s Details
Test / test_rebalance_verify_imm (push) Successful in 3m27s Details
Test / test_rebalance_verify_ec (push) Successful in 4m23s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m32s Details
Test / test_write (push) Successful in 31s Details
Test / test_write_xor (push) Successful in 33s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m37s Details
Test / test_heal_ec (push) Successful in 4m43s Details
Test / test_scrub (push) Successful in 45s Details
Test / test_scrub_zero_osd_2 (push) Successful in 37s Details
Test / test_scrub_xor (push) Successful in 39s Details
Test / test_scrub_pg_size_3 (push) Successful in 55s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 33s Details
Test / test_scrub_ec (push) Successful in 32s Details
2023-06-29 00:47:21 +03:00
Vitaliy Filippov b14220b4d0 Add a test for snapshot chain 2023-06-29 00:47:21 +03:00
Vitaliy Filippov 181d6ba407 Move npm install to CI build and remove it from tests 2023-06-28 23:41:46 +03:00
Vitaliy Filippov 63c2b9832c Fix chained (snapshot) reads often not working at all with chain size > 2
Test / test_change_pg_count (push) Successful in 42s Details
Test / test_change_pg_count_ec (push) Successful in 36s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 8s Details
Test / test_etcd_fail (push) Successful in 1m6s Details
Test / test_failure_domain (push) Successful in 10s Details
Test / test_interrupted_rebalance (push) Successful in 1m52s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m49s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m2s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m19s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 2m56s Details
Test / test_rebalance_verify_ec (push) Successful in 3m6s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m9s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_write (push) Successful in 1m30s Details
Test / test_write_xor (push) Successful in 2m14s Details
Test / test_write_no_same (push) Successful in 20s Details
Test / test_heal_pg_size_2 (push) Successful in 4m5s Details
Test / test_heal_ec (push) Successful in 4m11s Details
Test / test_scrub (push) Successful in 48s Details
Test / test_scrub_zero_osd_2 (push) Successful in 32s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 37s Details
Test / test_scrub_ec (push) Successful in 41s Details
2023-06-28 18:54:03 +03:00
Vitaliy Filippov 10e2e6a7c8 Add a patch for pve-qemu 8.0 2023-06-24 01:33:52 +03:00
Vitaliy Filippov a598428992 Add a note about PVE 8.0 2023-06-24 01:29:28 +03:00
Vitaliy Filippov 08a677b684 Notes about rebuilding pve-qemu 2023-06-24 01:00:21 +03:00
Vitaliy Filippov 7c8fbdad16 Fix typo 2023-06-22 02:07:33 +03:00
Vitaliy Filippov 2f9353df60 Markdown preprocessor to compile multiple .md files into one 2023-06-22 01:27:38 +03:00
Vitaliy Filippov 57c744f288 Fix some cross-references in docs 2023-06-22 01:16:03 +03:00
Vitaliy Filippov a11ca56fb1 Fix compatibility of the QEMU driver with iothread
Test / test_change_pg_count_ec (push) Successful in 55s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 57s Details
Test / test_failure_domain (push) Successful in 10s Details
Test / test_interrupted_rebalance (push) Successful in 1m53s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m39s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m46s Details
Test / test_minsize_1 (push) Successful in 55s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m13s Details
Test / test_rebalance_verify_imm (push) Successful in 2m52s Details
Test / test_rebalance_verify_ec (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m25s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot (push) Successful in 36s Details
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_write (push) Successful in 38s Details
Test / test_write_xor (push) Successful in 1m50s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m17s Details
Test / test_heal_ec (push) Successful in 4m18s Details
Test / test_scrub (push) Successful in 50s Details
Test / test_scrub_zero_osd_2 (push) Successful in 51s Details
Test / test_scrub_xor (push) Successful in 39s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 42s Details
Test / test_scrub_ec (push) Successful in 28s Details
2023-06-21 02:11:28 +03:00
Vitaliy Filippov b84927b340 Fix \n in nbd_proxy
Test / test_change_pg_count (push) Successful in 41s Details
Test / test_change_pg_count_ec (push) Successful in 37s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 9s Details
Test / test_etcd_fail (push) Successful in 1m9s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_interrupted_rebalance (push) Successful in 1m54s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m47s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m17s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m26s Details
Test / test_minsize_1 (push) Successful in 19s Details
Test / test_move_reappear (push) Successful in 50s Details
Test / test_rebalance_verify (push) Successful in 3m6s Details
Test / test_rebalance_verify_imm (push) Successful in 3m3s Details
Test / test_rebalance_verify_ec (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m55s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 1m38s Details
Test / test_write_xor (push) Successful in 2m13s Details
Test / test_write_no_same (push) Successful in 23s Details
Test / test_heal_pg_size_2 (push) Successful in 4m5s Details
Test / test_scrub (push) Successful in 42s Details
Test / test_scrub_zero_osd_2 (push) Successful in 33s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 40s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 34s Details
Test / test_scrub_ec (push) Successful in 30s Details
2023-06-19 01:48:58 +03:00
Vitaliy Filippov 83cacba226 Fix patched-qemu build 2023-06-19 01:47:55 +03:00
Vitaliy Filippov 2c8f0bc6d5 Add a note about Debian 12 2023-06-19 01:08:41 +03:00
Vitaliy Filippov 7ae5b0e368 Add patch for libvirt 9.0 2023-06-19 01:07:08 +03:00
Vitaliy Filippov 926be372fd Release 0.9.2
Test / test_change_pg_count (push) Successful in 35s Details
Test / test_change_pg_count_ec (push) Successful in 34s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 1m1s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_interrupted_rebalance (push) Successful in 1m26s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m15s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 2m49s Details
Test / test_rebalance_verify_imm (push) Successful in 2m45s Details
Test / test_rebalance_verify_ec (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m3s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 35s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_write (push) Successful in 1m10s Details
Test / test_write_xor (push) Successful in 2m23s Details
Test / test_write_no_same (push) Successful in 19s Details
Test / test_heal_pg_size_2 (push) Successful in 3m58s Details
Test / test_heal_ec (push) Successful in 4m8s Details
Test / test_scrub (push) Successful in 1m3s Details
Test / test_scrub_zero_osd_2 (push) Successful in 36s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 51s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 32s Details
Test / test_scrub_ec (push) Successful in 43s Details
- Measure and report scrub I/O statistics in vitastor-cli status
- Make aggregated statistics in vitastor-cli status much smoother
  (first derive, then sum instead of first summing and then deriving)
- Fix an old rare bug leading to journal corruption
  (try to use scrub if you think you're affected...)
- Do not start EC PGs without at least <data chunks> OSDs in each old set
  (prevents spurious read errors with EC during reconnections/restarts)
- Fix failed assert(!scrub_list_op) on OSD restart with pending scrubs
- Fix future planned scrubs not starting because of incorrect time comparison
- Build packages for Debian 12 (Bookworm)
2023-06-18 19:44:33 +03:00
Vitaliy Filippov 6222779b52 Support debian bookworm (12) build 2023-06-18 19:44:33 +03:00
Vitaliy Filippov a4186e20aa First derive, then sum per-OSD statistics instead of first summing and then deriving
Test / test_change_pg_count (push) Successful in 43s Details
Test / test_change_pg_count_ec (push) Successful in 37s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 8s Details
Test / test_failure_domain (push) Successful in 16s Details
Test / test_interrupted_rebalance (push) Successful in 1m49s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m38s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m49s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m23s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m2s Details
Test / test_rebalance_verify_imm (push) Successful in 2m53s Details
Test / test_rebalance_verify_ec (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m27s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot (push) Successful in 34s Details
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_write (push) Successful in 37s Details
Test / test_write_xor (push) Successful in 44s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_heal_pg_size_2 (push) Successful in 3m31s Details
Test / test_heal_ec (push) Successful in 4m20s Details
Test / test_scrub (push) Successful in 38s Details
Test / test_scrub_zero_osd_2 (push) Successful in 30s Details
Test / test_scrub_xor (push) Successful in 32s Details
Test / test_scrub_pg_size_3 (push) Successful in 42s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 37s Details
Test / test_scrub_ec (push) Successful in 34s Details
This makes statistics reported by vitastor-cli status much smoother
2023-06-18 01:32:24 +03:00
Vitaliy Filippov c74a424930 Report scrub I/O in vitastor-cli status 2023-06-17 21:11:21 +03:00
Vitaliy Filippov 32f2c4dd27 Measure scrub statistics 2023-06-17 20:56:26 +03:00
Vitaliy Filippov 3ad16b9a1a Fix auto_scrubs not starting because of < vs <= =))
Test / test_change_pg_count (push) Successful in 41s Details
Test / test_change_pg_count_ec (push) Successful in 36s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 9s Details
Test / test_etcd_fail (push) Successful in 1m20s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_interrupted_rebalance (push) Successful in 2m1s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m55s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m48s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s Details
Test / test_move_reappear (push) Successful in 51s Details
Test / test_rebalance_verify (push) Successful in 3m19s Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec (push) Successful in 3m21s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m4s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot (push) Successful in 23s Details
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 1m35s Details
Test / test_write_xor (push) Successful in 2m29s Details
Test / test_write_no_same (push) Successful in 29s Details
Test / test_heal_pg_size_2 (push) Successful in 4m11s Details
Test / test_heal_ec (push) Successful in 5m4s Details
Test / test_scrub (push) Successful in 55s Details
Test / test_scrub_zero_osd_2 (push) Successful in 41s Details
Test / test_scrub_xor (push) Successful in 37s Details
Test / test_scrub_pg_size_3 (push) Successful in 57s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 46s Details
Test / test_scrub_ec (push) Successful in 31s Details
2023-06-17 17:32:21 +03:00
Vitaliy Filippov 1c2df841c2 Fix failed assert(!scrub_list_op) on OSD restart with pending scrubs
Test / test_change_pg_count (push) Successful in 38s Details
Test / test_change_pg_count_ec (push) Successful in 38s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 8s Details
Test / test_etcd_fail (push) Successful in 53s Details
Test / test_failure_domain (push) Successful in 15s Details
Test / test_interrupted_rebalance (push) Successful in 2m3s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m55s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m43s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m31s Details
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rebalance_verify (push) Successful in 3m9s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec (push) Successful in 3m15s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m45s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot (push) Successful in 35s Details
Test / test_snapshot_ec (push) Successful in 18s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 1m12s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 3m41s Details
Test / test_scrub (push) Successful in 33s Details
Test / test_scrub_zero_osd_2 (push) Successful in 29s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 51s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 31s Details
Test / test_scrub_ec (push) Successful in 26s Details
2023-06-17 17:02:54 +03:00
Vitaliy Filippov aa5dacc7a9 Do not start EC PGs without at least pg_data_size connections to old OSDs from each set
Test / test_change_pg_count (push) Successful in 36s Details
Test / test_change_pg_count_ec (push) Successful in 38s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 9s Details
Test / test_etcd_fail (push) Successful in 1m13s Details
Test / test_failure_domain (push) Successful in 11s Details
Test / test_interrupted_rebalance (push) Successful in 1m51s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m43s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m47s Details
Test / test_minsize_1 (push) Successful in 43s Details
Test / test_move_reappear (push) Successful in 43s Details
Test / test_rebalance_verify (push) Successful in 3m16s Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec (push) Successful in 3m8s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m10s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot (push) Successful in 22s Details
Test / test_snapshot_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_write (push) Successful in 1m44s Details
Test / test_write_xor (push) Successful in 2m29s Details
Test / test_write_no_same (push) Successful in 22s Details
Test / test_heal_pg_size_2 (push) Successful in 4m37s Details
Test / test_heal_ec (push) Successful in 4m4s Details
Test / test_scrub (push) Successful in 48s Details
Test / test_scrub_zero_osd_2 (push) Successful in 41s Details
Test / test_scrub_xor (push) Successful in 39s Details
Test / test_scrub_pg_size_3 (push) Successful in 47s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 41s Details
Test / test_scrub_ec (push) Successful in 34s Details
2023-06-17 02:16:30 +03:00
Vitaliy Filippov affe8fc270 Raise timeout also for add_osd and rebalance_verify 2023-06-17 00:29:03 +03:00
Vitaliy Filippov 4fdc49bdc7 Add another assert-type check (it does not fire, just as a safety measure for the future) 2023-06-17 00:07:22 +03:00
Vitaliy Filippov 86b4682975 Put get_trim_pos into the "critical section". Fixes rare journal corruption issue
The consequence of this issue was that in some very rare cases (only reproduced
under load in CI when running 4+ tests in parallel) small write data written to
journal could overwrite journal entries.

Also add an assert-type safety check to be able to catch this issue in the
future again in case of a regression.
2023-06-17 00:06:42 +03:00
Vitaliy Filippov bdd48e4cf1 Release 0.9.1
Test / test_change_pg_count (push) Successful in 35s Details
Test / test_change_pg_count_ec (push) Successful in 32s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 6s Details
Test / test_etcd_fail (push) Successful in 49s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_interrupted_rebalance (push) Successful in 1m4s Details
Test / test_interrupted_rebalance_imm (push) Successful in 57s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m2s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 53s Details
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 1m45s Details
Test / test_rebalance_verify_imm (push) Successful in 1m41s Details
Test / test_rebalance_verify_ec (push) Successful in 1m53s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 1m48s Details
Test / test_rm (push) Successful in 10s Details
Test / test_snapshot (push) Successful in 14s Details
Test / test_snapshot_ec (push) Successful in 16s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 49s Details
Test / test_write_xor (push) Successful in 1m1s Details
Test / test_write_no_same (push) Successful in 11s Details
Test / test_heal_pg_size_2 (push) Successful in 3m12s Details
Test / test_scrub (push) Successful in 28s Details
Test / test_scrub_zero_osd_2 (push) Successful in 22s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 25s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s Details
Test / test_scrub_ec (push) Successful in 24s Details
- Fix "Client XX command out of sync" messages sometimes happening on OSD reconnections
- Fix a bug where EC reads parallel with writes to the same object failed with -ERANGE error
- Slightly reduce the amount of metadata writes during journal flushing
- Correctly unmap NBD volumes when Proxmox forces map_volume use (with SWTPM and maybe some other cases)
2023-06-10 11:42:49 +03:00
Vitaliy Filippov af8c3411cd Correctly unmap NBD when Proxmox forces map_volume use (with SWTPM and maybe something else) 2023-06-08 01:31:49 +03:00
Vitaliy Filippov 9c405009f3 Use randrw in test_heal
Test / test_change_pg_count (push) Successful in 33s Details
Test / test_change_pg_count_ec (push) Successful in 32s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 9s Details
Test / test_etcd_fail (push) Successful in 51s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_interrupted_rebalance (push) Successful in 1m4s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m0s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m20s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 55s Details
Test / test_minsize_1 (push) Successful in 22s Details
Test / test_move_reappear (push) Successful in 22s Details
Test / test_rebalance_verify (push) Successful in 1m47s Details
Test / test_rebalance_verify_imm (push) Successful in 1m45s Details
Test / test_rebalance_verify_ec (push) Successful in 2m3s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 1m52s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot (push) Successful in 15s Details
Test / test_snapshot_ec (push) Successful in 16s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_write (push) Successful in 31s Details
Test / test_write_xor (push) Successful in 47s Details
Test / test_write_no_same (push) Successful in 11s Details
Test / test_heal_pg_size_2 (push) Successful in 3m12s Details
Test / test_scrub (push) Successful in 24s Details
Test / test_scrub_zero_osd_2 (push) Successful in 26s Details
Test / test_scrub_xor (push) Successful in 26s Details
Test / test_scrub_pg_size_3 (push) Successful in 25s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 42s Details
Test / test_scrub_ec (push) Successful in 33s Details
2023-06-03 00:49:53 +03:00
Vitaliy Filippov f9fbea25a4 Remove double write when old and new locations are in the same metadata block
Also add another metadata entry fool-safety check which, ideally, will never fire %)
2023-06-03 00:47:10 +03:00
Vitaliy Filippov 2c9a10d081 Fix an idiotic bug leading to failed reads with -ERANGE with EC :D 2023-06-03 00:44:52 +03:00
Vitaliy Filippov 150968070f Slightly improve some debug prints
Test / test_change_pg_count (push) Successful in 30s Details
Test / test_change_pg_count_ec (push) Successful in 31s Details
Test / test_change_pg_size (push) Successful in 7s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 45s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_interrupted_rebalance (push) Successful in 1m3s Details
Test / test_interrupted_rebalance_imm (push) Successful in 55s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m30s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 57s Details
Test / test_minsize_1 (push) Successful in 20s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 1m49s Details
Test / test_rebalance_verify_imm (push) Successful in 1m40s Details
Test / test_rebalance_verify_ec (push) Successful in 2m4s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 1m51s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot (push) Successful in 16s Details
Test / test_snapshot_ec (push) Successful in 17s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 41s Details
Test / test_write_xor (push) Successful in 49s Details
Test / test_write_no_same (push) Successful in 10s Details
Test / test_heal_pg_size_2 (push) Successful in 2m58s Details
Test / test_scrub (push) Successful in 23s Details
Test / test_scrub_zero_osd_2 (push) Successful in 19s Details
Test / test_scrub_xor (push) Successful in 17s Details
Test / test_scrub_pg_size_3 (push) Successful in 24s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 28s Details
Test / test_scrub_ec (push) Successful in 25s Details
2023-05-29 01:04:16 +03:00
Vitaliy Filippov cdfc74665b Close client FDs only when destroying the client, after handling all async reads/writes
Test / test_change_pg_count (push) Successful in 50s Details
Test / test_change_pg_count_ec (push) Successful in 58s Details
Test / test_change_pg_size (push) Successful in 17s Details
Test / test_create_nomaxid (push) Successful in 19s Details
Test / test_etcd_fail (push) Successful in 58s Details
Test / test_failure_domain (push) Successful in 14s Details
Test / test_interrupted_rebalance (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m0s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m23s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m16s Details
Test / test_minsize_1 (push) Successful in 38s Details
Test / test_move_reappear (push) Successful in 54s Details
Test / test_rebalance_verify (push) Successful in 2m26s Details
Test / test_rebalance_verify_imm (push) Successful in 2m7s Details
Test / test_rebalance_verify_ec (push) Successful in 2m51s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m15s Details
Test / test_rm (push) Successful in 22s Details
Test / test_snapshot (push) Successful in 40s Details
Test / test_snapshot_ec (push) Successful in 34s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_write (push) Successful in 1m7s Details
Test / test_write_xor (push) Successful in 2m13s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_heal_pg_size_2 (push) Successful in 5m14s Details
Test / test_scrub (push) Successful in 36s Details
Test / test_scrub_zero_osd_2 (push) Successful in 40s Details
Test / test_scrub_xor (push) Successful in 54s Details
Test / test_scrub_pg_size_3 (push) Successful in 54s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m12s Details
Test / test_scrub_ec (push) Successful in 1m10s Details
Fixes "Client XX command out of sync" sometimes happening on reconnections
2023-05-25 00:52:43 +03:00
Vitaliy Filippov 3f60fecd7c Fix typo 2023-05-21 18:37:01 +03:00
Vitaliy Filippov 3b4cf29e65 Release 0.9.0
New features:
- Scrubbing! Check documentation: [auto_scrub](src/branch/master/docs/config/osd.en.md#auto_scrub)
- Document online-updatable configuration parameters

Bug fixes:
- Fix NaN during PG optimisation if there are nonexisting OSDs in node_placement
- Fix monitor crash on pool deletion
- Clear journal_device and meta_device before initialising the next OSD in automatic mode
- Sync unsynced deletes before overwriting them with a lower version
  (reproducted mostly/only after scrubbing)
2023-05-21 15:07:14 +03:00
Vitaliy Filippov eeaba11ebd Use fio 3.27-8 for alma9 2023-05-21 14:48:26 +03:00
Vitaliy Filippov aea567cfbd Slightly improve scrub docs
Test / test_cas (push) Successful in 9s Details
Test / test_change_pg_count (push) Successful in 52s Details
Test / test_change_pg_count_ec (push) Successful in 1m0s Details
Test / test_change_pg_size (push) Successful in 16s Details
Test / test_create_nomaxid (push) Successful in 16s Details
Test / test_etcd_fail (push) Successful in 56s Details
Test / test_failure_domain (push) Successful in 13s Details
Test / test_interrupted_rebalance (push) Successful in 1m24s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m10s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m9s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m6s Details
Test / test_minsize_1 (push) Failing after 19s Details
Test / test_move_reappear (push) Successful in 28s Details
Test / test_rebalance_verify (push) Successful in 2m25s Details
Test / test_rebalance_verify_imm (push) Successful in 2m19s Details
Test / test_rebalance_verify_ec (push) Successful in 3m3s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m20s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot (push) Successful in 21s Details
Test / test_snapshot_ec (push) Successful in 28s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_write_xor (push) Has started running Details
Test / test_heal_pg_size_2 (push) Has started running Details
Test / test_write (push) Has started running Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
2023-05-21 12:52:30 +03:00
Vitaliy Filippov ce02f47de6 Allow to disable scrub_find_best 2023-05-21 12:33:38 +03:00
Vitaliy Filippov 5fd3208616 Add version archive link to docs 2023-05-21 11:47:33 +03:00
Vitaliy Filippov 5997b76535 Remove -runtime=10 from fio params in test_scrub, it was breaking the test in CI :D
Test / test_change_pg_count (push) Successful in 49s Details
Test / test_change_pg_count_ec (push) Successful in 2m53s Details
Test / test_change_pg_size (push) Successful in 17s Details
Test / test_create_nomaxid (push) Successful in 13s Details
Test / test_etcd_fail (push) Successful in 1m0s Details
Test / test_failure_domain (push) Successful in 13s Details
Test / test_interrupted_rebalance (push) Successful in 1m18s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m3s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m40s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 52s Details
Test / test_minsize_1 (push) Successful in 22s Details
Test / test_move_reappear (push) Successful in 23s Details
Test / test_rebalance_verify (push) Successful in 2m32s Details
Test / test_rebalance_verify_imm (push) Successful in 2m29s Details
Test / test_rebalance_verify_ec (push) Successful in 2m55s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m30s Details
Test / test_rm (push) Successful in 22s Details
Test / test_snapshot (push) Successful in 26s Details
Test / test_snapshot_ec (push) Successful in 38s Details
Test / test_splitbrain (push) Successful in 28s Details
Test / test_write (push) Successful in 1m5s Details
Test / test_write_xor (push) Successful in 2m13s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_heal_ec (push) Successful in 5m27s Details
Test / test_scrub (push) Successful in 36s Details
Test / test_scrub_zero_osd_2 (push) Successful in 39s Details
Test / test_scrub_xor (push) Successful in 1m2s Details
Test / test_scrub_pg_size_3 (push) Successful in 50s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 29s Details
Test / test_scrub_ec (push) Successful in 1m25s Details
2023-05-21 11:03:59 +03:00
Vitaliy Filippov f1961157f0 Fix brute-force error locator for EC n+k with k > 2
Test / test_change_pg_count_ec (push) Successful in 2m23s Details
Test / test_change_pg_size (push) Successful in 20s Details
Test / test_create_nomaxid (push) Successful in 16s Details
Test / test_etcd_fail (push) Successful in 55s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_interrupted_rebalance (push) Successful in 1m18s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m9s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m23s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m8s Details
Test / test_minsize_1 (push) Successful in 22s Details
Test / test_move_reappear (push) Successful in 28s Details
Test / test_rebalance_verify (push) Successful in 2m17s Details
Test / test_rebalance_verify_imm (push) Successful in 2m19s Details
Test / test_rebalance_verify_ec (push) Successful in 3m4s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m22s Details
Test / test_rm (push) Successful in 23s Details
Test / test_snapshot (push) Successful in 20s Details
Test / test_snapshot_ec (push) Successful in 34s Details
Test / test_splitbrain (push) Successful in 33s Details
Test / test_write (push) Successful in 1m15s Details
Test / test_write_xor (push) Successful in 2m6s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_heal_pg_size_2 (push) Successful in 5m22s Details
Test / test_heal_ec (push) Successful in 5m31s Details
Test / test_scrub (push) Successful in 29s Details
Test / test_scrub_zero_osd_2 (push) Successful in 27s Details
Test / test_scrub_xor (push) Successful in 22s Details
Test / test_scrub_pg_size_3 (push) Failing after 37s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 29s Details
Test / test_scrub_ec (push) Failing after 33s Details
2023-05-21 00:57:14 +03:00
Vitaliy Filippov 88c1ba0790 Fix compile errors with gcc 10
Test / build (push) Has started running Details
Test / buildenv (push) Successful in 11s Details
Test / make_test (push) Has been cancelled Details
Test / test_add_osd (push) Has been cancelled Details
Test / test_cas (push) Has been cancelled Details
Test / test_change_pg_count (push) Has been cancelled Details
Test / test_change_pg_count_ec (push) Has been cancelled Details
Test / test_change_pg_size (push) Has been cancelled Details
Test / test_create_nomaxid (push) Has been cancelled Details
Test / test_etcd_fail (push) Has been cancelled Details
Test / test_failure_domain (push) Has been cancelled Details
Test / test_interrupted_rebalance (push) Has been cancelled Details
Test / test_interrupted_rebalance_imm (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec_imm (push) Has been cancelled Details
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot (push) Has been cancelled Details
Test / test_snapshot_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
2023-05-20 23:20:09 +03:00
Vitaliy Filippov b5bd611683 Add scrub tests to CI 2023-05-20 23:20:09 +03:00
Vitaliy Filippov fa90b5a4e7 Schedule automatic scrubs correctly (not just after previous scrub) 2023-05-20 23:20:09 +03:00
Vitaliy Filippov 8d40ad99a6 Add scrub documentation 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 3475772b07 Add configuration online update documentation 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 25fcedf6e7 Enable vitastor-cli fix in test 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 6ca20aa194 Allow scrub to fix corrupted object states 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 4bfd994341 Sync unsynced deletes before overwriting them with a lower version 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 59e959dcbb Do not die when "different versions are returned from subops" 2023-05-20 23:19:39 +03:00
Vitaliy Filippov a9581f0739 Handle dirty deletes during read correctly O_o 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 105a405b0a Implement vitastor-cli fix 2023-05-20 23:19:39 +03:00
Vitaliy Filippov d55d7d5326 Add scrub test 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 0e5d0e02a9 Add "vitastor-cli describe" command 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 0439981a66 Implement "describe object(s)" operation
Required to implement fixing inconsistent objects in vitastor-cli
2023-05-20 23:19:39 +03:00
Vitaliy Filippov 6648f6bb6e Implement ambiguity detection during scrub 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 281be547eb Implement brute-force error locator for EC 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 0c78dd7178 Add no_scrub flag 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 3c924397e7 Store next scrub timestamp instead of last scrub timestamp 2023-05-20 23:19:39 +03:00
Vitaliy Filippov c3bd26193d Implement PG scrub runner 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 43b77d7619 Implement scrubbing "data path" - OSD_OP_SCRUB 2023-05-20 23:19:39 +03:00
Vitaliy Filippov a6d846863b Add min/max stripe and limit to OP_LIST 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 8dc427b43c Retry failed reads (including chained and RMW) from other replicas 2023-05-20 23:19:39 +03:00
Vitaliy Filippov bf2112653b Refcount object_states 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 0538a484b3 Add corrupted object state 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 97720fa6b4 Remove unused capture
Test / buildenv (push) Successful in 12s Details
Test / build (push) Has started running Details
Test / make_test (push) Has been cancelled Details
Test / test_add_osd (push) Has been cancelled Details
Test / test_cas (push) Has been cancelled Details
Test / test_change_pg_count (push) Has been cancelled Details
Test / test_change_pg_count_ec (push) Has been cancelled Details
Test / test_change_pg_size (push) Has been cancelled Details
Test / test_create_nomaxid (push) Has been cancelled Details
Test / test_etcd_fail (push) Has been cancelled Details
Test / test_failure_domain (push) Has been cancelled Details
Test / test_interrupted_rebalance (push) Has been cancelled Details
Test / test_interrupted_rebalance_imm (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec_imm (push) Has been cancelled Details
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot (push) Has been cancelled Details
Test / test_snapshot_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
2023-05-20 22:58:51 +03:00
Vitaliy Filippov e60e352df6 Improve vitastor-nbd documentation 2023-05-20 22:58:51 +03:00
Vitaliy Filippov 98077a1712 Remove unused dependencies from CSI 2023-05-18 11:54:47 +03:00
Vitaliy Filippov 1c7d53996d Reweight only 2 OSDs to zero in test_rebalance_verify, otherwise the test does not pass with EC 3+2
Test / buildenv (push) Successful in 9s Details
Test / build (push) Successful in 2m20s Details
Test / test_cas (push) Successful in 11s Details
Test / make_test (push) Successful in 35s Details
Test / test_change_pg_size (push) Successful in 22s Details
Test / test_change_pg_count (push) Successful in 52s Details
Test / test_create_nomaxid (push) Successful in 19s Details
Test / test_change_pg_count_ec (push) Successful in 1m3s Details
Test / test_failure_domain (push) Successful in 13s Details
Test / test_etcd_fail (push) Successful in 1m0s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m3s Details
Test / test_interrupted_rebalance (push) Successful in 1m14s Details
Test / test_minsize_1 (push) Successful in 22s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m1s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m38s Details
Test / test_rebalance_verify (push) Successful in 2m20s Details
Test / test_rebalance_verify_imm (push) Successful in 2m1s Details
Test / test_rm (push) Successful in 26s Details
Test / test_rebalance_verify_ec (push) Successful in 2m30s Details
Test / test_snapshot (push) Successful in 22s Details
Test / test_snapshot_ec (push) Successful in 28s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_write (push) Successful in 48s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m11s Details
Test / test_write_xor (push) Successful in 1m28s Details
Test / test_heal_pg_size_2 (push) Successful in 4m48s Details
Test / test_heal_ec (push) Successful in 5m12s Details
Test / test_add_osd (push) Successful in 1m20s Details
2023-05-18 00:42:40 +03:00
Vitaliy Filippov 2ca07b1ea7 Raise timeout in test_rebalance_verify
Test / buildenv (push) Successful in 10s Details
Test / build (push) Successful in 2m27s Details
Test / test_cas (push) Successful in 11s Details
Test / make_test (push) Successful in 34s Details
Test / test_change_pg_size (push) Successful in 22s Details
Test / test_change_pg_count (push) Successful in 52s Details
Test / test_create_nomaxid (push) Successful in 8s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_etcd_fail (push) Successful in 1m0s Details
Test / test_interrupted_rebalance (push) Successful in 1m15s Details
Test / test_add_osd (push) Successful in 2m33s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m4s Details
Test / test_change_pg_count_ec (push) Successful in 2m52s Details
Test / test_minsize_1 (push) Successful in 19s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 53s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m36s Details
Test / test_rebalance_verify (push) Successful in 2m22s Details
Test / test_rebalance_verify_imm (push) Successful in 2m22s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_rebalance_verify_ec (push) Failing after 3m6s Details
Test / test_splitbrain (push) Successful in 17s Details
Test / test_write_no_same (push) Successful in 20s Details
Test / test_rebalance_verify_ec_imm (push) Failing after 3m9s Details
Test / test_write (push) Successful in 49s Details
Test / test_write_xor (push) Successful in 1m17s Details
Test / test_heal_ec (push) Successful in 4m53s Details
Test / test_heal_pg_size_2 (push) Failing after 10m10s Details
2023-05-17 01:58:01 +03:00
Vitaliy Filippov 022176aa98 Fix NaN during PG optimisation if there are nonexisting OSDs in node_placement
Test / buildenv (push) Successful in 11s Details
Test / build (push) Successful in 2m28s Details
Test / test_cas (push) Successful in 12s Details
Test / make_test (push) Successful in 40s Details
Test / test_change_pg_size (push) Successful in 23s Details
Test / test_change_pg_count (push) Successful in 1m1s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_failure_domain (push) Successful in 11s Details
Test / test_change_pg_count_ec (push) Successful in 1m35s Details
Test / test_etcd_fail (push) Successful in 51s Details
Test / test_add_osd (push) Successful in 2m27s Details
Test / test_interrupted_rebalance (push) Successful in 1m14s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m3s Details
Test / test_minsize_1 (push) Successful in 28s Details
Test / test_move_reappear (push) Successful in 41s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m13s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m49s Details
Test / test_rebalance_verify (push) Successful in 2m21s Details
Test / test_rm (push) Successful in 15s Details
Test / test_rebalance_verify_imm (push) Successful in 2m12s Details
Test / test_snapshot (push) Successful in 20s Details
Test / test_snapshot_ec (push) Successful in 28s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_write (push) Successful in 1m6s Details
Test / test_write_xor (push) Successful in 1m42s Details
Test / test_heal_pg_size_2 (push) Successful in 4m57s Details
Test / test_heal_ec (push) Successful in 4m42s Details
Test / test_rebalance_verify_ec_imm (push) Failing after 2m19s Details
Test / test_rebalance_verify_ec (push) Failing after 2m25s Details
2023-05-17 01:20:30 +03:00
Vitaliy Filippov 120e3fa7bc Fix pool deletion
Test / buildenv (push) Successful in 10s Details
Test / build (push) Successful in 2m32s Details
Test / test_cas (push) Successful in 13s Details
Test / make_test (push) Successful in 35s Details
Test / test_change_pg_size (push) Successful in 21s Details
Test / test_change_pg_count (push) Successful in 53s Details
Test / test_create_nomaxid (push) Successful in 17s Details
Test / test_change_pg_count_ec (push) Successful in 1m3s Details
Test / test_failure_domain (push) Successful in 16s Details
Test / test_etcd_fail (push) Successful in 1m3s Details
Test / test_add_osd (push) Successful in 2m36s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m10s Details
Test / test_interrupted_rebalance (push) Successful in 1m24s Details
Test / test_minsize_1 (push) Failing after 28s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m8s Details
Test / test_move_reappear (push) Failing after 1m2s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m8s Details
Test / test_rebalance_verify_imm (push) Successful in 2m12s Details
Test / test_rebalance_verify (push) Successful in 2m22s Details
Test / test_rm (push) Successful in 21s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m19s Details
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_rebalance_verify_ec (push) Successful in 2m33s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_write (push) Successful in 1m14s Details
Test / test_write_xor (push) Successful in 2m9s Details
Test / test_heal_ec (push) Successful in 4m25s Details
Test / test_heal_pg_size_2 (push) Successful in 4m59s Details
2023-05-17 00:45:59 +03:00
Vitaliy Filippov 629999f789 Clear journal_device and meta_device before initialising the next OSD in automatic mode 2023-05-15 23:58:55 +03:00
Vitaliy Filippov 93eca11ba2 Fix rhel 9 installation docs 2023-05-15 13:09:18 +03:00
Vitaliy Filippov 5a9e1ede52 Release 0.8.9
Test / buildenv (push) Successful in 9s Details
Test / build (push) Successful in 2m31s Details
Test / test_cas (push) Successful in 12s Details
Test / make_test (push) Successful in 33s Details
Test / test_change_pg_size (push) Successful in 19s Details
Test / test_change_pg_count (push) Successful in 55s Details
Test / test_create_nomaxid (push) Successful in 21s Details
Test / test_change_pg_count_ec (push) Successful in 58s Details
Test / test_failure_domain (push) Successful in 13s Details
Test / test_etcd_fail (push) Successful in 1m4s Details
Test / test_interrupted_rebalance (push) Successful in 1m13s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m7s Details
Test / test_add_osd (push) Successful in 2m59s Details
Test / test_move_reappear (push) Successful in 24s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m22s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m1s Details
Test / test_rebalance_verify (push) Successful in 2m12s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_rebalance_verify_imm (push) Successful in 2m4s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m9s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot (push) Successful in 23s Details
Test / test_rebalance_verify_ec (push) Successful in 2m31s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_write (push) Successful in 53s Details
Test / test_write_xor (push) Successful in 1m19s Details
Test / test_heal_pg_size_2 (push) Successful in 4m30s Details
Test / test_heal_ec (push) Successful in 4m32s Details
- The tests are now stable and run in a CI system based on Gitea CI
- The release includes final bug fixes for EC:
  - Implement missing EC recovery of allocation bitmap when built with ISA-L
  - Fix broken snapshot export with EC (allocation bitmap reads were giving incorrect results previously)
- Also fixed bugs manifesting under heavy load:
  - Fix monitor possibly applying incorrect PG history on retries
  - Fix monitor incorrectly changing PG count when last_clean_pgs contains less PGs than the new number
  - Allow writes to wait for free space again, but now correctly (previously dropped in 0.8.2)
  - Fix a rare segfault in client (handle client stop during incoming stream handling in 1 more place)
  - Make monitor correctly handle etcd connection errors - it could die instead of connecting to another etcd
  - Fix OSD rarely being unable to report PG states after a PG was taken over by another OSD
- Fixed return code for incomplete EC objects (now EIO) and made cluster client retry this error
- Made other small changes for tests: timeouts, nice/ionice for etcd, waiting conditions, NBD device checks and so on
2023-05-14 01:25:09 +03:00
Vitaliy Filippov 1c9a188600 Add tests to CI
Test / buildenv (push) Successful in 10s Details
Test / build (push) Successful in 10s Details
Test / test_cas (push) Successful in 12s Details
Test / make_test (push) Successful in 34s Details
Test / test_change_pg_size (push) Successful in 17s Details
Test / test_create_nomaxid (push) Successful in 9s Details
Test / test_change_pg_count (push) Successful in 1m29s Details
Test / test_failure_domain (push) Successful in 11s Details
Test / test_change_pg_count_ec (push) Successful in 1m35s Details
Test / test_etcd_fail (push) Successful in 52s Details
Test / test_add_osd (push) Successful in 2m13s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m4s Details
Test / test_interrupted_rebalance (push) Successful in 1m28s Details
Test / test_minsize_1 (push) Successful in 21s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m4s Details
Test / test_move_reappear (push) Successful in 30s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s Details
Test / test_rebalance_verify_imm (push) Successful in 2m14s Details
Test / test_rebalance_verify (push) Successful in 2m16s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m4s Details
Test / test_rm (push) Successful in 22s Details
Test / test_snapshot (push) Successful in 28s Details
Test / test_rebalance_verify_ec (push) Successful in 2m27s Details
Test / test_splitbrain (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 34s Details
Test / test_write_no_same (push) Successful in 19s Details
Test / test_write (push) Successful in 1m19s Details
Test / test_write_xor (push) Successful in 1m36s Details
Test / test_heal_pg_size_2 (push) Successful in 4m34s Details
Test / test_heal_ec (push) Successful in 4m21s Details
2023-05-14 00:06:09 +03:00
Vitaliy Filippov de3e609166 Add a FIXME about QEMU driver thread safety 2023-05-14 00:06:09 +03:00
Vitaliy Filippov 11481170f5 Add a FIXME about ENOSPC 2023-05-13 23:59:44 +03:00
Vitaliy Filippov e69d459d43 Allow rebalance to start in test_interrupted_rebalance, raise etcd start timeout 2023-05-13 15:16:28 +03:00
Vitaliy Filippov da82754baa Wait for conditions in test_move_reappear instead of waiting a fixed amount of time 2023-05-12 23:18:07 +03:00
Vitaliy Filippov d356aca030 Add missing $NO_SAME OSD argument to test_splitbrain 2023-05-12 23:18:07 +03:00
Vitaliy Filippov 04a273d213 Raise NBD timeout in tests 2023-05-12 23:18:07 +03:00
Vitaliy Filippov 6442010f93 Skip offline PGs during state reporting when the state is already deleted or taken over by another OSD
This fixes OSDs being unable to report PG states in rare conditions
2023-05-12 23:17:45 +03:00
Vitaliy Filippov 6f4dc16c59 Handle etcd connection errors correctly in mon (unhandled error events) 2023-05-11 11:02:44 +03:00
Vitaliy Filippov ce4a8067b5 Handle client stop during incoming stream handling in 1 more place 2023-05-11 01:53:41 +03:00
Vitaliy Filippov e431ecb715 Make tests more stable in CI 2023-05-11 01:53:41 +03:00
Vitaliy Filippov 8cac795445 Return EIO instead of EINVAL for incomplete EC objects 2023-05-11 01:15:23 +03:00
Vitaliy Filippov a409598b16 Wait for free space again, but count on big_write flushes instead of just flusher activity 2023-05-10 01:51:02 +03:00
Vitaliy Filippov f4c6765522 Ignore ENOENT in epoll_ctl 2023-05-08 20:39:20 +03:00
Vitaliy Filippov ad2916068a Fix test_add_osd rebalance timeout check 2023-05-08 20:39:20 +03:00
Vitaliy Filippov 321cb435a6 Fix monitor incorrectly changing PG count when last_clean_pgs contains less PGs than the new number 2023-05-08 20:39:20 +03:00
Vitaliy Filippov cfcf4f4355 Support checking /dev/nbdX nodes in Docker 2023-05-08 20:39:20 +03:00
Vitaliy Filippov e0fb17bfee Make etcd more stable in tests (add ionice and raise timeout) 2023-05-08 20:36:00 +03:00
Vitaliy Filippov 5b9031fecc Fix monitor possibly applying incorrect PG history under heavy load
Monitor could deceive itself by immediately saving PG configuration changes
which weren't applied to etcd yet in memory, and apply incorrect PG history
changes next time if the first update fails.

This usually only happened under heavy load and was caught in CI. :-)
2023-05-07 23:23:00 +03:00
Vitaliy Filippov 5da1d8e1b5 Fix EC just-bitmap reads (len=0) (fixes SCHEME=ec test_snapshot.sh) 2023-05-07 14:00:08 +03:00
Vitaliy Filippov 44f86f1999 Add a basic EC 2+2 recovery test (not really required, but let it be there) 2023-05-07 11:26:27 +03:00
Vitaliy Filippov 2d9a80c6f6 Implement missing bitmap recovery with ISA-L \(°□°)/ 2023-05-07 11:25:51 +03:00
Vitaliy Filippov 5e295e346e Do not make vitastor-mon part of vitastor.target 2023-04-29 00:17:47 +03:00
Vitaliy Filippov d9c0898b7c Notes about config and vitastor-disk cache status 2023-04-29 00:08:24 +03:00
Vitaliy Filippov 04cfb48361 Add a note about PVE 7.4 2023-04-28 11:37:11 +03:00
Vitaliy Filippov ab615849d6 Release 0.8.8
- Fix vitastor-cli rm/rm-data broken in 0.8.6 (missing messenger initialization)
- Prepare OSD read handler for upcoming version with scrub - allow "secondary reads" to return errors
- Fix OSDs re-peering PGs infinitely with a big number of PGs (reproduced in test_add_osd)
- Fix another variant of flusher sync-waiting stall (reproduced in test_write)
- Fix other tests in tests/ (will add them to Gitea CI soon)
- Add patches for QEMU 6.2-8.0
- Fix QEMU driver compatibility with QEMU 8.0
- Build packages for RHEL 9 clones (based on AlmaLinux 9)
2023-04-28 11:22:00 +03:00
Vitaliy Filippov 38be9a49c0 Add AlmaLinux 9 build to documentation 2023-04-28 02:00:52 +03:00
Vitaliy Filippov 7d6bf84a3e Add scripts/meson-buildoptions.sh to QEMU patches 2023-04-28 01:43:22 +03:00
Vitaliy Filippov 41a40a4123 Add QEMU spec patch for Alma/Rocky/RH 9 2023-04-28 01:32:06 +03:00
Vitaliy Filippov b94587ef0e Fix some build warnings 2023-04-28 00:44:27 +03:00
Vitaliy Filippov 2a2f4f6738 Add Almalinux 9 build 2023-04-28 00:40:50 +03:00
Vitaliy Filippov c768a9015f Fix QEMU driver compatibility with QEMU 8.0 2023-04-25 11:20:21 +03:00
Vitaliy Filippov 0d9e10cf96 Add patches for QEMU 6.2-8.0 2023-04-25 11:20:21 +03:00
Vitaliy Filippov b74ccb613c Fix another variant of flusher sync-waiting stall 2023-04-24 00:44:41 +03:00
Vitaliy Filippov 5052174918 Fix test_write_no_same (too large image) 2023-04-24 00:44:41 +03:00
Vitaliy Filippov eec9cf5575 Fix test_snapshot.sh - qemu-img requires explicit backing_fmt 2023-04-24 00:44:41 +03:00
Vitaliy Filippov a04dab0840 Initialize messenger in cluster_client listings 2023-04-24 00:44:41 +03:00
Vitaliy Filippov 160863f707 Print op pointer values in slow log 2023-04-23 17:54:00 +03:00
Vitaliy Filippov 2f16c32eb4 Fix test_minsize_1 (left_on_dead) 2023-04-23 17:54:00 +03:00
Vitaliy Filippov 2877cd0adb Allow OP_SEC_READ to return errors (do not hang the connection) 2023-04-23 17:54:00 +03:00
Vitaliy Filippov 480509f5b9 Fix pg_data_size > 1 for replicas (harmless bug) 2023-04-23 01:50:42 +03:00
Vitaliy Filippov 46462da45e Preload own PG history updates to fix PG state loop possibly applying the old metadata version 2023-04-23 01:50:30 +03:00
Vitaliy Filippov 024c8658f6 Fix missing } in quick start documentation 2023-04-12 18:26:38 +03:00
Vitaliy Filippov 7e958afeda Release 0.8.7
This release includes a bunch of important bugfixes for erasure-coded setups
with disabled immediate_commit. After these fixes, "test_heal" OSD killing test
now passes fine with EC:

- Fix cluster write stalls with "Error while doing flush on OSD xx: -16 (Device or resource busy)"
  in OSD logs possible in EC setups with disabled immediate_commit by selectively
  syncing nonsynced objects on STABILIZE/ROLLBACK (https://github.com/vitalif/vitastor/issues/51)
- Fix other EC + disabled immediate_commit problems:
  - Fix "opcode=5 retval=-2" errors happening on SYNC retries
  - Fix non-working "pagination" during PG dirty object flushing
  - Fix write operations not continued correctly after dirty object flushing
- Fix incorrect parity read-modify-write calculation when writing into a lost chunk
- Fix OSDs losing left_on_dead PG state of non-clean PGs and thus not removing junk data in the cluster
- Fix a small memory leak caused by bad indexing of EC recovery matrices
- Fix a rare use-after-free in cluster_client caused by a reenterability issue
- Fix vitastor-cli create command syntax in the CSI driver
- Allow to start OSDs without local store for tests
- Fix memory allocation error in disk_tool_meta for non-standard metadata block sizes
- Fix delete operations received before loading pool metadata crashing OSDs with "null pointer exception"
- Improve "theoretical performance" Russian documentation

New features:

- Implement online configuration update for some parameters. Documentation is coming soon :)
2023-04-11 02:11:57 +03:00
Vitaliy Filippov 2f5e769a29 Fix a small memory leak caused by bad indexing of EC recovery matrices 2023-04-11 00:30:36 +03:00
Vitaliy Filippov 28d5e53c6c Add test_heal to run_tests 2023-04-09 02:10:42 +03:00
Vitaliy Filippov d9f55f11d8 More logs (log_level 10), append to log instead of overwriting on restart in tests 2023-04-09 02:06:10 +03:00
Vitaliy Filippov 3237014608 Fix incorrect parity read-modify-write calculation when writing into a lost chunk 2023-04-09 02:06:10 +03:00
Vitaliy Filippov baaf8f6f44 Fix write operations not continued correctly after flush 2023-04-09 02:06:10 +03:00
Vitaliy Filippov 1d83fdcd17 Add debug logs to osd_flush 2023-04-09 02:06:10 +03:00
Vitaliy Filippov 0ddd787c38 Fix non-working "pagination" during PG dirty object flushing 2023-04-08 02:44:02 +03:00
Vitaliy Filippov 6eff3a60a5 Do not lose left_on_dead PG state of non-clean PGs 2023-04-08 02:44:02 +03:00
Vitaliy Filippov 888a6975ab Fix a rare use-after-free in cluster_client caused by a reenterability issue 2023-04-08 02:44:02 +03:00
Vitaliy Filippov cd1e890bd4 Fix "opcode=5 retval=-2" errors sometimes possible with EC 2023-04-08 02:44:02 +03:00
Vitaliy Filippov 0fbf4c6a08 Selectively sync nonsynced objects on STABILIZE/ROLLBACK (fix for github issue #51) 2023-04-08 02:44:02 +03:00
Vitaliy Filippov d06ed2b0e7 Implement online config update 2023-03-26 19:21:50 +03:00
Vitaliy Filippov 3bbc46543d Fix vitastor-cli create syntax 2023-03-17 11:12:58 +03:00
Vitaliy Filippov 2fb0c85618 Allow to start OSDs without local store (only for tests) 2023-03-15 01:13:59 +03:00
Vitaliy Filippov d81a6c04fc Update cmake min version so it does not complain about deprecation 2023-03-15 01:08:23 +03:00
Vitaliy Filippov 7b35801647 Fix possible bad realloc in disk_tool_meta for non-standard metadata block sizes 2023-03-15 01:08:23 +03:00
Vitaliy Filippov f3228d5c07 Fix typo (did not affect execution though) 2023-03-15 01:08:23 +03:00
Vitaliy Filippov 18366f5055 Fix read/write return type in rw_blocking 2023-03-15 01:08:14 +03:00
Vitaliy Filippov 851507c147 Add missing close() in test stubs 2023-03-15 00:23:56 +03:00
Vitaliy Filippov 9aaad28488 Fix "null pointer exception" for unhandled OSD_OP_DELETEs (when pool is not loaded yet) 2023-03-02 11:16:39 +03:00
Vitaliy Filippov dd57d086fe Add a missing part of the "theoretical performance" to the Russian version 2023-03-01 00:24:54 +03:00
Vitaliy Filippov 8810eae8fb Release 0.8.6
Important fixes:

- Fix possibly incorrect EC parity chunk updates with EC n+k, k > 1 and when
  the first parity chunk is missing

Minor fixes and improvements:

- Fix incorrect EC free space statistics in vitastor-cli df output
- Speedup vitastor-cli startup in clusters with RDMA
- Remove unused PG "peered" state (previously used to update PG epoch)
- Use sfdisk with just --json in vitastor-disk (--dump --json isn't needed)
- Allow trailing comma in sfdisk output (fixes sfdisk 2.36 compatibility)
- Slightly improve RDMA send/receive code
- Reduce RDMA memory consumption by default (rdma_max_recv/send = 16/8)
- Use vitastor-cli instead of direct etcd interaction in the CSI driver
2023-02-28 11:18:48 +03:00
Vitaliy Filippov c1365f46c9 Use vitastor-cli instead of direct etcd interaction in the CSI driver 2023-02-28 11:02:50 +03:00
Vitaliy Filippov 14d6acbcba Set default rdma_max_recv/send to 16/8, fix documentation 2023-02-28 11:00:56 +03:00
Vitaliy Filippov 1e307069bc Fix missing parity chunk calculation for EC n+k, k > 1 and first parity chunk missing 2023-02-28 02:40:19 +03:00
Vitaliy Filippov c3e80abad7 Allow to send more than 1 operation at a time 2023-02-26 02:01:04 +03:00
Vitaliy Filippov 138ffe4032 Reuse incoming RDMA buffers 2023-02-26 00:55:01 +03:00
Vitaliy Filippov 8139a34e97 Fix json11: allow trailing comma 2023-02-23 01:16:01 +03:00
Vitaliy Filippov 4ab630b44d Use just sfdisk --json, --dump is not needed 2023-02-23 00:55:47 +03:00
Vitaliy Filippov 2c8241b7db Remove PG "peered" state 2023-02-21 01:30:42 +03:00
Vitaliy Filippov 36a7dd3671 Move tests to "make test" 2023-02-21 01:30:42 +03:00
Vitaliy Filippov 936122bbcf Initialize msgr lazily in client to speedup vitastor-cli with RDMA enabled 2023-02-19 18:59:07 +03:00
Vitaliy Filippov 1a1ba0d1e7 Add set_immediate to ringloop and use it for bs/osd ops to prevent reenterability issues 2023-02-09 17:37:26 +03:00
Vitaliy Filippov 3d09c9cec7 Remove unused wait_sqe() from ringloop 2023-02-09 17:37:26 +03:00
Vitaliy Filippov 3d08a1ad6c Fix cluster_client test after last reenterability fixes 2023-02-05 01:47:32 +03:00
Vitaliy Filippov 499881d81c Fix typo 2023-01-27 01:52:02 +03:00
Vitaliy Filippov aba93b951b Fix incorrect EC free space statistics in vitastor-cli df output 2023-01-26 02:04:29 +03:00
Vitaliy Filippov d125fb1f30 Release 0.8.5
- Fix a possible "double free" bug in the client library happening on OSD restart
- Fix a possible write hang on PG history update when only epoch is changed
- Fix incorrect systemd target "local.target" in mon/make-etcd
- Allow "content" option in PVE storage plugin to allow to enable containers
- Build client library without tcmalloc which fixes "attempt to free invalid pointer"
  errors when, for example, trying to run QEMU with both Vitastor and Ceph RBD disks
2023-01-25 01:43:49 +03:00
Vitaliy Filippov 9d3fd72298 Require liburing < 2 in rpm specs 2023-01-25 01:43:49 +03:00
Vitaliy Filippov 8b552a01f9 Do not retry successful operation parts in client (could lead to "double free" bugs) 2023-01-25 01:30:36 +03:00
Vitaliy Filippov 0385b2f9e8 Fix write hangs on PG epoch update - always set pg.history_changed to true 2023-01-25 01:30:15 +03:00
Vitaliy Filippov 749c837045 Replace non-existing local.target with multi-user.target 2023-01-25 01:29:31 +03:00
Vitaliy Filippov 98001d845b Remove version from vitastor-release.rpm links 2023-01-23 14:03:33 +03:00
Vitaliy Filippov c96bcae74b Allow "content" option in PVE storage plugin to allow to enable containers 2023-01-16 18:14:45 +03:00
Vitaliy Filippov 9f4e34a8cc Build client library without tcmalloc
Fixes "[src/tcmalloc.cc:332] Attempt to free invalid pointer ..." when trying
to run QEMU with both Vitastor and Ceph RBD disks and other possible allocator
collisions.
2023-01-15 00:01:11 +03:00
Vitaliy Filippov 81fc8bb94c Release 0.8.4
New features:
- Implement QCOW2 image/snapshot export via qemu-img (bdrv_co_block_status in the driver)
- Remove OSDs from PG history during `vitastor-cli rm-osd` to prevent `left_on_dead` PG states after deletion
- Add a new recovery_pg_switch setting to mix all PGs during recovery, to almost
  fully reduce the probability of ENOSPC during rebalance
- Introduce partial ENOSPC ("OSD is full") handling - now ENOSPC doesn't turn
  into cascades of crashes
- Add migration support to Proxmox VE Vitastor driver
- Track last_clean_pgs on a per-pool basis thus reducing data movement in a cluster
  with pools remaining unclean/degraded for a long time

Bug fixes:
- Fix a bug where monitor could generate degraded PGs if one of the hosts had no OSDs
- Fix a bug where monitor could skip PG redistribution with a lot of OSDs in cluster
- Report PG history synchronously on the first write, which improves PG consistency
  and availability at the same time, because history now gets reported correctly
  and doesn't get reported without the need for it
- Fix possible write and recovery stalls which could happen in a cluster with both EC and replicated pools
- Make OSD and monitors sanitize & deduplicate PG history items in etcd
- Fix non-working OSD peer config safety check
- Fix a rare journal flush stall where flushing wasn't activated with full journal, but with empty flush queue
- Fix builds without ISA-L (jerasure-only) crashing with EC N+K, K>=2 due to the lack of 16-byte buffer alignment
- Fix a possible crash for EC N+K, K>=2 when calculating a parity chunk with previous parity chunk missing
- Fix a bug where vitastor-disk purge with suppressed warnings didn't work
2023-01-13 23:59:54 +03:00
Vitaliy Filippov bc465c16de Fix arithmetic on void* for clang 2023-01-13 23:58:42 +03:00
Vitaliy Filippov 8763e9211c Fix qemu driver compilation warning/error 2023-01-13 23:44:39 +03:00
Vitaliy Filippov 9e1a80bd17 Replace apt-key with trusted.gpg.d 2023-01-13 19:51:47 +03:00
Vitaliy Filippov 3e280f2f08 Mark vitastor as shared storage in PVE driver 2023-01-13 01:36:30 +03:00
Vitaliy Filippov fe87b4076b Fix backwards compatibility in cluster_client 2023-01-12 02:37:31 +03:00
Vitaliy Filippov a38957c1a7 Skip empty hosts in lp-optimizer 2023-01-09 16:26:16 +03:00
Vitaliy Filippov 137309cf29 Implement bdrv_co_block_status for snapshot export support 2023-01-07 17:06:58 +03:00
315 changed files with 33549 additions and 4804 deletions

View File

@ -0,0 +1,36 @@
FROM node:16-bullseye
WORKDIR /root
ADD ./docker/vitastor.gpg /etc/apt/trusted.gpg.d
RUN echo 'deb http://deb.debian.org/debian bullseye-backports main' >> /etc/apt/sources.list; \
echo 'deb http://vitastor.io/debian bullseye main' >> /etc/apt/sources.list; \
echo >> /etc/apt/preferences; \
echo 'Package: *' >> /etc/apt/preferences; \
echo 'Pin: release a=bullseye-backports' >> /etc/apt/preferences; \
echo 'Pin-Priority: 500' >> /etc/apt/preferences; \
echo >> /etc/apt/preferences; \
echo 'Package: *' >> /etc/apt/preferences; \
echo 'Pin: origin "vitastor.io"' >> /etc/apt/preferences; \
echo 'Pin-Priority: 1000' >> /etc/apt/preferences; \
grep '^deb ' /etc/apt/sources.list | perl -pe 's/^deb/deb-src/' >> /etc/apt/sources.list; \
echo 'APT::Install-Recommends false;' >> /etc/apt/apt.conf; \
echo 'APT::Install-Suggests false;' >> /etc/apt/apt.conf
RUN apt-get update
RUN apt-get -y install etcd qemu-system-x86 qemu-block-extra qemu-utils fio libasan5 \
liburing1 liburing-dev libgoogle-perftools-dev devscripts libjerasure-dev cmake libibverbs-dev libisal-dev
RUN apt-get -y build-dep fio qemu=`dpkg -s qemu-system-x86|grep ^Version:|awk '{print $2}'`
RUN apt-get -y install jq lp-solve sudo nfs-common
RUN apt-get --download-only source fio qemu=`dpkg -s qemu-system-x86|grep ^Version:|awk '{print $2}'`
RUN set -ex; \
mkdir qemu-build; \
cd qemu-build; \
dpkg-source -x /root/qemu*.dsc; \
cd qemu*/; \
debian/rules configure-qemu || debian/rules b/configure-stamp; \
cd b/qemu; \
make -j8 config-poison.h || true; \
make -j8 qapi/qapi-builtin-types.h

View File

@ -0,0 +1,19 @@
FROM git.yourcmc.ru/vitalif/vitastor/buildenv
ADD . /root/vitastor
RUN set -e -x; \
mkdir -p /root/fio-build/; \
cd /root/fio-build/; \
dpkg-source -x /root/fio*.dsc; \
cd /root/vitastor; \
ln -s /root/fio-build/fio-*/ ./fio; \
ln -s /root/qemu-build/qemu-*/ ./qemu; \
ls /usr/include/linux/raw.h || cp ./debian/raw.h /usr/include/linux/raw.h; \
cd mon; \
npm install; \
cd ..; \
mkdir build; \
cd build; \
cmake .. -DWITH_ASAN=yes -DWITH_QEMU=yes; \
make -j16

876
.gitea/workflows/test.yml Normal file
View File

@ -0,0 +1,876 @@
name: Test
on:
push:
branches:
- '*'
paths:
- '.gitea/**'
- 'src/**'
- 'mon/**'
- 'json11'
- 'cpp-btree'
- 'tests/**'
env:
BUILDENV_IMAGE: git.yourcmc.ru/vitalif/vitastor/buildenv
TEST_IMAGE: git.yourcmc.ru/vitalif/vitastor/test
OSD_ARGS: '--etcd_quick_timeout 2000'
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
buildenv:
runs-on: ubuntu-latest
container: git.yourcmc.ru/vitalif/gitea-ci-dind
steps:
- uses: actions/checkout@v3
- name: Build and push
run: |
set -ex
if ! docker manifest inspect $BUILDENV_IMAGE >/dev/null; then
docker build -t $BUILDENV_IMAGE -f .gitea/workflows/buildenv.Dockerfile .
docker login git.yourcmc.ru -u vitalif -p "${{secrets.TOKEN}}"
docker push $BUILDENV_IMAGE
fi
build:
runs-on: ubuntu-latest
needs: buildenv
container: git.yourcmc.ru/vitalif/gitea-ci-dind
steps:
- uses: actions/checkout@v3
with:
submodules: true
- name: Build and push
run: |
set -ex
if ! docker manifest inspect $TEST_IMAGE:$GITHUB_SHA >/dev/null; then
docker build -t $TEST_IMAGE:$GITHUB_SHA -f .gitea/workflows/test.Dockerfile .
docker login git.yourcmc.ru -u vitalif -p "${{secrets.TOKEN}}"
docker push $TEST_IMAGE:$GITHUB_SHA
fi
make_test:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
# leak sanitizer sometimes crashes
- run: cd /root/vitastor/build && ASAN_OPTIONS=detect_leaks=0 make -j16 test
test_add_osd:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: /root/vitastor/tests/test_add_osd.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_cas:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_cas.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_change_pg_count:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_change_pg_count.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_change_pg_count_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: SCHEME=ec /root/vitastor/tests/test_change_pg_count.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_change_pg_size:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_change_pg_size.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_create_nomaxid:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_create_nomaxid.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_etcd_fail:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: /root/vitastor/tests/test_etcd_fail.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_interrupted_rebalance:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: /root/vitastor/tests/test_interrupted_rebalance.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_interrupted_rebalance_imm:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: IMMEDIATE_COMMIT=1 /root/vitastor/tests/test_interrupted_rebalance.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_interrupted_rebalance_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: SCHEME=ec /root/vitastor/tests/test_interrupted_rebalance.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_interrupted_rebalance_ec_imm:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: SCHEME=ec IMMEDIATE_COMMIT=1 /root/vitastor/tests/test_interrupted_rebalance.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_failure_domain:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_failure_domain.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_snapshot:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_snapshot.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_snapshot_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: SCHEME=ec /root/vitastor/tests/test_snapshot.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_minsize_1:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_minsize_1.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_move_reappear:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_move_reappear.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_rm:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_rm.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_snapshot_chain:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_snapshot_chain.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_snapshot_chain_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 6
run: SCHEME=ec /root/vitastor/tests/test_snapshot_chain.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_snapshot_down:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_snapshot_down.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_snapshot_down_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: SCHEME=ec /root/vitastor/tests/test_snapshot_down.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_splitbrain:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_splitbrain.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_rebalance_verify:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: /root/vitastor/tests/test_rebalance_verify.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_rebalance_verify_imm:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: IMMEDIATE_COMMIT=1 /root/vitastor/tests/test_rebalance_verify.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_rebalance_verify_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: SCHEME=ec /root/vitastor/tests/test_rebalance_verify.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_rebalance_verify_ec_imm:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: SCHEME=ec IMMEDIATE_COMMIT=1 /root/vitastor/tests/test_rebalance_verify.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_switch_primary:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_switch_primary.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_write:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_write.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_write_xor:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: SCHEME=xor /root/vitastor/tests/test_write.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_write_no_same:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_write_no_same.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_heal_pg_size_2:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: PG_SIZE=2 /root/vitastor/tests/test_heal.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_heal_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: SCHEME=ec /root/vitastor/tests/test_heal.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_heal_csum_32k_dmj:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: TEST_NAME=csum_32k_dmj OSD_ARGS="--data_csum_type crc32c --csum_block_size 32k --inmemory_metadata false --inmemory_journal false" OFFSET_ARGS=$OSD_ARGS /root/vitastor/tests/test_heal.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_heal_csum_32k_dj:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: TEST_NAME=csum_32k_dj OSD_ARGS="--data_csum_type crc32c --csum_block_size 32k --inmemory_journal false" OFFSET_ARGS=$OSD_ARGS /root/vitastor/tests/test_heal.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_heal_csum_32k:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: TEST_NAME=csum_32k OSD_ARGS="--data_csum_type crc32c --csum_block_size 32k" OFFSET_ARGS=$OSD_ARGS /root/vitastor/tests/test_heal.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_heal_csum_4k_dmj:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: TEST_NAME=csum_4k_dmj OSD_ARGS="--data_csum_type crc32c --inmemory_metadata false --inmemory_journal false" OFFSET_ARGS=$OSD_ARGS /root/vitastor/tests/test_heal.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_heal_csum_4k_dj:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: TEST_NAME=csum_4k_dj OSD_ARGS="--data_csum_type crc32c --inmemory_journal false" OFFSET_ARGS=$OSD_ARGS /root/vitastor/tests/test_heal.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_heal_csum_4k:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 10
run: TEST_NAME=csum_4k OSD_ARGS="--data_csum_type crc32c" OFFSET_ARGS=$OSD_ARGS /root/vitastor/tests/test_heal.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_scrub:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_scrub.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_scrub_zero_osd_2:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: ZERO_OSD=2 /root/vitastor/tests/test_scrub.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_scrub_xor:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: SCHEME=xor /root/vitastor/tests/test_scrub.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_scrub_pg_size_3:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: PG_SIZE=3 /root/vitastor/tests/test_scrub.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: PG_SIZE=6 PG_MINSIZE=4 OSD_COUNT=6 SCHEME=ec /root/vitastor/tests/test_scrub.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_scrub_ec:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: SCHEME=ec /root/vitastor/tests/test_scrub.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done
test_nfs:
runs-on: ubuntu-latest
needs: build
container: ${{env.TEST_IMAGE}}:${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: 3
run: /root/vitastor/tests/test_nfs.sh
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- $i --------"
cat $i
echo ""
done

View File

@ -0,0 +1,79 @@
#!/usr/bin/perl
use strict;
for my $line (<>)
{
if ($line =~ /\.\/(test_[^\.]+)/s)
{
chomp $line;
my $base_name = $1;
my $test_name = $base_name;
my $timeout = 3;
if ($test_name eq 'test_etcd_fail' || $test_name eq 'test_heal' || $test_name eq 'test_add_osd' ||
$test_name eq 'test_interrupted_rebalance' || $test_name eq 'test_rebalance_verify')
{
$timeout = 10;
}
while ($line =~ /([^\s=]+)=(\S+)/gs)
{
if ($1 eq 'TEST_NAME')
{
$test_name = $base_name.'_'.$2;
last;
}
elsif ($1 eq 'SCHEME' && $2 eq 'ec')
{
$test_name .= '_ec';
}
elsif ($1 eq 'SCHEME' && $2 eq 'xor')
{
$test_name .= '_xor';
}
elsif ($1 eq 'IMMEDIATE_COMMIT')
{
$test_name .= '_imm';
}
else
{
$test_name .= '_'.lc($1).'_'.$2;
}
}
if ($test_name eq 'test_snapshot_chain_ec')
{
$timeout = 6;
}
$line =~ s!\./test_!/root/vitastor/tests/test_!;
# Gitea CI doesn't support artifacts yet, lol
#- name: Upload results
# uses: actions/upload-artifact\@v3
# if: always()
# with:
# name: ${test_name}_result
# path: |
# /root/vitastor/testdata
# !/root/vitastor/testdata/*.bin
# retention-days: 5
print <<"EOF"
$test_name:
runs-on: ubuntu-latest
needs: build
container: \${{env.TEST_IMAGE}}:\${{github.sha}}
steps:
- name: Run test
id: test
timeout-minutes: $timeout
run: $line
- name: Print logs
if: always() && steps.test.outcome == 'failure'
run: |
for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
echo "-------- \$i --------"
cat \$i
echo ""
done
EOF
;
}
}

115
CLA-en.md Normal file
View File

@ -0,0 +1,115 @@
## Contributor License Agreement
> This Agreement is made in the Russian and English languages. **The English
text of Agreement is for informational purposes only** and is not binding
for the Parties.
>
> In the event of a conflict between the provisions of the Russian and
English versions of this Agreement, the **Russian version shall prevail**.
>
> Russian version is published at https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-ru.md
This document represents the offer of Filippov Vitaliy Vladimirovich
("Author"), author and copyright holder of Vitastor software ("Program"),
acknowledged by a certificate of Federal Service for Intellectual
Property of Russian Federation (Rospatent) # 2021617829 dated 20 May 2021,
to "Contributors" to conclude this license agreement as follows
("Agreement" or "Offer").
In accordance with Art. 435, Art. 438 of the Civil Code of the Russian
Federation, this Agreement is an offer and in case of acceptance of the
offer, an agreement is considered concluded on the conditions specified
in the offer.
1. Applicable Terms. \
1.1. "Official Repository" shall mean the computer storage, operated by
the Author, containing all prior and future versions of the Source
Code of the Program, at Internet addresses https://git.yourcmc.ru/vitalif/vitastor/
or https://github.com/vitalif/vitastor/. \
1.2. "Contributions" shall mean results of intellectual activity
(including, but not limited to, source code, libraries, components,
texts, documentation) which can be software or elements of the software
and which are provided by Contributors to the Author for inclusion
in the Program. \
1.3. "Contributor" shall mean a person who provides Contributions to
the Author and agrees with all provisions of this Agreement.
A Сontributor can be: 1) an individual; or 2) a legal entity or an
individual entrepreneur in case when an individual provides Contributions
on behalf of third parties, including on behalf of his employer.
2. Subject of the Agreement. \
2.1. Subject of the Agreement shall be the Contributions sent to the Author by Contributors. \
2.2. The Contributor grants to the Author the right to use Contributions at his own
discretion and without any necessity to get a prior approval from Contributor or
any other third party in any way, under a simple (non-exclusive), royalty-free,
irrevocable license throughout the world by all means not contrary to law, in whole
or as a part of the Program, or other open-source or closed-source computer programs,
products or services (hereinafter -- the "License"), including, but not limited to: \
2.2.1. to execute Contributions and use them for any tasks; \
2.2.2. to publish and distribute Contributions in modified or unmodified form and/or to rent them; \
2.2.3. to modify Contributions, add comments, illustrations or any explanations to Contributions while using them; \
2.2.4. to create other results of intellectual activity based on Contributions, including derivative works and composite works; \
2.2.5. to translate Contributions into other languages, including other programming languages; \
2.2.6. to carry out rental and public display of Contributions; \
2.2.7. to use Contributions under the trade name and/or any trademark or any other label, or without it, as the Author thinks fit; \
2.3. The Contributor grants to the Author the right to sublicense any of the aforementioned
rights to third parties on any terms at the Author's discretion. \
2.4. The License is provided for the entire duration of Contributor's
exclusive intellectual property rights to the Contributions. \
2.5. The Contributor grants to the Author the right to decide how and where to mention,
or to not mention at all, the fact of his authorship, name, nickname and/or company
details when including Contributions into the Program or in any other computer
programs, products or services.
3. Acceptance of the Offer \
3.1. The Contributor may provide Contributions to the Author in the form of
a "Pull Request" in an Official Repository of the Program or by any
other electronic means of communication, including, but not limited to,
E-mail or messenger applications. \
3.2. The acceptance of the Offer shall be the fact of provision of Contributions
to the Author by the Contributor by any means with the following remark:
“I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md”
or “Я принимаю соглашение Vitastor CLA: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-ru.md”. \
3.3. Date of acceptance of the Offer shall be the date of such provision.
4. Rights and obligations of the parties. \
4.1. The Contributor reserves the right to use Contributions by any lawful means
not contrary to this Agreement. \
4.2. The Author has the right to refuse to include Contributions into the Program
at any moment with no explanation to the Contributor.
5. Representations and Warranties. \
5.1. The person providing Contributions for the purpose of their inclusion
in the Program represents and warrants that he is the Contributor
or legally acts on the Contributor's behalf. Name or company details
of the Contributor shall be provided with the Contribution at the moment
of their provision to the Author. \
5.2. The Contributor represents and warrants that he legally owns exclusive
intellectual property rights to the Contributions. \
5.3. The Contributor represents and warrants that any further use of
Contributions by the Author as provided by Contributor under the terms
of the Agreement does not infringe on intellectual and other rights and
legitimate interests of third parties. \
5.4. The Contributor represents and warrants that he has all rights and legal
capacity needed to accept this Offer; \
5.5. The Contributor represents and warrants that Contributions don't
contain malware or any information considered illegal under the law
of Russian Federation.
6. Termination of the Agreement \
6.1. The Agreement may be terminated at will of both Author and Contributor,
formalised in the written form or if the Agreement is terminated on
reasons prescribed by the law of Russian Federation.
7. Final Clauses \
7.1. The Contributor may optionally sign the Agreement in the written form. \
7.2. The Agreement is deemed to become effective from the Date of signing of
the Agreement and until the expiration of Contributor's exclusive
intellectual property rights to the Contributions. \
7.3. The Author may unilaterally alter the Agreement without informing Contributors.
The new version of the document shall come into effect 3 (three) days after
being published in the Official Repository of the Program at Internet address
[https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md).
Contributors should keep informed about the actual version of the Agreement themselves. \
7.4. If the Author and the Contributor fail to agree on disputable issues,
disputes shall be referred to the Moscow Arbitration court.

108
CLA-ru.md Normal file
View File

@ -0,0 +1,108 @@
## Лицензионное соглашение с участником
> Данная Оферта написана в Русской и Английской версиях. **Версия на английском
языке предоставляется в информационных целях** и не связывает стороны договора.
>
> В случае несоответствий между положениями Русской и Английской версий Договора,
**Русская версия имеет приоритет**.
>
> Английская версия опубликована по адресу https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
Настоящий договор-оферта (далее по тексту Оферта, Договор) адресована физическим
и юридическим лицам (далее Участникам) и является официальным публичным предложением
Филиппова Виталия Владимировича (далее Автора) программного обеспечения Vitastor,
свидетельство Федеральной службы по интеллектуальной собственности (Роспатент) № 2021617829
от 20 мая 2021 г. (далее Программа) о нижеследующем:
1. Термины и определения \
1.1. Репозиторий электронное хранилище, содержащее исходный код Программы. \
1.2. Доработка результат интеллектуальной деятельности Участника, включающий
в себя изменения или дополнения к исходному коду Программы, которые Участник
желает включить в состав Программы для дальнейшего использования и распространения
Автором и для этого направляет их Автору. \
1.3. Участник физическое или юридическое лицо, вносящее Доработки в код Программы. \
1.4. ГК РФ Гражданский кодекс Российской Федерации.
2. Предмет оферты \
2.1. Предметом настоящей оферты являются Доработки, отправляемые Участником Автору. \
2.2. Участник предоставляет Автору право использовать Доработки по собственному усмотрению
и без необходимости предварительного согласования с Участником или иным третьим лицом
на условиях простой (неисключительной) безвозмездной безотзывной лицензии, полностью
или фрагментарно, в составе Программы или других программ, продуктов или сервисов
как с открытым, так и с закрытым исходным кодом, любыми способами, не противоречащими
закону, включая, но не ограничиваясь следующими: \
2.2.1. Запускать и использовать Доработки для выполнения любых задач; \
2.2.2. Распространять, импортировать и доводить Доработки до всеобщего сведения; \
2.2.3. Вносить в Доработки изменения, сокращения и дополнения, снабжать Доработки
при их использовании комментариями, иллюстрациями или пояснениями; \
2.2.4. Создавать на основе Доработок иные результаты интеллектуальной деятельности,
в том числе производные и составные произведения; \
2.2.5. Переводить Доработки на другие языки, в том числе на другие языки программирования; \
2.2.6. Осуществлять прокат и публичный показ Доработок; \
2.2.7. Использовать Доработки под любым фирменным наименованием, товарным знаком
(знаком обслуживания) или иным обозначением, или без такового. \
2.3. Участник предоставляет Автору право сублицензировать полученные права на Доработки
третьим лицам на любых условиях на усмотрение Автора. \
2.4. Участник предоставляет Автору права на Доработки на территории всего мира. \
2.5. Участник предоставляет Автору права на весь срок действия исключительного права
Участника на Доработки. \
2.6. Участник предоставляет Автору права на Доработки на безвозмездной основе. \
2.7. Участник разрешает Автору самостоятельно определять порядок, способ и
место указания его имени, реквизитов и/или псевдонима при включении
Доработок в состав Программы или других программ, продуктов или сервисов.
3. Акцепт Оферты \
3.1. Участник может передавать Доработки в адрес Автора через зеркала официального
Репозитория Программы по адресам https://git.yourcmc.ru/vitalif/vitastor/ или
https://github.com/vitalif/vitastor/ в виде “запроса на слияние” (pull request),
либо в письменном виде или с помощью любых других электронных средств коммуникации,
например, электронной почты или мессенджеров. \
3.2. Факт передачи Участником Доработок в адрес Автора любым способом с одной из пометок
“I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md”
или “Я принимаю соглашение Vitastor CLA: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-ru.md”
является полным и безоговорочным акцептом (принятием) Участником условий настоящей
Оферты, т.е. Участник считается ознакомившимся с настоящим публичным договором и
в соответствии с ГК РФ признается лицом, вступившим с Автором в договорные отношения
на основании настоящей Оферты. \
3.3. Датой акцептирования настоящей Оферты считается дата такой передачи.
4. Права и обязанности Сторон \
4.1. Участник сохраняет за собой право использовать Доработки любым законным
способом, не противоречащим настоящему Договору. \
4.2. Автор вправе отказать Участнику во включении Доработок в состав
Программы без объяснения причин в любой момент по своему усмотрению.
5. Гарантии и заверения \
5.1. Лицо, направляющее Доработки для целей их включения в состав Программы,
гарантирует, что является Участником или представителем Участника. Имя или реквизиты
Участника должны быть указаны при их передаче в адрес Автора Программы. \
5.2. Участник гарантирует, что является законным обладателем исключительных прав
на Доработки. \
5.3. Участник гарантирует, что на момент акцептирования настоящей Оферты ему
ничего не известно (и не могло быть известно) о правах третьих лиц на
передаваемые Автору Доработки или их часть, которые могут быть нарушены
в связи с передачей Доработок по настоящему Договору. \
5.4. Участник гарантирует, что является дееспособным лицом и обладает всеми
необходимыми правами для заключения Договора. \
5.5. Участник гарантирует, что Доработки не содержат вредоносного ПО, а также
любой другой информации, запрещённой к распространению по законам Российской
Федерации.
6. Прекращение действия оферты \
6.1. Действие настоящего договора может быть прекращено по соглашению сторон,
оформленному в письменном виде, а также вследствие его расторжения по основаниям,
предусмотренным законом.
7. Заключительные положения \
7.1. Участник вправе по желанию подписать настоящий Договор в письменном виде. \
7.2. Настоящий договор действует с момента его заключения и до истечения срока
действия исключительных прав Участника на Доработки. \
7.3. Автор имеет право в одностороннем порядке вносить изменения и дополнения в договор
без специального уведомления об этом Участников. Новая редакция документа вступает
в силу через 3 (Три) календарных дня со дня опубликования в официальном Репозитории
Программы по адресу в сети Интернет
[https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-ru.md](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-ru.md).
Участники самостоятельно отслеживают действующие условия Оферты. \
7.4. Все споры, возникающие между сторонами в процессе их взаимодействия по настоящему
договору, решаются путём переговоров. В случае невозможности урегулирования споров
переговорным порядком стороны разрешают их в Арбитражном суде г.Москвы.

View File

@ -1,7 +1,7 @@
cmake_minimum_required(VERSION 2.8)
cmake_minimum_required(VERSION 2.8.12)
project(vitastor)
set(VERSION "0.8.3")
set(VERSION "1.5.0")
add_subdirectory(src)

View File

@ -6,8 +6,8 @@
Вернём былую скорость кластерному блочному хранилищу!
Vitastor - распределённая блочная SDS (программная СХД), прямой аналог Ceph RBD и
внутренних СХД популярных облачных провайдеров. Однако, в отличие от них, Vitastor
Vitastor - распределённая блочная и файловая SDS (программная СХД), прямой аналог Ceph RBD и CephFS,
а также внутренних СХД популярных облачных провайдеров. Однако, в отличие от них, Vitastor
быстрый и при этом простой. Только пока маленький :-).
Vitastor архитектурно похож на Ceph, что означает атомарность и строгую консистентность,
@ -15,7 +15,7 @@ Vitastor архитектурно похож на Ceph, что означает
и автоматическое распределение данных по любому числу дисков любого размера с настраиваемыми схемами
избыточности - репликацией или с произвольными кодами коррекции ошибок.
Vitastor нацелен на SSD и SSD+HDD кластеры с как минимум 10 Гбит/с сетью, поддерживает
Vitastor нацелен в первую очередь на SSD и SSD+HDD кластеры с как минимум 10 Гбит/с сетью, поддерживает
TCP и RDMA и на хорошем железе может достигать задержки 4 КБ чтения и записи на уровне ~0.1 мс,
что примерно в 10 раз быстрее, чем Ceph и другие популярные программные СХД.
@ -50,6 +50,7 @@ Vitastor поддерживает QEMU-драйвер, протоколы NBD и
- Параметры
- [Общие](docs/config/common.ru.md)
- [Сетевые](docs/config/network.ru.md)
- [Клиентский код](docs/config/client.en.md)
- [Глобальные дисковые параметры](docs/config/layout-cluster.ru.md)
- [Дисковые параметры OSD](docs/config/layout-osd.ru.md)
- [Прочие параметры OSD](docs/config/osd.ru.md)
@ -62,11 +63,12 @@ Vitastor поддерживает QEMU-драйвер, протоколы NBD и
- [fio](docs/usage/fio.ru.md) для тестов производительности
- [NBD](docs/usage/nbd.ru.md) для монтирования ядром
- [QEMU и qemu-img](docs/usage/qemu.ru.md)
- [NFS](docs/usage/nfs.ru.md)-прокси для VMWare и подобных
- [NFS](docs/usage/nfs.ru.md) кластерная файловая система и псевдо-ФС прокси
- Производительность
- [Понимание сути производительности](docs/performance/understanding.ru.md)
- [Теоретический максимум](docs/performance/theoretical.ru.md)
- [Пример сравнения с Ceph](docs/performance/comparison1.ru.md)
- [Более новый тест Vitastor 1.3.1](docs/performance/bench2.ru.md)
## Автор и лицензия

View File

@ -6,16 +6,16 @@
Make Clustered Block Storage Fast Again.
Vitastor is a distributed block SDS, direct replacement of Ceph RBD and internal SDS's
of public clouds. However, in contrast to them, Vitastor is fast and simple at the same time.
The only thing is it's slightly young :-).
Vitastor is a distributed block and file SDS, direct replacement of Ceph RBD and CephFS,
and also internal SDS's of public clouds. However, in contrast to them, Vitastor is fast
and simple at the same time. The only thing is it's slightly young :-).
Vitastor is architecturally similar to Ceph which means strong consistency,
primary-replication, symmetric clustering and automatic data distribution over any
number of drives of any size with configurable redundancy (replication or erasure codes/XOR).
Vitastor targets SSD and SSD+HDD clusters with at least 10 Gbit/s network, supports
TCP and RDMA and may achieve 4 KB read and write latency as low as ~0.1 ms
Vitastor targets primarily SSD and SSD+HDD clusters with at least 10 Gbit/s network,
supports TCP and RDMA and may achieve 4 KB read and write latency as low as ~0.1 ms
with proper hardware which is ~10 times faster than other popular SDS's like Ceph
or internal systems of public clouds.
@ -50,6 +50,7 @@ Read more details below in the documentation.
- Parameter Reference
- [Common](docs/config/common.en.md)
- [Network](docs/config/network.en.md)
- [Client](docs/config/client.en.md)
- [Global Disk Layout](docs/config/layout-cluster.en.md)
- [OSD Disk Layout](docs/config/layout-osd.en.md)
- [OSD Runtime Parameters](docs/config/osd.en.md)
@ -62,11 +63,12 @@ Read more details below in the documentation.
- [fio](docs/usage/fio.en.md) for benchmarks
- [NBD](docs/usage/nbd.en.md) for kernel mounts
- [QEMU and qemu-img](docs/usage/qemu.en.md)
- [NFS](docs/usage/nfs.en.md) emulator for VMWare and similar
- [NFS](docs/usage/nfs.en.md) clustered file system and pseudo-FS proxy
- Performance
- [Understanding storage performance](docs/performance/understanding.en.md)
- [Theoretical performance](docs/performance/theoretical.en.md)
- [Example comparison with Ceph](docs/performance/comparison1.en.md)
- [Newer benchmark of Vitastor 1.3.1](docs/performance/bench2.en.md)
## Author and License

View File

@ -48,9 +48,9 @@ Vitastor, составлены для того, чтобы убедиться,
интерфейс (прокси), опять же, без открытия в свободный публичный доступ как
самой программы, так и прокси.
Сетевая Публичная Лицензия Vitastor разработана специально чтобы
Сетевая Публичная Лицензия Vitastor разработана специально, чтобы
гарантировать, что в таких случаях и модифицированная версия программы, и
прокси оставались доступными сообществу. Для этого лицензия требует от
прокси останутся доступными сообществу. Для этого лицензия требует от
операторов сетевых серверов предоставлять исходный код оригинальной программы,
а также всех других программ, взаимодействующих с ней на их серверах,
пользователям этих серверов, на условиях свободных лицензий. Таким образом,

@ -1 +1 @@
Subproject commit 45e6d1f13196a0824e2089a586c53b9de0283f17
Subproject commit 8de8b467acbca50cfd8835c20e0e379110f3b32b

View File

@ -1,14 +1,15 @@
# Compile stage
FROM golang:buster AS build
FROM golang:bookworm AS build
ADD go.sum go.mod /app/
RUN cd /app; CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go mod download -x
ADD . /app
RUN perl -i -e '$/ = undef; while(<>) { s/\n\s*(\{\s*\n)/$1\n/g; s/\}(\s*\n\s*)else\b/$1} else/g; print; }' `find /app -name '*.go'`
RUN cd /app; CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -o vitastor-csi
RUN perl -i -e '$/ = undef; while(<>) { s/\n\s*(\{\s*\n)/$1\n/g; s/\}(\s*\n\s*)else\b/$1} else/g; print; }' `find /app -name '*.go'` && \
cd /app && \
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -o vitastor-csi
# Final stage
FROM debian:buster
FROM debian:bookworm
LABEL maintainers="Vitaliy Filippov <vitalif@yourcmc.ru>"
LABEL description="Vitastor CSI Driver"
@ -18,19 +19,30 @@ ENV CSI_ENDPOINT=""
RUN apt-get update && \
apt-get install -y wget && \
(echo deb http://deb.debian.org/debian buster-backports main > /etc/apt/sources.list.d/backports.list) && \
(echo "APT::Install-Recommends false;" > /etc/apt/apt.conf) && \
apt-get update && \
apt-get install -y e2fsprogs xfsprogs kmod && \
apt-get install -y e2fsprogs xfsprogs kmod iproute2 \
# dependencies of qemu-storage-daemon
libnuma1 liburing2 libglib2.0-0 libfuse3-3 libaio1 libzstd1 libnettle8 \
libgmp10 libhogweed6 libp11-kit0 libidn2-0 libunistring2 libtasn1-6 libpcre2-8-0 libffi8 && \
apt-get clean && \
(echo options nbd nbds_max=128 > /etc/modprobe.d/nbd.conf)
COPY --from=build /app/vitastor-csi /bin/
RUN (echo deb http://vitastor.io/debian buster main > /etc/apt/sources.list.d/vitastor.list) && \
RUN (echo deb http://vitastor.io/debian bookworm main > /etc/apt/sources.list.d/vitastor.list) && \
((echo 'Package: *'; echo 'Pin: origin "vitastor.io"'; echo 'Pin-Priority: 1000') > /etc/apt/preferences.d/vitastor.pref) && \
wget -q -O /etc/apt/trusted.gpg.d/vitastor.gpg https://vitastor.io/debian/pubkey.gpg && \
apt-get update && \
apt-get install -y vitastor-client && \
wget https://vitastor.io/archive/qemu/qemu-bookworm-8.1.2%2Bds-1%2Bvitastor1/qemu-utils_8.1.2%2Bds-1%2Bvitastor1_amd64.deb && \
wget https://vitastor.io/archive/qemu/qemu-bookworm-8.1.2%2Bds-1%2Bvitastor1/qemu-block-extra_8.1.2%2Bds-1%2Bvitastor1_amd64.deb && \
dpkg -x qemu-utils*.deb tmp1 && \
dpkg -x qemu-block-extra*.deb tmp1 && \
cp -a tmp1/usr/bin/qemu-storage-daemon /usr/bin/ && \
mkdir -p /usr/lib/x86_64-linux-gnu/qemu && \
cp -a tmp1/usr/lib/x86_64-linux-gnu/qemu/block-vitastor.so /usr/lib/x86_64-linux-gnu/qemu/ && \
rm -rf tmp1 *.deb && \
apt-get clean
ENTRYPOINT ["/bin/vitastor-csi"]

View File

@ -1,4 +1,4 @@
VERSION ?= v0.8.3
VERSION ?= v1.5.0
all: build push

View File

@ -2,6 +2,7 @@
apiVersion: v1
kind: ConfigMap
data:
# You can add multiple configuration files here to use a multi-cluster setup
vitastor.conf: |-
{"etcd_address":"http://192.168.7.2:2379","etcd_prefix":"/vitastor"}
metadata:

View File

@ -49,7 +49,7 @@ spec:
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
image: vitalif/vitastor-csi:v0.8.3
image: vitalif/vitastor-csi:v1.5.0
args:
- "--node=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"
@ -82,6 +82,8 @@ spec:
name: host-sys
- mountPath: /run/mount
name: host-mount
- mountPath: /run/vitastor-csi
name: run-vitastor-csi
- mountPath: /lib/modules
name: lib-modules
readOnly: true
@ -132,6 +134,9 @@ spec:
- name: host-mount
hostPath:
path: /run/mount
- name: run-vitastor-csi
hostPath:
path: /run/vitastor-csi
- name: lib-modules
hostPath:
path: /lib/modules

View File

@ -35,10 +35,13 @@ rules:
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list"]
verbs: ["get", "list", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots/status"]
verbs: ["get", "list", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["create", "get", "list", "watch", "update", "delete"]
verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotclasses"]
verbs: ["get", "list", "watch"]
@ -53,7 +56,7 @@ rules:
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents/status"]
verbs: ["update"]
verbs: ["update", "patch"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get"]

View File

@ -23,6 +23,11 @@ metadata:
name: csi-vitastor-provisioner
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 0
selector:
matchLabels:
app: csi-vitastor-provisioner
@ -46,7 +51,7 @@ spec:
priorityClassName: system-cluster-critical
containers:
- name: csi-provisioner
image: k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0
image: k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
args:
- "--csi-address=$(ADDRESS)"
- "--v=5"
@ -116,7 +121,7 @@ spec:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
image: vitalif/vitastor-csi:v0.8.3
image: vitalif/vitastor-csi:v1.5.0
args:
- "--node=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"

View File

@ -12,8 +12,6 @@ parameters:
etcdVolumePrefix: ""
poolId: "1"
# you can choose other configuration file if you have it in the config map
# different etcd URLs and prefixes should also be put in the config
#configPath: "/etc/vitastor/vitastor.conf"
# you can also specify etcdUrl here, maybe to connect to another Vitastor cluster
# multiple etcdUrls may be specified, delimited by comma
#etcdUrl: "http://192.168.7.2:2379"
#etcdPrefix: "/vitastor"
allowVolumeExpansion: true

View File

@ -0,0 +1,7 @@
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: vitastor-snapclass
driver: csi.vitastor.io
deletionPolicy: Delete
parameters:

View File

@ -0,0 +1,16 @@
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-vitastor-clone
spec:
storageClassName: vitastor
dataSource:
name: snap1
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi

View File

@ -0,0 +1,8 @@
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: snap1
spec:
volumeSnapshotClassName: vitastor-snapclass
source:
persistentVolumeClaimName: test-vitastor-pvc

View File

@ -4,26 +4,12 @@ go 1.15
require (
github.com/container-storage-interface/spec v1.4.0
github.com/coreos/bbolt v0.0.0-00010101000000-000000000000 // indirect
github.com/coreos/etcd v3.3.25+incompatible // indirect
github.com/coreos/go-semver v0.3.0 // indirect
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf // indirect
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/gorilla/websocket v1.4.2 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
github.com/jonboulle/clockwork v0.2.2 // indirect
github.com/kubernetes-csi/csi-lib-utils v0.9.1
github.com/soheilhy/cmux v0.1.5 // indirect
github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 // indirect
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
go.etcd.io/bbolt v0.0.0-00010101000000-000000000000 // indirect
go.etcd.io/etcd v3.3.25+incompatible
golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/grpc v1.33.1
google.golang.org/protobuf v1.24.0
k8s.io/klog v1.0.0
k8s.io/utils v0.0.0-20210305010621-2afb4311ab10
)

View File

@ -31,14 +31,11 @@ github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuy
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
@ -46,25 +43,12 @@ github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMn
github.com/container-storage-interface/spec v1.2.0/go.mod h1:6URME8mwIBbpVyZV93Ce5St17xBiQJQY67NDsuohiy4=
github.com/container-storage-interface/spec v1.4.0 h1:ozAshSKxpJnYUfmkpZCTYyF/4MYeYlhdXbAvPvfGmkg=
github.com/container-storage-interface/spec v1.4.0/go.mod h1:6URME8mwIBbpVyZV93Ce5St17xBiQJQY67NDsuohiy4=
github.com/coreos/bbolt v1.3.5 h1:XFv7xaq7701j8ZSEzR28VohFYSlyakMyqNMU5FQH6Ac=
github.com/coreos/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
github.com/coreos/etcd v3.3.25+incompatible h1:0GQEw6h3YnuOVdtwygkIfJ+Omx0tZ8/QkVyXI4LkbeY=
github.com/coreos/etcd v3.3.25+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-semver v0.3.0 h1:wkHLiw0WNATZnSG7epLsujiMCgPAc9xhjJ4tgnAxmfM=
github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf h1:iW4rZ826su+pqaw19uhpSCzhj44qo35pNgKFGqzDKkU=
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbpBpLoyyu8B6e44T7hJy6potg=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
@ -73,7 +57,6 @@ github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLi
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
@ -88,14 +71,10 @@ github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nA
github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls=
github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
@ -113,7 +92,6 @@ github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QD
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
@ -127,38 +105,24 @@ github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OI
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg=
github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 h1:+9834+KizmvFV7pXQGSXQTsaWhq2GjuNUt0aUU0YBYw=
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0/go.mod h1:z0ButlSOZa5vEBq9m2m2hlwIgKw+rp3sdCBRoJY+30Y=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.16.0 h1:gmcG1KaJ57LophUzW0Hy8NmPhnMZb4M0+kPpLofRdBo=
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/jonboulle/clockwork v0.2.2 h1:UOGuzwb1PwsrDAObMuhUnj0p5ULPj8V/xJ7Kx9qUBdQ=
github.com/jonboulle/clockwork v0.2.2/go.mod h1:Pkfl5aHPm1nk2H9h0bjmnJD/BcgbGXUBGnn1kMkgxc8=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
@ -171,14 +135,11 @@ github.com/kubernetes-csi/csi-lib-utils v0.9.1 h1:sGq6ifVujfMSkfTsMZip44Ttv8SDXv
github.com/kubernetes-csi/csi-lib-utils v0.9.1/go.mod h1:8E2jVUX9j3QgspwHXa6LwyN7IHQDjW9jX3kwoWnSC+M=
github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 h1:I0XW9+e1XWDxdcEniV4rQAIOPUGDq67JSCiRCgGCZLI=
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
@ -188,38 +149,28 @@ github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
github.com/prometheus/client_golang v1.7.1 h1:NTGy1Ja9pByO+xAeH/qiWnLrKtr3hJPNjaVUwnjpdpA=
github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.2.0 h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M=
github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.10.0 h1:RyRA7RzGXQZiW+tGMr7sxa85G1z0yOpM1qq5c8lNawc=
github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.1.3 h1:F0+tqvhOksq22sc6iCHF5WGlWjdwj92p0udFh1VFBS8=
github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I=
github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
github.com/soheilhy/cmux v0.1.5 h1:jjzc5WVemNEDTLwv9tlmemhC73tI08BNOIGwBOo10Js=
github.com/soheilhy/cmux v0.1.5/go.mod h1:T7TcVDs9LWfQgPlPsdngu6I6QIoyIFZDDC6sNE1GqG0=
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
@ -231,24 +182,11 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 h1:uruHq4dN7GR16kFc5fp3d1RIYzJW5onx8Ybykw2YQFA=
github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 h1:eY9dn8+vbi4tKz5Qo6v2eYzo7kUS51QINcR5jNpbZS8=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0=
go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
go.etcd.io/etcd v3.3.25+incompatible h1:V1RzkZJj9LqsJRy+TUBgpWSbZXITLB819lstuTFoZOY=
go.etcd.io/etcd v3.3.25+incompatible/go.mod h1:yaeTdrJi5lOmYerz05bd8+V7KubZs8YSFZfzsF9A6aI=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.uber.org/atomic v1.4.0 h1:cxzIVoETapQEqDhQu3QfnvXAV4AlzcvUCxkVUFw3+EU=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
@ -256,7 +194,6 @@ golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
@ -276,8 +213,6 @@ golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCc
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@ -291,26 +226,20 @@ golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb h1:eBmm0M9fYhWpKZLjQUUKka/LtIxf46G4fxeEz5KJr9U=
golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@ -326,11 +255,9 @@ golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4 h1:5/PjkGUjvEU5Gl6BxmvKRPpqo2uNMv4rcHBMwzk/st8=
golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f h1:+Nyd8tzPX9R7BWHguqsrbFdRx3WQ/1ib8I44HXV5yTA=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -341,7 +268,6 @@ golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@ -360,14 +286,10 @@ golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgw
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@ -388,8 +310,6 @@ google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/grpc v1.25.1 h1:wdKvqQk7IttEw92GoRyKG2IDrUIpgpj6H6m81yfeMW0=
@ -415,7 +335,6 @@ gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
@ -444,5 +363,4 @@ k8s.io/utils v0.0.0-20210305010621-2afb4311ab10/go.mod h1:jPW/WVKK9YHAvNhRxK0md/
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
sigs.k8s.io/structured-merge-diff/v4 v4.0.1/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw=
sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o=
sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q=
sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc=

View File

@ -5,7 +5,7 @@ package vitastor
const (
vitastorCSIDriverName = "csi.vitastor.io"
vitastorCSIDriverVersion = "0.8.3"
vitastorCSIDriverVersion = "1.5.0"
)
// Config struct fills the parameters of request or user input

View File

@ -6,11 +6,11 @@ package vitastor
import (
"context"
"encoding/json"
"fmt"
"strings"
"bytes"
"strconv"
"time"
"fmt"
"os"
"os/exec"
"io/ioutil"
@ -20,8 +20,7 @@ import (
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"go.etcd.io/etcd/clientv3"
"google.golang.org/protobuf/types/known/timestamppb"
"github.com/container-storage-interface/spec/lib/go/csi"
)
@ -47,6 +46,7 @@ type InodeConfig struct
ParentPool uint64 `json:"parent_pool,omitempty"`
ParentId uint64 `json:"parent_id,omitempty"`
Readonly bool `json:"readonly,omitempty"`
CreateTs uint64 `json:"create_ts,omitempty"`
}
type ControllerServer struct
@ -62,7 +62,7 @@ func NewControllerServer(driver *Driver) *ControllerServer
}
}
func GetConnectionParams(params map[string]string) (map[string]string, []string, string)
func GetConnectionParams(params map[string]string) (map[string]string, error)
{
ctxVars := make(map[string]string)
configPath := params["configPath"]
@ -75,43 +75,69 @@ func GetConnectionParams(params map[string]string) (map[string]string, []string,
ctxVars["configPath"] = configPath
}
config := make(map[string]interface{})
if configFD, err := os.Open(configPath); err == nil
configFD, err := os.Open(configPath)
if (err != nil)
{
defer configFD.Close()
data, _ := ioutil.ReadAll(configFD)
json.Unmarshal(data, &config)
return nil, err
}
// Try to load prefix & etcd URL from the config
defer configFD.Close()
data, _ := ioutil.ReadAll(configFD)
json.Unmarshal(data, &config)
// Check etcd URL in the config, but do not use the explicit etcdUrl
// parameter for CLI calls, otherwise users won't be able to later
// change them - storage class parameters are saved in volume IDs
var etcdUrl []string
if (params["etcdUrl"] != "")
switch config["etcd_address"].(type)
{
ctxVars["etcdUrl"] = params["etcdUrl"]
etcdUrl = strings.Split(params["etcdUrl"], ",")
case string:
url := strings.TrimSpace(config["etcd_address"].(string))
if (url != "")
{
etcdUrl = strings.Split(url, ",")
}
case []string:
etcdUrl = config["etcd_address"].([]string)
case []interface{}:
for _, url := range config["etcd_address"].([]interface{})
{
s, ok := url.(string)
if (ok)
{
etcdUrl = append(etcdUrl, s)
}
}
}
if (len(etcdUrl) == 0)
{
switch config["etcd_address"].(type)
{
case string:
etcdUrl = strings.Split(config["etcd_address"].(string), ",")
case []string:
etcdUrl = config["etcd_address"].([]string)
}
return nil, status.Error(codes.InvalidArgument, "etcd_address is missing in "+configPath)
}
etcdPrefix := params["etcdPrefix"]
if (etcdPrefix == "")
return ctxVars, nil
}
func system(program string, args ...string) ([]byte, []byte, error)
{
klog.Infof("Running "+program+" "+strings.Join(args, " "))
c := exec.Command(program, args...)
var stdout, stderr bytes.Buffer
c.Stdout, c.Stderr = &stdout, &stderr
err := c.Run()
if (err != nil)
{
etcdPrefix, _ = config["etcd_prefix"].(string)
if (etcdPrefix == "")
{
etcdPrefix = "/vitastor"
}
stdoutStr, stderrStr := string(stdout.Bytes()), string(stderr.Bytes())
klog.Errorf(program+" "+strings.Join(args, " ")+" failed: %s, status %s\n", stdoutStr+stderrStr, err)
return nil, nil, status.Error(codes.Internal, stdoutStr+stderrStr+" (status "+err.Error()+")")
}
else
return stdout.Bytes(), stderr.Bytes(), nil
}
func invokeCLI(ctxVars map[string]string, args []string) ([]byte, error)
{
if (ctxVars["configPath"] != "")
{
ctxVars["etcdPrefix"] = etcdPrefix
args = append(args, "--config_path", ctxVars["configPath"])
}
return ctxVars, etcdUrl, etcdPrefix
stdout, _, err := system("/usr/bin/vitastor-cli", args...)
return stdout, err
}
// Create the volume
@ -146,128 +172,57 @@ func (cs *ControllerServer) CreateVolume(ctx context.Context, req *csi.CreateVol
volSize = ((capRange.GetRequiredBytes() + MB - 1) / MB) * MB
}
// FIXME: The following should PROBABLY be implemented externally in a management tool
ctxVars, etcdUrl, etcdPrefix := GetConnectionParams(req.Parameters)
if (len(etcdUrl) == 0)
{
return nil, status.Error(codes.InvalidArgument, "no etcdUrl in storage class configuration and no etcd_address in vitastor.conf")
}
// Connect to etcd
cli, err := clientv3.New(clientv3.Config{
DialTimeout: ETCD_TIMEOUT,
Endpoints: etcdUrl,
})
ctxVars, err := GetConnectionParams(req.Parameters)
if (err != nil)
{
return nil, status.Error(codes.Internal, "failed to connect to etcd at "+strings.Join(etcdUrl, ",")+": "+err.Error())
return nil, err
}
defer cli.Close()
var imageId uint64 = 0
for
args := []string{ "create", volName, "-s", fmt.Sprintf("%v", volSize), "--pool", fmt.Sprintf("%v", poolId) }
// Support creation from snapshot
var src *csi.VolumeContentSource
if (req.VolumeContentSource.GetSnapshot() != nil)
{
// Check if the image exists
ctx, cancel := context.WithTimeout(context.Background(), ETCD_TIMEOUT)
resp, err := cli.Get(ctx, etcdPrefix+"/index/image/"+volName)
cancel()
if (err != nil)
snapId := req.VolumeContentSource.GetSnapshot().GetSnapshotId()
if (snapId != "")
{
return nil, status.Error(codes.Internal, "failed to read key from etcd: "+err.Error())
snapVars := make(map[string]string)
err := json.Unmarshal([]byte(snapId), &snapVars)
if (err != nil)
{
return nil, status.Error(codes.Internal, "volume ID not in JSON format")
}
args = append(args, "--parent", snapVars["name"]+"@"+snapVars["snapshot"])
src = &csi.VolumeContentSource{
Type: &csi.VolumeContentSource_Snapshot{
Snapshot: &csi.VolumeContentSource_SnapshotSource{
SnapshotId: snapId,
},
},
}
}
if (len(resp.Kvs) > 0)
}
// Create image using vitastor-cli
_, err = invokeCLI(ctxVars, args)
if (err != nil)
{
if (strings.Index(err.Error(), "already exists") > 0)
{
kv := resp.Kvs[0]
var v InodeIndex
err := json.Unmarshal(kv.Value, &v)
inodeCfg, err := invokeList(ctxVars, volName, true)
if (err != nil)
{
return nil, status.Error(codes.Internal, "invalid /index/image/"+volName+" key in etcd: "+err.Error())
return nil, err
}
poolId = v.PoolId
imageId = v.Id
inodeCfgKey := fmt.Sprintf("/config/inode/%d/%d", poolId, imageId)
ctx, cancel := context.WithTimeout(context.Background(), ETCD_TIMEOUT)
resp, err := cli.Get(ctx, etcdPrefix+inodeCfgKey)
cancel()
if (err != nil)
{
return nil, status.Error(codes.Internal, "failed to read key from etcd: "+err.Error())
}
if (len(resp.Kvs) == 0)
{
return nil, status.Error(codes.Internal, "missing "+inodeCfgKey+" key in etcd")
}
var inodeCfg InodeConfig
err = json.Unmarshal(resp.Kvs[0].Value, &inodeCfg)
if (err != nil)
{
return nil, status.Error(codes.Internal, "invalid "+inodeCfgKey+" key in etcd: "+err.Error())
}
if (inodeCfg.Size < uint64(volSize))
if (inodeCfg[0].Size < uint64(volSize))
{
return nil, status.Error(codes.Internal, "image "+volName+" is already created, but size is less than expected")
}
}
else
{
// Find a free ID
// Create image metadata in a transaction verifying that the image doesn't exist yet AND ID is still free
maxIdKey := fmt.Sprintf("%s/index/maxid/%d", etcdPrefix, poolId)
ctx, cancel := context.WithTimeout(context.Background(), ETCD_TIMEOUT)
resp, err := cli.Get(ctx, maxIdKey)
cancel()
if (err != nil)
{
return nil, status.Error(codes.Internal, "failed to read key from etcd: "+err.Error())
}
var modRev int64
var nextId uint64
if (len(resp.Kvs) > 0)
{
var err error
nextId, err = strconv.ParseUint(string(resp.Kvs[0].Value), 10, 64)
if (err != nil)
{
return nil, status.Error(codes.Internal, maxIdKey+" contains invalid ID")
}
modRev = resp.Kvs[0].ModRevision
nextId++
}
else
{
nextId = 1
}
inodeIdxJson, _ := json.Marshal(InodeIndex{
Id: nextId,
PoolId: poolId,
})
inodeCfgJson, _ := json.Marshal(InodeConfig{
Name: volName,
Size: uint64(volSize),
})
ctx, cancel = context.WithTimeout(context.Background(), ETCD_TIMEOUT)
txnResp, err := cli.Txn(ctx).If(
clientv3.Compare(clientv3.ModRevision(fmt.Sprintf("%s/index/maxid/%d", etcdPrefix, poolId)), "=", modRev),
clientv3.Compare(clientv3.CreateRevision(fmt.Sprintf("%s/index/image/%s", etcdPrefix, volName)), "=", 0),
clientv3.Compare(clientv3.CreateRevision(fmt.Sprintf("%s/config/inode/%d/%d", etcdPrefix, poolId, nextId)), "=", 0),
).Then(
clientv3.OpPut(fmt.Sprintf("%s/index/maxid/%d", etcdPrefix, poolId), fmt.Sprintf("%d", nextId)),
clientv3.OpPut(fmt.Sprintf("%s/index/image/%s", etcdPrefix, volName), string(inodeIdxJson)),
clientv3.OpPut(fmt.Sprintf("%s/config/inode/%d/%d", etcdPrefix, poolId, nextId), string(inodeCfgJson)),
).Commit()
cancel()
if (err != nil)
{
return nil, status.Error(codes.Internal, "failed to commit transaction in etcd: "+err.Error())
}
if (txnResp.Succeeded)
{
imageId = nextId
break
}
// Start over if the transaction fails
return nil, err
}
}
@ -278,6 +233,7 @@ func (cs *ControllerServer) CreateVolume(ctx context.Context, req *csi.CreateVol
// Ugly, but VolumeContext isn't passed to DeleteVolume :-(
VolumeId: string(volumeIdJson),
CapacityBytes: volSize,
ContentSource: src,
},
}, nil
}
@ -291,105 +247,24 @@ func (cs *ControllerServer) DeleteVolume(ctx context.Context, req *csi.DeleteVol
return nil, status.Error(codes.InvalidArgument, "request cannot be empty")
}
ctxVars := make(map[string]string)
err := json.Unmarshal([]byte(req.VolumeId), &ctxVars)
volVars := make(map[string]string)
err := json.Unmarshal([]byte(req.VolumeId), &volVars)
if (err != nil)
{
return nil, status.Error(codes.Internal, "volume ID not in JSON format")
}
volName := ctxVars["name"]
volName := volVars["name"]
_, etcdUrl, etcdPrefix := GetConnectionParams(ctxVars)
if (len(etcdUrl) == 0)
ctxVars, err := GetConnectionParams(volVars)
if (err != nil)
{
return nil, status.Error(codes.InvalidArgument, "no etcdUrl in storage class configuration and no etcd_address in vitastor.conf")
return nil, err
}
cli, err := clientv3.New(clientv3.Config{
DialTimeout: ETCD_TIMEOUT,
Endpoints: etcdUrl,
})
_, err = invokeCLI(ctxVars, []string{ "rm", volName })
if (err != nil)
{
return nil, status.Error(codes.Internal, "failed to connect to etcd at "+strings.Join(etcdUrl, ",")+": "+err.Error())
}
defer cli.Close()
// Find inode by name
ctx, cancel := context.WithTimeout(context.Background(), ETCD_TIMEOUT)
resp, err := cli.Get(ctx, etcdPrefix+"/index/image/"+volName)
cancel()
if (err != nil)
{
return nil, status.Error(codes.Internal, "failed to read key from etcd: "+err.Error())
}
if (len(resp.Kvs) == 0)
{
return nil, status.Error(codes.NotFound, "volume "+volName+" does not exist")
}
var idx InodeIndex
err = json.Unmarshal(resp.Kvs[0].Value, &idx)
if (err != nil)
{
return nil, status.Error(codes.Internal, "invalid /index/image/"+volName+" key in etcd: "+err.Error())
}
// Get inode config
inodeCfgKey := fmt.Sprintf("%s/config/inode/%d/%d", etcdPrefix, idx.PoolId, idx.Id)
ctx, cancel = context.WithTimeout(context.Background(), ETCD_TIMEOUT)
resp, err = cli.Get(ctx, inodeCfgKey)
cancel()
if (err != nil)
{
return nil, status.Error(codes.Internal, "failed to read key from etcd: "+err.Error())
}
if (len(resp.Kvs) == 0)
{
return nil, status.Error(codes.NotFound, "volume "+volName+" does not exist")
}
var inodeCfg InodeConfig
err = json.Unmarshal(resp.Kvs[0].Value, &inodeCfg)
if (err != nil)
{
return nil, status.Error(codes.Internal, "invalid "+inodeCfgKey+" key in etcd: "+err.Error())
}
// Delete inode data by invoking vitastor-cli
args := []string{
"rm-data", "--etcd_address", strings.Join(etcdUrl, ","),
"--pool", fmt.Sprintf("%d", idx.PoolId),
"--inode", fmt.Sprintf("%d", idx.Id),
}
if (ctxVars["configPath"] != "")
{
args = append(args, "--config_path", ctxVars["configPath"])
}
c := exec.Command("/usr/bin/vitastor-cli", args...)
var stderr bytes.Buffer
c.Stdout = nil
c.Stderr = &stderr
err = c.Run()
stderrStr := string(stderr.Bytes())
if (err != nil)
{
klog.Errorf("vitastor-cli rm-data failed: %s, status %s\n", stderrStr, err)
return nil, status.Error(codes.Internal, stderrStr+" (status "+err.Error()+")")
}
// Delete inode config in etcd
ctx, cancel = context.WithTimeout(context.Background(), ETCD_TIMEOUT)
txnResp, err := cli.Txn(ctx).Then(
clientv3.OpDelete(fmt.Sprintf("%s/index/image/%s", etcdPrefix, volName)),
clientv3.OpDelete(fmt.Sprintf("%s/config/inode/%d/%d", etcdPrefix, idx.PoolId, idx.Id)),
).Commit()
cancel()
if (err != nil)
{
return nil, status.Error(codes.Internal, "failed to delete keys in etcd: "+err.Error())
}
if (!txnResp.Succeeded)
{
return nil, status.Error(codes.Internal, "failed to delete keys in etcd: transaction failed")
return nil, err
}
return &csi.DeleteVolumeResponse{}, nil
@ -490,6 +365,8 @@ func (cs *ControllerServer) ControllerGetCapabilities(ctx context.Context, req *
csi.ControllerServiceCapability_RPC_LIST_VOLUMES,
csi.ControllerServiceCapability_RPC_EXPAND_VOLUME,
csi.ControllerServiceCapability_RPC_CREATE_DELETE_SNAPSHOT,
csi.ControllerServiceCapability_RPC_LIST_SNAPSHOTS,
// TODO: csi.ControllerServiceCapability_RPC_CLONE_VOLUME,
} {
controllerServerCapabilities = append(controllerServerCapabilities, functionControllerServerCapabilities(capability))
}
@ -499,28 +376,226 @@ func (cs *ControllerServer) ControllerGetCapabilities(ctx context.Context, req *
}, nil
}
func invokeList(ctxVars map[string]string, pattern string, expectExist bool) ([]InodeConfig, error)
{
stat, err := invokeCLI(ctxVars, []string{ "ls", "--json", pattern })
if (err != nil)
{
return nil, err
}
var inodeCfg []InodeConfig
err = json.Unmarshal(stat, &inodeCfg)
if (err != nil)
{
return nil, status.Error(codes.Internal, "Invalid JSON in vitastor-cli ls: "+err.Error())
}
if (expectExist && len(inodeCfg) == 0)
{
return nil, status.Error(codes.Internal, "Can't find expected image "+pattern+" via vitastor-cli ls")
}
return inodeCfg, nil
}
// CreateSnapshot create snapshot of an existing PV
func (cs *ControllerServer) CreateSnapshot(ctx context.Context, req *csi.CreateSnapshotRequest) (*csi.CreateSnapshotResponse, error)
{
return nil, status.Error(codes.Unimplemented, "")
klog.Infof("received controller create snapshot request %+v", protosanitizer.StripSecrets(req))
if (req == nil)
{
return nil, status.Errorf(codes.InvalidArgument, "request cannot be empty")
}
if (req.SourceVolumeId == "" || req.Name == "")
{
return nil, status.Error(codes.InvalidArgument, "source volume ID and snapshot name are required fields")
}
// snapshot name
snapName := req.Name
// req.VolumeId is an ugly json string in our case :)
ctxVars := make(map[string]string)
err := json.Unmarshal([]byte(req.SourceVolumeId), &ctxVars)
if (err != nil)
{
return nil, status.Error(codes.Internal, "volume ID not in JSON format")
}
volName := ctxVars["name"]
// Create image using vitastor-cli
_, err = invokeCLI(ctxVars, []string{ "create", "--snapshot", snapName, volName })
if (err != nil && strings.Index(err.Error(), "already exists") <= 0)
{
return nil, err
}
// Check created snapshot
inodeCfg, err := invokeList(ctxVars, volName+"@"+snapName, true)
if (err != nil)
{
return nil, err
}
// Use ugly JSON snapshot ID again, DeleteSnapshot doesn't have context :-(
ctxVars["snapshot"] = snapName
snapIdJson, _ := json.Marshal(ctxVars)
return &csi.CreateSnapshotResponse{
Snapshot: &csi.Snapshot{
SizeBytes: int64(inodeCfg[0].Size),
SnapshotId: string(snapIdJson),
SourceVolumeId: req.SourceVolumeId,
CreationTime: &timestamppb.Timestamp{ Seconds: int64(inodeCfg[0].CreateTs) },
ReadyToUse: true,
},
}, nil
}
// DeleteSnapshot delete provided snapshot of a PV
func (cs *ControllerServer) DeleteSnapshot(ctx context.Context, req *csi.DeleteSnapshotRequest) (*csi.DeleteSnapshotResponse, error)
{
return nil, status.Error(codes.Unimplemented, "")
klog.Infof("received controller delete snapshot request %+v", protosanitizer.StripSecrets(req))
if (req == nil)
{
return nil, status.Errorf(codes.InvalidArgument, "request cannot be empty")
}
if (req.SnapshotId == "")
{
return nil, status.Error(codes.InvalidArgument, "snapshot ID is a required field")
}
volVars := make(map[string]string)
err := json.Unmarshal([]byte(req.SnapshotId), &volVars)
if (err != nil)
{
return nil, status.Error(codes.Internal, "snapshot ID not in JSON format")
}
volName := volVars["name"]
snapName := volVars["snapshot"]
ctxVars, err := GetConnectionParams(volVars)
if (err != nil)
{
return nil, err
}
_, err = invokeCLI(ctxVars, []string{ "rm", volName+"@"+snapName })
if (err != nil)
{
return nil, err
}
return &csi.DeleteSnapshotResponse{}, nil
}
// ListSnapshots list the snapshots of a PV
func (cs *ControllerServer) ListSnapshots(ctx context.Context, req *csi.ListSnapshotsRequest) (*csi.ListSnapshotsResponse, error)
{
return nil, status.Error(codes.Unimplemented, "")
klog.Infof("received controller list snapshots request %+v", protosanitizer.StripSecrets(req))
if (req == nil)
{
return nil, status.Error(codes.InvalidArgument, "request cannot be empty")
}
volVars := make(map[string]string)
err := json.Unmarshal([]byte(req.SourceVolumeId), &volVars)
if (err != nil)
{
return nil, status.Error(codes.Internal, "volume ID not in JSON format")
}
volName := volVars["name"]
ctxVars, err := GetConnectionParams(volVars)
if (err != nil)
{
return nil, err
}
inodeCfg, err := invokeList(ctxVars, volName+"@*", false)
if (err != nil)
{
return nil, err
}
resp := &csi.ListSnapshotsResponse{}
for _, ino := range inodeCfg
{
snapName := ino.Name[len(volName)+1:]
if (len(req.StartingToken) > 0 && snapName < req.StartingToken)
{
}
else if (req.MaxEntries == 0 || len(resp.Entries) < int(req.MaxEntries))
{
volVars["snapshot"] = snapName
snapIdJson, _ := json.Marshal(volVars)
resp.Entries = append(resp.Entries, &csi.ListSnapshotsResponse_Entry{
Snapshot: &csi.Snapshot{
SizeBytes: int64(ino.Size),
SnapshotId: string(snapIdJson),
SourceVolumeId: req.SourceVolumeId,
CreationTime: &timestamppb.Timestamp{ Seconds: int64(ino.CreateTs) },
ReadyToUse: true,
},
})
}
else
{
resp.NextToken = snapName
break
}
}
return resp, nil
}
// ControllerExpandVolume resizes a volume
// ControllerExpandVolume increases the size of a volume
func (cs *ControllerServer) ControllerExpandVolume(ctx context.Context, req *csi.ControllerExpandVolumeRequest) (*csi.ControllerExpandVolumeResponse, error)
{
return nil, status.Error(codes.Unimplemented, "")
klog.Infof("received controller expand volume request %+v", protosanitizer.StripSecrets(req))
if (req == nil)
{
return nil, status.Error(codes.InvalidArgument, "request cannot be empty")
}
if (req.VolumeId == "" || req.CapacityRange == nil || req.CapacityRange.RequiredBytes == 0)
{
return nil, status.Error(codes.InvalidArgument, "VolumeId, CapacityRange and RequiredBytes are required fields")
}
volVars := make(map[string]string)
err := json.Unmarshal([]byte(req.VolumeId), &volVars)
if (err != nil)
{
return nil, status.Error(codes.Internal, "volume ID not in JSON format")
}
volName := volVars["name"]
ctxVars, err := GetConnectionParams(volVars)
if (err != nil)
{
return nil, err
}
inodeCfg, err := invokeList(ctxVars, volName, true)
if (err != nil)
{
return nil, err
}
if (req.CapacityRange.RequiredBytes > 0 && inodeCfg[0].Size < uint64(req.CapacityRange.RequiredBytes))
{
sz := ((req.CapacityRange.RequiredBytes+4095)/4096)*4096
_, err := invokeCLI(ctxVars, []string{ "modify", "--inc_size", "1", "--resize", fmt.Sprintf("%d", sz), volName })
if (err != nil)
{
return nil, err
}
inodeCfg, err = invokeList(ctxVars, volName, true)
if (err != nil)
{
return nil, err
}
}
return &csi.ControllerExpandVolumeResponse{
CapacityBytes: int64(inodeCfg[0].Size),
NodeExpansionRequired: false,
}, nil
}
// ControllerGetVolume get volume info

View File

@ -49,6 +49,13 @@ func (is *IdentityServer) GetPluginCapabilities(ctx context.Context, req *csi.Ge
},
},
},
{
Type: &csi.PluginCapability_VolumeExpansion_{
VolumeExpansion: &csi.PluginCapability_VolumeExpansion{
Type: csi.PluginCapability_VolumeExpansion_OFFLINE,
},
},
},
},
}, nil
}

View File

@ -5,11 +5,16 @@ package vitastor
import (
"context"
"errors"
"encoding/json"
"fmt"
"os"
"os/exec"
"encoding/json"
"path/filepath"
"strconv"
"strings"
"bytes"
"syscall"
"time"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
@ -25,16 +30,102 @@ import (
type NodeServer struct
{
*Driver
useVduse bool
stateDir string
mounter mount.Interface
restartInterval time.Duration
}
type DeviceState struct
{
ConfigPath string `json:"configPath"`
VdpaId string `json:"vdpaId"`
Image string `json:"image"`
Blockdev string `json:"blockdev"`
Readonly bool `json:"readonly"`
PidFile string `json:"pidFile"`
}
// NewNodeServer create new instance node
func NewNodeServer(driver *Driver) *NodeServer
{
return &NodeServer{
stateDir := os.Getenv("STATE_DIR")
if (stateDir == "")
{
stateDir = "/run/vitastor-csi"
}
if (stateDir[len(stateDir)-1] != '/')
{
stateDir += "/"
}
ns := &NodeServer{
Driver: driver,
useVduse: checkVduseSupport(),
stateDir: stateDir,
mounter: mount.New(""),
}
if (ns.useVduse)
{
ns.restoreVduseDaemons()
dur, err := time.ParseDuration(os.Getenv("RESTART_INTERVAL"))
if (err != nil)
{
dur = 10 * time.Second
}
ns.restartInterval = dur
if (ns.restartInterval != time.Duration(0))
{
go ns.restarter()
}
}
return ns
}
func checkVduseSupport() bool
{
// Check VDUSE support (vdpa, vduse, virtio-vdpa kernel modules)
vduse := true
for _, mod := range []string{"vdpa", "vduse", "virtio-vdpa"}
{
_, err := os.Stat("/sys/module/"+mod)
if (err != nil)
{
if (!errors.Is(err, os.ErrNotExist))
{
klog.Errorf("failed to check /sys/module/%s: %v", mod, err)
}
c := exec.Command("/sbin/modprobe", mod)
c.Stdout = os.Stderr
c.Stderr = os.Stderr
err := c.Run()
if (err != nil)
{
klog.Errorf("/sbin/modprobe %s failed: %v", mod, err)
vduse = false
break
}
}
}
// Check that vdpa tool functions
if (vduse)
{
c := exec.Command("/sbin/vdpa", "-j", "dev")
c.Stderr = os.Stderr
err := c.Run()
if (err != nil)
{
klog.Errorf("/sbin/vdpa -j dev failed: %v", err)
vduse = false
}
}
if (!vduse)
{
klog.Errorf(
"Your host apparently has no VDUSE support. VDUSE support disabled, NBD will be used to map devices."+
" For VDUSE you need at least Linux 5.15 and the following kernel modules: vdpa, virtio-vdpa, vduse.",
)
}
return vduse
}
// NodeStageVolume mounts the volume to a staging path on the node.
@ -61,6 +152,318 @@ func Contains(list []string, s string) bool
return false
}
func (ns *NodeServer) mapNbd(volName string, ctxVars map[string]string, readonly bool) (string, error)
{
// Map NBD device
// FIXME: Check if already mapped
args := []string{
"map", "--image", volName,
}
if (ctxVars["configPath"] != "")
{
args = append(args, "--config_path", ctxVars["configPath"])
}
if (readonly)
{
args = append(args, "--readonly", "1")
}
stdout, stderr, err := system("/usr/bin/vitastor-nbd", args...)
dev := strings.TrimSpace(string(stdout))
if (dev == "")
{
return "", fmt.Errorf("vitastor-nbd did not return the name of NBD device. output: %s", stderr)
}
return dev, err
}
func (ns *NodeServer) unmapNbd(devicePath string)
{
// unmap NBD device
unmapOut, unmapErr := exec.Command("/usr/bin/vitastor-nbd", "unmap", devicePath).CombinedOutput()
if (unmapErr != nil)
{
klog.Errorf("failed to unmap NBD device %s: %s, error: %v", devicePath, unmapOut, unmapErr)
}
}
func findByPidFile(pidFile string) (*os.Process, error)
{
pidBuf, err := os.ReadFile(pidFile)
if (err != nil)
{
return nil, err
}
pid, err := strconv.ParseInt(strings.TrimSpace(string(pidBuf)), 0, 64)
if (err != nil)
{
return nil, err
}
proc, err := os.FindProcess(int(pid))
if (err != nil)
{
return nil, err
}
return proc, nil
}
func killByPidFile(pidFile string) error
{
klog.Infof("killing process with PID from file %s", pidFile)
proc, err := findByPidFile(pidFile)
if (err != nil)
{
return err
}
return proc.Signal(syscall.SIGTERM)
}
func startStorageDaemon(vdpaId, volName, pidFile, configPath string, readonly bool) error
{
// Start qemu-storage-daemon
blockSpec := map[string]interface{}{
"node-name": "disk1",
"driver": "vitastor",
"image": volName,
"cache": map[string]bool{
"direct": true,
"no-flush": false,
},
"discard": "unmap",
}
if (configPath != "")
{
blockSpec["config-path"] = configPath
}
blockSpecJson, _ := json.Marshal(blockSpec)
writable := "true"
if (readonly)
{
writable = "false"
}
_, _, err := system(
"/usr/bin/qemu-storage-daemon", "--daemonize", "--pidfile", pidFile, "--blockdev", string(blockSpecJson),
"--export", "vduse-blk,id="+vdpaId+",node-name=disk1,name="+vdpaId+",num-queues=16,queue-size=128,writable="+writable,
)
return err
}
func (ns *NodeServer) mapVduse(volName string, ctxVars map[string]string, readonly bool) (string, string, error)
{
// Generate state file
stateFd, err := os.CreateTemp(ns.stateDir, "vitastor-vduse-*.json")
if (err != nil)
{
return "", "", err
}
stateFile := stateFd.Name()
stateFd.Close()
vdpaId := filepath.Base(stateFile)
vdpaId = vdpaId[0:len(vdpaId)-5] // remove ".json"
pidFile := ns.stateDir + vdpaId + ".pid"
// Map VDUSE device via qemu-storage-daemon
err = startStorageDaemon(vdpaId, volName, pidFile, ctxVars["configPath"], readonly)
if (err == nil)
{
// Add device to VDPA bus
_, _, err = system("/sbin/vdpa", "-j", "dev", "add", "name", vdpaId, "mgmtdev", "vduse")
if (err == nil)
{
// Find block device name
var matches []string
matches, err = filepath.Glob("/sys/bus/vdpa/devices/"+vdpaId+"/virtio*/block/*")
if (err == nil && len(matches) == 0)
{
err = errors.New("/sys/bus/vdpa/devices/"+vdpaId+"/virtio*/block/* is not found")
}
if (err == nil)
{
blockdev := "/dev/"+filepath.Base(matches[0])
_, err = os.Stat(blockdev)
if (err == nil)
{
// Generate state file
stateJSON, _ := json.Marshal(&DeviceState{
ConfigPath: ctxVars["configPath"],
VdpaId: vdpaId,
Image: volName,
Blockdev: blockdev,
Readonly: readonly,
PidFile: pidFile,
})
err = os.WriteFile(stateFile, stateJSON, 0600)
if (err == nil)
{
return blockdev, vdpaId, nil
}
}
}
}
killErr := killByPidFile(pidFile)
if (killErr != nil)
{
klog.Errorf("Failed to kill started qemu-storage-daemon: %v", killErr)
}
os.Remove(stateFile)
os.Remove(pidFile)
}
return "", "", err
}
func (ns *NodeServer) unmapVduse(devicePath string)
{
if (len(devicePath) < 6 || devicePath[0:6] != "/dev/v")
{
klog.Errorf("%s does not start with /dev/v", devicePath)
return
}
vduseDev, err := os.Readlink("/sys/block/"+devicePath[5:])
if (err != nil)
{
klog.Errorf("%s is not a symbolic link to VDUSE device (../devices/virtual/vduse/xxx): %v", devicePath, err)
return
}
vdpaId := ""
p := strings.Index(vduseDev, "/vduse/")
if (p >= 0)
{
vduseDev = vduseDev[p+7:]
p = strings.Index(vduseDev, "/")
if (p >= 0)
{
vdpaId = vduseDev[0:p]
}
}
if (vdpaId == "")
{
klog.Errorf("%s is not a symbolic link to VDUSE device (../devices/virtual/vduse/xxx), but is %v", devicePath, vduseDev)
return
}
ns.unmapVduseById(vdpaId)
}
func (ns *NodeServer) unmapVduseById(vdpaId string)
{
_, err := os.Stat("/sys/bus/vdpa/devices/"+vdpaId)
if (err != nil)
{
klog.Errorf("failed to stat /sys/bus/vdpa/devices/"+vdpaId+": %v", err)
}
else
{
_, _, _ = system("/sbin/vdpa", "-j", "dev", "del", vdpaId)
}
stateFile := ns.stateDir + vdpaId + ".json"
os.Remove(stateFile)
pidFile := ns.stateDir + vdpaId + ".pid"
_, err = os.Stat(pidFile)
if (os.IsNotExist(err))
{
// ok, already killed
}
else if (err != nil)
{
klog.Errorf("Failed to stat %v: %v", pidFile, err)
return
}
else
{
err = killByPidFile(pidFile)
if (err != nil)
{
klog.Errorf("Failed to kill started qemu-storage-daemon: %v", err)
}
os.Remove(pidFile)
}
}
func (ns *NodeServer) restarter()
{
// Restart dead VDUSE daemons at regular intervals
// Otherwise volume I/O may hang in case of a qemu-storage-daemon crash
// Moreover, it may lead to a kernel panic of the kernel is configured to
// panic on hung tasks
ticker := time.NewTicker(ns.restartInterval)
defer ticker.Stop()
for
{
<-ticker.C
ns.restoreVduseDaemons()
}
}
func (ns *NodeServer) restoreVduseDaemons()
{
pattern := ns.stateDir+"vitastor-vduse-*.json"
matches, err := filepath.Glob(pattern)
if (err != nil)
{
klog.Errorf("failed to list %s: %v", pattern, err)
}
if (len(matches) == 0)
{
return
}
devList := make(map[string]interface{})
// example output: {"dev":{"test1":{"type":"block","mgmtdev":"vduse","vendor_id":0,"max_vqs":16,"max_vq_size":128}}}
devListJSON, _, err := system("/sbin/vdpa", "-j", "dev", "list")
if (err != nil)
{
return
}
err = json.Unmarshal(devListJSON, &devList)
devs, ok := devList["dev"].(map[string]interface{})
if (err != nil || !ok)
{
klog.Errorf("/sbin/vdpa -j dev list returned bad JSON (error %v): %v", err, string(devListJSON))
return
}
for _, stateFile := range matches
{
vdpaId := filepath.Base(stateFile)
vdpaId = vdpaId[0:len(vdpaId)-5]
// Check if VDPA device is still added to the bus
if (devs[vdpaId] != nil)
{
// Check if the storage daemon is still active
pidFile := ns.stateDir + vdpaId + ".pid"
exists := false
proc, err := findByPidFile(pidFile)
if (err == nil)
{
exists = proc.Signal(syscall.Signal(0)) == nil
}
if (!exists)
{
// Restart daemon
stateJSON, err := os.ReadFile(stateFile)
if (err != nil)
{
klog.Warningf("error reading state file %v: %v", stateFile, err)
}
else
{
var state DeviceState
err := json.Unmarshal(stateJSON, &state)
if (err != nil)
{
klog.Warningf("state file %v contains invalid JSON (error %v): %v", stateFile, err, string(stateJSON))
}
else
{
klog.Warningf("restarting storage daemon for volume %v (VDPA ID %v)", state.Image, vdpaId)
_ = startStorageDaemon(vdpaId, state.Image, pidFile, state.ConfigPath, state.Readonly)
}
}
}
}
else
{
// Unused, clean it up
ns.unmapVduseById(vdpaId)
}
}
}
// NodePublishVolume mounts the volume mounted to the staging path to the target path
func (ns *NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublishVolumeRequest) (*csi.NodePublishVolumeResponse, error)
{
@ -70,10 +473,10 @@ func (ns *NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublis
isBlock := req.GetVolumeCapability().GetBlock() != nil
// Check that it's not already mounted
_, error := mount.IsNotMountPoint(ns.mounter, targetPath)
if (error != nil)
_, err := mount.IsNotMountPoint(ns.mounter, targetPath)
if (err != nil)
{
if (os.IsNotExist(error))
if (os.IsNotExist(err))
{
if (isBlock)
{
@ -81,13 +484,13 @@ func (ns *NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublis
if (err != nil)
{
klog.Errorf("failed to create block device mount target %s with error: %v", targetPath, err)
return nil, status.Error(codes.Internal, err.Error())
return nil, err
}
err = pathFile.Close()
if (err != nil)
{
klog.Errorf("failed to close %s with error: %v", targetPath, err)
return nil, status.Error(codes.Internal, err.Error())
return nil, err
}
}
else
@ -96,121 +499,112 @@ func (ns *NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublis
if (err != nil)
{
klog.Errorf("failed to create fs mount target %s with error: %v", targetPath, err)
return nil, status.Error(codes.Internal, err.Error())
return nil, err
}
}
}
else
{
return nil, status.Error(codes.Internal, error.Error())
return nil, err
}
}
ctxVars := make(map[string]string)
err := json.Unmarshal([]byte(req.VolumeId), &ctxVars)
err = json.Unmarshal([]byte(req.VolumeId), &ctxVars)
if (err != nil)
{
return nil, status.Error(codes.Internal, "volume ID not in JSON format")
}
volName := ctxVars["name"]
_, etcdUrl, etcdPrefix := GetConnectionParams(ctxVars)
if (len(etcdUrl) == 0)
{
return nil, status.Error(codes.InvalidArgument, "no etcdUrl in storage class configuration and no etcd_address in vitastor.conf")
}
// Map NBD device
// FIXME: Check if already mapped
args := []string{
"map", "--etcd_address", strings.Join(etcdUrl, ","),
"--etcd_prefix", etcdPrefix,
"--image", volName,
};
if (ctxVars["configPath"] != "")
{
args = append(args, "--config_path", ctxVars["configPath"])
}
if (req.GetReadonly())
{
args = append(args, "--readonly", "1")
}
c := exec.Command("/usr/bin/vitastor-nbd", args...)
var stdout, stderr bytes.Buffer
c.Stdout, c.Stderr = &stdout, &stderr
err = c.Run()
stdoutStr, stderrStr := string(stdout.Bytes()), string(stderr.Bytes())
_, err = GetConnectionParams(ctxVars)
if (err != nil)
{
klog.Errorf("vitastor-nbd map failed: %s, status %s\n", stdoutStr+stderrStr, err)
return nil, status.Error(codes.Internal, stdoutStr+stderrStr+" (status "+err.Error()+")")
}
devicePath := strings.TrimSpace(stdoutStr)
// Check existing format
diskMounter := &mount.SafeFormatAndMount{Interface: ns.mounter, Exec: utilexec.New()}
existingFormat, err := diskMounter.GetDiskFormat(devicePath)
if (err != nil)
{
klog.Errorf("failed to get disk format for path %s, error: %v", err)
// unmap NBD device
unmapOut, unmapErr := exec.Command("/usr/bin/vitastor-nbd", "unmap", devicePath).CombinedOutput()
if (unmapErr != nil)
{
klog.Errorf("failed to unmap NBD device %s: %s, error: %v", devicePath, unmapOut, unmapErr)
}
return nil, err
}
// Format the device (ext4 or xfs)
fsType := req.GetVolumeCapability().GetMount().GetFsType()
opt := req.GetVolumeCapability().GetMount().GetMountFlags()
opt = append(opt, "_netdev")
if ((req.VolumeCapability.AccessMode.Mode == csi.VolumeCapability_AccessMode_MULTI_NODE_READER_ONLY ||
req.VolumeCapability.AccessMode.Mode == csi.VolumeCapability_AccessMode_SINGLE_NODE_READER_ONLY) &&
!Contains(opt, "ro"))
var devicePath, vdpaId string
if (!ns.useVduse)
{
opt = append(opt, "ro")
}
if (fsType == "xfs")
{
opt = append(opt, "nouuid")
}
readOnly := Contains(opt, "ro")
if (existingFormat == "" && !readOnly)
{
args := []string{}
switch fsType
{
case "ext4":
args = []string{"-m0", "-Enodiscard,lazy_itable_init=1,lazy_journal_init=1", devicePath}
case "xfs":
args = []string{"-K", devicePath}
}
if (len(args) > 0)
{
cmdOut, cmdErr := diskMounter.Exec.Command("mkfs."+fsType, args...).CombinedOutput()
if (cmdErr != nil)
{
klog.Errorf("failed to run mkfs error: %v, output: %v", cmdErr, string(cmdOut))
// unmap NBD device
unmapOut, unmapErr := exec.Command("/usr/bin/vitastor-nbd", "unmap", devicePath).CombinedOutput()
if (unmapErr != nil)
{
klog.Errorf("failed to unmap NBD device %s: %s, error: %v", devicePath, unmapOut, unmapErr)
}
return nil, status.Error(codes.Internal, cmdErr.Error())
}
}
}
if (isBlock)
{
opt = append(opt, "bind")
err = diskMounter.Mount(devicePath, targetPath, fsType, opt)
devicePath, err = ns.mapNbd(volName, ctxVars, req.GetReadonly())
}
else
{
devicePath, vdpaId, err = ns.mapVduse(volName, ctxVars, req.GetReadonly())
}
if (err != nil)
{
return nil, err
}
diskMounter := &mount.SafeFormatAndMount{Interface: ns.mounter, Exec: utilexec.New()}
if (isBlock)
{
err = diskMounter.Mount(devicePath, targetPath, "", []string{"bind"})
}
else
{
// Check existing format
existingFormat, err := diskMounter.GetDiskFormat(devicePath)
if (err != nil)
{
klog.Errorf("failed to get disk format for path %s, error: %v", err)
goto unmap
}
// Format the device (ext4 or xfs)
fsType := req.GetVolumeCapability().GetMount().GetFsType()
opt := req.GetVolumeCapability().GetMount().GetMountFlags()
opt = append(opt, "_netdev")
if ((req.VolumeCapability.AccessMode.Mode == csi.VolumeCapability_AccessMode_MULTI_NODE_READER_ONLY ||
req.VolumeCapability.AccessMode.Mode == csi.VolumeCapability_AccessMode_SINGLE_NODE_READER_ONLY) &&
!Contains(opt, "ro"))
{
opt = append(opt, "ro")
}
if (fsType == "xfs")
{
opt = append(opt, "nouuid")
}
readOnly := Contains(opt, "ro")
if (existingFormat == "" && !readOnly)
{
var cmdOut []byte
switch fsType
{
case "ext4":
args := []string{"-m0", "-Enodiscard,lazy_itable_init=1,lazy_journal_init=1", devicePath}
cmdOut, err = diskMounter.Exec.Command("mkfs.ext4", args...).CombinedOutput()
case "xfs":
cmdOut, err = diskMounter.Exec.Command("mkfs.xfs", "-K", devicePath).CombinedOutput()
}
if (err != nil)
{
klog.Errorf("failed to run mkfs error: %v, output: %v", err, string(cmdOut))
goto unmap
}
}
err = diskMounter.FormatAndMount(devicePath, targetPath, fsType, opt)
// Try to run online resize on mount.
// FIXME: Implement online resize. It requires online resize support in vitastor-nbd.
if (err == nil && existingFormat != "" && !readOnly)
{
var cmdOut []byte
switch (fsType)
{
case "ext4":
cmdOut, err = diskMounter.Exec.Command("resize2fs", devicePath).CombinedOutput()
case "xfs":
cmdOut, err = diskMounter.Exec.Command("xfs_growfs", devicePath).CombinedOutput()
}
if (err != nil)
{
klog.Errorf("failed to run resizefs error: %v, output: %v", err, string(cmdOut))
goto unmap
}
}
}
if (err != nil)
{
@ -218,15 +612,20 @@ func (ns *NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublis
"failed to mount device path (%s) to path (%s) for volume (%s) error: %s",
devicePath, targetPath, volName, err,
)
// unmap NBD device
unmapOut, unmapErr := exec.Command("/usr/bin/vitastor-nbd", "unmap", devicePath).CombinedOutput()
if (unmapErr != nil)
{
klog.Errorf("failed to unmap NBD device %s: %s, error: %v", devicePath, unmapOut, unmapErr)
}
return nil, status.Error(codes.Internal, err.Error())
goto unmap
}
return &csi.NodePublishVolumeResponse{}, nil
unmap:
if (!ns.useVduse || len(devicePath) >= 8 && devicePath[0:8] == "/dev/nbd")
{
ns.unmapNbd(devicePath)
}
else
{
ns.unmapVduseById(vdpaId)
}
return nil, err
}
// NodeUnpublishVolume unmounts the volume from the target path
@ -241,25 +640,31 @@ func (ns *NodeServer) NodeUnpublishVolume(ctx context.Context, req *csi.NodeUnpu
{
return nil, status.Error(codes.NotFound, "Target path not found")
}
return nil, status.Error(codes.Internal, err.Error())
return nil, err
}
if (devicePath == "")
{
return nil, status.Error(codes.NotFound, "Volume not mounted")
// volume not mounted
klog.Warningf("%s is not a mountpoint, deleting", targetPath)
os.Remove(targetPath)
return &csi.NodeUnpublishVolumeResponse{}, nil
}
// unmount
err = mount.CleanupMountPoint(targetPath, ns.mounter, false)
if (err != nil)
{
return nil, status.Error(codes.Internal, err.Error())
return nil, err
}
// unmap NBD device
if (refCount == 1)
{
unmapOut, unmapErr := exec.Command("/usr/bin/vitastor-nbd", "unmap", devicePath).CombinedOutput()
if (unmapErr != nil)
if (!ns.useVduse)
{
klog.Errorf("failed to unmap NBD device %s: %s, error: %v", devicePath, unmapOut, unmapErr)
ns.unmapNbd(devicePath)
}
else
{
ns.unmapVduse(devicePath)
}
}
return &csi.NodeUnpublishVolumeResponse{}, nil

58
debian/build-pve-qemu.sh vendored Normal file
View File

@ -0,0 +1,58 @@
exit
git clone https://git.yourcmc.ru/vitalif/pve-qemu .
# bookworm
docker run -it -v `pwd`/pve-qemu:/root/pve-qemu --name pve-qemu-bullseye debian:bullseye bash
perl -i -pe 's/Types: deb$/Types: deb deb-src/' /etc/apt/sources.list.d/debian.sources
echo 'deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription' >> /etc/apt/sources.list
echo 'deb https://vitastor.io/debian bookworm main' >> /etc/apt/sources.list
echo 'APT::Install-Recommends false;' >> /etc/apt/apt.conf
echo 'ru_RU UTF-8' >> /etc/locale.gen
echo 'en_US UTF-8' >> /etc/locale.gen
apt-get update
apt-get install wget ca-certificates
wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
wget https://vitastor.io/debian/pubkey.gpg -O /etc/apt/trusted.gpg.d/vitastor.gpg
apt-get update
apt-get install git devscripts equivs wget mc libjemalloc-dev vitastor-client-dev lintian locales
mk-build-deps --install ./control
# bullseye
docker run -it -v `pwd`/pve-qemu:/root/pve-qemu --name pve-qemu-bullseye debian:bullseye bash
grep '^deb ' /etc/apt/sources.list | perl -pe 's/^deb /deb-src /' >> /etc/apt/sources.list
echo 'deb [arch=amd64] http://download.proxmox.com/debian/pve bullseye pve-no-subscription' >> /etc/apt/sources.list
echo 'deb https://vitastor.io/debian bullseye main' >> /etc/apt/sources.list
echo 'APT::Install-Recommends false;' >> /etc/apt/apt.conf
echo 'ru_RU UTF-8' >> /etc/locale.gen
echo 'en_US UTF-8' >> /etc/locale.gen
apt-get update
apt-get install wget
wget https://enterprise.proxmox.com/debian/proxmox-release-bullseye.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg
wget https://vitastor.io/debian/pubkey.gpg -O /etc/apt/trusted.gpg.d/vitastor.gpg
apt-get update
apt-get install git devscripts equivs wget mc libjemalloc-dev vitastor-client-dev lintian locales
mk-build-deps --install ./control
# buster
docker run -it -v `pwd`/pve-qemu:/root/pve-qemu --name pve-qemu-buster debian:buster bash
grep '^deb ' /etc/apt/sources.list | perl -pe 's/^deb /deb-src /' >> /etc/apt/sources.list
echo 'deb [arch=amd64] http://download.proxmox.com/debian/pve buster pve-no-subscription' >> /etc/apt/sources.list
echo 'deb https://vitastor.io/debian buster main' >> /etc/apt/sources.list
echo 'deb http://deb.debian.org/debian buster-backports main' >> /etc/apt/sources.list
echo 'APT::Install-Recommends false;' >> /etc/apt/apt.conf
echo 'ru_RU UTF-8' >> /etc/locale.gen
echo 'en_US UTF-8' >> /etc/locale.gen
apt-get update
apt-get install wget ca-certificates
wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
wget https://vitastor.io/debian/pubkey.gpg -O /etc/apt/trusted.gpg.d/vitastor.gpg
apt-get update
apt-get install git devscripts equivs wget mc libjemalloc-dev vitastor-client-dev lintian locales
mk-build-deps --install ./control

7
debian/build-vitastor-bookworm.sh vendored Executable file
View File

@ -0,0 +1,7 @@
#!/bin/bash
cat < vitastor.Dockerfile > ../Dockerfile
cd ..
mkdir -p packages
sudo podman build --build-arg DISTRO=debian --build-arg REL=bookworm -v `pwd`/packages:/root/packages -f Dockerfile .
rm Dockerfile

View File

@ -3,5 +3,5 @@
cat < vitastor.Dockerfile > ../Dockerfile
cd ..
mkdir -p packages
sudo podman build --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f Dockerfile .
sudo podman build --build-arg DISTRO=debian --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f Dockerfile .
rm Dockerfile

View File

@ -3,5 +3,5 @@
cat < vitastor.Dockerfile > ../Dockerfile
cd ..
mkdir -p packages
sudo podman build --build-arg REL=buster -v `pwd`/packages:/root/packages -f Dockerfile .
sudo podman build --build-arg DISTRO=debian --build-arg REL=buster -v `pwd`/packages:/root/packages -f Dockerfile .
rm Dockerfile

7
debian/build-vitastor-ubuntu-jammy.sh vendored Executable file
View File

@ -0,0 +1,7 @@
#!/bin/bash
cat < vitastor.Dockerfile > ../Dockerfile
cd ..
mkdir -p packages
sudo podman build --build-arg DISTRO=ubuntu --build-arg REL=jammy -v `pwd`/packages:/root/packages -f Dockerfile .
rm Dockerfile

4
debian/changelog vendored
View File

@ -1,10 +1,10 @@
vitastor (0.8.3-1) unstable; urgency=medium
vitastor (1.5.0-1) unstable; urgency=medium
* Bugfixes
-- Vitaliy Filippov <vitalif@yourcmc.ru> Fri, 03 Jun 2022 02:09:44 +0300
vitastor (0.8.3-1) unstable; urgency=medium
vitastor (0.7.0-1) unstable; urgency=medium
* Implement NFS proxy
* Add documentation

2
debian/control vendored
View File

@ -2,7 +2,7 @@ Source: vitastor
Section: admin
Priority: optional
Maintainer: Vitaliy Filippov <vitalif@yourcmc.ru>
Build-Depends: debhelper, liburing-dev (>= 0.6), g++ (>= 8), libstdc++6 (>= 8), linux-libc-dev, libgoogle-perftools-dev, libjerasure-dev, libgf-complete-dev, libibverbs-dev, libisal-dev
Build-Depends: debhelper, liburing-dev (>= 0.6), g++ (>= 8), libstdc++6 (>= 8), linux-libc-dev, libgoogle-perftools-dev, libjerasure-dev, libgf-complete-dev, libibverbs-dev, libisal-dev, cmake, pkg-config
Standards-Version: 4.5.0
Homepage: https://vitastor.io/
Rules-Requires-Root: no

View File

@ -1,13 +1,14 @@
# Build patched libvirt for Debian Buster or Bullseye/Sid inside a container
# cd ..; podman build --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f debian/libvirt.Dockerfile .
# cd ..; podman build --build-arg DISTRO=debian --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f debian/libvirt.Dockerfile .
ARG DISTRO=
ARG REL=
FROM debian:$REL
FROM $DISTRO:$REL
ARG REL=
WORKDIR /root
RUN if [ "$REL" = "buster" -o "$REL" = "bullseye" ]; then \
RUN if ([ "${DISTRO}" = "debian" ]) && ( [ "${REL}" = "buster" -o "${REL}" = "bullseye" ] ); then \
echo "deb http://deb.debian.org/debian $REL-backports main" >> /etc/apt/sources.list; \
echo >> /etc/apt/preferences; \
echo 'Package: *' >> /etc/apt/preferences; \
@ -23,7 +24,7 @@ RUN apt-get -y build-dep libvirt0
RUN apt-get -y install libglusterfs-dev
RUN apt-get --download-only source libvirt
ADD patches/libvirt-5.0-vitastor.diff patches/libvirt-7.0-vitastor.diff patches/libvirt-7.5-vitastor.diff patches/libvirt-7.6-vitastor.diff /root
ADD patches/libvirt-5.0-vitastor.diff patches/libvirt-7.0-vitastor.diff patches/libvirt-7.5-vitastor.diff patches/libvirt-7.6-vitastor.diff patches/libvirt-8.0-vitastor.diff /root
RUN set -e; \
mkdir -p /root/packages/libvirt-$REL; \
rm -rf /root/packages/libvirt-$REL/*; \

View File

@ -1,4 +1,4 @@
# Build patched QEMU for Debian Buster or Bullseye/Sid inside a container
# Build patched QEMU for Debian inside a container
# cd ..; podman build --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f debian/patched-qemu.Dockerfile .
ARG REL=
@ -7,7 +7,7 @@ ARG REL=
WORKDIR /root
RUN if [ "$REL" = "buster" -o "$REL" = "bullseye" ]; then \
RUN if [ "$REL" = "buster" -o "$REL" = "bullseye" -o "$REL" = "bookworm" ]; then \
echo "deb http://deb.debian.org/debian $REL-backports main" >> /etc/apt/sources.list; \
echo >> /etc/apt/preferences; \
echo 'Package: *' >> /etc/apt/preferences; \
@ -15,47 +15,47 @@ RUN if [ "$REL" = "buster" -o "$REL" = "bullseye" ]; then \
echo 'Pin-Priority: 500' >> /etc/apt/preferences; \
fi; \
grep '^deb ' /etc/apt/sources.list | perl -pe 's/^deb/deb-src/' >> /etc/apt/sources.list; \
perl -i -pe 's/Types: deb$/Types: deb deb-src/' /etc/apt/sources.list.d/debian.sources || true; \
echo 'APT::Install-Recommends false;' >> /etc/apt/apt.conf; \
echo 'APT::Install-Suggests false;' >> /etc/apt/apt.conf
RUN apt-get update
RUN apt-get -y install qemu fio liburing1 liburing-dev libgoogle-perftools-dev devscripts
RUN apt-get -y install fio liburing-dev libgoogle-perftools-dev devscripts
RUN apt-get -y build-dep qemu
# To build a custom version
#RUN cp /root/packages/qemu-orig/* /root
RUN apt-get --download-only source qemu
ADD patches/qemu-5.0-vitastor.patch patches/qemu-5.1-vitastor.patch patches/qemu-6.1-vitastor.patch src/qemu_driver.c /root/vitastor/patches/
ADD patches /root/vitastor/patches
ADD src/qemu_driver.c /root/vitastor/src/qemu_driver.c
#RUN set -e; \
# apt-get install -y wget; \
# wget -q -O /etc/apt/trusted.gpg.d/vitastor.gpg https://vitastor.io/debian/pubkey.gpg; \
# (echo deb http://vitastor.io/debian $REL main > /etc/apt/sources.list.d/vitastor.list); \
# (echo "APT::Install-Recommends false;" > /etc/apt/apt.conf) && \
# apt-get update; \
# apt-get install -y vitastor-client vitastor-client-dev quilt
RUN set -e; \
apt-get install -y wget; \
wget -q -O /etc/apt/trusted.gpg.d/vitastor.gpg https://vitastor.io/debian/pubkey.gpg; \
(echo deb http://vitastor.io/debian $REL main > /etc/apt/sources.list.d/vitastor.list); \
(echo "APT::Install-Recommends false;" > /etc/apt/apt.conf) && \
dpkg -i /root/packages/vitastor-$REL/vitastor-client_*.deb /root/packages/vitastor-$REL/vitastor-client-dev_*.deb; \
apt-get update; \
apt-get install -y vitastor-client vitastor-client-dev quilt; \
apt-get install -y quilt; \
mkdir -p /root/packages/qemu-$REL; \
rm -rf /root/packages/qemu-$REL/*; \
cd /root/packages/qemu-$REL; \
dpkg-source -x /root/qemu*.dsc; \
if ls -d /root/packages/qemu-$REL/qemu-5.0*; then \
D=$(ls -d /root/packages/qemu-$REL/qemu-5.0*); \
cp /root/vitastor/patches/qemu-5.0-vitastor.patch $D/debian/patches; \
echo qemu-5.0-vitastor.patch >> $D/debian/patches/series; \
elif ls /root/packages/qemu-$REL/qemu-6.1*; then \
D=$(ls -d /root/packages/qemu-$REL/qemu-6.1*); \
cp /root/vitastor/patches/qemu-6.1-vitastor.patch $D/debian/patches; \
echo qemu-6.1-vitastor.patch >> $D/debian/patches/series; \
else \
cp /root/vitastor/patches/qemu-5.1-vitastor.patch /root/packages/qemu-$REL/qemu-*/debian/patches; \
P=`ls -d /root/packages/qemu-$REL/qemu-*/debian/patches`; \
echo qemu-5.1-vitastor.patch >> $P/series; \
fi; \
QEMU_VER=$(ls -d qemu*/ | perl -pe 's!^.*?(\d+\.\d+).*!$1!'); \
D=$(ls -d qemu*/); \
cp /root/vitastor/patches/qemu-$QEMU_VER-vitastor.patch ./qemu-*/debian/patches; \
echo qemu-$QEMU_VER-vitastor.patch >> $D/debian/patches/series; \
cd /root/packages/qemu-$REL/qemu-*/; \
quilt push -a; \
quilt add block/vitastor.c; \
cp /root/vitastor/patches/qemu_driver.c block/vitastor.c; \
cp /root/vitastor/src/qemu_driver.c block/vitastor.c; \
quilt refresh; \
V=$(head -n1 debian/changelog | perl -pe 's/^.*\((.*?)(~bpo[\d\+]*)?\).*$/$1/')+vitastor1; \
V=$(head -n1 debian/changelog | perl -pe 's/5\.2\+dfsg-9/5.2+dfsg-11/; s/^.*\((.*?)(~bpo[\d\+]*)?\).*$/$1/')+vitastor4; \
if [ "$REL" = bullseye ]; then V=${V}bullseye; fi; \
DEBEMAIL="Vitaliy Filippov <vitalif@yourcmc.ru>" dch -D $REL -v $V 'Plug Vitastor block driver'; \
DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage --jobs=auto -sa; \
rm -rf /root/packages/qemu-$REL/qemu-*/

View File

@ -3,4 +3,6 @@ usr/bin/vitastor-cli
usr/bin/vitastor-rm
usr/bin/vitastor-nbd
usr/bin/vitastor-nfs
usr/bin/vitastor-kv
usr/bin/vitastor-kv-stress
usr/lib/*/libvitastor*.so*

View File

@ -1,8 +1,10 @@
# Build Vitastor packages for Debian Buster or Bullseye/Sid inside a container
# cd ..; podman build --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f debian/vitastor.Dockerfile .
# Build Vitastor packages for Debian inside a container
# cd ..; podman build --build-arg DISTRO=debian --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f debian/vitastor.Dockerfile .
ARG DISTRO=debian
ARG REL=
FROM debian:$REL
FROM $DISTRO:$REL
ARG DISTRO=debian
ARG REL=
WORKDIR /root
@ -15,11 +17,12 @@ RUN if [ "$REL" = "buster" -o "$REL" = "bullseye" ]; then \
echo 'Pin-Priority: 500' >> /etc/apt/preferences; \
fi; \
grep '^deb ' /etc/apt/sources.list | perl -pe 's/^deb/deb-src/' >> /etc/apt/sources.list; \
perl -i -pe 's/Types: deb$/Types: deb deb-src/' /etc/apt/sources.list.d/debian.sources || true; \
echo 'APT::Install-Recommends false;' >> /etc/apt/apt.conf; \
echo 'APT::Install-Suggests false;' >> /etc/apt/apt.conf
RUN apt-get update
RUN apt-get -y install fio liburing1 liburing-dev libgoogle-perftools-dev devscripts
RUN apt-get -y install fio liburing-dev libgoogle-perftools-dev devscripts
RUN apt-get -y build-dep fio
RUN apt-get --download-only source fio
RUN apt-get update && apt-get -y install libjerasure-dev cmake libibverbs-dev libisal-dev
@ -34,8 +37,8 @@ RUN set -e -x; \
mkdir -p /root/packages/vitastor-$REL; \
rm -rf /root/packages/vitastor-$REL/*; \
cd /root/packages/vitastor-$REL; \
cp -r /root/vitastor vitastor-0.8.3; \
cd vitastor-0.8.3; \
cp -r /root/vitastor vitastor-1.5.0; \
cd vitastor-1.5.0; \
ln -s /root/fio-build/fio-*/ ./fio; \
FIO=$(head -n1 fio/debian/changelog | perl -pe 's/^.*\((.*?)\).*$/$1/'); \
ls /usr/include/linux/raw.h || cp ./debian/raw.h /usr/include/linux/raw.h; \
@ -48,8 +51,8 @@ RUN set -e -x; \
rm -rf a b; \
echo "dep:fio=$FIO" > debian/fio_version; \
cd /root/packages/vitastor-$REL; \
tar --sort=name --mtime='2020-01-01' --owner=0 --group=0 --exclude=debian -cJf vitastor_0.8.3.orig.tar.xz vitastor-0.8.3; \
cd vitastor-0.8.3; \
tar --sort=name --mtime='2020-01-01' --owner=0 --group=0 --exclude=debian -cJf vitastor_1.5.0.orig.tar.xz vitastor-1.5.0; \
cd vitastor-1.5.0; \
V=$(head -n1 debian/changelog | perl -pe 's/^.*\((.*?)\).*$/$1/'); \
DEBFULLNAME="Vitaliy Filippov <vitalif@yourcmc.ru>" dch -D $REL -v "$V""$REL" "Rebuild for $REL"; \
DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage --jobs=auto -sa; \

Binary file not shown.

View File

@ -17,20 +17,23 @@ Configuration parameters can be set in 3 places:
- Configuration file (`/etc/vitastor/vitastor.conf` or other path)
- etcd key `/vitastor/config/global`. Most variables can be set there, but etcd
connection parameters should obviously be set in the configuration file.
- Command line of Vitastor components: OSD, mon, fio and QEMU options,
OpenStack/Proxmox/etc configuration. The latter doesn't allow to set all
variables directly, but it allows to override the configuration file and
set everything you need inside it.
- Command line of Vitastor components: OSD (when you run it without vitastor-disk),
mon, fio and QEMU options, OpenStack/Proxmox/etc configuration. The latter
doesn't allow to set all variables directly, but it allows to override the
configuration file and set everything you need inside it.
- OSD superblocks created by [vitastor-disk](usage/disk.en.md) contain
primarily disk layout parameters of specific OSDs. In fact, these parameters
are automatically passed into the command line of vitastor-osd process, so
they have the same "status" as command-line parameters.
In the future, additional configuration methods may be added:
- OSD superblock which will, by design, contain parameters related to the disk
layout and to one specific OSD.
- OSD-specific keys in etcd like `/vitastor/config/osd/<number>`.
## Parameter Reference
- [Common](config/common.en.md)
- [Network](config/network.en.md)
- [Client](config/client.en.md)
- [Global Disk Layout](config/layout-cluster.en.md)
- [OSD Disk Layout](config/layout-osd.en.md)
- [OSD Runtime Parameters](config/osd.en.md)

View File

@ -19,20 +19,24 @@
- Ключе в etcd `/vitastor/config/global`. Большая часть параметров может
задаваться там, кроме, естественно, самих параметров соединения с etcd,
которые должны задаваться в файле конфигурации
- В командной строке компонентов Vitastor: OSD, монитора, опциях fio и QEMU,
настроек OpenStack, Proxmox и т.п. Последние, как правило, не включают полный
набор параметров напрямую, но разрешают определить путь к файлу конфигурации
и задать любые параметры в нём.
- В командной строке компонентов Vitastor: OSD (при ручном запуске без vitastor-disk),
монитора, опциях fio и QEMU, настроек OpenStack, Proxmox и т.п. Последние,
как правило, не включают полный набор параметров напрямую, но позволяют
определить путь к файлу конфигурации и задать любые параметры в нём.
- В суперблоке OSD, записываемом [vitastor-disk](usage/disk.ru.md) - параметры,
связанные с дисковым форматом и с этим конкретным OSD. На самом деле,
при запуске OSD эти параметры автоматически передаются в командную строку
процесса vitastor-osd, то есть по "статусу" они эквивалентны параметрам
командной строки OSD.
В будущем также могут быть добавлены другие способы конфигурации:
- Суперблок OSD, в котором будут храниться параметры OSD, связанные с дисковым
форматом и с этим конкретным OSD.
- OSD-специфичные ключи в etcd типа `/vitastor/config/osd/<номер>`.
## Список параметров
- [Общие](config/common.ru.md)
- [Сеть](config/network.ru.md)
- [Клиентский код](config/client.ru.md)
- [Глобальные дисковые параметры](config/layout-cluster.ru.md)
- [Дисковые параметры OSD](config/layout-osd.ru.md)
- [Прочие параметры OSD](config/osd.ru.md)

159
docs/config/client.en.md Normal file
View File

@ -0,0 +1,159 @@
[Documentation](../../README.md#documentation) → [Configuration](../config.en.md) → Client Parameters
-----
[Читать на русском](client.ru.md)
# Client Parameters
These parameters apply only to Vitastor clients (QEMU, fio, NBD and so on) and
affect their interaction with the cluster.
- [client_retry_interval](#client_retry_interval)
- [client_eio_retry_interval](#client_eio_retry_interval)
- [client_max_dirty_bytes](#client_max_dirty_bytes)
- [client_max_dirty_ops](#client_max_dirty_ops)
- [client_enable_writeback](#client_enable_writeback)
- [client_max_buffered_bytes](#client_max_buffered_bytes)
- [client_max_buffered_ops](#client_max_buffered_ops)
- [client_max_writeback_iodepth](#client_max_writeback_iodepth)
- [nbd_timeout](#nbd_timeout)
- [nbd_max_devices](#nbd_max_devices)
- [nbd_max_part](#nbd_max_part)
## client_retry_interval
- Type: milliseconds
- Default: 50
- Minimum: 10
- Can be changed online: yes
Retry time for I/O requests failed due to inactive PGs or network
connectivity errors.
## client_eio_retry_interval
- Type: milliseconds
- Default: 1000
- Can be changed online: yes
Retry time for I/O requests failed due to data corruption or unfinished
EC object deletions (has_incomplete PG state). 0 disables such retries
and clients are not blocked and just get EIO error code instead.
## client_max_dirty_bytes
- Type: integer
- Default: 33554432
- Can be changed online: yes
Without [immediate_commit](layout-cluster.en.md#immediate_commit)=all this parameter sets the limit of "dirty"
(not committed by fsync) data allowed by the client before forcing an
additional fsync and committing the data. Also note that the client always
holds a copy of uncommitted data in memory so this setting also affects
RAM usage of clients.
## client_max_dirty_ops
- Type: integer
- Default: 1024
- Can be changed online: yes
Same as client_max_dirty_bytes, but instead of total size, limits the number
of uncommitted write operations.
## client_enable_writeback
- Type: boolean
- Default: false
- Can be changed online: yes
This parameter enables client-side write buffering. This means that write
requests are accumulated in memory for a short time before being sent to
a Vitastor cluster which allows to send them in parallel and increase
performance of some applications. Writes are buffered until client forces
a flush with fsync() or until the amount of buffered writes exceeds the
limit.
Write buffering significantly increases performance of some applications,
for example, CrystalDiskMark under Windows (LOL :-D), but also any other
applications if they do writes in one of two non-optimal ways: either if
they do a lot of small (4 kb or so) sequential writes, or if they do a lot
of small random writes, but without any parallelism or asynchrony, and also
without calling fsync().
With write buffering enabled, you can expect around 22000 T1Q1 random write
iops in QEMU more or less regardless of the quality of your SSDs, and this
number is in fact bound by QEMU itself rather than Vitastor (check it
yourself by adding a "driver=null-co" disk in QEMU). Without write
buffering, the current record is 9900 iops, but the number is usually
even lower with non-ideal hardware, for example, it may be 5000 iops.
Even when this parameter is enabled, write buffering isn't enabled until
the client explicitly allows it, because enabling it without the client
being aware of the fact that his writes may be buffered may lead to data
loss. Because of this, older versions of clients don't support write
buffering at all, newer versions of the QEMU driver allow write buffering
only if it's enabled in disk settings with `-blockdev cache.direct=false`,
and newer versions of FIO only allow write buffering if you don't specify
`-direct=1`. NBD and NFS drivers allow write buffering by default.
You can overcome this restriction too with the `client_writeback_allowed`
parameter, but you shouldn't do that unless you **really** know what you
are doing.
## client_max_buffered_bytes
- Type: integer
- Default: 33554432
- Can be changed online: yes
Maximum total size of buffered writes which triggers write-back when reached.
## client_max_buffered_ops
- Type: integer
- Default: 1024
- Can be changed online: yes
Maximum number of buffered writes which triggers write-back when reached.
Multiple consecutive modified data regions are counted as 1 write here.
## client_max_writeback_iodepth
- Type: integer
- Default: 256
- Can be changed online: yes
Maximum number of parallel writes when flushing buffered data to the server.
## nbd_timeout
- Type: seconds
- Default: 300
Timeout for I/O operations for [NBD](../usage/nbd.en.md). If an operation
executes for longer than this timeout, including when your cluster is just
temporarily down for more than timeout, the NBD device will detach by itself
(and possibly break the mounted file system).
You can set timeout to 0 to never detach, but in that case you won't be
able to remove the kernel device at all if the NBD process dies - you'll have
to reboot the host.
## nbd_max_devices
- Type: integer
- Default: 64
Maximum number of NBD devices in the system. This value is passed as
`nbds_max` parameter for the nbd kernel module when vitastor-nbd autoloads it.
## nbd_max_part
- Type: integer
- Default: 3
Maximum number of partitions per NBD device. This value is passed as
`max_part` parameter for the nbd kernel module when vitastor-nbd autoloads it.
Note that (nbds_max)*(1+max_part) usually can't exceed 256.

160
docs/config/client.ru.md Normal file
View File

@ -0,0 +1,160 @@
[Документация](../../README-ru.md#документация) → [Конфигурация](../config.ru.md) → Параметры клиентского кода
-----
[Read in English](client.en.md)
# Параметры клиентского кода
Данные параметры применяются только к клиентам Vitastor (QEMU, fio, NBD и т.п.) и
затрагивают логику их работы с кластером.
- [client_retry_interval](#client_retry_interval)
- [client_eio_retry_interval](#client_eio_retry_interval)
- [client_max_dirty_bytes](#client_max_dirty_bytes)
- [client_max_dirty_ops](#client_max_dirty_ops)
- [client_enable_writeback](#client_enable_writeback)
- [client_max_buffered_bytes](#client_max_buffered_bytes)
- [client_max_buffered_ops](#client_max_buffered_ops)
- [client_max_writeback_iodepth](#client_max_writeback_iodepth)
- [nbd_timeout](#nbd_timeout)
- [nbd_max_devices](#nbd_max_devices)
- [nbd_max_part](#nbd_max_part)
## client_retry_interval
- Тип: миллисекунды
- Значение по умолчанию: 50
- Минимальное значение: 10
- Можно менять на лету: да
Время повтора запросов ввода-вывода, неудачных из-за неактивных PG или
ошибок сети.
## client_eio_retry_interval
- Тип: миллисекунды
- Значение по умолчанию: 1000
- Можно менять на лету: да
Время повтора запросов ввода-вывода, неудачных из-за повреждения данных
или незавершённых удалений EC-объектов (состояния PG has_incomplete).
0 отключает повторы таких запросов и клиенты не блокируются, а вместо
этого просто получают код ошибки EIO.
## client_max_dirty_bytes
- Тип: целое число
- Значение по умолчанию: 33554432
- Можно менять на лету: да
При работе без [immediate_commit](layout-cluster.ru.md#immediate_commit)=all - это лимит объёма "грязных" (не
зафиксированных fsync-ом) данных, при достижении которого клиент будет
принудительно вызывать fsync и фиксировать данные. Также стоит иметь в виду,
что в этом случае до момента fsync клиент хранит копию незафиксированных
данных в памяти, то есть, настройка влияет на потребление памяти клиентами.
## client_max_dirty_ops
- Тип: целое число
- Значение по умолчанию: 1024
- Можно менять на лету: да
Аналогично client_max_dirty_bytes, но ограничивает количество
незафиксированных операций записи вместо их общего объёма.
## client_enable_writeback
- Тип: булево (да/нет)
- Значение по умолчанию: false
- Можно менять на лету: да
Данный параметр разрешает включать буферизацию записи в памяти. Буферизация
означает, что операции записи отправляются на кластер Vitastor не сразу, а
могут небольшое время накапливаться в памяти и сбрасываться сразу пакетами,
до тех пор, пока либо не будет превышен лимит неотправленных записей, либо
пока клиент не вызовет fsync.
Буферизация значительно повышает производительность некоторых приложений,
например, CrystalDiskMark в Windows (ха-ха :-D), но также и любых других,
которые пишут на диск неоптимально: либо последовательно, но мелкими блоками
(например, по 4 кб), либо случайно, но без параллелизма и без fsync - то
есть, например, отправляя 128 операций записи в разные места диска, но не
все сразу с помощью асинхронного I/O, а по одной.
В QEMU с буферизацией записи можно ожидать показателя примерно 22000
операций случайной записи в секунду в 1 поток и с глубиной очереди 1 (T1Q1)
без fsync, почти вне зависимости от того, насколько хороши ваши диски - эта
цифра упирается в сам QEMU. Без буферизации рекорд пока что - 9900 операций
в секунду, но на железе похуже может быть и поменьше, например, 5000 операций
в секунду.
При этом, даже если данный параметр включён, буферизация не включается, если
явно не разрешена клиентом, т.к. если клиент не знает, что запросы записи
буферизуются, это может приводить к потере данных. Поэтому в старых версиях
клиентских драйверов буферизация записи не включается вообще, в новых
версиях QEMU-драйвера включается, только если разрешена опцией диска
`-blockdev cache.direct=false`, а в fio - только если нет опции `-direct=1`.
В NBD и NFS драйверах буферизация записи разрешена по умолчанию.
Можно обойти и это ограничение с помощью параметра `client_writeback_allowed`,
но делать так не надо, если только вы не уверены в том, что делаете, на все
100%. :-)
## client_max_buffered_bytes
- Тип: целое число
- Значение по умолчанию: 33554432
- Можно менять на лету: да
Максимальный общий размер буферизованных записей, при достижении которого
начинается процесс сброса данных на сервер.
## client_max_buffered_ops
- Тип: целое число
- Значение по умолчанию: 1024
- Можно менять на лету: да
Максимальное количество буферизованных записей, при достижении которого
начинается процесс сброса данных на сервер. При этом несколько
последовательных изменённых областей здесь считаются 1 записью.
## client_max_writeback_iodepth
- Тип: целое число
- Значение по умолчанию: 256
- Можно менять на лету: да
Максимальное число параллельных операций записи при сбросе буферов на сервер.
## nbd_timeout
- Тип: секунды
- Значение по умолчанию: 300
Таймаут для операций чтения/записи через [NBD](../usage/nbd.ru.md). Если
операция выполняется дольше таймаута, включая временную недоступность
кластера на время, большее таймаута, NBD-устройство отключится само собой
(и, возможно, сломает примонтированную ФС).
Вы можете установить таймаут в 0, чтобы никогда не отключать устройство по
таймауту, но в этом случае вы вообще не сможете удалить устройство, если
процесс NBD умрёт - вам придётся перезагружать сервер.
## nbd_max_devices
- Тип: целое число
- Значение по умолчанию: 64
Максимальное число NBD-устройств в системе. Данное значение передаётся
модулю ядра nbd как параметр `nbds_max`, когда его загружает vitastor-nbd.
## nbd_max_part
- Тип: целое число
- Значение по умолчанию: 3
Максимальное число разделов на одном NBD-устройстве. Данное значение передаётся
модулю ядра nbd как параметр `max_part`, когда его загружает vitastor-nbd.
Имейте в виду, что (nbds_max)*(1+max_part) обычно не может превышать 256.

View File

@ -25,11 +25,16 @@ running if required parameters are specified.
## etcd_address
- Type: string or array of strings
- Can be changed online: yes
etcd connection endpoint(s). Multiple endpoints may be delimited by "," or
specified in a JSON array `["10.0.115.10:2379/v3","10.0.115.11:2379/v3"]`.
Note that https is not supported for etcd connections yet.
etcd connection endpoints can be changed online by updating global
configuration in etcd itself - this allows to switch the cluster to new
etcd addresses without downtime.
## etcd_prefix
- Type: string
@ -42,5 +47,6 @@ example, use a single etcd cluster for multiple Vitastor clusters.
- Type: integer
- Default: 0
- Can be changed online: yes
Log level. Raise if you want more verbose output.

View File

@ -24,10 +24,14 @@
## etcd_address
- Тип: строка или массив строк
- Можно менять на лету: да
Адрес(а) подключения к etcd. Несколько адресов могут разделяться запятой
или указываться в виде JSON-массива `["10.0.115.10:2379/v3","10.0.115.11:2379/v3"]`.
Адреса подключения к etcd можно поменять на лету, обновив конфигурацию в
самом etcd - это позволяет переключить кластер на новые etcd без остановки.
## etcd_prefix
- Тип: строка
@ -41,5 +45,6 @@
- Тип: целое число
- Значение по умолчанию: 0
- Можно менять на лету: да
Уровень логгирования. Повысьте, если хотите более подробный вывод.

View File

@ -33,12 +33,13 @@ Size of objects (data blocks) into which all physical and virtual drives
in Vitastor, affects memory usage, write amplification and I/O load
distribution effectiveness.
Recommended default block size is 128 KB for SSD and 4 MB for HDD. In fact,
it's possible to use 4 MB for SSD too - it will lower memory usage, but
Recommended default block size is 128 KB for SSD and 1 MB for HDD. In fact,
it's possible to use 1 MB for SSD too - it will lower memory usage, but
may increase average WA and reduce linear performance.
OSD memory usage is roughly (SIZE / BLOCK * 68 bytes) which is roughly
544 MB per 1 TB of used disk space with the default 128 KB block size.
With 1 MB it's 8 times lower.
## bitmap_granularity
@ -95,8 +96,9 @@ SSD cache or "media-cache" - for example, a lot of Seagate EXOS drives have
it (they have internal SSD cache even though it's not stated in datasheets).
Setting this parameter to "all" or "small" in OSD parameters requires enabling
disable_journal_fsync and disable_meta_fsync, setting it to "all" also requires
enabling disable_data_fsync.
[disable_journal_fsync](layout-osd.en.yml#disable_journal_fsync) and
[disable_meta_fsync](layout-osd.en.yml#disable_meta_fsync), setting it to
"all" also requires enabling [disable_data_fsync](layout-osd.en.yml#disable_data_fsync).
TLDR: For optimal performance, set immediate_commit to "all" if you only use
SSDs with supercapacitor-based power loss protection (nonvolatile

View File

@ -33,14 +33,14 @@ OSD) могут сосуществовать в одном кластере Vita
настроек, влияет на потребление памяти, объём избыточной записи (write
amplification) и эффективность распределения нагрузки по OSD.
Рекомендуемые по умолчанию размеры блока - 128 килобайт для SSD и 4
мегабайта для HDD. В принципе, для SSD можно тоже использовать 4 мегабайта,
Рекомендуемые по умолчанию размеры блока - 128 килобайт для SSD и 1 мегабайт
для HDD. В принципе, для SSD можно тоже использовать блок размером 1 мегабайт,
это понизит использование памяти, но ухудшит распределение нагрузки и в
среднем увеличит WA.
Потребление памяти OSD составляет примерно (РАЗМЕР / БЛОК * 68 байт),
т.е. примерно 544 МБ памяти на 1 ТБ занятого места на диске при
стандартном 128 КБ блоке.
стандартном 128 КБ блоке. При 1 МБ блоке памяти нужно в 8 раз меньше.
## bitmap_granularity
@ -103,8 +103,9 @@ HDD-дисках с внутренним SSD или "медиа" кэшем - н
указано в спецификациях).
Указание "all" или "small" в настройках / командной строке OSD требует
включения disable_journal_fsync и disable_meta_fsync, значение "all" также
требует включения disable_data_fsync.
включения [disable_journal_fsync](layout-osd.ru.yml#disable_journal_fsync) и
[disable_meta_fsync](layout-osd.ru.yml#disable_meta_fsync), значение "all"
также требует включения [disable_data_fsync](layout-osd.ru.yml#disable_data_fsync).
Итого, вкратце: для оптимальной производительности установите
immediate_commit в значение "all", если вы используете в кластере только SSD

View File

@ -24,6 +24,8 @@ initialization and can't be changed after it without losing data.
- [disable_journal_fsync](#disable_journal_fsync)
- [disable_device_lock](#disable_device_lock)
- [disk_alignment](#disk_alignment)
- [data_csum_type](#data_csum_type)
- [csum_block_size](#csum_block_size)
## data_device
@ -174,3 +176,43 @@ Intel Optane (probably, not tested yet).
Clients don't need to be aware of disk_alignment, so it's not required to
put a modified value into etcd key /vitastor/config/global.
## data_csum_type
- Type: string
- Default: none
Data checksum type to use. May be "crc32c" or "none". Set to "crc32c" to
enable data checksums.
## csum_block_size
- Type: integer
- Default: 4096
Checksum calculation block size.
Must be equal or a multiple of [bitmap_granularity](layout-cluster.en.md#bitmap_granularity)
(which is usually 4 KB).
Checksums increase metadata size by 4 bytes per each csum_block_size of data.
Checksums are always a tradeoff:
1. You either sacrifice +1 GB RAM per 1 TB of data
2. Or you raise csum_block_size, for example, to 32k and sacrifice
50% random write iops due to checksum read-modify-write
3. Or you turn off [inmemory_metadata](osd.en.md#inmemory_metadata) and
sacrifice 50% random read iops due to checksum reads
All-flash clusters usually have enough RAM to use default csum_block_size,
which uses 1 GB RAM per 1 TB of data. HDD clusters usually don't.
Thus, recommended setups are:
1. All-flash, 1 GB RAM per 1 TB data: default (csum_block_size=4k)
2. All-flash, less RAM: csum_block_size=4k + inmemory_metadata=false
3. Hybrid HDD+SSD: csum_block_size=4k + inmemory_metadata=false
4. HDD-only, faster random read: csum_block_size=32k
5. HDD-only, faster random write: csum_block_size=4k +
inmemory_metadata=false + meta_io=cached
See also [meta_io](osd.en.md#meta_io).

View File

@ -25,6 +25,8 @@
- [disable_journal_fsync](#disable_journal_fsync)
- [disable_device_lock](#disable_device_lock)
- [disk_alignment](#disk_alignment)
- [data_csum_type](#data_csum_type)
- [csum_block_size](#csum_block_size)
## data_device
@ -183,3 +185,47 @@ journal_block_size и meta_block_size. Однако единственные SSD
Клиентам не обязательно знать про disk_alignment, так что помещать значение
этого параметра в etcd в /vitastor/config/global не нужно.
## data_csum_type
- Тип: строка
- Значение по умолчанию: none
Тип используемых OSD контрольных сумм данных. Может быть "crc32c" или "none".
Установите в "crc32c", чтобы включить расчёт и проверку контрольных сумм данных.
Следует понимать, что контрольные суммы в зависимости от размера блока их
расчёта либо увеличивают потребление памяти, либо снижают производительность.
Подробнее смотрите в описании параметра [csum_block_size](#csum_block_size).
## csum_block_size
- Тип: целое число
- Значение по умолчанию: 4096
Размер блока расчёта контрольных сумм.
Должен быть равен или кратен [bitmap_granularity](layout-cluster.ru.md#bitmap_granularity)
(который обычно равен 4 КБ).
Контрольные суммы увеличивают размер метаданных на 4 байта на каждые
csum_block_size данных.
Контрольные суммы - это всегда компромисс:
1. Вы либо жертвуете потреблением +1 ГБ памяти на 1 ТБ дискового пространства
2. Либо вы повышаете csum_block_size до, скажем, 32k и жертвуете 50%
скорости случайной записи из-за цикла чтения-изменения-записи для расчёта
новых контрольных сумм
3. Либо вы отключаете [inmemory_metadata](osd.ru.md#inmemory_metadata) и
жертвуете 50% скорости случайного чтения из-за чтения контрольных сумм
с диска
Таким образом, рекомендуются следующие варианты настроек:
1. All-flash, 1 ГБ памяти на 1 ТБ данных: по умолчанию (csum_block_size=4k)
2. All-flash, меньше памяти: csum_block_size=4k + inmemory_metadata=false
3. Гибридные HDD+SSD: csum_block_size=4k + inmemory_metadata=false
4. Только HDD, быстрее случайное чтение: csum_block_size=32k
5. Только HDD, быстрее случайная запись: csum_block_size=4k +
inmemory_metadata=false + meta_io=cached
Смотрите также [meta_io](osd.ru.md#meta_io).

View File

@ -19,8 +19,8 @@ These parameters only apply to Monitors.
## etcd_mon_ttl
- Type: seconds
- Default: 30
- Minimum: 10
- Default: 1
- Minimum: 5
Monitor etcd lease refresh interval in seconds

View File

@ -19,8 +19,8 @@
## etcd_mon_ttl
- Тип: секунды
- Значение по умолчанию: 30
- Минимальное значение: 10
- Значение по умолчанию: 1
- Минимальное значение: 5
Интервал обновления etcd резервации (lease) монитором

View File

@ -19,17 +19,17 @@ between clients, OSDs and etcd.
- [rdma_max_sge](#rdma_max_sge)
- [rdma_max_msg](#rdma_max_msg)
- [rdma_max_recv](#rdma_max_recv)
- [rdma_max_send](#rdma_max_send)
- [rdma_odp](#rdma_odp)
- [peer_connect_interval](#peer_connect_interval)
- [peer_connect_timeout](#peer_connect_timeout)
- [osd_idle_timeout](#osd_idle_timeout)
- [osd_ping_timeout](#osd_ping_timeout)
- [up_wait_retry_interval](#up_wait_retry_interval)
- [max_etcd_attempts](#max_etcd_attempts)
- [etcd_quick_timeout](#etcd_quick_timeout)
- [etcd_slow_timeout](#etcd_slow_timeout)
- [etcd_keepalive_timeout](#etcd_keepalive_timeout)
- [etcd_ws_keepalive_timeout](#etcd_ws_keepalive_timeout)
- [client_dirty_limit](#client_dirty_limit)
- [etcd_ws_keepalive_interval](#etcd_ws_keepalive_interval)
## tcp_header_buffer_size
@ -68,11 +68,20 @@ but they are not connected to the cluster.
- Type: string
RDMA device name to use for Vitastor OSD communications (for example,
"rocep5s0f0"). Please note that Vitastor RDMA requires Implicit On-Demand
Paging (Implicit ODP) and Scatter/Gather (SG) support from the RDMA device
to work. For example, Mellanox ConnectX-3 and older adapters don't have
Implicit ODP, so they're unsupported by Vitastor. Run `ibv_devinfo -v` as
root to list available RDMA devices and their features.
"rocep5s0f0"). Now Vitastor supports all adapters, even ones without
ODP support, like Mellanox ConnectX-3 and non-Mellanox cards.
Versions up to Vitastor 1.2.0 required ODP which is only present in
Mellanox ConnectX >= 4. See also [rdma_odp](#rdma_odp).
Run `ibv_devinfo -v` as root to list available RDMA devices and their
features.
Remember that you also have to configure your network switches if you use
RoCE/RoCEv2, otherwise you may experience unstable performance. Refer to
the manual of your network vendor for details about setting up the switch
for RoCEv2 correctly. Usually it means setting up Lossless Ethernet with
PFC (Priority Flow Control) and ECN (Explicit Congestion Notification).
## rdma_port_num
@ -116,26 +125,59 @@ required to change this parameter.
## rdma_max_msg
- Type: integer
- Default: 1048576
- Default: 132096
Maximum size of a single RDMA send or receive operation in bytes.
## rdma_max_recv
- Type: integer
- Default: 16
Maximum number of RDMA receive buffers per connection (RDMA requires
preallocated buffers to receive data). Each buffer is `rdma_max_msg` bytes
in size. So this setting directly affects memory usage: a single Vitastor
RDMA client uses `rdma_max_recv * rdma_max_msg * OSD_COUNT` bytes of memory.
Default is roughly 2 MB * number of OSDs.
## rdma_max_send
- Type: integer
- Default: 8
Maximum number of parallel RDMA receive operations. Note that this number
of receive buffers `rdma_max_msg` in size are allocated for each client,
so this setting actually affects memory usage. This is because RDMA receive
operations are (sadly) still not zero-copy in Vitastor. It may be fixed in
later versions.
Maximum number of outstanding RDMA send operations per connection. Should be
less than `rdma_max_recv` so the receiving side doesn't run out of buffers.
Doesn't affect memory usage - additional memory isn't allocated for send
operations.
## rdma_odp
- Type: boolean
- Default: false
Use RDMA with On-Demand Paging. ODP is currently only available on Mellanox
ConnectX-4 and newer adapters. ODP allows to not register memory explicitly
for RDMA adapter to be able to use it. This, in turn, allows to skip memory
copying during sending. One would think this should improve performance, but
**in reality** RDMA performance with ODP is **drastically** worse. Example
3-node cluster with 8 NVMe in each node and 2*25 GBit/s ConnectX-6 RDMA network
without ODP pushes 3950000 read iops, but only 239000 iops with ODP...
This happens because Mellanox ODP implementation seems to be based on
message retransmissions when the adapter doesn't know about the buffer yet -
it likely uses standard "RNR retransmissions" (RNR = receiver not ready)
which is generally slow in RDMA/RoCE networks. Here's a presentation about
it from ISPASS-2021 conference: https://tkygtr6.github.io/pub/ISPASS21_slides.pdf
ODP support is retained in the code just in case a good ODP implementation
appears one day.
## peer_connect_interval
- Type: seconds
- Default: 5
- Minimum: 1
- Can be changed online: yes
Interval before attempting to reconnect to an unavailable OSD.
@ -144,6 +186,7 @@ Interval before attempting to reconnect to an unavailable OSD.
- Type: seconds
- Default: 5
- Minimum: 1
- Can be changed online: yes
Timeout for OSD connection attempts.
@ -152,6 +195,7 @@ Timeout for OSD connection attempts.
- Type: seconds
- Default: 5
- Minimum: 1
- Can be changed online: yes
OSD connection inactivity time after which clients and other OSDs send
keepalive requests to check state of the connection.
@ -161,25 +205,17 @@ keepalive requests to check state of the connection.
- Type: seconds
- Default: 5
- Minimum: 1
- Can be changed online: yes
Maximum time to wait for OSD keepalive responses. If an OSD doesn't respond
within this time, the connection to it is dropped and a reconnection attempt
is scheduled.
## up_wait_retry_interval
- Type: milliseconds
- Default: 500
- Minimum: 50
OSDs respond to clients with a special error code when they receive I/O
requests for a PG that's not synchronized and started. This parameter sets
the time for the clients to wait before re-attempting such I/O requests.
## max_etcd_attempts
- Type: integer
- Default: 5
- Can be changed online: yes
Maximum number of attempts for etcd requests which can't be retried
indefinitely.
@ -188,6 +224,7 @@ indefinitely.
- Type: milliseconds
- Default: 1000
- Can be changed online: yes
Timeout for etcd requests which should complete quickly, like lease refresh.
@ -195,6 +232,7 @@ Timeout for etcd requests which should complete quickly, like lease refresh.
- Type: milliseconds
- Default: 5000
- Can be changed online: yes
Timeout for etcd requests which are allowed to wait for some time.
@ -202,27 +240,16 @@ Timeout for etcd requests which are allowed to wait for some time.
- Type: seconds
- Default: max(30, etcd_report_interval*2)
- Can be changed online: yes
Timeout for etcd connection HTTP Keep-Alive. Should be higher than
etcd_report_interval to guarantee that keepalive actually works.
## etcd_ws_keepalive_timeout
## etcd_ws_keepalive_interval
- Type: seconds
- Default: 30
- Can be changed online: yes
etcd websocket ping interval required to keep the connection alive and
detect disconnections quickly.
## client_dirty_limit
- Type: integer
- Default: 33554432
Without immediate_commit=all this parameter sets the limit of "dirty"
(not committed by fsync) data allowed by the client before forcing an
additional fsync and committing the data. Also note that the client always
holds a copy of uncommitted data in memory so this setting also affects
RAM usage of clients.
This parameter doesn't affect OSDs themselves.

View File

@ -19,17 +19,17 @@
- [rdma_max_sge](#rdma_max_sge)
- [rdma_max_msg](#rdma_max_msg)
- [rdma_max_recv](#rdma_max_recv)
- [rdma_max_send](#rdma_max_send)
- [rdma_odp](#rdma_odp)
- [peer_connect_interval](#peer_connect_interval)
- [peer_connect_timeout](#peer_connect_timeout)
- [osd_idle_timeout](#osd_idle_timeout)
- [osd_ping_timeout](#osd_ping_timeout)
- [up_wait_retry_interval](#up_wait_retry_interval)
- [max_etcd_attempts](#max_etcd_attempts)
- [etcd_quick_timeout](#etcd_quick_timeout)
- [etcd_slow_timeout](#etcd_slow_timeout)
- [etcd_keepalive_timeout](#etcd_keepalive_timeout)
- [etcd_ws_keepalive_timeout](#etcd_ws_keepalive_timeout)
- [client_dirty_limit](#client_dirty_limit)
- [etcd_ws_keepalive_interval](#etcd_ws_keepalive_interval)
## tcp_header_buffer_size
@ -71,12 +71,22 @@ RDMA может быть нужно только если у клиентов е
- Тип: строка
Название RDMA-устройства для связи с Vitastor OSD (например, "rocep5s0f0").
Имейте в виду, что поддержка RDMA в Vitastor требует функций устройства
Implicit On-Demand Paging (Implicit ODP) и Scatter/Gather (SG). Например,
адаптеры Mellanox ConnectX-3 и более старые не поддерживают Implicit ODP и
потому не поддерживаются в Vitastor. Запустите `ibv_devinfo -v` от имени
суперпользователя, чтобы посмотреть список доступных RDMA-устройств, их
параметры и возможности.
Сейчас Vitastor поддерживает все модели адаптеров, включая те, у которых
нет поддержки ODP, то есть вы можете использовать RDMA с ConnectX-3 и
картами производства не Mellanox.
Версии Vitastor до 1.2.0 включительно требовали ODP, который есть только
на Mellanox ConnectX 4 и более новых. См. также [rdma_odp](#rdma_odp).
Запустите `ibv_devinfo -v` от имени суперпользователя, чтобы посмотреть
список доступных RDMA-устройств, их параметры и возможности.
Обратите внимание, что если вы используете RoCE/RoCEv2, вам также необходимо
правильно настроить для него коммутаторы, иначе вы можете столкнуться с
нестабильной производительностью. Подробную информацию о настройке
коммутатора для RoCEv2 ищите в документации производителя. Обычно это
подразумевает настройку сети без потерь на основе PFC (Priority Flow
Control) и ECN (Explicit Congestion Notification).
## rdma_port_num
@ -121,28 +131,62 @@ OSD в любом случае согласовывают реальное зн
## rdma_max_msg
- Тип: целое число
- Значение по умолчанию: 1048576
- Значение по умолчанию: 132096
Максимальный размер одной RDMA-операции отправки или приёма.
## rdma_max_recv
- Тип: целое число
- Значение по умолчанию: 16
Максимальное число буферов для RDMA-приёма данных на одно соединение
(RDMA требует заранее выделенных буферов для приёма данных). Каждый буфер
имеет размер `rdma_max_msg` байт. Таким образом, настройка прямо влияет на
потребление памяти - один Vitastor-клиент с RDMA использует
`rdma_max_recv * rdma_max_msg * ЧИСЛО_OSD` байт памяти, по умолчанию -
примерно 2 МБ * число OSD.
## rdma_max_send
- Тип: целое число
- Значение по умолчанию: 8
Максимальное число параллельных RDMA-операций получения данных. Следует
иметь в виду, что данное число буферов размером `rdma_max_msg` выделяется
для каждого подключённого клиентского соединения, так что данная настройка
влияет на потребление памяти. Это так потому, что RDMA-приём данных в
Vitastor, увы, всё равно не является zero-copy, т.е. всё равно 1 раз
копирует данные в памяти. Данная особенность, возможно, будет исправлена в
более новых версиях Vitastor.
Максимальное число RDMA-операций отправки, отправляемых в очередь одного
соединения. Желательно, чтобы оно было меньше `rdma_max_recv`, чтобы
у принимающей стороны в процессе работы не заканчивались буферы на приём.
Не влияет на потребление памяти - дополнительная память на операции отправки
не выделяется.
## rdma_odp
- Тип: булево (да/нет)
- Значение по умолчанию: false
Использовать RDMA с On-Demand Paging. ODP - функция, доступная пока что
исключительно на адаптерах Mellanox ConnectX-4 и более новых. ODP позволяет
не регистрировать память для её использования RDMA-картой. Благодаря этому
можно не копировать данные при отправке их в сеть и, казалось бы, это должно
улучшать производительность - но **по факту** получается так, что
производительность только ухудшается, причём сильно. Пример - на 3-узловом
кластере с 8 NVMe в каждом узле и сетью 2*25 Гбит/с на чтение с RDMA без ODP
удаётся снять 3950000 iops, а с ODP - всего 239000 iops...
Это происходит из-за того, что реализация ODP у Mellanox неоптимальная и
основана на повторной передаче сообщений, когда карте не известен буфер -
вероятно, на стандартных "RNR retransmission" (RNR = receiver not ready).
А данные повторные передачи в RDMA/RoCE - всегда очень медленная штука.
Презентация на эту тему с конференции ISPASS-2021: https://tkygtr6.github.io/pub/ISPASS21_slides.pdf
Возможность использования ODP сохранена в коде на случай, если вдруг в один
прекрасный день появится хорошая реализация ODP.
## peer_connect_interval
- Тип: секунды
- Значение по умолчанию: 5
- Минимальное значение: 1
- Можно менять на лету: да
Время ожидания перед повторной попыткой соединиться с недоступным OSD.
@ -151,6 +195,7 @@ Vitastor, увы, всё равно не является zero-copy, т.е. вс
- Тип: секунды
- Значение по умолчанию: 5
- Минимальное значение: 1
- Можно менять на лету: да
Максимальное время ожидания попытки соединения с OSD.
@ -159,6 +204,7 @@ Vitastor, увы, всё равно не является zero-copy, т.е. вс
- Тип: секунды
- Значение по умолчанию: 5
- Минимальное значение: 1
- Можно менять на лету: да
Время неактивности соединения с OSD, после которого клиенты или другие OSD
посылают запрос проверки состояния соединения.
@ -168,27 +214,17 @@ Vitastor, увы, всё равно не является zero-copy, т.е. вс
- Тип: секунды
- Значение по умолчанию: 5
- Минимальное значение: 1
- Можно менять на лету: да
Максимальное время ожидания ответа на запрос проверки состояния соединения.
Если OSD не отвечает за это время, соединение отключается и производится
повторная попытка соединения.
## up_wait_retry_interval
- Тип: миллисекунды
- Значение по умолчанию: 500
- Минимальное значение: 50
Когда OSD получают от клиентов запросы ввода-вывода, относящиеся к не
поднятым на данный момент на них PG, либо к PG в процессе синхронизации,
они отвечают клиентам специальным кодом ошибки, означающим, что клиент
должен некоторое время подождать перед повторением запроса. Именно это время
ожидания задаёт данный параметр.
## max_etcd_attempts
- Тип: целое число
- Значение по умолчанию: 5
- Можно менять на лету: да
Максимальное число попыток выполнения запросов к etcd для тех запросов,
которые нельзя повторять бесконечно.
@ -197,6 +233,7 @@ Vitastor, увы, всё равно не является zero-copy, т.е. вс
- Тип: миллисекунды
- Значение по умолчанию: 1000
- Можно менять на лету: да
Максимальное время выполнения запросов к etcd, которые должны завершаться
быстро, таких, как обновление резервации (lease).
@ -205,6 +242,7 @@ Vitastor, увы, всё равно не является zero-copy, т.е. вс
- Тип: миллисекунды
- Значение по умолчанию: 5000
- Можно менять на лету: да
Максимальное время выполнения запросов к etcd, для которых не обязательно
гарантировать быстрое выполнение.
@ -213,26 +251,15 @@ Vitastor, увы, всё равно не является zero-copy, т.е. вс
- Тип: секунды
- Значение по умолчанию: max(30, etcd_report_interval*2)
- Можно менять на лету: да
Таймаут для HTTP Keep-Alive в соединениях к etcd. Должен быть больше, чем
etcd_report_interval, чтобы keepalive гарантированно работал.
## etcd_ws_keepalive_timeout
## etcd_ws_keepalive_interval
- Тип: секунды
- Значение по умолчанию: 30
- Можно менять на лету: да
Интервал проверки живости вебсокет-подключений к etcd.
## client_dirty_limit
- Тип: целое число
- Значение по умолчанию: 33554432
При работе без immediate_commit=all - это лимит объёма "грязных" (не
зафиксированных fsync-ом) данных, при достижении которого клиент будет
принудительно вызывать fsync и фиксировать данные. Также стоит иметь в виду,
что в этом случае до момента fsync клиент хранит копию незафиксированных
данных в памяти, то есть, настройка влияет на потребление памяти клиентами.
Параметр не влияет на сами OSD.

View File

@ -7,9 +7,11 @@
# Runtime OSD Parameters
These parameters only apply to OSDs, are not fixed at the moment of OSD drive
initialization and can be changed with an OSD restart.
initialization and can be changed - either with an OSD restart or, for some of
them, even without restarting by updating configuration in etcd.
- [etcd_report_interval](#etcd_report_interval)
- [etcd_stats_interval](#etcd_stats_interval)
- [run_primary](#run_primary)
- [osd_network](#osd_network)
- [bind_address](#bind_address)
@ -17,6 +19,7 @@ initialization and can be changed with an OSD restart.
- [autosync_interval](#autosync_interval)
- [autosync_writes](#autosync_writes)
- [recovery_queue_depth](#recovery_queue_depth)
- [recovery_sleep_us](#recovery_sleep_us)
- [recovery_pg_switch](#recovery_pg_switch)
- [recovery_sync_batch](#recovery_sync_batch)
- [readonly](#readonly)
@ -30,6 +33,9 @@ initialization and can be changed with an OSD restart.
- [max_flusher_count](#max_flusher_count)
- [inmemory_metadata](#inmemory_metadata)
- [inmemory_journal](#inmemory_journal)
- [data_io](#data_io)
- [meta_io](#meta_io)
- [journal_io](#journal_io)
- [journal_sector_buffer_count](#journal_sector_buffer_count)
- [journal_no_same_sector_overwrites](#journal_no_same_sector_overwrites)
- [throttle_small_writes](#throttle_small_writes)
@ -38,17 +44,43 @@ initialization and can be changed with an OSD restart.
- [throttle_target_parallelism](#throttle_target_parallelism)
- [throttle_threshold_us](#throttle_threshold_us)
- [osd_memlock](#osd_memlock)
- [auto_scrub](#auto_scrub)
- [no_scrub](#no_scrub)
- [scrub_interval](#scrub_interval)
- [scrub_queue_depth](#scrub_queue_depth)
- [scrub_sleep](#scrub_sleep)
- [scrub_list_limit](#scrub_list_limit)
- [scrub_find_best](#scrub_find_best)
- [scrub_ec_max_bruteforce](#scrub_ec_max_bruteforce)
- [recovery_tune_interval](#recovery_tune_interval)
- [recovery_tune_util_low](#recovery_tune_util_low)
- [recovery_tune_util_high](#recovery_tune_util_high)
- [recovery_tune_client_util_low](#recovery_tune_client_util_low)
- [recovery_tune_client_util_high](#recovery_tune_client_util_high)
- [recovery_tune_agg_interval](#recovery_tune_agg_interval)
- [recovery_tune_sleep_min_us](#recovery_tune_sleep_min_us)
- [recovery_tune_sleep_cutoff_us](#recovery_tune_sleep_cutoff_us)
## etcd_report_interval
- Type: seconds
- Default: 5
Interval at which OSDs report their state to etcd. Affects OSD lease time
Interval at which OSDs report their liveness to etcd. Affects OSD lease time
and thus the failover speed. Lease time is equal to this parameter value
plus max_etcd_attempts * etcd_quick_timeout because it should be guaranteed
that every OSD always refreshes its lease in time.
## etcd_stats_interval
- Type: seconds
- Default: 30
Interval at which OSDs report their statistics to etcd. Highly affects the
imposed load on etcd, because statistics include a key for every OSD and
for every PG. At the same time, low statistic intervals make `vitastor-cli`
statistics more responsive.
## run_primary
- Type: boolean
@ -91,6 +123,7 @@ OSD by hand.
- Type: seconds
- Default: 5
- Can be changed online: yes
Time interval at which automatic fsyncs/flushes are issued by each OSD when
the immediate_commit mode if disabled. fsyncs are required because without
@ -103,6 +136,7 @@ issue fsyncs at all.
- Type: integer
- Default: 128
- Can be changed online: yes
Same as autosync_interval, but sets the maximum number of uncommitted write
operations before issuing an fsync operation internally.
@ -110,16 +144,30 @@ operations before issuing an fsync operation internally.
## recovery_queue_depth
- Type: integer
- Default: 4
- Default: 1
- Can be changed online: yes
Maximum recovery operations per one primary OSD at any given moment of time.
Currently it's the only parameter available to tune the speed or recovery
and rebalancing, but it's planned to implement more.
Maximum recovery and rebalance operations initiated by each OSD in parallel.
Note that each OSD talks to a lot of other OSDs so actual number of parallel
recovery operations per each OSD is greater than just recovery_queue_depth.
Increasing this parameter can speedup recovery if [auto-tuning](#recovery_tune_interval)
allows it or if it is disabled.
## recovery_sleep_us
- Type: microseconds
- Default: 0
- Can be changed online: yes
Delay for all recovery- and rebalance- related operations. If non-zero,
such operations are artificially slowed down to reduce the impact on
client I/O.
## recovery_pg_switch
- Type: integer
- Default: 128
- Can be changed online: yes
Number of recovery operations before switching to recovery of the next PG.
The idea is to mix all PGs during recovery for more even space and load
@ -130,6 +178,7 @@ Degraded PGs are anyway scanned first.
- Type: integer
- Default: 16
- Can be changed online: yes
Maximum number of recovery operations before issuing an additional fsync.
@ -145,6 +194,7 @@ the underlying device. This may be useful for recovery purposes.
- Type: boolean
- Default: false
- Can be changed online: yes
Disable automatic background recovery of objects. Note that it doesn't
affect implicit recovery of objects happening during writes - a write is
@ -154,6 +204,7 @@ always made to a full set of at least pg_minsize OSDs.
- Type: boolean
- Default: false
- Can be changed online: yes
Disable background movement of data between different OSDs. Disabling it
means that PGs in the `has_misplaced` state will be left in it indefinitely.
@ -162,6 +213,7 @@ means that PGs in the `has_misplaced` state will be left in it indefinitely.
- Type: seconds
- Default: 3
- Can be changed online: yes
Time interval at which OSDs print simple human-readable operation
statistics on stdout.
@ -170,6 +222,7 @@ statistics on stdout.
- Type: seconds
- Default: 10
- Can be changed online: yes
Time interval at which OSDs dump slow or stuck operations on stdout, if
they're any. Also it's the time after which an operation is considered
@ -179,6 +232,7 @@ they're any. Also it's the time after which an operation is considered
- Type: seconds
- Default: 60
- Can be changed online: yes
Number of seconds after which a deleted inode is removed from OSD statistics.
@ -186,6 +240,7 @@ Number of seconds after which a deleted inode is removed from OSD statistics.
- Type: integer
- Default: 128
- Can be changed online: yes
Parallel client write operation limit per one OSD. Operations that exceed
this limit are pushed to a temporary queue instead of being executed
@ -195,6 +250,7 @@ immediately.
- Type: integer
- Default: 1
- Can be changed online: yes
Flusher is a micro-thread that moves data from the journal to the data
area of the device. Their number is auto-tuned between minimum and maximum.
@ -204,6 +260,7 @@ Minimum number is set by this parameter.
- Type: integer
- Default: 256
- Can be changed online: yes
Maximum number of journal flushers (see above min_flusher_count).
@ -233,6 +290,60 @@ is typically very small because it's sufficient to have 16-32 MB journal
for SSD OSDs. However, in theory it's possible that you'll want to turn it
off for hybrid (HDD+SSD) OSDs with large journals on quick devices.
## data_io
- Type: string
- Default: direct
I/O mode for *data*. One of "direct", "cached" or "directsync". Corresponds
to O_DIRECT, O_SYNC and O_DIRECT|O_SYNC, respectively.
Choose "cached" to use Linux page cache. This may improve read performance
for hot data and slower disks - HDDs and maybe SATA SSDs - but will slightly
decrease write performance for fast disks because page cache is an overhead
itself.
Choose "directsync" to use [immediate_commit](layout-cluster.ru.md#immediate_commit)
(which requires disable_data_fsync) with drives having write-back cache
which can't be turned off, for example, Intel Optane. Also note that *some*
desktop SSDs (for example, HP EX950) may ignore O_SYNC thus making
disable_data_fsync unsafe even with "directsync".
## meta_io
- Type: string
- Default: direct
I/O mode for *metadata*. One of "direct", "cached" or "directsync".
"cached" may improve read performance, but only under the following conditions:
1. your drives are relatively slow (HDD, SATA SSD), and
2. checksums are enabled, and
3. [inmemory_metadata](#inmemory_metadata) is disabled.
Under all these conditions, metadata blocks are read from disk on every
read request to verify checksums and caching them may reduce this extra
read load. Without (3) metadata is never read from the disk after starting,
and without (2) metadata blocks are read from disk only during journal
flushing.
"directsync" is the same as above.
If the same device is used for data and metadata, meta_io by default is set
to the same value as [data_io](#data_io).
## journal_io
- Type: string
- Default: direct
I/O mode for *journal*. One of "direct", "cached" or "directsync".
Here, "cached" may only improve read performance for recent writes and
only if [inmemory_journal](#inmemory_journal) is turned off.
If the same device is used for metadata and journal, journal_io by default
is set to the same value as [meta_io](#meta_io).
## journal_sector_buffer_count
- Type: integer
@ -260,6 +371,7 @@ Most (99%) other SSDs don't need this option.
- Type: boolean
- Default: false
- Can be changed online: yes
Enable soft throttling of small journaled writes. Useful for hybrid OSDs
with fast journal/metadata devices and slow data devices. The idea is that
@ -277,6 +389,7 @@ fills up.
- Type: integer
- Default: 100
- Can be changed online: yes
Target maximum number of throttled operations per second under the condition
of full journal. Set it to approximate random write iops of your data devices
@ -286,6 +399,7 @@ of full journal. Set it to approximate random write iops of your data devices
- Type: integer
- Default: 100
- Can be changed online: yes
Target maximum bandwidth in MB/s of throttled operations per second under
the condition of full journal. Set it to approximate linear write
@ -295,6 +409,7 @@ performance of your data devices (HDDs).
- Type: integer
- Default: 1
- Can be changed online: yes
Target maximum parallelism of throttled operations under the condition of
full journal. Set it to approximate internal parallelism of your data
@ -304,6 +419,7 @@ devices (1 for HDDs, 4-8 for SSDs).
- Type: microseconds
- Default: 50
- Can be changed online: yes
Minimal computed delay to be applied to throttled operations. Usually
doesn't need to be changed.
@ -313,4 +429,190 @@ doesn't need to be changed.
- Type: boolean
- Default: false
Lock all OSD memory to prevent it from being unloaded into swap with mlockall(). Requires sufficient ulimit -l (max locked memory).
Lock all OSD memory to prevent it from being unloaded into swap with
mlockall(). Requires sufficient ulimit -l (max locked memory).
## auto_scrub
- Type: boolean
- Default: false
- Can be changed online: yes
Data scrubbing is the process of background verification of copies to find
and repair corrupted blocks. It's not run automatically by default since
it's a new feature. Set this parameter to true to enable automatic scrubs.
This parameter makes OSDs automatically schedule data scrubbing of clean PGs
every `scrub_interval` (see below). You can also start/schedule scrubbing
manually by setting `next_scrub` JSON key to the desired UNIX time of the
next scrub in `/pg/history/...` values in etcd.
## no_scrub
- Type: boolean
- Default: false
- Can be changed online: yes
Temporarily disable scrubbing and stop running scrubs.
## scrub_interval
- Type: string
- Default: 30d
- Can be changed online: yes
Default automatic scrubbing interval for all pools. Numbers without suffix
are treated as seconds, possible unit suffixes include 's' (seconds),
'm' (minutes), 'h' (hours), 'd' (days), 'M' (months) and 'y' (years).
## scrub_queue_depth
- Type: integer
- Default: 1
- Can be changed online: yes
Number of parallel scrubbing operations per one OSD.
## scrub_sleep
- Type: milliseconds
- Default: 0
- Can be changed online: yes
Additional interval between two consecutive scrubbing operations on one OSD.
Can be used to slow down scrubbing if it affects user load too much.
## scrub_list_limit
- Type: integer
- Default: 1000
- Can be changed online: yes
Number of objects to list in one listing operation during scrub.
## scrub_find_best
- Type: boolean
- Default: true
- Can be changed online: yes
Find and automatically restore best versions of objects with unmatched
copies. In replicated setups, the best version is the version with most
matching replicas. In EC setups, the best version is the subset of data
and parity chunks without mismatches.
The hypothetical situation where you might want to disable it is when
you have 3 replicas and you are paranoid that 2 HDDs out of 3 may silently
corrupt an object in the same way (for example, zero it out) and only
1 HDD will remain good. In this case disabling scrub_find_best may help
you to recover the data! See also scrub_ec_max_bruteforce below.
## scrub_ec_max_bruteforce
- Type: integer
- Default: 100
- Can be changed online: yes
Vitastor can locate corrupted chunks in EC setups with more than 1 parity
chunk by brute-forcing all possible error locations. This configuration
value limits the maximum number of checked combinations. You can try to
increase it if you have EC N+K setup with N and K large enough for
combination count `C(N+K-1, K-1) = (N+K-1)! / (K-1)! / N!` to be greater
than the default 100.
If there are too many possible combinations or if multiple combinations give
correct results then objects are marked inconsistent and aren't recovered
automatically.
In replicated setups bruteforcing isn't needed, Vitastor just assumes that
the variant with most available equal copies is correct. For example, if
you have 3 replicas and 1 of them differs, this one is considered to be
corrupted. But if there is no "best" version with more copies than all
others have then the object is also marked as inconsistent.
## recovery_tune_interval
- Type: seconds
- Default: 1
- Can be changed online: yes
Interval at which OSD re-considers client and recovery load and automatically
adjusts [recovery_sleep_us](#recovery_sleep_us). Recovery auto-tuning is
disabled if recovery_tune_interval is set to 0.
Auto-tuning targets utilization. Utilization is a measure of load and is
equal to the product of iops and average latency (so it may be greater
than 1). You set "low" and "high" client utilization thresholds and two
corresponding target recovery utilization levels. OSD calculates desired
recovery utilization from client utilization using linear interpolation
and auto-tunes recovery operation delay to make actual recovery utilization
match desired.
This allows to reduce recovery/rebalance impact on client operations. It is
of course impossible to remove it completely, but it should become adequate.
In some tests rebalance could earlier drop client write speed from 1.5 GB/s
to 50-100 MB/s, with default auto-tuning settings it now only reduces
to ~1 GB/s.
## recovery_tune_util_low
- Type: number
- Default: 0.1
- Can be changed online: yes
Desired recovery/rebalance utilization when client load is high, i.e. when
it is at or above recovery_tune_client_util_high.
## recovery_tune_util_high
- Type: number
- Default: 1
- Can be changed online: yes
Desired recovery/rebalance utilization when client load is low, i.e. when
it is at or below recovery_tune_client_util_low.
## recovery_tune_client_util_low
- Type: number
- Default: 0
- Can be changed online: yes
Client utilization considered "low".
## recovery_tune_client_util_high
- Type: number
- Default: 0.5
- Can be changed online: yes
Client utilization considered "high".
## recovery_tune_agg_interval
- Type: integer
- Default: 10
- Can be changed online: yes
The number of last auto-tuning iterations to use for calculating the
delay as average. Lower values result in quicker response to client
load change, higher values result in more stable delay. Default value of 10
is usually fine.
## recovery_tune_sleep_min_us
- Type: microseconds
- Default: 10
- Can be changed online: yes
Minimum possible value for auto-tuned recovery_sleep_us. Lower values
are changed to 0.
## recovery_tune_sleep_cutoff_us
- Type: microseconds
- Default: 10000000
- Can be changed online: yes
Maximum possible value for auto-tuned recovery_sleep_us. Higher values
are treated as outliers and ignored in aggregation.

View File

@ -8,9 +8,11 @@
Данные параметры используются только OSD, но, в отличие от дисковых параметров,
не фиксируются в момент инициализации дисков OSD и могут быть изменены в любой
момент с перезапуском OSD.
момент с помощью перезапуска OSD, а некоторые и без перезапуска, с помощью
изменения конфигурации в etcd.
- [etcd_report_interval](#etcd_report_interval)
- [etcd_stats_interval](#etcd_stats_interval)
- [run_primary](#run_primary)
- [osd_network](#osd_network)
- [bind_address](#bind_address)
@ -18,6 +20,7 @@
- [autosync_interval](#autosync_interval)
- [autosync_writes](#autosync_writes)
- [recovery_queue_depth](#recovery_queue_depth)
- [recovery_sleep_us](#recovery_sleep_us)
- [recovery_pg_switch](#recovery_pg_switch)
- [recovery_sync_batch](#recovery_sync_batch)
- [readonly](#readonly)
@ -31,6 +34,9 @@
- [max_flusher_count](#max_flusher_count)
- [inmemory_metadata](#inmemory_metadata)
- [inmemory_journal](#inmemory_journal)
- [data_io](#data_io)
- [meta_io](#meta_io)
- [journal_io](#journal_io)
- [journal_sector_buffer_count](#journal_sector_buffer_count)
- [journal_no_same_sector_overwrites](#journal_no_same_sector_overwrites)
- [throttle_small_writes](#throttle_small_writes)
@ -39,17 +45,43 @@
- [throttle_target_parallelism](#throttle_target_parallelism)
- [throttle_threshold_us](#throttle_threshold_us)
- [osd_memlock](#osd_memlock)
- [auto_scrub](#auto_scrub)
- [no_scrub](#no_scrub)
- [scrub_interval](#scrub_interval)
- [scrub_queue_depth](#scrub_queue_depth)
- [scrub_sleep](#scrub_sleep)
- [scrub_list_limit](#scrub_list_limit)
- [scrub_find_best](#scrub_find_best)
- [scrub_ec_max_bruteforce](#scrub_ec_max_bruteforce)
- [recovery_tune_interval](#recovery_tune_interval)
- [recovery_tune_util_low](#recovery_tune_util_low)
- [recovery_tune_util_high](#recovery_tune_util_high)
- [recovery_tune_client_util_low](#recovery_tune_client_util_low)
- [recovery_tune_client_util_high](#recovery_tune_client_util_high)
- [recovery_tune_agg_interval](#recovery_tune_agg_interval)
- [recovery_tune_sleep_min_us](#recovery_tune_sleep_min_us)
- [recovery_tune_sleep_cutoff_us](#recovery_tune_sleep_cutoff_us)
## etcd_report_interval
- Тип: секунды
- Значение по умолчанию: 5
Интервал, с которым OSD обновляет своё состояние в etcd. Значение параметра
влияет на время резервации (lease) OSD и поэтому на скорость переключения
Интервал, с которым OSD сообщает о том, что жив, в etcd. Значение параметра
влияет на время резервации (lease) OSD и поэтому - на скорость переключения
при падении OSD. Время lease равняется значению этого параметра плюс
max_etcd_attempts * etcd_quick_timeout.
## etcd_stats_interval
- Тип: секунды
- Значение по умолчанию: 30
Интервал, с которым OSD обновляет свою статистику в etcd. Сильно влияет на
создаваемую нагрузку на etcd, потому что статистика содержит по ключу на
каждый OSD и на каждую PG. В то же время низкий интервал делает
статистику, печатаемую `vitastor-cli`, отзывчивей.
## run_primary
- Тип: булево (да/нет)
@ -93,6 +125,7 @@ RUNNING), подходящий под заданную маску. Также н
- Тип: секунды
- Значение по умолчанию: 5
- Можно менять на лету: да
Временной интервал отправки автоматических fsync-ов (операций очистки кэша)
каждым OSD для случая, когда режим immediate_commit отключён. fsync-и нужны
@ -105,6 +138,7 @@ OSD, чтобы успевать очищать журнал - без них OSD
- Тип: целое число
- Значение по умолчанию: 128
- Можно менять на лету: да
Аналогично autosync_interval, но задаёт не временной интервал, а
максимальное количество незафиксированных операций записи перед
@ -113,17 +147,31 @@ OSD, чтобы успевать очищать журнал - без них OSD
## recovery_queue_depth
- Тип: целое число
- Значение по умолчанию: 4
- Значение по умолчанию: 1
- Можно менять на лету: да
Максимальное число операций восстановления на одном первичном OSD в любой
момент времени. На данный момент единственный параметр, который можно менять
для ускорения или замедления восстановления и перебалансировки данных, но
в планах реализация других параметров.
Максимальное число параллельных операций восстановления, инициируемых одним
OSD в любой момент времени. Имейте в виду, что каждый OSD обычно работает с
многими другими OSD, так что на практике параллелизм восстановления больше,
чем просто recovery_queue_depth. Увеличение значения этого параметра может
ускорить восстановление если [автотюнинг скорости](#recovery_tune_interval)
разрешает это или если он отключён.
## recovery_sleep_us
- Тип: микросекунды
- Значение по умолчанию: 0
- Можно менять на лету: да
Delay for all recovery- and rebalance- related operations. If non-zero,
such operations are artificially slowed down to reduce the impact on
client I/O.
## recovery_pg_switch
- Тип: целое число
- Значение по умолчанию: 128
- Можно менять на лету: да
Число операций восстановления перед переключением на восстановление другой PG.
Идея заключается в том, чтобы восстанавливать все PG одновременно для более
@ -135,6 +183,7 @@ OSD, чтобы успевать очищать журнал - без них OSD
- Тип: целое число
- Значение по умолчанию: 16
- Можно менять на лету: да
Максимальное число операций восстановления перед дополнительным fsync.
@ -150,6 +199,7 @@ OSD, чтобы успевать очищать журнал - без них OSD
- Тип: булево (да/нет)
- Значение по умолчанию: false
- Можно менять на лету: да
Отключить автоматическое фоновое восстановление объектов. Обратите внимание,
что эта опция не отключает восстановление объектов, происходящее при
@ -160,6 +210,7 @@ OSD.
- Тип: булево (да/нет)
- Значение по умолчанию: false
- Можно менять на лету: да
Отключить фоновое перемещение объектов между разными OSD. Отключение
означает, что PG, находящиеся в состоянии `has_misplaced`, будут оставлены
@ -169,6 +220,7 @@ OSD.
- Тип: секунды
- Значение по умолчанию: 3
- Можно менять на лету: да
Временной интервал, с которым OSD печатают простую человекочитаемую
статистику выполнения операций в стандартный вывод.
@ -177,6 +229,7 @@ OSD.
- Тип: секунды
- Значение по умолчанию: 10
- Можно менять на лету: да
Временной интервал, с которым OSD выводят в стандартный вывод список
медленных или зависших операций, если таковые имеются. Также время, при
@ -186,6 +239,7 @@ OSD.
- Тип: секунды
- Значение по умолчанию: 60
- Можно менять на лету: да
Число секунд, через которое удалённые инод удаляется и из статистики OSD.
@ -193,6 +247,7 @@ OSD.
- Тип: целое число
- Значение по умолчанию: 128
- Можно менять на лету: да
Максимальное число одновременных клиентских операций записи на один OSD.
Операции, превышающие этот лимит, не исполняются сразу, а сохраняются во
@ -202,6 +257,7 @@ OSD.
- Тип: целое число
- Значение по умолчанию: 1
- Можно менять на лету: да
Flusher - это микро-поток (корутина), которая копирует данные из журнала в
основную область устройства данных. Их число настраивается динамически между
@ -211,6 +267,7 @@ Flusher - это микро-поток (корутина), которая коп
- Тип: целое число
- Значение по умолчанию: 256
- Можно менять на лету: да
Максимальное число микро-потоков очистки журнала (см. выше min_flusher_count).
@ -241,6 +298,63 @@ Flusher - это микро-поток (корутина), которая коп
параметра может оказаться полезным для гибридных OSD (HDD+SSD) с большими
журналами, расположенными на быстром по сравнению с HDD устройстве.
## data_io
- Тип: строка
- Значение по умолчанию: direct
Режим ввода-вывода для *данных*. Одно из значений "direct", "cached" или
"directsync", означающих O_DIRECT, O_SYNC и O_DIRECT|O_SYNC, соответственно.
Выберите "cached", чтобы использовать системный кэш Linux (page cache) при
чтении и записи. Это может улучшить скорость чтения горячих данных с
относительно медленных дисков - HDD и, возможно, SATA SSD - но немного
снижает производительность записи для быстрых дисков, так как кэш сам по
себе тоже добавляет накладные расходы.
Выберите "directsync", если хотите задействовать
[immediate_commit](layout-cluster.ru.md#immediate_commit) (требующий
включенияd disable_data_fsync) на дисках с неотключаемым кэшем. Пример таких
дисков - Intel Optane. При этом также стоит иметь в виду, что *некоторые*
настольные SSD (например, HP EX950) игнорируют флаг O_SYNC, делая отключение
fsync небезопасным даже с режимом "directsync".
## meta_io
- Тип: строка
- Значение по умолчанию: direct
Режим ввода-вывода для *метаданных*. Одно из значений "direct", "cached" или
"directsync".
"cached" может улучшить скорость чтения, если:
1. у вас медленные диски (HDD, SATA SSD)
2. контрольные суммы включены
3. параметр [inmemory_metadata](#inmemory_metadata) отключён.
При этих условиях блоки метаданных читаются с диска при каждом запросе чтения
для проверки контрольных сумм и их кэширование может снизить дополнительную
нагрузку на диск. Без (3) метаданные никогда не читаются с диска после
запуска OSD, а без (2) блоки метаданных читаются только при сбросе журнала.
Если одно и то же устройство используется для данных и метаданных, режим
ввода-вывода метаданных по умолчанию устанавливается равным [data_io](#data_io).
## journal_io
- Тип: строка
- Значение по умолчанию: direct
Режим ввода-вывода для *журнала*. Одно из значений "direct", "cached" или
"directsync".
Здесь "cached" может улучшить скорость чтения только недавно записанных
данных и только если параметр [inmemory_journal](#inmemory_journal)
отключён.
Если одно и то же устройство используется для метаданных и журнала,
режим ввода-вывода журнала по умолчанию устанавливается равным
[meta_io](#meta_io).
## journal_sector_buffer_count
- Тип: целое число
@ -270,6 +384,7 @@ Flusher - это микро-поток (корутина), которая коп
- Тип: булево (да/нет)
- Значение по умолчанию: false
- Можно менять на лету: да
Разрешить мягкое ограничение скорости журналируемой записи. Полезно для
гибридных OSD с быстрыми устройствами метаданных и медленными устройствами
@ -288,6 +403,7 @@ Flusher - это микро-поток (корутина), которая коп
- Тип: целое число
- Значение по умолчанию: 100
- Можно менять на лету: да
Расчётное максимальное число ограничиваемых операций в секунду при условии
отсутствия свободного места в журнале. Устанавливайте приблизительно равным
@ -298,6 +414,7 @@ Flusher - это микро-поток (корутина), которая коп
- Тип: целое число
- Значение по умолчанию: 100
- Можно менять на лету: да
Расчётный максимальный размер в МБ/с ограничиваемых операций в секунду при
условии отсутствия свободного места в журнале. Устанавливайте приблизительно
@ -308,6 +425,7 @@ Flusher - это микро-поток (корутина), которая коп
- Тип: целое число
- Значение по умолчанию: 1
- Можно менять на лету: да
Расчётный максимальный параллелизм ограничиваемых операций в секунду при
условии отсутствия свободного места в журнале. Устанавливайте приблизительно
@ -318,6 +436,7 @@ Flusher - это микро-поток (корутина), которая коп
- Тип: микросекунды
- Значение по умолчанию: 50
- Можно менять на лету: да
Минимальная применимая к ограничиваемым операциям задержка. Обычно не
требует изменений.
@ -327,4 +446,203 @@ Flusher - это микро-поток (корутина), которая коп
- Тип: булево (да/нет)
- Значение по умолчанию: false
Блокировать всю память OSD с помощью mlockall, чтобы запретить её выгрузку в пространство подкачки. Требует достаточного значения ulimit -l (лимита заблокированной памяти).
Блокировать всю память OSD с помощью mlockall, чтобы запретить её выгрузку
в пространство подкачки. Требует достаточного значения ulimit -l (лимита
заблокированной памяти).
## auto_scrub
- Тип: булево (да/нет)
- Значение по умолчанию: false
- Можно менять на лету: да
Скраб - процесс фоновой проверки копий данных, предназначенный, чтобы
находить и исправлять повреждённые блоки. По умолчанию эти проверки ещё не
запускаются автоматически, так как являются новой функцией. Чтобы включить
автоматическое планирование скрабов, установите данный параметр в true.
Включённый параметр заставляет OSD автоматически планировать фоновую
проверку чистых PG раз в `scrub_interval` (см. ниже). Вы также можете
запустить или запланировать проверку вручную, установив значение ключа JSON
`next_scrub` внутри ключей etcd `/pg/history/...` в UNIX-время следующей
желаемой проверки.
## no_scrub
- Тип: булево (да/нет)
- Значение по умолчанию: false
- Можно менять на лету: да
Временно отключить и остановить запущенные скрабы.
## scrub_interval
- Тип: строка
- Значение по умолчанию: 30d
- Можно менять на лету: да
Интервал автоматической фоновой проверки по умолчанию для всех пулов.
Значения без указанной единицы измерения считаются в секундах, допустимые
символы единиц измерения в конце: 's' (секунды),
'm' (минуты), 'h' (часы), 'd' (дни), 'M' (месяца) или 'y' (годы).
## scrub_queue_depth
- Тип: целое число
- Значение по умолчанию: 1
- Можно менять на лету: да
Число параллельных операций фоновой проверки на один OSD.
## scrub_sleep
- Тип: миллисекунды
- Значение по умолчанию: 0
- Можно менять на лету: да
Дополнительный интервал ожидания после фоновой проверки каждого объекта на
одном OSD. Может использоваться для замедления скраба, если он слишком
сильно влияет на пользовательскую нагрузку.
## scrub_list_limit
- Тип: целое число
- Значение по умолчанию: 1000
- Можно менять на лету: да
Размер загружаемых за одну операцию списков объектов в процессе фоновой
проверки.
## scrub_find_best
- Тип: булево (да/нет)
- Значение по умолчанию: true
- Можно менять на лету: да
Находить и автоматически восстанавливать "лучшие версии" объектов с
несовпадающими копиями/частями. При использовании репликации "лучшая"
версия - версия, доступная в большем числе экземпляров, чем другие. При
использовании кодов коррекции ошибок "лучшая" версия - это подмножество
частей данных и чётности, полностью соответствующих друг другу.
Гипотетическая ситуация, в которой вы можете захотеть отключить этот
поиск - это если у вас 3 реплики и вы боитесь, что 2 диска из 3 могут
незаметно и одинаково повредить данные одного и того же объекта, например,
занулив его, и только 1 диск останется неповреждённым. В этой ситуации
отключение этого параметра поможет вам восстановить данные! Смотрите также
описание следующего параметра - scrub_ec_max_bruteforce.
## scrub_ec_max_bruteforce
- Тип: целое число
- Значение по умолчанию: 100
- Можно менять на лету: да
Vitastor старается определить повреждённые части объектов при использовании
EC (кодов коррекции ошибок) с более, чем 1 диском чётности, путём перебора
всех возможных комбинаций ошибочных частей. Данное значение конфигурации
ограничивает число перебираемых комбинаций. Вы можете попробовать поднять
его, если используете схему кодирования EC N+K с N и K, достаточно большими
для того, чтобы число сочетаний `C(N+K-1, K-1) = (N+K-1)! / (K-1)! / N!`
было больше, чем стандартное значение 100.
Если возможных комбинаций слишком много или если корректная комбинаций не
определяется однозначно, объекты помечаются неконсистентными (inconsistent)
и не восстанавливаются автоматически.
При использовании репликации перебор не нужен, Vitastor просто предполагает,
что вариант объекта с наибольшим количеством одинаковых копий корректен.
Например, если вы используете 3 реплики и 1 из них отличается, эта 1 копия
считается некорректной. Однако, если "лучшую" версию с числом доступных
копий большим, чем у всех других версий, найти невозможно, то объект тоже
маркируется неконсистентным.
## recovery_tune_interval
- Тип: секунды
- Значение по умолчанию: 1
- Можно менять на лету: да
Интервал, с которым OSD пересматривает клиентскую нагрузку и нагрузку
восстановления и автоматически подстраивает [recovery_sleep_us](#recovery_sleep_us).
Автотюнинг (автоподстройка) отключается, если recovery_tune_interval
устанавливается в значение 0.
Автотюнинг регулирует утилизацию. Утилизация является мерой нагрузки
и равна произведению числа операций в секунду и средней задержки
(то есть, она может быть выше 1). Вы задаёте два уровня клиентской
утилизации - "низкий" и "высокий" (low и high) и два соответствующих
целевых уровня утилизации операциями восстановления. OSD рассчитывает
желаемый уровень утилизации восстановления линейной интерполяцией от
клиентской утилизации и подстраивает задержку операций восстановления
так, чтобы фактическая утилизация восстановления совпадала с желаемой.
Это позволяет снизить влияние восстановления и ребаланса на клиентские
операции. Конечно, невозможно исключить такое влияние полностью, но оно
должно становиться адекватнее. В некоторых тестах перебалансировка могла
снижать клиентскую скорость записи с 1.5 ГБ/с до 50-100 МБ/с, а теперь, с
настройками автотюнинга по умолчанию, она снижается только до ~1 ГБ/с.
## recovery_tune_util_low
- Тип: число
- Значение по умолчанию: 0.1
- Можно менять на лету: да
Желаемая утилизация восстановления в моменты, когда клиентская нагрузка
высокая, то есть, находится на уровне или выше recovery_tune_client_util_high.
## recovery_tune_util_high
- Тип: число
- Значение по умолчанию: 1
- Можно менять на лету: да
Желаемая утилизация восстановления в моменты, когда клиентская нагрузка
низкая, то есть, находится на уровне или ниже recovery_tune_client_util_low.
## recovery_tune_client_util_low
- Тип: число
- Значение по умолчанию: 0
- Можно менять на лету: да
Клиентская утилизация, которая считается "низкой".
## recovery_tune_client_util_high
- Тип: число
- Значение по умолчанию: 0.5
- Можно менять на лету: да
Клиентская утилизация, которая считается "высокой".
## recovery_tune_agg_interval
- Тип: целое число
- Значение по умолчанию: 10
- Можно менять на лету: да
Число последних итераций автоподстройки для расчёта задержки как среднего
значения. Меньшие значения параметра ускоряют отклик на изменение нагрузки,
большие значения делают задержку стабильнее. Значение по умолчанию 10
обычно нормальное и не требует изменений.
## recovery_tune_sleep_min_us
- Тип: микросекунды
- Значение по умолчанию: 10
- Можно менять на лету: да
Минимальное возможное значение авто-подстроенного recovery_sleep_us.
Меньшие значения заменяются на 0.
## recovery_tune_sleep_cutoff_us
- Тип: микросекунды
- Значение по умолчанию: 10000000
- Можно менять на лету: да
Максимальное возможное значение авто-подстроенного recovery_sleep_us.
Большие значения считаются случайными выбросами и игнорируются в
усреднении.

View File

@ -40,6 +40,8 @@ Parameters:
- [root_node](#root_node)
- [osd_tags](#osd_tags)
- [primary_affinity_tags](#primary_affinity_tags)
- [scrub_interval](#scrub_interval)
- [used_for_fs](#used_for_fs)
Examples:
@ -153,6 +155,26 @@ That is, if it becomes impossible to place PG data on at least (pg_minsize)
OSDs, PG is deactivated for both read and write. So you know that a fresh
write always goes to at least (pg_minsize) OSDs (disks).
For example, the difference between pg_minsize 2 and 1 in a 3-way replicated
pool (pg_size=3) is:
- If 2 hosts go down with pg_minsize=2, the pool becomes inactive and remains
inactive for [osd_out_time](monitor.en.md#osd_out_time) (10 minutes). After
this timeout, the monitor selects replacement hosts/OSDs and the pool comes
up and starts to heal. Therefore, if you don't have replacement OSDs, i.e.
if you only have 3 hosts with OSDs and 2 of them are down, the pool remains
inactive until you add or return at least 1 host (or change failure_domain
to "osd").
- If 2 hosts go down with pg_minsize=1, the pool only experiences a short
I/O pause until the monitor notices that OSDs are down (5-10 seconds with
the default [etcd_report_interval](osd.en.md#etcd_report_interval)). After
this pause, I/O resumes, but new data is temporarily written in only 1 copy.
Then, after osd_out_time, the monitor also selects replacement OSDs and the
pool starts to heal.
So, pg_minsize regulates the number of failures that a pool can tolerate
without temporary downtime for [osd_out_time](monitor.en.md#osd_out_time),
but at a cost of slightly reduced storage reliability.
FIXME: pg_minsize behaviour may be changed in the future to only make PGs
read-only instead of deactivating them.
@ -164,8 +186,8 @@ read-only instead of deactivating them.
Number of PGs for this pool. The value should be big enough for the monitor /
LP solver to be able to optimize data placement.
"Enough" is usually around 64-128 PGs per OSD, i.e. you set pg_count for pool
to (total OSD count * 100 / pg_size). You can round it to the closest power of 2,
"Enough" is usually around 10-100 PGs per OSD, i.e. you set pg_count for pool
to (total OSD count * 10 / pg_size). You can round it to the closest power of 2,
because it makes it easier to reduce or increase PG count later by dividing or
multiplying it by 2.
@ -204,9 +226,8 @@ This parameter usually doesn't require to be changed.
- Default: 131072
Block size for this pool. The value from /vitastor/config/global is used when
unspecified. If your cluster has OSDs with different block sizes then pool must
be restricted by [osd_tags](#osd_tags) to only include OSDs with matching block
size.
unspecified. Only OSDs with matching block_size are used for each pool. If you
want to further restrict OSDs for the pool, use [osd_tags](#osd_tags).
Read more about this parameter in [Cluster-Wide Disk Layout Parameters](layout-cluster.en.md#block_size).
@ -215,10 +236,9 @@ Read more about this parameter in [Cluster-Wide Disk Layout Parameters](layout-c
- Type: integer
- Default: 4096
"Sector" size of virtual disks in this pool. The value from
/vitastor/config/global is used when unspecified. Similar to block_size, the
pool must be restricted by [osd_tags](#osd_tags) to only include OSDs with
matching bitmap_granularity.
"Sector" size of virtual disks in this pool. The value from /vitastor/config/global
is used when unspecified. Similarly to block_size, only OSDs with matching
bitmap_granularity are used for each pool.
Read more about this parameter in [Cluster-Wide Disk Layout Parameters](layout-cluster.en.md#bitmap_granularity).
@ -228,10 +248,11 @@ Read more about this parameter in [Cluster-Wide Disk Layout Parameters](layout-c
- Default: none
Immediate commit setting for this pool. The value from /vitastor/config/global
is used when unspecified. Similar to block_size, the pool must be restricted by
[osd_tags](#osd_tags) to only include OSDs with compatible immediate_commit.
Compatible means that a pool with non-immediate commit will work with OSDs with
immediate commit enabled, but not vice versa.
is used when unspecified. Similarly to block_size, only OSDs with compatible
bitmap_granularity are used for each pool. "Compatible" means that a pool with
non-immediate commit will use OSDs with immediate commit enabled, but not vice
versa. I.e., pools with "none" use all OSDs, pools with "small" only use OSDs
with "all" or "small", and pools with "all" only use OSDs with "all".
Read more about this parameter in [Cluster-Wide Disk Layout Parameters](layout-cluster.en.md#immediate_commit).
@ -272,6 +293,32 @@ Specifies OSD tags to prefer putting primary OSDs in this pool to.
Note that for EC/XOR pools Vitastor always prefers to put primary OSD on one
of the OSDs containing a data chunk for a PG.
## scrub_interval
- Type: time interval (number + unit s/m/h/d/M/y)
Automatic scrubbing interval for this pool. Overrides
[global scrub_interval setting](osd.en.md#scrub_interval).
## used_for_fs
- Type: string
If non-empty, the pool is marked as used for VitastorFS with metadata stored
in block image (regular Vitastor volume) named as the value of this pool parameter.
When a pool is marked as used for VitastorFS, regular block volume creation in it
is disabled (vitastor-cli refuses to create images without --force) to protect
the user from block volume and FS file ID collisions and data loss.
[vitastor-nfs](../usage/nfs.ru.md), in its turn, refuses to use pools not marked
for the corresponding FS when starting. This also implies that you can use one
pool only for one VitastorFS.
The second thing that is disabled for VitastorFS pools is reporting per-inode space
usage statistics in etcd because a FS pool may store a very large number of files
and statistics for them all would take a lot of space in etcd.
# Examples
## Replicated pool

View File

@ -39,6 +39,8 @@
- [root_node](#root_node)
- [osd_tags](#osd_tags)
- [primary_affinity_tags](#primary_affinity_tags)
- [scrub_interval](#scrub_interval)
- [used_for_fs](#used_for_fs)
Примеры:
@ -156,6 +158,26 @@
OSD, PG деактивируется на чтение и запись. Иными словами, всегда известно,
что новые блоки данных всегда записываются как минимум на pg_minsize дисков.
Для примера, разница между pg_minsize 2 и 1 в реплицированном пуле с 3 копиями
данных (pg_size=3), проявляется следующим образом:
- Если 2 сервера отключаются при pg_minsize=2, пул становится неактивным и
остаётся неактивным в течение [osd_out_time](monitor.en.md#osd_out_time)
(10 минут), после чего монитор назначает другие OSD/серверы на замену, пул
поднимается и начинает восстанавливать недостающие копии данных. Соответственно,
если OSD на замену нет - то есть, если у вас всего 3 сервера с OSD и 2 из них
недоступны - пул так и остаётся недоступным до тех пор, пока вы не вернёте
или не добавите хотя бы 1 сервер (или не переключите failure_domain на "osd").
- Если 2 сервера отключаются при pg_minsize=1, ввод-вывод лишь приостанавливается
на короткое время, до тех пор, пока монитор не поймёт, что OSD отключены
(что занимает 5-10 секунд при стандартном [etcd_report_interval](osd.en.md#etcd_report_interval)).
После этого ввод-вывод восстанавливается, но новые данные временно пишутся
всего в 1 копии. Когда же проходит osd_out_time, монитор точно так же назначает
другие OSD на замену выбывшим и пул начинает восстанавливать копии данных.
То есть, pg_minsize регулирует число отказов, которые пул может пережить без
временной остановки обслуживания на [osd_out_time](monitor.ru.md#osd_out_time),
но ценой немного пониженных гарантий надёжности.
FIXME: Поведение pg_minsize может быть изменено в будущем с полной деактивации
PG на перевод их в режим только для чтения.
@ -167,8 +189,8 @@ PG на перевод их в режим только для чтения.
Число PG для данного пула. Число должно быть достаточно большим, чтобы монитор
мог равномерно распределить по ним данные.
Обычно это означает примерно 64-128 PG на 1 OSD, т.е. pg_count можно устанавливать
равным (общему числу OSD * 100 / pg_size). Значение можно округлить до ближайшей
Обычно это означает примерно 10-100 PG на 1 OSD, т.е. pg_count можно устанавливать
равным (общему числу OSD * 10 / pg_size). Значение можно округлить до ближайшей
степени 2, чтобы потом было легче уменьшать или увеличивать число PG, умножая
или деля его на 2.
@ -207,8 +229,9 @@ PG в Vitastor эферемерны, то есть вы можете менят
Размер блока для данного пула. Если не задан, используется значение из
/vitastor/config/global. Если в вашем кластере есть OSD с разными размерами
блока, пул должен быть ограничен только OSD, блок которых равен блоку пула,
с помощью [osd_tags](#osd_tags).
блока, пул будет использовать только OSD с размером блока, равным размеру блока
пула. Если вы хотите сильнее ограничить набор используемых для пула OSD -
используйте [osd_tags](#osd_tags).
О самом параметре читайте в разделе [Дисковые параметры уровня кластера](layout-cluster.ru.md#block_size).
@ -218,9 +241,8 @@ PG в Vitastor эферемерны, то есть вы можете менят
- По умолчанию: 4096
Размер "сектора" виртуальных дисков в данном пуле. Если не задан, используется
значение из /vitastor/config/global. Аналогично block_size, пул должен быть
ограничен OSD со значением bitmap_granularity, равным значению пула, с помощью
[osd_tags](#osd_tags).
значение из /vitastor/config/global. Аналогично block_size, каждый пул будет
использовать только OSD с совпадающей с пулом настройкой bitmap_granularity.
О самом параметре читайте в разделе [Дисковые параметры уровня кластера](layout-cluster.ru.md#bitmap_granularity).
@ -230,11 +252,13 @@ PG в Vitastor эферемерны, то есть вы можете менят
- По умолчанию: none
Настройка мгновенного коммита для данного пула. Если не задана, используется
значение из /vitastor/config/global. Аналогично block_size, пул должен быть
ограничен OSD со значением bitmap_granularity, совместимым со значением пула, с
помощью [osd_tags](#osd_tags). Совместимость означает, что пул с отключенным
мгновенным коммитом может работать на OSD с включённым мгновенным коммитом, но
не наоборот.
значение из /vitastor/config/global. Аналогично block_size, каждый пул будет
использовать только OSD с *совместимыми* настройками immediate_commit.
"Совместимыми" означает, что пул с отключенным мгновенным коммитом будет
использовать OSD с включённым мгновенным коммитом, но не наоборот. То есть,
пул со значением "none" будет использовать все OSD, пул со "small" будет
использовать OSD с "all" или "small", а пул с "all" будет использовать только
OSD с "all".
О самом параметре читайте в разделе [Дисковые параметры уровня кластера](layout-cluster.ru.md#immediate_commit).
@ -276,6 +300,34 @@ PG в Vitastor эферемерны, то есть вы можете менят
для PG этого пула. Имейте в виду, что для EC-пулов Vitastor также всегда
предпочитает помещать первичный OSD на один из OSD с данными, а не с чётностью.
## scrub_interval
- Тип: временной интервал (число + единица измерения s/m/h/d/M/y)
Интервал скраба, то есть, автоматической фоновой проверки данных для данного пула.
Переопределяет [глобальную настройку scrub_interval](osd.ru.md#scrub_interval).
## used_for_fs
- Type: string
Если непусто, пул помечается как используемый для файловой системы VitastorFS с
метаданными, хранимыми в блочном образе Vitastor с именем, равным значению
этого параметра.
Когда пул помечается как используемый для VitastorFS, создание обычных блочных
образов в нём отключается (vitastor-cli отказывается создавать образы без --force),
чтобы защитить пользователя от коллизий ID файлов и блочных образов и, таким
образом, от потери данных.
[vitastor-nfs](../usage/nfs.ru.md), в свою очередь, при запуске отказывается
использовать для ФС пулы, не выделенные для неё. Это также означает, что один
пул может использоваться только для одной VitastorFS.
Также для ФС-пулов отключается передача статистики в etcd по отдельным инодам,
так как ФС-пул может содержать очень много файлов и статистика по ним всем
заняла бы очень много места в etcd.
# Примеры
## Реплицированный пул

View File

@ -0,0 +1,4 @@
# Client Parameters
These parameters apply only to Vitastor clients (QEMU, fio, NBD and so on) and
affect their interaction with the cluster.

View File

@ -0,0 +1,4 @@
# Параметры клиентского кода
Данные параметры применяются только к клиентам Vitastor (QEMU, fio, NBD и т.п.) и
затрагивают логику их работы с кластером.

192
docs/config/src/client.yml Normal file
View File

@ -0,0 +1,192 @@
- name: client_retry_interval
type: ms
min: 10
default: 50
online: true
info: |
Retry time for I/O requests failed due to inactive PGs or network
connectivity errors.
info_ru: |
Время повтора запросов ввода-вывода, неудачных из-за неактивных PG или
ошибок сети.
- name: client_eio_retry_interval
type: ms
default: 1000
online: true
info: |
Retry time for I/O requests failed due to data corruption or unfinished
EC object deletions (has_incomplete PG state). 0 disables such retries
and clients are not blocked and just get EIO error code instead.
info_ru: |
Время повтора запросов ввода-вывода, неудачных из-за повреждения данных
или незавершённых удалений EC-объектов (состояния PG has_incomplete).
0 отключает повторы таких запросов и клиенты не блокируются, а вместо
этого просто получают код ошибки EIO.
- name: client_max_dirty_bytes
type: int
default: 33554432
online: true
info: |
Without [immediate_commit](layout-cluster.en.md#immediate_commit)=all this parameter sets the limit of "dirty"
(not committed by fsync) data allowed by the client before forcing an
additional fsync and committing the data. Also note that the client always
holds a copy of uncommitted data in memory so this setting also affects
RAM usage of clients.
info_ru: |
При работе без [immediate_commit](layout-cluster.ru.md#immediate_commit)=all - это лимит объёма "грязных" (не
зафиксированных fsync-ом) данных, при достижении которого клиент будет
принудительно вызывать fsync и фиксировать данные. Также стоит иметь в виду,
что в этом случае до момента fsync клиент хранит копию незафиксированных
данных в памяти, то есть, настройка влияет на потребление памяти клиентами.
- name: client_max_dirty_ops
type: int
default: 1024
online: true
info: |
Same as client_max_dirty_bytes, but instead of total size, limits the number
of uncommitted write operations.
info_ru: |
Аналогично client_max_dirty_bytes, но ограничивает количество
незафиксированных операций записи вместо их общего объёма.
- name: client_enable_writeback
type: bool
default: false
online: true
info: |
This parameter enables client-side write buffering. This means that write
requests are accumulated in memory for a short time before being sent to
a Vitastor cluster which allows to send them in parallel and increase
performance of some applications. Writes are buffered until client forces
a flush with fsync() or until the amount of buffered writes exceeds the
limit.
Write buffering significantly increases performance of some applications,
for example, CrystalDiskMark under Windows (LOL :-D), but also any other
applications if they do writes in one of two non-optimal ways: either if
they do a lot of small (4 kb or so) sequential writes, or if they do a lot
of small random writes, but without any parallelism or asynchrony, and also
without calling fsync().
With write buffering enabled, you can expect around 22000 T1Q1 random write
iops in QEMU more or less regardless of the quality of your SSDs, and this
number is in fact bound by QEMU itself rather than Vitastor (check it
yourself by adding a "driver=null-co" disk in QEMU). Without write
buffering, the current record is 9900 iops, but the number is usually
even lower with non-ideal hardware, for example, it may be 5000 iops.
Even when this parameter is enabled, write buffering isn't enabled until
the client explicitly allows it, because enabling it without the client
being aware of the fact that his writes may be buffered may lead to data
loss. Because of this, older versions of clients don't support write
buffering at all, newer versions of the QEMU driver allow write buffering
only if it's enabled in disk settings with `-blockdev cache.direct=false`,
and newer versions of FIO only allow write buffering if you don't specify
`-direct=1`. NBD and NFS drivers allow write buffering by default.
You can overcome this restriction too with the `client_writeback_allowed`
parameter, but you shouldn't do that unless you **really** know what you
are doing.
info_ru: |
Данный параметр разрешает включать буферизацию записи в памяти. Буферизация
означает, что операции записи отправляются на кластер Vitastor не сразу, а
могут небольшое время накапливаться в памяти и сбрасываться сразу пакетами,
до тех пор, пока либо не будет превышен лимит неотправленных записей, либо
пока клиент не вызовет fsync.
Буферизация значительно повышает производительность некоторых приложений,
например, CrystalDiskMark в Windows (ха-ха :-D), но также и любых других,
которые пишут на диск неоптимально: либо последовательно, но мелкими блоками
(например, по 4 кб), либо случайно, но без параллелизма и без fsync - то
есть, например, отправляя 128 операций записи в разные места диска, но не
все сразу с помощью асинхронного I/O, а по одной.
В QEMU с буферизацией записи можно ожидать показателя примерно 22000
операций случайной записи в секунду в 1 поток и с глубиной очереди 1 (T1Q1)
без fsync, почти вне зависимости от того, насколько хороши ваши диски - эта
цифра упирается в сам QEMU. Без буферизации рекорд пока что - 9900 операций
в секунду, но на железе похуже может быть и поменьше, например, 5000 операций
в секунду.
При этом, даже если данный параметр включён, буферизация не включается, если
явно не разрешена клиентом, т.к. если клиент не знает, что запросы записи
буферизуются, это может приводить к потере данных. Поэтому в старых версиях
клиентских драйверов буферизация записи не включается вообще, в новых
версиях QEMU-драйвера включается, только если разрешена опцией диска
`-blockdev cache.direct=false`, а в fio - только если нет опции `-direct=1`.
В NBD и NFS драйверах буферизация записи разрешена по умолчанию.
Можно обойти и это ограничение с помощью параметра `client_writeback_allowed`,
но делать так не надо, если только вы не уверены в том, что делаете, на все
100%. :-)
- name: client_max_buffered_bytes
type: int
default: 33554432
online: true
info: |
Maximum total size of buffered writes which triggers write-back when reached.
info_ru: |
Максимальный общий размер буферизованных записей, при достижении которого
начинается процесс сброса данных на сервер.
- name: client_max_buffered_ops
type: int
default: 1024
online: true
info: |
Maximum number of buffered writes which triggers write-back when reached.
Multiple consecutive modified data regions are counted as 1 write here.
info_ru: |
Максимальное количество буферизованных записей, при достижении которого
начинается процесс сброса данных на сервер. При этом несколько
последовательных изменённых областей здесь считаются 1 записью.
- name: client_max_writeback_iodepth
type: int
default: 256
online: true
info: |
Maximum number of parallel writes when flushing buffered data to the server.
info_ru: |
Максимальное число параллельных операций записи при сбросе буферов на сервер.
- name: nbd_timeout
type: sec
default: 300
online: false
info: |
Timeout for I/O operations for [NBD](../usage/nbd.en.md). If an operation
executes for longer than this timeout, including when your cluster is just
temporarily down for more than timeout, the NBD device will detach by itself
(and possibly break the mounted file system).
You can set timeout to 0 to never detach, but in that case you won't be
able to remove the kernel device at all if the NBD process dies - you'll have
to reboot the host.
info_ru: |
Таймаут для операций чтения/записи через [NBD](../usage/nbd.ru.md). Если
операция выполняется дольше таймаута, включая временную недоступность
кластера на время, большее таймаута, NBD-устройство отключится само собой
(и, возможно, сломает примонтированную ФС).
Вы можете установить таймаут в 0, чтобы никогда не отключать устройство по
таймауту, но в этом случае вы вообще не сможете удалить устройство, если
процесс NBD умрёт - вам придётся перезагружать сервер.
- name: nbd_max_devices
type: int
default: 64
online: false
info: |
Maximum number of NBD devices in the system. This value is passed as
`nbds_max` parameter for the nbd kernel module when vitastor-nbd autoloads it.
info_ru: |
Максимальное число NBD-устройств в системе. Данное значение передаётся
модулю ядра nbd как параметр `nbds_max`, когда его загружает vitastor-nbd.
- name: nbd_max_part
type: int
default: 3
online: false
info: |
Maximum number of partitions per NBD device. This value is passed as
`max_part` parameter for the nbd kernel module when vitastor-nbd autoloads it.
Note that (nbds_max)*(1+max_part) usually can't exceed 256.
info_ru: |
Максимальное число разделов на одном NBD-устройстве. Данное значение передаётся
модулю ядра nbd как параметр `max_part`, когда его загружает vitastor-nbd.
Имейте в виду, что (nbds_max)*(1+max_part) обычно не может превышать 256.

View File

@ -11,13 +11,21 @@
- name: etcd_address
type: string or array of strings
type_ru: строка или массив строк
online: true
info: |
etcd connection endpoint(s). Multiple endpoints may be delimited by "," or
specified in a JSON array `["10.0.115.10:2379/v3","10.0.115.11:2379/v3"]`.
Note that https is not supported for etcd connections yet.
etcd connection endpoints can be changed online by updating global
configuration in etcd itself - this allows to switch the cluster to new
etcd addresses without downtime.
info_ru: |
Адрес(а) подключения к etcd. Несколько адресов могут разделяться запятой
или указываться в виде JSON-массива `["10.0.115.10:2379/v3","10.0.115.11:2379/v3"]`.
Адреса подключения к etcd можно поменять на лету, обновив конфигурацию в
самом etcd - это позволяет переключить кластер на новые etcd без остановки.
- name: etcd_prefix
type: string
default: "/vitastor"
@ -31,5 +39,6 @@
- name: log_level
type: int
default: 0
online: true
info: Log level. Raise if you want more verbose output.
info_ru: Уровень логгирования. Повысьте, если хотите более подробный вывод.

145
docs/config/src/include.js Executable file
View File

@ -0,0 +1,145 @@
#!/usr/bin/nodejs
const fsp = require('fs').promises;
run(process.argv).catch(console.error);
async function run(argv)
{
if (argv.length < 3)
{
console.log('Markdown preprocessor\nUSAGE: ./include.js file.md');
return;
}
const index_file = await fsp.realpath(argv[2]);
const re = /(\{\{[\s\S]*?\}\}|\[[^\]]+\]\([^\)]+\)|(?:^|\n)#[^\n]+)/;
let text = await fsp.readFile(index_file, { encoding: 'utf-8' });
text = text.split(re);
let included = {};
let heading = 0, heading_name = '', m;
for (let i = 0; i < text.length; i++)
{
if (text[i].substr(0, 2) == '{{')
{
// Inclusion
let incfile = text[i].substr(2, text[i].length-4);
let section = null;
let indent = heading;
incfile = incfile.replace(/\s*\|\s*indent\s*=\s*(-?\d+)\s*$/, (m, m1) => { indent = parseInt(m1); return ''; });
incfile = incfile.replace(/\s*#\s*([^#]+)$/, (m, m1) => { section = m1; return ''; });
let inc_heading = section;
incfile = rel2abs(index_file, incfile);
let inc = await fsp.readFile(incfile, { encoding: 'utf-8' });
inc = inc.trim().replace(/^[\s\S]+?\n#/, '#'); // remove until the first header
inc = inc.split(re);
const indent_str = new Array(indent+1).join('#');
let section_start = -1, section_end = -1;
for (let j = 0; j < inc.length; j++)
{
if ((m = /^(\n?)(#+\s*)([\s\S]+)$/.exec(inc[j])))
{
if (!inc_heading)
{
inc_heading = m[3].trim();
}
if (section)
{
if (m[3].trim() == section)
section_start = j;
else if (section_start >= 0)
{
section_end = j;
break;
}
}
inc[j] = m[1] + indent_str + m[2] + m[3];
}
else if ((m = /^(\[[^\]]+\]\()([^\)]+)(\))$/.exec(inc[j])) && !/^https?:(\/\/)|^#/.exec(m[2]))
{
const abs_m2 = rel2abs(incfile, m[2]);
const rel_m = abs2rel(__filename, abs_m2);
if (rel_m.substr(0, 9) == '../../../') // outside docs
inc[j] = m[1] + 'https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/'+rel2abs('docs/config/src/include.js', rel_m) + m[3];
else
inc[j] = m[1] + abs_m2 + m[3];
}
}
if (section)
{
inc = section_start >= 0 ? inc.slice(section_start, section_end < 0 ? inc.length : section_end) : [];
}
if (inc.length)
{
if (!inc_heading)
inc_heading = heading_name||'';
included[incfile+(section ? '#'+section : '')] = '#'+inc_heading.toLowerCase().replace(/\P{L}+/ug, '-').replace(/^-|-$/g, '');
inc[0] = inc[0].replace(/^\s+/, '');
inc[inc.length-1] = inc[inc.length-1].replace(/\s+$/, '');
}
text.splice(i, 1, ...inc);
i = i + inc.length - 1;
}
else if ((m = /^\n?(#+)\s*([\s\S]+)$/.exec(text[i])))
{
// Heading
heading = m[1].length;
heading_name = m[2].trim();
}
}
for (let i = 0; i < text.length; i++)
{
if ((m = /^(\[[^\]]+\]\()([^\)]+)(\))$/.exec(text[i])) && !/^https?:(\/\/)|^#/.exec(m[2]))
{
const p = m[2].indexOf('#');
if (included[m[2]])
{
text[i] = m[1]+included[m[2]]+m[3];
}
else if (p >= 0 && included[m[2].substr(0, p)])
{
text[i] = m[1]+m[2].substr(p)+m[3];
}
}
}
console.log(text.join(''));
}
function rel2abs(ref, rel)
{
rel = [ ...ref.replace(/^(.*)\/[^\/]+$/, '$1').split(/\/+/), ...rel.split(/\/+/) ];
return killdots(rel).join('/');
}
function abs2rel(ref, abs)
{
ref = ref.split(/\/+/);
abs = abs.split(/\/+/);
while (ref.length > 1 && ref[0] == abs[0])
{
ref.shift();
abs.shift();
}
for (let i = 1; i < ref.length; i++)
{
abs.unshift('..');
}
return killdots(abs).join('/');
}
function killdots(rel)
{
for (let i = 0; i < rel.length; i++)
{
if (rel[i] == '.')
{
rel.splice(i, 1);
i--;
}
else if (i >= 1 && rel[i] == '..' && rel[i-1] != '..')
{
rel.splice(i-1, 2);
i -= 2;
}
}
return rel;
}

View File

@ -0,0 +1,69 @@
# Vitastor
{{../../../README.md#The Idea}}
{{../../../README.md#Talks and presentations}}
{{../../intro/features.en.md}}
{{../../intro/quickstart.en.md}}
{{../../intro/architecture.en.md}}
## Installation
{{../../installation/packages.en.md}}
{{../../installation/proxmox.en.md}}
{{../../installation/openstack.en.md}}
{{../../installation/kubernetes.en.md}}
{{../../installation/source.en.md}}
{{../../config.en.md|indent=1}}
{{../../config/common.en.md|indent=2}}
{{../../config/network.en.md|indent=2}}
{{../../config/client.en.md|indent=2}}
{{../../config/layout-cluster.en.md|indent=2}}
{{../../config/layout-osd.en.md|indent=2}}
{{../../config/osd.en.md|indent=2}}
{{../../config/monitor.en.md|indent=2}}
{{../../config/pool.en.md|indent=2}}
{{../../config/inode.en.md|indent=2}}
## Usage
{{../../usage/cli.en.md}}
{{../../usage/disk.en.md}}
{{../../usage/fio.en.md}}
{{../../usage/nbd.en.md}}
{{../../usage/qemu.en.md}}
{{../../usage/nfs.en.md}}
## Performance
{{../../performance/understanding.en.md}}
{{../../performance/theoretical.en.md}}
{{../../performance/comparison1.en.md}}
{{../../performance/bench2.en.md}}
{{../../intro/author.en.md|indent=1}}

View File

@ -0,0 +1,69 @@
# Vitastor
{{../../../README-ru.md#Идея|indent=0}}
{{../../../README-ru.md#Презентации и записи докладов|indent=0}}
{{../../intro/features.ru.md}}
{{../../intro/quickstart.ru.md}}
{{../../intro/architecture.ru.md}}
## Установка
{{../../installation/packages.ru.md}}
{{../../installation/proxmox.ru.md}}
{{../../installation/openstack.ru.md}}
{{../../installation/kubernetes.ru.md}}
{{../../installation/source.ru.md}}
{{../../config.ru.md|indent=1}}
{{../../config/common.ru.md|indent=2}}
{{../../config/network.ru.md|indent=2}}
{{../../config/client.ru.md|indent=2}}
{{../../config/layout-cluster.ru.md|indent=2}}
{{../../config/layout-osd.ru.md|indent=2}}
{{../../config/osd.ru.md|indent=2}}
{{../../config/monitor.ru.md|indent=2}}
{{../../config/pool.ru.md|indent=2}}
{{../../config/inode.ru.md|indent=2}}
## Использование
{{../../usage/cli.ru.md}}
{{../../usage/disk.ru.md}}
{{../../usage/fio.ru.md}}
{{../../usage/nbd.ru.md}}
{{../../usage/qemu.ru.md}}
{{../../usage/nfs.ru.md}}
## Производительность
{{../../performance/understanding.ru.md}}
{{../../performance/theoretical.ru.md}}
{{../../performance/comparison1.ru.md}}
{{../../performance/bench2.ru.md}}
{{../../intro/author.ru.md|indent=1}}

View File

@ -7,26 +7,27 @@
in Vitastor, affects memory usage, write amplification and I/O load
distribution effectiveness.
Recommended default block size is 128 KB for SSD and 4 MB for HDD. In fact,
it's possible to use 4 MB for SSD too - it will lower memory usage, but
Recommended default block size is 128 KB for SSD and 1 MB for HDD. In fact,
it's possible to use 1 MB for SSD too - it will lower memory usage, but
may increase average WA and reduce linear performance.
OSD memory usage is roughly (SIZE / BLOCK * 68 bytes) which is roughly
544 MB per 1 TB of used disk space with the default 128 KB block size.
With 1 MB it's 8 times lower.
info_ru: |
Размер объектов (блоков данных), на которые делятся физические и виртуальные
диски в Vitastor (в рамках каждого пула). Одна из ключевых на данный момент
настроек, влияет на потребление памяти, объём избыточной записи (write
amplification) и эффективность распределения нагрузки по OSD.
Рекомендуемые по умолчанию размеры блока - 128 килобайт для SSD и 4
мегабайта для HDD. В принципе, для SSD можно тоже использовать 4 мегабайта,
Рекомендуемые по умолчанию размеры блока - 128 килобайт для SSD и 1 мегабайт
для HDD. В принципе, для SSD можно тоже использовать блок размером 1 мегабайт,
это понизит использование памяти, но ухудшит распределение нагрузки и в
среднем увеличит WA.
Потребление памяти OSD составляет примерно (РАЗМЕР / БЛОК * 68 байт),
т.е. примерно 544 МБ памяти на 1 ТБ занятого места на диске при
стандартном 128 КБ блоке.
стандартном 128 КБ блоке. При 1 МБ блоке памяти нужно в 8 раз меньше.
- name: bitmap_granularity
type: int
default: 4096
@ -86,8 +87,9 @@
it (they have internal SSD cache even though it's not stated in datasheets).
Setting this parameter to "all" or "small" in OSD parameters requires enabling
disable_journal_fsync and disable_meta_fsync, setting it to "all" also requires
enabling disable_data_fsync.
[disable_journal_fsync](layout-osd.en.yml#disable_journal_fsync) and
[disable_meta_fsync](layout-osd.en.yml#disable_meta_fsync), setting it to
"all" also requires enabling [disable_data_fsync](layout-osd.en.yml#disable_data_fsync).
TLDR: For optimal performance, set immediate_commit to "all" if you only use
SSDs with supercapacitor-based power loss protection (nonvolatile
@ -139,8 +141,9 @@
указано в спецификациях).
Указание "all" или "small" в настройках / командной строке OSD требует
включения disable_journal_fsync и disable_meta_fsync, значение "all" также
требует включения disable_data_fsync.
включения [disable_journal_fsync](layout-osd.ru.yml#disable_journal_fsync) и
[disable_meta_fsync](layout-osd.ru.yml#disable_meta_fsync), значение "all"
также требует включения [disable_data_fsync](layout-osd.ru.yml#disable_data_fsync).
Итого, вкратце: для оптимальной производительности установите
immediate_commit в значение "all", если вы используете в кластере только SSD

View File

@ -204,3 +204,73 @@
Клиентам не обязательно знать про disk_alignment, так что помещать значение
этого параметра в etcd в /vitastor/config/global не нужно.
- name: data_csum_type
type: string
default: none
info: |
Data checksum type to use. May be "crc32c" or "none". Set to "crc32c" to
enable data checksums.
info_ru: |
Тип используемых OSD контрольных сумм данных. Может быть "crc32c" или "none".
Установите в "crc32c", чтобы включить расчёт и проверку контрольных сумм данных.
Следует понимать, что контрольные суммы в зависимости от размера блока их
расчёта либо увеличивают потребление памяти, либо снижают производительность.
Подробнее смотрите в описании параметра [csum_block_size](#csum_block_size).
- name: csum_block_size
type: int
default: 4096
info: |
Checksum calculation block size.
Must be equal or a multiple of [bitmap_granularity](layout-cluster.en.md#bitmap_granularity)
(which is usually 4 KB).
Checksums increase metadata size by 4 bytes per each csum_block_size of data.
Checksums are always a tradeoff:
1. You either sacrifice +1 GB RAM per 1 TB of data
2. Or you raise csum_block_size, for example, to 32k and sacrifice
50% random write iops due to checksum read-modify-write
3. Or you turn off [inmemory_metadata](osd.en.md#inmemory_metadata) and
sacrifice 50% random read iops due to checksum reads
All-flash clusters usually have enough RAM to use default csum_block_size,
which uses 1 GB RAM per 1 TB of data. HDD clusters usually don't.
Thus, recommended setups are:
1. All-flash, 1 GB RAM per 1 TB data: default (csum_block_size=4k)
2. All-flash, less RAM: csum_block_size=4k + inmemory_metadata=false
3. Hybrid HDD+SSD: csum_block_size=4k + inmemory_metadata=false
4. HDD-only, faster random read: csum_block_size=32k
5. HDD-only, faster random write: csum_block_size=4k +
inmemory_metadata=false + meta_io=cached
See also [meta_io](osd.en.md#meta_io).
info_ru: |
Размер блока расчёта контрольных сумм.
Должен быть равен или кратен [bitmap_granularity](layout-cluster.ru.md#bitmap_granularity)
(который обычно равен 4 КБ).
Контрольные суммы увеличивают размер метаданных на 4 байта на каждые
csum_block_size данных.
Контрольные суммы - это всегда компромисс:
1. Вы либо жертвуете потреблением +1 ГБ памяти на 1 ТБ дискового пространства
2. Либо вы повышаете csum_block_size до, скажем, 32k и жертвуете 50%
скорости случайной записи из-за цикла чтения-изменения-записи для расчёта
новых контрольных сумм
3. Либо вы отключаете [inmemory_metadata](osd.ru.md#inmemory_metadata) и
жертвуете 50% скорости случайного чтения из-за чтения контрольных сумм
с диска
Таким образом, рекомендуются следующие варианты настроек:
1. All-flash, 1 ГБ памяти на 1 ТБ данных: по умолчанию (csum_block_size=4k)
2. All-flash, меньше памяти: csum_block_size=4k + inmemory_metadata=false
3. Гибридные HDD+SSD: csum_block_size=4k + inmemory_metadata=false
4. Только HDD, быстрее случайное чтение: csum_block_size=32k
5. Только HDD, быстрее случайная запись: csum_block_size=4k +
inmemory_metadata=false + meta_io=cached
Смотрите также [meta_io](osd.ru.md#meta_io).

View File

@ -14,6 +14,7 @@ const L = {
toc_config: '[Configuration](../config.en.md)',
toc_usage: 'Usage',
toc_performance: 'Performance',
online: 'Can be changed online: yes',
},
ru: {
Documentation: 'Документация',
@ -28,6 +29,7 @@ const L = {
toc_config: '[Конфигурация](../config.ru.md)',
toc_usage: 'Использование',
toc_performance: 'Производительность',
online: 'Можно менять на лету: да',
},
};
const types = {
@ -36,6 +38,7 @@ const types = {
bool: 'boolean',
int: 'integer',
sec: 'seconds',
float: 'number',
ms: 'milliseconds',
us: 'microseconds',
},
@ -44,6 +47,7 @@ const types = {
bool: 'булево (да/нет)',
int: 'целое число',
sec: 'секунды',
float: 'число',
ms: 'миллисекунды',
us: 'микросекунды',
},
@ -70,6 +74,8 @@ for (const file of params_files)
out += `- ${L[lang]['Default'] || 'Default'}: ${c.default}\n`;
if (c.min !== undefined)
out += `- ${L[lang]['Minimum'] || 'Minimum'}: ${c.min}\n`;
if (c.online)
out += `- ${L[lang]['online'] || 'Can be changed online: yes'}\n`;
out += `\n`+(c["info_"+lang] || c["info"]).replace(/\s+$/, '');
}
const head = fs.readFileSync(__dirname+'/'+file+'.'+lang+'.md', { encoding: 'utf-8' });

View File

@ -1,7 +1,7 @@
- name: etcd_mon_ttl
type: sec
min: 10
default: 30
min: 5
default: 1
info: Monitor etcd lease refresh interval in seconds
info_ru: Интервал обновления etcd резервации (lease) монитором
- name: etcd_mon_timeout

View File

@ -48,19 +48,38 @@
type: string
info: |
RDMA device name to use for Vitastor OSD communications (for example,
"rocep5s0f0"). Please note that Vitastor RDMA requires Implicit On-Demand
Paging (Implicit ODP) and Scatter/Gather (SG) support from the RDMA device
to work. For example, Mellanox ConnectX-3 and older adapters don't have
Implicit ODP, so they're unsupported by Vitastor. Run `ibv_devinfo -v` as
root to list available RDMA devices and their features.
"rocep5s0f0"). Now Vitastor supports all adapters, even ones without
ODP support, like Mellanox ConnectX-3 and non-Mellanox cards.
Versions up to Vitastor 1.2.0 required ODP which is only present in
Mellanox ConnectX >= 4. See also [rdma_odp](#rdma_odp).
Run `ibv_devinfo -v` as root to list available RDMA devices and their
features.
Remember that you also have to configure your network switches if you use
RoCE/RoCEv2, otherwise you may experience unstable performance. Refer to
the manual of your network vendor for details about setting up the switch
for RoCEv2 correctly. Usually it means setting up Lossless Ethernet with
PFC (Priority Flow Control) and ECN (Explicit Congestion Notification).
info_ru: |
Название RDMA-устройства для связи с Vitastor OSD (например, "rocep5s0f0").
Имейте в виду, что поддержка RDMA в Vitastor требует функций устройства
Implicit On-Demand Paging (Implicit ODP) и Scatter/Gather (SG). Например,
адаптеры Mellanox ConnectX-3 и более старые не поддерживают Implicit ODP и
потому не поддерживаются в Vitastor. Запустите `ibv_devinfo -v` от имени
суперпользователя, чтобы посмотреть список доступных RDMA-устройств, их
параметры и возможности.
Сейчас Vitastor поддерживает все модели адаптеров, включая те, у которых
нет поддержки ODP, то есть вы можете использовать RDMA с ConnectX-3 и
картами производства не Mellanox.
Версии Vitastor до 1.2.0 включительно требовали ODP, который есть только
на Mellanox ConnectX 4 и более новых. См. также [rdma_odp](#rdma_odp).
Запустите `ibv_devinfo -v` от имени суперпользователя, чтобы посмотреть
список доступных RDMA-устройств, их параметры и возможности.
Обратите внимание, что если вы используете RoCE/RoCEv2, вам также необходимо
правильно настроить для него коммутаторы, иначе вы можете столкнуться с
нестабильной производительностью. Подробную информацию о настройке
коммутатора для RoCEv2 ищите в документации производителя. Обычно это
подразумевает настройку сети без потерь на основе PFC (Priority Flow
Control) и ECN (Explicit Congestion Notification).
- name: rdma_port_num
type: int
default: 1
@ -114,42 +133,97 @@
так что менять этот параметр обычно не нужно.
- name: rdma_max_msg
type: int
default: 1048576
default: 132096
info: Maximum size of a single RDMA send or receive operation in bytes.
info_ru: Максимальный размер одной RDMA-операции отправки или приёма.
- name: rdma_max_recv
type: int
default: 16
info: |
Maximum number of RDMA receive buffers per connection (RDMA requires
preallocated buffers to receive data). Each buffer is `rdma_max_msg` bytes
in size. So this setting directly affects memory usage: a single Vitastor
RDMA client uses `rdma_max_recv * rdma_max_msg * OSD_COUNT` bytes of memory.
Default is roughly 2 MB * number of OSDs.
info_ru: |
Максимальное число буферов для RDMA-приёма данных на одно соединение
(RDMA требует заранее выделенных буферов для приёма данных). Каждый буфер
имеет размер `rdma_max_msg` байт. Таким образом, настройка прямо влияет на
потребление памяти - один Vitastor-клиент с RDMA использует
`rdma_max_recv * rdma_max_msg * ЧИСЛО_OSD` байт памяти, по умолчанию -
примерно 2 МБ * число OSD.
- name: rdma_max_send
type: int
default: 8
info: |
Maximum number of parallel RDMA receive operations. Note that this number
of receive buffers `rdma_max_msg` in size are allocated for each client,
so this setting actually affects memory usage. This is because RDMA receive
operations are (sadly) still not zero-copy in Vitastor. It may be fixed in
later versions.
Maximum number of outstanding RDMA send operations per connection. Should be
less than `rdma_max_recv` so the receiving side doesn't run out of buffers.
Doesn't affect memory usage - additional memory isn't allocated for send
operations.
info_ru: |
Максимальное число параллельных RDMA-операций получения данных. Следует
иметь в виду, что данное число буферов размером `rdma_max_msg` выделяется
для каждого подключённого клиентского соединения, так что данная настройка
влияет на потребление памяти. Это так потому, что RDMA-приём данных в
Vitastor, увы, всё равно не является zero-copy, т.е. всё равно 1 раз
копирует данные в памяти. Данная особенность, возможно, будет исправлена в
более новых версиях Vitastor.
Максимальное число RDMA-операций отправки, отправляемых в очередь одного
соединения. Желательно, чтобы оно было меньше `rdma_max_recv`, чтобы
у принимающей стороны в процессе работы не заканчивались буферы на приём.
Не влияет на потребление памяти - дополнительная память на операции отправки
не выделяется.
- name: rdma_odp
type: bool
default: false
online: false
info: |
Use RDMA with On-Demand Paging. ODP is currently only available on Mellanox
ConnectX-4 and newer adapters. ODP allows to not register memory explicitly
for RDMA adapter to be able to use it. This, in turn, allows to skip memory
copying during sending. One would think this should improve performance, but
**in reality** RDMA performance with ODP is **drastically** worse. Example
3-node cluster with 8 NVMe in each node and 2*25 GBit/s ConnectX-6 RDMA network
without ODP pushes 3950000 read iops, but only 239000 iops with ODP...
This happens because Mellanox ODP implementation seems to be based on
message retransmissions when the adapter doesn't know about the buffer yet -
it likely uses standard "RNR retransmissions" (RNR = receiver not ready)
which is generally slow in RDMA/RoCE networks. Here's a presentation about
it from ISPASS-2021 conference: https://tkygtr6.github.io/pub/ISPASS21_slides.pdf
ODP support is retained in the code just in case a good ODP implementation
appears one day.
info_ru: |
Использовать RDMA с On-Demand Paging. ODP - функция, доступная пока что
исключительно на адаптерах Mellanox ConnectX-4 и более новых. ODP позволяет
не регистрировать память для её использования RDMA-картой. Благодаря этому
можно не копировать данные при отправке их в сеть и, казалось бы, это должно
улучшать производительность - но **по факту** получается так, что
производительность только ухудшается, причём сильно. Пример - на 3-узловом
кластере с 8 NVMe в каждом узле и сетью 2*25 Гбит/с на чтение с RDMA без ODP
удаётся снять 3950000 iops, а с ODP - всего 239000 iops...
Это происходит из-за того, что реализация ODP у Mellanox неоптимальная и
основана на повторной передаче сообщений, когда карте не известен буфер -
вероятно, на стандартных "RNR retransmission" (RNR = receiver not ready).
А данные повторные передачи в RDMA/RoCE - всегда очень медленная штука.
Презентация на эту тему с конференции ISPASS-2021: https://tkygtr6.github.io/pub/ISPASS21_slides.pdf
Возможность использования ODP сохранена в коде на случай, если вдруг в один
прекрасный день появится хорошая реализация ODP.
- name: peer_connect_interval
type: sec
min: 1
default: 5
online: true
info: Interval before attempting to reconnect to an unavailable OSD.
info_ru: Время ожидания перед повторной попыткой соединиться с недоступным OSD.
- name: peer_connect_timeout
type: sec
min: 1
default: 5
online: true
info: Timeout for OSD connection attempts.
info_ru: Максимальное время ожидания попытки соединения с OSD.
- name: osd_idle_timeout
type: sec
min: 1
default: 5
online: true
info: |
OSD connection inactivity time after which clients and other OSDs send
keepalive requests to check state of the connection.
@ -160,6 +234,7 @@
type: sec
min: 1
default: 5
online: true
info: |
Maximum time to wait for OSD keepalive responses. If an OSD doesn't respond
within this time, the connection to it is dropped and a reconnection attempt
@ -168,23 +243,10 @@
Максимальное время ожидания ответа на запрос проверки состояния соединения.
Если OSD не отвечает за это время, соединение отключается и производится
повторная попытка соединения.
- name: up_wait_retry_interval
type: ms
min: 50
default: 500
info: |
OSDs respond to clients with a special error code when they receive I/O
requests for a PG that's not synchronized and started. This parameter sets
the time for the clients to wait before re-attempting such I/O requests.
info_ru: |
Когда OSD получают от клиентов запросы ввода-вывода, относящиеся к не
поднятым на данный момент на них PG, либо к PG в процессе синхронизации,
они отвечают клиентам специальным кодом ошибки, означающим, что клиент
должен некоторое время подождать перед повторением запроса. Именно это время
ожидания задаёт данный параметр.
- name: max_etcd_attempts
type: int
default: 5
online: true
info: |
Maximum number of attempts for etcd requests which can't be retried
indefinitely.
@ -194,6 +256,7 @@
- name: etcd_quick_timeout
type: ms
default: 1000
online: true
info: |
Timeout for etcd requests which should complete quickly, like lease refresh.
info_ru: |
@ -202,6 +265,7 @@
- name: etcd_slow_timeout
type: ms
default: 5000
online: true
info: Timeout for etcd requests which are allowed to wait for some time.
info_ru: |
Максимальное время выполнения запросов к etcd, для которых не обязательно
@ -209,36 +273,19 @@
- name: etcd_keepalive_timeout
type: sec
default: max(30, etcd_report_interval*2)
online: true
info: |
Timeout for etcd connection HTTP Keep-Alive. Should be higher than
etcd_report_interval to guarantee that keepalive actually works.
info_ru: |
Таймаут для HTTP Keep-Alive в соединениях к etcd. Должен быть больше, чем
etcd_report_interval, чтобы keepalive гарантированно работал.
- name: etcd_ws_keepalive_timeout
- name: etcd_ws_keepalive_interval
type: sec
default: 30
online: true
info: |
etcd websocket ping interval required to keep the connection alive and
detect disconnections quickly.
info_ru: |
Интервал проверки живости вебсокет-подключений к etcd.
- name: client_dirty_limit
type: int
default: 33554432
info: |
Without immediate_commit=all this parameter sets the limit of "dirty"
(not committed by fsync) data allowed by the client before forcing an
additional fsync and committing the data. Also note that the client always
holds a copy of uncommitted data in memory so this setting also affects
RAM usage of clients.
This parameter doesn't affect OSDs themselves.
info_ru: |
При работе без immediate_commit=all - это лимит объёма "грязных" (не
зафиксированных fsync-ом) данных, при достижении которого клиент будет
принудительно вызывать fsync и фиксировать данные. Также стоит иметь в виду,
что в этом случае до момента fsync клиент хранит копию незафиксированных
данных в памяти, то есть, настройка влияет на потребление памяти клиентами.
Параметр не влияет на сами OSD.

View File

@ -1,4 +1,5 @@
# Runtime OSD Parameters
These parameters only apply to OSDs, are not fixed at the moment of OSD drive
initialization and can be changed with an OSD restart.
initialization and can be changed - either with an OSD restart or, for some of
them, even without restarting by updating configuration in etcd.

View File

@ -2,4 +2,5 @@
Данные параметры используются только OSD, но, в отличие от дисковых параметров,
не фиксируются в момент инициализации дисков OSD и могут быть изменены в любой
момент с перезапуском OSD.
момент с помощью перезапуска OSD, а некоторые и без перезапуска, с помощью
изменения конфигурации в etcd.

View File

@ -2,15 +2,28 @@
type: sec
default: 5
info: |
Interval at which OSDs report their state to etcd. Affects OSD lease time
Interval at which OSDs report their liveness to etcd. Affects OSD lease time
and thus the failover speed. Lease time is equal to this parameter value
plus max_etcd_attempts * etcd_quick_timeout because it should be guaranteed
that every OSD always refreshes its lease in time.
info_ru: |
Интервал, с которым OSD обновляет своё состояние в etcd. Значение параметра
влияет на время резервации (lease) OSD и поэтому на скорость переключения
Интервал, с которым OSD сообщает о том, что жив, в etcd. Значение параметра
влияет на время резервации (lease) OSD и поэтому - на скорость переключения
при падении OSD. Время lease равняется значению этого параметра плюс
max_etcd_attempts * etcd_quick_timeout.
- name: etcd_stats_interval
type: sec
default: 30
info: |
Interval at which OSDs report their statistics to etcd. Highly affects the
imposed load on etcd, because statistics include a key for every OSD and
for every PG. At the same time, low statistic intervals make `vitastor-cli`
statistics more responsive.
info_ru: |
Интервал, с которым OSD обновляет свою статистику в etcd. Сильно влияет на
создаваемую нагрузку на etcd, потому что статистика содержит по ключу на
каждый OSD и на каждую PG. В то же время низкий интервал делает
статистику, печатаемую `vitastor-cli`, отзывчивей.
- name: run_primary
type: bool
default: true
@ -66,6 +79,7 @@
- name: autosync_interval
type: sec
default: 5
online: true
info: |
Time interval at which automatic fsyncs/flushes are issued by each OSD when
the immediate_commit mode if disabled. fsyncs are required because without
@ -83,6 +97,7 @@
- name: autosync_writes
type: int
default: 128
online: true
info: |
Same as autosync_interval, but sets the maximum number of uncommitted write
operations before issuing an fsync operation internally.
@ -92,19 +107,33 @@
принудительной отправкой fsync-а.
- name: recovery_queue_depth
type: int
default: 4
default: 1
online: true
info: |
Maximum recovery operations per one primary OSD at any given moment of time.
Currently it's the only parameter available to tune the speed or recovery
and rebalancing, but it's planned to implement more.
Maximum recovery and rebalance operations initiated by each OSD in parallel.
Note that each OSD talks to a lot of other OSDs so actual number of parallel
recovery operations per each OSD is greater than just recovery_queue_depth.
Increasing this parameter can speedup recovery if [auto-tuning](#recovery_tune_interval)
allows it or if it is disabled.
info_ru: |
Максимальное число операций восстановления на одном первичном OSD в любой
момент времени. На данный момент единственный параметр, который можно менять
для ускорения или замедления восстановления и перебалансировки данных, но
в планах реализация других параметров.
Максимальное число параллельных операций восстановления, инициируемых одним
OSD в любой момент времени. Имейте в виду, что каждый OSD обычно работает с
многими другими OSD, так что на практике параллелизм восстановления больше,
чем просто recovery_queue_depth. Увеличение значения этого параметра может
ускорить восстановление если [автотюнинг скорости](#recovery_tune_interval)
разрешает это или если он отключён.
- name: recovery_sleep_us
type: us
default: 0
online: true
info: |
Delay for all recovery- and rebalance- related operations. If non-zero,
such operations are artificially slowed down to reduce the impact on
client I/O.
- name: recovery_pg_switch
type: int
default: 128
online: true
info: |
Number of recovery operations before switching to recovery of the next PG.
The idea is to mix all PGs during recovery for more even space and load
@ -119,6 +148,7 @@
- name: recovery_sync_batch
type: int
default: 16
online: true
info: Maximum number of recovery operations before issuing an additional fsync.
info_ru: Максимальное число операций восстановления перед дополнительным fsync.
- name: readonly
@ -133,6 +163,7 @@
- name: no_recovery
type: bool
default: false
online: true
info: |
Disable automatic background recovery of objects. Note that it doesn't
affect implicit recovery of objects happening during writes - a write is
@ -145,6 +176,7 @@
- name: no_rebalance
type: bool
default: false
online: true
info: |
Disable background movement of data between different OSDs. Disabling it
means that PGs in the `has_misplaced` state will be left in it indefinitely.
@ -155,6 +187,7 @@
- name: print_stats_interval
type: sec
default: 3
online: true
info: |
Time interval at which OSDs print simple human-readable operation
statistics on stdout.
@ -164,6 +197,7 @@
- name: slow_log_interval
type: sec
default: 10
online: true
info: |
Time interval at which OSDs dump slow or stuck operations on stdout, if
they're any. Also it's the time after which an operation is considered
@ -175,6 +209,7 @@
- name: inode_vanish_time
type: sec
default: 60
online: true
info: |
Number of seconds after which a deleted inode is removed from OSD statistics.
info_ru: |
@ -182,6 +217,7 @@
- name: max_write_iodepth
type: int
default: 128
online: true
info: |
Parallel client write operation limit per one OSD. Operations that exceed
this limit are pushed to a temporary queue instead of being executed
@ -193,6 +229,7 @@
- name: min_flusher_count
type: int
default: 1
online: true
info: |
Flusher is a micro-thread that moves data from the journal to the data
area of the device. Their number is auto-tuned between minimum and maximum.
@ -204,6 +241,7 @@
- name: max_flusher_count
type: int
default: 256
online: true
info: |
Maximum number of journal flushers (see above min_flusher_count).
info_ru: |
@ -247,6 +285,96 @@
достаточно 16- или 32-мегабайтного журнала. Однако в теории отключение
параметра может оказаться полезным для гибридных OSD (HDD+SSD) с большими
журналами, расположенными на быстром по сравнению с HDD устройстве.
- name: data_io
type: string
default: direct
info: |
I/O mode for *data*. One of "direct", "cached" or "directsync". Corresponds
to O_DIRECT, O_SYNC and O_DIRECT|O_SYNC, respectively.
Choose "cached" to use Linux page cache. This may improve read performance
for hot data and slower disks - HDDs and maybe SATA SSDs - but will slightly
decrease write performance for fast disks because page cache is an overhead
itself.
Choose "directsync" to use [immediate_commit](layout-cluster.ru.md#immediate_commit)
(which requires disable_data_fsync) with drives having write-back cache
which can't be turned off, for example, Intel Optane. Also note that *some*
desktop SSDs (for example, HP EX950) may ignore O_SYNC thus making
disable_data_fsync unsafe even with "directsync".
info_ru: |
Режим ввода-вывода для *данных*. Одно из значений "direct", "cached" или
"directsync", означающих O_DIRECT, O_SYNC и O_DIRECT|O_SYNC, соответственно.
Выберите "cached", чтобы использовать системный кэш Linux (page cache) при
чтении и записи. Это может улучшить скорость чтения горячих данных с
относительно медленных дисков - HDD и, возможно, SATA SSD - но немного
снижает производительность записи для быстрых дисков, так как кэш сам по
себе тоже добавляет накладные расходы.
Выберите "directsync", если хотите задействовать
[immediate_commit](layout-cluster.ru.md#immediate_commit) (требующий
включенияd disable_data_fsync) на дисках с неотключаемым кэшем. Пример таких
дисков - Intel Optane. При этом также стоит иметь в виду, что *некоторые*
настольные SSD (например, HP EX950) игнорируют флаг O_SYNC, делая отключение
fsync небезопасным даже с режимом "directsync".
- name: meta_io
type: string
default: direct
info: |
I/O mode for *metadata*. One of "direct", "cached" or "directsync".
"cached" may improve read performance, but only under the following conditions:
1. your drives are relatively slow (HDD, SATA SSD), and
2. checksums are enabled, and
3. [inmemory_metadata](#inmemory_metadata) is disabled.
Under all these conditions, metadata blocks are read from disk on every
read request to verify checksums and caching them may reduce this extra
read load. Without (3) metadata is never read from the disk after starting,
and without (2) metadata blocks are read from disk only during journal
flushing.
"directsync" is the same as above.
If the same device is used for data and metadata, meta_io by default is set
to the same value as [data_io](#data_io).
info_ru: |
Режим ввода-вывода для *метаданных*. Одно из значений "direct", "cached" или
"directsync".
"cached" может улучшить скорость чтения, если:
1. у вас медленные диски (HDD, SATA SSD)
2. контрольные суммы включены
3. параметр [inmemory_metadata](#inmemory_metadata) отключён.
При этих условиях блоки метаданных читаются с диска при каждом запросе чтения
для проверки контрольных сумм и их кэширование может снизить дополнительную
нагрузку на диск. Без (3) метаданные никогда не читаются с диска после
запуска OSD, а без (2) блоки метаданных читаются только при сбросе журнала.
Если одно и то же устройство используется для данных и метаданных, режим
ввода-вывода метаданных по умолчанию устанавливается равным [data_io](#data_io).
- name: journal_io
type: string
default: direct
info: |
I/O mode for *journal*. One of "direct", "cached" or "directsync".
Here, "cached" may only improve read performance for recent writes and
only if [inmemory_journal](#inmemory_journal) is turned off.
If the same device is used for metadata and journal, journal_io by default
is set to the same value as [meta_io](#meta_io).
info_ru: |
Режим ввода-вывода для *журнала*. Одно из значений "direct", "cached" или
"directsync".
Здесь "cached" может улучшить скорость чтения только недавно записанных
данных и только если параметр [inmemory_journal](#inmemory_journal)
отключён.
Если одно и то же устройство используется для метаданных и журнала,
режим ввода-вывода журнала по умолчанию устанавливается равным
[meta_io](#meta_io).
- name: journal_sector_buffer_count
type: int
default: 32
@ -284,6 +412,7 @@
- name: throttle_small_writes
type: bool
default: false
online: true
info: |
Enable soft throttling of small journaled writes. Useful for hybrid OSDs
with fast journal/metadata devices and slow data devices. The idea is that
@ -312,6 +441,7 @@
- name: throttle_target_iops
type: int
default: 100
online: true
info: |
Target maximum number of throttled operations per second under the condition
of full journal. Set it to approximate random write iops of your data devices
@ -324,6 +454,7 @@
- name: throttle_target_mbs
type: int
default: 100
online: true
info: |
Target maximum bandwidth in MB/s of throttled operations per second under
the condition of full journal. Set it to approximate linear write
@ -336,6 +467,7 @@
- name: throttle_target_parallelism
type: int
default: 1
online: true
info: |
Target maximum parallelism of throttled operations under the condition of
full journal. Set it to approximate internal parallelism of your data
@ -348,6 +480,7 @@
- name: throttle_threshold_us
type: us
default: 50
online: true
info: |
Minimal computed delay to be applied to throttled operations. Usually
doesn't need to be changed.
@ -357,10 +490,260 @@
- name: osd_memlock
type: bool
default: false
info: >
info: |
Lock all OSD memory to prevent it from being unloaded into swap with
mlockall(). Requires sufficient ulimit -l (max locked memory).
info_ru: >
info_ru: |
Блокировать всю память OSD с помощью mlockall, чтобы запретить её выгрузку
в пространство подкачки. Требует достаточного значения ulimit -l (лимита
заблокированной памяти).
- name: auto_scrub
type: bool
default: false
online: true
info: |
Data scrubbing is the process of background verification of copies to find
and repair corrupted blocks. It's not run automatically by default since
it's a new feature. Set this parameter to true to enable automatic scrubs.
This parameter makes OSDs automatically schedule data scrubbing of clean PGs
every `scrub_interval` (see below). You can also start/schedule scrubbing
manually by setting `next_scrub` JSON key to the desired UNIX time of the
next scrub in `/pg/history/...` values in etcd.
info_ru: |
Скраб - процесс фоновой проверки копий данных, предназначенный, чтобы
находить и исправлять повреждённые блоки. По умолчанию эти проверки ещё не
запускаются автоматически, так как являются новой функцией. Чтобы включить
автоматическое планирование скрабов, установите данный параметр в true.
Включённый параметр заставляет OSD автоматически планировать фоновую
проверку чистых PG раз в `scrub_interval` (см. ниже). Вы также можете
запустить или запланировать проверку вручную, установив значение ключа JSON
`next_scrub` внутри ключей etcd `/pg/history/...` в UNIX-время следующей
желаемой проверки.
- name: no_scrub
type: bool
default: false
online: true
info: |
Temporarily disable scrubbing and stop running scrubs.
info_ru: |
Временно отключить и остановить запущенные скрабы.
- name: scrub_interval
type: string
default: 30d
online: true
info: |
Default automatic scrubbing interval for all pools. Numbers without suffix
are treated as seconds, possible unit suffixes include 's' (seconds),
'm' (minutes), 'h' (hours), 'd' (days), 'M' (months) and 'y' (years).
info_ru: |
Интервал автоматической фоновой проверки по умолчанию для всех пулов.
Значения без указанной единицы измерения считаются в секундах, допустимые
символы единиц измерения в конце: 's' (секунды),
'm' (минуты), 'h' (часы), 'd' (дни), 'M' (месяца) или 'y' (годы).
- name: scrub_queue_depth
type: int
default: 1
online: true
info: |
Number of parallel scrubbing operations per one OSD.
info_ru: |
Число параллельных операций фоновой проверки на один OSD.
- name: scrub_sleep
type: ms
default: 0
online: true
info: |
Additional interval between two consecutive scrubbing operations on one OSD.
Can be used to slow down scrubbing if it affects user load too much.
info_ru: |
Дополнительный интервал ожидания после фоновой проверки каждого объекта на
одном OSD. Может использоваться для замедления скраба, если он слишком
сильно влияет на пользовательскую нагрузку.
- name: scrub_list_limit
type: int
default: 1000
online: true
info: |
Number of objects to list in one listing operation during scrub.
info_ru: |
Размер загружаемых за одну операцию списков объектов в процессе фоновой
проверки.
- name: scrub_find_best
type: bool
default: true
online: true
info: |
Find and automatically restore best versions of objects with unmatched
copies. In replicated setups, the best version is the version with most
matching replicas. In EC setups, the best version is the subset of data
and parity chunks without mismatches.
The hypothetical situation where you might want to disable it is when
you have 3 replicas and you are paranoid that 2 HDDs out of 3 may silently
corrupt an object in the same way (for example, zero it out) and only
1 HDD will remain good. In this case disabling scrub_find_best may help
you to recover the data! See also scrub_ec_max_bruteforce below.
info_ru: |
Находить и автоматически восстанавливать "лучшие версии" объектов с
несовпадающими копиями/частями. При использовании репликации "лучшая"
версия - версия, доступная в большем числе экземпляров, чем другие. При
использовании кодов коррекции ошибок "лучшая" версия - это подмножество
частей данных и чётности, полностью соответствующих друг другу.
Гипотетическая ситуация, в которой вы можете захотеть отключить этот
поиск - это если у вас 3 реплики и вы боитесь, что 2 диска из 3 могут
незаметно и одинаково повредить данные одного и того же объекта, например,
занулив его, и только 1 диск останется неповреждённым. В этой ситуации
отключение этого параметра поможет вам восстановить данные! Смотрите также
описание следующего параметра - scrub_ec_max_bruteforce.
- name: scrub_ec_max_bruteforce
type: int
default: 100
online: true
info: |
Vitastor can locate corrupted chunks in EC setups with more than 1 parity
chunk by brute-forcing all possible error locations. This configuration
value limits the maximum number of checked combinations. You can try to
increase it if you have EC N+K setup with N and K large enough for
combination count `C(N+K-1, K-1) = (N+K-1)! / (K-1)! / N!` to be greater
than the default 100.
If there are too many possible combinations or if multiple combinations give
correct results then objects are marked inconsistent and aren't recovered
automatically.
In replicated setups bruteforcing isn't needed, Vitastor just assumes that
the variant with most available equal copies is correct. For example, if
you have 3 replicas and 1 of them differs, this one is considered to be
corrupted. But if there is no "best" version with more copies than all
others have then the object is also marked as inconsistent.
info_ru: |
Vitastor старается определить повреждённые части объектов при использовании
EC (кодов коррекции ошибок) с более, чем 1 диском чётности, путём перебора
всех возможных комбинаций ошибочных частей. Данное значение конфигурации
ограничивает число перебираемых комбинаций. Вы можете попробовать поднять
его, если используете схему кодирования EC N+K с N и K, достаточно большими
для того, чтобы число сочетаний `C(N+K-1, K-1) = (N+K-1)! / (K-1)! / N!`
было больше, чем стандартное значение 100.
Если возможных комбинаций слишком много или если корректная комбинаций не
определяется однозначно, объекты помечаются неконсистентными (inconsistent)
и не восстанавливаются автоматически.
При использовании репликации перебор не нужен, Vitastor просто предполагает,
что вариант объекта с наибольшим количеством одинаковых копий корректен.
Например, если вы используете 3 реплики и 1 из них отличается, эта 1 копия
считается некорректной. Однако, если "лучшую" версию с числом доступных
копий большим, чем у всех других версий, найти невозможно, то объект тоже
маркируется неконсистентным.
- name: recovery_tune_interval
type: sec
default: 1
online: true
info: |
Interval at which OSD re-considers client and recovery load and automatically
adjusts [recovery_sleep_us](#recovery_sleep_us). Recovery auto-tuning is
disabled if recovery_tune_interval is set to 0.
Auto-tuning targets utilization. Utilization is a measure of load and is
equal to the product of iops and average latency (so it may be greater
than 1). You set "low" and "high" client utilization thresholds and two
corresponding target recovery utilization levels. OSD calculates desired
recovery utilization from client utilization using linear interpolation
and auto-tunes recovery operation delay to make actual recovery utilization
match desired.
This allows to reduce recovery/rebalance impact on client operations. It is
of course impossible to remove it completely, but it should become adequate.
In some tests rebalance could earlier drop client write speed from 1.5 GB/s
to 50-100 MB/s, with default auto-tuning settings it now only reduces
to ~1 GB/s.
info_ru: |
Интервал, с которым OSD пересматривает клиентскую нагрузку и нагрузку
восстановления и автоматически подстраивает [recovery_sleep_us](#recovery_sleep_us).
Автотюнинг (автоподстройка) отключается, если recovery_tune_interval
устанавливается в значение 0.
Автотюнинг регулирует утилизацию. Утилизация является мерой нагрузки
и равна произведению числа операций в секунду и средней задержки
(то есть, она может быть выше 1). Вы задаёте два уровня клиентской
утилизации - "низкий" и "высокий" (low и high) и два соответствующих
целевых уровня утилизации операциями восстановления. OSD рассчитывает
желаемый уровень утилизации восстановления линейной интерполяцией от
клиентской утилизации и подстраивает задержку операций восстановления
так, чтобы фактическая утилизация восстановления совпадала с желаемой.
Это позволяет снизить влияние восстановления и ребаланса на клиентские
операции. Конечно, невозможно исключить такое влияние полностью, но оно
должно становиться адекватнее. В некоторых тестах перебалансировка могла
снижать клиентскую скорость записи с 1.5 ГБ/с до 50-100 МБ/с, а теперь, с
настройками автотюнинга по умолчанию, она снижается только до ~1 ГБ/с.
- name: recovery_tune_util_low
type: float
default: 0.1
online: true
info: |
Desired recovery/rebalance utilization when client load is high, i.e. when
it is at or above recovery_tune_client_util_high.
info_ru: |
Желаемая утилизация восстановления в моменты, когда клиентская нагрузка
высокая, то есть, находится на уровне или выше recovery_tune_client_util_high.
- name: recovery_tune_util_high
type: float
default: 1
online: true
info: |
Desired recovery/rebalance utilization when client load is low, i.e. when
it is at or below recovery_tune_client_util_low.
info_ru: |
Желаемая утилизация восстановления в моменты, когда клиентская нагрузка
низкая, то есть, находится на уровне или ниже recovery_tune_client_util_low.
- name: recovery_tune_client_util_low
type: float
default: 0
online: true
info: Client utilization considered "low".
info_ru: Клиентская утилизация, которая считается "низкой".
- name: recovery_tune_client_util_high
type: float
default: 0.5
online: true
info: Client utilization considered "high".
info_ru: Клиентская утилизация, которая считается "высокой".
- name: recovery_tune_agg_interval
type: int
default: 10
online: true
info: |
The number of last auto-tuning iterations to use for calculating the
delay as average. Lower values result in quicker response to client
load change, higher values result in more stable delay. Default value of 10
is usually fine.
info_ru: |
Число последних итераций автоподстройки для расчёта задержки как среднего
значения. Меньшие значения параметра ускоряют отклик на изменение нагрузки,
большие значения делают задержку стабильнее. Значение по умолчанию 10
обычно нормальное и не требует изменений.
- name: recovery_tune_sleep_min_us
type: us
default: 10
online: true
info: |
Minimum possible value for auto-tuned recovery_sleep_us. Lower values
are changed to 0.
info_ru: |
Минимальное возможное значение авто-подстроенного recovery_sleep_us.
Меньшие значения заменяются на 0.
- name: recovery_tune_sleep_cutoff_us
type: us
default: 10000000
online: true
info: |
Maximum possible value for auto-tuned recovery_sleep_us. Higher values
are treated as outliers and ignored in aggregation.
info_ru: |
Максимальное возможное значение авто-подстроенного recovery_sleep_us.
Большие значения считаются случайными выбросами и игнорируются в
усреднении.

View File

@ -17,4 +17,27 @@ and apply all `NNN-*.yaml` manifests to your Kubernetes installation:
for i in ./???-*.yaml; do kubectl apply -f $i; done
```
After that you'll be able to create PersistentVolumes. See example in [csi/deploy/example-pvc.yaml](../../csi/deploy/example-pvc.yaml).
After that you'll be able to create PersistentVolumes.
**Important:** For best experience, use Linux kernel at least 5.15 with [VDUSE](../usage/qemu.en.md#vduse)
kernel modules enabled (vdpa, vduse, virtio-vdpa). If your distribution doesn't
have them pre-built - build them yourself ([instructions](../usage/qemu.en.md#vduse)),
I promise it's worth it :-). When VDUSE is unavailable, CSI driver uses [NBD](../usage/nbd.en.md)
to map Vitastor devices. NBD is slower and prone to timeout issues: if Vitastor
cluster becomes unresponsible for more than [nbd_timeout](../config/client.en.md#nbd_timeout),
the NBD device detaches and breaks pods using it.
## Features
Vitastor CSI supports:
- Kubernetes starting with 1.20 (or 1.17 for older vitastor-csi <= 1.1.0)
- Filesystem RWO (ReadWriteOnce) volumes. Example: [PVC](../../csi/deploy/example-pvc.yaml), [pod](../../csi/deploy/example-test-pod.yaml)
- Raw block RWX (ReadWriteMany) volumes. Example: [PVC](../../csi/deploy/example-pvc-block.yaml), [pod](../../csi/deploy/example-test-pod-block.yaml)
- Volume expansion
- Volume snapshots. Example: [snapshot class](../../csi/deploy/example-snapshot-class.yaml), [snapshot](../../csi/deploy/example-snapshot.yaml), [clone](../../csi/deploy/example-snapshot-clone.yaml)
- [VDUSE](../usage/qemu.en.md#vduse) (preferred) and [NBD](../usage/nbd.en.md) device mapping methods
- Upgrades with VDUSE - new handler processes are restarted when CSI pods are restarted themselves
- VDUSE daemon auto-restart - handler processes are automatically restarted if they crash due to a bug in Vitastor client code
- Multiple clusters by using multiple configuration files in ConfigMap.
Remember that to use snapshots with CSI you also have to install [Snapshot Controller and CRDs](https://kubernetes-csi.github.io/docs/snapshot-controller.html#deployment).

View File

@ -8,13 +8,36 @@
У Vitastor есть CSI-плагин для Kubernetes, поддерживающий RWO, а также блочные RWX, тома.
Для установки возьмите манифесты из директории [csi/deploy/](../csi/deploy/), поместите
вашу конфигурацию подключения к Vitastor в [csi/deploy/001-csi-config-map.yaml](../csi/deploy/001-csi-config-map.yaml),
настройте StorageClass в [csi/deploy/009-storage-class.yaml](../csi/deploy/009-storage-class.yaml)
Для установки возьмите манифесты из директории [csi/deploy/](../../csi/deploy/), поместите
вашу конфигурацию подключения к Vitastor в [csi/deploy/001-csi-config-map.yaml](../../csi/deploy/001-csi-config-map.yaml),
настройте StorageClass в [csi/deploy/009-storage-class.yaml](../../csi/deploy/009-storage-class.yaml)
и примените все `NNN-*.yaml` к вашей инсталляции Kubernetes.
```
for i in ./???-*.yaml; do kubectl apply -f $i; done
```
После этого вы сможете создавать PersistentVolume. Пример смотрите в файле [csi/deploy/example-pvc.yaml](../csi/deploy/example-pvc.yaml).
После этого вы сможете создавать PersistentVolume.
**Важно:** Лучше всего использовать ядро Linux версии не менее 5.15 с включёнными модулями
[VDUSE](../usage/qemu.ru.md#vduse) (vdpa, vduse, virtio-vdpa). Если в вашем дистрибутиве
они не собраны из коробки - соберите их сами, обещаю, что это стоит того ([инструкция](../usage/qemu.ru.md#vduse)) :-).
Когда VDUSE недоступно, CSI-плагин использует [NBD](../usage/nbd.ru.md) для подключения
дисков, а NBD медленнее и имеет проблему таймаута - если кластер остаётся недоступным
дольше, чем [nbd_timeout](../config/client.ru.md#nbd_timeout), NBD-устройство отключается
и ломает поды, использующие его.
## Возможности
CSI-плагин Vitastor поддерживает:
- Версии Kubernetes, начиная с 1.20 (или с 1.17 для более старых vitastor-csi <= 1.1.0)
- Файловые RWO (ReadWriteOnce) тома. Пример: [PVC](../../csi/deploy/example-pvc.yaml), [под](../../csi/deploy/example-test-pod.yaml)
- Сырые блочные RWX (ReadWriteMany) тома. Пример: [PVC](../../csi/deploy/example-pvc-block.yaml), [под](../../csi/deploy/example-test-pod-block.yaml)
- Расширение размера томов
- Снимки томов. Пример: [класс снимков](../../csi/deploy/example-snapshot-class.yaml), [снимок](../../csi/deploy/example-snapshot.yaml), [клон снимка](../../csi/deploy/example-snapshot-clone.yaml)
- Способы подключения устройств [VDUSE](../usage/qemu.ru.md#vduse) (предпочитаемый) и [NBD](../usage/nbd.ru.md)
- Обновление при использовании VDUSE - новые процессы-обработчики устройств успешно перезапускаются вместе с самими подами CSI
- Автоперезауск демонов VDUSE - процесс-обработчик автоматически перезапустится, если он внезапно упадёт из-за бага в коде клиента Vitastor
- Несколько кластеров через задание нескольких файлов конфигурации в ConfigMap.
Не забывайте, что для использования снимков нужно сначала установить [контроллер снимков и CRD](https://kubernetes-csi.github.io/docs/snapshot-controller.html#deployment).

View File

@ -36,5 +36,5 @@ vitastor_pool_id = 1
image_upload_use_cinder_backend = True
```
To put Glance images in Vitastor, use [https://docs.openstack.org/cinder/pike/admin/blockstorage-volume-backed-image.html](volume-backed images),
To put Glance images in Vitastor, use [volume-backed images](https://docs.openstack.org/cinder/pike/admin/blockstorage-volume-backed-image.html),
although the support has not been verified yet.

View File

@ -36,5 +36,5 @@ image_upload_use_cinder_backend = True
```
Чтобы помещать в Vitastor Glance-образы, нужно использовать
[https://docs.openstack.org/cinder/pike/admin/blockstorage-volume-backed-image.html](образы на основе томов Cinder),
[образы на основе томов Cinder](https://docs.openstack.org/cinder/pike/admin/blockstorage-volume-backed-image.html),
однако, поддержка этой функции ещё не проверялась.

View File

@ -9,26 +9,32 @@
## Debian
- Trust Vitastor package signing key:
`wget -q -O - https://vitastor.io/debian/pubkey | sudo apt-key add -`
`wget https://vitastor.io/debian/pubkey.gpg -O /etc/apt/trusted.gpg.d/vitastor.gpg`
- Add Vitastor package repository to your /etc/apt/sources.list:
- Debian 11 (Bullseye/Sid): `deb https://vitastor.io/debian bullseye main`
- Debian 12 (Bookworm/Sid): `deb https://vitastor.io/debian bookworm main`
- Debian 11 (Bullseye): `deb https://vitastor.io/debian bullseye main`
- Debian 10 (Buster): `deb https://vitastor.io/debian buster main`
- Add `-oldstable` to bookworm/bullseye/buster in this line to install the last
stable version from 0.9.x branch instead of 1.x
- For Debian 10 (Buster) also enable backports repository:
`deb http://deb.debian.org/debian buster-backports main`
- Install packages: `apt update; apt install vitastor lp-solve etcd linux-image-amd64 qemu`
- Install packages: `apt update; apt install vitastor lp-solve etcd linux-image-amd64 qemu-system-x86`
## CentOS
- Add Vitastor package repository:
- CentOS 7: `yum install https://vitastor.io/rpms/centos/7/vitastor-release-1.0-1.el7.noarch.rpm`
- CentOS 8: `dnf install https://vitastor.io/rpms/centos/8/vitastor-release-1.0-1.el8.noarch.rpm`
- CentOS 7: `yum install https://vitastor.io/rpms/centos/7/vitastor-release.rpm`
- CentOS 8: `dnf install https://vitastor.io/rpms/centos/8/vitastor-release.rpm`
- AlmaLinux 9 and other RHEL 9 clones (Rocky, Oracle...): `dnf install https://vitastor.io/rpms/centos/9/vitastor-release.rpm`
- Enable EPEL: `yum/dnf install epel-release`
- Enable additional CentOS repositories:
- CentOS 7: `yum install centos-release-scl`
- CentOS 8: `dnf install centos-release-advanced-virtualization`
- RHEL 9 clones: not required
- Enable elrepo-kernel:
- CentOS 7: `yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm`
- CentOS 8: `dnf install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm`
- RHEL 9 clones: `dnf install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm`
- Install packages: `yum/dnf install vitastor lpsolve etcd kernel-ml qemu-kvm`
## Installation requirements
@ -42,3 +48,10 @@
- etcd 3.4.15 or newer. Earlier versions won't work because of various bugs,
for example [#12402](https://github.com/etcd-io/etcd/pull/12402).
- node.js 10 or newer
## Version archive
All previous Vitastor and other components (QEMU, etcd...) package builds
can be found here:
https://vitastor.io/archive/

View File

@ -9,26 +9,32 @@
## Debian
- Добавьте ключ репозитория Vitastor:
`wget -q -O - https://vitastor.io/debian/pubkey | sudo apt-key add -`
`wget https://vitastor.io/debian/pubkey.gpg -O /etc/apt/trusted.gpg.d/vitastor.gpg`
- Добавьте репозиторий Vitastor в /etc/apt/sources.list:
- Debian 11 (Bullseye/Sid): `deb https://vitastor.io/debian bullseye main`
- Debian 12 (Bookworm/Sid): `deb https://vitastor.io/debian bookworm main`
- Debian 11 (Bullseye): `deb https://vitastor.io/debian bullseye main`
- Debian 10 (Buster): `deb https://vitastor.io/debian buster main`
- Добавьте `-oldstable` к слову bookworm/bullseye/buster в этой строке, чтобы
установить последнюю стабильную версию из ветки 0.9.x вместо 1.x
- Для Debian 10 (Buster) также включите репозиторий backports:
`deb http://deb.debian.org/debian buster-backports main`
- Установите пакеты: `apt update; apt install vitastor lp-solve etcd linux-image-amd64 qemu`
- Установите пакеты: `apt update; apt install vitastor lp-solve etcd linux-image-amd64 qemu-system-x86`
## CentOS
- Добавьте в систему репозиторий Vitastor:
- CentOS 7: `yum install https://vitastor.io/rpms/centos/7/vitastor-release-1.0-1.el7.noarch.rpm`
- CentOS 8: `dnf install https://vitastor.io/rpms/centos/8/vitastor-release-1.0-1.el8.noarch.rpm`
- CentOS 7: `yum install https://vitastor.io/rpms/centos/7/vitastor-release.rpm`
- CentOS 8: `dnf install https://vitastor.io/rpms/centos/8/vitastor-release.rpm`
- AlmaLinux 9 и другие клоны RHEL 9 (Rocky, Oracle...): `dnf install https://vitastor.io/rpms/centos/9/vitastor-release.rpm`
- Включите EPEL: `yum/dnf install epel-release`
- Включите дополнительные репозитории CentOS:
- CentOS 7: `yum install centos-release-scl`
- CentOS 8: `dnf install centos-release-advanced-virtualization`
- Клоны RHEL 9: не нужно
- Включите elrepo-kernel:
- CentOS 7: `yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm`
- CentOS 8: `dnf install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm`
- Клоны RHEL 9: `dnf install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm`
- Установите пакеты: `yum/dnf install vitastor lpsolve etcd kernel-ml qemu-kvm`
## Установочные требования
@ -41,3 +47,10 @@
- etcd 3.4.15 или новее. Более старые версии не будут работать из-за разных багов,
например, [#12402](https://github.com/etcd-io/etcd/pull/12402).
- node.js 10 или новее
## Архив предыдущих версий
Все предыдущие сборки пакетов Vitastor и других компонентов, таких, как QEMU
и etcd, можно скачать по следующей ссылке:
https://vitastor.io/archive/

View File

@ -6,10 +6,10 @@
# Proxmox VE
To enable Vitastor support in Proxmox Virtual Environment (6.4-7.3 are supported):
To enable Vitastor support in Proxmox Virtual Environment (6.4-8.1 are supported):
- Add the corresponding Vitastor Debian repository into sources.list on Proxmox hosts:
buster for 6.4, bullseye for 7.3, pve7.1 for 7.1, pve7.2 for 7.2
bookworm for 8.1, pve8.0 for 8.0, bullseye for 7.4, pve7.3 for 7.3, pve7.2 for 7.2, pve7.1 for 7.1, buster for 6.4
- Install vitastor-client, pve-qemu-kvm, pve-storage-vitastor (* or see note) packages from Vitastor repository
- Define storage in `/etc/pve/storage.cfg` (see below)
- Block network access from VMs to Vitastor network (to OSDs and etcd),
@ -25,7 +25,7 @@ vitastor: vitastor
vitastor_pool testpool
# path to the configuration file
vitastor_config_path /etc/vitastor/vitastor.conf
# etcd address(es), required only if missing in the configuration file
# etcd address(es), OPTIONAL, required only if missing in the configuration file
vitastor_etcd_address 192.168.7.2:2379/v3
# prefix for keys in etcd
vitastor_etcd_prefix /vitastor
@ -35,5 +35,5 @@ vitastor: vitastor
vitastor_nbd 0
```
\* Note: you can also manually copy [patches/VitastorPlugin.pm](patches/VitastorPlugin.pm) to Proxmox hosts
\* Note: you can also manually copy [patches/VitastorPlugin.pm](../../patches/VitastorPlugin.pm) to Proxmox hosts
as `/usr/share/perl5/PVE/Storage/Custom/VitastorPlugin.pm` instead of installing pve-storage-vitastor.

View File

@ -1,15 +1,15 @@
[Документация](../../README-ru.md#документация) → Установка → Proxmox
[Документация](../../README-ru.md#документация) → Установка → Proxmox VE
-----
[Read in English](proxmox.en.md)
# Proxmox
# Proxmox VE
Чтобы подключить Vitastor к Proxmox Virtual Environment (поддерживаются версии 6.4-7.3):
Чтобы подключить Vitastor к Proxmox Virtual Environment (поддерживаются версии 6.4-8.1):
- Добавьте соответствующий Debian-репозиторий Vitastor в sources.list на хостах Proxmox:
buster для 6.4, bullseye для 7.3, pve7.1 для 7.1, pve7.2 для 7.2
bookworm для 8.1, pve8.0 для 8.0, bullseye для 7.4, pve7.3 для 7.3, pve7.2 для 7.2, pve7.1 для 7.1, buster для 6.4
- Установите пакеты vitastor-client, pve-qemu-kvm, pve-storage-vitastor (* или см. сноску) из репозитория Vitastor
- Определите тип хранилища в `/etc/pve/storage.cfg` (см. ниже)
- Обязательно заблокируйте доступ от виртуальных машин к сети Vitastor (OSD и etcd), т.к. Vitastor (пока) не поддерживает аутентификацию
@ -24,7 +24,7 @@ vitastor: vitastor
vitastor_pool testpool
# Путь к файлу конфигурации
vitastor_config_path /etc/vitastor/vitastor.conf
# Адрес(а) etcd, нужны, только если не указаны в vitastor.conf
# Адрес(а) etcd, ОПЦИОНАЛЬНЫ, нужны, только если не указаны в vitastor.conf
vitastor_etcd_address 192.168.7.2:2379/v3
# Префикс ключей метаданных в etcd
vitastor_etcd_prefix /vitastor
@ -35,5 +35,5 @@ vitastor: vitastor
```
\* Примечание: вместо установки пакета pve-storage-vitastor вы можете вручную скопировать файл
[patches/VitastorPlugin.pm](patches/VitastorPlugin.pm) на хосты Proxmox как
[patches/VitastorPlugin.pm](../../patches/VitastorPlugin.pm) на хосты Proxmox как
`/usr/share/perl5/PVE/Storage/Custom/VitastorPlugin.pm`.

View File

@ -21,7 +21,7 @@
## Basic instructions
Download source, for example using git: `git clone --recurse-submodules https://yourcmc.ru/git/vitalif/vitastor/`
Download source, for example using git: `git clone --recurse-submodules https://git.yourcmc.ru/vitalif/vitastor/`
Get `fio` source and symlink it into `<vitastor>/fio`. If you don't want to build fio engine,
you can disable it by passing `-DWITH_FIO=no` to cmake.
@ -41,7 +41,7 @@ It's recommended to build the QEMU driver (qemu_driver.c) in-tree, as a part of
QEMU build process. To do that:
- Install vitastor client library headers (from source or from vitastor-client-dev package)
- Take a corresponding patch from `patches/qemu-*-vitastor.patch` and apply it to QEMU source
- Copy `src/qemu_driver.c` to QEMU source directory as `block/block-vitastor.c`
- Copy `src/qemu_driver.c` to QEMU source directory as `block/vitastor.c`
- Build QEMU as usual
But it is also possible to build it out-of-tree. To do that:

View File

@ -21,7 +21,7 @@
## Базовая инструкция
Скачайте исходные коды, например, из git: `git clone --recurse-submodules https://yourcmc.ru/git/vitalif/vitastor/`
Скачайте исходные коды, например, из git: `git clone --recurse-submodules https://git.yourcmc.ru/vitalif/vitastor/`
Скачайте исходные коды пакета `fio`, распакуйте их и создайте символическую ссылку на них
в директории исходников Vitastor: `<vitastor>/fio`. Либо, если вы не хотите собирать плагин fio,
@ -41,7 +41,7 @@ cmake .. && make -j8 install
Драйвер QEMU (qemu_driver.c) рекомендуется собирать вместе с самим QEMU. Для этого:
- Установите заголовки клиентской библиотеки Vitastor (из исходников или из пакета vitastor-client-dev)
- Возьмите соответствующий патч из `patches/qemu-*-vitastor.patch` и примените его к исходникам QEMU
- Скопируйте [src/qemu_driver.c](../../src/qemu_driver.c) в директорию исходников QEMU как `block/block-vitastor.c`
- Скопируйте [src/qemu_driver.c](../../src/qemu_driver.c) в директорию исходников QEMU как `block/vitastor.c`
- Соберите QEMU как обычно
Однако в целях отладки драйвер также можно собирать отдельно от QEMU. Для этого:
@ -60,7 +60,7 @@ cmake .. && make -j8 install
* Для QEMU 2.0+: `<qemu>/qapi-types.h` &rarr; `<vitastor>/qemu/b/qemu/qapi-types.h`
- `config-host.h` и `qapi` нужны, т.к. в них содержатся автогенерируемые заголовки
- Сконфигурируйте cmake Vitastor с `WITH_QEMU=yes` (`cmake .. -DWITH_QEMU=yes`) и, если вы
используете RHEL-подобый дистрибутив, также с `QEMU_PLUGINDIR=qemu-kvm`.
используете RHEL-подобный дистрибутив, также с `QEMU_PLUGINDIR=qemu-kvm`.
- После этого в процессе сборки Vitastor также будет собираться подходящий для вашей
версии QEMU `block-vitastor.so`.
- Таким образом можно использовать драйвер даже с немодифицированным QEMU, но в этом случае

View File

@ -44,7 +44,7 @@
depends linearly on drive capacity and data store block size which is 128 KB by default.
With 128 KB blocks metadata takes around 512 MB per 1 TB (which is still less than Ceph wants).
Journal is also kept in memory by default, but in SSD-only clusters it's only 32 MB, and in SSD+HDD
clusters, where it's beneficial to increase it, [inmemory_journal](docs/config/osd.en.md#inmemory_journal) can be disabled.
clusters, where it's beneficial to increase it, [inmemory_journal](../config/osd.en.md#inmemory_journal) can be disabled.
- Vitastor storage layer doesn't have internal copy-on-write or redirect-write. I know that maybe
it's possible to create a good copy-on-write storage, but it's much harder and makes performance
less deterministic, so CoW isn't used in Vitastor.

View File

@ -54,7 +54,8 @@
виртуальные диски, их снимки и клоны.
- **Драйвер QEMU** — подключаемый модуль QEMU, позволяющий QEMU/KVM виртуальным машинам работать
с виртуальными дисками Vitastor напрямую из пространства пользователя с помощью клиентской
библиотеки, без необходимости отображения дисков в виде блочных устройств.
библиотеки, без необходимости отображения дисков в виде блочных устройств. Тот же драйвер
позволяет подключать диски в систему через [VDUSE](../usage/qemu.ru.md#vduse).
- **vitastor-nbd** — утилита, позволяющая монтировать образы Vitastor в виде блочных устройств
с помощью NBD (Network Block Device), на самом деле скорее работающего как "BUSE"
(Block Device In Userspace). Модуля ядра Linux для выполнения той же задачи в Vitastor нет
@ -156,7 +157,7 @@
блока хранилища (block_size, по умолчанию 128 КБ). С 128 КБ блоком потребление памяти
составляет примерно 512 МБ на 1 ТБ данных. Журналы по умолчанию тоже хранятся в памяти,
но в SSD-кластерах нужный размер журнала составляет всего 32 МБ, а в гибридных (SSD+HDD)
кластерах, в которых есть смысл делать журналы больше, можно отключить [inmemory_journal](../docs/config/osd.ru.md#inmemory_journal).
кластерах, в которых есть смысл делать журналы больше, можно отключить [inmemory_journal](../config/osd.ru.md#inmemory_journal).
- В Vitastor нет внутреннего copy-on-write. Я считаю, что реализация CoW-хранилища гораздо сложнее,
поэтому сложнее добиться устойчиво хороших результатов. Возможно, в один прекрасный день
я придумаю красивый алгоритм для CoW-хранилища, но пока нет — внутреннего CoW в Vitastor не будет.

View File

@ -13,7 +13,7 @@
## Server-side features
- Basic part: highly-available block storage with symmetric clustering and no SPOF
- [Performance](../performance/comparison1.en.md) ;-D
- [Performance](../performance/bench2.en.md) ;-D
- [Multiple redundancy schemes](../config/pool.en.md#scheme): Replication, XOR n+1, Reed-Solomon erasure codes
based on jerasure and ISA-L libraries with any number of data and parity drives in a group
- Configuration via simple JSON data structures in etcd (parameters, pools and images)
@ -29,12 +29,17 @@
- Snapshots and copy-on-write image clones
- [Write throttling to smooth random write workloads in SSD+HDD configurations](../config/osd.en.md#throttle_small_writes)
- [RDMA/RoCEv2 support via libibverbs](../config/network.en.md#rdma_device)
- [Scrubbing](../config/osd.en.md#auto_scrub) (verification of copies)
- [Checksums](../config/layout-osd.en.md#data_csum_type)
- [Client write-back cache](../config/client.en.md#client_enable_writeback)
- [Intelligent recovery auto-tuning](../config/osd.en.md#recovery_tune_interval)
- [Clustered file system](../usage/nfs.en.md#vitastorfs)
## Plugins and tools
- [Debian and CentOS packages](../installation/packages.en.md)
- [Image management CLI (vitastor-cli)](../usage/cli.en.md)
- [Disk management CLI (vitastor-disk)](docs/usage/disk.en.md)
- [Disk management CLI (vitastor-disk)](../usage/disk.en.md)
- Generic user-space client library
- [Native QEMU driver](../usage/qemu.en.md)
- [Loadable fio engine for benchmarks](../usage/fio.en.md)
@ -42,21 +47,20 @@
- [CSI plugin for Kubernetes](../installation/kubernetes.en.md)
- [OpenStack support: Cinder driver, Nova and libvirt patches](../installation/openstack.en.md)
- [Proxmox storage plugin and packages](../installation/proxmox.en.md)
- [Simplified NFS proxy for file-based image access emulation (suitable for VMWare)](../usage/nfs.en.md)
- [Simplified NFS proxy for file-based image access emulation (suitable for VMWare)](../usage/nfs.en.md#pseudo-fs)
## Roadmap
The following features are planned for the future:
- Control plane optimisation
- Other administrative tools
- Web GUI
- OpenNebula plugin
- iSCSI proxy
- iSCSI and NVMeoF gateways
- Multi-threaded client
- Faster failover
- Scrubbing without checksums (verification of replicas)
- Checksums
- S3
- Tiered storage (SSD caching)
- NVDIMM support
- Compression (possibly)
- Read caching using system page cache (possibly)

View File

@ -13,7 +13,7 @@
## Серверные функции
- Базовая часть - надёжное кластерное блочное хранилище без единой точки отказа
- [Производительность](../comparison1.ru.md) ;-D
- [Производительность](../performance/bench2.ru.md) ;-D
- [Несколько схем отказоустойчивости](../config/pool.ru.md#scheme): репликация, XOR n+1 (1 диск чётности), коды коррекции ошибок
Рида-Соломона на основе библиотек jerasure и ISA-L с любым числом дисков данных и чётности в группе
- Конфигурация через простые человекочитаемые JSON-структуры в etcd
@ -31,12 +31,17 @@
- Снапшоты и copy-on-write клоны
- [Сглаживание производительности случайной записи в SSD+HDD конфигурациях](../config/osd.ru.md#throttle_small_writes)
- [Поддержка RDMA/RoCEv2 через libibverbs](../config/network.ru.md#rdma_device)
- [Фоновая проверка целостности](../config/osd.ru.md#auto_scrub) (сверка копий)
- [Контрольные суммы](../config/layout-osd.ru.md#data_csum_type)
- [Буферизация записи на стороне клиента](../config/client.ru.md#client_enable_writeback)
- [Интеллектуальная автоподстройка скорости восстановления](../config/osd.ru.md#recovery_tune_interval)
- [Кластерная файловая система](../usage/nfs.ru.md#vitastorfs)
## Драйверы и инструменты
- [Пакеты для Debian и CentOS](../installation/packages.ru.md)
- [Консольный интерфейс управления образами (vitastor-cli)](../usage/cli.ru.md)
- [Инструмент управления дисками (vitastor-disk)](docs/usage/disk.ru.md)
- [Инструмент управления дисками (vitastor-disk)](../usage/disk.ru.md)
- Общая пользовательская клиентская библиотека для работы с кластером
- [Драйвер диска для QEMU](../usage/qemu.ru.md)
- [Драйвер диска для утилиты тестирования производительности fio](../usage/fio.ru.md)
@ -44,18 +49,18 @@
- [CSI-плагин для Kubernetes](../installation/kubernetes.ru.md)
- [Базовая поддержка OpenStack: драйвер Cinder, патчи для Nova и libvirt](../installation/openstack.ru.md)
- [Плагин для Proxmox](../installation/proxmox.ru.md)
- [Упрощённая NFS-прокси для эмуляции файлового доступа к образам (подходит для VMWare)](../usage/nfs.ru.md)
- [Упрощённая NFS-прокси для эмуляции файлового доступа к образам (подходит для VMWare)](../usage/nfs.ru.md#псевдо-фс)
## Планы развития
- Оптимизация слоя управления
- Другие инструменты администрирования
- Web-интерфейс
- Плагин для OpenNebula
- iSCSI-прокси
- iSCSI и NVMeoF прокси
- Многопоточный клиент
- Более быстрое переключение при отказах
- Фоновая проверка целостности без контрольных сумм (сверка реплик)
- Контрольные суммы
- S3
- Поддержка SSD-кэширования (tiered storage)
- Поддержка NVDIMM
- Возможно, сжатие

View File

@ -7,22 +7,34 @@
# Quick Start
- [Preparation](#preparation)
- [Recommended drives](#recommended-drives)
- [Configure monitors](#configure-monitors)
- [Configure OSDs](#configure-osds)
- [Create a pool](#create-a-pool)
- [Check cluster status](#check-cluster-status)
- [Create an image](#create-an-image)
- [Install plugins](#install-plugins)
- [Create VitastorFS](#create-vitastorfs)
## Preparation
- Get some SATA or NVMe SSDs with capacitors (server-grade drives). You can use desktop SSDs
with lazy fsync, but prepare for inferior single-thread latency. Read more about capacitors
[here](../config/layout-cluster.en.md#immediate_commit).
- If you want to use HDDs, get modern HDDs with Media Cache or SSD Cache: HGST Ultrastar,
Toshiba MG08, Seagate EXOS or something similar. If your drives don't have such cache then
you also need small SSDs for journal and metadata (even 2 GB per 1 TB of HDD space is enough).
- Get a fast network (at least 10 Gbit/s). Something like Mellanox ConnectX-4 with RoCEv2 is ideal.
- Disable CPU powersaving: `cpupower idle-set -D 0 && cpupower frequency-set -g performance`.
- [Install Vitastor packages](../installation/packages.en.md).
## Recommended drives
- SATA SSD: Micron 5100/5200/5300/5400, Samsung PM863/PM883/PM893, Intel D3-S4510/4520/4610/4620, Kingston DC500M
- NVMe: Micron 9100/9200/9300/9400, Micron 7300/7450, Samsung PM983/PM9A3, Samsung PM1723/1735/1743,
Intel DC-P3700/P4500/P4600, Intel D7-P5500/P5600, Intel Optane, Kingston DC1000B/DC1500M
- HDD: HGST Ultrastar, Toshiba MG06/MG07/MG08, Seagate EXOS
## Configure monitors
On the monitor hosts:
@ -45,7 +57,10 @@ On the monitor hosts:
}
```
- Initialize OSDs:
- SSD-only: `vitastor-disk prepare /dev/sdXXX [/dev/sdYYY ...]`
- SSD-only or HDD-only: `vitastor-disk prepare /dev/sdXXX [/dev/sdYYY ...]`.
Add `--disable_data_fsync off` to leave disk write cache enabled if you use
desktop SSDs without capacitors. Do NOT add `--disable_data_fsync off` if you
use HDDs or SSD+HDD.
- Hybrid, SSD+HDD: `vitastor-disk prepare --hybrid /dev/sdXXX [/dev/sdYYY ...]`.
Pass all your devices (HDD and SSD) to this script &mdash; it will partition disks and initialize journals on its own.
This script skips HDDs which are already partitioned so if you want to use non-empty disks for
@ -53,28 +68,32 @@ On the monitor hosts:
but some free unpartitioned space must be available because the script creates new partitions for journals.
- You can change OSD configuration in units or in `vitastor.conf`.
Check [Configuration Reference](../config.en.md) for parameter descriptions.
- If all your drives have capacitors, create global configuration in etcd: \
- If all your drives have capacitors, and even if not, but if you ran `vitastor-disk`
without `--disable_data_fsync off` at the first step, then put the following
setting into etcd: \
`etcdctl --endpoints=... put /vitastor/config/global '{"immediate_commit":"all"}'`
- Start all OSDs: `systemctl start vitastor.target`
## Create a pool
Create pool configuration in etcd:
Create a pool using vitastor-cli:
```
etcdctl --endpoints=... put /vitastor/config/pools '{"1":{"name":"testpool",
"scheme":"replicated","pg_size":2,"pg_minsize":1,"pg_count":256,"failure_domain":"host"}}'
vitastor-cli create-pool testpool --pg_size 2 --pg_count 256
```
For EC pools the configuration should look like the following:
```
etcdctl --endpoints=... put /vitastor/config/pools '{"2":{"name":"ecpool",
"scheme":"ec","pg_size":4,"parity_chunks":2,"pg_minsize":2,"pg_count":256,"failure_domain":"host"}'
vitastor-cli create-pool testpool --ec 2+2 --pg_count 256
```
After you do this, one of the monitors will configure PGs and OSDs will start them.
If you use HDDs you should also add `"block_size": 1048576` to pool configuration.
The other option is to add it into /vitastor/config/global, in this case it will
apply to all pools by default.
## Check cluster status
`vitastor-cli status`
@ -96,3 +115,9 @@ After that, you can [run benchmarks](../usage/fio.en.md) or [start QEMU manually
- [Proxmox](../installation/proxmox.en.md)
- [OpenStack](../installation/openstack.en.md)
- [Kubernetes CSI](../installation/kubernetes.en.md)
## Create VitastorFS
If you want to use clustered file system in addition to VM or container images:
- [Follow the instructions here](../usage/nfs.en.md#vitastorfs)

View File

@ -7,22 +7,34 @@
# Быстрый старт
- [Подготовка](#подготовка)
- [Рекомендуемые диски](#рекомендуемые-диски)
- [Настройте мониторы](#настройте-мониторы)
- [Настройте OSD](#настройте-osd)
- [Создайте пул](#создайте-пул)
- [Проверьте состояние кластера](#проверьте-состояние-кластера)
- [Создайте образ](#создайте-образ)
- [Установите плагины](#установите-плагины)
- [Создайте VitastorFS](#создайте-vitastorfs)
## Подготовка
- Возьмите серверы с SSD (SATA или NVMe), желательно с конденсаторами (серверные SSD). Можно
использовать и десктопные SSD, включив режим отложенного fsync, но производительность будет хуже.
О конденсаторах читайте [здесь](../config/layout-cluster.ru.md#immediate_commit).
- Если хотите использовать HDD, берите современные модели с Media или SSD кэшем - HGST Ultrastar,
Toshiba MG08, Seagate EXOS или что-то похожее. Если такого кэша у ваших дисков нет,
обязательно возьмите SSD под метаданные и журнал (маленькие, буквально 2 ГБ на 1 ТБ HDD-места).
- Возьмите быструю сеть, минимум 10 гбит/с. Идеал - что-то вроде Mellanox ConnectX-4 с RoCEv2.
- Для лучшей производительности отключите энергосбережение CPU: `cpupower idle-set -D 0 && cpupower frequency-set -g performance`.
- [Установите пакеты Vitastor](../installation/packages.ru.md).
## Рекомендуемые диски
- SATA SSD: Micron 5100/5200/5300/5400, Samsung PM863/PM883/PM893, Intel D3-S4510/4520/4610/4620, Kingston DC500M
- NVMe: Micron 9100/9200/9300/9400, Micron 7300/7450, Samsung PM983/PM9A3, Samsung PM1723/1735/1743,
Intel DC-P3700/P4500/P4600, Intel D7-P5500/P5600, Intel Optane, Kingston DC1000B/DC1500M
- HDD: HGST Ultrastar, Toshiba MG06/MG07/MG08, Seagate EXOS
## Настройте мониторы
На хостах, выделенных под мониторы:
@ -45,7 +57,10 @@
}
```
- Инициализуйте OSD:
- SSD: `vitastor-disk prepare /dev/sdXXX [/dev/sdYYY ...]`
- Только SSD или только HDD: `vitastor-disk prepare /dev/sdXXX [/dev/sdYYY ...]`.
Если вы используете десктопные SSD без конденсаторов, добавьте опцию `--disable_data_fsync off`,
чтобы оставить кэш записи диска включённым. НЕ добавляйте эту опцию, если используете
жёсткие диски (HDD).
- Гибридные, SSD+HDD: `vitastor-disk prepare --hybrid /dev/sdXXX [/dev/sdYYY ...]`.
Передайте все ваши SSD и HDD скрипту в командной строке подряд, скрипт автоматически выделит
разделы под журналы на SSD и данные на HDD. Скрипт пропускает HDD, на которых уже есть разделы
@ -54,28 +69,33 @@
для журналов, на SSD должно быть доступно свободное нераспределённое место.
- Вы можете менять параметры OSD в юнитах systemd или в `vitastor.conf`. Описания параметров
смотрите в [справке по конфигурации](../config.ru.md).
- Если все ваши диски - серверные с конденсаторами, пропишите это в глобальную конфигурацию в etcd: \
`etcdctl --endpoints=... put /vitastor/config/global '{"immediate_commit":"all"}'`
- Если все ваши диски - серверные с конденсаторами, и даже если нет, но при этом
вы не добавляли опцию `--disable_data_fsync off` на первом шаге, а `vitastor-disk`
не ругался на невозможность отключения кэша дисков, пропишите следующую настройку
в глобальную конфигурацию в etcd: \
`etcdctl --endpoints=... put /vitastor/config/global '{"immediate_commit":"all"}'`.
- Запустите все OSD: `systemctl start vitastor.target`
## Создайте пул
Создайте конфигурацию пула с помощью etcdctl:
Создайте пул с помощью vitastor-cli:
```
etcdctl --endpoints=... put /vitastor/config/pools '{"1":{"name":"testpool",
"scheme":"replicated","pg_size":2,"pg_minsize":1,"pg_count":256,"failure_domain":"host"}}'
vitastor-cli create-pool testpool --pg_size 2 --pg_count 256
```
Для пулов с кодами коррекции ошибок конфигурация должна выглядеть примерно так:
```
etcdctl --endpoints=... put /vitastor/config/pools '{"2":{"name":"ecpool",
"scheme":"ec","pg_size":4,"parity_chunks":2,"pg_minsize":2,"pg_count":256,"failure_domain":"host"}'
vitastor-cli create-pool testpool --ec 2+2 --pg_count 256
```
После этого один из мониторов должен сконфигурировать PG, а OSD должны запустить их.
Если вы используете HDD-диски, то добавьте в конфигурацию пулов опцию `"block_size": 1048576`.
Также эту опцию можно добавить в /vitastor/config/global, в этом случае она будет
применяться ко всем пулам по умолчанию.
## Проверьте состояние кластера
`vitastor-cli status`
@ -97,3 +117,10 @@ vitastor-cli create -s 10G testimg
- [Proxmox](../installation/proxmox.ru.md)
- [OpenStack](../installation/openstack.ru.md)
- [Kubernetes CSI](../installation/kubernetes.ru.md)
## Создайте VitastorFS
Если вы хотите использовать не только блочные образы виртуальных машин или контейнеров,
а также кластерную файловую систему, то:
- [Следуйте инструкциям](../usage/nfs.en.md#vitastorfs)

View File

@ -0,0 +1,154 @@
[Documentation](../../README.md#documentation) → Performance → Newer benchmark of Vitastor 1.3.1
-----
[Читать на русском](bench2.ru.md)
# Newer benchmark of Vitastor 1.3.1
- [Test environment](#test-environment)
- [Notes](#notes)
- [Raw drive performance](#raw-drive-performance)
- [2 replicas](#2-replicas)
- [3 replicas](#3-replicas)
- [EC 2+1](#ec-2-1)
## Test environment
Hardware configuration: 3 nodes, each with:
- 8x NVMe Samsung PM9A3 1.92 TB
- 2x Xeon Gold 6342 (24 cores @ 2.8 GHz)
- 256 GB RAM
- Dual-port 25 GbE Mellanox ConnectX-4 LX network card with RoCEv2
- Connected to 2 Mellanox SN2010 switches with MLAG
## Notes
Vitastor version was 1.3.1.
Tests were ran from the storage nodes - 4 fio clients per each of 3 nodes.
The same large 3 TB image was tested from all hosts because Vitastor has no
performance penalties related to running multiple clients over a single inode.
CPU power saving was disabled. 4 OSDs were created per each NVMe.
Checksums were not enabled. Tests with checksums will be conducted later,
along with the newer version of Vitastor, and results will be updated.
CPU configuration was not optimal because of NUMA. It's better to avoid 2-socket
platforms. It was especially noticeable in RDMA tests - in the form of ksoftirqd
processes (usually 1 per server) eating 100 % of one CPU core and actual bandwidth
of one network port reduced to 3-5 Gbit/s instead of 25 Gbit/s - probably because
of RFS (Receive Flow Steering) misses. Many network configurations were tried during
tests, but nothing helped to solve this problem, so final tests were conducted with
the default settings.
# Raw drive performance
- Linear write ~1000-2000 MB/s, depending on current state of the drive's garbage collector
- Linear read ~3300 MB/s
- T1Q1 random write ~60000 iops (latency ~0.015ms)
- T1Q1 random read ~14700 iops (latency ~0.066ms)
- T1Q16 random write ~180000 iops
- T1Q16 random read ~120000 iops
- T1Q32 random write ~180000 iops
- T1Q32 random read ~195000 iops
- T1Q128 random write ~180000 iops
- T1Q128 random read ~195000 iops
- T4Q128 random write ~525000 iops
- T4Q128 random read ~750000 iops
These numbers make obvious that results could be much better if a faster network
was available, because NVMe drives obviously weren't a bottleneck. For example,
theoretical maximum linear read performance for 24 drives could be 79.2 GByte/s,
which is 633 Gbit/s. Real Vitastor read speed (both linear and random) was around
16 Gbyte/s, which is 130 Gbit/s. It's important to note that it was still much
larger than the network bandwidth of one server (50 Gbit/s). This is also correct
because tests were conducted from all 3 nodes.
## 2 replicas
| | TCP | RDMA |
|------------------------------|--------------|--------------|
| Linear read (4M T6 Q16) | 13.13 GB/s | 16.25 GB/s |
| Linear write (4M T6 Q16) | 8.16 GB/s | 7.88 GB/s |
| Read 4k T1 Q1 | 8745 iops | 10252 iops |
| Write 4k T1 Q1 | 8097 iops | 11488 iops |
| Read 4k T12 Q128 | 1305936 iops | 4265861 iops |
| Write 4k T12 Q128 | 660490 iops | 1384033 iops |
CPU consumption OSD per 1 disk:
| | TCP | RDMA |
|------------------------------|---------|---------|
| Linear read (4M T6 Q16) | 29.7 % | 29.8 % |
| Linear write (4M T6 Q16) | 84.4 % | 33.2 % |
| Read 4k T12 Q128 | 98.4 % | 119.1 % |
| Write 4k T12 Q128 | 173.4 % | 175.9 % |
CPU consumption per 1 client (fio):
| | TCP | RDMA |
|------------------------------|--------|--------|
| Linear read (4M T6 Q16) | 100 % | 85.2 % |
| Linear write (4M T6 Q16) | 55.8 % | 48.8 % |
| Read 4k T12 Q128 | 99.9 % | 96 % |
| Write 4k T12 Q128 | 71.6 % | 48.5 % |
## 3 replicas
| | TCP | RDMA |
|------------------------------|--------------|--------------|
| Linear read (4M T6 Q16) | 13.98 GB/s | 16.54 GB/s |
| Linear write (4M T6 Q16) | 5.38 GB/s | 5.7 GB/s |
| Read 4k T1 Q1 | 8969 iops | 9980 iops |
| Write 4k T1 Q1 | 8126 iops | 11672 iops |
| Read 4k T12 Q128 | 1358818 iops | 4279088 iops |
| Write 4k T12 Q128 | 433890 iops | 993506 iops |
CPU consumption OSD per 1 disk:
| | TCP | RDMA |
|------------------------------|--------|---------|
| Linear read (4M T6 Q16) | 24.9 % | 25.4 % |
| Linear write (4M T6 Q16) | 99.3 % | 38.4 % |
| Read 4k T12 Q128 | 95.3 % | 111.7 % |
| Write 4k T12 Q128 | 173 % | 194 % |
CPU consumption per 1 client (fio):
| | TCP | RDMA |
|------------------------------|--------|--------|
| Linear read (4M T6 Q16) | 99.9 % | 85.8 % |
| Linear write (4M T6 Q16) | 38.9 % | 38.1 % |
| Read 4k T12 Q128 | 100 % | 96.1 % |
| Write 4k T12 Q128 | 51.6 % | 41.9 % |
## EC 2+1
| | TCP | RDMA |
|------------------------------|--------------|--------------|
| Linear read (4M T6 Q16) | 10.07 GB/s | 11.43 GB/s |
| Linear write (4M T6 Q16) | 7.74 GB/s | 8.32 GB/s |
| Read 4k T1 Q1 | 7408 iops | 8891 iops |
| Write 4k T1 Q1 | 3525 iops | 4903 iops |
| Read 4k T12 Q128 | 1216496 iops | 2552765 iops |
| Write 4k T12 Q128 | 278110 iops | 821261 iops |
CPU consumption OSD per 1 disk:
| | TCP | RDMA |
|------------------------------|---------|---------|
| Linear read (4M T6 Q16) | 68.6 % | 33.6 % |
| Linear write (4M T6 Q16) | 108.3 % | 50.2 % |
| Read 4k T12 Q128 | 138.1 % | 97.9 % |
| Write 4k T12 Q128 | 168.7 % | 188.5 % |
CPU consumption per 1 client (fio):
| | TCP | RDMA |
|------------------------------|--------|--------|
| Linear read (4M T6 Q16) | 88.2 % | 52.4 % |
| Linear write (4M T6 Q16) | 51.8 % | 46.8 % |
| Read 4k T12 Q128 | 99.7 % | 61.3 % |
| Write 4k T12 Q128 | 35.1 % | 31.3 % |

View File

@ -0,0 +1,157 @@
[Документация](../../README-ru.md#документация) → Производительность → Более новый тест Vitastor 1.3.1
-----
[Read in English](bench2.en.md)
# Более новый тест Vitastor 1.3.1
- [Описание стенда](#описание-стенда)
- [Примечания](#примечания)
- [Производительность голых дисков](#производительность-голых-дисков)
- [2 реплики](#2-реплики)
- [3 реплики](#3-реплики)
- [EC 2+1](#ec-2-1)
## Описание стенда
Железо: 3 сервера, в каждом:
- 8x NVMe Samsung PM9A3 1.92 TB
- 2x Xeon Gold 6342 (24 ядра @ 2.8 GHz)
- 256 GB RAM
- Двухпортовая 25 GbE сетевая карта Mellanox ConnectX-4 LX с поддержкой RoCEv2
- Подключение к 2 коммутаторам Mellanox SN2010 в MLAG
## Примечания
Версия Vitastor 1.3.1.
Тесты проводились с самих серверов хранения - по 4 клиента fio с каждого из 3 серверов.
Тестировался один большой образ размером 3 ТБ со всех хостов - создавать отдельные образы
для тестов в Vitastor необязательно, т.к. в Vitastor нет замедления при записи в один
узел несколькими клиентами.
Экономия энергии CPU отключена. На каждый NVMe создавалось 4 OSD.
Контрольные суммы не включались. Тесты с контрольными суммами будут проведены
позднее. Тогда же будет протестирована более новая версия Vitastor, и результаты
будут обновлены.
Конфигурация CPU стенда неоптимальна из-за NUMA - двухпроцессорных серверов лучше
избегать. Особенно это проявлялось во время тестов с RDMA - выражалось это в потреблении
100% одного ядра CPU одним процессом ksoftirqd и работой одного из двух сетевых портов
на скорости 3-5 ГБит/с вместо 25 ГБит/с - предположительно, из-за "непопаданий" RFS
(Receive Flow Steering) на нужные ядра. Решить эту проблему во время проведения тестов
не получилось. Было перепробовано множество различных настроек, но в итоге тесты проведены
с настройками по умолчанию, т.к. улучшения добиться не удалось.
# Производительность голых дисков
- Линейная запись ~1000-2000 МБ/с, в зависимости от состояния сборщика мусора диска
- Линейное чтение ~3300 МБ/с
- T1Q1 запись ~60000 iops (задержка ~0.015ms)
- T1Q1 чтение ~14700 iops (задержка ~0.066ms)
- T1Q16 запись ~180000 iops
- T1Q16 чтение ~120000 iops
- T1Q32 запись ~180000 iops
- T1Q32 чтение ~195000 iops
- T1Q128 запись ~180000 iops
- T1Q128 чтение ~195000 iops
- T4Q128 запись ~525000 iops
- T4Q128 чтение ~750000 iops
Из данных цифр очевидно, что при наличии более быстрой сети результаты были бы
значительно лучше, так как в диски тест, очевидно, не упирался. Например, теоретический предел по
линейному чтению для 24 таких дисков был бы около 79.2 ГБайт/с, то есть,
633 гигабита в секунду. Реальная скорость чтения (и случайного, и линейного)
Vitastor составила примерно 16 ГБайт/с, то есть 130 гигабит в секунду. При этом
следует заметить, что этот результат всё равно значительно лучше пропускной способности
сети отдельно взятого узла - что тоже вполне логично, так как тест выполнялся со
всех трёх узлов.
## 2 реплики
| | TCP | RDMA |
|------------------------------|--------------|--------------|
| Линейное чтение (4M T6 Q16) | 13.13 ГБ/с | 16.25 ГБ/с |
| Линейная запись (4M T6 Q16) | 8.16 ГБ/с | 7.88 ГБ/с |
| Чтение 4k T1 Q1 | 8745 iops | 10252 iops |
| Запись 4k T1 Q1 | 8097 iops | 11488 iops |
| Чтение 4k T12 Q128 | 1305936 iops | 4265861 iops |
| Запись 4k T12 Q128 | 660490 iops | 1384033 iops |
Потребление CPU OSD на 1 диск:
| | TCP | RDMA |
|------------------------------|---------|---------|
| Линейное чтение (4M T6 Q16) | 29.7 % | 29.8 % |
| Линейная запись (4M T6 Q16) | 84.4 % | 33.2 % |
| Чтение 4k T12 Q128 | 98.4 % | 119.1 % |
| Запись 4k T12 Q128 | 173.4 % | 175.9 % |
Потребление CPU на 1 клиента (fio):
| | TCP | RDMA |
|------------------------------|--------|--------|
| Линейное чтение (4M T6 Q16) | 100 % | 85.2 % |
| Линейная запись (4M T6 Q16) | 55.8 % | 48.8 % |
| Чтение 4k T12 Q128 | 99.9 % | 96 % |
| Запись 4k T12 Q128 | 71.6 % | 48.5 % |
## 3 реплики
| | TCP | RDMA |
|------------------------------|--------------|--------------|
| Линейное чтение (4M T6 Q16) | 13.98 ГБ/с | 16.54 ГБ/с |
| Линейная запись (4M T6 Q16) | 5.38 ГБ/с | 5.7 ГБ/с |
| Чтение 4k T1 Q1 | 8969 iops | 9980 iops |
| Запись 4k T1 Q1 | 8126 iops | 11672 iops |
| Чтение 4k T12 Q128 | 1358818 iops | 4279088 iops |
| Запись 4k T12 Q128 | 433890 iops | 993506 iops |
Потребление CPU OSD на 1 диск:
| | TCP | RDMA |
|------------------------------|--------|---------|
| Линейное чтение (4M T6 Q16) | 24.9 % | 25.4 % |
| Линейная запись (4M T6 Q16) | 99.3 % | 38.4 % |
| Чтение 4k T12 Q128 | 95.3 % | 111.7 % |
| Запись 4k T12 Q128 | 173 % | 194 % |
Потребление CPU на 1 клиента (fio):
| | TCP | RDMA |
|------------------------------|--------|--------|
| Линейное чтение (4M T6 Q16) | 99.9 % | 85.8 % |
| Линейная запись (4M T6 Q16) | 38.9 % | 38.1 % |
| Чтение 4k T12 Q128 | 100 % | 96.1 % |
| Запись 4k T12 Q128 | 51.6 % | 41.9 % |
## EC 2+1
| | TCP | RDMA |
|------------------------------|--------------|--------------|
| Линейное чтение (4M T6 Q16) | 10.07 ГБ/с | 11.43 ГБ/с |
| Линейная запись (4M T6 Q16) | 7.74 ГБ/с | 8.32 ГБ/с |
| Чтение 4k T1 Q1 | 7408 iops | 8891 iops |
| Запись 4k T1 Q1 | 3525 iops | 4903 iops |
| Чтение 4k T12 Q128 | 1216496 iops | 2552765 iops |
| Запись 4k T12 Q128 | 278110 iops | 821261 iops |
Потребление CPU OSD на 1 диск:
| | TCP | RDMA |
|------------------------------|---------|---------|
| Линейное чтение (4M T6 Q16) | 68.6 % | 33.6 % |
| Линейная запись (4M T6 Q16) | 108.3 % | 50.2 % |
| Чтение 4k T12 Q128 | 138.1 % | 97.9 % |
| Запись 4k T12 Q128 | 168.7 % | 188.5 % |
Потребление CPU на 1 клиента (fio):
| | TCP | RDMA |
|------------------------------|--------|--------|
| Линейное чтение (4M T6 Q16) | 88.2 % | 52.4 % |
| Линейная запись (4M T6 Q16) | 51.8 % | 46.8 % |
| Чтение 4k T12 Q128 | 99.7 % | 61.3 % |
| Запись 4k T12 Q128 | 35.1 % | 31.3 % |

View File

@ -11,19 +11,26 @@ Replicated setups:
- Single-threaded write+fsync latency:
- With immediate commit: 2 network roundtrips + 1 disk write.
- With lazy commit: 4 network roundtrips + 1 disk write + 1 disk flush.
- Saturated parallel read iops: min(network bandwidth, sum(disk read iops)).
- Saturated parallel write iops: min(network bandwidth, sum(disk write iops / number of replicas / write amplification)).
- Linear read: `min(total network bandwidth, sum(disk read MB/s))`.
- Linear write: `min(total network bandwidth, sum(disk write MB/s / number of replicas))`.
- Saturated parallel read iops: `min(total network bandwidth, sum(disk read iops))`.
- Saturated parallel write iops: `min(total network bandwidth / number of replicas, sum(disk write iops / number of replicas / (write amplification = 4)))`.
EC/XOR setups:
EC/XOR setups (EC N+K):
- Single-threaded (T1Q1) read latency: 1.5 network roundtrips + 1 disk read.
- Single-threaded write+fsync latency:
- With immediate commit: 3.5 network roundtrips + 1 disk read + 2 disk writes.
- With lazy commit: 5.5 network roundtrips + 1 disk read + 2 disk writes + 2 disk fsyncs.
- 0.5 in actually (k-1)/k which means that an additional roundtrip doesn't happen when
- 0.5 in actually `(N-1)/N` which means that an additional roundtrip doesn't happen when
the read sub-operation can be served locally.
- Saturated parallel read iops: min(network bandwidth, sum(disk read iops)).
- Saturated parallel write iops: min(network bandwidth, sum(disk write iops * number of data drives / (number of data + parity drives) / write amplification)).
In fact, you should put disk write iops under the condition of ~10% reads / ~90% writes in this formula.
- Linear read: `min(total network bandwidth, sum(disk read MB/s))`.
- Linear write: `min(total network bandwidth, sum(disk write MB/s * N/(N+K)))`.
- Saturated parallel read iops: `min(total network bandwidth, sum(disk read iops))`.
- Saturated parallel write iops: roughly `total iops / (N+K) / WA`. More exactly,
`min(total network bandwidth * N/(N+K), sum(disk randrw iops / (N*4 + K*5 + 1)))` with
random read/write mix corresponding to `(N-1)/(N*4 + K*5 + 1)*100 % reads`.
- For example, with EC 2+1 it is: `(7% randrw iops) / 14`.
- With EC 6+3 it is: `(12.5% randrw iops) / 40`.
Write amplification for 4 KB blocks is usually 3-5 in Vitastor:
1. Journal block write
@ -35,15 +42,24 @@ Write amplification for 4 KB blocks is usually 3-5 in Vitastor:
If you manage to get an SSD which handles 512 byte blocks well (Optane?) you may
lower 1, 3 and 4 to 512 bytes (1/8 of data size) and get WA as low as 2.375.
Implemented NVDIMM support can basically eliminate WA at all - all extra writes will
go to DRAM memory. But this requires a test cluster with NVDIMM - please contact me
if you want to provide me with such cluster for tests.
Lazy fsync also reduces WA for parallel workloads because journal blocks are only
written when they fill up or fsync is requested.
## In Practice
In practice, using tests from [Understanding Performance](understanding.en.md)
and good server-grade SSD/NVMe drives, you should head for:
In practice, using tests from [Understanding Performance](understanding.en.md), decent TCP network,
good server-grade SSD/NVMe drives and disabled CPU power saving, you should head for:
- At least 5000 T1Q1 replicated read and write iops (maximum 0.2ms latency)
- At least 5000 T1Q1 EC read IOPS and at least 2200 EC write IOPS (maximum 0.45ms latency)
- At least ~80k parallel read iops or ~30k write iops per 1 core (1 OSD)
- Disk-speed or wire-speed linear reads and writes, whichever is the bottleneck in your case
Lower results may mean that you have bad drives, bad network or some kind of misconfiguration.
Current latency records:
- 9668 T1Q1 replicated write iops (0.103 ms latency) with TCP and NVMe
- 9143 T1Q1 replicated read iops (0.109 ms latency) with TCP and NVMe

View File

@ -11,20 +11,27 @@
- Запись+fsync в 1 поток:
- С мгновенным сбросом: 2 RTT + 1 запись.
- С отложенным ("ленивым") сбросом: 4 RTT + 1 запись + 1 fsync.
- Параллельное чтение: сумма IOPS всех дисков либо производительность сети, если в сеть упрётся раньше.
- Параллельная запись: сумма IOPS всех дисков / число реплик / WA либо производительность сети, если в сеть упрётся раньше.
- Линейное чтение: сумма МБ/с чтения всех дисков, либо общая производительность сети (сумма пропускной способности сети всех нод), если в сеть упрётся раньше.
- Линейная запись: сумма МБ/с записи всех дисков / число реплик, либо производительность сети / число реплик, если в сеть упрётся раньше.
- Параллельное случайное мелкое чтение: сумма IOPS чтения всех дисков, либо производительность сети, если в сеть упрётся раньше.
- Параллельная случайная мелкая запись: сумма IOPS записи всех дисков / число реплик / WA, либо производительность сети / число реплик, если в сеть упрётся раньше.
При использовании кодов коррекции ошибок (EC):
При использовании кодов коррекции ошибок (EC N+K):
- Задержка чтения в 1 поток (T1Q1): 1.5 RTT + 1 чтение.
- Запись+fsync в 1 поток:
- С мгновенным сбросом: 3.5 RTT + 1 чтение + 2 записи.
- С отложенным ("ленивым") сбросом: 5.5 RTT + 1 чтение + 2 записи + 2 fsync.
- Под 0.5 на самом деле подразумевается (k-1)/k, где k - число дисков данных,
- Под 0.5 на самом деле подразумевается (N-1)/N, где N - число дисков данных,
что означает, что дополнительное обращение по сети не нужно, когда операция
чтения обслуживается локально.
- Параллельное чтение: сумма IOPS всех дисков либо производительность сети, если в сеть упрётся раньше.
- Параллельная запись: сумма IOPS всех дисков / общее число дисков данных и чётности / WA либо производительность сети, если в сеть упрётся раньше.
Примечание: IOPS дисков в данном случае надо брать в смешанном режиме чтения/записи в пропорции, аналогичной формулам выше.
- Линейное чтение: сумма МБ/с чтения всех дисков, либо общая производительность сети, если в сеть упрётся раньше.
- Линейная запись: сумма МБ/с записи всех дисков * N/(N+K), либо производительность сети * N / (N+K), если в сеть упрётся раньше.
- Параллельное случайное мелкое чтение: сумма IOPS чтения всех дисков либо производительность сети, если в сеть упрётся раньше.
- Параллельная случайная мелкая запись: грубо `(сумма IOPS / (N+K) / WA)`. Если точнее, то:
сумма смешанного IOPS всех дисков при `(N-1)/(N*4 + K*5 + 1)*100 %` чтения, делённая на `(N*4 + K*5 + 1)`.
Либо, производительность сети * N/(N+K), если в сеть упрётся раньше.
- Например, при EC 2+1 это: `(сумма IOPS при 7% чтения) / 14`.
- При EC 6+3 это: `(сумма IOPS при 12.5% чтения) / 40`.
WA (мультипликатор записи) для 4 КБ блоков в Vitastor обычно составляет 3-5:
1. Запись метаданных в журнал
@ -36,6 +43,25 @@ WA (мультипликатор записи) для 4 КБ блоков в Vit
Если вы найдёте SSD, хорошо работающий с 512-байтными блоками данных (Optane?),
то 1, 3 и 4 можно снизить до 512 байт (1/8 от размера данных) и получить WA всего 2.375.
Если реализовать поддержку NVDIMM, то WA можно, условно говоря, ликвидировать вообще - все
дополнительные операции записи смогут обслуживаться DRAM памятью. Но для этого необходим
тестовый кластер с NVDIMM - пишите, если готовы предоставить такой для тестов.
Кроме того, WA снижается при использовании отложенного/ленивого сброса при параллельной
нагрузке, т.к. блоки журнала записываются на диск только когда они заполняются или явным
образом запрашивается fsync.
## На практике
На практике, используя тесты fio со страницы [Понимание сути производительности систем хранения](understanding.ru.md),
нормальную TCP-сеть, хорошие серверные SSD/NVMe, при отключённом энергосбережении процессоров вы можете рассчитывать на:
- От 5000 IOPS в 1 поток (T1Q1) и на чтение, и на запись при использовании репликации (задержка до 0.2мс)
- От 5000 IOPS в 1 поток (T1Q1) на чтение и 2200 IOPS в 1 поток на запись при использовании EC (задержка до 0.45мс)
- От 80000 IOPS на чтение в параллельном режиме на 1 ядро, от 30000 IOPS на запись на 1 ядро (на 1 OSD)
- Скорость параллельного линейного чтения и записи, равная меньшему значению из скорости дисков или сети
Худшие результаты означают, что у вас либо медленные диски, либо медленная сеть, либо что-то неправильно настроено.
Зафиксированный на данный момент рекорд задержки:
- 9668 IOPS (0.103 мс задержка) в 1 поток (T1Q1) на запись с TCP и NVMe при использовании репликации
- 9143 IOPS (0.109 мс задержка) в 1 поток (T1Q1) на чтение с TCP и NVMe при использовании репликации

View File

@ -14,18 +14,26 @@ It supports the following commands:
- [df](#df)
- [ls](#ls)
- [create](#create)
- [snap-create](#create)
- [modify](#modify)
- [rm](#rm)
- [flatten](#flatten)
- [rm-data](#rm-data)
- [merge-data](#merge-data)
- [describe](#describe)
- [fix](#fix)
- [alloc-osd](#alloc-osd)
- [rm-osd](#rm-osd)
- [create-pool](#create-pool)
- [modify-pool](#modify-pool)
- [ls-pools](#ls-pools)
- [rm-pool](#rm-pool)
Global options:
```
--etcd_address ADDR Etcd connection address
--config_file FILE Path to Vitastor configuration file
--etcd_address URL Etcd connection address
--iodepth N Send N operations in parallel to each OSD when possible (default 32)
--parallel_osds M Work with M osds in parallel when possible (default 4)
--progress 1|0 Report progress (default 1)
@ -123,21 +131,22 @@ vitastor-cli snap-create [-p|--pool <id|name>] <image>@<snapshot>
Create a snapshot of image `<name>` (either form can be used). May be used live if only a single writer is active.
See also about [how to export snapshots](qemu.en.md#exporting-snapshots).
## modify
`vitastor-cli modify <name> [--rename <new-name>] [--resize <size>] [--readonly | --readwrite] [-f|--force]`
`vitastor-cli modify <name> [--rename <new-name>] [--resize <size>] [--readonly | --readwrite] [-f|--force] [--down-ok]`
Rename, resize image or change its readonly status. Images with children can't be made read-write.
If the new size is smaller than the old size, extra data will be purged.
You should resize file system in the image, if present, before shrinking it.
```
-f|--force Proceed with shrinking or setting readwrite flag even if the image has children.
```
* `-f|--force` - Proceed with shrinking or setting readwrite flag even if the image has children.
* `--down-ok` - Proceed with shrinking even if some data will be left on unavailable OSDs.
## rm
`vitastor-cli rm <from> [<to>] [--writers-stopped]`
`vitastor-cli rm <from> [<to>] [--writers-stopped] [--down-ok]`
Remove `<from>` or all layers between `<from>` and `<to>` (`<to>` must be a child of `<from>`),
rebasing all their children accordingly. --writers-stopped allows merging to be a bit
@ -145,6 +154,10 @@ more effective in case of a single 'slim' read-write child and 'fat' removed par
the child is merged into parent and parent is renamed to child in that case.
In other cases parent layers are always merged into children.
Other options:
* `--down-ok` - Continue deletion/merging even if some data will be left on unavailable OSDs.
## flatten
`vitastor-cli flatten <layer>`
@ -171,6 +184,51 @@ Merge layer data without changing metadata. Merge `<from>`..`<to>` to `<target>`
`<to>` must be a child of `<from>` and `<target>` may be one of the layers between
`<from>` and `<to>`, including `<from>` and `<to>`.
## describe
`vitastor-cli describe [--osds <osds>] [--object-state <states>] [--pool <pool>]
[--inode <ino>] [--min-inode <ino>] [--max-inode <ino>]
[--min-offset <offset>] [--max-offset <offset>]`
Describe unclean object locations in the cluster.
```
--osds <osds>
Only list objects from primary OSD(s) <osds>.
--object-state <states>
Only list objects in given state(s). State(s) may include:
degraded, misplaced, incomplete, corrupted, inconsistent.
--pool <pool name or number>
Only list objects in the given pool.
--inode, --min-inode, --max-inode
Restrict listing to specific inode numbers.
--min-offset, --max-offset
Restrict listing to specific offsets inside inodes.
```
## fix
`vitastor-cli fix [--objects <objects>] [--bad-osds <osds>] [--part <part>] [--check no]`
Fix inconsistent objects in the cluster by deleting some copies.
```
--objects <objects>
Objects to fix, either in plain text or JSON format. If not specified,
object list will be read from STDIN in one of the same formats.
Plain text format: 0x<inode>:0x<stripe> <any delimiter> 0x<inode>:0x<stripe> ...
JSON format: [{"inode":"0x...","stripe":"0x..."},...]
--bad-osds <osds>
Remove inconsistent copies/parts of objects from these OSDs, effectively
marking them bad and allowing Vitastor to recover objects from other copies.
--part <number>
Only remove EC part <number> (from 0 to pg_size-1), required for extreme
edge cases where one OSD has multiple parts of a EC object.
--check no
Do not recheck that requested objects are actually inconsistent,
delete requested copies/parts anyway.
```
## alloc-osd
`vitastor-cli alloc-osd`
@ -187,3 +245,91 @@ Refuses to remove OSDs with data without `--force` and `--allow-data-loss`.
With `--dry-run` only checks if deletion is possible without data loss and
redundancy degradation.
## create-pool
`vitastor-cli create-pool|pool-create <name> (-s <pg_size>|--ec <N>+<K>) -n <pg_count> [OPTIONS]`
Create a pool. Required parameters:
| <!-- --> | <!-- --> |
|--------------------------|---------------------------------------------------------------------------------------|
| `-s R` or `--pg_size R` | Number of replicas for replicated pools |
| `--ec N+K` | Number of data (N) and parity (K) chunks for erasure-coded pools |
| `-n N` or `--pg_count N` | PG count for the new pool (start with 10*<OSD count>/pg_size rounded to a power of 2) |
Optional parameters:
| <!-- --> | <!-- --> |
|--------------------------------|----------------------------------------------------------------------------|
| `--pg_minsize <number>` | R or N+K minus number of failures to tolerate without downtime ([details](../config/pool.en.md#pg_minsize)) |
| `--failure_domain host` | Failure domain: host, osd or a level from placement_levels. Default: host |
| `--root_node <node>` | Put pool only on child OSDs of this placement tree node |
| `--osd_tags <tag>[,<tag>]...` | Put pool only on OSDs tagged with all specified tags |
| `--block_size 128k` | Put pool only on OSDs with this data block size |
| `--bitmap_granularity 4k` | Put pool only on OSDs with this logical sector size |
| `--immediate_commit none` | Put pool only on OSDs with this or larger immediate_commit (none < small < all) |
| `--primary_affinity_tags tags` | Prefer to put primary copies on OSDs with all specified tags |
| `--scrub_interval <time>` | Enable regular scrubbing for this pool. Format: number + unit s/m/h/d/M/y |
| `--used_for_fs <name>` | Mark pool as used for VitastorFS with metadata in image <name> |
| `--pg_stripe_size <number>` | Increase object grouping stripe |
| `--max_osd_combinations 10000` | Maximum number of random combinations for LP solver input |
| `--wait` | Wait for the new pool to come online |
| `-f` or `--force` | Do not check that cluster has enough OSDs to create the pool |
See also [Pool configuration](../config/pool.en.md) for detailed parameter descriptions.
Examples:
`vitastor-cli create-pool test_x4 -s 4 -n 32`
`vitastor-cli create-pool test_ec42 --ec 4+2 -n 32`
## modify-pool
`vitastor-cli modify-pool|pool-modify <id|name> [--name <new_name>] [PARAMETERS...]`
Modify an existing pool. Modifiable parameters:
```
[-s|--pg_size <number>] [--pg_minsize <number>] [-n|--pg_count <count>]
[--failure_domain <level>] [--root_node <node>] [--osd_tags <tags>] [--no_inode_stats 0|1]
[--max_osd_combinations <number>] [--primary_affinity_tags <tags>] [--scrub_interval <time>]
```
Non-modifiable parameters (changing them WILL lead to data loss):
```
[--block_size <size>] [--bitmap_granularity <size>]
[--immediate_commit <all|small|none>] [--pg_stripe_size <size>]
```
These, however, can still be modified with -f|--force.
See [create-pool](#create-pool) for parameter descriptions.
Examples:
`vitastor-cli modify-pool pool_A --name pool_B`
`vitastor-cli modify-pool 2 --pg_size 4 -n 128`
## rm-pool
`vitastor-cli rm-pool|pool-rm [--force] <id|name>`
Remove a pool. Refuses to remove pools with images without `--force`.
## ls-pools
`vitastor-cli ls-pools|pool-ls|ls-pool|pools [-l] [--detail] [--sort FIELD] [-r] [-n N] [--stats] [<glob> ...]`
List pools (only matching <glob> patterns if passed).
| <!-- --> | <!-- --> |
|----------------------|-------------------------------------------------------|
| `-l` or `--long` | Also report I/O statistics |
| `--detail` | Use list format (not table), show all details |
| `--sort FIELD` | Sort by specified field (see fields in --json output) |
| `-r` or `--reverse` | Sort in descending order |
| `-n` or `--count N` | Only list first N items |

View File

@ -15,6 +15,7 @@ vitastor-cli - интерфейс командной строки для адм
- [df](#df)
- [ls](#ls)
- [create](#create)
- [snap-create](#create)
- [modify](#modify)
- [rm](#rm)
- [flatten](#flatten)
@ -22,11 +23,16 @@ vitastor-cli - интерфейс командной строки для адм
- [merge-data](#merge-data)
- [alloc-osd](#alloc-osd)
- [rm-osd](#rm-osd)
- [create-pool](#create-pool)
- [modify-pool](#modify-pool)
- [ls-pools](#ls-pools)
- [rm-pool](#rm-pool)
Глобальные опции:
```
--etcd_address ADDR Адрес соединения с etcd
--config_file FILE Путь к файлу конфигурации Vitastor
--etcd_address URL Адрес соединения с etcd
--iodepth N Отправлять параллельно N операций на каждый OSD (по умолчанию 32)
--parallel_osds M Работать параллельно с M OSD (по умолчанию 4)
--progress 1|0 Печатать прогресс выполнения (по умолчанию 1)
@ -83,8 +89,8 @@ kaveri 2/1 32 0 B 10 G 0 B 100% 0%
`vitastor-cli ls [-l] [-p POOL] [--sort FIELD] [-r] [-n N] [<glob> ...]`
Показать список образов, если переданы шаблоны `<glob>`, то только с именами,
соответствующими этим шаблонам (стандартные ФС-шаблоны с * и ?).
Показать список образов, если передан(ы) шаблон(ы) `<glob>`, то только с именами,
соответствующими одному из шаблонов (стандартные ФС-шаблоны с * и ?).
Опции:
@ -126,9 +132,11 @@ vitastor-cli snap-create [-p|--pool <id|name>] <image>@<snapshot>
Создать снимок образа `<name>` (можно использовать любую форму команды). Снимок можно создавать без остановки
клиентов, если пишущий клиент максимум 1.
Смотрите также информацию о том, [как экспортировать снимки](qemu.ru.md#экспорт-снимков).
## modify
`vitastor-cli modify <name> [--rename <new-name>] [--resize <size>] [--readonly | --readwrite] [-f|--force]`
`vitastor-cli modify <name> [--rename <new-name>] [--resize <size>] [--readonly | --readwrite] [-f|--force] [--down-ok]`
Изменить размер, имя образа или флаг "только для чтения". Снимать флаг "только для чтения"
и уменьшать размер образов, у которых есть дочерние клоны, без `--force` нельзя.
@ -136,13 +144,12 @@ vitastor-cli snap-create [-p|--pool <id|name>] <image>@<snapshot>
Если новый размер меньше старого, "лишние" данные будут удалены, поэтому перед уменьшением
образа сначала уменьшите файловую систему в нём.
```
-f|--force Разрешить уменьшение или перевод в чтение-запись образа, у которого есть клоны.
```
* `-f|--force` - Разрешить уменьшение или перевод в чтение-запись образа, у которого есть клоны.
* `--down-ok` - Разрешить уменьшение, даже если часть данных останется неудалённой на недоступных OSD.
## rm
`vitastor-cli rm <from> [<to>] [--writers-stopped]`
`vitastor-cli rm <from> [<to>] [--writers-stopped] [--down-ok]`
Удалить образ `<from>` или все слои от `<from>` до `<to>` (`<to>` должен быть дочерним
образом `<from>`), одновременно меняя родительские образы их клонов (если таковые есть).
@ -154,6 +161,10 @@ vitastor-cli snap-create [-p|--pool <id|name>] <image>@<snapshot>
В других случаях родительские слои вливаются в дочерние.
Другие опции:
* `--down-ok` - Продолжать удаление/слияние, даже если часть данных останется неудалённой на недоступных OSD.
## flatten
`vitastor-cli flatten <layer>`
@ -181,6 +192,59 @@ vitastor-cli snap-create [-p|--pool <id|name>] <image>@<snapshot>
в целевой образ `<target>`. `<to>` должен быть дочерним образом `<from>`, а `<target>`
должен быть одним из слоёв между `<from>` и `<to>`, включая сами `<from>` и `<to>`.
## describe
`vitastor-cli describe [--osds <osds>] [--object-state <состояния>] [--pool <пул>]
[--inode <номер>] [--min-inode <номер>] [--max-inode <номер>]
[--min-offset <смещение>] [--max-offset <смещение>]`
Описать состояние "грязных" объектов в кластере, то есть таких объектов, копии
или части которых хранятся на наборе OSD, не равном целевому.
```
--osds <osds>
Перечислять только объекты с первичных OSD из списка <osds>.
--object-state <состояния>
Перечислять только объекты в указанных состояниях. Возможные состояния
объектов:
- degraded - деградированная избыточность
- misplaced - перемещённый
- incomplete - нечитаемый из-за потери большего числа частей, чем допустимо
- corrupted - с одной или более повреждённой частью
- inconsistent - неконсистентный, с неоднозначным расхождением копий/частей
--pool <имя или ID пула>
Перечислять только объекты из заданного пула.
--inode, --min-inode, --max-inode
Перечислять только объекты из указанных номеров инодов (образов).
--min-offset, --max-offset
Перечислять только объекты с заданных смещений внутри образов.
```
## fix
`vitastor-cli fix [--objects <объекты>] [--bad-osds <osds>] [--part <номер>] [--check no]`
Исправить неконсистентные (неоднозначные) объекты путём удаления части копий.
```
--objects <объекты>
Объекты для исправления - в простом текстовом или JSON формате. Если опция
не указана, список объектов читается со стандартного ввода в тех же форматах.
Простой формат: 0x<инод>:0x<смещение> <любой разделитель> 0x<инод>:0x<смещение> ...
Формат JSON: [{"inode":"0x<инод>","stripe":"0x<смещение>"},...]
--bad-osds <osds>
Удалить неконсистентные копии/части объектов с данных OSD, таким образом
признавая потерю этих копий и позволяя Vitastor-у восстановить объекты из
других копий.
--part <номер>
Удалить только части EC с заданным номером (от 0 до pg_size-1). Нужно только
в редких граничных случаях, когда один и тот же OSD содержит несколько частей
одного EC-объекта.
--check no
Не перепроверять, что заданные объекты действительно в неконсистентном
состоянии и просто удалять заданные части.
```
## alloc-osd
`vitastor-cli alloc-osd`
@ -198,3 +262,91 @@ vitastor-cli snap-create [-p|--pool <id|name>] <image>@<snapshot>
С опцией `--dry-run` только проверяет, возможно ли удаление без потери данных и деградации
избыточности.
## create-pool
`vitastor-cli create-pool|pool-create <name> (-s <pg_size>|--ec <N>+<K>) -n <pg_count> [OPTIONS]`
Создать пул. Обязательные параметры:
| <!-- --> | <!-- --> |
|---------------------------|---------------------------------------------------------------------------------------------|
| `-s R` или `--pg_size R` | Число копий данных для реплицированных пулов |
| `--ec N+K` | Число частей данных (N) и чётности (K) для пулов с кодами коррекции ошибок |
| `-n N` или `--pg_count N` | Число PG для нового пула (начните с 10*<число OSD>/pg_size, округлённого до степени двойки) |
Необязательные параметры:
| <!-- --> | <!-- --> |
|--------------------------------|----------------------------------------------------------------------------|
| `--pg_minsize <number>` | (R или N+K) минус число разрешённых отказов без остановки пула ([подробнее](../config/pool.ru.md#pg_minsize)) |
| `--failure_domain host` | Домен отказа: host, osd или другой из placement_levels. По умолчанию: host |
| `--root_node <node>` | Использовать для пула только дочерние OSD этого узла дерева размещения |
| `--osd_tags <tag>[,<tag>]...` | ...только OSD со всеми заданными тегами |
| `--block_size 128k` | ...только OSD с данным размером блока |
| `--bitmap_granularity 4k` | ...только OSD с данным размером логического сектора |
| `--immediate_commit none` | ...только OSD с этим или большим immediate_commit (none < small < all) |
| `--primary_affinity_tags tags` | Предпочитать OSD со всеми данными тегами для роли первичных |
| `--scrub_interval <time>` | Включить скрабы с заданным интервалом времени (число + единица s/m/h/d/M/y) |
| `--pg_stripe_size <number>` | Увеличить блок группировки объектов по PG |
| `--max_osd_combinations 10000` | Максимальное число случайных комбинаций OSD для ЛП-солвера |
| `--wait` | Подождать, пока новый пул будет активирован |
| `-f` или `--force` | Не проверять, что в кластере достаточно доменов отказа для создания пула |
Подробно о параметрах см. [Конфигурация пулов](../config/pool.ru.md).
Примеры:
`vitastor-cli create-pool test_x4 -s 4 -n 32`
`vitastor-cli create-pool test_ec42 --ec 4+2 -n 32`
## modify-pool
`vitastor-cli modify-pool|pool-modify <id|name> [--name <new_name>] [PARAMETERS...]`
Изменить настройки существующего пула. Изменяемые параметры:
```
[-s|--pg_size <number>] [--pg_minsize <number>] [-n|--pg_count <count>]
[--failure_domain <level>] [--root_node <node>] [--osd_tags <tags>]
[--max_osd_combinations <number>] [--primary_affinity_tags <tags>] [--scrub_interval <time>]
```
Неизменяемые параметры (их изменение ПРИВЕДЁТ к потере данных):
```
[--block_size <size>] [--bitmap_granularity <size>]
[--immediate_commit <all|small|none>] [--pg_stripe_size <size>]
```
Эти параметры можно изменить, только если явно передать опцию -f или --force.
Описания параметров смотрите в [create-pool](#create-pool).
Примеры:
`vitastor-cli modify-pool pool_A --name pool_B`
`vitastor-cli modify-pool 2 --pg_size 4 -n 128`
## rm-pool
`vitastor-cli rm-pool|pool-rm [--force] <id|name>`
Удалить пул. Отказывается удалять пул, в котором ещё есть образы, без `--force`.
## ls-pools
`vitastor-cli ls-pools|pool-ls|ls-pool|pools [-l] [--detail] [--sort FIELD] [-r] [-n N] [--stats] [<glob> ...]`
Показать список пулов. Если передан(ы) шаблон(ы) `<glob>`, то только с именами,
соответствующими одному из шаблонов (стандартные ФС-шаблоны с * и ?).
| <!-- --> | <!-- --> |
|-----------------------|------------------------------------------------------------|
| `-l` или `--long` | Вывести также статистику ввода-вывода |
| `--detail` | Максимально подробный вывод в виде списка (а не таблицы) |
| `--sort FIELD` | Сортировать по заданному полю (поля см. в выводе с --json) |
| `-r` или `--reverse` | Сортировать в обратном порядке |
| `-n` или `--count N` | Выводить только первые N записей |

View File

@ -1,4 +1,4 @@
[Documentation](../../README.md#documentation) → Usage → Disk Tool
[Documentation](../../README.md#documentation) → Usage → Disk management tool
-----
@ -17,6 +17,7 @@ It supports the following commands:
- [purge](#purge)
- [read-sb](#read-sb)
- [write-sb](#write-sb)
- [update-sb](#update-sb)
- [udev](#udev)
- [exec-osd](#exec-osd)
- [pre-exec](#pre-exec)
@ -86,6 +87,8 @@ Options (both modes):
--journal_size 1G/32M Set journal size (area or partition size)
--block_size 1M/128k Set blockstore object size
--bitmap_granularity 4k Set bitmap granularity
--data_csum_type none Set data checksum type (crc32c or none)
--csum_block_size 4k Set data checksum block size
--data_device_block 4k Override data device block size
--meta_device_block 4k Override metadata device block size
--journal_device_block 4k Override journal device block size
@ -100,8 +103,9 @@ checks the device cache status on start and tries to disable cache for SATA/SAS
If it doesn't succeed it issues a warning in the system log.
You can also pass other OSD options here as arguments and they'll be persisted
to the superblock: max_write_iodepth, max_write_iodepth, min_flusher_count,
max_flusher_count, inmemory_metadata, inmemory_journal, journal_sector_buffer_count,
in the superblock: cached_io_data, cached_io_meta, cached_io_journal,
inmemory_metadata, inmemory_journal, max_write_iodepth,
min_flusher_count, max_flusher_count, journal_sector_buffer_count,
journal_no_same_sector_overwrites, throttle_small_writes, throttle_target_iops,
throttle_target_mbs, throttle_target_parallelism, throttle_threshold_us.
See [Runtime OSD Parameters](../config/osd.en.md) for details.
@ -179,6 +183,14 @@ Try to read Vitastor OSD superblock from `<device>` and print it in JSON format.
Read JSON from STDIN and write it into Vitastor OSD superblock on `<device>`.
## update-sb
`vitastor-disk update-sb <device> [--force] [--<parameter> <value>] [...]`
Read Vitastor OSD superblock from <device>, update parameters in it and write it back.
`--force` allows to ignore validation errors.
## udev
`vitastor-disk udev <device>`
@ -250,6 +262,8 @@ Options (see also [Cluster-Wide Disk Layout Parameters](../config/layout-cluster
--object_size 128k Set blockstore block size
--bitmap_granularity 4k Set bitmap granularity
--journal_size 32M Set journal size
--data_csum_type none Set data checksum type (crc32c or none)
--csum_block_size 4k Set data checksum block size
--device_block_size 4k Set device block size
--journal_offset 0 Set journal offset
--device_size 0 Set device size

View File

@ -1,4 +1,4 @@
[Документация](../../README-ru.md#документация) → Использование → Управление дисками
[Документация](../../README-ru.md#документация) → Использование → Инструмент управления дисками
-----
@ -17,6 +17,7 @@ vitastor-disk - инструмент командной строки для уп
- [purge](#purge)
- [read-sb](#read-sb)
- [write-sb](#write-sb)
- [update-sb](#update-sb)
- [udev](#udev)
- [exec-osd](#exec-osd)
- [pre-exec](#pre-exec)
@ -87,6 +88,8 @@ vitastor-disk - инструмент командной строки для уп
--journal_size 1G/32M Задать размер журнала (области или раздела журнала)
--block_size 1M/128k Задать размер объекта хранилища
--bitmap_granularity 4k Задать гранулярность битовых карт
--data_csum_type none Задать тип контрольных сумм (crc32c или none)
--csum_block_size 4k Задать размер блока расчёта контрольных сумм
--data_device_block 4k Задать размер блока устройства данных
--meta_device_block 4k Задать размер блока метаданных
--journal_device_block 4k Задать размер блока журнала
@ -101,8 +104,9 @@ vitastor-disk - инструмент командной строки для уп
это не удаётся, в системный журнал выводится предупреждение.
Вы можете передать данной команде и некоторые другие опции OSD в качестве аргументов
и они тоже будут сохранены в суперблок: max_write_iodepth, max_write_iodepth, min_flusher_count,
max_flusher_count, inmemory_metadata, inmemory_journal, journal_sector_buffer_count,
и они тоже будут сохранены в суперблок: cached_io_data, cached_io_meta,
cached_io_journal, inmemory_metadata, inmemory_journal, max_write_iodepth,
min_flusher_count, max_flusher_count, journal_sector_buffer_count,
journal_no_same_sector_overwrites, throttle_small_writes, throttle_target_iops,
throttle_target_mbs, throttle_target_parallelism, throttle_threshold_us.
Читайте об этих параметрах подробнее в разделе [Изменяемые параметры OSD](../config/osd.ru.md).
@ -184,6 +188,15 @@ throttle_target_mbs, throttle_target_parallelism, throttle_threshold_us.
Прочитать JSON со стандартного ввода и записать его в суперблок OSD на диск `<device>`.
## update-sb
`vitastor-disk update-sb <device> [--force] [--<параметр> <значение>] [...]`
Прочитать суперблок OSD с диска `<device>`, изменить в нём заданные параметры и записать обратно.
Опция `--force` позволяет читать суперблок, даже если он считается некорректным
из-за ошибок валидации.
## udev
`vitastor-disk udev <device>`
@ -255,6 +268,8 @@ OSD отключены fsync-и.
--object_size 128k Размер блока хранилища
--bitmap_granularity 4k Гранулярность битовых карт
--journal_size 32M Размер журнала
--data_csum_type none Задать тип контрольных сумм (crc32c или none)
--csum_block_size 4k Задать размер блока расчёта контрольных сумм
--device_block_size 4k Размер блока устройства
--journal_offset 0 Смещение журнала
--device_size 0 Размер устройства

View File

@ -14,10 +14,13 @@ Vitastor has a fio driver which can be installed from the package vitastor-fio.
Use the following command as an example to run tests with fio against a Vitastor cluster:
```
fio -thread -ioengine=libfio_vitastor.so -name=test -bs=4M -direct=1 -iodepth=16 -rw=write -etcd=10.115.0.10:2379/v3 -image=testimg
fio -thread -ioengine=libfio_vitastor.so -name=test -bs=4M -direct=1 -iodepth=16 -rw=write -image=testimg
```
If you don't want to access your image by name, you can specify pool number, inode number and size
(`-pool=1 -inode=1 -size=400G`) instead of the image name (`-image=testimg`).
See exact fio commands to use for benchmarking [here](../performance/understanding.en.md#команды-fio).
You can also specify etcd address(es) explicitly by adding `-etcd=10.115.0.10:2379/v3`, or you
can override configuration file path by adding `-conf=/etc/vitastor/vitastor.conf`.
See exact fio commands to use for benchmarking [here](../performance/understanding.en.md#fio-commands).

View File

@ -14,10 +14,13 @@
Используйте следующую команду как пример для запуска тестов кластера Vitastor через fio:
```
fio -thread -ioengine=libfio_vitastor.so -name=test -bs=4M -direct=1 -iodepth=16 -rw=write -etcd=10.115.0.10:2379/v3 -image=testimg
fio -thread -ioengine=libfio_vitastor.so -name=test -bs=4M -direct=1 -iodepth=16 -rw=write -image=testimg
```
Вместо обращения к образу по имени (`-image=testimg`) можно указать номер пула, номер инода и размер:
`-pool=1 -inode=1 -size=400G`.
Вы также можете задать адрес(а) подключения к etcd явно, добавив `-etcd=10.115.0.10:2379/v3`,
или переопределить путь к файлу конфигурации, добавив `-conf=/etc/vitastor/vitastor.conf`.
Конкретные команды fio для тестирования производительности можно посмотреть [здесь](../performance/understanding.ru.md#команды-fio).

View File

@ -11,20 +11,42 @@ NBD stands for "Network Block Device", but in fact it also functions as "BUSE"
NBD slighly lowers the performance due to additional overhead, but performance still
remains decent (see an example [here](../performance/comparison1.en.md#vitastor-0-4-0-nbd)).
Vitastor Kubernetes CSI driver is based on NBD.
See also [VDUSE](qemu.en.md#vduse) as a better alternative to NBD.
Vitastor Kubernetes CSI driver uses NBD when VDUSE is unavailable.
## Map image
To create a local block device for a Vitastor image run:
```
vitastor-nbd map --etcd_address 10.115.0.10:2379/v3 --image testimg
vitastor-nbd map --image testimg
```
It will output a block device name like /dev/nbd0 which you can then use as a normal disk.
You can also use `--pool <POOL> --inode <INODE> --size <SIZE>` instead of `--image <IMAGE>` if you want.
vitastor-nbd supports all usual Vitastor configuration options like `--config_file <path_to_config>` plus NBD-specific:
* `--nbd_timeout 300` \
Timeout for I/O operations in seconds after exceeding which the kernel stops
the device. You can set it to 0 to disable the timeout, but beware that you
won't be able to stop the device at all if vitastor-nbd process dies.
* `--nbd_max_devices 64 --nbd_max_part 3` \
Options for the `nbd` kernel module when modprobing it (`nbds_max` and `max_part`).
note that maximum allowed (nbds_max)*(1+max_part) is 256.
* `--logfile /path/to/log/file.txt` \
Write log messages to the specified file instead of dropping them (in background mode)
or printing them to the standard output (in foreground mode).
* `--dev_num N` \
Use the specified device /dev/nbdN instead of automatic selection.
* `--foreground 1` \
Stay in foreground, do not daemonize.
Note that `nbd_timeout`, `nbd_max_devices` and `nbd_max_part` options may also be specified
in `/etc/vitastor/vitastor.conf` or in other configuration file specified with `--config_file`.
## Unmap image
To unmap the device run:
@ -32,3 +54,27 @@ To unmap the device run:
```
vitastor-nbd unmap /dev/nbd0
```
## List mapped images
```
vitastor-nbd ls [--json]
```
Example output (normal format):
```
/dev/nbd0
image: bench
pid: 584536
/dev/nbd1
image: bench1
pid: 584546
```
Example output (JSON format):
```
{"/dev/nbd0": {"image": "bench", "pid": 584536}, "/dev/nbd1": {"image": "bench1", "pid": 584546}}
```

View File

@ -14,14 +14,16 @@ NBD на данный момент необходимо, чтобы монтир
NBD немного снижает производительность из-за дополнительных копирований памяти,
но она всё равно остаётся на неплохом уровне (см. для примера [тест](../performance/comparison1.ru.md#vitastor-0-4-0-nbd)).
CSI-драйвер Kubernetes Vitastor основан на NBD.
Смотрите также [VDUSE](qemu.ru.md#vduse), как лучшую альтернативу NBD.
CSI-драйвер Kubernetes Vitastor использует NBD, когда VDUSE недоступен.
## Подключить устройство
Чтобы создать локальное блочное устройство для образа, выполните команду:
```
vitastor-nbd map --etcd_address 10.115.0.10:2379/v3 --image testimg
vitastor-nbd map --image testimg
```
Команда напечатает название блочного устройства вида /dev/nbd0, которое потом можно
@ -30,6 +32,32 @@ vitastor-nbd map --etcd_address 10.115.0.10:2379/v3 --image testimg
Для обращения по номеру инода, аналогично другим командам, можно использовать опции
`--pool <POOL> --inode <INODE> --size <SIZE>` вместо `--image testimg`.
vitastor-nbd поддерживает все обычные опции Vitastor, например, `--config_file <path_to_config>`,
плюс специфичные для NBD:
* `--nbd_timeout 30` \
Максимальное время выполнения любой операции чтения/записи в секундах, при
превышении которого ядро остановит NBD-устройство. Вы можете установить опцию
в 0, чтобы отключить ограничение времени, но имейте в виду, что в этом случае
вы вообще не сможете отключить NBD-устройство при нештатном завершении процесса
vitastor-nbd.
* `--nbd_max_devices 64 --nbd_max_part 3` \
Опции, передаваемые модулю ядра nbd, если его загружает vitastor-nbd
(`nbds_max` и `max_part`). Имейте в виду, что (nbds_max)*(1+max_part)
обычно не должно превышать 256.
* `--logfile /path/to/log/file.txt` \
Писать сообщения о процессе работы в заданный файл, вместо пропуска их
при фоновом режиме запуска или печати на стандартный вывод при запуске
в консоли с `--foreground 1`.
* `--dev_num N` \
Использовать заданное устройство `/dev/nbdN` вместо автоматического подбора.
* `--foreground 1` \
Не уводить процесс в фоновый режим.
Обратите внимание, что опции `nbd_timeout`, `nbd_max_devices` и `nbd_max_part` можно
также задавать в `/etc/vitastor/vitastor.conf` или в другом файле конфигурации,
заданном опцией `--config_file`.
## Отключить устройство
Для отключения устройства выполните:
@ -37,3 +65,27 @@ vitastor-nbd map --etcd_address 10.115.0.10:2379/v3 --image testimg
```
vitastor-nbd unmap /dev/nbd0
```
## Вывести подключённые устройства
```
vitastor-nbd ls [--json]
```
Пример вывода в обычном формате:
```
/dev/nbd0
image: bench
pid: 584536
/dev/nbd1
image: bench1
pid: 584546
```
Пример вывода в JSON-формате:
```
{"/dev/nbd0": {"image": "bench", "pid": 584536}, "/dev/nbd1": {"image": "bench1", "pid": 584546}}
```

Some files were not shown because too many files have changed in this diff Show More