851 Commits (master)

Author	SHA1	Message	Date
Vitaliy Filippov	77167e2920	Do not use \r if output is not a terminal	2024-02-29 00:21:17 +03:00
Vitaliy Filippov	5af23672d0	Fix rm/rm-data error return code, add --down-ok option to bypass the error	2024-02-29 00:20:10 +03:00
Vitaliy Filippov	6bf1f539a6	Add EIO retry timeout and allow to disable these retries, rename up_wait_retry_interval to client_retry_interval	2024-02-28 13:10:02 +03:00
Vitaliy Filippov	4eab26f968	Add documentation and a very basic test for pool management commands	2024-02-28 13:08:04 +03:00
Vitaliy Filippov	86243b7101	Rework & fix pool-create / pool-modify / pool-ls	2024-02-28 13:08:04 +03:00
idelson	dc92851322	vitastor-cli: add commands to control pools: pool-create, pool-ls, pool-modify, pool-rm ... PR #59 - https://github.com/vitalif/vitastor/pull/58/commits By MIND Software LLC By submitting this pull request, I accept Vitastor CLA	2024-02-28 13:08:04 +03:00
Vitaliy Filippov	fc413038d1	Wait for blockstore initialisation before starting OSD	2024-02-27 02:20:04 +03:00
Vitaliy Filippov	1bc0b5aab3	Fix a rare use-after-free in automatic sync after delete in blockstore ... ASan report: [0] READ of size 16 at operator() /root/vitastor/src/blockstore_write.cpp:100 ...[5] blockstore_impl_t::ack_sync(blockstore_op_t*) /root/vitastor/src/blockstore_sync.cpp:232	2024-02-24 00:06:36 +03:00
Vitaliy Filippov	5e934264cf	Release 1.4.7 ... - Fix another old "BUG: Attempt to overwrite used offset" in a very simple case: bs=4k rw=write iodepth=16 from OSD start; add this case to tests - Fix a rare crash with "unexpected state during flush: 0x51" possible with EC since 1.4.2 during rebalance and OSD outages - Fix a rare write stall with EC & immediate_commit=none caused by sync operations reserving unneeded space in the journal - Fix 32-bit build warnings, most in printf/scanf format strings	2024-02-22 12:45:52 +03:00
Vitaliy Filippov	f20564b44b	Fix 32-bit build warnings (99.9% in printf)	2024-02-22 12:22:16 +03:00
Vitaliy Filippov	b3c15db331	32M journal by default in simple-offsets	2024-02-21 15:25:02 +03:00
Vitaliy Filippov	685bcd6ef9	Do not reserve extra space for big_writes during sync - sync itself is needed to commit and clear them	2024-02-21 13:00:14 +03:00
Vitaliy Filippov	3eb389b321	Supposed fix for "unexpected state during flush: 0x51" with EC	2024-02-21 01:32:06 +03:00
Vitaliy Filippov	3d16cde23c	Fix assertions, add small sequential write test	2024-02-20 19:41:48 +03:00
Vitaliy Filippov	c6406d67fc	Fix journal space_check incorrectly checking for space at the beginning	2024-02-20 19:40:56 +03:00
Vitaliy Filippov	f87964861d	Release 1.4.6 ... Unwavering stabilization of 1.4.x, continued :-) - Include the accidentally lost part of 1.4.5 journal trimming fix - Fix a possible OSD crash with "BUG: Attempt to overwrite used offset" which was probably present for long time, but became apparent after fixing flapping tests in CI - Fix remaining flapping tests in CI. It was the first time when tests actually passed without retries :-)	2024-02-20 17:01:26 +03:00
Vitaliy Filippov	7048228678	Supposed fix for "BUG: Attempt to overwrite used offset"	2024-02-20 15:56:48 +03:00
Vitaliy Filippov	ea73857450	Add asserts to catch "BUG: Attempt to overwrite used offset"	2024-02-20 15:56:48 +03:00
Vitaliy Filippov	6cfe38ec04	Followup to empty cur.oid as stop condition for forced trim fix	2024-02-20 15:56:38 +03:00
Vitaliy Filippov	f882c7dd87	Release 1.4.5 ... - Fix a write stall caused by incorrect journal trimming introduced in 1.4.4 :) - Fix PGs sometimes hanging in "starting" state on mass OSD restarts - Fix a rare crash with "map::at" during OSD pings - Use new defaults for non-capacitor (desktop) SSDs - improves T1Q256 random write from ~6k iops to ~45k iops - Make journal_trim_interval configurable	2024-02-16 10:13:33 +03:00
Vitaliy Filippov	26dd863c8d	Fix sometimes possible crash on clients.at() during pings	2024-02-16 10:13:33 +03:00
Vitaliy Filippov	2ae859fbc6	Use min/max_flusher_count=32/256, 128M journal and autosync_writes=512 for non-capacitor SSDs by default	2024-02-16 10:13:33 +03:00
Vitaliy Filippov	8389c0f33b	Fix PGs sometimes hanging in "starting" state on mass OSD restarts	2024-02-15 23:38:52 +03:00
Vitaliy Filippov	9db2196aef	Make journal_trim_interval configurable	2024-02-15 23:38:51 +03:00
Vitaliy Filippov	8d6ae662fe	Use empty cur.oid as stop condition for forced trim, not journal_trim_counter	2024-02-15 23:27:17 +03:00
Vitaliy Filippov	c777a0041a	Release 1.4.4 ... A couple of fixes for EC pools - Fix a segfault possible on partial EC overwrite in 1234 -> 5030 rebalance scenario - Fix two problems leading to EC pools stalling on rebalance & parallel sudden stops of OSDs, for example during a sudden poweroff of a host: - Recovery auto-tuning (1.4.0 feature) could apply too large delays and stall the EC journal - fixed by limiting delays with a new recovery_tune_sleep_cutoff_us parameter (10 seconds by default) and applying recovery pauses before write operations, not after them, to not occupy space in the journal for long time - Dynamic journal space reservation (1.3.0 feature) wasn't accounting new writes when checking the limit so OSDs could still fill the journal fully and stall - fixed by including new writes into the limit - Print etcd dbSize instead of dbSizeInUse in status	2024-02-11 16:23:08 +03:00
Vitaliy Filippov	978bdc128a	Apply recovery pause before writes, after commits, and do not apply it to syncs to not block EC pools from functioning	2024-02-11 16:13:52 +03:00
Vitaliy Filippov	bb2f395f1e	Add cutoff threshold for recovery auto-tuning	2024-02-11 16:13:52 +03:00
Vitaliy Filippov	b127da40f7	Add a FIXME about incomplete PGs	2024-02-11 13:42:51 +03:00
Vitaliy Filippov	ca34a6047a	Fix dynamic journal space reservation: include the new write itself, too	2024-02-11 13:42:51 +03:00
Vitaliy Filippov	38ba76e893	Fix flusher sometimes being unable to trim journal when the flush queue is empty	2024-02-11 13:42:51 +03:00
Vitaliy Filippov	1e3c4edea0	Print etcd dbSize instead of dbSizeInUse in status	2024-02-11 13:42:51 +03:00
Vitaliy Filippov	e7ac855b07	Fix that EC segfault (1234 -> 5030 partial overwrite)	2024-02-11 13:42:51 +03:00
Vitaliy Filippov	c53357ac45	Add a test for EC segfault with partial overwrite in 1234 -> 5030 rebalance scenario	2024-02-11 13:42:51 +03:00
Vitaliy Filippov	27e9f244ec	Release 1.4.3 ... Hotfix for hotfix O:-) - "Write stall fix" was incomplete and EC write stalls could continue even on 1.4.2. Now they're finally fixed O:-) - Make monitor ignore statistics of stopped OSDs. Previously if you stopped all OSDs the last total I/O numbers would remain the same indefinitely	2024-02-09 00:29:31 +03:00
Vitaliy Filippov	5d3317e4f2	Followup to 1.4.2 write stall fix - sadly, the previous version was not working correctly :)	2024-02-08 19:34:29 +03:00
Vitaliy Filippov	016115c0d4	Release 1.4.2 ... - Log to systemd by default - Fix excessive autosyncs after every operation with disabled immediate_commit (introduced in 1.1.0) - Fix a possible write stall with EC due to the lack of OSD wakeup after stabilizing previous writes - Change sync operation semantics as a final fix to possible write stalls with EC and disabled immediate_commit - Sync after deleting data in CLI rm / rm-data if immediate_commit is disabled - Fix OSDs ignoring syncs & autosyncs for delete operations - Fix OSD space reporting sometimes adding garbage zeros for deleted inodes (causing extra pool/stats etcd keys for deleted pools) - Speed up monitor failover - change default etcd_mon_ttl from 30 to 5 seconds - Speed up operation retries - change default up_wait_retry_interval to 50 ms - Add patch for libvirt 9.10	2024-02-04 02:23:49 +03:00
Vitaliy Filippov	77c10fd1f8	In fact, do not autosync blockstore when autosync_writes=0	2024-02-03 20:37:36 +03:00
Vitaliy Filippov	581d02e581	Mark secondary OSDs with deletions as dirty to not forget to sync & autosync them	2024-02-03 20:31:08 +03:00
Vitaliy Filippov	f03a9db4d9	Fix OSD space reporting sometimes adding garbage zeros for deleted inodes (causing extra pool/stats etcd keys for deleted pools)	2024-02-03 20:31:08 +03:00
Vitaliy Filippov	cb9c30bc31	Sync after sending all deletes to each PG in cli rm-data	2024-02-03 20:31:08 +03:00
Vitaliy Filippov	a86a380d20	Fix invalid parsing of autosync_writes in blockstore leading to autosyncs after every operation with disabled immediate_commit :D	2024-02-03 20:31:08 +03:00
Vitaliy Filippov	1cec62d25d	Sync only completed writes ... Should be a final remaining fix to EC + non-capacitor (non-immediate-commit) write hangs :). First it was breaking non-EC ("instantly stable") writes because they sometimes complete out of order which was leading to the following error: terminate called after throwing an instance of 'std::runtime_error' what(): BUG: Unexpected dirty_entry 1000000000001:29480000 v65540 unstable state during flush: 0x151 But it is easily fixed by scanning previous and next dirty_entries in mark_stable.	2024-01-27 15:17:22 +03:00
Vitaliy Filippov	1c322b33ed	Change default up_wait_retry_interval to 50 ms	2024-01-26 01:51:08 +03:00
Vitaliy Filippov	ba55f91409	Release 1.4.1 ... - Fix a monitor crash on primary OSD switching introduced in 1.4.0 - Fix "partly outside array bounds" warnings for GCC 12 in cpp-btree - Fix a realloc memory leak in theory possible with too large listings (OSD_OP_LIST)	2024-01-18 02:31:42 +03:00
Vitaliy Filippov	d00d4dbac0	Initialize mod_revision field in etcd_state_client	2024-01-13 01:30:28 +03:00
Vitaliy Filippov	5d9d6f32a0	Fix common realloc memory leak mistakes found by cppcheck	2024-01-13 01:30:28 +03:00
Vitaliy Filippov	5280d1d561	Release 1.4.0 ... New features: - Intelligent recovery/rebalance speed auto-tuning to reduce its impact on clients (see README -> Features) - Auto-restoration of dead VDUSE daemons in CSI plugin - Add vitastor-disk update-sb command - Update QEMU for Debian Bookworm to 8.1 and use it for CSI plugin Bug fixes: - Fix pools SOMETIMES staying inactive after stopping a node due to OSDs not reacting to PG state changes caused by incorrect full reload of state from etcd on reconnection - Make monitors retry pool configuration changes quickier which fixes them being unable to apply changes when an ongoing rebalance is quickly making a lot of PGs clean - Fix CSI plugin not accepting array of strings as etcd address in /etc/vitastor/vitastor.conf - Allow multiple interfaces with the same IP address, for "simple routed" full mesh network - Do not ignore loopback addresses for OSD network (to make ECMP setups with frr possible) - Fix a rare client crash during OSD reconnections - Only treat data partitions as existing OSDs in vitastor-disk prepare - Remove etcd parameter from default command examples - Fix reported free space sometimes changing non-immediately after deletion of data from OSDs - Fix a possible OSD crash on print_slow when bs_op is NULL - Use the same etcd_ws_keepalive_interval in mon as in OSD - Fix mon not using values from config when /config/global is not present - Remove pve-storage-portal-dns-list format for vitastor_etcd_address - Parse log_level in cluster_client - Fix vitastor-nbd image existence check not working because of non-zeroed inode_watch fields - Do not warn on EPIPE in client unless log_level is raised explicitly - Fix incorrect error in CSI when searching for the device in /sys - Remove 2 last prints to stdout in etcd_state_client - Fix a possible OSD crash when checking corrupted journal entries	2024-01-12 01:28:33 +03:00
Vitaliy Filippov	2f228fa96a	Only treat data partitions as existing OSDs in vitastor-disk prepare	2023-12-31 11:46:47 +03:00
Vitaliy Filippov	a6ab54b1ba	Do not allow negative util_low/high	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	99ee8596ea	Rename min/max_util to util_low/high	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	c4928e6ecd	Protect from try_send completing the operation immediately ... Fixes a possible use-after-free in case of continue_ops() calling try_send(), then connect_peer() -> set_timer() -> trigger_nearest() -> handle_op_part() -> continue_ops() again	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	8b8c1179a7	Use a separate used_blocks counter for free space stats to hide possibly delayed on-flush deallocation	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	d5a6fa6dd7	Fix possible crash on print_slow when bs_op is NULL	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	1edf86ed26	Aggregate recovery delay using simple mean over last 10 observations (EWMA is shit)	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	5ca7cde612	Experiment/WIP: Try to track "secondary" recovery ops separately	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	751935ddd8	WIP Auto-tune recovery speed	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	d84dee7098	Track recovery op latencies + refactor into a structure	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	f72f14e6a7	Clear old PG states, history, and OSD states on etcd state reload ... Also add protection from etcd watcher messages being split into multiple websocket messages - I'm not sure if etcd actually does that, but it's better to have extra protection anyway. Also check that all etcd watchers are started in the keepalive routine, otherwise it sometimes tries to revive etcd watchers starting with revision=1 which obviously always fails because this revision is nearly always compacted. All these changes should fix an old rarely reproduced bug where SOMETIMES OSDs didn't react to PG config changes which was leading to offline pools on node reboot. It happened on the full reload of state from etcd.	2023-12-24 02:02:13 +03:00
Vitaliy Filippov	178bb0e701	Prevent re-entry into timerfd set_nearest	2023-12-22 02:32:40 +03:00
Vitaliy Filippov	7239cfb91a	Parse log_level in cluster_client	2023-12-20 02:21:23 +03:00
Vitaliy Filippov	7cea642f4a	Fix vitastor-nbd image existence check not working because of non-zeroed inode_watch fields	2023-12-19 01:11:37 +03:00
Vitaliy Filippov	dc615403d9	Do not warn on EPIPE in client unless log_level is raised explicitly	2023-12-17 13:42:26 +03:00
Vitaliy Filippov	1a704e06ab	Allow multiple interfaces with the same IP address, for "simple routed" full mesh network	2023-12-17 13:25:56 +03:00
Vitaliy Filippov	575475de71	Do not ignore loopback addresses for OSD network (to make ECMP setups with frr possible)	2023-12-17 11:55:13 +03:00
Vitaliy Filippov	aca2bef15f	Add vitastor-disk update-sb command	2023-12-14 01:11:42 +03:00
Vitaliy Filippov	691ebd991a	Move 2 last log printfs to stderr from stdout in etcd_state_client	2023-12-08 00:01:52 +03:00
Vitaliy Filippov	6d5df908a3	Fix possible out of bounds when checking invalid journal entries	2023-12-08 00:01:07 +03:00
Vitaliy Filippov	a1c7cc3d8d	Release 1.3.1 ... Hotfix to 1.3.0 - new "journal space reservation" had a bug which caused OSDs to crash with EC and without immediate_commit.	2023-12-04 18:35:09 +03:00
Vitaliy Filippov	a5e3dfbc5a	Oops, 1.3.0 needs a hotfix	2023-12-04 13:45:54 +03:00
Vitaliy Filippov	7972502eaf	Release 1.3.0 ... New features: - RDMA without ODP - much faster and all cards are now supported, not just Mellanox - VDUSE in CSI - faster, more stable and can even recover after CSI pod restart! - Reserve journal space for stabilize requests dynamically to prevent stalls under load with EC - Raise default NBD timeout from 30 to 300 seconds and allow to take it from /etc/vitastor/vitastor.conf - Remove explicit etcdUrl/etcdPrefix K8S storage class parameter support to prevent etcd migration issues for volumes created with these parameters - Support QEMU 8.1 and pve-qemu 8.1 Bug fixes: - Fix RDMA connection (and thus memory) leak - Fix rare crashes under load due to incorrect io_uring queue size tracking - Fix monitor statistics aggregation in case of empty /osd/stats keys - Fix crash on unknown long argument to vitastor-disk - Allow trailing comma in JSONs again - Fix crash on attempts to dump a long listing of objects "to stabilize" or "to rollback" in a slow op	2023-12-04 02:36:43 +03:00
Vitaliy Filippov	845454742d	Fix warning with QEMU 8.1	2023-12-04 01:59:07 +03:00
Vitaliy Filippov	628aa59574	Raise default NBD timeout from 30 to 300 seconds and allow to take it from /etc/vitastor/vitastor.conf	2023-12-02 14:11:14 +03:00
Vitaliy Filippov	19e2d9d6fa	Fix crash on unknown long argument to vitastor-disk	2023-12-01 00:55:51 +03:00
Vitaliy Filippov	b5c020ce0b	Use io_uring SQ size for ringloop capacity - otherwise get_sqe could return NULL when space_left() was > 0 under load ... Raise default io_uring size to 1024 for the same effective capacity as previously	2023-11-20 03:04:06 +03:00
Vitaliy Filippov	6b33ae973d	%d -> %lu	2023-11-20 03:02:26 +03:00
Vitaliy Filippov	cf36445359	Reserve journal space for stabilize requests dynamically to prevent stalls	2023-11-20 03:01:57 +03:00
Vitaliy Filippov	3fd873d263	Add -fno-omit-frame-pointer by default	2023-11-20 02:59:54 +03:00
Vitaliy Filippov	a00e8ae9ed	Fix mismatch journal pos format in vitastor-disk	2023-11-19 15:19:54 +03:00
Vitaliy Filippov	75674545dc	Limit the number of printed object versions in slow op dump (otherwise it may overflow the fixed buffer)	2023-11-13 01:10:28 +03:00
Vitaliy Filippov	225eb2fe3d	Support RDMA without ODP by stupidly copying memory. Disable ODP by default ... ODP is slower than regular RDMA even with memory copy overhead Example numbers: - 3950000 random read iops without ODP vs 240000 iops with ODP - 1447000 random write iops without ODP vs 101000 iops with ODP Reference: https://tkygtr6.github.io/pub/ISPASS21_slides.pdf	2023-11-12 15:03:47 +03:00
Vitaliy Filippov	7e82573ed0	Fix RDMA connection leak which was preventing stable functioning of RDMA :)	2023-11-11 23:40:47 +03:00
Vitaliy Filippov	5524dbdab7	Release 1.2.0 ... New features: - Implement CSI volume expansion - Implement CSI volume snapshots - CSI driver now requires Kubernetes >= 1.20 Bug fixes: - Important bug fix for EC: fix EC n+k, k>=2 read recovery in ISA-L version returning incorrect data when reading at least the second chunk out of multiple missing chunks without reading the first one. All users of EC n+k, k>=2 should upgrade as soon as possible, and upgrade should be conducted with downtime: first stop all clients (VMs/containers), then all OSDs, then upgrade and restart everything. - Fix unstable statistics aggregation in monitor (affecting vitastor-cli status and df) - Make udev not wait for OSDs to start during boot - Do not report negative numbers of offline PGs in vitastor-cli status when changing PG count - Report both old and new PG counts in vitastor-cli df when changing it - Fix OSDs sometimes not starting with "The code only supports journal versions 1 and 2, but it is 2 on disk" error after upgrading from pre-1.0 versions and letting OSDs run for some time - Fix monitors sometimes returning old PG count back after OSD configuration changes - Make monitor PG changes more stable and timeout errors less probable	2023-11-05 01:48:57 +03:00
Vitaliy Filippov	cd3dec06ac	Remove spaces from old->new PG count in df	2023-11-05 01:45:45 +03:00
Vitaliy Filippov	e15b6e7805	Fix "cannot be narrowed" in clang	2023-11-04 18:14:44 +03:00
Vitaliy Filippov	31017d8412	Allow to start with V2 journal with header size from V1, as incorrectly updated by previous versions	2023-11-04 18:13:42 +03:00
Vitaliy Filippov	4819854064	Fix OSDs incorrectly updating journal superblock after upgrade to 1.x from pre-1.x and refusing to start after it	2023-11-04 15:02:24 +03:00
Vitaliy Filippov	1f509cca77	Fix unused capture warnings and void* arithmetic (clang)	2023-11-04 14:55:12 +03:00
Vitaliy Filippov	aa8e8e8271	Add version info to --help output	2023-11-04 13:32:12 +03:00
Vitaliy Filippov	4d79e531c5	Do not print "-X offline" in status when changing pool PG count, print it in df instead	2023-11-04 13:12:13 +03:00
Vitaliy Filippov	30dff8893f	Fix ISA-L version EC recovery with first missing data chunk not being read ... (Yes, all EC n + k with k >= 2 users should upgrade as soon as possible)	2023-11-04 01:34:18 +03:00
Vitaliy Filippov	becf14a705	Add a test for EC with multiple missing data chunks, but without recovery of first of them	2023-11-04 01:34:18 +03:00
Vitaliy Filippov	7c054c6f10	Add "id" to df --json output	2023-11-01 12:46:16 +03:00
Vitaliy Filippov	b4740acf62	Fix operations paused for 0.5-1 second when it happens that io_uring submit is not triggered	2023-10-28 13:18:21 +03:00
Vitaliy Filippov	8222e3c77d	Release 1.1.0 ... New features: - Implement [client writeback cache](docs/config/client.en.md#client_enable_writeback) - Add the third I/O mode: [O_DIRECT|O_SYNC](docs/config/osd.en.md#data_io) (good for Optane) - Reduce load on etcd by splitting OSD lease and statistics reporting intervals: [etcd_stats_interval](docs/config/osd.en.md#etcd_stats_interval) (default 30 sec) - Make MON automatically filter OSDs by layout (block_size/immediate_commit/bitmap_granularity) to prevent "refusing to start PGs of this pool" errors on misconfiguration - Support running fio benchmarks on systems without io_uring - Make QEMU driver compatible with QEMU 8.1 - Document usage of [vhost-user-blk](docs/usage/qemu.en.md#vhost-user-blk) Bug fixes: - Fix resizing disks in QEMU driver (for example, in Proxmox) - Fix "unexpected result" in Proxmox driver by making CLI flush output on exit - Remove unneeded block_size mismatch warnings on pools without matching PGs - Fix possible segfault in vitastor-cli ls -l (usually with deleted pools) - Fix QEMU driver compatibility with systems without io_uring - Fix monitor eating 100% CPU when etcd is down (caused by infinite retries) - Fix potential incorrect write processing with snapshots (not caught in tests but could probably lead to client hangs) - Fix buffer insertion in cluster_client (not caught in tests but could probably lead to incorrect writes in rare cases) - Fix rare OSD crash during sync operation processing - Fix a reenterability issue in cluster_client not reproducible in QEMU/fio, but reproducible with the currently developed K/V database implementation - Fix deletion of the first modified object - OSDs could crash if you modified the same object a lot of times, then deleted it, and then modified it again - Fix the fio_sec_osd test tool	2023-10-28 00:33:06 +03:00
Vitaliy Filippov	be7e76f849	Split etcd_stats_interval out of etcd_report_interval	2023-10-27 01:26:26 +03:00
Vitaliy Filippov	294a754c9e	Allow write-back by default in NBD & NFS	2023-10-27 01:26:26 +03:00
Vitaliy Filippov	8bfea6e7de	Support vitastor_c_create_epoll() in fio driver	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	bac9e34836	Allow to create vitastor_c with plain epoll without uring :-)	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	8aa4d492c1	Allow to use epoll_manager without ringloop	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	9336ee5476	Correctly free manual "small vector" in cluster_client %-)	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	ad30b11519	Add the missing ringloop creation check to vitastor_c_create_uring_json()	2023-10-26 18:07:23 +03:00
Vitaliy Filippov	a061246997	Do not attempt to initialize QEMU driver via vitastor_c_create_qemu_uring() ... It doesn't add any compatibility because vitastor_c_uring_register_eventfd() is added in the same VITASTOR_C_API_VERSION 2.	2023-10-26 17:46:19 +03:00
Vitaliy Filippov	5066e35a49	Fix write-over-delete failing for the very first entry in dirty_db	2023-10-21 17:00:14 +03:00
Vitaliy Filippov	93dc31f3fc	Fix possible segfault in vitastor-cli ls -l	2023-10-18 11:11:41 +03:00
Vitaliy Filippov	f245b56176	Fix another possible reenterability issue in cluster_client ... Non-reproducible in QEMU/FIO, only caught during K/V DB debugging	2023-10-08 11:02:53 +03:00
Vitaliy Filippov	38db53f5ee	Implement client writeback cache ... - Disabled by default, enable with client_enable_writeback=true - Even then only enabled in FIO when -direct is disabled and in QEMU when block device cache is enabled in settings - Can also be enabled in other clients like vitastor-cli using parameter client_writeback_allowed=true, but not recommended	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	cd543a90bc	Prevent stack overflows in cli_merge with CAS and writeback cache	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	f600cc07b0	Autosync in blockstore every autosync_writes, too	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	6a8e530e6b	Add FIXME to timerfd_manager	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	5cadb170b9	Fix possible OSD crash during sync due to missing min_flushed_journal_sector reset	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	e72d4ed1d4	Remove unused bs_sync fields	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	ff479a102d	Make MON filter OSDs by block layout to prevent "refusing to start PGs of this pool" errors on misconfiguration	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	27d0d5b06a	Reads do not have to wait for buffer flushes anymore	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	33950c1ec8	Fix fio_sec_osd attr_len	2023-09-16 17:49:10 +03:00
Vitaliy Filippov	cc0fdc6253	Remove erroneous block_size mismatch warnings on pools without matching PGs	2023-09-08 23:19:04 +03:00
Vitaliy Filippov	79ecd59b10	Flush STDOUT and STDERR before exiting from cli to fix Proxmox "Unexpected result"	2023-09-07 17:30:26 +03:00
Vitaliy Filippov	b7d398be5b	Fix sscanf validation usage (field count instead of null_byte == 0)	2023-09-07 02:34:35 +03:00
Vitaliy Filippov	85e9f67d9d	Add supported_truncate_flags	2023-09-06 17:37:52 +03:00
Vitaliy Filippov	79c6d6f323	Make QEMU driver compatible with QEMU 8.1	2023-08-24 02:23:55 +03:00
Vitaliy Filippov	ae760dbc1d	Fix co_truncate size division by BDRV_SECTOR_SIZE	2023-08-24 01:55:35 +03:00
Vitaliy Filippov	65487da4b1	Do not include msgr_rdma.h into messenger.h	2023-08-24 01:55:35 +03:00
Vitaliy Filippov	7862282938	Extract validation to check_rw(), remove duplicate code with OP_SYNC	2023-08-13 23:49:52 +03:00
Vitaliy Filippov	30ce2bd951	Fix buffer insert in cluster_client	2023-08-12 11:08:50 +03:00
Vitaliy Filippov	b1a0afd10a	Aggregate buffer flushes	2023-08-11 11:26:13 +03:00
Vitaliy Filippov	85b6134910	Return dirty buffers on read in client ... Required at least to return buffers when they need to be replayed, but until they are actually replayed	2023-08-09 00:57:08 +03:00
Vitaliy Filippov	b1b07a393d	Fix incorrect marking op parts as done with snapshots (could probably lead to client hangs)	2023-08-09 00:57:08 +03:00
Vitaliy Filippov	7333022adf	Add a third I/O mode: O_DIRECT|O_SYNC, change parameters to data_io/meta_io/journal_io	2023-08-09 00:57:08 +03:00
Vitaliy Filippov	6acf562e01	Release 1.0.0 ... New features: - Data and metadata checksums! - Metadata checksums are always used with new disk format - Data checksums can be turned on with --data_csum_type crc32c for new OSDs - Checksum block size can be configured - inmemory_metadata now also affects keeping checksums in memory - Linux page cache I/O caching support which can be enabled separately for data, metadata (including checksums) and journal (O_SYNC instead of O_DIRECT) - Details [here](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/config/layout-osd.en.md#data_csum_type) - Backwards compatibility is preserved, you can use new OSDs with old disks Release also includes bug fixes from [0.9.6](https://git.yourcmc.ru/vitalif/vitastor/releases/tag/v0.9.6). 0.9.6 is moved to "-oldstable" repositories and will be available for some additional time.	2023-07-29 18:57:19 +03:00
Vitaliy Filippov	564df2eb5d	Support using buffered I/O with O_SYNC instead of direct I/O	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	1a4ceb420d	Track used blocks, not object versions	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	21b5124a4b	Document data_csum_type and csum_block_size parameters	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	4181add1f4	Remove creepy "metadata copying" during overwrite ... Instead of it, just do not verify checksums of currently mutated objects. When clean data modification during flush runs in parallel to a read request, that request may read a mix of old and new data. It may even read a mix of multiple flushed versions if it lasts too long... And attempts to verify it using temporary copies of metadata make the algorithm too complex and creepy.	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	a8464c19af	Support keeping checksums on disk (not in memory) ... Definitely beneficial for SSD+HDD setups	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	3c8e4c6b72	Use clean_dyn_size for space check	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	8ef4cf89dc	Log more details about checksum mismatch in big_writes	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	7bfb1639ea	Use find_holes() in flusher for unification	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	628e481c32	Fill journal header to know checksum type & size when dumping journal with --all	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	af6f2046fc	Fix journal read checksum verification with inmemory_journal=false	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	9357e5293e	Call fill_partial_checksum_blocks() correctly in regard to COPY_BUF_CSUM_FILL	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	12851dc07d	Wait for journal reads before checking them in clear_incomplete_csum_block_bits	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	d6ee1ca17c	Use zero checksum size for zero-length writes	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	71674d00cf	Fix journal data checksum mangling on corrupted block overwrite	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	ddb078d5a7	Check journal entry size when checking block checksums	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	d22d56f90a	Fix journal data checksum verification on start	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	eb1331a079	Add more details to "journal entry data is corrupt" messages	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	c5274f655b	...and partially remove the perversion with bitmap inlining	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	45e07d6294	Sadly we have to refcount dyn_data...	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	a8ee391e05	Fix clean block checksum read	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	de48fa3fd2	Allow to forcibly set meta_format	2023-07-29 12:17:18 +03:00