Skip to content

virtio-blk: add sync option to select libkrun's sync mode - #117

Merged
tylerfanelli merged 1 commit into
libkrun:mainfrom
ArtBlnd:blk-sync-mode
Oct 14, 2026
Merged

tylerfanelli merged 1 commit into
libkrun:mainfrom
ArtBlnd:blk-sync-mode

Conversation

@ArtBlnd

@ArtBlnd ArtBlnd commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

krunkit adds every disk with krun_add_disk2, which leaves the flush behavior to libkrun. In libkrun 1.19.6 that call hardcodes the sync mode per platform: SyncMode::Full on Linux, SyncMode::Relaxed on macOS (src/libkrun/src/lib.rs, krun_add_disk2). With Relaxed, a guest VIRTIO_BLK_T_FLUSH ends in imago's File::sync, which calls fsync(2) rather than fcntl(F_FULLFSYNC) when relaxed_sync is set.

On macOS that distinction matters. From fsync(2):

Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.

Specifically, if the drive loses power or the OS crashes, the application may find that only some or none of their data was written. The disk drive may also re-order the data so that later writes may be present, while earlier writes are not.

So a flush the guest sees complete (an ext4 journal commit, a database fsync) is not guaranteed to survive a host power loss, and ordering between flushed writes is not guaranteed either. The virtio 1.2 spec (5.2.6.2) asks a device backed by persistent storage to commit stable writes before reporting the flush complete.

libkrun already lets the caller choose: krun_add_disk3(..., direct_io, sync_mode) with KRUN_SYNC_NONE, KRUN_SYNC_RELAXED and KRUN_SYNC_FULL. krunkit has no way to reach it.

Change

  • virtio-blk accepts an optional sync=none|relaxed|full.
  • Without sync=, krunkit keeps calling krun_add_disk2, so the default is unchanged.
  • With sync=, krunkit calls krun_add_disk3 with read_only=false and direct_io=false (what krun_add_disk2 uses) and the chosen mode.
  • docs/usage.md documents the option; cmdline.rs tests cover parsing and the unset case.

Testing

cargo fmt -- --check, cargo clippy --target aarch64-apple-darwin -- -D warnings and cargo test --release pass (37 tests).

To check what reaches the host, I booted the same Ubuntu 24.04 arm64 disk image three times, each from a fresh APFS clone, with the patched krunkit and no network device, and traced krunkit with fs_usage -w -f filesys for 30 seconds, counting calls on the disk image's file descriptor. Mac mini (Mac16,10, M4), macOS 26.5.1, libkrun 1.19.6 (Homebrew bottle from the slp/krun tap).

virtio-blk argument fsync fcntl(F_FULLFSYNC) boot to getty
(no sync=) 502 0 9 s
sync=relaxed 500 0 8 s
sync=full 0 392 8 s

The counts differ between runs because boot progress within the 30-second window varies; what changes with the option is which call the guest's flushes become. I have not measured the throughput cost of sync=full beyond boot time, and I have not reproduced data loss on power loss; the change rests on the documented semantics above.

Whether full should become the default on macOS is left to you; this PR only makes it selectable.

krun_add_disk2 picks the flush behavior itself, and on macOS that is
KRUN_SYNC_RELAXED: guest flushes become fsync(2), which does not ask the
drive to flush its buffers. Expose krun_add_disk3's sync_mode as
sync=none|relaxed|full; without the option krunkit keeps calling
krun_add_disk2, so the default is unchanged.

Signed-off-by: Jun Ryoung Ju <junryoungju@gmail.com>
@ArtBlnd

ArtBlnd commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Ran into what looks like the failure this is about. A Mac mini (M4, macOS 26.5.1) running krunkit 1.3.2 / libkrun 1.19.6 with no sync= (so relaxed) hosts a VM that is an etcd member. I pulled the Mac's power plug while it was running, and once it was back, etcd in the guest panicked on the first heartbeat from the leader:

panic: tocommit(113807833) is out of range [lastIndex(113807744)]. Was the raft log corrupted, truncated, or lost?

As far as I understand raft, the commit index in a heartbeat is capped at what that follower has acknowledged, and etcd acknowledges entries only after fsyncing its WAL. So it looks like ~89 entries the guest had already flushed didn't make it through the host power loss. etcd had no unsafe-no-fsync and the guest disk reports write_cache = write back, so those flushes went through krunkit.

Not 100% sure nothing else played a part, but it lines up with the fsync(2) note above. I haven't repeated the same pull with sync=full yet; will follow up if I do.

@ArtBlnd

ArtBlnd commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

I've unplugged my mac mini while running etcd for 3 or 4 times. looks like the sync=full fixes the etcd corruption.

@slp

slp commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

Change and code LGTM. The build issue is not related to this PR, should be fixed by #119

Thanks @ArtBlnd !

@tylerfanelli tylerfanelli left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@tylerfanelli
tylerfanelli merged commit 9e39476 into libkrun:main Oct 14, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants