Skip to content

kvm: fix delete of failed Instance Snapshot on stopped VM with RBD storage - #14456

Open
weizhouapache wants to merge 1 commit into
apache:4.22from
weizhouapache:4.22-fix-remove-error-vmsnapshot
Open

weizhouapache wants to merge 1 commit into
apache:4.22from
weizhouapache:4.22-fix-remove-error-vmsnapshot

Conversation

@weizhouapache

Copy link
Copy Markdown
Member

Description

StorageVMSnapshotStrategy.canHandle() required either State.Allocated or a persisted vm_snapshot_details/STORAGE_SNAPSHOT row before it would claim a VM snapshot. A snapshot that fails before any underlying disk snapshot is taken (e.g. VM freeze fails because the QEMU guest agent is not connected) transitions to State.Error without ever persisting such a row, so canHandle() returned CANT_HANDLE and deletion fell back to DefaultVMSnapshotStrategy. That strategy sends a DeleteVMSnapshotCommand to the KVM agent, which requires a live libvirt domain - failing with 'Domain not found' once the VM is stopped, and blocking both snapshot deletion and VM destruction.

Let StorageVMSnapshotStrategy also claim ownership of snapshots in State.Error even when no disk-snapshot details were persisted: there is nothing on the hypervisor to clean up in that case, and deleteVMSnapshot()/deleteDiskSnapshot() are already safe no-ops when the details list is empty - no libvirt domain is required.

Fixes #13972

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

…orage

StorageVMSnapshotStrategy.canHandle() required either State.Allocated or
a persisted vm_snapshot_details/STORAGE_SNAPSHOT row before it would
claim a VM snapshot. A snapshot that fails before any underlying disk
snapshot is taken (e.g. VM freeze fails because the QEMU guest agent is
not connected) transitions to State.Error without ever persisting such
a row, so canHandle() returned CANT_HANDLE and deletion fell back to
DefaultVMSnapshotStrategy. That strategy sends a DeleteVMSnapshotCommand
to the KVM agent, which requires a live libvirt domain - failing with
'Domain not found' once the VM is stopped, and blocking both snapshot
deletion and VM destruction.

Let StorageVMSnapshotStrategy also claim ownership of snapshots in
State.Error even when no disk-snapshot details were persisted: there is
nothing on the hypervisor to clean up in that case, and
deleteVMSnapshot()/deleteDiskSnapshot() are already safe no-ops when the
details list is empty - no libvirt domain is required.

Fixes apache#13972
@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 18.07%. Comparing base (a7697fe) to head (c18ebe2).
⚠️ Report is 1 commits behind head on 4.22.

Files with missing lines Patch % Lines
.../storage/vmsnapshot/StorageVMSnapshotStrategy.java 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               4.22   #14456   +/-   ##
=========================================
  Coverage     18.06%   18.07%           
- Complexity    16356    16362    +6     
=========================================
  Files          5937     5937           
  Lines        537015   537015           
  Branches      65830    65830           
=========================================
+ Hits          97026    97059   +33     
+ Misses       428913   428869   -44     
- Partials      11076    11087   +11     
Flag Coverage Δ
uitests 4.04% <ø> (ø)
unittests 19.15% <0.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@weizhouapache

Copy link
Copy Markdown
Member Author

Manual test of this change on KVM with Ceph RBD primary storage (4.23.0 build with this commit, deployed on both management servers):

Setup

  • kvm.vmstoragesnapshot.enabled=true
  • Debian 12 VM on a KVM host, root volume on an RBD pool, no connected QEMU guest agent (the cloud image has none by default)

Steps and results

  1. Create a quiesced disk-only Instance Snapshot (createVMSnapshot with quiescevm=true). It fails as in Unable to delete failed Instance Snapshot on stopped KVM VM with RAW/RBD storage #13972:
    Could not freeze Instance. Failed to frozen VM - i-2-1070-VM due to Guest agent is not responding: QEMU guest agent is not connected
    The snapshot is left in Error state (type Disk), no disk snapshot was taken.
  2. Stop the VM.
  3. deleteVMSnapshot on the failed snapshot: success: true. The VM has no snapshots left, and there is no Domain not found error.
  4. Destroy and expunge the VM: succeeds, the VM is gone.

Unit tests

  • VMSnapshotStrategyKVMTest: 7 tests pass (the 5 existing ones plus 2 for canHandle()):
    • an Error snapshot with no STORAGE_SNAPSHOT details is handled by StorageVMSnapshotStrategy (returns HYPERVISOR);
    • a Ready snapshot with no details is still CANT_HANDLE (existing behaviour kept).

Not tested

  • I did not run the same steps on a build without this change, so I have not seen the original Domain not found failure in this environment. The reasoning is from the code: canHandle() returned CANT_HANDLE for such a snapshot, so the delete went to DefaultVMSnapshotStrategy, which needs a libvirt domain.
  • Other storage types, and a failure after some disk snapshots were already taken (this case was already handled, because the details exist).
  • The 4.22 branch itself (the unit tests ran on a 4.23.0 base).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to delete failed Instance Snapshot on stopped KVM VM with RAW/RBD storage

1 participant