Unable to delete failed Instance Snapshot on stopped KVM VM with RAW/RBD storage #13972

problem

When using KVM with RBD primary storage and kvm.vmstoragesnapshot.enabled=true, an Instance Snapshot that fails during creation before the underlying storage snapshot is created cannot be deleted while the VM is stopped.

The failed Instance Snapshot remains in CloudStack in an error state. When CloudStack later attempts to delete the snapshot, the snapshot is handled by DefaultVMSnapshotStrategy instead of StorageVMSnapshotStrategy.

For a stopped VM backed by RAW/RBD storage, DeleteVMSnapshotCommand then fails because the libvirt domain does not exist:

Delete Instance Snapshot failed due to org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'

Starting the VM allows the same failed Instance Snapshot to be deleted successfully.

This appears to be a RAW/RBD variant of #11673. PR #11687 fixes a similar case for QCOW2/stopped VMs, but the fallback implemented there only handles QCOW2 volumes.

versions

Apache CloudStack: 4.22.1
Hypervisor: KVM
Primary storage: Ceph RBD
Volume format: RAW
Instance Snapshot type: Disk
kvm.vmstoragesnapshot.enabled=true

The steps to reproduce the bug

Deploy a KVM VM whose ROOT volume is stored on Ceph RBD.
Enable: kvm.vmstoragesnapshot.enabled=true
Run the VM without a working QEMU guest agent.
Create a disk-only Instance Snapshot with VM quiescing enabled.
Snapshot creation fails during the freeze operation: StorageVMSnapshotStrategy

Failed to frozen VM - i-2-43-VM due to Guest agent is not responding:
QEMU guest agent is not connected

Failed to create Instance Snapshot
The Instance Snapshot remains in CloudStack in a failed/error state.
Stop the VM.
Try to delete the failed Instance Snapshot, or destroy the VM so that CloudStack runs VmWorkDeleteAllVMSnapshots.

Actual result

CloudStack does not use StorageVMSnapshotStrategy to clean up the failed snapshot.

The deletion is handled by DefaultVMSnapshotStrategy:

KvmFileBasedStorageVmSnapshotStrategy ... cannot handle [...]
...
DeleteVMSnapshotCommand
...
DefaultVMSnapshotStrategy

The command sent to the KVM agent contains an RBD/RAW ROOT volume:

"poolType":"RBD"
"format":"RAW"
"volumeType":"ROOT"
"vmName":"i-2-43-VM"

The agent then fails because the stopped VM has no libvirt domain:

DeleteVMSnapshotAnswer:
result=false

Delete Instance Snapshot failed due to org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'

The VM destroy operation also fails because orchestrateDeleteAllVMSnapshots() cannot remove the failed Instance Snapshot:

CloudRuntimeException:
Failed to destroy vm with specified vmId

Caused by:
Delete Instance Snapshot ... failed due to
org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'

Expected result

A failed storage-based Instance Snapshot whose creation stopped before any underlying storage snapshot was created should be removable while the VM is stopped.

The cleanup should not require a libvirt domain to exist.

In particular, if:

the Instance Snapshot is in Error state,
it was created using the storage-based KVM snapshot strategy,
there are no kvmStorageSnapshot details / underlying volume snapshots to remove,

then deleting the Instance Snapshot should clean up the CloudStack metadata and return success.

VM destruction should also not be blocked by such a failed Instance Snapshot.

Additional observations

The Instance Snapshot creation originally uses StorageVMSnapshotStrategy:

StorageVMSnapshotStrategy
Freeze of virtual machine took 0 milliseconds.

Failed to create Instance Snapshot ...
Could not freeze Instance.
QEMU guest agent is not connected

The failure happens during the VM freeze operation, before the actual RBD snapshot creation.
As a result, there is no underlying RBD snapshot associated with the failed Instance Snapshot.

During a later delete, however, CloudStack falls back to DefaultVMSnapshotStrategy.
For stopped RAW/RBD VMs, the KVM LibvirtDeleteVMSnapshotCommandWrapper cannot use the QCOW2 offline fallback, because the ROOT volume format is RAW.

Workaround

Starting the VM creates the libvirt domain again.
After starting the VM, deleting the exact same failed Instance Snapshot succeeds:

DeleteVMSnapshotAnswer: result=true

No RBD snapshot is actually being deleted in this case. The presence of the libvirt domain merely allows the wrapper to determine that the requested libvirt snapshot does not exist and return success.

The current workaround is therefore:

Start VM

• delete failed Instance Snapshot
• stop/destroy VM

This should not be necessary for a failed storage-based Instance Snapshot.

What to do about it?

No response