Repository navigation
Conversation
Flip StoragePoolType.RBD from EncryptionSupport.Unsupported to Hypervisor so the existing encryption control plane (allocator, endpoint selector, offerings) treats RBD pools as encryption-capable. The agent-side encrypted RBD create path is not implemented yet; it is delivered by two follow-up tracks (qemu-native engine='qemu' and ceph-native engine='librbd'). Until then, fail closed at the two RBD create chokepoints in LibvirtStorageAdaptor (createPhysicalDisk and createDiskFromTemplate) when a passphrase is present, so we never silently produce a plaintext volume that the control plane believes is encrypted. No change for existing unencrypted RBD volumes (guards only fire when a passphrase is set; supportsEncryption() only affects volumes that require encryption).
Implements encrypted RBD data and root disks using librbd's native LUKS2 encryption, decrypted at runtime by libvirt/qemu via <encryption engine='librbd'>. CloudStack manages the passphrase (existing model). - RbdEncryption: isolated helper wrapping `rbd encryption format luks2`, cephx via --id + keyfile (secret not on the command line), LUKS passphrase via KeyFile. Kept separate so the CLI can later be swapped for a JNA binding (rados-java has no rbd_encryption_format API). - LibvirtStorageAdaptor: create/clone the raw RBD image, then apply `rbd encryption format luks2`; mark the disk LUKS2 so encrypt_format propagates to the volume. Replaces the fail-closed guards. - QemuObject.EncryptFormat: add LUKS2. - LibvirtVMDef: render <encryption format='luks2' engine='librbd'>; the encrypt details now carry an optional engine. - attach (KVMStorageProcessor) and boot (LibvirtComputingResource): set engine='librbd' for RBD-backed encrypted volumes. NOTE: the CoW-clone-then-format path (encrypted root from an unencrypted template) needs live-cluster validation for the parent-grow / usable-size behaviour described in the Ceph image-encryption docs. Builds: api + plugins/hypervisors/kvm (JDK11).
hostSupportsVolumeEncryption() now advertises encryption capability if the host supports EITHER qemu-native LUKS (qemu-img LUKS + cryptsetup) OR librbd native encryption (rbd CLI with the encryption subcommand). Previously a Ceph-only host that lacked cryptsetup would not advertise encryption even though librbd can encrypt RBD volumes. Split into hostSupportsQemuNativeVolumeEncryption() and hostSupportsRbdVolumeEncryption(); kept HOST_VOLUME_ENCRYPTION as the single host-wide flag (documented limitation: not per-pool).
Encrypted RBD volumes are encrypted natively by librbd and must be resized with `rbd resize --encryption-passphrase-file` so librbd grows the encrypted payload and keeps the LUKS header consistent. The existing encrypted-resize path (resizeEncryptedQcowFile) uses qemu-img --object secret, which is for qemu-native LUKS and does not fit the librbd LUKS2 layout. - RbdEncryption.resize(): new `rbd resize` wrapper (cephx via --id + keyfile, passphrase via KeyFile, optional --allow-shrink). - LibvirtResizeVolumeCommandWrapper: detect encrypted RBD and route to the rbd resize path, bypassing the libvirt v.resize and qemu-img paths. Snapshot/revert, RBD<->RBD copy, and migration of encrypted RBD volumes need no code changes: they operate on the raw (LUKS-containing) image at the block level, and the destination passphrase secret is already created engine-agnostic in LibvirtPrepareForMigrationCommandWrapper. These still require live validation. Builds: plugins/hypervisors/kvm (JDK11).
For a running VM, an librbd-encrypted RBD volume must be resized in-band by qemu/librbd, not out-of-band by the rbd CLI. Gate the CLI rbd-resize path on !vmIsRunning so: - offline -> `rbd resize --encryption-passphrase-file` (librbd-aware), and - online -> existing NOTIFYONLY path -> virsh blockresize, where qemu's block_resize delegates to librbd to grow the encrypted payload and notify the guest in one step (no passphrase needed; qemu holds the secret). This avoids notify-less out-of-band growth and qemu/librbd size divergence while the image is open. Online behaviour still needs live validation that blockresize resizes the encrypted payload for engine='librbd' disks.
Root disks could not be encrypted: cloning a plaintext template and then `rbd encryption format`ing the clone leaves the inherited OS data unreadable (the LUKS header offsets it), so the guest could not mount root. Fix, in createDiskFromTemplateOnRBD, with two paths: - Option A (same-cluster cached RBD template): grow the template base to reserve LUKS2 header space, snapshot+protect it (cloudstack-base-snap-luks), clone from it, apply the LUKS2 header, resize the clone to the requested size. Inherited template data stays readable through the clone's encryption and the clone is a thin CoW image (only the header is written). - Option B (first-use / non-RBD template): create an empty image, apply a LUKS2 header, then import the template THROUGH the encryption layer via RbdEncryption.importTemplate (qemu-img convert -n into encrypt.key-secret). Correct but a full copy. Validated end-to-end on Ubuntu 26.04 / libvirt 12.0.0: both boot; A is thin (3.5 GiB provisioned, ~120 MiB used); LUKS2 verified at rest on Ceph.
Review pass over the librbd LUKS2 encryption feature to fix latent issues
and bring it in line with CloudStack conventions:
- RbdEncryption: reject empty/null passphrase with a clear error; round
rbd --size up to MiB so a non-aligned request never shrinks the volume
below what was asked for; create the temporary cephx conf/keyring 0600
explicitly instead of relying on the umask.
- LibvirtStorageAdaptor: close Rados/IoCTX/RbdImage in a finally block on
the encrypted-root paths (mirrors deleteVolume) so handles are not
leaked on exceptions; use parameterized log messages instead of string
concatenation; extract the encrypted-root Option A/B logic into
createEncryptedRootCoWClone / createEncryptedRootFullCopy.
- RbdEncryption: use an instance logger (matching the plugin convention)
and split argv construction into build{Format,Resize,Convert}Script so
the generated commands can be unit-tested.
Assert the rbd/qemu-img argv built for format, resize and convert-through-encryption (RBD and file sources), and that empty/null passphrases are rejected. Command construction is verified without a live Ceph cluster.
libvirt 10.0.0 has an object apply-order bug (fixed in 10.1.0) that breaks hot-plug of an encrypted rbd blockdev: on attach the disk is opened before its LUKS secret object is defined, so the attach fails with "No secret with id '...-format-encryption-secret0'". Booting a VM from an encrypted RBD disk is unaffected (the QEMU command line resolves all -object before -blockdev). Refuse the attach up front with a clear error (mirroring the existing openvswitch/io_uring libvirt-version gates) instead of letting libvirt fail opaquely. Only the RBD hot-plug path is gated; boot/root/detach are untouched.
RbdEncryption built its own 'qemu-img convert' command line, which duplicated qemu-img knowledge outside of QemuImg. QemuImg could only write to a plain filename destination, so importing a template through the librbd encryption layer was not expressible with it. QemuImg now supports a destination described by image options (--target-image-opts, with -n implied since such a target always exists already), exposed as convertIntoExistingTarget(). QemuImageOptions can render its parameters under either image-opts flag. RbdEncryption.importTemplate now composes QemuImageOptions and a QemuObject secret and delegates to QemuImg; its hand-built convert script is removed. The rbd CLI calls (encryption format, resize, support probe) stay, as qemu-img cannot perform them. No functional change to the generated command.
Review feedback: single-letter and abbreviated names are hard to read. Renamed in the two methods added by this PR only (renaming the rest of the class is out of scope here): r -> radosConnection, io -> ioContext, rbd -> rbdClient, base -> templateImage, s -> snapshotInfo, encSnap -> luksReservedSnapshotName, haveEncSnap -> luksSnapshotExists, createSize -> imageSizeWithLuksHeader, srcIsRbd -> sourceIsRbdPool. No functional change.
Review feedback: distinguish the two volume encryption mechanisms instead of advertising them under one host flag. host.volume.encryption goes back to meaning qemu-native LUKS only (qemu-img LUKS + cryptsetup), as it did before this PR, and hosts now additionally report host.volume.encryption.rbd for librbd encryption (rbd encryption format). The deployment planner requires the flag matching the pool type of each encrypted volume - librbd for volumes on RBD pools, qemu-native for any other pool type - at all three places it validated encryption support before. The pool is taken from the pools proposed alongside the host when present, so first deployments are matched accurately too; an encrypted volume with no pool yet accepts either mechanism and the storage pool allocator picks a pool the host can serve. This also stops a host whose librbd is too old for 'rbd encryption format' from being selected for encrypted RBD volumes; it previously advertised encryption through the qemu stack and the VM failed to start.
takeRbdVolumeSnapshotOfStoppedVm() called image.snapCreate(snapshotName)
twice. The first call creates the RBD snapshot, the second one always
throws RbdException ("Failed to create snapshot <uuid>") because the
snapshot already exists.
The duplicate is a merge artifact: 30d3066 ("Merge branch '4.20' into
4.22") resolved a conflict by keeping the call from both sides - each
parent had exactly one.
Because there was no finally block, that exception skipped rbd.close(image)
and r.ioCtxDestroy(io), so the agent kept the image open and held its RBD
exclusive-lock indefinitely. The exception is only logged, so the snapshot
job still reported success and the fault stayed invisible.
Consequences observed on a KVM + Ceph/RBD cluster:
- revertSnapshot fails with "com.ceph.rbd.RbdException: Failed to rollback
snapshot <uuid>". librbd returns EROFS because a live peer holds the
exclusive-lock; 'rbd snap rollback' only succeeds once that client dies
and librbd can break the lock, which makes the failure look intermittent.
- getRbdSnapshotSize() is never reached, so every snapshot is reported with
physical size 0 when snapshot.backup.to.secondary is false.
- The leaked watchers keep the image busy, so 'rbd rm' fails and the volume
cannot be expunged - it stays stuck in state Destroy.
Note the method also runs for RUNNING VMs: createSnapshot() branches on
"RUNNING && !primaryPool.isExternalSnapshot()", and RBD is an
external-snapshot pool, so every RBD volume snapshot took this path.
Remove the duplicated call and move the image/IO-context cleanup into a
finally block so the lock is released even if the snapshot itself fails.
…napshot createRBDvolumeFromRBDSnapshot() closed the source image, the cloned image and the RADOS IO context only on the success path, and called snapUnprotect() only there too. Two paths escaped that cleanup: - the early "Could not find snapshot ... on RBD" return, and - any RadosException/RbdException from clone(), resize() or flatten(), which is caught and turned into a null disk. Both leave the images open, so this client keeps the RBD exclusive-lock. That later makes 'rbd snap rollback' (revertSnapshot) fail with EROFS from another host, and keeps the image busy so 'rbd rm' cannot remove it - the volume then stays stuck in state Destroy. The failure paths after snapProtect() are worse: the snapshot stays protected, and a protected snapshot can be deleted neither on its own nor together with its volume. Move the cleanup into a finally block, tracking whether the snapshot was actually protected so it is unprotected exactly when it needs to be. Failures during cleanup are logged and never mask the original outcome; a failed snapUnprotect is logged at ERROR since it needs manual intervention. This is the same class of defect as the leak fixed in takeRbdVolumeSnapshotOfStoppedVm(); no behaviour changes on the success path.
Two tests around takeRbdVolumeSnapshotOfStoppedVm, using the MockedConstruction pattern already used in this test class (the Rbd instance is created inside the method under test, so it cannot be injected): - createsSnapshotExactlyOnce guards the duplicated snapCreate call from coming back, and checks the image and IO context are released. - releasesHandlesWhenSnapshotFails makes snapCreate throw and asserts the image is still closed and the IO context destroyed, so a future failure cannot leak the RBD exclusive-lock again. takeRbdVolumeSnapshotOfStoppedVm, radosConnect and getRbdSnapshotSize widened from private to protected so the test can stub the Ceph interactions.
Address review feedback: the nested try-catch teardown blocks in the finally clauses of takeRbdVolumeSnapshotOfStoppedVm and createRBDvolumeFromRBDSnapshot are extracted into two reusable, null-safe helpers that log but never throw: - closeRbdImage (3 call sites) - destroyRadosIoCtx (2 call sites) No behavior change.
Follow-up to the teardown helpers: the snapUnprotect block in the finally clause of createRBDvolumeFromRBDSnapshot moves into a never-throwing unprotectRbdSnapshot helper, so the finally clause is now free of inline try-catch constructions entirely. No behavior change.
An in-place restore of a KVM instance with more than one data disk can write a backed up disk into a different volume than the one it came from. NASBackupProvider.restoreVMBackup pairs the backed up volumes with the instance's current volumes by position in two lists that are each sorted by device id, and LibvirtRestoreBackupCommandWrapper.restoreVolumesOfExistingVM consumes those lists index by index. The volume UUID is carried along but only used to build error messages. That pairing is only correct while the instance's device ids still match the ones recorded in the backup, and the restore itself breaks that: on completion KVMGuru.importVirtualMachineFromBackup re-attaches every data disk with getNextAvailableDeviceId(). The volumes are never detached first, so their current ids still count as in use and the helper cannot return the id a volume already holds - every data disk is shifted by one slot on each restore. The root disk is unaffected because it is always re-attached at device id 0. A second restore therefore maps the backup files onto the wrong volumes. With equally sized disks the contents are silently exchanged and the API reports success. With disks of different sizes the first volume is overwritten and the restore then fails converting an image into a smaller volume, leaving that volume half rewritten with no rollback. Three changes: - KVMGuru: re-attach data disks with the device id recorded in the backup, falling back to the next free id only when the backup has none. - LibvirtRestoreBackupCommandWrapper: select the target volume by UUID instead of by list position, and fail explicitly when a backed up volume is no longer attached to the instance. - NASBackupProvider: derive the UUID list and the backup file list from the same sorted collection so the two cannot drift apart. The create-instance-from-backup path (restoreVolumesOfDestroyedVMs) is left alone: it provisions new volumes with new UUIDs, so it has nothing to match on.
LibvirtRestoreBackupCommandWrapperTest: restoring an instance whose volumes arrive in a different order than the backed up volumes must still write each backup into the volume it was taken from, and must fail explicitly when a backed up volume is no longer attached to the instance. KVMGuruTest: importing an instance from a backup must re-attach its data disks on the device ids recorded in the backup rather than allocating new ones.
… uuid Matching backed up volumes to the instance's volumes by uuid breaks creating an instance from a backup: that instance gets brand new volumes, so none of the uuids recorded in the backup can ever match and the restore always failed with the volume 'is not attached to the instance any more' error. When none of the recorded uuids match, the volumes are new and the device id ordering both lists already carry is the only mapping available, so fall back to it. When some of them do match, a missing one really is a detached volume and is still rejected, which is what the uuid matching was added for. The fallback also refuses to run if the two lists differ in length.
Two defects in one line, both fatal for a stopped-instance backup:
mount ... 2>&1 | tee -a "$logFile"
if [ $? -eq 0 ]; then
1. `tee` writes to the log AND to stdout, so anything the mount helper says
lands in the stream the agent parses. `LibvirtTakeBackupCommandWrapper
.parseBackupSize` sums EVERY line of that output as a number when volume
paths are set - which is only for a STOPPED instance - so a single benign
warning throws NumberFormatException. A Ceph client reporting a missing
keyring (harmless: the mount succeeds) is enough. The record then sits in
BackingUp for ever: there is no timeout, no transition to Failed, and it
blocks every later restore of that instance.
Measured on 4.23.0.0-SNAPSHOT with Ceph primary storage: without the
keyring the backup wedges with
`NumberFormatException: For input string: "2026-08-25T17:38:38.403+0000"`
- the timestamp of the mount warning. A running instance is unaffected,
because with no volume paths the parser reads only the last line.
2. `$?` after a pipeline is the exit status of the LAST command, i.e. `tee`,
which practically never fails. A mount that did fail was therefore
reported as "Successfully mounted" and the backup carried on against an
empty directory.
Same change as the pending upstream PRs apache#12843 and apache#12898.
parseBackupSize() parsed every line of the backup script's stdout as a byte count. The script is not the only writer on that stream: mount helpers and storage clients emit warnings of their own, and one such line was enough to throw NumberFormatException out of the wrapper, e.g. java.lang.NumberFormatException: For input string: "2026-08-25T17:38:38.403+0000" emitted by a Ceph client as "unable to find a keyring on /etc/ceph/ceph.keyring". The cost of that is out of proportion to the cause. The exception escapes after the backup has already been written, so the data is on the repository but the record stays in BackingUp - there is no timeout and no transition to Failed - and it blocks every later restore of that instance until the row is corrected by hand. Ignore lines that do not begin with a byte count instead, logging them at debug. Only the multi-volume branch was reachable in practice, which is the branch taken for a stopped instance, so running instances backed up normally and the failure went unnoticed. Adds LibvirtTakeBackupCommandWrapperTest; four of its seven cases fail without this change.
An encrypted volume was backed up in the clear: libvirt's backup-begin copies from the top of the disk's block graph, which is the decrypted guest view, and the backup target carried no encryption. Restore then wrote that plaintext back over a volume whose record still says luks, leaving an image the hypervisor could not open. This affected every encrypted volume, not just RBD ones, since the leak is independent of which engine decrypts the source. QEMU now writes the backup already LUKS-encrypted, using the volume's own passphrase, via the encryption element libvirt allows inside the backup target. The stopped-VM path writes the same artefact through qemu-img. Restore reads the backup with that passphrase and writes it back through the volume's own encryption, so the restored volume keeps its LUKS header. The passphrases travel on the take/restore commands, are marked so they never reach the logs, and are wiped once handled. The agent registers an ephemeral libvirt secret per volume under a distinct consumer name, so cleanup cannot remove the secret a running VM is using for the same disk. Encrypted backups onto non-RBD pools are rejected explicitly rather than silently corrupting the volume.
Makes the existing take-backup test explicit about its volumes being unencrypted, and adds a test asserting the passphrase of an encrypted volume is sent on the command, which is what lets the agent have QEMU write the backup already encrypted.
Three problems only a real restore surfaced: The pre-restore 'qemu-img check' ran without a key, so an encrypted backup looked corrupt and the restore refused to start. The passphrase map is keyed by the volume path as the management server knows it, but the agent sees that path with a pool prefix, so the lookup missed and the code silently fell through to the unencrypted path. QemuImageOptions built from a raw map has no format set, and in that case renders as a bare filename, dropping the backup's own encrypt.key-secret; the source options now force the image-opts form.
The backport of the restore-volume-mapping regression tests carried an unused-import cleanup that was correct on the branch it was written against but not on 4.23.0.0, where ArrayList and Arrays are used by the backup repository mount handling. EnumMap is needed by restoreEncryptedBlockDevice.
LibvirtRestoreBackupCommandWrapperTest: the restore path in 4.23 first asks qemu-img whether the backup references a parent and flattens the chain with qemu-img convert when it does; only a backup with no backing file is rsynced. The tests carried over from the 4.22 branch stubbed runSimpleBashScriptForExitValue to 0 for every command, which made the probe report a backing chain and take the convert path, so the rsync argv they assert on was never executed. NASBackupProviderTest: drop an ArgumentCaptor import duplicated when the encrypted-volume test was merged onto 4.23's import block, and pass the isolated flag that takeBackup gained in 4.23.
…ance countPendingNonPseudoJobs() filtered on instance_type != 'Thread'. VmWork jobs (VmWorkStart, VmWorkStop, VmWorkAttachVolume, ...) carry a NULL instance_type, and NULL != 'Thread' is not true in SQL, so they were never counted. The maintenance drain then reported 0 pending jobs and moved the agents away while VM work was still running on the management server, which failed that work. Count NULL instance types as well, and use the same ownership rule as the jobs a restarting management server fails (getResetJobs): executing on this server, or queued on it and not picked up yet.
…ng commands Preparing a management server for maintenance migrates its indirect agents with MigrateAgentConnectionCommand. The agent reconnected 3 s later whatever was still running, and the old management server treated the closed link as a failure: commands in flight lost their answers, the host went Alert with an agent investigation and HA work, and commands sent meanwhile failed with AgentUnavailableException. Under load every agent move failed operations, and once a live migration completed in libvirt while CloudStack recorded it as failed. Make the move a planned handoff: - Before the move command, put the host into Rebalancing. A disconnect while Rebalancing stays Rebalancing (no Alert, no investigation), and the agent's ping does not ask for a new startup that would pull it back. - Wait until this management server has no command queued or awaiting an answer for the host (indirect.agent.migration.idle.wait, default 600 s); otherwise leave the agent where it is. - Send the move command with send(): easySend() refuses hosts that are not Up/Connecting. - Hold new commands for a host that is Rebalancing or Connecting until it is Up on its new owner (agent.handoff.wait, default 180 s), instead of failing them. The move command itself is never held. - On the agent, finish running and queued commands before reconnecting, so their answers still go back over the current link. - Move one host at a time (indirect.agent.migration.parallelism, default 1), because the planner skips Rebalancing hosts. - If the agent does not come Up elsewhere, reconnect it instead of leaving the host Rebalancing.
When a management server restarts quickly, ClusterManagerImpl only notices the
new runid ("left and rejoined quickly") and raises no left/joined event, so the
other server keeps its cached channel to the old process. The first write to a
channel the peer has closed still succeeds locally and its data is lost; only
the next write fails with "Broken pipe" and triggers a reconnect. The lost write
is typically an agent answer routed back to the restarted server, so the job
waiting for it hangs until its timeout.
Peer channels are only ever written to. Before reusing a cached one, do a
non-blocking read: -1 means the peer closed it, so open a new connection first.
The VM detail rootDiskController=virtio-blk has never been able to start a VM. DiskBus renders its own value into the libvirt disk XML, and 'virtio-blk' is not one of libvirt's target buses (ide, scsi, virtio, xen, usb, sata, sd, fdc, uml), so libvirt rejects the domain and every host fails the start with: XML error: Invalid value for attribute 'bus' in element 'target': 'virtio-blk' The point of the controller was a virtio disk with discard='unmap', so that combination has been unreachable: only SCSI disks get discard on KVM today, and plain virtio disks keep libvirt's default discard='ignore', which drops a guest's TRIM before it reaches the storage. Render the libvirt bus name rather than the enum value, leaving toString() and fromValue() alone so the detail keeps parsing, and label virtio-blk disks vd* as virtio disks are labelled. Data disks now follow a virtio-blk root as they already follow a SCSI one, so a VM that opts in gets discard on all its disks, and the second attach path gives a hot-plugged disk the same discard the volume-attach path does. A disk being attached cannot recover the controller from the running domain XML, where a virtio-blk disk is indistinguishable from a virtio one, so a data disk with no controller of its own follows the root detail; without that a hot-plugged disk silently loses discard until the next stop/start. Verified on a KVM host with Ceph RBD primary storage: the VM starts, every disk is bus='virtio' with discard='unmap', guest device names are stable across reboot and stop/start, and after an fstrim in the guest 700 MiB of the 728 MiB written was returned to the pool. (cherry picked from commit 430ecf0) (cherry picked from commit 95397d23696ad42e780d2637497b6aefe379a1b1)
… win on attach A disk with iothreads enabled got its iothread= binding only when its bus was VIRTIO, although a virtio-blk disk is rendered with the same libvirt bus. A virtio-blk VM with the iothreads detail therefore ran every disk with io='threads' but no iothread, and data disks following a virtio-blk root now lost it as well. Treat VIRTIOBLK like VIRTIO there. When a data disk is attached without a controller of its own, the virtio-blk root fallback ran before the scan of the running domain. The root detail can be changed on a running VM, so a VM started on SCSI whose detail was later set to virtio-blk would get a virtio disk hot-plugged next to its sd* ones. Scan the running disks first and use the fallback only when no SCSI disk is there. (cherry picked from commit 3d7ab65) (cherry picked from commit daa034857f23f179990b2a9a82914fd4ab9f279d)
…orageProcessorTest (cherry picked from commit 3af6ee15b33c968c8ed792192529cc3b01b98021)
A virtio-blk disk is written to the domain XML with libvirt's 'virtio' bus, so the running domain reports it back as VIRTIO and the VIRTIOBLK checks on the parsed disks are dead code. Attaching to a virtio-blk VM is handled by the fallback to the VM's rootDiskController detail in getAttachDiskBusType(). LibvirtComputingResource.attachOrDetachDisk() is back to what it was; nothing calls it. (cherry picked from commit 7233d17)
An encrypted root is a librbd CoW clone of a plaintext template carrying a LUKS2 header. librbd serves the ranges the clone's own objects do not hold from that parent, as plaintext, which is what makes the inherited filesystem readable. That stops the moment an object exists in the clone: one 4 KiB guest write copies up the whole 4 MiB object, and the ranges inside it that the parent never materialised are from then on read through the crypto layer - where zeros decrypted with AES-XTS are not zeros. Before taking the snapshot encrypted roots are cloned from, rewrite the template image from its own cloudstack-base-snap with qemu-img convert -S 0, so the snapshot has no holes left. The snapshot gets a new name, -luks2, so templates prepared before get a dense one too; roots already cloned from the old -luks snapshot keep it. If another host prepares the same template at the same time, its protected snapshot is used instead of failing the deploy. QemuImg grows setWriteZeroRanges() for the -S 0 this needs, since qemu-img skips the source's zero ranges by default. (cherry picked from commit 764b22b)
On a redundant router dnsmasq listens on the gateway (VIP) and uses it as its DHCP server identifier. dhcp_release puts the first address of the interface in that option, which is the router's own address, so dnsmasq drops every DHCPRELEASE without logging anything. The lease of an expunged VM stays in dnsmasq's memory and the next VM given that address is refused it. The fallback added by apache#13194 cannot help: it replaces the leases file and reloads dnsmasq, but dnsmasq reads the file only when it starts, so the lease stays in memory, and from then on dnsmasq writes to the deleted file. Send the DHCPRELEASE with the address dnsmasq listens on in the VM's network as the server identifier. Give dnsmasq a moment to drop the lease before falling back, and make the fallback edit the file in place and restart a running dnsmasq. (cherry picked from commit d0ca8c5)
dnsmasq does not run on the backup of a redundant pair, and on routers with IPv6 on the guest NIC it keeps the leases file read-only (leasefile-ro). In both cases it never takes a released lease out of the file, so every expunge waited the full 2 s and then restarted dnsmasq; on a dual-stack router that restart also dropped every other lease dnsmasq held in memory. There the line is now taken out of the file right away, without a restart, and the backup no longer sends a release to the primary's VIP. Also read the leases file before going through it: dnsmasq and remove_lease rewrite it in place, and reading it while that happens skipped leases once the file outgrew one read buffer. (cherry picked from commit d9058a9)
When the release of several leases did not take, each of them waited up to 2 s and restarted dnsmasq on its own, so a run with many leftover leases kept DHCP and DNS on the router going down again and again. Send all the releases first, wait for them together, then take whatever is left out of the leases file in one pass and restart dnsmasq at most once. When dnsmasq releases the leases, as it normally does, nothing is restarted, as before. (cherry picked from commit 70a586e)
Each backup poll checks every schedule that carries an async job id before it submits the scheduled backups that are due. It dereferenced getAsyncJob(id).getStatus() without handling a missing job, but the async job GC expunges expired jobs, unfinished ones included, and does not clear the backup schedules that still reference them. The NullPointerException then ended the whole poll before scheduleBackups() ran, on every tick, so no schedule handled by that poll fired again until a management server restart reset the job ids. A schedule whose job no longer exists is now treated as finished and moved on to its next run, and each schedule is checked on its own, so one bad record can no longer stop the others. Also log the failure to clean up an Instance after a failed create Instance from backup at WARN rather than DEBUG: the Instance remains, and its volumes may hold none of the backup's data. (cherry picked from commit ceb8336)
haproxy cannot balance UDP, so the virtual router refused UDP load balancer rules although it lists udp as a supported protocol. UDP rules are now left out of the haproxy configuration and sent to the router as a separate list. The router serves them with IPVS in NAT mode (it is the gateway of the backends), keeping the IPVS table in sync with the rules instead of flushing it, and opens the UDP port in its firewall for the rule's source CIDRs. - algorithms map to IPVS schedulers: roundrobin -> rr, leastconn -> lc, source -> sh; source based stickiness becomes IPVS persistence with its expire value as the timeout - other stickiness methods and health check policies are refused for UDP rules - a UDP rule gets a UDP firewall rule, and its protocol cannot be switched between udp and the tcp based protocols
calvix
force-pushed
the
integration/all-fixes-4.23.0.0
branch
from
October 29, 2026 13:06
1e9186c to
d5907dc
Compare
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sketch: UDP load balancer rules on the virtual router, served by IPVS (NAT mode) instead of haproxy.
Changes
expirebecomes the IPVS persistence timeout), health check policies are refused. A UDP rule gets a UDP firewall rule, and an update cannot switch a rule between udp and the tcp based protocols.haproxy.cfgand haproxy's (tcp) firewall rules and sent as a newudp_ruleslist (VIP, port, scheduler, persistence, backends, source CIDRs). Algorithms: roundrobin -> rr, leastconn -> lc, source -> sh.CsLoadBalancer.py): opens the UDP port per source CIDR, syncs the IPVS table withipvsadm(adds/removes only the differences), setsnet.ipv4.vs.conntrack=1.Tested
Unit tests only (
HAProxyConfiguratorTest,NetworkHelperImplTest,LoadBalancingRulesManagerImplTest,VirtualRouterElementTest,ConfigHelperTest), build with-P developer. Not tried on a router yet.Open
ilb.sh) not covered