Unreleased regression (4.20, 4.22, main): the orphaned-network cleanup from #12409 shuts down networks while a VM start is implementing them, so the start fails with "Network is not yet fully implemented"

@Pearl1594

problem

While testing instance boot groups (#13907) on a main nightly build, boot group starts failed with "Network is not yet fully implemented". The cause turned out to be unrelated to boot groups. It is the orphaned-network cleanup added by #12409.

#12409 solves a real problem: a network left in Implementing after a crash or a lost job could not be used again, and cleaning those up is valuable. The problem is only in how a "stuck" network is detected. The cleanup also catches networks that a VM start is implementing at that moment.

#12409 is in no release yet:

It was merged into 4.20 on 2026-09-09 (milestone 4.20.4) and merged forward into 4.22 and main on 2026-09-10.
None of 4.20.3.x, 4.22.1.x or 4.23.0.0 contains it.
Unless it is fixed, it ships with 4.20.4 and with the next releases cut from 4.22 and main. The next release from main would ship it together with boot groups.

What the code does (AsyncJobManagerImpl.cleanupOrphanedNetworks(), framework/jobs):

It runs in the async-job GC task every 10 seconds (GC_INTERVAL).
It lists networks in state Implementing. It skips a network whose creation time is within job.expire.minutes (default 1440, one day). Once a network is more than a day old, this check always passes, however briefly the network has been Implementing.
It treats a network as busy only if a pending async job has instance type Network and that network's id. When a VM start or deployment implements a network, the work runs inside the VM's own job (VmWorkStart, instance type VirtualMachine), so the cleanup never sees it.
It fires Network.Event.OperationFailed (Implementing to Shutdown). If the state machine refuses, it sets Shutdown directly in the database. It does this without the network lock, which the implementing thread holds (NetworkOrchestrator.implementNetwork, acquireInLockTable).
Implementing an isolated network includes booting its virtual router. That took about 36 s here, so a 10-second tick lands inside the implementation almost every time. When the start then prepares the NICs, the network is in Shutdown, and the start fails:
ResourceUnavailableException: Resource [Network:209] is unreachable: Network is not yet fully implemented

Trace (management-server.log, network id 209):

01:19:46-50: the network GC shut the network down (router stopped).
01:20:23.709: a VM start took the network's lock and asked the guru and VirtualRouter to implement it. The router's StartCommand went out at 01:20:24.217.
01:20:31.650: Found orphaned network 209 in Implementing state without async job. Network created: 2026-10-27T14:32:37, age: 2087 minutes, expiration threshold: 1440 minutes. Transitioning to Shutdown state., followed by Successfully transitioned orphaned network 209 to Shutdown state using state machine.
01:21:00.246: the router start finished, and the lock was released at 01:21:02.030.
01:21:02.048: NIC prepare failed with "Network is not yet fully implemented", and the start failed.

Why we are confident about the cause: the test networks were created on 2026-10-27 at 14:32 UTC.

For the first 24 hours, the same check found networks in Implementing and skipped them as too young. Nothing failed.
From 2026-10-28 14:32 on, every cleanup run that found a network in Implementing marked it as orphaned, and the start implementing it failed. That was 7 cleanup runs and 17 failed boot group starts.
One plain startVirtualMachine call also failed its first attempt this way, and the API's retry passed.

Impact:

Which networks: any network older than job.expire.minutes (one day by default, so practically every production network) whenever it has to be implemented again. Typical cases:
- the first VM start after the network GC shut an idle network down (with the default network.gc.* settings, about 10-20 minutes after its last VM stopped);
- the first deployment into a network created more than a day ago.
Network types: the cleanup does not filter by type. Any network that passes through Implementing is exposed; we saw it with isolated networks.
Plain start and deploy API: VMEntityManagerImpl.deployVirtualMachine retries once, which usually passes because the router is up by then. So the problem is partly hidden: the first attempt fails, a WARN is logged, and the start takes longer. The retry can be hit too; we saw that once.
Paths without a retry fail outright:
- Instance boot groups (feature: instance boot group and readiness rules #13907, on main) start VMs through advanceStart once (InstanceBootGroupManagerImpl line 201 on main), so the whole group halts.
- HA restarts (startVirtualMachineForHA) also call advanceStart once. The HA worker may try again later on its own schedule. HA was not tested here.
State consistency: the cleanup changes the state of a network that another thread is implementing under the lock, and falls back to writing Shutdown directly to the database.

versions

Observed on: the main nightly cloudstack-management-24.0.0-SNAPSHOT.20261027, which includes the merged feature: instance boot group and readiness rules #13907 (f03cffc; it does not touch this code).
Code: AsyncJobManagerImpl.cleanupOrphanedNetworks(), added by Allow cleaning up of networks stuck in Implementing state #12409 (215ef98). It is present on the 4.20 and 4.22 branches and on main.
Environment:
- KVM, a single host, EL9, MySQL 8.0.
- Isolated networks on DefaultIsolatedNetworkOfferingWithSourceNatService (not persistent).
- Default network.gc.interval and network.gc.wait (600/600) and job.expire.minutes (1440).

The steps to reproduce the bug

As observed:

Take an isolated network on a non-persistent offering, created more than job.expire.minutes ago (default one day).
Stop all its VMs and wait until the network GC shuts it down (network in Shutdown, router stopped). With the defaults this takes about 10-20 minutes.
Start a VM on it.
- With a boot group (or any path that calls advanceStart once), the start fails with "Network is not yet fully implemented".
- With the plain startVirtualMachine API, the first attempt fails and the API's retry usually passes.
management-server.log shows Found orphaned network <id> in Implementing state without async job a few seconds after the implementation started.

Quicker variant (not yet run in this exact form):

Set job.expire.minutes to 5. It is dynamic, but note that it also shortens job retention on the test system.
Create a new isolated network, which stays Allocated until it is first used.
Wait more than 5 minutes, then deploy a VM into it. The cleanup should cut the first implementation the same way.

Control: a network younger than job.expire.minutes is not affected. A controlled A/B confirmation (a young network against an old one, each case twice) is running, and its results will follow as a comment.

What to do about it?

Possible fixes (any of these would do):

Measure the time stuck in Implementing, not the age since creation. Treat a network as stuck only if it has had no implement activity for X minutes since it entered Implementing, for example from a timestamp set on that transition. Use an X tied to how long an implementation can take, not job.expire.minutes.
Recognise the implementer. Count VmWork jobs (and any other job that implements networks) as active. Or, before the transition, try the network's lock without waiting: skip the network if the lock is held, and otherwise re-read its state under the lock and transition only if it is still Implementing. Avoid the direct database write to Shutdown while another thread may own the network.
Tests:
- Start a VM through advanceStart (no retry) on a network older than the threshold, with the cleanup running. It must succeed, and no orphan may be logged.
- A network that is truly stuck (lock free, idle for X minutes) is still cleaned up.

If a fix cannot make 4.20.4, it may be safer to disable this cleanup on the release branches for now, so that the regression does not ship.

Workarounds until fixed:

Use a persistent network offering. The network GC never collects those networks, so they stay Implemented.
Or make sure networks are Implemented before starts that do not retry, for example by first starting one VM per network with the plain start API.

cc @Pearl1594: thank you for #12409. We are raising this now only because it has not shipped yet and there is still time to adjust it before 4.20.4.

A/B confirmation, as promised in the issue. Same build (main nightly cloudstack-management-24.0.0-SNAPSHOT.20261027), KVM, a single host, isolated networks on DefaultIsolatedNetworkOfferingWithSourceNatService (not persistent). The global settings were unchanged: network.gc.interval / network.gc.wait 600/600 and job.expire.minutes 1440, read back after the test.

Method. Each case starts one VM on one network that needs implementing. The cases ran one at a time, about 30 s apart, with young and old interleaved. Before the test, the network GC had shut every network in A, B and P down (state Allocated, router stopped).

A (control): networks 211 and 212, created 2026-10-29 04:50 UTC (22 and 26 minutes old at the test). Each was implemented once by a first VM start, then the VM was stopped and the GC shut the network down. A one-VM boot group starts the VM (advanceStart, no retry).
B: networks 206 and 207, created 2026-10-27 (about 38.6 h old). The same one-VM boot group start.
P: networks 208 and 209 (old, shut down by the GC). A plain startVirtualMachine (the API path, which retries once).
Q (age by backdating): networks 213 and 214, new and never implemented (Allocated). The age condition comes from backdating networks.created on these two test networks only, while they were Allocated:
UPDATE networks SET created = created - INTERVAL 2 DAY WHERE id IN (213, 214);
- Q1: deployVirtualMachine into network 213.
- Q2: a one-VM boot group on network 214.

Results:

Case	Network	Age at test	Path	Result	Cleanup during the implementation
B1	206	2318 min	boot group	failed after 44 s	orphaned 3.9 s after the implementation started
A1	211	22 min	boot group	passed, 75 s	saw it Implementing 4 times, no orphan
B2	207	2322 min	boot group	failed after 45 s	orphaned 1.9 s after the implementation started
A2	212	26 min	boot group	passed, 76 s	saw it Implementing 4 times, no orphan
P1	208	2326 min	plain start	passed, 55 s; first attempt failed, retry passed	orphaned 5.9 s after the implementation started
P2	209	2327 min	plain start	passed, 53 s; first attempt failed, retry passed	orphaned 9.9 s after the implementation started
Q1	213	2880 min (backdated)	deploy	passed, 51 s; first attempt failed, retry passed	orphaned 9.9 s after the implementation started
Q2	214	2881 min (backdated)	boot group	failed after 43 s	orphaned 7.9 s after the implementation started

In every old or backdated case, 38-41 s passed from the start of the implementation to the router being up, and a 10-second cleanup run landed inside that window.
The young networks took about as long (37-38 s) and were seen by 4 cleanup runs each, which left them alone.
After the failed boot-group starts, the networks stayed in Allocated.

Log lines (management-server.log, 2026-10-29 UTC; times exact, lines shortened, network ids only):

B1  05:11:07.717  lock acquired for network 206 (implement)
    05:11:11.647  Found 1 networks in Implementing state, checking for orphaned networks
    05:11:11.648  Found orphaned network 206 in Implementing state without async job. ... age: 2318 minutes, expiration threshold: 1440 minutes. Transitioning to Shutdown state.
    05:11:46.994  router start completed
    05:11:48.859  ResourceUnavailableException: Resource [Network:206] is unreachable: Network is not yet fully implemented
    05:11:49.266  boot group start halted (tier 0: failed to start a VM)

A1  05:12:43.711  lock acquired for network 211 (implement)
    05:12:51.641 / 05:13:01.648 / 05:13:11.639 / 05:13:21.644  Found 1 networks in Implementing state (no orphan logged)
    05:13:21.052  router start completed
    05:13:25.797  VM start completed

B2  05:14:49.751  lock acquired for network 207 (implement)
    05:14:51.645  Found orphaned network 207 ... age: 2322 minutes ... Transitioning to Shutdown state.
    05:15:30.810  router start completed
    05:15:32.480  Resource [Network:207] is unreachable: Network is not yet fully implemented

A2  05:16:27.717  lock acquired for network 212 (implement)
    05:16:31.641 / 05:16:41.639 / 05:16:51.649 / 05:17:01.642  Found 1 networks in Implementing state (no orphan logged)
    05:17:05.217  router start completed
    05:17:10.593  VM start completed

P1  05:18:35.740  lock acquired for network 208 (implement), first attempt
    05:18:41.655  Found orphaned network 208 ... age: 2326 minutes ... Transitioning to Shutdown state.
    05:19:14.872  router start completed
    05:19:17.107  Resource [Network:208] is unreachable: Network is not yet fully implemented
    05:19:19.751  lock acquired for network 208 (implement), retry; router already running
    05:19:26.278  VM start completed

P2  05:20:21.732  lock acquired for network 209 (implement), first attempt
    05:20:31.647  Found orphaned network 209 ... age: 2327 minutes ...
    05:21:01.127  router start completed
    05:21:02.889  Resource [Network:209] is unreachable: Network is not yet fully implemented
    05:21:03.748  lock acquired for network 209 (implement), retry
    05:21:10.961  VM start completed

Q1  05:21:41.743  lock acquired for network 213 (implement), first attempt
    05:21:51.642  Found orphaned network 213 ... Network created: 2026-10-27T05:21:34 (backdated), age: 2880 minutes ...
    05:22:19.817  router start completed
    05:22:20.923  Resource [Network:213] is unreachable: Network is not yet fully implemented
    05:22:21.756  lock acquired for network 213 (implement), retry
    05:22:28.415  VM start completed

Q2  05:23:23.775  lock acquired for network 214 (implement)
    05:23:31.646  Found orphaned network 214 ... age: 2881 minutes ...
    05:24:01.758  router start completed
    05:24:02.807  Resource [Network:214] is unreachable: Network is not yet fully implemented
    05:24:03.144  boot group start halted (tier 0: failed to start a VM)

Conclusion. The network's age since creation alone decides the outcome.

Every start that implemented a network older than job.expire.minutes had its network moved to Shutdown mid-implementation: 6 of 6, including 2 that were only backdated.
Every start on a young network passed: 2 of 2.
Paths without a retry (boot groups) failed: 3 of 3.
The API path passed only because of its one retry: 3 of 3, each with a failed first attempt.

The retry is not a safe net either. Shortly after the A/B, a plain deployVirtualMachine into old network 209 (shut down by the GC) failed outright. The cleanup caught both attempts:

06:26:09.819  lock acquired for network 209 (implement), first attempt
06:26:13.743  Found orphaned network 209 ... age: 2393 minutes ... Transitioning to Shutdown state.
06:26:47.197  router start completed
06:26:49.501  Resource [Network:209] is unreachable: Network is not yet fully implemented
06:26:51.917  lock acquired for network 209 (implement), retry; router already running
06:26:53.745  Found orphaned network 209 ... age: 2394 minutes ... Transitioning to Shutdown state.
06:26:53.871  Resource [Network:209] is unreachable: Network is not yet fully implemented
06:26:54.337  deploy job FAILED (530); the VM ends in Error, with the alert "Failed to deploy Instance"

The retry's implementation took about 2 s, and a 10-second cleanup run landed inside it. A new deploy about two minutes later passed.

For maintainers reproducing it: backdating a test network's created, as above, reproduces it immediately. Lowering job.expire.minutes also reproduces it, but it deletes completed job history; backdating a test network's created has no side effects.

Updated counts from the soak described in the issue: the issue was written at 7 cleanup runs and 17 failed boot group starts. By the end of the soak (95 cycles, ending 2026-10-29 04:49 UTC), it was 10 cleanup runs and 26 failed boot group starts. Leaving aside readiness-rule timing (a separate boot-group topic), all but one of the soak's failed group starts came from this cleanup.

Navigation Menu

Repository navigation

Unreleased regression (4.20, 4.22, main): the orphaned-network cleanup from #12409 shuts down networks while a VM start is implementing them, so the start fails with "Network is not yet fully implemented" #14522

problem

versions

The steps to reproduce the bug

What to do about it?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unreleased regression (4.20, 4.22, main): the orphaned-network cleanup from #12409 shuts down networks while a VM start is implementing them, so the start fails with "Network is not yet fully implemented" #14522

Description

problem

versions

The steps to reproduce the bug

What to do about it?

Activity

andrijapanicsb commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions