OpenStack: Machine does not recover after the volume quota is increased; manual MachineDeployment rollout required

[mihiragrawal]

Current behavior:

Observed on KKP v2.31.0 with machine-controller v1.66.2 on OpenStack.

A worker Machine is created and the provider accepts the request, and the instance then fails because the volume
quota is exhausted, leaving it in an error state. That part is expected and is not what this issue is about.

The quota is then increased, by enough that the same request would now succeed, and nothing changes. The Machine
stays in CreateError and goes on reporting the original quota message, and the MachineDeployment stays below its
desired worker count because no replacement Machine is created. The Machine is still being reconciled throughout,
so this is not a stalled controller; the create is simply never attempted again while the failed instance is still
there, so the increase never reaches it. Recovering the lost capacity needs a manual step: either deleting the
failed Machine or rolling the MachineDeployment.

Two things appear to combine here. Both are worth confirming independently.

1. The failed instance is never retried. ensureInstanceExistsForMachine only reaches the create path when the
provider reports that no instance exists, and the OpenStack provider returns a terminal error for an instance that
is present but in an error state:

https://github.com/kubermatic/machine-controller/blob/v1.66.2/pkg/controller/machine/controller.go#L818
https://github.com/kubermatic/machine-controller/blob/v1.66.2/pkg/cloudprovider/provider/openstack/provider.go#L917-L922

The comment on the branch that handles it states the outcome directly:

https://github.com/kubermatic/machine-controller/blob/v1.66.2/pkg/controller/machine/controller.go#L864

// case 2.2: terminal error was returned and manual interaction is required to recover

If that is deliberate, the rest of this issue is really a question about whether it should stay that way.

2. The failed Machine still counts towards the replica count. syncReplicas compares the number of Machines the
MachineSet owns against its desired replicas, and shouldExcludeMachine excludes a Machine only for a foreign
controller reference, a deletion timestamp, or mismatched labels:

https://github.com/kubermatic/machine-controller/blob/v1.66.2/pkg/controller/machineset/controller.go#L249
https://github.com/kubermatic/machine-controller/blob/v1.66.2/pkg/controller/machineset/controller.go#L337-L355

So the desired count already looks satisfied and no replacement is created, even though the MachineDeployment
reports unavailableReplicas: 1. This part is not specific to OpenStack.

Steps to reproduce

1. Create an OpenStack MachineDeployment whose root volume is larger than the available volume quota.
2. The instance creation is accepted and then fails on quota, and the Machine ends up in CreateError reporting the
   quota message.
3. Increase the volume quota so that the requested volume now fits.
4. Observe that the existing Machine stays in CreateError, still reporting the original message, and that no
   replacement Machine is created. The worker count stays short indefinitely.
5. Delete the Machine, or roll the MachineDeployment, and the capacity comes back.

What we checked

• The Machine's status.errorReason is CreateError and status.errorMessage carries the provider's original
  quota message, repeated unchanged on every reconcile for as long as the failed instance exists.
• Restarting the machine-controller Deployment does not help. After the restart the error still names the same
  provider instance ID, so no new create is attempted. Deleting the Machine produces a different instance ID, which
  is what a retry looks like by comparison. This is the clearest evidence that nothing re-attempts the create, and
  therefore that no change made on the provider side can reach the existing Machine.
• Deleting the failed Machine recovers the capacity, and so does rolling the MachineDeployment. Both only work once
  the underlying cause is actually resolved; rolling while the cause is still present simply produces a second
  failed Machine alongside the first, because the old MachineSet is not scaled down until the new one becomes
  available.
• One thing we did not test directly: we did not increase a volume quota on our own environment, because it is a
  shared one. We made the request satisfiable instead, and the MachineDeployment then recovered through a rollout.

Expected behavior:

After the underlying cause has been resolved, a MachineDeployment should return to its desired worker count without
someone having to notice the failure and intervene by hand. At minimum it would help if a Machine in a terminal
error state did not count as a healthy replica, so that the shortfall is visible to whatever is meant to act on it.

We are not attached to a particular approach, and there may be good reasons for the current behaviour that we are
not seeing. If it is intended, documenting the recovery step would still be worthwhile, since the natural reaction
today is to roll the whole MachineDeployment when deleting the single Machine is enough.