Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Tiltfile
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ local('kubectl wait --namespace cert-manager --for=condition=available deploymen
# Pinned SHAs must match the versions in go.mod.
url = 'https://raw.githubusercontent.com/cobaltcore-dev/openstack-hypervisor-operator/df757ad112ea027f47c198266cb3a323d52ee2fe/charts/openstack-hypervisor-operator/crds/kvm.cloud.sap_hypervisors.yaml'
local('curl -L ' + url + ' | kubectl apply -f -')
url = 'https://raw.githubusercontent.com/cobaltcore-dev/vm-operator/66e5e459efeb01bfc599a1cda5921528ab47be19/dist/files/crds/cobaltcore.cloud_virtualmachines.yaml'
url = 'https://raw.githubusercontent.com/cobaltcore-dev/vm-operator/4876b6fc19aa/dist/files/crds/cobaltcore.cloud_virtualmachines.yaml'
local('curl -L ' + url + ' | kubectl apply -f -')

########### Cortex Manager & CRDs
Expand Down
18 changes: 11 additions & 7 deletions docs/02-external-scheduler-api/02-nova-smart-scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,15 +153,19 @@ resource (the vm-operator CRD, cluster-scoped) so operators can inspect it direc
```bash
kubectl get virtualmachines -o custom-columns=\
NAME:.metadata.name,\
EXPLANATION:.spec.cortex.explanation,\
INTENT:.spec.cortex.intent,\
HOST:.spec.cortex.hostCandidates[0].name
EXPLANATION:.spec.cortex.schedulingExplanation,\
INTENT:.spec.cortex.schedulingIntent,\
HOST:.spec.cortex.schedulingCandidates[0].name
```

`.spec.cortex.explanation` carries a human-readable summary of why the chosen host won,
`.spec.cortex.intent` records the scheduling intent (e.g. initial placement or evacuation), and
`.spec.cortex.hostCandidates` lists the candidate hosts in ranked order; `-o yaml` on a single
VM shows the full detail. The detector pipeline emits **`Descheduling`** resources for VMs it
`.spec.cortex.schedulingExplanation` carries a human-readable summary of why the chosen host won,
`.spec.cortex.schedulingIntent` records the scheduling intent (e.g. initial placement or evacuation), and
`.spec.cortex.schedulingCandidates` lists the candidate hosts in ranked order (`.spec.cortex.schedulingError`
is set instead when scheduling failed); `-o yaml` on a single
VM shows the full detail. Cortex only ever writes this `spec.cortex` subspec (plus the immutable
spec and labels on create); the VM's `status.conditions` are owned exclusively by the vm-operator
and reflect its own observations (Running, Migrating, Resizing, …), not the scheduling decision.
The detector pipeline emits **`Descheduling`** resources for VMs it
wants to move:

```bash
Expand Down
4 changes: 2 additions & 2 deletions docs/06-cortex-library/01-multicluster-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -630,8 +630,8 @@ kubectl --context kind-cortex-remote-az-b get virtualmachines 2>/dev/null || ech
```

You should see the `VirtualMachine` named `cortex-test-instance-001` only in `kind-cortex-remote-az-b`.
Describe it to see the full scheduling result — the target host under `spec.targetHost`, the ranked
`spec.cortex.hostCandidates`, and `spec.cortex.explanation`, the human-readable summary of why that host
Describe it to see the full scheduling result — the ranked `spec.cortex.schedulingCandidates` (the first entry is
the selected host), and `spec.cortex.schedulingExplanation`, the human-readable summary of why that host
won:

```bash
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.27

require (
github.com/cobaltcore-dev/openstack-hypervisor-operator v1.4.0
github.com/cobaltcore-dev/vm-operator v0.2.10
github.com/cobaltcore-dev/vm-operator v0.2.11-0.20261117124215-4876b6fc19aa
github.com/go-gorp/gorp v2.2.0+incompatible
github.com/go-logr/logr v1.4.4
github.com/google/uuid v1.6.0
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ github.com/cobaltcore-dev/openstack-hypervisor-operator v1.4.0 h1:lAb+qBhyfgf77Y
github.com/cobaltcore-dev/openstack-hypervisor-operator v1.4.0/go.mod h1:VIbkpZpnTeLQrEesadTeANk0MB0t+enfIGfJIX/a2vY=
github.com/cobaltcore-dev/vm-operator v0.2.10 h1:W/cNZBDva2pq9kn4QYTcfn4VPxyNEoYqxJBU9hc8Ulg=
github.com/cobaltcore-dev/vm-operator v0.2.10/go.mod h1:0Lanmhb7onfOIHX+4h1lGLC7vga/nNYabJ6a/y/APk4=
github.com/cobaltcore-dev/vm-operator v0.2.11-0.20261117084340-f859ba2eb3bf h1:12SpyudWEoEB8QZ4OGS/R6iYYoY+BwjlDUc8Xf6WFVk=
github.com/cobaltcore-dev/vm-operator v0.2.11-0.20261117084340-f859ba2eb3bf/go.mod h1:0Lanmhb7onfOIHX+4h1lGLC7vga/nNYabJ6a/y/APk4=
github.com/cobaltcore-dev/vm-operator v0.2.11-0.20261117124215-4876b6fc19aa h1:GoxVGt2o0jbvQtlRfEs9ipVm4KKB+k6FehP0hAS5v+U=
github.com/cobaltcore-dev/vm-operator v0.2.11-0.20261117124215-4876b6fc19aa/go.mod h1:0Lanmhb7onfOIHX+4h1lGLC7vga/nNYabJ6a/y/APk4=
github.com/containerd/continuity v0.4.5 h1:ZRoN1sXq9u7V6QoHMcVWGhOwDFqZ4B9i5H6un1Wh0x4=
github.com/containerd/continuity v0.4.5/go.mod h1:/lNJvtJKUQStBzpVQ1+rasXO1LAWtUQssk28EZvJ3nE=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
Expand Down
1 change: 0 additions & 1 deletion helm/bundles/cortex-nova/templates/rbac.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ rules:
- cobaltcore.cloud
resources:
- virtualmachines
- virtualmachines/status
verbs:
- create
- get
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
vmv1alpha1 "github.com/cobaltcore-dev/vm-operator/api/v1alpha1"
"github.com/prometheus/client_golang/prometheus"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/meta"
"k8s.io/apimachinery/pkg/api/resource"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
Expand Down Expand Up @@ -120,7 +121,9 @@ func (k *FailoverPartitionGraphKPI) Collect(ch chan<- prometheus.Metric) {
}
// Count the vms running on each host so the node series can expose it. A vm
// Nova hasn't placed yet has no host and is skipped (it also nil-panics on
// Status.Nova otherwise).
// Status.Nova otherwise). The vm-operator retains the last-known Nova fields
// after a server disappears, so skip vms confirmed deleted or scheduled for
// deletion to avoid counting ghosts against a host.
vmList := &vmv1alpha1.VirtualMachineList{}
if err := k.Client.List(ctx, vmList); err != nil {
failoverPartitionGraphKPILogger.Error(err, "Failed to list virtual machines")
Expand All @@ -131,6 +134,10 @@ func (k *FailoverPartitionGraphKPI) Collect(ch chan<- prometheus.Metric) {
if vm.Status.Nova == nil || vm.Status.Nova.Host == "" {
continue
}
if vm.Status.DeletionTime != nil ||
meta.IsStatusConditionTrue(vm.Status.Conditions, vmv1alpha1.ConditionTypeDeleted) {
continue
}
vmsByHost[vm.Status.Nova.Host]++
}
for _, hv := range hypervisorList.Items {
Expand Down
98 changes: 35 additions & 63 deletions internal/scheduling/nova/vm_handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ import (
vmv1alpha1 "github.com/cobaltcore-dev/vm-operator/api/v1alpha1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
Expand All @@ -36,9 +35,10 @@ type VirtualMachineHandler struct {

// Upsert creates or patches the VirtualMachine resource described by the
// scheduling result and request. Region and AZ are immutable on the CRD and only
// written on create; mutable fields (target host, cortex spec, labels) are
// patched on updates. A human-readable explanation of the result and the intent
// are recorded on the VM's cortex spec.
// written on create; mutable fields (cortex spec, labels) are patched on
// updates. A human-readable explanation of the result and the intent are
// recorded on the VM's cortex spec. Conditions are owned by the vm-operator and
// never written here.
func (h *VirtualMachineHandler) Upsert(ctx context.Context, result compute.PlacementResult, request api.ExternalSchedulerRequest, intent v1alpha1.SchedulingIntent) error {
log := ctrl.LoggerFrom(ctx)

Expand Down Expand Up @@ -75,20 +75,14 @@ func (h *VirtualMachineHandler) Upsert(ctx context.Context, result compute.Place
vm.ObjectMeta = metav1.ObjectMeta{
Name: request.Spec.Data.InstanceUUID,
Labels: map[string]string{
vmv1alpha1.LabelRegion: targetHvRegion,
vmv1alpha1.LabelAZ: targetHvAZ,
vmv1alpha1.LabelTargetHost: targetHv.Name,
vmv1alpha1.LabelRegion: targetHvRegion,
vmv1alpha1.LabelAZ: targetHvAZ,
},
}

// Add hierarchical location information to the virtual machine spec.
vm.Spec.Region = targetHvRegion
vm.Spec.AZ = targetHvAZ
vm.Spec.TargetHost = &vmv1alpha1.HostRef{
APIVersion: hv1.GroupVersion.String(),
Kind: "Hypervisor",
Name: targetHost,
}

// Derive the hypervisor type from the flavor extra spec hypervisor type.
if hvType, ok := request.Spec.Data.Flavor.
Expand Down Expand Up @@ -145,42 +139,18 @@ func (h *VirtualMachineHandler) Upsert(ctx context.Context, result compute.Place

// Write the cortex spec (our scheduling result). The intent was resolved by
// the API layer (with a graceful fallback to CreateIntent for initial
// placements) and passed in.
// placements) and passed in. We only reach this point with a target host, so
// scheduling succeeded and SchedulingError stays empty.
vm.Spec.Cortex = &vmv1alpha1.CortexSpec{}
for _, host := range result.OrderedHosts {
vm.Spec.Cortex.HostCandidates = append(vm.Spec.Cortex.HostCandidates, vmv1alpha1.HostRef{
vm.Spec.Cortex.SchedulingCandidates = append(vm.Spec.Cortex.SchedulingCandidates, vmv1alpha1.HostRef{
APIVersion: hv1.GroupVersion.String(),
Kind: "Hypervisor",
Name: host,
})
}
vm.Spec.Cortex.Explanation = explanation
vm.Spec.Cortex.Intent = string(intent)

// Write the ConditionTypeScheduled condition to indicate that the VM has
// been successfully scheduled. For that we need to extract the scheduling
// intent from request and verify that the operation
var reason string
switch intent {
case api.LiveMigrationIntent:
reason = vmv1alpha1.ConditionReasonScheduledLiveMigration
case api.RebuildIntent:
reason = vmv1alpha1.ConditionReasonScheduledRebuild
case api.ResizeIntent:
reason = vmv1alpha1.ConditionReasonScheduledResize
case api.EvacuateIntent:
reason = vmv1alpha1.ConditionReasonScheduledEvacuation
case api.CreateIntent:
reason = vmv1alpha1.ConditionReasonScheduledInitialPlacement
default:
reason = "Unknown intent: " + string(intent)
}
meta.SetStatusCondition(&vm.Status.Conditions, metav1.Condition{
Type: vmv1alpha1.ConditionTypeScheduled,
Status: metav1.ConditionTrue,
Reason: reason,
Message: "VM has been successfully scheduled by Cortex",
})
vm.Spec.Cortex.SchedulingExplanation = explanation
vm.Spec.Cortex.SchedulingIntent = string(intent)

// If the vm resource doesn't exist yet, create it.
vmInCluster := &vmv1alpha1.VirtualMachine{}
Expand All @@ -189,29 +159,25 @@ func (h *VirtualMachineHandler) Upsert(ctx context.Context, result compute.Place
Name: vm.Name,
}, vmInCluster)
if err != nil && apierrors.IsNotFound(err) {
// Create ignores the status subresource and writes the server response
// (with an empty status) back into vm, so snapshot the conditions and
// restore them before persisting the status.
conditions := vm.Status.Conditions
// First time we see this VM: this intent is its first transition. We
// cannot report a scheduled generation yet (nothing is persisted), so we
// leave ScheduledGeneration unset. Conditions are owned by the vm-operator
// and never written here, so the status subresource is left untouched.
vm.Spec.Cortex.SchedulingTime = new(metav1.Now())
if err := h.Create(ctx, vm); err != nil {
return fmt.Errorf("failed to create VM: %w", err)
}
vm.Status.Conditions = conditions
if err := h.Status().Update(ctx, vm); err != nil {
return fmt.Errorf("failed to update VM status: %w", err)
}
log.Info("created virtual machine resource", "instanceUUID", vm.Name, "targetHost", targetHost, "intent", intent)
return nil
} else if err != nil {
return fmt.Errorf("error talking to apiserver: %w", err)
}

// The vm already exists, so update the existing resource with the new
// object meta, spec, and status. For map data like the labels, we preserve
// object meta and spec. For map data like the labels, we preserve
// existing map entries that are not inside the new object, and only
// store/override the entries that are present in the new object. For status
// conditions, we only update the conditions that are present in the new
// object. In contrast, for the host candidates we completely replace the
// store/override the entries that are present in the new object.
// In contrast, for the scheduling candidates we completely replace the
// existing list with the new list from the object. For the remaining spec
// fields, we follow the same approach as for labels: preserve existing
// entries that are not present in the new object, and only store/override
Expand All @@ -222,28 +188,34 @@ func (h *VirtualMachineHandler) Upsert(ctx context.Context, result compute.Place
maps.Copy(vmInCluster.Labels, vm.Labels)
// Region and AZ are immutable and are only ever written on create, so we
// leave vmInCluster.Spec.Region and vmInCluster.Spec.AZ untouched.
vmInCluster.Spec.TargetHost = vm.Spec.TargetHost
vmInCluster.Spec.HypervisorType = vm.Spec.HypervisorType
if vmInCluster.Spec.Resources == nil {
vmInCluster.Spec.Resources = map[vmv1alpha1.ResourceName]resource.Quantity{}
}
maps.Copy(vmInCluster.Spec.Resources, vm.Spec.Resources)
// Nova does not expose the project's domain; it is populated out of band
// (via keystone) and must not be clobbered by a scheduling write.
domain := vmInCluster.Spec.Nova.Ownership.Domain
vmInCluster.Spec.Nova = vm.Spec.Nova
vmInCluster.Spec.Nova.Ownership.Domain = domain
if vmInCluster.Spec.Cortex == nil {
vmInCluster.Spec.Cortex = &vmv1alpha1.CortexSpec{}
}
vmInCluster.Spec.Cortex.HostCandidates = vm.Spec.Cortex.HostCandidates
vmInCluster.Spec.Cortex.Explanation = vm.Spec.Cortex.Explanation
vmInCluster.Spec.Cortex.Intent = vm.Spec.Cortex.Intent
vmInCluster.Spec.Cortex.SchedulingCandidates = vm.Spec.Cortex.SchedulingCandidates
vmInCluster.Spec.Cortex.SchedulingExplanation = vm.Spec.Cortex.SchedulingExplanation
// This scheduling succeeded, so clear any error left from a previous attempt.
vmInCluster.Spec.Cortex.SchedulingError = ""
// SchedulingTime marks a change of intent; only bump it when the intent
// actually transitioned, otherwise carry the existing timestamp forward.
if vmInCluster.Spec.Cortex.SchedulingIntent != vm.Spec.Cortex.SchedulingIntent {
vmInCluster.Spec.Cortex.SchedulingTime = new(metav1.Now())
}
vmInCluster.Spec.Cortex.SchedulingIntent = vm.Spec.Cortex.SchedulingIntent
// Record the generation we observed on the resource we just read.
vmInCluster.Spec.Cortex.ScheduledGeneration = vmInCluster.Generation
if err := h.Update(ctx, vmInCluster); err != nil {
return fmt.Errorf("failed to update VM: %w", err)
}
for _, cond := range vm.Status.Conditions {
meta.SetStatusCondition(&vmInCluster.Status.Conditions, cond)
}
if err := h.Status().Update(ctx, vmInCluster); err != nil {
return fmt.Errorf("failed to update VM status: %w", err)
}
log.Info("updated virtual machine resource", "instanceUUID", vm.Name, "targetHost", targetHost, "intent", intent)
return nil
}
Loading
Loading