Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
1bab2ce
Renovate: Update kube-prometheus-stack Docker tag to v88.5.3 (#1152)
renovate[bot] Aug 24, 2026
e709fdf
Support ingress in cortex-shim library chart (#1158)
PhilippMatthes Aug 25, 2026
cd19dcb
Renovate: Update debian:trixie-slim Docker digest to d7e1218 (#1159)
renovate[bot] Aug 25, 2026
cf70e17
fix(postgres): rebuild image to resolve CVEs (#1164)
github-actions[bot] Aug 26, 2026
cce4eac
feat: pending cache for configurable list of gvks (#1042)
SoWieMarkus Aug 26, 2026
b32774b
feat: add cache monitoring (#1160)
SoWieMarkus Aug 26, 2026
6226ae5
Renovate: Update github.com/sapcc/go-bits digest to 2ac89aa (#1168)
renovate[bot] Aug 27, 2026
bcdb964
feat: add physical host size label to vmware capacity metrics (#1165)
SoWieMarkus Aug 27, 2026
eb0073f
refactor: scope host details extractor to vmware only (#1166)
SoWieMarkus Aug 27, 2026
f7cc194
fix: update Go version in Dockerfile to 1.27 and remove go constraint…
SoWieMarkus Aug 28, 2026
c181f19
docs: document host oversubscription detection and remediation (#1153)
cortex-ai-agents[bot] Aug 28, 2026
c21930c
Renovate: Update github.com/sapcc/go-bits digest to 7669cbd (#1171)
renovate[bot] Aug 28, 2026
2cb34af
feat: implement forced destination behavior in Nova external schedule…
SoWieMarkus Aug 28, 2026
17e0a4f
Renovate: Update External dependencies to v0.37.0 (#1169)
renovate[bot] Aug 28, 2026
e9b9282
fix(nova): evaluate forced destination before canRunScheduler (#1176)
cortex-ai-agents[bot] Aug 31, 2026
0ed6abc
docs: add pending cache overlay documentation (#1175)
cortex-ai-agents[bot] Aug 31, 2026
23f975d
docs: document forced destination behavior in Nova external scheduler…
cortex-ai-agents[bot] Aug 31, 2026
91df27a
Release cortex 0.4.0 (#1178)
cortex-ai-agents[bot] Aug 31, 2026
c368fba
bump app version [skip ci] (#1157)
github-actions[bot] Aug 31, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions .github/renovate.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
],
"commitMessageAction": "Renovate: Update",
"constraints": {
"go": "1.26",
"python": "3.13"
},
"dependencyDashboardOSVVulnerabilitySummary": "all",
Expand Down Expand Up @@ -73,12 +72,6 @@
}
],
"packageRules": [
{
"matchPackageNames": [
"golang"
],
"allowedVersions": "1.26.x"
},
{
"matchPackageNames": [
"postgresql-18"
Expand Down
67 changes: 67 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,72 @@
# Changelog

## 2026-08-31 — [#1177](https://github.com/cobaltcore-dev/cortex/pull/1177)

### cortex v0.4.0 (sha-23f975db)

New features:
- Pending cache overlay — a transparent, in-process write-through cache on `client.Client` that eliminates informer lag for configurable GVKs; after a successful write the overlay keeps the object in memory so subsequent reads return the fresh version immediately instead of the stale informer snapshot; includes per-object keyed mutex, tombstone tracking, field selector support, and fallback TTL expiry ([#1042](https://github.com/cobaltcore-dev/cortex/pull/1042))
- Cache monitoring — new `cortex_cache_overlay_entries` (current count) and `cortex_cache_overlay_entries_max` (high-watermark) Prometheus gauge metrics per GVK/host ([#1160](https://github.com/cobaltcore-dev/cortex/pull/1160))
- Forced destination behavior in Nova external scheduler — replicates Nova's native `force_hosts`/`force_nodes` behavior; when a scheduling request specifies forced destinations and no `_nova_check_type` hint is set, Cortex skips the entire filter/weigher pipeline and returns only the forced hosts directly; configurable via `forcedDestinationEnabled` ([#1163](https://github.com/cobaltcore-dev/cortex/pull/1163))
- Forced destination evaluated before `canRunScheduler` — critical because Nova may not send weights for forced requests ([#1176](https://github.com/cobaltcore-dev/cortex/pull/1176))
- Physical host size label — adds a `physical_host_size` label derived from memory inventory (e.g. `"4TiB"`, `"512GiB"`) to VMware host capacity metrics ([#1165](https://github.com/cobaltcore-dev/cortex/pull/1165))

Breaking changes:
- Knowledge resource renamed from `host-details` to `vmware-host-details` — existing Knowledge CRDs with the old name will be orphaned and must be deleted manually; extractor renamed from `sap_host_details_extractor` to `vmware_host_details_extractor`; SQL rewritten to filter at the query level, eliminating Go-side hypervisor type/family filtering ([#1166](https://github.com/cobaltcore-dev/cortex/pull/1166))

Non-breaking changes:
- Document forced destination behavior in Nova external scheduler ([#1174](https://github.com/cobaltcore-dev/cortex/pull/1174))
- Document pending cache overlay architecture ([#1175](https://github.com/cobaltcore-dev/cortex/pull/1175))
- Document host oversubscription detection and remediation ([#1153](https://github.com/cobaltcore-dev/cortex/pull/1153))
- Update Go version to 1.27, remove Go version constraint from Renovate ([#1170](https://github.com/cobaltcore-dev/cortex/pull/1170))
- Update `k8s.io/api`, `k8s.io/apimachinery`, `k8s.io/client-go` to v0.37.0 ([#1169](https://github.com/cobaltcore-dev/cortex/pull/1169))
- Update `github.com/sapcc/go-bits` ([#1168](https://github.com/cobaltcore-dev/cortex/pull/1168), [#1171](https://github.com/cobaltcore-dev/cortex/pull/1171))
- Update `kube-prometheus-stack` to v88.6.0 ([#1152](https://github.com/cobaltcore-dev/cortex/pull/1152))

### cortex-shim v0.1.15 (sha-23f975db)

New features:
- Ingress resource template — new `ingress.yaml` template with configurable className, annotations, rules, and TLS support ([#1158](https://github.com/cobaltcore-dev/cortex/pull/1158))

Includes updated dependencies (Go 1.27, k8s.io v0.37.0).

### cortex-postgres v0.6.15 (sha-23f975db)

- Rebuild image to resolve CVEs ([#1164](https://github.com/cobaltcore-dev/cortex/pull/1164))
- Update `debian:trixie-slim` base image digest ([#1159](https://github.com/cobaltcore-dev/cortex/pull/1159))

### cortex-nova v0.0.89

- New alert `CortexNovaCacheOverlayNotDraining` (5m window, warning severity)
- New values: `cache.enabled` (true), `cache.gvks`, `forcedDestinationEnabled` (true)
- Knowledge renamed: `host-details` → `vmware-host-details`

Includes updated charts cortex v0.4.0 and cortex-postgres v0.6.15.

### cortex-cinder v0.0.89

Includes updated charts cortex v0.4.0 and cortex-postgres v0.6.15.

### cortex-manila v0.0.89

Includes updated charts cortex v0.4.0 and cortex-postgres v0.6.15.

### cortex-crds v0.0.89

Includes updated chart cortex v0.4.0.

### cortex-ironcore v0.0.89

Includes updated chart cortex v0.4.0.

### cortex-pods v0.0.89

Includes updated chart cortex v0.4.0.

### cortex-placement-shim v0.1.15

Includes updated chart cortex-shim v0.1.15.

## 2026-08-24 — [#1154](https://github.com/cobaltcore-dev/cortex/pull/1154)

### cortex v0.3.8 (sha-1742e3ee)
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build the manager binary
FROM golang:1.26 AS builder
FROM golang:1.27 AS builder
ARG TARGETOS
ARG TARGETARCH
# Path of our go.mod
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ GOLANGCI_LINT = $(LOCALBIN)/golangci-lint
GOTESTSUM = $(LOCALBIN)/gotestsum

CONTROLLER_TOOLS_VERSION ?= v0.21.0
GOLANGCI_LINT_VERSION ?= v2.13.1
GOLANGCI_LINT_VERSION ?= v2.13.2
GOTESTSUM_VERSION ?= v1.13.0

.PHONY: controller-gen
Expand Down
69 changes: 69 additions & 0 deletions api/external/nova/messages.go
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,75 @@ func (r ExternalSchedulerRequest) Filter(includedHosts map[string]float64) lib.F
return r
}

// IsForcedDestination reports whether Nova forced this request onto specific
// hosts/nodes and the scheduler filters should be skipped entirely.
//
// This replicates Nova's native behavior: when force_hosts or force_nodes is
// set and the scheduler hint "_nova_check_type" is not set, Nova skips the
// filters and returns only the forced destinations.
// See: https://github.com/sapcc/nova/blob/05f384a938e3d6a8740a8f404d79d767d6ebdbd7/nova/scheduler/host_manager.py#L610-L620
func (r ExternalSchedulerRequest) IsForcedDestination() bool {
forceHosts := r.Spec.Data.ForceHosts != nil && len(*r.Spec.Data.ForceHosts) > 0
forceNodes := r.Spec.Data.ForceNodes != nil && len(*r.Spec.Data.ForceNodes) > 0
if !forceHosts && !forceNodes {
return false
}
// If _nova_check_type is set (e.g. rebuild/evacuate/resize), the host is
// forced but must still be validated by the filters, so don't skip.
checkType, err := r.Spec.Data.GetSchedulerHintStr("_nova_check_type")
if err == nil && checkType != "" {
return false
}
return true
}

// ForcedHosts returns the subset of request hosts that match the forced
// destinations (force_hosts / force_nodes). Matching mirrors Nova:
// - force_hosts matches on the compute host name, case-insensitive.
// - force_nodes matches on the hypervisor hostname, case-sensitive.
// - When both are set, a host must match both (intersection).
//
// The returned slice preserves the order of the request hosts. If nothing
// matches, an empty slice is returned (Nova would then raise NoValidHost).
func (r ExternalSchedulerRequest) ForcedHosts() []string {
var forceHosts []string
if r.Spec.Data.ForceHosts != nil {
forceHosts = *r.Spec.Data.ForceHosts
}
var forceNodes []string
if r.Spec.Data.ForceNodes != nil {
forceNodes = *r.Spec.Data.ForceNodes
}
forceHostSet := make(map[string]bool, len(forceHosts))
for _, h := range forceHosts {
forceHostSet[strings.ToLower(h)] = true
}
forceNodeSet := make(map[string]bool, len(forceNodes))
for _, n := range forceNodes {
forceNodeSet[n] = true
}
matched := make([]string, 0, len(r.Hosts))
// Candidates are (host, node) pairs, so a single compute host may appear
// multiple times (once per node). Deduplicate by compute host name, since
// the response only carries host names.
seen := make(map[string]bool, len(r.Hosts))
for _, host := range r.Hosts {
if len(forceHostSet) > 0 && !forceHostSet[strings.ToLower(host.ComputeHost)] {
continue
}
if len(forceNodeSet) > 0 && !forceNodeSet[host.HypervisorHostname] {
continue
}
key := strings.ToLower(host.ComputeHost)
if seen[key] {
continue
}
seen[key] = true
matched = append(matched, host.ComputeHost)
}
return matched
}

type FlavorType string

const (
Expand Down
175 changes: 175 additions & 0 deletions api/external/nova/messages_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -569,3 +569,178 @@ func TestNovaImageMeta_GetHypervisorType(t *testing.T) {
})
}
}

func strPtrSlice(s ...string) *[]string {
out := s
return &out
}

func TestIsForcedDestination(t *testing.T) {
tests := []struct {
name string
forceHosts *[]string
forceNodes *[]string
hints map[string]any
expected bool
}{
{
name: "no force",
expected: false,
},
{
name: "empty force_hosts slice",
forceHosts: strPtrSlice(),
expected: false,
},
{
name: "force_hosts set, no check_type",
forceHosts: strPtrSlice("node017-bb545"),
expected: true,
},
{
name: "force_nodes set, no check_type",
forceNodes: strPtrSlice("domain-c123"),
expected: true,
},
{
name: "force_hosts set with check_type rebuild",
forceHosts: strPtrSlice("node017-bb545"),
hints: map[string]any{"_nova_check_type": "rebuild"},
expected: false,
},
{
name: "force_hosts set with check_type as list",
forceHosts: strPtrSlice("node017-bb545"),
hints: map[string]any{"_nova_check_type": []any{"resize"}},
expected: false,
},
{
name: "force_hosts set with empty check_type",
forceHosts: strPtrSlice("node017-bb545"),
hints: map[string]any{"_nova_check_type": ""},
expected: true,
},
{
name: "force_hosts set, hints without check_type",
forceHosts: strPtrSlice("node017-bb545"),
hints: map[string]any{"some_other_hint": "value"},
expected: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := ExternalSchedulerRequest{
Spec: NovaObject[NovaSpec]{
Data: NovaSpec{
ForceHosts: tt.forceHosts,
ForceNodes: tt.forceNodes,
SchedulerHints: tt.hints,
},
},
}
if got := req.IsForcedDestination(); got != tt.expected {
t.Errorf("expected %v, got %v", tt.expected, got)
}
})
}
}

func TestForcedHosts(t *testing.T) {
hosts := []ExternalSchedulerHost{
{ComputeHost: "node017-bb545", HypervisorHostname: "domain-c17"},
{ComputeHost: "node018-bb545", HypervisorHostname: "domain-c18"},
{ComputeHost: "NODE019-bb545", HypervisorHostname: "domain-c19"},
}
tests := []struct {
name string
hosts []ExternalSchedulerHost
forceHosts *[]string
forceNodes *[]string
expected []string
}{
{
name: "match single host",
forceHosts: strPtrSlice("node017-bb545"),
expected: []string{"node017-bb545"},
},
{
name: "match host case-insensitive",
forceHosts: strPtrSlice("node019-bb545"),
expected: []string{"NODE019-bb545"},
},
{
name: "match node case-sensitive",
forceNodes: strPtrSlice("domain-c18"),
expected: []string{"node018-bb545"},
},
{
name: "node case-sensitive no match",
forceNodes: strPtrSlice("DOMAIN-C18"),
expected: []string{},
},
{
name: "intersection of host and node",
forceHosts: strPtrSlice("node017-bb545", "node018-bb545"),
forceNodes: strPtrSlice("domain-c18"),
expected: []string{"node018-bb545"},
},
{
name: "intersection empty",
forceHosts: strPtrSlice("node017-bb545"),
forceNodes: strPtrSlice("domain-c18"),
expected: []string{},
},
{
name: "no match returns empty",
forceHosts: strPtrSlice("unknown-host"),
expected: []string{},
},
{
name: "multi-node host deduplicated by force_hosts",
hosts: []ExternalSchedulerHost{
{ComputeHost: "hostA", HypervisorHostname: "node1"},
{ComputeHost: "hostA", HypervisorHostname: "node2"},
{ComputeHost: "hostB", HypervisorHostname: "node1"},
},
forceHosts: strPtrSlice("hostA"),
expected: []string{"hostA"},
},
{
name: "multi-node host intersection picks single node",
hosts: []ExternalSchedulerHost{
{ComputeHost: "hostA", HypervisorHostname: "node1"},
{ComputeHost: "hostA", HypervisorHostname: "node2"},
{ComputeHost: "hostB", HypervisorHostname: "node1"},
},
forceHosts: strPtrSlice("hostA"),
forceNodes: strPtrSlice("node1"),
expected: []string{"hostA"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
candidates := tt.hosts
if candidates == nil {
candidates = hosts
}
req := ExternalSchedulerRequest{
Hosts: candidates,
Spec: NovaObject[NovaSpec]{
Data: NovaSpec{
ForceHosts: tt.forceHosts,
ForceNodes: tt.forceNodes,
},
},
}
got := req.ForcedHosts()
if len(got) != len(tt.expected) {
t.Fatalf("expected %v, got %v", tt.expected, got)
}
for i, h := range tt.expected {
if got[i] != h {
t.Errorf("expected host[%d]=%s, got %s", i, h, got[i])
}
}
})
}
}
12 changes: 12 additions & 0 deletions cmd/manager/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ import (
"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/failover"
"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/inflight"
"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/quota"
"github.com/cobaltcore-dev/cortex/pkg/cache"
"github.com/cobaltcore-dev/cortex/pkg/conf"
"github.com/cobaltcore-dev/cortex/pkg/monitoring"
"github.com/cobaltcore-dev/cortex/pkg/multicluster"
Expand Down Expand Up @@ -391,6 +392,14 @@ func main() {
},
}
multiclusterClientConfig := conf.GetConfigOrDie[multicluster.ClientConfig]()

var cacheMonitor *cache.Monitor
var cacheWrapper *cache.Wrapper
if c := conf.GetConfigOrDie[cache.RootConfig](); c.Cache.Enabled {
cacheMonitor = cache.NewMonitor("cortex_")
cacheWrapper = cache.NewWrapper(mgr, c.Cache, cacheMonitor)
multiclusterClient.Wrappers = append(multiclusterClient.Wrappers, cacheWrapper)
}
if err := multiclusterClient.InitFromConf(ctx, mgr, multiclusterClientConfig); err != nil {
setupLog.Error(err, "unable to initialize multicluster client")
os.Exit(1)
Expand All @@ -402,6 +411,9 @@ func main() {
metrics.Registry = monitoring.WrapRegistry(metrics.Registry, metricsConfig)
metrics.Registry.MustRegister(&logMetricsMonitor)
metrics.Registry.MustRegister(multiclusterMonitor)
if cacheMonitor != nil {
metrics.Registry.MustRegister(cacheMonitor)
}

// TODO: Remove me after scheduling pipeline steps don't require DB connections anymore.
metrics.Registry.MustRegister(&db.Monitor)
Expand Down
Loading