diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index a4c0c71c..25f3ac33 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,44 +1,53 @@ @@ -155,15 +164,46 @@ it is bounded: once; with an unbounded one, a stuck writer would hang every request. Five seconds is far above the milliseconds a write transaction takes once no transaction spans external work (Article II). +**Permitted exception — re-establishing a dependency in the background** (added in v1.5.0). A +background loop that restores the process's own connection to an external dependency, or completes a +one-time preparation that depends on it, does not violate this principle when all of the following +hold: + +- No request waits on the loop. While the dependency is unreachable or the preparation incomplete, a + request that needs it fails at once, as it would without the loop. The request may make at most one + attempt of its own — or wait for one already running — within the dependency's bound. +- Every attempt, the loop's and a request's, is bounded as the dependency's calls are: 1 second for + libvirt. +- The pause between the loop's attempts is capped — 1 to 5 seconds for libvirt — and the loop stops + when the process does. +- The loop carries no queue, no job records and no per-request state, and accepts no work from a + request. + +**Rationale**: the libvirt reconnect loop shipped with `6ed30f7` (2026-09-24) — without it a libvirtd +restart left real mode dead until the pod was restarted by hand — contradicted "Never: retry loops, +exponential backoff", and no spec or plan recorded it. SPEC-031 extends it to the first connection and +to the console socket directory preparation, which real-mode readiness waits for: a retry driven only +by guest starts would deadlock, since no guest start reaches a pod that is not Ready. What the article +protects is unchanged: a request still gets immediate feedback. + ### IV. Multi-Mode Architecture Every service supports stub mode (development) and real mode (production). - **Stub mode**: Returns fake data, no external dependencies, safe on macOS - **Real mode**: Full implementation with libvirt/Ceph/iptables, requires Linux -- Mode detection: Check `svc.libvirtMode == "stub"` or `svc.vmManager == nil` -- Default to stub mode when dependencies unavailable +- **Mode selection**: the configuration chooses the mode; an unset mode defaults to stub +- **Mode detection**: check the configured mode (`svc.libvirtMode == "stub"`, + `svc.networkingMode == "stub"`). A nil `svc.vmManager` means no hypervisor was wired in (stub-mode + tests); it never stands for real mode +- **No runtime fallback**: a configured real mode never falls back to stub. An unreachable dependency + is an outage — the operations that need it fail as they do when it is lost later — never fake data **Requirement**: All features must work in stub mode for development/testing without Linux. +**Rationale** (v1.5.0): read at runtime, "default to stub mode when dependencies unavailable" was how a +real-mode deployment whose libvirt was unreachable at startup reported servers `ACTIVE` with no guest +behind them, while the console path already treated the same state as an outage (SPEC-031). Stub mode +stays the default when no mode is configured, which is what the rule was for. + ### V. Project Isolation All resources scoped by `project_id` from JWT token. - Database queries auto-filter by project_id @@ -231,7 +271,8 @@ undocumented divergence does not merely go unfixed; it silently bounds the value - **No message queues**: RabbitMQ, Kafka, Redis streams - **No async workers**: Celery, background job processors, or any loop an API response waits on (stateless reconciliation loops and the correction of interrupted reservations — at startup and by - a periodic pass — are permitted; see Article II) + a periodic pass — are permitted, see Article II; so is re-establishing a dependency in the + background, see Article III) - **No state machines**: Conductor-style workflows - **No service discovery**: Consul, etcd (services on fixed ports) - **No container orchestration required**: Runs standalone or in Docker Compose @@ -293,6 +334,29 @@ Amendments require: ## Amendment Log +### v1.5.0 — 2026-11-06 — re-establishing a dependency in the background (Article III); no runtime fallback to stub (Article IV) + +**Rationale**: SPEC-031 (cloud-storage#630) found two places where the text no longer described the +product, or would have forbidden its fix: + +- Article III forbids "retry loops, exponential backoff", yet `main` has re-dialed libvirt in the + background with a 1-to-5-second backoff since `6ed30f7` (2026-09-24), and no spec or plan recorded + it. SPEC-031 needs the same loop for the first connection and for the console socket directory + preparation, because real-mode readiness waits for that preparation and no guest start can reach a + pod that is not Ready. +- Article IV's "default to stub mode when dependencies unavailable", read at runtime, is exactly how a + real-mode deployment with an unreachable libvirt reported servers `ACTIVE` with no guest behind + them. SPEC-031 FR-019 removes that path. + +**Scope**: an exception with four limits, as v1.1.0 and v1.3.0 added theirs: no request waits on the +loop, every attempt keeps the dependency's bound, the pause is capped, and the loop takes no work. +Article IV is restated to its original intent — stub when no mode is configured — and its runtime +reading is ruled out. Neither change relaxes what a request experiences. + +**Migration plan**: none for the constitution itself. The reconnect loop has already shipped. SPEC-031 +ships the first-connection and preparation retries and ends the runtime fallback; its +`docs/UPGRADING.md` notes (FR-024) cover what an existing real-mode deployment will notice. + ### v1.4.0 — 2026-11-04 — correcting interrupted reservations: a periodic pass, reclaim on delete (Article II) **Rationale**: SPEC-028's review found two ways in which the reservation exception's fourth limit no @@ -370,4 +434,4 @@ forbidden, and the synchronous-operations rule for API operations is unchanged. **Migration plan**: none required. No behaviour changes. One shipped component stops being a deviation, and SPEC-025's compute state reconciler and action-history reclamation are permitted on the same terms. -**Version**: 1.4.0 | **Ratified**: 2026-03-13 | **Last Amended**: 2026-11-04 +**Version**: 1.5.0 | **Ratified**: 2026-03-13 | **Last Amended**: 2026-11-06 diff --git a/CHANGELOG.md b/CHANGELOG.md index f61084ac..c09626f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Volume-transfer auth keys** are compared in constant time. ### 🐛 Correctness +- **Real mode on modular-libvirt hosts** (cloud-storage#630): the binary dials `virtqemud-sock` where it exists and `libvirt-sock` otherwise, and a non-default `realMode.libvirtSocketPath` is the socket it dials; the console socket directory's preparation is retried until libvirt answers, and real-mode readiness waits for it; with libvirt unreachable a create ends `ERROR` with a fault instead of a fake `ACTIVE`; an unsupported `libvirt_uri` stops startup; host-networked pods resolve in-cluster Service names. See [UPGRADING](docs/UPGRADING.md). +- **A server create must name its boot source**: a create with neither `imageRef` nor `block_device_mapping_v2` is refused with `400` "Missing imageRef attribute", as upstream does, instead of ending `ERROR`. +- **Failed background actions say so**: a shelve libvirt cannot perform ends `ERROR` instead of reading `SHELVED`, and suspend, resume and shelve log the cause. - **Glance checksums describe the stored bytes**: normalizing backends rewrite a qcow2 upload to a raw base, but the recorded checksum hashed the upload stream, so `EnsureLocalBase` failed on every node without the local base and boot-from-image errored. The checksum is now taken from the stored object; unreadable stored data or failed state writes fail the upload instead of returning success. Concurrent uploads atomically claim the queued image, and failed uploads reset it for retry. - **Neutron IP allocation is collision-free**: allocation drew purely at random, so ports in one subnet could land on the same address; allocation and insertion share a subnet lock across port creation, server boot and interface attach. Canonical IPv6 comparisons prevent duplicate spellings, a complete fallback finds the last free address, and tiny subnets (≤ /29) no longer panic `crypto/rand.Int`. - **Metadata `network_data.json` works**: the query selected a nonexistent `fixed_ip` column and answered 500 to every guest; it now resolves each port's JSONB addresses with their own subnets: one link per port, unique network IDs, correct IPv4/IPv6 masks, routes and DNS services, and empty arrays instead of null. diff --git a/CLAUDE.md b/CLAUDE.md index 62db651c..23fcdfc4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -238,7 +238,8 @@ Safe for macOS and non-Linux systems. No external dependencies required. **Nova stub mode**: - Returns fake VM instances, no actual VMs created -- Check: `svc.libvirtMode == "stub"` or `svc.vmManager == nil` +- Check: `svc.libvirtMode == "stub"`. A nil `svc.vmManager` means no hypervisor was wired in (stub-mode + tests) — never real mode - VMs tracked in database only **Neutron stub mode**: @@ -256,6 +257,8 @@ Requires Linux with appropriate dependencies. - Requires libvirt + KVM on Linux - Creates actual VMs via `github.com/digitalocean/go-libvirt` - XML templates in `pkg/hypervisor/xml_template.go` +- Never falls back to stub (Constitution IV, v1.5.0): an unreachable libvirt is an outage, so the operations + that need it fail, never return fake data **Neutron real mode** (`networking_mode: iptables` or `ebpf`): - Requires Linux with network namespaces @@ -396,8 +399,9 @@ if errors.Is(err, database.ErrNoRows) { /* not found — on either backend */ } ### Mode Detection ```go -// Nova: Check libvirtMode or vmManager -if svc.libvirtMode == "stub" || svc.vmManager == nil { +// Nova: check the configured mode. A nil vmManager means no hypervisor was wired in +// (stub-mode tests); real mode never falls back to stub (Constitution IV, v1.5.0). +if svc.libvirtMode == "stub" { // Stub mode: return fake data } @@ -486,8 +490,10 @@ if requestedVersion == "" { - PostgreSQL (unchanged) **or** one SQLite file chosen by the `database.url` scheme (`sqlite:///abs/path`, WAL, `synchronous=FULL`, 0600). One `migrations/` folder read through a per-backend view; SQLite starts at a generated `073_baseline.sqlite`; migration `074_reservation_node` (`volumes`/`snapshots.reserved_by`) on both (028-sqlite-database-layer) - Helm 3.12+ chart (Go `text/template` + Sprig); JSON Schema **draft-07** (`values.schema.json`); Go 1.27.3 for the one `internal/teardown` change; Markdown docs. (029-sqlite-chart-compose) - node-local hostPath SQLite directory at `/db` (`lightstack.db` + `-wal` + `-shm`), **no PVC**. External PostgreSQL unchanged. **No schema change, no migration, no persisted data owned by the chart.** (029-sqlite-chart-compose) -- Go 1.27.3 (module pins `go 1.26`; Constitution requires 1.26+). Markdown docs. (030-sqlite-backup-restore) +- Go 1.27.3 (`go.mod`; Constitution requires 1.26+). Markdown docs. (030-sqlite-backup-restore) - the operator's SQLite database file and a sibling `backups/` directory under the node storage root (`state_path` / `realMode.hostStoragePath`, default `/var/lib/lightstack`). **No schema change, no migration, no persisted data owned by the chart.** PostgreSQL is untouched except the backend-neutral ceiling check. (030-sqlite-backup-restore) +- Go 1.27.3 (`go.mod`; the constitution requires 1.26+). Helm 3.12+ chart templates (Go `text/template` + Sprig), JSON Schema draft-07. Markdown. + `digitalocean/go-libvirt` at the pinned `v0.0.0-20260609165003-6254771e63a8` (`socket.Dialer`, `ConnectToURI`, `Disconnected()`); `golang.org/x/sync/singleflight` (already required, now direct); `gin`; `internal/health`; helm-unittest. **No new module.** (031-realmode-dns-libvirt-fixes) +- nothing new. No database schema change, no migration. Node-local ownership only: `/console-sockets` (as today) and `/instances/` (now ensured before every launch). (031-realmode-dns-libvirt-fixes) ## Recent Changes - 028-sqlite-database-layer: plan — SQLite as a second backend behind a backend-neutral `internal/database` contract whose method sets mirror pgx's, so the 736 call sites change types only. PostgreSQL passes SQL through untouched; SQLite translates casts, `= ANY($n)`, `FOR UPDATE` and the console expiry, registers `now()`/`gen_random_uuid()`/no-op advisory locks, and converts times, JSON, lists and `[]byte` (a STRICT TEXT column refuses BLOBs). Ten rewrite sites, the same for both backends; `name ~` stays a database match (`REGEXP` on SQLite), as upstream's. Cinder creates reserve a `creating` row and build the backing outside any transaction (+ a per-node startup sweep); Nova gains a binding quota recheck under an advisory lock and answers **403** instead of 413 (with FR-036a's 400s, the two API changes). Probe-verified: deferred read-then-write hits `SQLITE_BUSY_SNAPSHOT` (517) → immediate transactions expected (SPEC-028 plan) diff --git a/charts/lightstack/CONTRACT.md b/charts/lightstack/CONTRACT.md index 93876a33..f3449800 100644 --- a/charts/lightstack/CONTRACT.md +++ b/charts/lightstack/CONTRACT.md @@ -91,14 +91,18 @@ kubectl -n lightstack get pods -l app.kubernetes.io/name=lightstack \ A **Ready** release's API is genuinely serving: the pod readiness probe is `GET /readyz` on the identity port — DB-gated, so readiness is `true` only when the database is reachable and all listeners are up (it returns `503` otherwise), and the Pod `Ready` condition is the machine-readable signal to -consume (not a custom status protocol). `--wait` blocks until ready or the timeout; on timeout/failure +consume (not a custom status protocol). In real mode readiness also waits until the console socket +directory has been prepared once, which takes the host's libvirt answering (`503 hypervisor +unavailable` until then); later libvirt outages do not affect it. A host where it never succeeds is +restarted by the startup probe, ending in `CrashLoopBackOff` with the cause at the top of each start's +log, so the release becomes **Failed**. `--wait` blocks until ready or the timeout; on timeout/failure Helm exits non-zero and `.info.status` becomes `failed`. > The health endpoints (`/readyz`, `/livez`, the OpenStack `/healthcheck` alias) are **unauthenticated** > and served on the identity port. If you expose the identity port externally, restrict these paths at > your load balancer / Ingress (per OpenStack guidance). They expose only a coarse status + a short -> reason (e.g. `database unavailable`) — never error detail. There is no Lightstack metrics endpoint; scrape -> the Pod `Ready` condition via `kube-state-metrics`. +> reason (e.g. `database unavailable`, `hypervisor unavailable`) — never error detail. There is no Lightstack +> metrics endpoint; scrape the Pod `Ready` condition via `kube-state-metrics`. ## 3. Endpoints diff --git a/charts/lightstack/README.md b/charts/lightstack/README.md index ea5b7211..bc8588f4 100644 --- a/charts/lightstack/README.md +++ b/charts/lightstack/README.md @@ -24,7 +24,7 @@ Lightstack exposes three **unauthenticated** endpoints on the identity port (353 | Path | Purpose | Behavior | |------|---------|----------| -| `GET /readyz` | readiness | `200` only when the database is reachable **and** all HTTP listeners are bound; `503` + a short reason (e.g. `database unavailable`) otherwise. | +| `GET /readyz` | readiness | `200` only when the database is reachable **and** all HTTP listeners are bound; `503` + a short reason (e.g. `database unavailable`) otherwise. In real mode it also waits until the console socket directory has been prepared once, which takes the host's libvirt answering (`hypervisor unavailable` until then); later libvirt outages do not affect it, and a host where it never succeeds is restarted by the startup probe. | | `GET /livez` | liveness | `200` while the process is up; **never** touches the database — a DB outage must not restart the pod. | | `GET\|HEAD /healthcheck` | OpenStack/oslo alias | alias of `/readyz`; content-negotiates `text/plain` (default) and `application/json`. | @@ -134,8 +134,8 @@ Defaults target the single-node stub MVP. "Mode" = where the value applies. | Key | Default | Mode | Purpose | |-----|---------|------|---------| | `realMode.enabled` | `false` | real | gate privileged pod + host mounts | -| `realMode.privileged` / `hostNetwork` | `false` | real | host-level access | -| `realMode.libvirtSocketPath` | `/var/run/libvirt/libvirt-sock` | real | host libvirtd socket; pods mount its directory (hostPath), so a re-created socket stays reachable | +| `realMode.privileged` / `hostNetwork` | `false` | real | host-level access; host networking also renders `dnsPolicy: ClusterFirstWithHostNet` on the Lightstack pod and the purge Job, so in-cluster Service names (`..svc`) resolve | +| `realMode.libvirtSocketPath` | `/var/run/libvirt/libvirt-sock` | real | host libvirt socket; pods mount its directory (hostPath), so a re-created socket stays reachable. The default lets the binary choose `virtqemud-sock`, else `libvirt-sock`; any other value is also the socket it dials. With `secrets.existingSecret`, your own config's `nova.libvirt_uri` must name the same socket — absolute | | `realMode.hostStoragePath` | `/var/lib/lightstack` | real | node-local VM/volume/image data (hostPath); mounted at the same path and passed to the binary as `state_path` — absolute | | `realMode.legacyStatePath` | `""` | real | pre-rename root to migrate into `hostStoragePath` once; empty renders nothing ([UPGRADING](../../docs/UPGRADING.md)) | | `realMode.floatingIpPool` | `""` | real | operator pool routed to nodes (FR-009c) | @@ -239,6 +239,10 @@ The purge is blocking and fail-closed — if it cannot complete, `helm uninstall - **Irreversible on the node.** `delete` destroys the node's local VM disks / images / volumes / ISOs. The external **database is operator-owned and is NOT touched** (read-only for enumeration) — a pristine redeploy requires the operator to provide a clean database. - **Operator-owned, NOT purged:** networked backends (Ceph RBD / S3). Reclaim those yourself if desired. +- **One attempt to reach libvirt.** In real mode the purge dials libvirt once, bounded at 1 second, and + fails closed when that is missed. A socket-activated daemon that had exited is started by the attempt + itself and may miss the bound: `helm uninstall` then aborts, and the purge Job's log names the socket + it tried. Run `helm uninstall` again — the daemon is up by then. - **Escape hatch:** `helm uninstall lightstack --no-hooks` removes the release *without* purging (e.g. the hosting node is gone). You are then responsible for any orphaned host state. - **Single-node only** today; multi-node DaemonSet host purge is tracked in issue #52. diff --git a/charts/lightstack/templates/NOTES.txt b/charts/lightstack/templates/NOTES.txt index 324b7d74..cb5b0673 100644 --- a/charts/lightstack/templates/NOTES.txt +++ b/charts/lightstack/templates/NOTES.txt @@ -22,6 +22,11 @@ Verify the deployment: # The Pod reports Ready only once GET /readyz passes — database reachable + all # listeners bound (SPEC-013). Gate automation on the native Pod Ready condition: # kubectl wait --for=condition=Ready pod -l app.kubernetes.io/instance={{ .Release.Name }} --timeout=180s + {{- if .Values.realMode.enabled }} + # Real mode: /readyz also waits until the console socket directory has been prepared + # once, which takes the host's libvirt answering. A host where it never does is + # restarted by the startup probe; the cause heads each start's log. + {{- end }} Authenticate (port-forward example): kubectl port-forward svc/{{ include "lightstack.fullname" . }} 35357:35357 diff --git a/charts/lightstack/templates/_helpers.tpl b/charts/lightstack/templates/_helpers.tpl index a8b62201..964eb9a4 100644 --- a/charts/lightstack/templates/_helpers.tpl +++ b/charts/lightstack/templates/_helpers.tpl @@ -250,7 +250,10 @@ compute: heartbeat_interval: 30s nova: port: 8774 - libvirt_uri: "qemu:///system" + {{- /* The default socket leaves the choice to the binary: virtqemud-sock where it exists, + else libvirt-sock. Any other is the socket it dials, as well as the one the pods + mount (SPEC-031 FR-004). Compared here only. */}} + libvirt_uri: {{ if eq .Values.realMode.libvirtSocketPath "/var/run/libvirt/libvirt-sock" }}"qemu:///system"{{ else }}{{ .Values.realMode.libvirtSocketPath | quote }}{{ end }} libvirt_mode: {{ .Values.nova.libvirtMode | quote }} console_proxy_base_url: {{ .Values.nova.consoleProxyBaseUrl | quote }} console_token_ttl: {{ .Values.nova.consoleTokenTtl | quote }} @@ -410,6 +413,11 @@ imagePullSecrets: {{- end }} {{- if .Values.realMode.enabled }} hostNetwork: {{ .Values.realMode.hostNetwork }} +{{- if .Values.realMode.hostNetwork }} +# A host-networked pod resolves through the node unless told otherwise, so in-cluster +# Service names (an operator-managed PostgreSQL, RGW, Redis) would never resolve. +dnsPolicy: ClusterFirstWithHostNet +{{- end }} {{- end }} {{- include "lightstack.scheduling" . }} {{- if .Values.realMode.enabled }} diff --git a/charts/lightstack/templates/teardown-purge-job.yaml b/charts/lightstack/templates/teardown-purge-job.yaml index 823e6434..73b8ee39 100644 --- a/charts/lightstack/templates/teardown-purge-job.yaml +++ b/charts/lightstack/templates/teardown-purge-job.yaml @@ -35,6 +35,9 @@ spec: {{- end }} {{- if and .Values.realMode.enabled .Values.realMode.hostNetwork }} hostNetwork: true + # The purge reads the database, which may be an in-cluster Service: resolve through the + # cluster DNS, as the workload does (see lightstack.podSpec). + dnsPolicy: ClusterFirstWithHostNet {{- end }} containers: - name: purge diff --git a/charts/lightstack/tests/config_render_test.yaml b/charts/lightstack/tests/config_render_test.yaml index 77ef1c38..cf907ec6 100644 --- a/charts/lightstack/tests/config_render_test.yaml +++ b/charts/lightstack/tests/config_render_test.yaml @@ -91,3 +91,32 @@ tests: - matchRegex: path: stringData["lightstack.yaml"] pattern: '(?m)^ console_enabled: false$' + # SPEC-031 FR-004: a non-default libvirt socket is the one the binary dials, not only the one + # the pod mounts. The default leaves the choice to the binary — virtqemud-sock, else + # libvirt-sock — so the default render does not change. + - it: hands a non-default libvirt socket to the binary as libvirt_uri + set: + database.url: postgres://lightstack@db/lightstack + realMode.enabled: true + realMode.libvirtSocketPath: /run/libvirt/virtqemud-sock + asserts: + - matchRegex: + path: stringData["lightstack.yaml"] + pattern: '(?m)^ libvirt_uri: "/run/libvirt/virtqemud-sock"$' + - it: leaves the socket choice to the binary by default + set: + database.url: postgres://lightstack@db/lightstack + realMode.enabled: true + asserts: + - matchRegex: + path: stringData["lightstack.yaml"] + pattern: '(?m)^ libvirt_uri: "qemu:///system"$' + - it: leaves the socket choice to the binary when the default socket is set explicitly + set: + database.url: postgres://lightstack@db/lightstack + realMode.enabled: true + realMode.libvirtSocketPath: /var/run/libvirt/libvirt-sock + asserts: + - matchRegex: + path: stringData["lightstack.yaml"] + pattern: '(?m)^ libvirt_uri: "qemu:///system"$' diff --git a/charts/lightstack/tests/dns_policy_test.yaml b/charts/lightstack/tests/dns_policy_test.yaml new file mode 100644 index 00000000..c93d144e --- /dev/null +++ b/charts/lightstack/tests/dns_policy_test.yaml @@ -0,0 +1,94 @@ +# helm-unittest specs for the pod DNS policy (SPEC-031 FR-001, FR-002, FR-003). +# A host-networked pod treats the default ClusterFirst like Default and resolves through the +# node, so ..svc names never resolve. ClusterFirstWithHostNet sends it to +# the cluster DNS, as the console proxy already does. Without host networking nothing changes. +suite: pod DNS policy +set: + image.repository: ghcr.io/lightstack/lightstack + secrets.jwtSecret: jwt + secrets.adminPassword: pw + database.url: postgres://lightstack@db.lightstack.svc/lightstack +tests: + - it: renders ClusterFirstWithHostNet on the single-node Deployment with host networking + template: templates/deployment.yaml + set: + realMode.enabled: true + realMode.hostNetwork: true + asserts: + - equal: + path: spec.template.spec.hostNetwork + value: true + - equal: + path: spec.template.spec.dnsPolicy + value: ClusterFirstWithHostNet + + - it: renders ClusterFirstWithHostNet on the multi-node DaemonSet with host networking + template: templates/daemonset.yaml + set: + deploymentMode: multi-node + glance.storageMode: rbd # the multi-node real-mode gates refuse node-local storage + cinder.storageMode: rbd + realMode.enabled: true + realMode.hostNetwork: true + asserts: + - equal: + path: spec.template.spec.dnsPolicy + value: ClusterFirstWithHostNet + + - it: renders ClusterFirstWithHostNet on the teardown purge Job with host networking + template: templates/teardown-purge-job.yaml + set: + dataRetentionPolicy: delete + teardown.kubectlImage: rancher/kubectl:v1.31 + realMode.enabled: true + realMode.hostNetwork: true + asserts: + - equal: + path: spec.template.spec.hostNetwork + value: true + - equal: + path: spec.template.spec.dnsPolicy + value: ClusterFirstWithHostNet + + - it: renders no dnsPolicy in stub mode, even with realMode.hostNetwork set + template: templates/deployment.yaml + set: + realMode.hostNetwork: true + asserts: + - notExists: + path: spec.template.spec.dnsPolicy + + - it: renders no dnsPolicy on the Deployment in real mode without host networking + template: templates/deployment.yaml + set: + realMode.enabled: true + realMode.hostNetwork: false + asserts: + - equal: + path: spec.template.spec.hostNetwork + value: false + - notExists: + path: spec.template.spec.dnsPolicy + + - it: renders no dnsPolicy on the DaemonSet in real mode without host networking + template: templates/daemonset.yaml + set: + deploymentMode: multi-node + glance.storageMode: rbd # the multi-node real-mode gates refuse node-local storage + cinder.storageMode: rbd + realMode.enabled: true + realMode.hostNetwork: false + asserts: + - notExists: + path: spec.template.spec.dnsPolicy + + - it: renders no dnsPolicy on the purge Job in real mode without host networking + template: templates/teardown-purge-job.yaml + set: + dataRetentionPolicy: delete + teardown.kubectlImage: rancher/kubectl:v1.31 + realMode.enabled: true + realMode.hostNetwork: false + asserts: + - notExists: + path: spec.template.spec.dnsPolicy diff --git a/charts/lightstack/tests/schema_test.yaml b/charts/lightstack/tests/schema_test.yaml index b0079dd0..e865e6c5 100644 --- a/charts/lightstack/tests/schema_test.yaml +++ b/charts/lightstack/tests/schema_test.yaml @@ -73,6 +73,28 @@ tests: asserts: - failedTemplate: errorPattern: "hostStoragePath|does not match pattern|schema" + # SPEC-031 FR-006: the socket path is mounted by its directory and handed to the binary, + # which accepts only an absolute path; the pattern moves that failure to `helm install`. + - it: rejects a relative libvirtSocketPath (pattern) + set: + image.repository: ghcr.io/lightstack/lightstack + secrets.jwtSecret: x + secrets.adminPassword: x + database.url: postgres://lightstack@db/lightstack + realMode.libvirtSocketPath: libvirt-sock + asserts: + - failedTemplate: + errorPattern: "libvirtSocketPath|does not match pattern|schema" + - it: rejects an empty libvirtSocketPath (pattern) + set: + image.repository: ghcr.io/lightstack/lightstack + secrets.jwtSecret: x + secrets.adminPassword: x + database.url: postgres://lightstack@db/lightstack + realMode.libvirtSocketPath: "" + asserts: + - failedTemplate: + errorPattern: "libvirtSocketPath|does not match pattern|schema" - it: rejects a console proxy image digest that is not a sha256 digest (pattern) # The template renders repository@digest, so a tag here yields a reference the # runtime cannot pull. diff --git a/charts/lightstack/values.schema.json b/charts/lightstack/values.schema.json index 4f4cad52..e3c21668 100644 --- a/charts/lightstack/values.schema.json +++ b/charts/lightstack/values.schema.json @@ -223,8 +223,8 @@ "properties": { "enabled": { "type": "boolean", "description": "Gate privileged pod + host mounts." }, "privileged": { "type": "boolean", "description": "Run the pod privileged." }, - "hostNetwork": { "type": "boolean", "description": "Use host networking." }, - "libvirtSocketPath": { "type": "string", "description": "Host libvirtd socket. Pods mount the directory that holds it (hostPath), so a re-created socket stays reachable." }, + "hostNetwork": { "type": "boolean", "description": "Use host networking. Renders dnsPolicy: ClusterFirstWithHostNet on the Lightstack pod and the purge Job, so in-cluster Service names resolve." }, + "libvirtSocketPath": { "type": "string", "pattern": "^/.+", "description": "Host libvirt socket. Pods mount the directory that holds it (hostPath), so a re-created socket stays reachable. The default lets the binary choose virtqemud-sock, else libvirt-sock, in that directory; any other value is also the socket the binary dials. With secrets.existingSecret the chart renders no config, so the operator's own nova.libvirt_uri must name the same socket. Absolute." }, "hostStoragePath": { "type": "string", "pattern": "^/.+", "description": "Node-local data root (hostPath), mounted at the same path in the pod and passed to the binary as state_path. Absolute: libvirtd on the host opens the paths Lightstack writes into the domain XML." }, "legacyStatePath": { "type": "string", "description": "Pre-rename node-local root to migrate into hostStoragePath once; empty disables." }, "floatingIpPool": { "type": "string", "description": "Operator pool routed to nodes (FR-009c)." } @@ -386,12 +386,12 @@ "probes": { "type": "object", "additionalProperties": false, - "description": "Kubernetes health probes (SPEC-013), unauthenticated on the identity port: /readyz (DB-gated readiness), /livez (process-up liveness), and the OpenStack /healthcheck alias. Paths are fixed; tune timing/thresholds only.", + "description": "Kubernetes health probes (SPEC-013), unauthenticated on the identity port: /readyz (DB-gated readiness), /livez (process-up liveness), and the OpenStack /healthcheck alias. Paths are fixed; tune timing/thresholds only. In real mode /readyz also waits until the console socket directory has been prepared once, which takes the host's libvirt answering.", "properties": { "startup": { "type": "object", "additionalProperties": false, - "description": "startupProbe -> /readyz. failureThreshold x periodSeconds is the slow-DB-connect boot grace before CrashLoopBackOff.", + "description": "startupProbe -> /readyz. failureThreshold x periodSeconds is the boot grace before CrashLoopBackOff: a slow DB connect, and in real mode the host's libvirt answering once. A host where libvirt never answers is restarted.", "properties": { "enabled": { "type": "boolean" }, "periodSeconds": { "type": "integer", "minimum": 1 }, @@ -402,7 +402,7 @@ "readiness": { "type": "object", "additionalProperties": false, - "description": "readinessProbe -> /readyz. Failure removes the pod from the Service (no restart).", + "description": "readinessProbe -> /readyz. Failure removes the pod from the Service (no restart). In real mode it passes only once the console socket directory has been prepared; later libvirt outages do not fail it.", "properties": { "enabled": { "type": "boolean" }, "periodSeconds": { "type": "integer", "minimum": 1 }, diff --git a/charts/lightstack/values.yaml b/charts/lightstack/values.yaml index 5c403035..3c274549 100644 --- a/charts/lightstack/values.yaml +++ b/charts/lightstack/values.yaml @@ -137,9 +137,14 @@ extraEnv: [] realMode: enabled: false privileged: false + # -- Host networking. Renders dnsPolicy: ClusterFirstWithHostNet on the Lightstack pod and + # the purge Job, so in-cluster Service names (database, RGW, Redis) still resolve. hostNetwork: false - # -- Host libvirtd socket. Pods mount the directory that holds it, not the socket - # itself, so a libvirtd that re-creates its socket stays reachable. + # -- Host libvirt socket. Pods mount the directory that holds it, not the socket + # itself, so a daemon that re-creates its socket stays reachable. The default lets the + # binary choose: virtqemud-sock where the modular daemons run, else libvirt-sock. Any + # other value is also the socket the binary dials. With secrets.existingSecret the chart + # renders no config: name the same socket in your own nova.libvirt_uri. Absolute. libvirtSocketPath: /var/run/libvirt/libvirt-sock # -- Node-local data root. Mounted at the same path inside the pod and passed to the # binary as state_path — the two must match, because libvirtd on the host opens the @@ -279,14 +284,16 @@ logging: # port: /readyz (DB-gated readiness), /livez (process-up liveness), and the # OpenStack /healthcheck alias. Paths are fixed — tune timing/thresholds only. # Defaults: ~150s startup grace, ~30s readiness-failure detection, ~60s -# liveness-failure restart. +# liveness-failure restart. In real mode /readyz also waits until the console +# socket directory has been prepared once, which takes the host's libvirt +# answering; a host where it never does is restarted by the startup probe. probes: - startup: # gates on /readyz; tolerates a slow DB connect at boot + startup: # gates on /readyz; tolerates a slow DB connect (real mode: a slow libvirt) at boot enabled: true periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 30 # 30 × 5s ≈ 150s boot grace before CrashLoopBackOff - readiness: # gates Service traffic on /readyz (DB reachable + listeners up) + readiness: # gates Service traffic on /readyz (DB reachable + listeners up; real mode: libvirt answered once) enabled: true periodSeconds: 10 timeoutSeconds: 2 diff --git a/cmd/lightstack/config_test.go b/cmd/lightstack/config_test.go index fc351e16..9731611f 100644 --- a/cmd/lightstack/config_test.go +++ b/cmd/lightstack/config_test.go @@ -1,8 +1,10 @@ package main import ( + "fmt" "os" "path/filepath" + "strings" "testing" "github.com/cobaltcore-dev/lightstack/pkg/paths" @@ -26,3 +28,28 @@ func TestLoadConfigAppliesStatePath(t *testing.T) { t.Errorf("ConsoleTokenDir = %q; want %q", got, want) } } + +// SPEC-031 FR-009: in real mode loadConfig refuses a libvirt_uri Lightstack cannot honour, +// naming it. Stub mode never dials libvirt, so it does not check the value. +func TestLoadConfigLibvirtURI(t *testing.T) { + t.Cleanup(func() { paths.SetBase(paths.DefaultBase) }) + load := func(mode, uri string) error { + path := filepath.Join(t.TempDir(), "lightstack.yaml") + config := fmt.Sprintf("nova:\n libvirt_mode: %s\n libvirt_uri: %q\n", mode, uri) + if err := os.WriteFile(path, []byte(config), 0o600); err != nil { + t.Fatal(err) + } + _, err := loadConfig(path) + return err + } + + if err := load("real", "qemu:///session"); err == nil || !strings.Contains(err.Error(), "qemu:///session") { + t.Errorf("real mode, qemu:///session: %v; want an error naming the value", err) + } + if err := load("stub", "qemu:///session"); err != nil { + t.Errorf("stub mode, qemu:///session: %v; stub mode does not check libvirt_uri", err) + } + if err := load("real", "qemu+unix:///system?socket=/run/x/s"); err != nil { + t.Errorf("real mode, an explicit socket: %v", err) + } +} diff --git a/cmd/lightstack/main.go b/cmd/lightstack/main.go index 7224f5d4..b092d245 100644 --- a/cmd/lightstack/main.go +++ b/cmd/lightstack/main.go @@ -32,6 +32,7 @@ import ( "github.com/cobaltcore-dev/lightstack/internal/placement" "github.com/cobaltcore-dev/lightstack/internal/teardown" "github.com/cobaltcore-dev/lightstack/pkg/cache" + "github.com/cobaltcore-dev/lightstack/pkg/hypervisor" "github.com/cobaltcore-dev/lightstack/pkg/networking" "github.com/cobaltcore-dev/lightstack/pkg/paths" "github.com/cobaltcore-dev/lightstack/pkg/storage" @@ -104,10 +105,6 @@ func main() { log.Printf("Database connection established (pool: max=%d, min=%d)", poolConfig.MaxConns, poolConfig.MinConns) - // Health checker for the Kubernetes probes (SPEC-013): readiness gates on the - // database being reachable + all listeners bound; liveness is dependency-free. - healthChecker := health.New(zlog.Logger, health.DatabaseCheck(pool)) - // Run migrations log.Println("Running database migrations...") if err := database.MigrateUp(cfg.Database.URL, *migrationsPath); err != nil { @@ -151,22 +148,26 @@ func main() { } novaService := nova.NewService(pool, cfg.Nova.LibvirtURI, libvirtMode, cacheInstance) - // Initialize hypervisor + // Initialize the hypervisor. In real mode the connection to libvirt is made in the + // background, and the console socket directory every domain names (SPEC-021 FR-017a) is + // prepared after every connection, so startup never waits on libvirt (SPEC-031). The only + // error left is a real-mode libvirt_uri Lightstack cannot honour, which configuration load + // refuses first; it is fatal anyway, because real mode never runs without a hypervisor + // (Constitution IV). Stub mode cannot fail here. if err := novaService.InitHypervisor(); err != nil { - log.Printf("WARNING: Failed to initialize hypervisor: %v", err) - log.Printf("Nova will run in %s mode", libvirtMode) + log.Fatalf("Failed to initialize hypervisor: %v", err) + } + if libvirtMode == "real" { + log.Printf("Hypervisor started in real mode; connecting to libvirt in the background") } else { log.Printf("Hypervisor initialized successfully in %s mode", libvirtMode) - if libvirtMode == "real" { - // Before any guest can start: every domain names its console socket in this - // directory, which libvirt does not prepare (SPEC-021 FR-017a). On failure the - // hypervisor refuses guest starts and says why; every other service keeps running. - if err := novaService.PrepareConsoleSocketDir(ctx, paths.ConsoleSocketDir()); err != nil { - log.Printf("ERROR: console socket directory not ready, guest starts are refused until it is: %v", err) - } - } } + // Health checker for the Kubernetes probes (SPEC-013): readiness gates on the database + // being reachable and all listeners bound, and in real mode on the hypervisor being able + // to start guests (SPEC-031 FR-017); liveness is dependency-free. + healthChecker := health.New(zlog.Logger, readinessChecks(pool, novaService, libvirtMode)...) + // Set default networking mode if not specified networkingMode := cfg.Neutron.NetworkingMode if networkingMode == "" { @@ -494,6 +495,14 @@ func loadConfig(path string) (*common.Config, error) { if err != nil { return nil, err } + // Real mode dials libvirt, so a libvirt_uri it cannot honour stops every entry point here, + // before anything is opened, naming the value and the accepted forms (SPEC-031 FR-009). + // Stub mode never dials libvirt and does not check it. + if cfg.Nova.LibvirtMode == "real" { + if _, err := hypervisor.ParseLibvirtURI(cfg.Nova.LibvirtURI); err != nil { + return nil, err + } + } paths.SetBase(cfg.StatePath) return cfg, nil } @@ -619,6 +628,17 @@ func checkDatabaseConfig(cfg *common.Config, server bool) error { return nil } +// readinessChecks are what readiness gates on besides the listeners: the database, and in real +// mode the hypervisor, until it has prepared the console socket directory once (SPEC-031 +// FR-017). Stub mode's readiness is unchanged. +func readinessChecks(pool database.DB, novaSvc *nova.Service, libvirtMode string) []health.Check { + checks := []health.Check{health.DatabaseCheck(pool)} + if libvirtMode == "real" { + checks = append(checks, health.Check{Name: "hypervisor", Probe: novaSvc.HypervisorReady}) + } + return checks +} + // logDatabaseReady writes the line that says which database the server runs on // (FR-013a): the backend, where it is — never with credentials — the schema version, the // node name and, on SQLite, the connection settings as read back from a pooled connection. diff --git a/cmd/lightstack/readiness_checks_test.go b/cmd/lightstack/readiness_checks_test.go new file mode 100644 index 00000000..72cc925e --- /dev/null +++ b/cmd/lightstack/readiness_checks_test.go @@ -0,0 +1,56 @@ +package main + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "slices" + "strings" + "testing" + + "github.com/gin-gonic/gin" + "github.com/rs/zerolog" + + "github.com/cobaltcore-dev/lightstack/internal/health" + "github.com/cobaltcore-dev/lightstack/internal/nova" +) + +// SPEC-031 FR-017, FR-018a: in real mode readiness also waits for the hypervisor — the +// console socket directory prepared once — and stub mode's readiness is unchanged. +func TestReadinessChecks(t *testing.T) { + names := func(checks []health.Check) []string { + var out []string + for _, c := range checks { + out = append(out, c.Name) + } + return out + } + svc := nova.NewService(nil, "", "real", nil) + if got := names(readinessChecks(nil, svc, "real")); !slices.Equal(got, []string{"database", "hypervisor"}) { + t.Errorf("real mode checks %v; want [database hypervisor]", got) + } + if got := names(readinessChecks(nil, svc, "stub")); !slices.Equal(got, []string{"database"}) { + t.Errorf("stub mode checks %v; want [database]", got) + } + + readyz := func(checks ...health.Check) (int, string) { + gin.SetMode(gin.TestMode) + c := health.New(zerolog.Nop(), checks...) + c.MarkStarted() + r := gin.New() + c.Register(r) + w := httptest.NewRecorder() + r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/readyz", nil)) + return w.Code, strings.TrimSpace(w.Body.String()) + } + // The check as main wires it, so the probe behind the name is the one that runs here. + hypervisorCheck := readinessChecks(nil, svc, "real")[1] + if code, body := readyz(hypervisorCheck); code != http.StatusServiceUnavailable || body != "hypervisor unavailable" { + t.Errorf("/readyz = %d %q; want 503 \"hypervisor unavailable\"", code, body) + } + dbDown := health.Check{Name: "database", Probe: func(context.Context) error { return errors.New("down") }} + if code, body := readyz(dbDown, hypervisorCheck); code != http.StatusServiceUnavailable || body != "database,hypervisor unavailable" { + t.Errorf("/readyz = %d %q; want 503 \"database,hypervisor unavailable\"", code, body) + } +} diff --git a/cmd/lightstack/startup_refusal_test.go b/cmd/lightstack/startup_refusal_test.go index 08a378a3..0eab91a2 100644 --- a/cmd/lightstack/startup_refusal_test.go +++ b/cmd/lightstack/startup_refusal_test.go @@ -215,3 +215,35 @@ func TestMissingSQLiteFile(t *testing.T) { }) } } + +// TestUnsupportedLibvirtURIRefused pins SPEC-031 FR-009: in real mode, a libvirt_uri Lightstack +// cannot honour stops every entry point that loads the configuration, naming the value and the +// forms it accepts, before the database is opened. lightstack-migrate never reads libvirt. +func TestUnsupportedLibvirtURIRefused(t *testing.T) { + const uri = "qemu+tcp://h/system" + for _, ep := range entryPoints { + if ep.name == "lightstack-migrate" { + continue + } + t.Run(ep.name, func(t *testing.T) { + watch := filepath.Join(t.TempDir(), "watch") + // Real mode allows a state_path of 50 characters at most; t.TempDir's are longer. + state, err := os.MkdirTemp("", "ls") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.RemoveAll(state) }) + extra := fmt.Sprintf("state_path: %q\nnova:\n libvirt_mode: real\n libvirt_uri: %q\n", state, uri) + out := run(t, ep, "sqlite://"+filepath.Join(watch, "db", "x.db"), extra, nil) + if out.code != ep.exit { + t.Errorf("exit code = %d; want %d. stderr: %s", out.code, ep.exit, out.stderr) + } + if !strings.Contains(out.stderr, uri) || !strings.Contains(out.stderr, "accepted") { + t.Errorf("stderr does not name %s and the accepted forms:\n%s", uri, out.stderr) + } + if _, err := os.Stat(watch); !os.IsNotExist(err) { + t.Errorf("the refusal created %s (stat: %v); it must come at configuration load, before the database is opened", watch, err) + } + }) + } +} diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 73bc9b83..5462da22 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -433,11 +433,18 @@ nova: libvirt_mode: stub # "stub" or "real" ``` -**libvirt_uri options:** -- `qemu:///system` - System-wide QEMU/KVM (requires root) -- `qemu:///session` - User session QEMU/KVM -- `qemu+tcp://host/system` - Remote libvirt over TCP -- `qemu+ssh://user@host/system` - Remote libvirt over SSH +**libvirt_uri** names the local libvirt socket Lightstack dials. Read in real mode only: +- `qemu:///system` (the default; empty and `qemu+unix:///system` mean the same) - Lightstack chooses + the socket on every connection attempt: `/var/run/libvirt/virtqemud-sock` where the modular + daemons run, else `/var/run/libvirt/libvirt-sock` (the monolithic `libvirtd`, or `virtproxyd`). +- `qemu:///system?socket=/path/to/sock` or `qemu+unix:///system?socket=/path/to/sock` - that + socket, exactly. +- `/path/to/sock` - an absolute socket path: that socket, exactly. + +Anything else - a session URI, another driver, a remote host or transport (`qemu+tcp://`, +`qemu+ssh://`, ...), a relative path, any other parameter - stops real-mode startup, and the +`cleanup` and `console-tokens` commands, with an error naming the value and these forms. +Lightstack only ever dials the libvirt on its own node. **libvirt_mode:** - `stub` - Mock libvirt (for testing, no actual VMs) diff --git a/docs/HORIZON_INTEGRATION.md b/docs/HORIZON_INTEGRATION.md index 9176a619..ef23a567 100644 --- a/docs/HORIZON_INTEGRATION.md +++ b/docs/HORIZON_INTEGRATION.md @@ -361,7 +361,8 @@ sudo -u postgres psql lightstack -c "UPDATE endpoints SET url = REPLACE(url, 'ht **Solutions**: 1. Verify real mode: `grep libvirt_mode /opt/lightstack/config/lightstack.yaml` 2. Check privileged mode: `docker inspect lightstack | jq '.[].HostConfig.Privileged'` -3. Fix libvirt socket permissions: `sudo chmod 666 /var/run/libvirt/libvirt-sock` +3. Check that a libvirt socket is mounted: `docker exec lightstack ls -l /var/run/libvirt/` lists + `virtqemud-sock` with the modular daemons or `libvirt-sock` with `libvirtd` 4. Verify namespaces: `docker exec lightstack ip netns list` **Problem**: VMs have no network connectivity diff --git a/docs/HOST_TESTING_OPENSUSE.md b/docs/HOST_TESTING_OPENSUSE.md index 8649cdea..9904eac2 100644 --- a/docs/HOST_TESTING_OPENSUSE.md +++ b/docs/HOST_TESTING_OPENSUSE.md @@ -37,11 +37,9 @@ Why each line: - **SELinux permissive** — enforcing blocks k3s from exec'ing its binary (`status=203/EXEC`, no k3s-selinux policy on openSUSE) and svirt blocks QEMU on the `var_lib_t` state; the `sed` persists it, `setenforce 0` applies it now. -- **modular libvirt sockets** must be up (openSUSE has no `libvirt-sock`). -- **`/run/libvirt/libvirt-sock` symlink** — `/run` is tmpfs, so it vanishes on reboot; the Helm chart - hard-codes that path (Part C). +- **modular libvirt sockets** must be up (openSUSE has no `libvirt-sock`; Lightstack dials + `virtqemud-sock` itself, so no symlink is needed). - **default libvirt network** must be active for minikube's kvm2 driver (Part C). -- **warm libvirt** so the 1 s console-dir capability read at real-mode startup doesn't time out. **Host (sudo):** ``` @@ -49,16 +47,13 @@ sudo systemctl disable --now firewalld sudo sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config sudo setenforce 0 sudo systemctl enable --now virtqemud.socket virtnetworkd.socket virtstoraged.socket -sudo ln -sf virtqemud-sock /run/libvirt/libvirt-sock sudo virsh -c qemu:///system net-start default 2>/dev/null || true -sudo virsh -c qemu:///system capabilities >/dev/null && echo "libvirt warm" ``` -Verify (want `firewalld=inactive`, `selinux_enforce=0`, the symlink resolving to `virtqemud-sock`, and -`default` active): +Verify (want `firewalld=inactive`, `selinux_enforce=0`, `virtqemud-sock` listed, and `default` active): ``` echo "firewalld=$(systemctl is-active firewalld) selinux_enforce=$(cat /sys/fs/selinux/enforce)" -ls -l /run/libvirt/libvirt-sock +ls -l /run/libvirt/virtqemud-sock sudo virsh -c qemu:///system net-list ``` @@ -179,28 +174,15 @@ host's modular libvirt. Part A (stub/CI) is **not** affected. (the running QEMU's current i440fx machine), so real-mode `server create` defines on QEMU 11 with no workaround. Nothing to do; this note stays so an older checkout's define failure is recognisable. -**Blocker — libvirt socket path.** The binary dials `/var/run/libvirt/libvirt-sock` whenever -`nova.libvirt_uri` is `qemu:///system`. Modular libvirt on this host exposes only -`/run/libvirt/virtqemud-sock`. -- **Part B (compose-kvm)** avoids it cleanly: the config file this runbook writes sets - `nova.libvirt_uri: "/run/libvirt/virtqemud-sock"`, which the binary uses verbatim. No symlink. -- **Part C (Helm chart)** can't set `libvirt_uri` (the chart hard-codes `qemu:///system` in - `_helpers.tpl`), so create a compatibility symlink on the host before installing: +**Fixed in this tree — libvirt socket path (SPEC-031).** With `nova.libvirt_uri: qemu:///system` the +binary dials `virtqemud-sock` where it exists and `libvirt-sock` otherwise, so modular libvirt needs no +symlink and no `virtproxyd`. Through the chart, a non-default `realMode.libvirtSocketPath` is also the +socket the binary dials. A `libvirt-sock` symlink left from an earlier run is harmless and can go. - ```bash - sudo ln -sf virtqemud-sock /run/libvirt/libvirt-sock - ls -l /run/libvirt/libvirt-sock - ``` - - `/run` is tmpfs — re-create the symlink after a reboot. - -**Watch-point — cold libvirt (SPEC-021).** Real-mode startup reads libvirt capabilities once with a -1 s deadline; a freshly socket-activated `virtqemud` can answer slower, after which **every** guest -start is refused. Warm it before starting Lightstack, and grep the log if starts are refused: - -```bash -sudo virsh -c qemu:///system capabilities >/dev/null && echo "libvirt warm" -``` +**Fixed in this tree — cold libvirt (SPEC-031).** The console socket directory's preparation is +retried until libvirt answers, and in real mode `/readyz` waits for it, so no warm-up is needed. A +freshly socket-activated `virtqemud` that misses the first 1 s attempt logs one WARN or ERROR line, then +`console socket directory ready; guests can start` once it answers. **Watch-point — SELinux (svirt).** SELinux is enforcing here. If a guest fails to start or the console socket can't be created, check for denials: @@ -226,9 +208,8 @@ for real-mode guests, both validated 2026-11-02: `svirt_image_t` or set `security_driver="none"` in the host qemu.conf). `setenforce 1` when done. (The per-instance-dir **ownership** gap is a separate DAC layer, fixed in code — see §0.4's note.) -The machine-type fix landed in this tree (`machine='pc'`). The remaining libvirt-socket item is left -as the symlink workaround above; I can turn it into a proper commit (a chart-settable `libvirt_uri` -plus a correct default on modular libvirt) if you want it fixed rather than worked around. +The machine-type, libvirt-socket and cold-libvirt fixes have all landed in this tree: beyond the +firewalld and SELinux relaxations above, real mode needs no host workaround. --- @@ -383,8 +364,9 @@ Boots a **real libvirt/KVM VM** under `deployments/docker-compose-kvm.yml` and c create, SSH-key injection, SSH login, console login, internet egress. SQLite backend (one file, no Postgres container). -The machine-type fix is already in this tree (§0.4), so the VM defines on QEMU 11 unchanged. The -libvirt-socket item is handled by the config's `libvirt_uri` override below — no symlink needed here. +The machine-type and libvirt-socket fixes are in this tree (§0.4), so the VM defines on QEMU 11 and the +binary finds `virtqemud-sock` itself. The config below still names the socket; `qemu:///system` would +choose the same one. > **Run the KVM stack with `sudo` (rootful podman).** Real mode needs real root: neutron `iptables` > mode creates bridges/TAPs in the **host** network namespace and Lightstack talks to the host libvirt @@ -397,7 +379,6 @@ libvirt-socket item is handled by the config's `libvirt_uri` override below — ### B1 — Host prep and config ```bash -sudo virsh -c qemu:///system capabilities >/dev/null && echo "libvirt warm" sudo mkdir -p /var/lib/lightstack/images /var/lib/lightstack/volumes /var/lib/lightstack/instances /var/lib/lightstack/cloud-init /var/lib/lightstack/dhcp /var/lib/lightstack/db /var/lib/lightstack/console-sockets sudo mkdir -p /opt/lightstack/config ``` @@ -458,12 +439,13 @@ sudo podman rm -f lightstack 2>/dev/null || true sudo docker compose -f deployments/docker-compose-kvm.yml up -d --build timeout 120 bash -c 'until curl -sf http://localhost:35357/v3 >/dev/null; do sleep 3; done' sudo docker compose -f deployments/docker-compose-kvm.yml logs --tail=40 lightstack -sudo docker compose -f deployments/docker-compose-kvm.yml logs lightstack 2>/dev/null | grep -i "console socket directory not ready" && echo "WARN cold-libvirt: restart lightstack after warming" || echo "console dir OK" +sudo docker compose -f deployments/docker-compose-kvm.yml logs lightstack 2>/dev/null | grep -i "console socket directory ready" || echo "WAIT: libvirt has not answered yet; the preparation is retried, check again" ``` Desired outcome: Keystone answers; the log shows migrations applied, real hypervisor + iptables neutron -+ local cinder/glance, and **no** "console socket directory not ready". If that warning appears, warm -libvirt (§0.4) and `sudo docker compose -f deployments/docker-compose-kvm.yml restart lightstack`. ++ local cinder/glance, and `console socket directory ready; guests can start`. A cold `virtqemud` may log +`console socket directory not ready` first: the preparation is retried, so check again — no restart and +no warm-up. ### B3 — Authenticate and seed image + keypair + network @@ -593,10 +575,10 @@ Three shells, each pinned to a context — each block is labelled with the shell - **T2 — OpenStack CLI**: drives Lightstack over the node IP. No kubeconfig. - **T3 — Rook/minikube**: the default kubeconfig (`~/.kube/config`, context `rook`). -The machine-type fix is already in this tree (§0.4), so the chart image boots a guest unchanged. -Create the `libvirt-sock` symlink first (the chart can't override `libvirt_uri`) — C0 does this. +The machine-type fix is already in this tree (§0.4), so the chart image boots a guest unchanged. The +chart's default `realMode.libvirtSocketPath` lets the binary choose `virtqemud-sock`: no symlink. -### C0 — Fresh k3s, blockers, kubeconfig (T1) +### C0 — Fresh k3s, kubeconfig (T1) Install a **clean k3s each run**: uninstall any prior k3s first (its uninstaller is idempotent and only present when k3s is installed), then install. A fresh install wipes k3s's containerd image @@ -609,13 +591,10 @@ curl -sfL https://get.k3s.io | sudo INSTALL_K3S_SELINUX_WARN=true INSTALL_K3S_SK sudo systemctl is-active k3s ``` -Then the real-mode blockers (libvirt socket + warm) and a user-readable kubeconfig pinned to T1: +Then a user-readable kubeconfig pinned to T1: **T1 — host:** ```bash -sudo ln -sf virtqemud-sock /run/libvirt/libvirt-sock -ls -l /run/libvirt/libvirt-sock -sudo virsh -c qemu:///system capabilities >/dev/null && echo "libvirt warm" mkdir -p ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/k3s.yaml sudo chown "$(id -u):$(id -g)" ~/.kube/k3s.yaml @@ -624,8 +603,7 @@ export KUBECONFIG=~/.kube/k3s.yaml kubectl get nodes ``` -Desired outcome: k3s is `active`, the symlink resolves to `virtqemud-sock`, libvirt is warm, and the -k3s node is `Ready`. Keep `KUBECONFIG` exported in T1 for the rest of Part C. +Desired outcome: k3s is `active` and the k3s node is `Ready`. Keep `KUBECONFIG` exported in T1 for the rest of Part C. ### C1 — Rook/Ceph + RGW on minikube (T3) @@ -758,13 +736,13 @@ Confirm every service is **real** and the RGW endpoint is reachable from the pod **T1 — host:** ```bash kubectl -n lightstack get secret lightstack-lightstack-config -o jsonpath='{.data.lightstack\.yaml}' | base64 -d | grep -E "_mode:|libvirt_mode:" -kubectl -n lightstack logs deploy/lightstack-lightstack | grep -i "console socket directory not ready" && echo "WARN cold-libvirt: kubectl -n lightstack rollout restart deploy/lightstack-lightstack after warming" || echo "console dir OK" +kubectl -n lightstack logs deploy/lightstack-lightstack | grep -i "console socket directory ready" || echo "WAIT: libvirt has not answered yet; the pod turns Ready once it has" kubectl -n lightstack exec deploy/lightstack-lightstack -- sh -c "wget -qO- $RGW_URL 2>&1 | head -c 120; echo" ``` Desired outcome: `libvirt_mode: "real"`, `networking_mode: "iptables"`, -`security_group_mode: "iptables"`, `storage_mode: "s3"` (glance) and `"local"` (cinder); no -cold-libvirt warning; the `wget` prints S3 XML (`` or +`security_group_mode: "iptables"`, `storage_mode: "s3"` (glance) and `"local"` (cinder); the +`console socket directory ready` line; the `wget` prints S3 XML (`` or `AccessDenied`). If the pod can't reach RGW, add the cross-cluster rules: **T1 — host (only if the wget failed):** @@ -1010,10 +988,9 @@ sudo /usr/local/bin/k3s-uninstall.sh deployments/rook/setup-rook-rgw.sh --delete ``` -**Remove the compatibility symlink, and (for a truly fresh run) the node-local state** that lives -outside k3s and survives its uninstall: +**For a truly fresh run, remove the node-local state** that lives outside k3s and survives its +uninstall: ```bash -sudo rm -f /run/libvirt/libvirt-sock sudo rm -rf /var/lib/lightstack ``` diff --git a/docs/REAL_LIBVIRT_MODE.md b/docs/REAL_LIBVIRT_MODE.md index 98cbe8c7..39b61482 100644 --- a/docs/REAL_LIBVIRT_MODE.md +++ b/docs/REAL_LIBVIRT_MODE.md @@ -98,8 +98,9 @@ virsh version # List default network virsh net-list --all -# Check libvirt socket -ls -la /var/run/libvirt/libvirt-sock +# Check libvirt socket: Lightstack dials virtqemud-sock where the modular daemons run, +# else libvirt-sock (libvirtd); one of them must exist +ls -la /var/run/libvirt/virtqemud-sock /var/run/libvirt/libvirt-sock # Test connection virsh --connect qemu:///system list --all @@ -573,8 +574,8 @@ sudo systemctl status libvirtd # Start if not running sudo systemctl start libvirtd -# Check socket permissions -ls -la /var/run/libvirt/libvirt-sock +# Check socket permissions (virtqemud-sock with the modular daemons, libvirt-sock with libvirtd) +ls -la /var/run/libvirt/virtqemud-sock /var/run/libvirt/libvirt-sock # Add user to libvirt group sudo usermod -aG libvirt $USER @@ -631,8 +632,8 @@ ls -la /dev/kvm # Add user to kvm group sudo usermod -aG kvm $USER -# Set permissions -sudo chmod 666 /dev/kvm +# Log out and back in (or run `newgrp kvm`) for the group to take effect. +# Do not make /dev/kvm world-writable: every local user would get KVM. ``` --- diff --git a/docs/REAL_MODE_TESTING.md b/docs/REAL_MODE_TESTING.md index 41a853ef..d44302c1 100644 --- a/docs/REAL_MODE_TESTING.md +++ b/docs/REAL_MODE_TESTING.md @@ -25,16 +25,16 @@ nova: ### Result: ``` -WARNING: Failed to initialize hypervisor: - failed to connect to libvirt: - failed to dial libvirt socket: - dial unix /var/run/libvirt/libvirt-sock: connect: no such file or directory +Hypervisor started in real mode; connecting to libvirt in the background +{"level":"warn","error":"no libvirt socket: neither /var/run/libvirt/virtqemud-sock nor /var/run/libvirt/libvirt-sock exists","next_attempt_in":1000,"message":"libvirt connection attempt failed; retrying in the background"} ``` ### Behavior: -✅ **Graceful degradation** - Lightstack started successfully -✅ **Services available** - All 5 services running -⚠️ **VM operations disabled** - Nova will return "hypervisor not available" +✅ **Startup does not wait on libvirt** - all services start, and Lightstack keeps retrying in +the background (1 s attempts, 1-5 s apart), connecting once a socket appears +⚠️ **Not Ready** - `/readyz` answers `503 hypervisor unavailable` until libvirt has answered once +⚠️ **No guest is faked** - a server create ends `ERROR` with the fault "the hypervisor is not +ready to start guests"; a server action answers `500` ## Real Mode Requirements @@ -53,8 +53,9 @@ WARNING: Failed to initialize hypervisor: 3. **libvirt socket** available ```bash - ls -la /var/run/libvirt/libvirt-sock - # Should exist and be accessible + ls -la /var/run/libvirt/virtqemud-sock /var/run/libvirt/libvirt-sock + # One should exist and be accessible: virtqemud-sock with the modular daemons, + # libvirt-sock with libvirtd ``` 4. **User permissions** @@ -159,13 +160,14 @@ virsh list --all # Should be gone ``` 1. lightstack starts with libvirt_mode: real ↓ -2. VMManager.connectLibvirt() called +2. hypervisor.StartVMManager() returns at once; a background loop connects ↓ -3. Dial Unix socket: /var/run/libvirt/libvirt-sock +3. Each attempt dials /var/run/libvirt/virtqemud-sock if it exists, else + /var/run/libvirt/libvirt-sock (or the socket libvirt_uri names) ↓ -4. Connect to libvirt daemon via go-libvirt +4. Connect to libvirt daemon via go-libvirt, dial and handshake bounded at 1 second ↓ -5. Connection established (2-second timeout) +5. Connection established; on failure or a lost connection, retried 1-5 s apart ``` ### VM Creation Flow (Real Mode): diff --git a/docs/SCALING.md b/docs/SCALING.md index ccf1c06a..d9855ab0 100644 --- a/docs/SCALING.md +++ b/docs/SCALING.md @@ -613,7 +613,7 @@ nova: host: "0.0.0.0" port: 8774 libvirt_mode: real - libvirt_uri: "qemu+tcp://compute1/system" # Will be load-balanced + libvirt_uri: "qemu:///system" # each node dials its own local libvirt # The noVNC console is single-node only in v1 (SPEC-021): each node's token projection # and guest console sockets are node-local. With this, console requests answer 400 # "Unavailable console type novnc" instead of a URL that leads nowhere; diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index f112ca56..6407d3e0 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -192,8 +192,8 @@ docker logs lightstack 2>&1 | grep DEBUG 3. **Check file permissions**: ```bash - # libvirt socket - ls -la /var/run/libvirt/libvirt-sock + # libvirt socket: virtqemud-sock where the modular daemons run, else libvirt-sock + ls -la /var/run/libvirt/virtqemud-sock /var/run/libvirt/libvirt-sock # Image directory ls -la /var/lib/lightstack/images/ diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index 7e701422..8326566e 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -602,3 +602,51 @@ it could not read the database and purged only what it found on the node: the re host networking stayed behind. Now it removes them, as it does for a release with `database.url`. Without the URL, `lightstack cleanup` — which now refuses an empty one — would fail the hook and stop the uninstall. + +## Real mode: the libvirt socket, readiness and cluster DNS + +Five things an existing real-mode release meets: + +- **A `libvirt_uri` Lightstack cannot honour stops startup.** Lightstack only ever dials the libvirt + on its own node: `qemu:///system` (the default) or `qemu+unix:///system`, either with + `?socket=`, or an absolute socket path. Anything else — `qemu:///session`, + `qemu+tcp://`, `qemu+ssh://`, a relative path — used to be dialed as a file name and failed as a + missing socket. The server, `lightstack cleanup` and `lightstack console-tokens` now refuse it at + configuration load, naming the value and the accepted forms. With the default, Lightstack chooses + the socket on every connection attempt: `virtqemud-sock` where the modular daemons run, else + `libvirt-sock` ([CONFIGURATION](CONFIGURATION.md)). +- **A pod that cannot prepare the console socket directory is no longer Ready.** Before, it was + Ready and refused every guest start. Lightstack now connects to libvirt in the background and + prepares the directory after every connection; until the first preparation succeeds, `/readyz` + answers `503 hypervisor unavailable`, and on a single node the Service has no endpoint. A host + where libvirt never answers is restarted by the startup probe (30 × 5 s by default) and ends in + `CrashLoopBackOff`, with the cause at the top of each start's log, so `helm install --wait` fails. + Once prepared, later libvirt outages do not affect readiness. Docker Compose and bare metal are + not gated: their healthchecks probe `GET /v3`. +- **Host-networked pods resolve through the cluster DNS.** With `realMode.hostNetwork: true` the + chart now renders `dnsPolicy: ClusterFirstWithHostNet` on the Lightstack pod and the purge Job, + so in-cluster Service names (`..svc`) resolve. A name only the node's own + resolver answers — an `/etc/hosts` entry, a zone only the node's resolver serves — must now be + reachable through the cluster DNS, for example through a CoreDNS forward or stub domain. +- **The host workarounds can go.** A `libvirt-sock` symlink to `virtqemud-sock`, enabling + `virtproxyd.socket` so that `libvirt-sock` exists, and warming libvirt before Lightstack starts + (`virsh capabilities`) are no longer needed: Lightstack finds `virtqemud-sock` itself, bounds each + attempt at 1 second, and retries the preparation until libvirt answers. +- **The uninstall purge waits one second for libvirt, once.** With `dataRetentionPolicy: delete`, + `lightstack cleanup` used to wait on libvirt's handshake without a bound. It now makes one attempt + bounded at 1 second and fails closed. A socket-activated daemon that had exited is started by that + attempt and may miss it; `helm uninstall` then aborts. Run it again: the daemon is up by then. + +## A server create must name its boot source + +A server create that names neither `imageRef` nor `block_device_mapping_v2` is now refused at once +with `400` "Missing imageRef attribute", in stub and real mode, as upstream Nova refuses it. It used +to be accepted with `202`, and the server then ended `ERROR` with the fault "the image reference is +not a valid image ID". Lightstack reads `block_device_mapping_v2` only, so a request that boots from +the legacy `block_device_mapping` alone was never honoured either, and is now refused the same way. + +## A shelve libvirt cannot perform ends `ERROR` + +Suspend, resume and shelve answer `202` and only then ask libvirt. A failed suspend or resume already +turned the server `ERROR`; a failed shelve was dropped, and the server read `SHELVED` while its guest +ran on. It now ends `ERROR` too, and all three log the cause (`shelve failed in libvirt`, and so on). diff --git a/docs/blueprint.md b/docs/blueprint.md index db0dbf0f..e22430aa 100644 --- a/docs/blueprint.md +++ b/docs/blueprint.md @@ -524,12 +524,13 @@ recipe: `helm status`, `kubectl rollout`/DaemonSet readiness, and terminal-failu Keystone-first, then discovered from the service catalog. **Health signal (SPEC-013 / #475).** Lightstack exposes three unauthenticated endpoints on the identity listener -(`:35357`): `GET /livez` (liveness — process only, never touches the DB), `GET /readyz` (readiness — `200` -only when all listeners are bound, the DB `Ping` succeeds, and the pod isn't draining; `503` with a short -curated reason otherwise, bounded to <1 s), and `GET|HEAD /healthcheck` (an OpenStack-compatible, -content-negotiated alias of `/readyz`). The **machine-readable signal sci-in-a-box consumes is the -Kubernetes Pod `Ready` condition** (the aggregate of the readiness probe and its roll-ups) — *not* a bespoke -Lightstack status protocol. The chart wires `liveness`/`readiness`/`startup` probes accordingly. +(`:35357`): `GET /livez` (liveness — process only, never touches the DB), `GET /readyz` (readiness — `200` only when +all listeners are bound, the DB `Ping` succeeds, and the pod isn't draining, and in real mode once the console +socket directory has been prepared, which takes the host's libvirt answering once (SPEC-031); `503` with a short +curated reason otherwise, bounded to <1 s), and `GET|HEAD /healthcheck` (an OpenStack-compatible, content-negotiated +alias of `/readyz`). The **machine-readable signal sci-in-a-box consumes is the Kubernetes Pod `Ready` condition** +(the aggregate of the readiness probe and its roll-ups) — *not* a bespoke Lightstack status protocol. The chart +wires `liveness`/`readiness`/`startup` probes accordingly. **Lifecycle automation (#421).** Maps onto this contract directly: **deploy** = `helm install` with schema-validated values → wait on Pod `Ready`; **populate** = seed users / projects / quotas (the binary diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/HOST_TESTING.md b/docs/specs/031-realmode-dns-libvirt-fixes/HOST_TESTING.md new file mode 100644 index 00000000..1722901e --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/HOST_TESTING.md @@ -0,0 +1,659 @@ +# Host testing guide — real mode: cluster DNS, the libvirt socket and the console directory retry (SPEC-031) + +Validates cloud-storage#630 (SPEC-031) by hand: the five host checks FR-021 (a)–(e) asks for, with the chart in +real mode on single-node k3s. It proves SC-001, SC-002, SC-003, SC-004, SC-004a and SC-007 on a real host; the +unit, chart and contract suites cover the rest (`baseline-gates.md`, T041 and T042). + +Written for Ubuntu 24.04, whose libvirt is the monolithic `libvirtd` only: its packages ship no `virtqemud`. An +Ubuntu host therefore proves four of the five checks. (b), a host running only the modular daemons, needs a +libvirt that ships them, such as openSUSE Tumbleweed or Fedora. §1 records which daemon the host runs and the +steps use it, so on such a host the same blocks prove (b) in 2.1 and (e) in 2.4. Nothing in the runbook is +specific to one distribution. + +**Run all commands from the repo root**, in one shell. Paths below are repo-relative (`charts/lightstack`, +`build/package/Dockerfile`); host paths (`/run/libvirt`, `/var/lib/…`, `/tmp/…`, `~/.kube`) are not. `sudo` +is needed throughout: k3s, libvirt's units and the node-local state are root's. Each section is one block. +Every block from §2 on starts by sourcing the helpers §1 writes, so any of them can be re-run on its own. + +Assumptions: +- One host with KVM (nested is fine), libvirt, Docker or Podman, and a running single-node k3s. The runbook + installs nothing and removes nothing: §0 only checks the host. +- The chart runs real mode with `privileged` and `hostNetwork`, PostgreSQL inside the cluster behind a + Service, and `dataRetentionPolicy: delete`. Node-local state goes to a dedicated root, + `/var/lib/lightstack-spec031`, never `/var/lib/lightstack`. + +**Validates:** + +| FR-021 | What the host must show | Proves | Ubuntu 24.04 | Modular host | +|---|---|---|---|---| +| (a) | a database URL naming a Service resolves under host networking: the pod becomes Ready, and the delete-policy uninstall's purge reaches the database | SC-001 | 2.1 | 2.1 | +| (b) | modular daemons only, default values: dials `virtqemud-sock`, a VM boots `ACTIVE`, the uninstall destroys it | SC-002 | — | 2.1 | +| (c) | cold libvirt: a failed first attempt, a later success, Ready, the first VM `ACTIVE`, no restart | SC-004, SC-004a | 2.2, 2.2b | 2.2, 2.2b | +| (d) | libvirt unreachable at start: not Ready, a create ends `ERROR` with a fault; sockets back → connects, Ready, VM `ACTIVE`, no restart | SC-004a, SC-007 | 2.3 | 2.3 | +| (e) | monolithic `libvirtd`: dials `libvirt-sock`, a VM boots `ACTIVE` | SC-003 | 2.1 | 2.4 | + +On Ubuntu, (c) and (d) run against `libvirtd`, where the ticket's (c) names `virtqemud`. For Lightstack it is +the same path: it dials a socket and does not know which daemon answers. + +--- + +## 0. Prerequisites + +The steps need these on the host. Install anything missing yourself; this section only checks. + +- **`sudo`** — k3s, libvirt's units and the node-local state are root's. +- **Docker or Podman** — builds the Lightstack image in §1. Either one; the steps use whichever is present. +- **k3s**, running, single node, at `/usr/local/bin/k3s` (the installer's path) — the cluster the chart is + installed into. §1 imports the image into its containerd. Its node must be Ready and its CoreDNS available: + step (a) resolves through it. A host firewall must let k3s's pod and service networks through + (`10.42.0.0/16`, `10.43.0.0/16`), as k3s itself requires. +- **`kubectl`** — the helpers drive the cluster with it (k3s ships one). +- **Helm 3.12 or later** — installs and uninstalls the chart. +- **KVM and libvirt** — `/dev/kvm`, `qemu-system-x86_64`, and libvirt socket-activated: `libvirtd.socket` (Ubuntu) + or `virtqemud.socket` (a modular host) active. +- **`virsh`** — lists and checks the host's domains. +- **The OpenStack client** (`openstack`) — creates the image, network and servers. +- **`curl`, `jq`** — probes `/readyz`, downloads the cirros image, reads the server's fault. +- **`ip`, `ss`, `awk`** — the node address and the port check. + +The block also fails if one of Lightstack's ports is already in use on the host: the pod is host-networked. + +```bash +missing=0 +for t in sudo curl jq helm kubectl openstack virsh qemu-system-x86_64 ip ss awk; do command -v "$t" >/dev/null || { echo "MISSING: $t"; missing=1; }; done +command -v docker >/dev/null || command -v podman >/dev/null || { echo "MISSING: docker or podman"; missing=1; } +test -x /usr/local/bin/k3s || { echo "MISSING: k3s at /usr/local/bin/k3s"; missing=1; } +test -c /dev/kvm || { echo "MISSING: /dev/kvm"; missing=1; } +systemctl is-active --quiet libvirtd.socket || systemctl is-active --quiet virtqemud.socket || { echo "MISSING: libvirt, neither libvirtd.socket nor virtqemud.socket is active"; missing=1; } +HV=$(helm version --template '{{.Version}}' 2>/dev/null); case "$HV" in v2.*|v3.[0-9].*|v3.1[01].*) echo "TOO OLD: helm $HV, the chart needs 3.12 or later"; missing=1;; esac +sudo /usr/local/bin/k3s kubectl get nodes --no-headers 2>/dev/null | grep -qw Ready || { echo "NOT READY: the k3s node"; missing=1; } +test "$(sudo /usr/local/bin/k3s kubectl -n kube-system get deploy coredns -o jsonpath='{.status.availableReplicas}' 2>/dev/null)" -ge 1 2>/dev/null || { echo "NOT READY: CoreDNS"; missing=1; } +ss -ltnH '( sport = :35357 or sport = :8774 or sport = :9696 or sport = :8776 or sport = :9292 or sport = :8775 or sport = :8778 )' | grep -q . && { echo "BUSY: a Lightstack port is in use on this host"; missing=1; } +echo "== versions" +docker --version 2>/dev/null || podman --version 2>/dev/null +helm version --short 2>/dev/null +kubectl version --client 2>/dev/null | head -1 +/usr/local/bin/k3s --version 2>/dev/null | head -1 +jq --version 2>/dev/null +openstack --version 2>/dev/null +virsh --version 2>/dev/null +qemu-system-x86_64 --version 2>/dev/null | head -1 +echo "== libvirt: libvirtd.socket=$(systemctl is-active libvirtd.socket) virtqemud.socket=$(systemctl is-active virtqemud.socket)" +ls -l /run/libvirt/libvirt-sock /run/libvirt/virtqemud-sock 2>&1 +[ "$missing" = 0 ] && echo "PREREQS OK" || echo "PREREQS FAILED: install or fix what is listed above, then re-run" +``` + +**Pass =** `PREREQS OK`, with no `MISSING:`, `TOO OLD:`, `NOT READY:` or `BUSY:` line above it, and: +- a version for each tool; Helm `v3.12` or later; +- on Ubuntu `libvirtd.socket=active virtqemud.socket=inactive`, `libvirt-sock` listed and `virtqemud-sock` + `No such file or directory`. A modular host reads the other way round. + +Fix whatever the block reports, then run it again before going on. + +On a modular host, a `libvirt-sock` beside `virtqemud-sock` is the symlink workaround or a running +`virtproxyd` — §1 clears both. A running `libvirtd.socket` there must be stopped by hand first +(`sudo systemctl stop libvirtd.service libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket`), or 2.1 tests +the monolithic daemon instead of the modular one. + +--- + +## 1. Setup steps + +One block, which does this in order: +- **Copies k3s's kubeconfig** to `~/.kube/lightstack-spec031.yaml`, owned by you, mode 0600. Every later block + uses that copy, so none of them can act on another cluster. +- **Records** which libvirt socket units are active, so §3 restores exactly those. From that record it also takes + the daemon that serves QEMU: `libvirtd` on Ubuntu, `virtqemud` on a modular host. A re-run keeps the first + record. +- **Modular hosts only** (both lines do nothing on Ubuntu): + - stops an active `virtproxyd`, which would provide `libvirt-sock`. §3 starts it again, since it is in the + record. + - removes a `libvirt-sock` symlink workaround, for good: the fix retires it, and `/run` is tmpfs anyway. +- **Builds the image** from the working tree, so the change under test is in the image whether or not it is + committed. It streams the image into k3s's own containerd. The tag is fully qualified, since Podman would + otherwise prefix `localhost/`, which the chart's `image.repository: lightstack` does not resolve to. Uses + `sudo docker` if your user cannot reach the Docker daemon, and Podman if there is no Docker. This part takes + a few minutes. +- **Writes three files:** + - `/tmp/ls031-host.sh` — the client environment, `NODE_IP`, `LIBVIRT_DAEMON` and the functions below. + `NODE_IP` is the address of the default route's interface, computed once now, before any Lightstack bridge + exists. The catalog advertises it (`endpointHost`), and the pod is host-networked, so the API answers on it + even while the pod is not Ready. + - `/tmp/ls031-values.yaml` — names the database by Service name only, `pg.ls031.svc`, which resolves only + through the cluster DNS (a). It keeps `realMode.libvirtSocketPath` at its default, so the binary chooses the + socket (b–e). `dataRetentionPolicy: delete` makes each uninstall purge the release. + - `/tmp/ls031-pg.yaml` — a PostgreSQL Deployment and Service. Its readiness probe checks TCP, so it turns + Ready only once the final server listens, not during the image's init-time restart. + +The functions: + +| Function | Does | +|---|---| +| `ls_fresh` | recreates namespace `ls031` with a new PostgreSQL, so each step starts on an empty database | +| `ls_install [helm args]` | installs release `ls`, waits until Keystone answers on `NODE_IP`, and prints its HTTP status (`000`: no answer) | +| `ls_readyz` | prints `/readyz`'s body and status | +| `ls_pods` | the Lightstack pod's READY and RESTARTS | +| `ls_log` | the log lines about the libvirt connection and the console socket directory | +| `ls_seed` | the cirros image (downloaded once), network `net031` and its subnet | +| `ls_boot NAME` | creates a server, waits, prints its status and fault | +| `ls_domains` | the host's Lightstack domains (`instance-…`) | +| `ls_uninstall` | the delete-policy uninstall, then `ls_domains` | +| `ls_sockets` | lists `libvirt-sock` and `virtqemud-sock` | +| `ls_cold` | stops `LIBVIRT_DAEMON`'s service, leaving its sockets, and empties the QEMU capabilities cache (2.2) | +| `ls_libvirt_off`, `ls_libvirt_on` | stop `LIBVIRT_DAEMON` with its socket units, and start the socket units again (2.3) | +| `ls_to_monolithic`, `ls_to_modular` | modular hosts: switch libvirt to `libvirtd` and back (2.4) | + +```bash +sudo -v +mkdir -p "$HOME/.kube" +sudo install -m 600 -o "$(id -un)" -g "$(id -gn)" /etc/rancher/k3s/k3s.yaml "$HOME/.kube/lightstack-spec031.yaml" +test -f /tmp/ls031-sockets-before || systemctl list-units --type=socket --state=active --plain --no-legend 'virt*' 'libvirtd*' | awk '{print $1}' > /tmp/ls031-sockets-before +LIBVIRT_DAEMON=libvirtd; grep -qx virtqemud.socket /tmp/ls031-sockets-before && LIBVIRT_DAEMON=virtqemud +systemctl is-active --quiet virtproxyd.socket && sudo systemctl stop virtproxyd.service virtproxyd.socket virtproxyd-ro.socket virtproxyd-admin.socket +test -L /run/libvirt/libvirt-sock && sudo rm -f /run/libvirt/libvirt-sock && echo "removed the libvirt-sock symlink workaround" +NODE_IP=$(ip -4 route get 1.1.1.1 | awk '{for (i = 1; i < NF; i++) if ($i == "src") print $(i + 1)}') +if command -v docker >/dev/null; then CE=(docker); docker info >/dev/null 2>&1 || CE=(sudo docker); else CE=(podman); fi +"${CE[@]}" build -t docker.io/library/lightstack:spec031 -f build/package/Dockerfile . +sudo -v +"${CE[@]}" save docker.io/library/lightstack:spec031 | sudo /usr/local/bin/k3s ctr images import - +printf 'export NODE_IP=%s LIBVIRT_DAEMON=%s\n' "$NODE_IP" "$LIBVIRT_DAEMON" > /tmp/ls031-host.sh +cat >> /tmp/ls031-host.sh <<'EOF' +export NS=ls031 +export KUBECONFIG=$HOME/.kube/lightstack-spec031.yaml +export OS_AUTH_URL=http://$NODE_IP:35357/v3 +export OS_USERNAME=admin OS_PASSWORD=secret OS_PROJECT_NAME=default OS_IDENTITY_API_VERSION=3 +export OS_USER_DOMAIN_NAME=Default OS_PROJECT_DOMAIN_NAME=Default +LS_SEL=app.kubernetes.io/name=lightstack,app.kubernetes.io/instance=ls +ls_fresh() { + kubectl delete namespace "$NS" --ignore-not-found --wait=true + kubectl create namespace "$NS" + kubectl apply -f /tmp/ls031-pg.yaml + kubectl -n "$NS" rollout status deploy/pg --timeout=300s +} +ls_install() { + helm install ls charts/lightstack -n "$NS" -f /tmp/ls031-values.yaml "$@" + for i in $(seq 1 150); do curl -s -o /dev/null "http://$NODE_IP:35357/v3" && break; sleep 2; done + curl -s -o /dev/null -w "Keystone on $NODE_IP, HTTP %{http_code}\n" "http://$NODE_IP:35357/v3" +} +ls_readyz() { + curl -s -w ' (HTTP %{http_code})\n' "http://$NODE_IP:35357/readyz" +} +ls_pods() { + kubectl -n "$NS" get pods -l "$LS_SEL" -o 'custom-columns=POD:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount' +} +ls_log() { + kubectl -n "$NS" logs deploy/ls-lightstack -c lightstack | grep -E 'Hypervisor started|connected to libvirt|connection attempt failed|connection lost|console socket directory' +} +ls_seed() { + stat -c %s /tmp/cirros-0.6.2.img 2>/dev/null | grep -qx 21430272 || curl -fsSL -o /tmp/cirros-0.6.2.img https://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img + openstack image create cirros --disk-format qcow2 --container-format bare --public --file /tmp/cirros-0.6.2.img -f value -c status + openstack network create net031 -f value -c status + openstack subnet create net031-sub --network net031 --subnet-range 10.31.0.0/24 -f value -c cidr +} +ls_boot() { + openstack server create "$1" --flavor m1.tiny --image cirros --network net031 --wait -f value -c status + openstack server show "$1" -f json | jq -r '"status=\(.status) fault=\(.fault.message // "none")"' +} +ls_domains() { + sudo virsh -c qemu:///system list --all | grep instance- || echo "no Lightstack domains" +} +ls_uninstall() { + helm uninstall ls -n "$NS" --timeout 10m + ls_domains +} +ls_sockets() { + ls -l /run/libvirt/libvirt-sock /run/libvirt/virtqemud-sock 2>&1 +} +ls_cold() { + sudo systemctl stop "$LIBVIRT_DAEMON.service" + sudo find /var/cache/libvirt/qemu/capabilities -name '*.xml' -delete + echo "cached capability files left: $(sudo find /var/cache/libvirt/qemu/capabilities -name '*.xml' | wc -l)" + echo "$LIBVIRT_DAEMON.service=$(systemctl is-active "$LIBVIRT_DAEMON.service") $LIBVIRT_DAEMON.socket=$(systemctl is-active "$LIBVIRT_DAEMON.socket")" +} +ls_libvirt_off() { + sudo systemctl stop "$LIBVIRT_DAEMON.service" "$LIBVIRT_DAEMON.socket" "$LIBVIRT_DAEMON-ro.socket" "$LIBVIRT_DAEMON-admin.socket" + ls_sockets +} +ls_libvirt_on() { + sudo systemctl start "$LIBVIRT_DAEMON.socket" "$LIBVIRT_DAEMON-ro.socket" "$LIBVIRT_DAEMON-admin.socket" + ls_sockets +} +ls_to_monolithic() { + : > /tmp/ls031-modular-daemons + for d in virtqemud virtnetworkd virtnodedevd virtsecretd virtstoraged virtinterfaced virtnwfilterd; do + systemctl is-active --quiet "$d.socket" && echo "$d" >> /tmp/ls031-modular-daemons + done + for d in $(cat /tmp/ls031-modular-daemons); do + sudo systemctl stop "$d.service" "$d.socket" "$d-ro.socket" "$d-admin.socket" + done + sudo systemctl start libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket + ls_sockets +} +ls_to_modular() { + sudo systemctl stop libvirtd.service libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket + for d in $(cat /tmp/ls031-modular-daemons 2>/dev/null); do + sudo systemctl start "$d.socket" "$d-ro.socket" "$d-admin.socket" + done + rm -f /tmp/ls031-modular-daemons + ls_sockets +} +EOF +cat > /tmp/ls031-values.yaml < /tmp/ls031-pg.yaml <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: pg + namespace: ls031 +spec: + selector: + matchLabels: + app: pg + template: + metadata: + labels: + app: pg + spec: + containers: + - name: pg + image: docker.io/library/postgres:18.3 + env: + - name: POSTGRES_USER + value: lightstack + - name: POSTGRES_PASSWORD + value: secret + - name: POSTGRES_DB + value: lightstack + ports: + - containerPort: 5432 + readinessProbe: + exec: + command: ["pg_isready", "-h", "127.0.0.1", "-U", "lightstack", "-d", "lightstack"] + periodSeconds: 2 +--- +apiVersion: v1 +kind: Service +metadata: + name: pg + namespace: ls031 +spec: + selector: + app: pg + ports: + - port: 5432 + targetPort: 5432 +EOF +source /tmp/ls031-host.sh +echo "== NODE_IP=$NODE_IP LIBVIRT_DAEMON=$LIBVIRT_DAEMON" && ls_sockets +echo "== recorded libvirt sockets:" $(cat /tmp/ls031-sockets-before) +echo "== image" && sudo /usr/local/bin/k3s ctr images ls -q | grep lightstack:spec031 +echo "== helpers" && type ls_fresh ls_boot ls_cold ls_libvirt_off ls_to_monolithic +grep -nE 'endpointHost|url:' /tmp/ls031-values.yaml +``` + +**Pass =** +- `NODE_IP= LIBVIRT_DAEMON=libvirtd` (`virtqemud` on a modular host). +- That daemon's socket is listed, and the other says `No such file or directory`. +- The recorded sockets include `libvirtd.socket`; on a modular host, `virtqemud.socket`. +- `docker.io/library/lightstack:spec031` is listed. +- The five functions are reported as shell functions. +- The values file shows `endpointHost: ` and + `url: postgres://lightstack:secret@pg.ls031.svc:5432/lightstack?sslmode=disable`. + +--- + +## 2. Test steps + +Run the steps in order. Each starts from a fresh namespace and database and ends with a delete-policy +uninstall, which purges the release from the host, so the next step starts clean. On Ubuntu, 2.4 only prints +that it does not apply. + +### 2.1 Cluster DNS and default values on the host's daemon — FR-021 (a), with (e) on Ubuntu and (b) on a modular host + +Installs with default values on the host as §1 found it. + +The pod reaching Ready proves the first half of (a): the database check passes only once `pg.ls031.svc` +resolves, which takes the cluster DNS. Before the fix a host-networked pod used the node's resolver and never +got there. + +The uninstall proves the second half, together with the daemon check: the purge Job finds the guests to destroy +in the database, through the same Service name. With no database it logs `database unreachable` and destroys +nothing, so a VM left behind means the purge could not resolve the name. + +```bash +source /tmp/ls031-host.sh +ls_fresh +ls_install +kubectl -n "$NS" rollout status deploy/ls-lightstack --timeout=300s +kubectl -n "$NS" get pods -l "$LS_SEL" -o jsonpath='{.items[0].spec.hostNetwork} {.items[0].spec.dnsPolicy}{"\n"}' +kubectl -n "$NS" exec deploy/ls-lightstack -c lightstack -- cat /etc/resolv.conf +ls_readyz +ls_pods +ls_log +ls_seed +ls_boot vm-default +ls_domains +sudo ls -ln /var/lib/lightstack-spec031/instances +sudo stat -c '%u:%g %a %n' /var/lib/lightstack-spec031/console-sockets +ls_uninstall +``` + +**Expect:** +- `deployment "ls-lightstack" successfully rolled out` — Ready (a). +- `true ClusterFirstWithHostNet`, and `/etc/resolv.conf` with `nameserver 10.43.0.10` and + `search ls031.svc.cluster.local svc.cluster.local cluster.local`. +- `ok (HTTP 200)`; the pod `READY true`, `RESTARTS 0`. +- The log has two lines: + - `connected to libvirt`, with the host daemon's socket: `"socket":"/var/run/libvirt/libvirt-sock"` on + Ubuntu (e), `"/var/run/libvirt/virtqemud-sock"` on a modular host (b). + - Then `console socket directory ready; guests can start`, naming + `"dir":"/var/lib/lightstack-spec031/console-sockets"` and a `qemu_user` uid:gid. On Ubuntu that is + `libvirt-qemu`'s uid and the `kvm` group's gid (`id libvirt-qemu`). + + A warm daemon shows no failure line here. +- `ls_seed`: `active`, `ACTIVE`, `10.31.0.0/24`. `ls_boot`: `ACTIVE`, then `status=ACTIVE fault=none`. One + `instance-… running` domain. +- The instance's directory and `console-sockets` both belong to the `qemu_user` uid:gid from the log, and + `console-sockets` has mode `750`. +- `release "ls" uninstalled`, then `no Lightstack domains`. The purge reached the database by Service name + (a) and destroyed the VM: (e) on Ubuntu, (b) on a modular host. + +### 2.2 Cold libvirt, forced — FR-021 (c) + +`ls_cold` stops the daemon's service — `libvirtd` on Ubuntu, `virtqemud` on a modular host. Its sockets stay, +so the next connection starts it. It also empties the daemon's QEMU capabilities cache, so its first answers +come slowly, as after a fresh boot. Nothing may touch libvirt between this and the install: no `virsh`. + +```bash +source /tmp/ls031-host.sh +ls_fresh +ls_cold +ls_install +kubectl -n "$NS" rollout status deploy/ls-lightstack --timeout=300s +ls_log +ls_pods +ls_seed +ls_boot vm-cold +ls_domains +ls_uninstall +``` + +**Expect:** +- `cached capability files left: 0`; `libvirtd.service=inactive libvirtd.socket=active` on Ubuntu, + `virtqemud…` on a modular host. systemd may warn that the stopped service can still be activated by its + sockets — that is the point. +- Rolled out — Ready, with no restart in between. +- The log shows a failed first attempt, then success: + - a failure — `libvirt connection attempt failed; retrying in the background` (`did not answer within + 1s`), or `console socket directory not ready; guest starts are refused until it is` with + `reading libvirt capabilities: libvirt did not answer within 1s`, or both; + - then `connected to libvirt` with the daemon's socket, and `console socket directory ready; guests can start`. +- `READY true`, `RESTARTS 0`; `vm-cold` `ACTIVE` with `fault=none` — the first guest after a cold start, with + no restart and no warm-up (SC-004); one `instance-…` domain; the uninstall ends with + `no Lightstack domains`. +- If the log has no failure line, this host answered even the cold read within 1 s, and the retry was not + exercised here. Run 2.2b, which forces it. + +### 2.2b Cold libvirt, frozen — FR-021 (c), the retry forced + +Run this when 2.2's log shows no failure line: the host answered even the cold read within 1 s, so 2.2 could +not show the retry. Here the daemon cannot answer in time. The block starts the daemon's service and then +freezes the process with `SIGSTOP`. Its socket still accepts connections, but nothing answers, so each of +Lightstack's attempts runs into its 1-second limit. After 20 s the block thaws the daemon with `SIGCONT` and +watches Lightstack connect by itself. + +The freeze covers the connection itself, so this shows the connection retry. The narrower case, a connection +that opens and a capabilities read that comes too late, cannot be timed by hand; +`TestPreparationRetriesUntilLibvirtAnswers` covers it. As in 2.3, the install raises the startup probe's +budget, so the probe cannot restart the pod while the daemon is frozen. Nothing may call `virsh` until the +thaw: it would hang too. + +```bash +source /tmp/ls031-host.sh +ls_fresh +sudo systemctl start "$LIBVIRT_DAEMON.service" +sudo pkill -STOP -x "$LIBVIRT_DAEMON" +echo "$LIBVIRT_DAEMON process state: $(ps -o stat= -C "$LIBVIRT_DAEMON")" +ls_install --set probes.startup.failureThreshold=120 +sleep 20 +ls_readyz +ls_pods +ls_log +echo "== thawing $LIBVIRT_DAEMON" +sudo pkill -CONT -x "$LIBVIRT_DAEMON" +kubectl -n "$NS" wait --for=condition=Ready pod -l "$LS_SEL" --timeout=180s +ls_log +ls_readyz +ls_pods +ls_seed +ls_boot vm-frozen +ls_domains +ls_uninstall +``` + +**Expect** (while frozen): +- The process state starts with `T` (stopped), e.g. `Tsl`. +- `Keystone on , HTTP 200`: the API starts without libvirt. +- `hypervisor unavailable (HTTP 503)`; `READY false`, `RESTARTS 0`. +- The log: `Hypervisor started in real mode; connecting to libvirt in the background`, then one WARN + `libvirt connection attempt failed; retrying in the background` with + `libvirt at /var/run/libvirt/virtqemud-sock did not answer within 1s` (`libvirt-sock` on Ubuntu). Later + attempts log at DEBUG only. No `connected to libvirt` yet. + +**Expect** (after the thaw): +- `condition met` within a few seconds — the loop retries every 1–5 s. +- The log adds `connected to libvirt` with the same socket, then `console socket directory ready; guests can + start`: the failed first attempt, then the success, with no restart (FR-021 c). +- `ok (HTTP 200)`; `READY true`, `RESTARTS 0`. +- `vm-frozen`: `ACTIVE`, `status=ACTIVE fault=none`; one `instance-…` domain; the uninstall ends with + `no Lightstack domains`. + +### 2.3 libvirt unreachable at start, forced — FR-021 (d) + +`ls_libvirt_off` stops the daemon's socket units, which remove the socket (`RemoveOnStop=yes`), and then the +block installs. The pod stays up but not Ready, so the Service has no endpoint. The client talks to the pod on +the node's own address instead, which host networking allows. `ls_libvirt_on` then starts the socket units +again, and the block watches Lightstack connect by itself. + +The block runs straight through. The install raises the startup probe's budget from 30 to 120 tries, 5 s apart. +The step keeps the pod unready on purpose, and a kubelet restart in the middle would hide whether Lightstack +recovers on its own. + +```bash +source /tmp/ls031-host.sh +ls_fresh +ls_libvirt_off +ls_install --set probes.startup.failureThreshold=120 +sleep 15 +ls_readyz +ls_pods +ls_seed +ls_boot vm-unreachable +ls_log +echo "== starting the $LIBVIRT_DAEMON sockets again" +ls_libvirt_on +kubectl -n "$NS" wait --for=condition=Ready pod -l "$LS_SEL" --timeout=180s +ls_log +ls_readyz +ls_pods +ls_boot vm-reconnected +ls_domains +ls_uninstall +``` + +**Expect** (before the sockets come back): +- Both sockets: `No such file or directory`. +- `hypervisor unavailable (HTTP 503)`; the pod `READY false`, `RESTARTS 0`. +- `ls_seed` works — the API needs no libvirt. In `ls_boot vm-unreachable` the create is accepted and `--wait` + reports the server in error. Then `status=ERROR fault=the hypervisor is not ready to start guests` — never + `ACTIVE` (SC-007). +- The log, in order: + - `Hypervisor started in real mode; connecting to libvirt in the background`; + - one WARN `libvirt connection attempt failed; retrying in the background` with + `no libvirt socket: neither /var/run/libvirt/virtqemud-sock nor /var/run/libvirt/libvirt-sock exists` + (later attempts log at DEBUG only); + - one ERROR `Failed to create VM via libvirt` naming `vm-unreachable`'s instance id, with the error + `console socket directory not ready: … not connected to libvirt`: the create's refusal. The + preparation attempt the start made logs at DEBUG only — the WARN above already reports the outage. + +**Expect** (after): +- The daemon's socket is listed again: `libvirt-sock` on Ubuntu. +- `condition met` within a few seconds — the loop retries every 1–5 s. +- The log adds `connected to libvirt` with that socket, and `console socket directory ready; guests can start`. +- `ok (HTTP 200)`; `READY true`, `RESTARTS 0` — no restart (SC-004a). +- `vm-reconnected`: `ACTIVE`, `status=ACTIVE fault=none`. One `instance-…` domain, `vm-reconnected`'s; + `vm-unreachable` never had one. The uninstall ends with `no Lightstack domains`. + +### 2.4 Monolithic libvirtd on a modular host — FR-021 (e) there + +**On a modular host only.** The block stops every active modular driver daemon with its sockets and starts +`libvirtd.socket`. It installs with default values, then switches back. On Ubuntu, 2.1 has already tested +`libvirtd`, and the block only prints that it skipped. + +```bash +source /tmp/ls031-host.sh +if [ "$LIBVIRT_DAEMON" = virtqemud ]; then + ls_fresh + ls_to_monolithic + ls_install + kubectl -n "$NS" rollout status deploy/ls-lightstack --timeout=300s + ls_log + ls_pods + ls_seed + ls_boot vm-monolithic + ls_domains + ls_uninstall + ls_to_modular +else + echo "skipped: this host runs libvirtd, which 2.1 tested; (b) needs a host with the modular daemons" +fi +``` + +**Expect** on a modular host: +- After the switch, `libvirt-sock` is listed and `virtqemud-sock` says `No such file or directory`. +- Rolled out — Ready. The log: `connected to libvirt` with `"socket":"/var/run/libvirt/libvirt-sock"`, then + `console socket directory ready; guests can start` (SC-003). A freshly started `libvirtd` may miss the + first attempt: a WARN `libvirt connection attempt failed; retrying in the background` with + `libvirt at /var/run/libvirt/libvirt-sock did not answer within 1s` before `connected to libvirt` is the cold + path recovering by itself, as in 2.2b, not a failure. +- `READY true`, `RESTARTS 0`; `vm-monolithic` `ACTIVE`, `fault=none`; one `instance-…` domain, listed through + `libvirtd`; the uninstall ends with `no Lightstack domains`. +- After the switch back, `virtqemud-sock` is listed again and `libvirt-sock` says + `No such file or directory`. + +On Ubuntu: `skipped: …`. (b) cannot be shown on this host. + +--- + +## 3. Teardown steps + +Removes what this run created and puts back what it changed. It goes in this order: +1. **libvirt first**, so the release's purge can reach it after an interrupted 2.2b, 2.3 or 2.4: it thaws a + frozen daemon, undoes any leftover switch to `libvirtd`, then starts the socket units §1 recorded. +2. The release and its namespace. +3. The test state root, the image, the kubeconfig copy and the temporary files. + +k3s, libvirt's packages and the tools stay: this runbook installed none of them. + +It reads the kubeconfig copy directly and touches the cluster only while that copy exists, so it cannot act on +another cluster. Safe to re-run. + +```bash +sudo pkill -CONT -x virtqemud; sudo pkill -CONT -x libvirtd +test -f /tmp/ls031-modular-daemons && sudo systemctl stop libvirtd.service libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket +test -f /tmp/ls031-sockets-before && sudo systemctl start $(cat /tmp/ls031-sockets-before) +export KUBECONFIG=$HOME/.kube/lightstack-spec031.yaml +test -f "$KUBECONFIG" && helm uninstall ls -n ls031 --timeout 10m +test -f "$KUBECONFIG" && kubectl delete namespace ls031 --ignore-not-found --wait=true +sudo virsh -c qemu:///system list --all | grep instance- || echo "no Lightstack domains" +sudo rm -rf /var/lib/lightstack-spec031 +if command -v docker >/dev/null; then CE=(docker); docker info >/dev/null 2>&1 || CE=(sudo docker); else CE=(podman); fi +"${CE[@]}" rmi docker.io/library/lightstack:spec031 2>/dev/null || true +sudo /usr/local/bin/k3s ctr images rm docker.io/library/lightstack:spec031 >/dev/null 2>&1 || true +rm -f /tmp/ls031-host.sh /tmp/ls031-values.yaml /tmp/ls031-pg.yaml /tmp/ls031-sockets-before /tmp/ls031-modular-daemons /tmp/cirros-0.6.2.img "$HOME/.kube/lightstack-spec031.yaml" +sudo /usr/local/bin/k3s ctr images ls -q | grep lightstack:spec031 || echo "image gone from k3s" +ls -l /run/libvirt/libvirt-sock /run/libvirt/virtqemud-sock 2>&1 +echo "teardown done" +``` + +**Pass =** +- `teardown done`, `no Lightstack domains` and `image gone from k3s`. +- After a complete run, `helm uninstall` reports `release: not found`: the last step already uninstalled it. +- The daemon's socket is listed — `libvirt-sock` on Ubuntu, `virtqemud-sock` alone on a modular host. +- `/var/lib/lightstack-spec031` is gone. + +Not put back: +- On a modular host, the `libvirt-sock` symlink, if §1 removed one. The fix makes it unnecessary, and a reboot + clears `/run` anyway. +- On a modular host where `virtproxyd.socket` was enabled only to provide `libvirt-sock`, it is no longer + needed either: `sudo systemctl disable --now virtproxyd.socket`. This runbook restarts it as it found it. + +--- + +## Troubleshooting + +- **§0 reports `NOT READY: the k3s node` or `CoreDNS`** — `sudo journalctl -u k3s -n 50` and + `sudo /usr/local/bin/k3s kubectl -n kube-system get pods`. On an SELinux host, a k3s service failing with + `status=203/EXEC` means SELinux blocks it. +- **`ls_install` prints `HTTP 000`, or the pod restarts with `Failed to connect to database`** — in 2.1 this is + the failure step (a) tests. Check that the pod's `/etc/resolv.conf` names `10.43.0.10` and the + `ls031.svc.cluster.local` search domain, and that `kubectl -n ls031 get pods -l app=pg` is Ready. The log is + in `kubectl -n ls031 logs deploy/ls-lightstack -c lightstack --previous`. An active host firewall (ufw, + firewalld) that drops k3s's pod and service networks gives the same failure. +- **`ImagePullBackOff`** — for `lightstack`, §1 did not import the image, or the tag differs: run + `sudo /usr/local/bin/k3s ctr images ls -q | grep lightstack`. For `postgres` or `alpine/k8s`, the host + cannot reach Docker Hub. +- **`helm uninstall` times out** — look at the purge Job: `kubectl -n ls031 get jobs` and + `kubectl -n ls031 logs job/ls-lightstack-teardown-purge`. The purge makes one 1-second attempt to reach + libvirt and fails closed (FR-020a). If 2.3 was interrupted with the sockets stopped, start them and uninstall + again: `source /tmp/ls031-host.sh && ls_libvirt_on && ls_uninstall`. +- **The uninstall fails with `relation "instances" does not exist`, and the release stays `uninstalling`** — + Lightstack never started, so the database was never migrated and the purge has nothing to read. It created + nothing on the host either, so remove the release without its hooks: + `helm uninstall ls -n ls031 --no-hooks`, then `ls_domains` should print `no Lightstack domains`. +- **A server ends `ERROR` while libvirt is up** — its fault says which step failed: + - `the image could not be made available on this host` or `the root disk could not be staged`: disk staging + failed before libvirt was asked. + - `the hypervisor could not create the guest`: libvirt refused the domain. Look at + `sudo journalctl -u libvirtd -n 50` (`-u virtqemud` on a modular host), and at `sudo dmesg | grep -i denied` + for an AppArmor (Ubuntu) or SELinux denial naming the path QEMU could not open. +- **2.3 shows `RESTARTS 1`** — the startup probe ran out of tries: the block was held up for longer than + 120 × 5 s. Run 2.3 again. +- **2.2 shows no failure line** — the host answered the cold read within 1 s. 2.2b forces the retry. +- **2.2b: `virsh` or anything else using libvirt hangs** — the daemon is still frozen: + `sudo pkill -CONT -x virtqemud` (`libvirtd` on Ubuntu). §3 does this too. +- **Domains left behind after an interrupted run** — list them with + `sudo virsh -c qemu:///system list --all`, then for each `instance-…`: + `sudo virsh -c qemu:///system destroy ` and `sudo virsh -c qemu:///system undefine `. diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/baseline-gates.md b/docs/specs/031-realmode-dns-libvirt-fixes/baseline-gates.md new file mode 100644 index 00000000..4e6d54a2 --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/baseline-gates.md @@ -0,0 +1,122 @@ +# Baseline gates — SPEC-031 + +The oracle for SC-008 ("the existing suites stay green"): the gates on the commit this branch's code +work starts from, before any SPEC-031 change, and the approval of the test strategy. + +## T001 — Baseline (2026-11-06, `dd9e069`) + +`dd9e069` was this branch's base before it was rebased onto `d81f081`, the `main` commit it is based +on now; it is no longer in the repository. The review of 2026-11-16 re-ran the gates on the rebased +branch (last section). + +Go 1.27.3, Helm with helm-unittest 1.2.1, PostgreSQL 18 from `make db-up`. Unit runs exclude +`test/contract` (as `make test` does), use `-count=1` (the test cache ignores `TestMain`'s +environment) and `-p 1` (packages sharing the seeded project race in parallel). + +| Gate | Command | Result | +|---|---|---| +| Build | `make build` | ok | +| Lint | `make lint` | 0 issues | +| Unit, PostgreSQL | `LIGHTSTACK_TEST_DB_URL=postgres://lightstack:secret@localhost:5432/lightstack?sslmode=disable go test -count=1 -p 1 $(go list ./... \| grep -v /test/contract)` | 19 packages ok, 0 failed | +| Unit, SQLite | the same with `LIGHTSTACK_TEST_DB_URL=sqlite:///tmp/ls-test/t.db` | 19 packages ok, 0 failed | +| Chart | `helm unittest charts/lightstack` | 24 suites, 169 tests passed | +| Contract, PostgreSQL | `make test-contract` against a stub server on a fresh `make db-up` database | 353 passed, 0 failed, 12 skipped | +| Contract, SQLite | `make test-contract` against a stub server on a new SQLite file | 353 passed, 0 failed, 12 skipped | + +The contract runs mirror CI's SQLite contract job: `config/lightstack.yaml` with `state_path` set to a +temporary directory, `LIGHTSTACK_DB_URL`, `LIGHTSTACK_JWT_SECRET`, `LIGHTSTACK_ENDPOINT_HOST=localhost` +and `LIGHTSTACK_METADATA_TRUST_INSTANCE_HEADER=true`. + +No failure predates this branch. + +## T002 — Test strategy approved (2026-11-06) + +The user approved the test strategy as written in research.md R10 and the test tasks of tasks.md, with +no changes. Host validation (T044) needs root on the openSUSE host, which the agent's sandbox does not +have: the agent generates the runbook (T043), and the user runs it. + +## T041 — Gates after the change (2026-11-06) + +T001's commands and environment, on the working tree (SPEC-031's changes, uncommitted, on `dd9e069`). +`make fmt` was checked without rewriting files, as `gofmt -l cmd internal pkg test tools`. + +| Gate | Baseline (T001) | After | +|---|---|---| +| Format | — | no file to reformat | +| Vet | — | `go vet ./...` clean | +| Build | ok | ok | +| Lint | 0 issues | 0 issues | +| Unit, PostgreSQL | 19 packages ok, 0 failed | 19 packages ok, 0 failed | +| Unit, SQLite | 19 packages ok, 0 failed | 19 packages ok, 0 failed | +| Chart | 24 suites, 169 tests passed | 25 suites, 181 tests passed | +| Contract, PostgreSQL | 353 passed, 0 failed, 12 skipped | 353 passed, 0 failed, 13 skipped | +| Contract, SQLite | 353 passed, 0 failed, 12 skipped | 353 passed, 0 failed, 13 skipped | +| No-hypervisor contract case | RED: the server went `ACTIVE` | PASS: `ERROR` with a fault | + +- The chart's 12 new tests: the new `dns_policy_test.yaml` suite (7), three `config_render_test.yaml` + cases and two `schema_test.yaml` cases. +- The 13th contract skip is the new gated `TestNoHypervisorCreateEndsErrorWithFault`, which skips against + a stub server by design. +- The no-hypervisor run (quickstart §2): a real-mode server whose `libvirt_uri` names a socket that does + not exist, on SQLite in a temporary directory. `/readyz` answered `503 hypervisor unavailable`; the gated + case passed with `LIGHTSTACK_CONTRACT_NO_HYPERVISOR=1`. +- After these runs, T042 changed one test in `pkg/hypervisor` (below). That package was re-run with + `-race` (ok) and `make lint` repeated (0 issues). + +No gate regressed. + +## T042 — Mutation check (2026-11-06) + +Each fix was reverted in a `git clone --shared` scratch clone carrying the worktree's changes. The +worktree was never edited. First, the unmutated clone passed every test command used below. After each +mutation the file was restored, and at the end the clone matched the worktree. + +| # | Fix reverted | Caught by | +|---|---|---| +| M1a | `dnsPolicy` removed from the pod spec (T007) | `dns_policy_test.yaml`: the Deployment and DaemonSet cases | +| M1b | `dnsPolicy` removed from the purge Job (T007) | `dns_policy_test.yaml`: the purge Job case | +| M2 | the socket chosen once per dialer, not per attempt (T004, T013) | `TestSocketDialer/a socket that appears later…`; `TestStartVMManagerConnectsWhenSocketAppears` (see below) | +| M3 | `dialWithin`'s 1 s bound removed (T011, T013, T014) | `TestNewVMManagerBoundsTheAttempt`, `TestStartVMManagerAbandonsSilentHandshake`, `TestCleanupBoundsASilentDaemon` | +| M4 | the single-flight join replaced by a lock, then an attempt of its own (T020) | `TestStartJoinsInFlightAttempt` | +| M5 | `prepared` cleared when a redo fails (T020) | `TestRedoAfterReconnect` | +| M6a–d | the ownership step removed from create, hard reboot, start, resume — one at a time (T021) | `TestGuestStartMakesTheInstanceDirQEMUs`, the matching subtest each time | +| M7a | real mode with no manager turns `ACTIVE` again (T015) | `TestNoHypervisorCreateEndsError/no_manager_wired_in` | +| M7b | a `CreateVM` failure written inline again, with no fault (T015, T016) | `TestNoHypervisorCreateEndsError/unconnected_manager`; the contract case against a binary built with M7b (fails at the fault assertion) | +| M8 | the `loadConfig` check removed (T034, T035) | `TestLoadConfigLibvirtURI`; `TestUnsupportedLibvirtURIRefused` for server, cleanup and console-tokens | +| X1 | `HypervisorReady` always ready (T022; beyond T042's list) | `TestHypervisorReady` | +| X2 | `readinessChecks` without the hypervisor check (T023; beyond T042's list) | `TestReadinessChecks` | + +All 16 were caught, none by a compile error. M7a and M7b ran on SQLite. M7a's branch is unreachable +from the real binary — `InitHypervisor` always leaves a manager — so it has no contract counterpart; before +the change, the contract case went RED on exactly that fake `ACTIVE`. + +**One finding, fixed.** `TestStartVMManagerConnectsWhenSocketAppears` raced its own background loop. When +the loop's first dial ran after the test had created the socket, it connected at once. The test then +passed in milliseconds without exercising a later attempt, and in the combined M2 run it passed. It now +waits for the logged failed attempt before the socket appears. It takes one backoff step (1 s) every +time, and fails under M2 (10 s timeout). + +## Review (2026-11-16) — gates after the review's fixes + +On `a599f0b` (the branch on `d81f081`) plus the review's uncommitted fixes. Go 1.27.3, Helm 4.3.1 +with helm-unittest 1.2.1. The PostgreSQL runs used a throwaway PostgreSQL 18 container, the SQLite +runs a temporary file; the contract run used a stub server on SQLite, as CI's SQLite job does. + +| Gate | Result | +|---|---| +| Format, vet, `go mod tidy -diff` | clean | +| Lint | 0 issues | +| Unit, PostgreSQL, `-race` | 19 packages ok, 0 failed | +| Unit, SQLite, `-race` | 19 packages ok, 0 failed | +| Chart | 25 suites, 181 tests passed; Horizon chart 3 tests passed | +| Contract, SQLite | 353 passed, 0 failed, 13 skipped | +| Contract, PostgreSQL | not re-run in the review | +| No-hypervisor contract case | PASS: `ERROR` with a fault; `/readyz` `503 hypervisor unavailable` | +| Changed-line coverage against `d81f081` | 230 of 238 executable lines covered | + +The eight uncovered lines are four in `main()`, which no unit test runs; a failed rollback write; the +create goroutine's panic recovery; `afterConnect` returning on a dropped connection; and the DEBUG line +for a second consecutive connection failure. + +Every new test was seen failing first against the code it pins, or, for behaviour that already held, +against a mutation applied through `go test -overlay` — the worktree was never edited for a mutation. diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/checklists/requirements.md b/docs/specs/031-realmode-dns-libvirt-fixes/checklists/requirements.md new file mode 100644 index 00000000..9dd50bcf --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/checklists/requirements.md @@ -0,0 +1,42 @@ +# Specification Quality Checklist: Real mode — cluster DNS under host networking, the libvirt socket, and a capabilities read that is never retried + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-11-06 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Iteration 1 (2026-11-06): two markers open — FR-017 (whether the preparation state reaches readiness or the hypervisor API, against SPEC-013 FR-001) and User Story 6 (whether the fake-`ACTIVE` server on an unreachable libvirt is in scope). +- Iteration 2 (2026-11-06): both answered and recorded under Clarifications. Readiness waits for the one-time preparation in real mode (FR-017, FR-017a, FR-018a, FR-023, FR-024; User Story 3 scenarios 6–8; new edge cases on a preparation that never succeeds, single node, multi-node, Compose/bare metal; SC-004a). User Story 6 is in scope as written (FR-019, FR-020). Zero markers remain; every FR/SC referenced is defined. +- Iteration 3 (2026-11-06, clarify): five more answers, all integrated — a server action without a libvirt connection answers as a lost connection does (FR-019); ownership of the per-instance directory is ensured before every guest start, self-heal included (FR-014, new Key Entity "Guest start"); host validation forces the cold and unreachable paths and adds a monolithic step (FR-010, FR-021 a–e); `lightstack cleanup` makes one attempt and fails closed (FR-020a); the chart docs and `docs/UPGRADING.md` cover the DNS change and the retired workarounds (FR-003a, FR-024). Also fixed: FR-016's "promptly" is now bounded, SC-001 and SC-003 no longer over-claim, "schema" in Assumptions now says database schema. Zero markers; every FR/SC referenced is defined. All items still pass. +- Iteration 4 (2026-11-06, analysis remediation): all 17 findings fixed across spec, plan, research, data model, contracts, quickstart and tasks. The spec gained a second clarification session (analysis): `lightstack cleanup`'s attempt is bounded at 1 s (FR-020a); a guest start takes the result of an attempt already running (FR-013, FR-016); every real-mode create that ends `ERROR` records a fault (FR-019, SC-007; the out-of-scope line removed). FR-005 now holds the `libvirtSocketPath` documentation sentence FR-022 used to repeat; FR-009 names every entry point. Zero markers; every FR/SC referenced is defined. All items still pass. +- Iteration 5 (2026-11-06, second analysis): 6 findings fixed. FR-019 and US6 scenario 2 now say each handler's own lost-connection message (a fourth analysis-session entry records it; the clarify record points to it); the contract and research list the per-step fault messages; T015 boots from an existing volume under a temporary state path, adds `TestCreateFault`, and seeds the `os-stop` target `ACTIVE`; T024 checks the `go mod tidy` diff; the plan's dependency row is exact. Zero markers; every FR/SC referenced is defined. All items still pass. +- Iteration 6 (2026-11-16, review): a third clarification session (review) adds FR-019a — a create must name its boot source — and FR-019b — an action libvirt fails after its `202` ends `ERROR`; FR-017a and FR-024 gained a clause each. +- Implementation detail, read against this project's convention: chart values (`realMode.*`, `dnsPolicy`, `probes.*`), config keys (`nova.libvirt_uri`) and socket paths are the operator's interface, so they stay in requirements. Code locations appear only in Context, as the evidence for each defect. FR-014's ordering clause and FR-017's "from the recorded state, never from a libvirt call" are behaviour (no start against a root-owned directory; readiness that cannot flap with libvirt), not design choices. +- Audience: operators of a single-binary OpenStack. "Non-technical" is read as "no code knowledge needed to verify a requirement": every FR is checkable by rendering the chart, running the binary, asking `/readyz`, or using the OpenStack CLI. +- Success criteria name `ACTIVE`, Ready, a chart install and libvirt: domain terms an operator verifies directly, not technology choices. diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/contracts/chart-render.md b/docs/specs/031-realmode-dns-libvirt-fixes/contracts/chart-render.md new file mode 100644 index 00000000..f9a38661 --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/contracts/chart-render.md @@ -0,0 +1,49 @@ +# Contract: Chart render — cluster DNS and the libvirt socket + +**Feature**: `031-realmode-dns-libvirt-fixes` + +The chart's interface is what `helm template` produces and which values the schema refuses. Each clause is a helm-unittest assertion. "Real values" means `realMode.enabled: true` plus the image, secrets and a PostgreSQL `database.url`, as the existing suites set them. + +## C-DNS-1 — Host-networked pods use the cluster DNS (FR-001) + +Given real values with `realMode.hostNetwork: true`: + +| Template | Path | Value | +|---|---|---| +| `templates/deployment.yaml` | `spec.template.spec.dnsPolicy` | `ClusterFirstWithHostNet` | +| `templates/daemonset.yaml` (`deploymentMode: multi-node`) | `spec.template.spec.dnsPolicy` | `ClusterFirstWithHostNet` | +| `templates/teardown-purge-job.yaml` (`dataRetentionPolicy: delete`) | `spec.template.spec.dnsPolicy` | `ClusterFirstWithHostNet` | + +`hostNetwork: true` is rendered beside it in each, as today. + +## C-DNS-2 — No policy without host networking (FR-002) + +- Stub mode, with or without `realMode.hostNetwork: true`: no `dnsPolicy` on any of the three. +- Real values with `realMode.hostNetwork: false`: no `dnsPolicy` on any of the three. The pod spec still says `hostNetwork: false`, as today. +- The console proxy keeps its own `dnsPolicy: ClusterFirstWithHostNet` (unchanged, `console_proxy_test.yaml`). + +## C-SOCK-1 — A non-default socket reaches the binary (FR-004) + +Given real values with `realMode.libvirtSocketPath: /run/libvirt/virtqemud-sock`: + +- `templates/secret.yaml` → `stringData["lightstack.yaml"]` matches `libvirt_uri: "/run/libvirt/virtqemud-sock"`. +- The Deployment and the purge Job mount `/run/libvirt` as the `libvirt-run` hostPath (`type: Directory`), as today (`libvirt_socket_mount_test.yaml`). +- Both read that one config Secret, so no second assertion per workload is needed beyond the existing `config` volume. + +## C-SOCK-2 — The default render is unchanged (FR-004) + +Given real values without `realMode.libvirtSocketPath`, or with it set to `/var/run/libvirt/libvirt-sock`: `stringData["lightstack.yaml"]` matches `libvirt_uri: "qemu:///system"`, and the mount is `/var/run/libvirt`. + +## C-SOCK-3 — The schema refuses a non-absolute socket (FR-006) + +`realMode.libvirtSocketPath: libvirt-sock` and `realMode.libvirtSocketPath: ""` both fail schema validation, as a relative `realMode.hostStoragePath` already does (`"pattern": "^/.+"`). + +## C-DOC-1 — What the chart's own docs say (FR-003a, FR-005, FR-023) + +Not helm-unittest; reviewed in the diff: + +| Where | Says | +|---|---| +| `values.yaml` `realMode.hostNetwork` comment; `values.schema.json` description; README row | host networking renders `dnsPolicy: ClusterFirstWithHostNet`, so the Lightstack pod and the purge Job resolve in-cluster Service names | +| `values.yaml` `realMode.libvirtSocketPath` comment; schema description; README row | a non-default value is also the socket dialed; the default lets the binary choose `virtqemud-sock`, else `libvirt-sock`; with `secrets.existingSecret` the operator's own config must name the same socket | +| README health table; `CONTRACT.md` §2; `NOTES.txt`; `values.yaml` `probes` comments; schema probe descriptions | in real mode, `/readyz` also waits until the console socket directory has been prepared once; a host where that never succeeds is restarted by the startup probe | diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/contracts/guest-operations.md b/docs/specs/031-realmode-dns-libvirt-fixes/contracts/guest-operations.md new file mode 100644 index 00000000..91595dea --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/contracts/guest-operations.md @@ -0,0 +1,62 @@ +# Contract: Compute guest operations while libvirt is unreachable or unprepared + +**Feature**: `031-realmode-dns-libvirt-fixes` (FR-014, FR-016, FR-019) + +Real mode only. "Unreachable" covers both "never connected yet" and "connection lost" — they answer alike (spec Clarifications). + +## C-GUEST-1 — Create (FR-019, FR-019a) + +A create that names neither `imageRef` nor `block_device_mapping_v2` is refused before anything is +created, in either mode: `400` "Missing imageRef attribute", as upstream answers. Every other create: + +| State | `POST /servers` | Server ends | +|---|---|---| +| unreachable | `202` (accepted, as always) | `ERROR` with a `fault`, never `ACTIVE` | +| connected, not prepared, one on-demand attempt fails | `202` | `ERROR` with a `fault` | +| connected, prepared | `202` | `ACTIVE` with a guest | + +Every real-mode create that ends `ERROR` records a `fault` through `markInstanceError` (code 500). The message names the step that failed, curated as `allocationFault` does for networks; the detail is in the log: + +| Step that failed | `fault.message` | +|---|---| +| libvirt create (`createFault`): the console socket directory, or the connection it needs, not ready | "the hypervisor is not ready to start guests" | +| libvirt create (`createFault`): anything else | "the hypervisor could not create the guest" | +| no hypervisor wired in (real mode) | "the hypervisor is not ready to start guests" | +| per-instance directory (image or volume boot) | "the per-instance directory could not be prepared" | +| root disk staging (overlay) | "the root disk could not be staged" | +| image base not available on the host | "the image could not be made available on this host" | +| no root device Lightstack boots from: an empty `imageRef`, and mappings with no root volume or image | "the image reference is not a valid image ID" | +| panic in the create goroutine | "the server could not be created" | + +The faults the create already records (network allocation, boot volume checks) are unchanged. `server show` returns the fault as upstream does for a failed build. + +## C-GUEST-2 — Server actions (FR-016, FR-019) + +| Action | Preparation never succeeded (includes never connected) | Prepared, connection lost later | +|---|---|---| +| start, hard reboot | one on-demand attempt; if it fails, `500` naming the console socket directory and its cause — FR-016's existing refusal. While unconnected the cause reads "not connected to libvirt" | `500` "failed to perform ", as a lost connection answers today (`sendLifecycleError`) | +| resume | `202`; the start's one on-demand attempt fails in the background, and the server ends `ERROR`, FR-016's refusal — directory and cause — in the log | `202`; the server ends `ERROR`, the cause in the log | +| suspend, shelve | `202`; the server ends `ERROR`, the cause in the log (FR-019b) | the same | +| stop, soft reboot, delete, volume attach/detach | `500` "failed to perform " (or the handler's existing generic `500`) | the same | +| console (`remote-consoles`) | `500` — the console readback's existing answer for a failed read (`sendConsoleEndpointError`, default case) | the same | + +- `startRefusal` names the directory only for `ErrConsoleSocketDirNotReady`. Every other libvirt failure keeps the generic message. +- Suspend, resume and shelve answer `202` first and ask libvirt afterwards, so a failure can reach only the log and the server's status. Shelve used to drop the failure and read SHELVED while the guest ran on (review, 2026-11-16). +- No `503`, and no new message. +- A delete that cannot reach libvirt leaves the server in place, as during a lost connection today. + +## C-GUEST-3 — Ownership before every launch (FR-014) + +Immediately before QEMU launches — create (image or volume boot), start, hard reboot, resume, self-heal: + +1. If not yet prepared, one on-demand preparation attempt — or the result of the one already running — within 1 s. +2. `/instances/` exists and belongs to QEMU's recorded uid:gid. It is created or chowned on the spot, idempotently. + +If step 2 fails, the guest is not launched: a create ends `ERROR` with a `fault` and its defined domain is undefined; an action answers `500`. + +This repairs a directory made while QEMU's user was unknown, one left root-owned by an earlier release, and one whose owner changed under a reconnect redo. + +## C-GUEST-4 — What does not change + +- Stub mode, and every read path (`server list`, `server show`): no libvirt involved. +- Existing guests keep running while libvirt is unreachable: libvirt being down does not stop QEMU processes, and the state reconciler treats an unanswered read as "unknown" (SPEC-025). diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/contracts/libvirt-connection.md b/docs/specs/031-realmode-dns-libvirt-fixes/contracts/libvirt-connection.md new file mode 100644 index 00000000..76198474 --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/contracts/libvirt-connection.md @@ -0,0 +1,74 @@ +# Contract: libvirt connection — configuration, socket choice, retries + +**Feature**: `031-realmode-dns-libvirt-fixes` + +What an operator configures, and what the binary does with it, in real mode. Stub mode never reads `nova.libvirt_uri`. + +## C-URI-1 — Accepted forms (FR-007, FR-008) + +| `nova.libvirt_uri` | Dials | +|---|---| +| `""`, `qemu:///system`, `qemu+unix:///system` | the automatic choice (C-SOCK-AUTO) | +| `qemu:///system?socket=/abs`, `qemu+unix:///system?socket=/abs` | `/abs` | +| `/abs` | `/abs` | + +An explicit socket is dialed whatever else exists. + +## C-URI-2 — Refused forms stop startup (FR-009) + +In real mode, any other value stops the process when its configuration loads: + +| Entry point | Exit | Output contains | +|---|---|---| +| `lightstack --config …` | 1 | the value and the accepted forms | +| `lightstack cleanup --config …` | 2 | the same | +| `lightstack console-tokens --config … list` | 2 | the same | + +Message shape (wording settled in implementation; the value and all three accepted shapes must appear): + +```text +nova.libvirt_uri "" is not supported; accepted: qemu:///system or qemu+unix:///system (chooses virtqemud-sock, else libvirt-sock), either with ?socket=, or an absolute socket path +``` + +Refused: another driver, `qemu:///session`, `qemu:///embed`, a host or remote transport (`qemu://h/system`, `qemu+tcp://`, `qemu+ssh://`, `qemu+tls://`, `qemu+libssh2://`, `qemu+ext://`), a relative path, a relative or repeated `socket=`, any other parameter, a fragment. In stub mode the same values are accepted unread. + +## C-SOCK-AUTO — The automatic choice (FR-007, FR-012) + +On **every** connection attempt: + +1. `/var/run/libvirt/virtqemud-sock` exists → dial it. +2. else `/var/run/libvirt/libvirt-sock` exists → dial it. +3. else → the attempt fails naming both paths. + +"Exists" follows symlinks. A monolithic `libvirtd` host has only `libvirt-sock`, so it dials what it dials today. + +## C-LOG-1 — What the log says (FR-010) + +| Event | Level | Names | +|---|---|---| +| connection established | INFO | the socket dialed | +| connection attempt failed | DEBUG (first failure: WARN) | the socket or sockets tried, and the cause | +| connection lost | WARN | — (as today) | + +## C-CONN-SERVER — The server never waits for libvirt (FR-020) + +- Startup does not block on libvirt. In real mode the hypervisor starts unconnected, and a background loop dials. +- Each attempt (dial + handshake) is bounded at 1 s. Between attempts the loop waits 1 s, doubling to at most 5 s. It stops at shutdown. +- On every connection, the first and each reconnect, the loop runs the console socket directory preparation until it succeeds (C-PREP). +- Until connected, libvirt calls fail as during a lost connection; nothing falls back to stub (see `guest-operations.md`). + +## C-CONN-CLEANUP — `lightstack cleanup` makes one attempt (FR-020a) + +- One attempt with the C-SOCK-AUTO choice (or the explicit socket), bounded at 1 s for dial plus handshake — the same bound as the server's attempts (Article III). +- A missing socket fails it at once; a daemon that does not answer within the bound fails it too. Either way: exit non-zero, naming the socket or sockets tried, and the uninstall stops (fail-closed, SPEC-010). +- No background dialing, no retry. A socket-activated daemon that had exited is started by the attempt itself, so re-running the uninstall finds it warm. + +## C-PREP — Console socket directory preparation (FR-013, FR-015, FR-017a) + +| Rule | Detail | +|---|---| +| Attempt | read libvirt's capabilities under 1 s, then make the directory QEMU's (0750); while an earlier read is still unanswered, fail at once without a second read | +| Background | after every connection, attempts with the 1 → 5 s pause until one succeeds | +| On demand | one attempt per guest start while not yet prepared — or, if an attempt is already running, that attempt's result; never a wait followed by a second attempt (single-flight) | +| Sticky | the first success is never withdrawn; a later success replaces QEMU's owner | +| Log | first failure: ERROR with the cause (as today); an attempt that failed only for want of a connection is DEBUG, since the connection loop reports the outage · success after a failure: INFO, directory and uid:gid · redo after reconnect: DEBUG, or INFO if the owner changed · other retries: DEBUG | diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/contracts/readiness.md b/docs/specs/031-realmode-dns-libvirt-fixes/contracts/readiness.md new file mode 100644 index 00000000..3d4b25db --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/contracts/readiness.md @@ -0,0 +1,41 @@ +# Contract: Readiness in real mode + +**Feature**: `031-realmode-dns-libvirt-fixes` — amends SPEC-013 FR-001 for one case (spec Clarifications, FR-017) + +## C-READY-1 — Inputs + +| Mode | `GET /readyz` answers `200` when | +|---|---| +| stub | the database is reachable, every listener is bound, the pod is not draining (unchanged) | +| real | all of the above **and** the console socket directory has been prepared at least once | + +## C-READY-2 — Stickiness + +Once the preparation has succeeded, nothing about libvirt affects readiness again: + +- a lost connection does not; +- a reconnect does not; +- a redo of the preparation that fails does not. + +SPEC-013 FR-001 keeps holding for outages. + +## C-READY-3 — The answer + +| State | Status | Reason (text and JSON `reasons`) | +|---|---|---| +| not yet prepared | `503` | `hypervisor unavailable` | +| not yet prepared, database down | `503` | `database,hypervisor unavailable` | +| prepared | `200` | `OK` (unchanged) | + +- No error detail, as the database reason (CONTRACT.md §2). +- The answer reads recorded state and makes no libvirt call, so the 800 ms per-check bound is never at stake. +- `GET|HEAD /healthcheck` mirrors it. + +## C-READY-4 — Consequences an operator sees + +| Situation | Effect | +|---|---| +| single node, not yet prepared | the Service has no endpoint: all six APIs are unreachable through it. The listeners still answer on the pod's own address — the node's, with host networking | +| preparation never succeeds | the startup probe (30 × 5 s by default) restarts the pod, ending in CrashLoopBackOff with the cause at the top of each start's log; `helm install --wait` fails | +| multi-node | only the affected node's pod leaves the Service | +| Docker Compose, bare metal | nothing gates on it; the Compose healthchecks probe `GET /v3`, not `/readyz` | diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/data-model.md b/docs/specs/031-realmode-dns-libvirt-fixes/data-model.md new file mode 100644 index 00000000..56b90782 --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/data-model.md @@ -0,0 +1,104 @@ +# Data Model: Real mode — cluster DNS under host networking, the libvirt socket, and a capabilities read that is never retried + +**Feature**: `031-realmode-dns-libvirt-fixes` | **Date**: 2026-11-06 + +No database entity, schema change or migration. What this feature models is configuration and in-process state: one configuration key, two chart values, and three pieces of state inside the real-mode hypervisor manager. Requirement numbers refer to `spec.md`; research items (R*n*) to `research.md`. + +--- + +## 1. libvirt connection setting — `nova.libvirt_uri` + +| Value | Target | Notes | +|---|---|---| +| `""` | automatic | as `qemu:///system` | +| `qemu:///system` | automatic | the default; what the chart renders for the default socket | +| `qemu+unix:///system` | automatic | same meaning | +| `qemu:///system?socket=` or `qemu+unix:///system?socket=` | explicit `` | FR-008 | +| `` (no scheme) | explicit `` | kept from today; what the chart renders for a non-default socket | + +**Validation** (real mode only, at configuration load — R3): anything else is refused with one message naming the value and the accepted forms — another driver, `qemu:///session`, `qemu:///embed`, a host or remote transport, a relative path or relative `socket=`, a repeated `socket=`, any other parameter, a fragment. Stub mode does not check it (FR-009). + +## 2. Socket choice + +| Target | Each connection attempt dials | +|---|---| +| explicit `` | ``, whatever else exists | +| automatic | `/var/run/libvirt/virtqemud-sock` if it exists, else `/var/run/libvirt/libvirt-sock` if it exists, else fails naming both | + +- Decided on **every** attempt, reconnects included (FR-007, R4). +- "Exists" follows symlinks. +- The socket dialed is logged on success; a failure names what was tried (FR-010). + +## 3. Connection state (server) + +```text + start (real mode) + │ + ▼ + ┌─────────────┐ attempt fails or times out (1 s) + │ unconnected │◄──────────── back off 1 s → 5 s ──┐ + └──────┬──────┘ │ + │ attempt succeeds (socket logged) │ + ▼ │ + ┌─────────────┐ Disconnected() fires │ + │ connected │───────────────────────────────────┘ + └──────┬──────┘ + │ on entering: run preparation (§4) until it succeeds + ▼ + shutdown → loop stops (Close) +``` + +- While **unconnected**, every libvirt call fails as during a lost connection (FR-019). Nothing falls back to stub. +- `lightstack cleanup` has no loop: one attempt, bounded at 1 s like the loop's. A missing socket, or a daemon that does not answer in time, fails it (FR-020a, R5). + +## 4. Console socket directory preparation + +| Field | Meaning | +|---|---| +| `prepared` | set by the first success; **never cleared** | +| owner | uid:gid QEMU runs as, from libvirt's capabilities; **replaced by every success** | +| directory | `/console-sockets` | + +**Transitions**: + +| Event | Effect | +|---|---| +| attempt succeeds | `prepared` ← true; owner ← reported; directory chowned 0750 to it | +| attempt fails while not prepared | guest starts keep being refused | +| attempt fails while prepared | nothing in effect changes (FR-015) | + +No cause is stored: a refused guest start carries the cause of the attempt it made or joined. + +**Attempts**: +- background, after every (re)connection, until one succeeds; +- on demand, one per guest start that finds `prepared` false. + +Callers share one in-flight attempt (single-flight): a guest start that arrives while an attempt runs takes that attempt's result. Each attempt is bounded at 1 s, so no caller waits longer than one bound (FR-013, FR-016, R6). + +## 5. Guest start + +Any launch of a guest's QEMU process: **create** (image or volume boot), **start**, **hard reboot**, **resume** (from managed save), **self-heal** re-creation. All of them reach `DomainCreate` in one of four functions (R7). + +**Preconditions, checked immediately before the launch**: +1. **Prepared** — if not, one on-demand attempt; if it fails, refused with the existing message naming the directory and the cause (FR-016). +2. **Per-instance directory** `/instances/` exists and belongs to the recorded owner — created or chowned on the spot, idempotently (FR-014). The domain UUID is the instance ID. + +## 6. Readiness inputs + +| Mode | `/readyz` ready when | +|---|---| +| stub | database reachable **and** all listeners bound (unchanged) | +| real | the above **and** `prepared` (sticky, §4) | + +- Reason when unprepared: `hypervisor unavailable` (combined with others as `database,hypervisor unavailable`). +- `/healthcheck` mirrors `/readyz` (FR-017). + +## 7. Chart values + +| Value | Renders | +|---|---| +| `realMode.enabled` + `realMode.hostNetwork` both true | `dnsPolicy: ClusterFirstWithHostNet` on the Deployment or DaemonSet pod and the purge Job (FR-001); otherwise no `dnsPolicy` (FR-002) | +| `realMode.libvirtSocketPath` = default `/var/run/libvirt/libvirt-sock` | mount `/var/run/libvirt`; `libvirt_uri: "qemu:///system"` (unchanged) | +| `realMode.libvirtSocketPath` = anything else (absolute) | mount its directory; `libvirt_uri: ""` in the rendered config (FR-004) | +| `realMode.libvirtSocketPath` relative or empty | refused by the schema (FR-006) | +| `secrets.existingSecret` set | no rendered config: the operator's config must name the mounted socket (FR-005, documentation only) | diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/plan.md b/docs/specs/031-realmode-dns-libvirt-fixes/plan.md new file mode 100644 index 00000000..c4196f13 --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/plan.md @@ -0,0 +1,149 @@ +# Implementation Plan: Real mode — cluster DNS under host networking, the libvirt socket, and a capabilities read that is never retried + +**Branch**: `bugfix/630` (feature id: `031-realmode-dns-libvirt-fixes`) | **Date**: 2026-11-06 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `docs/specs/031-realmode-dns-libvirt-fixes/spec.md` (cloud-storage#630) + +## Summary + +Three real-mode defects, and the silent failure behind two of them. The fixes are small and local: one template condition, one rendered config line, one parser, one dialer, one connection loop that replaces today's reconnect loop, one retried preparation, one step before every guest launch, one readiness check. **No new Go module, no new chart value, no database schema change, no migration, no API shape change.** One constitution amendment, v1.5.0, was ratified on 2026-11-06 after the user chose it at the gate. + +- **Cluster DNS (Problem 1, R1).** The shared pod spec and the purge Job render `dnsPolicy: ClusterFirstWithHostNet` beside `hostNetwork` when real mode runs with host networking, as the console proxy already does. Without host networking nothing is rendered. +- **The socket reaches the binary (Problem 2, R2).** The rendered config carries `libvirt_uri: ""` when `realMode.libvirtSocketPath` is not the default; the default renders `qemu:///system` as today. The schema requires an absolute path. +- **The binary chooses the socket (Problem 2, R3–R4).** + - `ParseLibvirtURI` accepts the system URI (automatic choice), the same with `?socket=`, or a bare absolute path. + - Anything else stops real-mode startup at configuration load, in the server, `lightstack cleanup` and `console-tokens`. + - The automatic choice is a dialer of our own (go-libvirt's `socket.Dialer` is one method). On every attempt it dials `virtqemud-sock` if present, else `libvirt-sock`, and logs which. +- **Connection (US6, R5).** + - The server's hypervisor starts unconnected and returns at once. A background loop dials — each attempt bounded at 1 s, pauses of 1 → 5 s — runs the preparation on every connection, and watches for loss. It replaces `keepConnected`/`redial`. + - Startup never waits on libvirt. + - `lightstack cleanup` keeps one synchronous attempt, bounded at 1 s like the loop's, and fails closed (FR-020a). +- **Preparation (Problem 3, R6).** + - Attempts are retried in the background after every connection until one succeeds, and once on demand per guest start while not yet prepared. Callers share one in-flight attempt (single-flight), and each is bounded at 1 s, so a guest start waits at most one bound. + - The first success is sticky; later successes refresh QEMU's owner. + - The log reports the first failure (ERROR), the success after it (INFO) and nothing louder than DEBUG in between. +- **Every launch (R7).** (*Launch* = a *guest start*, as the spec's Key Entities define it.) The four functions that reach `DomainCreate` — create (and self-heal), start, hard reboot, resume — replace their `consoleDirReady()` check with the on-demand attempt. They then make the instance's directory QEMU's immediately before `DomainCreate` (`PrepareInstanceDir`, idempotent; the domain UUID is the instance ID). +- **Readiness (R8).** A `hypervisor` check, real mode only, reads the sticky flag: `503 hypervisor unavailable` until the first preparation succeeds, then never again because of libvirt. The checker is created after the hypervisor starts. +- **Never fake (R9).** + - Real mode always has a manager, so an unreachable libvirt fails like a lost connection: a create ends `ERROR` with a `fault`, an action gets the generic `500`. + - `main.go` treats an `InitHypervisor` error as fatal in real mode. + - The create's stub branch marks `ERROR` in real mode, and every real-mode create that ends `ERROR` records a `fault` through `markInstanceError`, with a curated message (spec Clarifications, analysis session). + +Deferred from clarification and settled here: every connection attempt is bounded at 1 s — the server's in its loop and cleanup's single one (R5); concurrent preparation attempts share one in-flight call (R6); the readiness reason is `hypervisor unavailable` (R8). The analysis pass (2026-11-06) tightened the first two and added the `fault` (spec Clarifications). + +## Technical Context + +**Language/Version**: Go 1.27.3 (`go.mod`; the constitution requires 1.26+). Helm 3.12+ chart templates (Go `text/template` + Sprig), JSON Schema draft-07. Markdown. +**Primary Dependencies**: `digitalocean/go-libvirt` at the pinned `v0.0.0-20260609165003-6254771e63a8` (`socket.Dialer`, `ConnectToURI`, `Disconnected()`); `gin`; `internal/health`; `golang.org/x/sync/singleflight` (already required in `go.mod`, indirect → direct); helm-unittest. **No new module.** +**Storage**: nothing new. No database schema change, no migration. Node-local ownership only: `/console-sockets` (as today) and `/instances/` (now ensured before every launch). +**Testing**: `go test`, with database-backed tests on PostgreSQL and SQLite (`LIGHTSTACK_TEST_DB_URL`): +- the existing fake libvirtd, which speaks libvirt's wire protocol, for connection, retry and launch tests; +- real unix listeners in `t.TempDir()` for the socket choice; +- the built binaries in `cmd/lightstack/startup_refusal_test.go` for refusals; +- helm-unittest for the chart. + +Host validation follows FR-021 (a)–(e) on k3s and the openSUSE host (R12, `quickstart.md`). +**Target Platform**: Linux real mode — k3s via the chart, Docker Compose, bare metal. Stub mode is unchanged and still runs on macOS. +**Project Type**: single Go project, plus the Helm chart and docs. +**Performance Goals**: none new. The readiness probe is an atomic load. Each libvirt attempt is ≤ 1 s, background pauses 1–5 s. A guest start adds at most one bounded attempt, and only while unprepared. +**Constraints**: +- Article III fail-fast: request-path attempts ≤ 1 s, and no request waits on a loop (v1.5.0 exception). +- Article IV: no runtime stub fallback in real mode (v1.5.0). +- Article I: no API shape change. A create still answers `202` and ends `ERROR`; actions keep SPEC-025's `500`. +- SPEC-013 readiness amended for the one-time preparation only. +- The default chart render changes only by `dnsPolicy` under host networking. + +**Scale/Scope** (from the file reads in research): + +| Area | Files | +|---|---| +| Go | `pkg/hypervisor/libvirt.go` (constructors, loop, dialer use, 4 launch functions); `pkg/hypervisor/console_dir.go` (state, attempts, logging); `pkg/hypervisor/disk.go` (comment); a new `pkg/hypervisor/libvirt_uri.go` (parser and dialer); `internal/nova/handlers.go` (`InitHypervisor`, the create branch, the `fault` on every create `ERROR`, `HypervisorReady`); `cmd/lightstack/main.go` (`loadConfig` check, startup order, fatal init, readiness check). `internal/teardown` keeps its code: `NewVMManager`'s contract is unchanged | +| Tests | `pkg/hypervisor` (URI table, dialer, connection loop, preparation retry, launch ownership; fake libvirtd knobs); `internal/nova` (create and action without a connection, the fault, the manager guarantee); `test/contract/nova/no_hypervisor_test.go` (gophercloud, gated); `cmd/lightstack` (readiness checks, startup refusals, `loadConfig`); `internal/teardown` (fail-fast on a missing and on a silent socket) | +| Chart | `templates/_helpers.tpl`, `templates/teardown-purge-job.yaml`, `templates/NOTES.txt`, `values.yaml`, `values.schema.json`, `README.md`, `CONTRACT.md`; tests: a new DNS-policy suite, `libvirt_socket_mount_test.yaml` or `config_render_test.yaml`, `schema_test.yaml` | +| Docs | `docs/CONFIGURATION.md`, `SCALING.md`, `TROUBLESHOOTING.md`, `REAL_LIBVIRT_MODE.md`, `REAL_MODE_TESTING.md`, `HORIZON_INTEGRATION.md`, `blueprint.md`, `UPGRADING.md`; `CLAUDE.md` (Mode Detection, with the amendment) | +| Governance | `.specify/memory/constitution.md` → v1.5.0, ratified 2026-11-06 | + +No `NEEDS CLARIFICATION` remains: the spec's Clarifications settled scope, answers, ownership, validation, cleanup and docs, and research R1–R13 settle every design choice. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design (below the table).* + +| Principle (`.specify/memory/constitution.md` v1.4.0 at planning; v1.5.0 ratified 2026-11-06) | Verdict | Why | +|---|---|---| +| I. OpenStack Flamingo API compatibility | PASS | No endpoint, field or status is added or reshaped. A create against an unreachable libvirt answers `202` and ends `ERROR` with a `fault` — upstream's outcome when no compute can host the server — instead of a fake `ACTIVE`. Every real-mode create that ends `ERROR` now records the `fault` SPEC-025 added, which the libvirt failure path used to skip. Actions keep SPEC-025's deliberate generic `500`. Readiness is a Kubernetes endpoint; the oslo-style `/healthcheck` alias already answers `503` on a failed check. No entry for the Known Divergences register. | +| II. Synchronous operations | PASS | Server create is unchanged and stays asynchronous, as upstream's is: it answers `202 BUILD` and its goroutine does the libvirt work (`handlers.go:1097-1100`), so FR-019's `ERROR` is written there. Article II's VM-creation bullet yields to Article I under Governance rule 1 (compatibility first); this feature adds no new asynchrony. Every other operation completes before it answers. The loops accept no work from a request. A guest start makes one bounded attempt, or takes the result of the one already running — never both, never a wait for the loop. | +| III. Fail-fast | **PASS (v1.5.0)** — failed against v1.4.0 | "Never: retry loops, exponential backoff" is already contradicted on `main` by the reconnect loop of `6ed30f7`, recorded in no spec or plan. This feature extends that loop to the first connection and adds the preparation retry, which the clarified readiness gate requires (an on-demand-only retry deadlocks: no guest start reaches a not-Ready pod). Every attempt — the loop's, a guest start's, and `lightstack cleanup`'s single one — is bounded at 1 s, matching "libvirt connection failures: 1 second timeout". A guest start joins an attempt already running instead of following it with its own. Requests fail at once while the loop runs. The ratified exception (research R13) states those limits. | +| IV. Multi-mode architecture | **PASS (v1.5.0)** — failed against v1.4.0 | "Default to stub mode when dependencies unavailable" and "Mode detection: … or `svc.vmManager == nil`", read at runtime, are the fake-`ACTIVE` fallback FR-019 removes. The draft keeps stub as the default for an unconfigured mode and forbids runtime fallback from a configured real mode. Stub mode itself is unchanged and every test still runs without Linux. | +| V. Project isolation | PASS | No query or scoping change. | +| VI. Single binary | PASS | Nothing added outside the binary; no new process or service. | +| Technology constraints | PASS | No new module. `golang.org/x/sync`, already required in `go.mod`, becomes a direct dependency for `singleflight` (R6). | +| Forbidden patterns | PASS with the III exception | No queue, worker, state machine or service discovery. The two loops are the drafted exception's subject: no queue, no job records, no per-request state. | +| Test-first | PASS by plan | Each requirement's test is written first and seen failing (research R10). | +| Backwards compatibility | PASS | The default chart render changes only by `dnsPolicy`, and only under host networking. `docs/UPGRADING.md` states the behaviour changes (FR-024): an unsupported `libvirt_uri` now stops startup, an unpreparable real-mode pod is no longer Ready, host-networked names resolve through the cluster DNS, and the workarounds can go. No migration, nothing to reverse. | +| Security | PASS | No credential, token or scoping change. The readiness reason stays coarse (CONTRACT.md §2). The socket path is schema-checked as absolute. DNS queries go to the cluster DNS, as every other pod's do. | +| Quality gates before merge | PASS by plan | Both modes: stub unchanged and tested, real mode on the host (FR-021). Both database backends run the `internal/nova` and `cmd/lightstack` tests, though no SQL changes. No migration, so the up/down gate is N/A. | + +**Gate decision (2026-11-06)**: the user chose to amend the constitution rather than justify the departures in this plan. Research R13 holds the v1.5.0 text. Phase 0 and Phase 1 proceeded, and v1.5.0 was ratified on 2026-11-06 (`.specify/memory/constitution.md`); **task generation may proceed.** + +**Post-design re-check (after Phase 1)**: unchanged. +- The design adds no abstraction beyond the one dialer go-libvirt's own interface asks for, and the parser. +- The two loops collapse into one connection loop that runs the preparation, so the exception covers one place in the code. +- The contracts (`contracts/*.md`) add no API surface; they pin today's answers for the new states. +- No new divergence. Articles III and IV pass under v1.5.0, and nothing else needed amending. + +## Project Structure + +### Documentation (this feature) + +```text +docs/specs/031-realmode-dns-libvirt-fixes/ +├── spec.md # 3 clarification sessions (13 Q&As) +├── plan.md # this file +├── research.md # Phase 0: R1–R13 (R13 = constitution v1.5.0 draft) +├── data-model.md # Phase 1: configuration and in-process state (no DB entities) +├── quickstart.md # Phase 1: gates, render and binary checks, host validation (a)–(e) +├── contracts/ +│ ├── chart-render.md # dnsPolicy, libvirt_uri rendering, schema, chart docs +│ ├── libvirt-connection.md# URI forms, refusals, socket choice, server loop, cleanup, preparation +│ ├── readiness.md # /readyz in real mode +│ └── guest-operations.md # create/actions while unreachable or unprepared; ownership before launch +├── checklists/requirements.md +└── tasks.md # Phase 2 (not created here) +``` + +### Source Code (repository root) + +```text +charts/lightstack/ +├── templates/_helpers.tpl # podSpec: dnsPolicy; lightstack.config: libvirt_uri +├── templates/teardown-purge-job.yaml # dnsPolicy +├── templates/NOTES.txt # readiness wording +├── values.yaml / values.schema.json # comments, descriptions; libvirtSocketPath pattern +├── README.md / CONTRACT.md # value rows, health table, §2 readiness +└── tests/ # DNS-policy suite; libvirt_uri render; schema + +pkg/hypervisor/ +├── libvirt_uri.go # NEW: ParseLibvirtURI, the per-attempt dialer +├── libvirt.go # StartVMManager + connection loop (replaces keepConnected/redial); +│ # NewVMManager one-shot; the 4 launch functions +├── console_dir.go # sticky state, shared (single-flight) attempts, logging +├── disk.go # PrepareInstanceDir comment +└── *_test.go # URI table, dialer, loop, retry, launch ownership; fake libvirtd knobs + +internal/nova/handlers.go # InitHypervisor → StartVMManager; create branch; fault on every create ERROR; HypervisorReady +cmd/lightstack/main.go # loadConfig check; hypervisor before health.New; fatal init; hypervisor check +internal/teardown/ # test only: cleanup fails fast on a missing or silent socket +test/contract/nova/no_hypervisor_test.go # gophercloud, gated: create ends ERROR with a fault +docs/*.md, CLAUDE.md # per FR-022/023/024 and the amendment +``` + +**Structure Decision**: the existing single-project layout. The one new file, `pkg/hypervisor/libvirt_uri.go`, holds the parser and the dialer so `libvirt.go` keeps its role. No new package. `internal/common` does not learn about libvirt (R3). + +## Complexity Tracking + +Both rows are resolved by the constitution amendment (v1.5.0, research R13), not by this table. They are listed because the departures are real, and the v1.2.0 rationale requires that they be visible. + +| Violation | Why needed | Simpler alternative rejected because | +|---|---|---| +| Background retry with capped backoff (Article III) — the existing reconnect loop, extended to the first connection, plus the preparation retry | Readiness waits for the first preparation, and no guest start can reach a not-Ready pod to trigger one. A libvirtd restart must not leave real mode dead (`6ed30f7`). | Retrying only on guest start deadlocks under the readiness gate. Retrying from `/readyz` makes readiness call libvirt (FR-017). No retry at all is the defect. | +| No runtime fallback to stub (Article IV) | FR-019: a real-mode deployment must not report servers it did not create. | Keeping the fallback is the fake-`ACTIVE` defect. Failing startup on an unreachable libvirt would turn a socket-activated host's ordinary boot race into a crash loop, and contradicts the clarified retry. | diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/quickstart.md b/docs/specs/031-realmode-dns-libvirt-fixes/quickstart.md new file mode 100644 index 00000000..75e53209 --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/quickstart.md @@ -0,0 +1,57 @@ +# Quickstart: verifying SPEC-031 + +**Feature**: `031-realmode-dns-libvirt-fixes` (cloud-storage#630) | **Date**: 2026-11-06 + +An outline of how the result is checked: the gates, then the host validation of FR-021. The step-by-step host runbook (paste-safe, with teardown) is generated separately into `HOST_TESTING.md`. + +## 0. Gates + +```bash +make build && make lint +make db-up +LIGHTSTACK_TEST_DB_URL='postgres://lightstack:secret@localhost:5432/lightstack?sslmode=disable' \ + go test ./pkg/hypervisor/... ./internal/nova/... ./internal/health/... ./internal/teardown/... ./cmd/lightstack/... +LIGHTSTACK_TEST_DB_URL='sqlite:///tmp/ls-test/t.db' \ + go test ./internal/nova/... ./cmd/lightstack/... +helm unittest charts/lightstack +# contract suite: against a running stub Lightstack, once on PostgreSQL and once on SQLite +make test-contract +``` + +Expected: +- the URI table, the socket-choice tests, the background-connection and preparation-retry tests, the per-launch ownership tests, the readiness check, the startup refusals, the cleanup fail-fast tests and the chart suites pass on both backends; +- the existing suites, the contract suite included, stay green (SC-008). + +The no-hypervisor contract case runs separately, against the §2 server: `LIGHTSTACK_CONTRACT_NO_HYPERVISOR=1 go test ./test/contract/nova/ -run NoHypervisor`. + +## 1. Render checks (no cluster needed) + +```bash +helm template t charts/lightstack --set image.repository=x --set secrets.jwtSecret=j --set secrets.adminPassword=p \ + --set database.url=postgres://u@db.ns.svc/l --set realMode.enabled=true --set realMode.hostNetwork=true \ + | grep -n 'dnsPolicy\|hostNetwork' +``` + +Expected: `hostNetwork: true` followed by `dnsPolicy: ClusterFirstWithHostNet` on the Deployment. + +Add `--set dataRetentionPolicy=delete` to see the purge Job carry the same pair. Add `--set realMode.libvirtSocketPath=/run/libvirt/virtqemud-sock` and look at the config Secret's `libvirt_uri` (C-SOCK-1). Without it, `libvirt_uri: "qemu:///system"` (C-SOCK-2). + +## 2. Binary checks (no libvirt needed) + +- Real mode with `nova.libvirt_uri: "qemu+tcp://h/system"` → the server exits 1 and `lightstack cleanup` exits 2, each naming the value and the accepted forms (C-URI-2). With `libvirt_mode: stub` the same config starts. +- Real mode with libvirt absent → the server starts, logs failed connection attempts naming both sockets, and `/readyz` answers `503 hypervisor unavailable` (C-READY-3). +- A `server create` against it ends `ERROR` with a `fault` (C-GUEST-1). The message names the step that failed first: an image boot on a host without the image base fails at disk staging, a boot from an existing volume reaches the hypervisor ("the hypervisor is not ready to start guests"). This is the server the gated contract case (§0) runs against. + +## 3. Host validation (FR-021, research R12) + +Single-node k3s on the openSUSE host: `virtqemud` socket-activated, no `libvirtd`, no `virtproxyd`, no symlink. Chart in real mode with `realMode.privileged` and `realMode.hostNetwork`. + +| Step | Do | Expect | +|---|---|---| +| (a) DNS | `database.url` naming the PostgreSQL Service as `..svc`; `dataRetentionPolicy: delete` | the pod becomes Ready; `helm uninstall` completes (the purge Job resolves the name) | +| (b) modular host, defaults | install with default `realMode.libvirtSocketPath`; create a server | the log names `/var/run/libvirt/virtqemud-sock`; the server is `ACTIVE` with a guest in `virsh list`; the delete uninstall destroys it | +| (c) cold path | before installing: `systemctl stop virtqemud`, empty `/var/cache/libvirt/qemu/capabilities/` | the log shows the failed first capabilities read, then the success line (directory, uid:gid); Ready; the first server boots `ACTIVE`; no restart. A fast host that answers within 1 s anyway is recorded as "retry not exercised" | +| (d) unreachable path | stop the `virtqemud` socket units (their `RemoveOnStop=yes` removes the socket), then install | not Ready, `503 hypervisor unavailable`; a create on the node's own address (`localhost:8774`) ends `ERROR`. Start the socket units again: the log shows the connection and the socket; Ready; a server boots `ACTIVE`; no restart | +| (e) monolithic | stop the modular socket units, start `libvirtd.socket`, reinstall | the log names `/var/run/libvirt/libvirt-sock`; a server boots `ACTIVE` | + +Afterwards, restore the host's daemon setup and remove any leftover workaround: the symlink, `virtproxyd.socket`, the warm-up. diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/research.md b/docs/specs/031-realmode-dns-libvirt-fixes/research.md new file mode 100644 index 00000000..2e9a36dc --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/research.md @@ -0,0 +1,210 @@ +# Research: Real mode — cluster DNS under host networking, the libvirt socket, and a capabilities read that is never retried + +**Feature**: `031-realmode-dns-libvirt-fixes` (cloud-storage#630) | **Date**: 2026-11-06 +**Method**: read the code this branch is based on (`d81f081`; the line numbers below refer to it): `charts/lightstack` (templates, values, schema, tests), `pkg/hypervisor` (connection, console directory, disk, fake libvirtd tests), `internal/nova` (create, actions, self-heal, console), `internal/health`, `internal/teardown`, `cmd/lightstack`; the pinned go-libvirt module (`v0.0.0-20260609165003-6254771e63a8`); the libvirt and Kubernetes documentation the spec cites; and the openSUSE host's own systemd units. + +Each item: **Decision / Rationale / Alternatives**. + +--- + +## R1 — Cluster DNS for host-networked pods + +**Decision**: Render `dnsPolicy: ClusterFirstWithHostNet` next to `hostNetwork` in `lightstack.podSpec` (`_helpers.tpl:411-413`, shared by the Deployment and the DaemonSet) and in the purge Job (`teardown-purge-job.yaml:36-38`), only when `realMode.enabled` and `realMode.hostNetwork` are both true. The pod spec already renders `hostNetwork: ` in real mode, `false` included, so the condition is on the value, not on real mode. + +**Rationale**: For a host-networked pod Kubernetes demotes `ClusterFirst` to the node's resolver; `ClusterFirstWithHostNet` is the documented policy for exactly this case, and the console proxy already uses it (`console-proxy.yaml:42-43`). Where the kubelet has no cluster DNS configured it falls back to the node's resolver and records `MissingClusterDNS`, so no cluster loses resolution it had. The node's `/etc/hosts` keeps applying to host-networked pods; other names go through the cluster DNS, which forwards upstream (CoreDNS in k3s forwards to the node's resolvers). Guests are untouched: dnsmasq serves DHCP only (`port=0`, `pkg/networking/dhcp.go:213`). + +**Alternatives**: a `dnsPolicy` or `dnsConfig` chart value (spec Out of Scope; nothing needs a choice); `dnsPolicy: None` with an explicit nameserver (the cluster DNS address differs per cluster); `hostAliases` (static, breaks when a Service is re-created — the defect itself). + +## R2 — The chart passes a non-default socket to the binary + +**Decision**: In `lightstack.config` (`_helpers.tpl:253`), render `libvirt_uri` as the bare `realMode.libvirtSocketPath` when it differs from `/var/run/libvirt/libvirt-sock`, else `"qemu:///system"` as today. The default is compared in one place. The schema gains `"pattern": "^/.+"` for `realMode.libvirtSocketPath`, as `realMode.hostStoragePath` already has. The rendered config is the Secret both the Lightstack pod and the purge Job mount, so one change reaches both. + +**Rationale**: A bare absolute path is a form the binary has always taken and keeps taking (FR-008), the form real mode under Docker Compose on openSUSE already uses, and it needs no URL escaping. Rendering it only when non-default keeps the default render byte-identical (spec FR-004, the ticket's acceptance criterion) and leaves the default meaning "let the binary choose" (R4). With `secrets.existingSecret` the chart renders no config at all, so only documentation can reach that case (FR-005). + +**Alternatives**: render `qemu+unix:///system?socket=` (equivalent, needs escaping, gains nothing); always render the path (the default would then force `libvirt-sock` and defeat R4 on modular hosts, and change the default render); a new `realMode.libvirtUri` value (spec: no new value). + +## R3 — The `libvirt_uri` grammar, and where it is refused + +**Decision**: One pure function in `pkg/hypervisor`, `ParseLibvirtURI(string) (target, error)`, parsed with `net/url`: + +| Form | Meaning | +|---|---| +| `""`, `qemu:///system`, `qemu+unix:///system` | automatic socket choice (R4) | +| either URI with `?socket=` | that socket, exactly | +| `` (no scheme) | that socket, exactly (kept) | +| anything else | refused | + +Refused, each with one message naming the value and the accepted forms: another driver (`lxc://`, `xen://`, `test://`), a session or embedded URI (`qemu:///session`, `qemu:///embed`), any host or remote transport (`qemu://host/system`, `qemu+tcp://`, `qemu+ssh://`, `qemu+tls://`, `qemu+libssh2://`, `qemu+ext://`), a relative path, a relative `socket=`, a repeated `socket=`, any other parameter, a fragment. + +It is called from `cmd/lightstack`'s `loadConfig` when `nova.libvirt_mode` is `real`. Every entry point already goes through `loadConfig`: the server (exit 1), `lightstack cleanup` (exit 2) and `lightstack console-tokens` (exit 2) — the exit codes `startup_refusal_test.go` already pins for refused configurations. The manager constructors call it again, so no path dials a refused value. Stub mode never dials libvirt and does not check the value. + +**Rationale**: `connectLibvirt` (`libvirt.go:98-113`) uses any value other than `qemu:///system` verbatim as a socket path, so every form `docs/CONFIGURATION.md:436-440` advertises — session, TCP, SSH — has only ever failed as "no such file". A configuration error belongs at configuration load, like the existing real-mode checks there. `internal/common` stays free of the hypervisor package: importing it would pull go-libvirt into every package that reads the config. `pkg/hypervisor` imports only `pkg/paths`, so the parser adds no cycle. + +**Alternatives**: validate in `internal/common.LoadConfig` (acyclic, but makes the config package depend on the hypervisor package); honour libvirt's `mode=legacy|direct` parameter (no requirement for it; the explicit socket already forces either daemon); keep the verbatim fallback (the defect). + +## R4 — Choosing the socket on every attempt + +**Decision**: A dialer of our own, implementing go-libvirt's one-method `socket.Dialer` (`Dial() (net.Conn, error)`, `socket/socket.go:60-62`). For the automatic form it checks `/var/run/libvirt/virtqemud-sock`, then `/var/run/libvirt/libvirt-sock`, with `os.Stat` on **every** `Dial`, dials the first that exists, and remembers which one, for the log line FR-010 requires. Neither present → an error naming both. An explicit socket dials that path only. The run directory is a constant that tests can point at a temporary directory. + +**Rationale**: +- go-libvirt asks the dialer for a connection on every connect and reconnect, so a per-`Dial` choice is what makes FR-007's "made at every connection attempt" hold with no further machinery. +- `os.Stat` follows symlinks, so a host still carrying the `libvirt-sock → virtqemud-sock` workaround resolves to the same daemon. +- Stale socket files are not a practical risk: the openSUSE host's `virtqemud.socket` unit has `ListenStream=/run/libvirt/virtqemud-sock` and `RemoveOnStop=yes`, so a stopped socket unit leaves no file behind (`systemctl cat virtqemud.socket`). +- The QEMU daemon's socket is sufficient: Lightstack calls only `Domain*`, `ConnectGetCapabilities` and `ConnectGetLibVersion` — no network, storage, node-device or secret API (grep of `pkg/`, `internal/`, `cmd/`). +- libvirt's own daemon page says its RPC client "still prefers connecting to the monolithic daemon". Lightstack deliberately prefers the modular socket when it exists. Where `libvirt-sock` is `virtproxyd`'s, both reach the same daemon; a host running both daemon kinds is the misconfiguration the spec's edge case names. + +**Alternatives**: choose once at startup (a daemon whose socket appears later would never be found — the reason FR-007 exists); prefer `libvirt-sock` (the defect on modular hosts); fall back to `libvirt-sock` when dialing `virtqemud-sock` fails (silently switches daemons and hides a broken one). + +## R5 — Connecting: the server in the background, `lightstack cleanup` once + +**Decision**: +- **Server**: a new constructor, `StartVMManager(uri, mode)`, returns at once in real mode. A background loop owns the connection: + 1. Dial with the R4 dialer. Each attempt — dial plus handshake — is bounded at 1 second. On timeout the attempt closes the connection its dialer returned, which unblocks go-libvirt; a connection the dialer returns after the timeout is closed as it arrives. + 2. Back off between attempts with the existing `reconnectBackoffMin`/`reconnectBackoffMax` (1 s → 5 s). + 3. Once connected, publish the connection through the existing atomic pointer, log the socket, and run the preparation (R6). + 4. Watch `Disconnected()` and start again. This subsumes today's `keepConnected`/`redial`. + + Startup no longer waits on libvirt; readiness covers the gap (R8). +- **`lightstack cleanup`**: `NewVMManager` keeps its contract — one synchronous attempt with the R4 socket choice, an error if it fails (FR-020a) — and that attempt is bounded exactly as the loop's are: 1 second for dial plus handshake, through the same helper. A missing socket fails at the dial at once. A socket-activated `virtqemud` that has exited since the quiesce hook stopped Lightstack (`--timeout 120`) is started by the attempt itself; if it has not answered within the bound, the purge fails closed and re-running the uninstall finds it warm (spec Clarifications, analysis session). + +**Rationale (go-libvirt, read)**: `ConnectToURI` (`libvirt.go:286-308`) dials, then runs `initLibvirtComms` — auth list and connect-open RPCs — with no deadline; today's dialer timeout (2 s) covers only the dial, so the bound is imposed around the whole connect, by closing the connection the dialer returned. `Disconnected()` returns a channel created only at the end of a successful connect, so a never-connected `*Libvirt` must never be watched; the loop keeps `m.conn` nil until a connect succeeds. Of the 15 `m.conn.Load()` call sites, 14 already handle nil: 13 return "not connected to libvirt" and `Close` skips it. The 15th is the reconnect loop's own watch (`libvirt.go:166`), which would panic on nil, and this loop is the one being rewritten. A never-connected manager therefore fails exactly as a disconnected one does, with no change at the other 14 sites. A reconnect builds a fresh `*Libvirt`, as `6ed30f7` does: re-opening one in place races in-flight calls. + +**Alternatives**: keep the server's first attempt synchronous (a wedged daemon blocks startup, as it can today); leave any attempt unbounded — the loop's (a stuck attempt never retries) or cleanup's (a wedged daemon holds the uninstall until the Job's deadline) — against Article III's 1-second libvirt connection timeout. + +## R6 — Preparing the console socket directory: retried, shared, sticky + +**Decision**: The manager keeps: +- `prepared` — set by the first success, never cleared; +- the QEMU owner — replaced by every success; +- the directory: `paths.ConsoleSocketDir()`, the one the domain XML names; +- single-flight (`golang.org/x/sync/singleflight`): every attempt — the loop's or a guest start's — goes through one `Do`, so callers that arrive while an attempt runs share its result. + +Attempts are made in two places: +- **Background**: after every connection, the first and each reconnect, the loop makes attempts with the same 1 → 5 s backoff until one succeeds, even when an earlier success is in effect (FR-015). A failed redo changes nothing that is in effect. +- **On demand**: a guest start that finds `prepared` false joins the attempt already running, or starts one if none is, and takes its result — one bounded attempt either way, never a wait followed by a second attempt (Article III v1.5.0; FR-013, FR-016). + +Every attempt is today's `prepareConsoleSocketDir` under `withDeadline` (1 s, `console_endpoint.go:19`). Logging (FR-017a): + +| Event | Level | +|---|---| +| First failure, with its cause | ERROR (today's line) | +| An attempt with no connection to make it on | DEBUG: the connection loop reports the outage (review, 2026-11-16) | +| Success after a failure — directory, QEMU's uid:gid | INFO | +| A redo after a reconnect | DEBUG, or INFO if QEMU's user changed | +| Further retries | DEBUG | + +**Rationale**: The clarified readiness gate needs a retry that does not depend on a guest start. No guest start reaches a not-Ready pod through its Service, so on-demand-only retrying, the ticket's minimum, would deadlock. A capabilities call that outlives its bound finishes in the background (`withDeadline` lets it), warming the daemon for the next attempt. Sharing the in-flight result is what keeps a guest start within one bound. `golang.org/x/sync/singleflight` does exactly that and is already required in `go.mod` (indirect, v0.23.0): it becomes a direct dependency, not a new module. + +**Alternatives**: retry only on guest start (the deadlock above); let `/readyz` drive attempts (readiness would call libvirt; FR-017 forbids it); a mutex with a re-check (a start could wait for one attempt and then make another — up to 2 s, against the v1.5.0 exception and FR-016); lengthen the 1-second bound (Article III; the 2026-10-14 host run that found this defect pointed to retrying, not lengthening). + +## R7 — One step before every guest launch + +*Launch*, here and in the plan and tasks, means a *guest start* as the spec's Key Entities define it. + +**Decision**: Every QEMU launch goes through `DomainCreate` in four functions — `createVMReal` (create and self-heal, both via `CreateVM`), `startVMReal`, `hardRebootVMReal`, `resumeVMReal`. These are the four existing `consoleDirReady()` call sites (`libvirt.go:224, 373, 535, 787`). In each: +- replace `consoleDirReady()` with `ensurePrepared(ctx)` — FR-013's on-demand attempt, refusing with FR-016's existing named error when it fails; +- call `PrepareInstanceDir()` (`disk.go:41-59`: `MkdirAll` plus a chown to the recorded owner, idempotent) immediately before `DomainCreate`. + +In `createVMReal` the step runs after define, using the defined domain's UUID, and a failure undefines the domain on the existing path. `hardRebootVMReal` has no context and uses a background one; the attempt is bounded by `withDeadline` anyway. The `disk.go` comment claiming a chown-less directory "never gets used" goes, since a retried preparation makes that false. + +**Rationale**: the domain UUID is the instance ID (`UUID: instanceID`, `internal/nova/handlers.go:1338`; `selfheal.go:73`), so `InstanceDir(uuid)` is the instance's directory. Self-heal calls `CreateVM` without `PrepareInstanceDir` (`selfheal.go:200`), and directories from releases before the ownership fix are root-owned. Placing the step in the four launch functions covers both with no change in `internal/nova`, and keeps FR-014's ordering — preparation before ownership — inside one function. + +**Alternatives**: add the step at each `internal/nova` call site (create, self-heal, start, reboot, resume across several files — easy to miss one); chown only at create (the clarification rejected it); read the UUID out of the XML before define (an extra parse to avoid a path that already exists). + +## R8 — Readiness waits for the first preparation (real mode only) + +**Decision**: A `hypervisor` check (`health.Check`) whose probe reads the manager's sticky `prepared` — never a libvirt call — registered only when `libvirt_mode` is `real`. `health.New` moves from `main.go:109` to after the hypervisor starts; nothing uses the checker in between (its uses: `main.go:371, 415, 436`). The existing reason format then reads `hypervisor unavailable`, or `database,hypervisor unavailable`. `/healthcheck` mirrors readiness already (`TestHealthcheck_MirrorsReadiness`). + +**Rationale**: FR-017 as clarified: not-ready until the first success, then never again because of libvirt. That keeps SPEC-013 FR-001's rule for outages and adds no error detail (CONTRACT.md §2 promises a coarse reason). The 800 ms per-check timeout (`health.go`, `defaultCheckTimeout`) is never at stake: the probe is an atomic load. + +**Alternatives**: a `Checker.AddCheck` method (unneeded once the checker is created later); a separate endpoint (Kubernetes reads only the probes, and the clarification chose readiness); a live libvirt call (flaps on every outage; FR-017 forbids it). + +## R9 — Real mode never fakes a server + +**Decision**: +- `nova.Service.InitHypervisor` uses `StartVMManager`, so real mode always has a manager, connected or not. An unreachable libvirt fails each call as a lost connection does: the create goroutine marks the server `ERROR` (`handlers.go:1355-1366`) — now through `markInstanceError` (`handlers.go:650`), so the `fault` SPEC-025 added is recorded — and an action answers `sendLifecycleError`'s generic `500` (`handlers.go:2758-2766`). A guest start before the first preparation meets FR-016's named refusal (R7). +- In real mode `main.go` treats an `InitHypervisor` error as fatal. The only error left is an unsupported URI, which `loadConfig` has already refused. +- The create's stub branch (`handlers.go:1390-1398`) marks `ERROR` in real mode instead of `ACTIVE`, also through `markInstanceError`. That is now reachable only in tests that run real mode without a manager. +- The fault's message names the step that failed, curated as `allocationFault` does for networks; the detail stays in the log. For the libvirt create, `createFault`: `ErrConsoleSocketDirNotReady` (which also covers a missing connection) → "the hypervisor is not ready to start guests", anything else → "the hypervisor could not create the guest". The no-hypervisor branch reuses the first. The other fault-less `ERROR` writes in `CreateServer` — per-instance directory, root disk staging, image base, invalid `imageRef`, panic — get their own messages, listed in `contracts/guest-operations.md` C-GUEST-1. Every real-mode create that ends `ERROR` thus records a fault, closing the gap where a libvirt failure left none (spec Clarifications, analysis session). +- The other 12 nil-manager branches in `internal/nova` stay: real mode cannot reach them in production — `InitHypervisor` always leaves a manager in real mode, which a test now pins — and the tests that model a missing hypervisor keep their behaviour. + +**Rationale**: FR-019 as clarified — the lost-connection answer, no new HTTP status or action message — plus the fault the analysis session added. Making the manager unconditional removes the fake path by construction. Hardening only the create branch keeps tests that model a missing hypervisor honest: `TestConsoleWithoutAHypervisorIsAnOutage` and the state reconciler's "blind" service keep their assertions. + +**Alternatives**: harden all 12 other branches (no production path reaches them, and the guarantee is tested); answer `503` (rejected in clarification: upstream compute does not, Article I); leave the fault out (an `ERROR` with no reason, unlike upstream's). + +## R10 — Test strategy (Test-first; real protocol over mocks) + +**Decision**: RED first, per Development Workflow. The strategy below is reviewed with the user before the first test is written (step 2, "get approval"). +- **`pkg/hypervisor`**: + - a table test for `ParseLibvirtURI` covering every accepted and refused form (FR-011); + - the R4 dialer against real unix listeners named `virtqemud-sock`/`libvirt-sock` in `t.TempDir()`: only one, the other, both, neither, and one appearing between attempts (FR-007, FR-010); + - `StartVMManager` against the existing fake libvirtd: a first dial that fails and a later one that succeeds, with no restart (FR-020); a handshake the fake never answers is abandoned after 1 s and retried (R5); `NewVMManager` against a daemon that never answers fails after ≈1 s (R5); + - the preparation: the fake delays its first `ConnectGetCapabilities` reply past 1 s, then answers at once → prepared with no restart, and the next start launches with its directory chowned (FR-018); `prepared` survives a dropped connection; a redo after a reconnect replaces the owner; a guest start arriving while an attempt runs takes that attempt's result within one bound (R6); + - waits use an explicit deadline (`waitForWithin`, 10 s): the fake helpers' `waitFor` gives up after 2 s, too close to a 1.5 s first reply plus a retry; + - each of the four launch functions chowns a root-owned directory before `DomainCreate` (FR-018, second sentence), observed without root through the existing `otherOwner` helper (`disk_test.go`). +- **`internal/nova`** (database-backed, both backends): real mode with an unconnected manager — `StartVMManager` given an explicit socket path that does not exist, since `internal/nova` reaches neither the fake libvirtd nor the run directory — booting from an existing volume under a temporary state path, so the create reaches `CreateVM` (an image boot would fail at disk staging first) → create ends `ERROR` with the hypervisor fault, and an `ACTIVE` server's `os-stop` answers the generic `500` (FR-019); `createFault`'s mapping has its own pure test; real mode without a manager → `ERROR` with a fault, not `ACTIVE`; real-mode `InitHypervisor` with no socket returns no error and a manager (the guarantee R9 relies on); `HypervisorReady` errors without a prepared manager — the prepared and sticky cases are `pkg/hypervisor`'s. +- **`test/contract/nova`**: a gophercloud test against a real-mode Lightstack whose libvirt socket is absent — a create ends `ERROR` with a fault, never `ACTIVE` (FR-019; Development Workflow step 1, contract tests with OpenStack clients). Gated by `LIGHTSTACK_CONTRACT_NO_HYPERVISOR=1` and skipped otherwise, as SPEC-025 ran its real-mode contract cases on the host. +- **`internal/health` / `cmd/lightstack`**: the `hypervisor` check — reason while unprepared, ready after, ready through a later loss, absent in stub mode (FR-018a). +- **`cmd/lightstack/startup_refusal_test.go`**: an unsupported `libvirt_uri` in real mode → server exit 1, cleanup and console-tokens exit 2, each with the message; the same value in stub mode starts (FR-009). +- **`internal/teardown`**: real mode with a socket that does not exist → `Run` fails at once, with no dialing afterwards; a socket that accepts and never answers → `Run` fails within ≈1 s (FR-020a). +- **`charts/lightstack/tests`**: `dnsPolicy` on the Deployment, the DaemonSet and the purge Job, and its absence in stub mode and in real mode without host networking (FR-003); `libvirt_uri` rendered for a non-default socket and unchanged for the default (FR-004); the schema refusing a relative path (FR-006). + +The fake libvirtd gains a configurable domain UUID with a `DomainDefineXML` reply, a delay before the first capabilities reply, and a mode that never answers the handshake. + +**Rationale**: the fake speaks libvirt's real wire protocol and echoes serials, which is what makes reconnect tests possible: go-libvirt's `libvirttest` mock cannot follow a client across a reconnect, as `fake_libvirtd_test.go` records. Real unix listeners exercise `os.Stat` and `net.Dial` as production does. Chown is observed without root, as the existing ownership tests do. + +**Alternatives**: mocks of the manager in `internal/nova` (would not exercise the nil-guard and lost-connection paths that make FR-019 hold); host-only verification of the retry (not deterministic — R12). + +## R11 — Documentation the requirements name + +**Decision**: These files change, per FR-003a, FR-005, FR-022, FR-023 and FR-024: +- **chart** — `charts/lightstack/README.md` (the `libvirtSocketPath` row per FR-005, the `hostNetwork` row, the health table), `values.yaml` (the `realMode` and `probes` comments), `values.schema.json` (the `libvirtSocketPath`, `hostNetwork` and probe descriptions), `templates/NOTES.txt`, `CONTRACT.md` §2; +- **`docs/`** — `CONFIGURATION.md` (the `libvirt_uri` forms), `SCALING.md` (drop `qemu+tcp://`), `TROUBLESHOOTING.md`, `REAL_LIBVIRT_MODE.md`, `REAL_MODE_TESTING.md`, `HORIZON_INTEGRATION.md` (name `virtqemud-sock` beside `libvirt-sock`), `blueprint.md` (readiness), `UPGRADING.md` (five notes). + +`CLAUDE.md`'s "Mode Detection" idiom (`svc.vmManager == nil` as stub) follows the Article IV amendment (R13). + +**Rationale**: each file states something this feature makes false. The `NOTES.txt` change is text only; `tests/notes_test.yaml` asserts nothing about readiness (grep). + +**Alternatives**: none worth recording — the spec names the files. + +## R12 — Host validation that exercises the retry paths + +**Decision**: The FR-021 steps: + +| Step | How it is forced | It passes when | +|---|---|---| +| (c) cold | `systemctl stop virtqemud`, then empty `/var/cache/libvirt/qemu/capabilities/`, before Lightstack starts | the log shows a failed first read and a later success. If a fast host still answers within 1 s, the step records "retry not exercised", and FR-018's unit test is the proof | +| (d) unreachable | stop the `virtqemud` socket units, which `RemoveOnStop=yes` makes remove the socket, then start Lightstack; create a server on the node's own address, since the pod is not Ready and its Service has no endpoint | the create ends `ERROR`; after the socket units are started again, the log shows the later connection and the socket dialed, the pod becomes Ready, and a VM boots | +| (e) monolithic | stop the modular socket units and start `libvirtd.socket` — enabled but inactive on the openSUSE host | the log names `libvirt-sock` and a VM boots | + +**Rationale**: a warm daemon answers capabilities in about 0.04 s and caches QEMU's capabilities on disk, so a plain reboot may never hit the slow path. The cache directory exists on the host (root-only, `ls` → permission denied for the user). + +**Alternatives**: a fresh boot only (may pass without exercising the fix — the clarification rejected it). + +## R13 — Constitution v1.5.0 (ratified 2026-11-06) + +Ratified in `.specify/memory/constitution.md` with two refinements of the draft below. Article III's first limit also lets a request "wait for one [attempt] already running", which R6's shared (single-flight) preparation attempts need. Article IV's Mode detection names `svc.networkingMode` beside `svc.libvirtMode`, since the article covers every service. The constitution file is authoritative. + +The plan's Constitution Check finds two articles contradicted by the clarified behaviour, and the user chose amendment over a plan-only justification (2026-11-06), following the v1.1.0–v1.4.0 precedent. Draft text: + +**Article III — add after "Never: …":** + +> **Permitted exception — re-establishing a dependency in the background** (added in v1.5.0). A background loop that restores the process's own connection to an external dependency, or completes a one-time preparation that depends on it, does not violate this principle when all of the following hold: +> - No request waits on the loop. While the dependency is unreachable or the preparation incomplete, a request that needs it fails at once, as it would without the loop; it may make at most one attempt of its own, under the dependency's bound. +> - Every attempt is bounded as the dependency's calls are — 1 second for libvirt. +> - The pause between the loop's attempts is capped (1 to 5 seconds for libvirt), and the loop stops with the process. +> - The loop carries no queue, no job records and no per-request state, and accepts no work from a request. +> +> **Rationale**: the libvirt reconnect loop merged with `6ed30f7` (2026-09-24) — without it a libvirtd restart left real mode dead until the pod was restarted — contradicted "Never: retry loops, exponential backoff" without any spec or plan recording it. SPEC-031 extends it to the first connection and to the console socket directory preparation that real-mode readiness waits for. What the article protects is unchanged: a request still gets immediate feedback. + +**Article IV — replace the "Mode detection" and "Default to stub mode" bullets:** + +> - **Mode selection**: the configuration chooses the mode; an unset mode defaults to stub. +> - **Mode detection**: check `svc.libvirtMode == "stub"`. A nil `svc.vmManager` means no hypervisor was wired in (stub-mode tests); it never stands for real mode. +> - **No runtime fallback**: a configured real mode never falls back to stub. An unreachable dependency is an outage — the operations that need it fail as they do when it is lost later — never fake data. +> +> **Rationale**: read at runtime, "default to stub mode when dependencies unavailable" is how a real-mode deployment whose libvirt was unreachable at startup reported servers `ACTIVE` with no guest behind them (SPEC-031 Context, `internal/nova/handlers.go:1390-1398`), while the console path already treated the same state as an outage (`errNoHypervisor`). Stub mode stays the default when no mode is configured, which is what the rule was for. + +**Runtime guidance to update with the amendment**: `CLAUDE.md` "Mode Detection" (`if svc.libvirtMode == "stub" || svc.vmManager == nil`). The known mismatch between `CLAUDE.md`'s "Nine Articles" and this constitution, flagged since v1.2.0, is unchanged by this amendment. + +**Migration plan for the amendment**: none. Article III's exception records behaviour already shipped; Article IV's change ships with SPEC-031 and is documented in `docs/UPGRADING.md` (FR-024). diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/spec.md b/docs/specs/031-realmode-dns-libvirt-fixes/spec.md new file mode 100644 index 00000000..bfc52951 --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/spec.md @@ -0,0 +1,290 @@ +# Feature Specification: Real mode — cluster DNS under host networking, the libvirt socket, and a capabilities read that is never retried + +**Feature Branch**: `bugfix/630` (feature id: `031-realmode-dns-libvirt-fixes`) +**Created**: 2026-11-06 +**Status**: Draft +**Input**: cobaltcore-dev/cloud-storage#630 — "[BUG] - lightstack - Real mode - cluster DNS under hostNetwork, the libvirt socket path, and a one-shot capabilities read" + +## Context + +Three real-mode defects. They surfaced while planning a consumer that installs the chart in single-node real mode on k3s (`realMode.enabled`, `realMode.privileged` and `realMode.hostNetwork` all `true`), and while running real mode on a host with libvirt's modular daemons (openSUSE Tumbleweed: `virtqemud` socket-activated, no `libvirtd`). Each has a workaround, on the host or in the consumer, and each forces every real-mode consumer into it. Problems 2 and 3 block real mode on modular-daemon hosts until the workaround is in place. Paths and line numbers refer to `d81f081`, the `main` commit this branch is based on. + +**Problem 1 — host-networked pods do not use cluster DNS.** The shared pod spec (`lightstack.podSpec`, `charts/lightstack/templates/_helpers.tpl:411-413`), used by the single-node Deployment and the multi-node DaemonSet, sets `hostNetwork` and no `dnsPolicy`. The teardown purge Job does the same (`charts/lightstack/templates/teardown-purge-job.yaml:36-38`). For a host-networked pod Kubernetes treats the default `ClusterFirst` like `Default`: the pod uses the node's resolver, and `..svc` does not resolve. Every in-cluster endpoint Lightstack is given must therefore be a cluster IP or a node address: a PostgreSQL URL (in `database.url` or `database.existingSecret` — an operator-managed PostgreSQL such as CloudNativePG's `-rw` Service is the common case, and the purge Job reads the database too), `glance.s3.endpoint` for an in-cluster RGW, `cache.redisUrl` for an in-cluster Redis. A cluster IP changes when its Service is re-created, and every consumer then renders the URL again. The console proxy already sets `dnsPolicy: ClusterFirstWithHostNet` (`charts/lightstack/templates/console-proxy.yaml:42-43`). A SQLite URL is not affected: that database is a node-local file. + +**Problem 2 — `realMode.libvirtSocketPath` only picks what is mounted.** The value decides which host directory the pods mount (`_helpers.tpl:584,614`, `teardown-purge-job.yaml:60,78`). The rendered config always says `libvirt_uri: "qemu:///system"` (`_helpers.tpl:253`), and for that URI the binary dials the fixed path `/var/run/libvirt/libvirt-sock` (`connectLibvirt`, `pkg/hypervisor/libvirt.go:98-113`). Setting the value to another socket mounts the right directory and still dials `libvirt-sock`. The binary does take a socket path — any other `libvirt_uri` is used verbatim as one (`libvirt.go:101-107`), which is how real mode runs under Docker Compose on openSUSE (`nova.libvirt_uri: "/run/libvirt/virtqemud-sock"`) — but the chart offers no way to set it. The comment there says the scheme is stripped from a URI. It is not, so a real URI such as `qemu+unix:///system?socket=/run/libvirt/virtqemud-sock` fails as a nonexistent path, and so does every session and remote form `docs/CONFIGURATION.md:436-440` lists as an option. + +Hosts running the modular daemons — the default on Fedora and openSUSE Tumbleweed — have no `libvirt-sock` unless `virtproxyd.socket` is enabled. The workarounds are on the host: enable `virtproxyd.socket`, or symlink `libvirt-sock` to `virtqemud-sock` in tmpfs, gone after every reboot. Lightstack calls only libvirt's domain and connect APIs, all served by the QEMU daemon, so that daemon's own socket is sufficient. + +What follows a failed connection is worse than a refusal. `InitHypervisor` failing is logged as a warning and startup continues with no hypervisor (`cmd/lightstack/main.go:155-157`). `server create` then takes the stub branch (`internal/nova/handlers.go:1390-1398`) and reports the server `ACTIVE` with no guest behind it, while `/readyz` stays green. The console path already treats the same state as an outage (`errNoHypervisor`, `internal/nova/console.go:226-228`). The purge Job cannot connect either, so a `dataRetentionPolicy: delete` uninstall fails (`internal/teardown/teardown.go:80-83`). + +**Problem 3 — one startup read of libvirt's capabilities, never retried.** At real-mode startup `PrepareConsoleSocketDir` reads libvirt's capabilities once (`cmd/lightstack/main.go:164`, `pkg/hypervisor/console_dir.go:53`) under the 1-second `consoleReadTimeout` (`pkg/hypervisor/console_endpoint.go:19`). If the read fails, the error is recorded and every guest start is refused with `console socket directory not ready` (`console_dir.go:67-77`). Nothing reads again — not a reconnect, not a later guest start — so the refusal lasts until the process restarts, although the startup log says starts are refused "until it is" ready. With socket activation the first call may have to start `virtqemud` and probe QEMU, which can take longer than a second. Lightstack then refuses every guest start while `/readyz` stays green: Kubernetes neither restarts the pod nor marks it unready, and an orchestrator sees a Ready Lightstack that cannot boot VMs. The workaround today: warm libvirt (`virsh -c qemu:///system capabilities`) before Lightstack starts. + +The same read caches the user QEMU runs as. Each guest's per-instance directory is chowned to that user so QEMU can write the guest's `console.log`. While the user is unknown, `PrepareInstanceDir` (`pkg/hypervisor/disk.go:41-59`) creates the directory root-owned and relies on the refusal to make that harmless. A retry that succeeds lifts the refusal, so it must not leave a root-owned directory behind it. + +Guests are not affected by the DNS change: the per-network dnsmasq serves DHCP only (`port=0`, `pkg/networking/dhcp.go:213`) and hands guests the subnet's `dns_nameservers`, not the pod's resolver. + +## Clarifications + +### Session 2026-11-06 + +- Q: The console socket directory preparation now retries until it succeeds. How does "guests cannot start yet" become visible? The issue proposes the readiness probe "or at least a log line", and SPEC-013 FR-001 rules that libvirt never affects readiness. → A: **Readiness waits for the one-time preparation, in real mode only.** `/readyz` reports not-ready until the directory has been prepared once. From then on nothing about libvirt affects readiness again — an outage, a reconnect or a failed redo does not — so SPEC-013 FR-001 keeps holding for outages. Until then the pod is out of the Service (on a single node: all six APIs), and the startup probe restarts it if the preparation does not succeed within its budget. This amends SPEC-013 FR-001 for real mode (FR-017). +- Q: Not in the issue, found in the code: when real-mode Lightstack cannot reach libvirt at startup, it carries on with no hypervisor, and `server create` reports servers `ACTIVE` with no guest behind them. Is that in scope? → A: **Yes — retry, and never fake.** While unconnected, guest operations fail — answered as a lost connection is, per the next answer — and a create ends in `ERROR`. The first connection is retried in the background like a lost one, with the socket choice made on every attempt. Once connected, the preparation runs and guests start without a restart (User Story 6, FR-019, FR-020). +- Q: While real-mode Lightstack has no libvirt connection — never connected yet, or lost — how does a server action answer? The spec said "an error naming libvirt"; SPEC-025 deliberately answers a lost connection with a generic `500`. → A: **As a lost connection.** One answer for an unreachable libvirt, never connected or lost: a server action answers `500` with the generic "failed to perform " (lifecycle actions; other handlers keep their own message — analysis session), the cause in the log; a create is accepted and ends in `ERROR`. No new message, no `503`. A guest start before the first preparation keeps FR-016's refusal, which already names the directory and its cause (FR-019). +- Q: A guest must not start with a root-owned per-instance directory, or a modular `virtqemud` cannot write its `console.log`. Only the create paths prepare the directory today; self-heal re-creates a guest without that step, and guests from earlier releases keep root-owned directories. Where is the ownership ensured? → A: **Immediately before every guest start** — create (image or volume boot), start, hard reboot, resume and self-heal re-creation. Idempotent, so it also repairs a directory made while QEMU's user was unknown, one left by an earlier release, and one whose QEMU user changed under a reconnect redo (FR-014). +- Q: How does host validation prove the retry paths? A warm libvirt answers capabilities in about 0.04 s and caches QEMU's capabilities on disk, so a plain reboot may never hit the slow first read; the unreachable-libvirt path and the monolithic path had no host step. → A: **Force both paths, and add a monolithic step.** Cold: stop `virtqemud` and clear its QEMU capabilities cache before Lightstack starts. Unreachable: start Lightstack with libvirt's sockets stopped, then start them. Each passes when a VM boots to `ACTIVE` with no restart and the log shows the failed first attempt, the later success and the socket dialed. Monolithic: a default install against `libvirtd` — the same host switched over will do — still dials `libvirt-sock` (FR-010, FR-021). +- Q: The server now keeps dialing libvirt in the background. What does `lightstack cleanup` — the uninstall purge Job, fail-closed per SPEC-010 — do when libvirt is unreachable? → A: **One attempt, fail closed, as today.** One connection attempt with the socket choice of FR-007; if it fails, the purge fails at once and the uninstall stops. No background dialing in the CLI, no waiting (FR-020a). +- Q: Nothing documented Problem 1's change — host-networked pods now resolve through the cluster DNS — or the host workarounds the fixes retire. What do the docs say? → A: **The chart docs and `docs/UPGRADING.md`.** The chart README and the `realMode.hostNetwork` comment and description say host networking renders `dnsPolicy: ClusterFirstWithHostNet`, so in-cluster Service names resolve. `docs/UPGRADING.md` says names only the node's own resolver can answer must be reachable through the cluster DNS, and that the `libvirt-sock` symlink, enabling `virtproxyd.socket` and warming libvirt before start can be removed (FR-003a, FR-024). + +### Session 2026-11-06 (analysis) + +- Q: `lightstack cleanup`'s one connection attempt kept a 2-second dial timeout and no bound on the handshake, against Article III's "libvirt connection failures: 1 second timeout" and this spec's "no waiting". Bound it? → A: **Yes — the same 1-second bound as every other attempt.** A socket-activated daemon that has not answered within it fails the purge closed; the attempt itself started the daemon, so re-running the uninstall finds it warm (FR-020a). +- Q: A guest start that finds a preparation attempt already running could wait for it and then make its own — up to 2 seconds, against the v1.5.0 exception's "one attempt … or wait for one already running … within the dependency's bound". → A: **It takes the result of the attempt already running; only a start that finds none makes its own** (FR-013, FR-016). +- Q: A create that ends `ERROR` because libvirt is unreachable carried no `fault` — the libvirt failure path bypasses the one SPEC-025 added — while upstream reports why a build failed. Keep that out of scope? → A: **No. Every real-mode create that ends `ERROR` records a fault**, with a curated message, as the network-allocation failure already does (FR-019). +- Q: "failed to perform " is what lifecycle actions answer; delete, console and volume attach/detach have their own generic `500` messages. Which does a non-start action give while libvirt is unreachable? → A: **Each handler's existing message for a lost connection** — no new text (FR-019, US6 scenario 2). + +### Session 2026-11-16 (review) + +- Q: A create that names neither `imageRef` nor `block_device_mapping_v2` is accepted with `202` and ends `ERROR` with the fault "the image reference is not a valid image ID" — in stub mode too, since the server's stub hypervisor runs the same create. Upstream refuses it before it looks anything up (`_image_from_req_data`: `400` "Missing imageRef attribute"). Leave it for another ticket, or refuse it here? → A: **Refuse it here, in both modes**, with upstream's status and message, before anything is created (FR-019a). A create whose mappings name no root device Lightstack boots from — a snapshot or blank root, no boot index 0 — still ends `ERROR` with that fault. +- Q: Suspend, resume and shelve answer `202` and only then ask libvirt; the action contract said they answer `500`. Shelve also dropped the stop's failure, so the server read SHELVED while its guest ran on, and suspend's failure reached no log. Keep shelve as it is? → A: **No: a shelve libvirt cannot perform ends `ERROR`, as a failed suspend or resume already does, and all three log the cause** (FR-019b, C-GUEST-2). + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 — Reach in-cluster services by name in real mode (Priority: P1) + +An operator installs the chart in real mode with host networking on k3s and points Lightstack at an in-cluster PostgreSQL by its Service name (`..svc`), and at an in-cluster RGW and Redis the same way if they have them. The pod resolves the names, becomes Ready, and keeps working when a Service is re-created. An uninstall with `dataRetentionPolicy: delete` completes, because the purge Job resolves the same names. + +**Why this priority**: An operator-managed database in the same cluster is the common case, and today it works only through a cluster IP that changes under the consumer. It is also the smallest fix. + +**Independent Test**: Render the chart with real mode and host networking and confirm the policy on all three workloads. On k3s, install with a Service-name PostgreSQL URL; the pod becomes Ready (`/readyz` is gated on the database); uninstall with the delete policy completes. + +**Acceptance Scenarios**: + +1. **Given** `realMode.enabled: true` and `realMode.hostNetwork: true`, **When** the chart renders, **Then** the Deployment, the multi-node DaemonSet and the purge Job carry `dnsPolicy: ClusterFirstWithHostNet`. +2. **Given** stub mode, or real mode with `realMode.hostNetwork: false`, **When** the chart renders, **Then** no `dnsPolicy` is rendered — as today. +3. **Given** real mode with host networking on k3s and a PostgreSQL URL naming `..svc`, **When** the chart is installed, **Then** the pod becomes Ready. +4. **Given** that release with `dataRetentionPolicy: delete`, **When** it is uninstalled, **Then** the purge Job reaches the database by its Service name and the uninstall completes. + +--- + +### User Story 2 — Run real mode on a host with only libvirt's modular daemons (Priority: P1) + +An operator runs real mode on a host where libvirt runs as modular daemons: `virtqemud` with socket activation, no `libvirtd`, no `virtproxyd`, no symlink. They install the chart with default values. Lightstack connects to the QEMU daemon's own socket, a VM boots to `ACTIVE`, and an uninstall with the delete policy destroys it. On a host running the monolithic `libvirtd`, nothing changes. + +**Why this priority**: Real mode does not work on these hosts today without a host-side workaround that does not survive a reboot — and without the workaround, Lightstack reports servers it never created. + +**Independent Test**: Unit tests cover the socket choice with each socket present or absent. On a modular-daemon host with no workaround, install the chart with default values, create a server, see it `ACTIVE` with its guest in `virsh list`, uninstall with the delete policy, and see the guest gone. Switch the host to monolithic `libvirtd`, install again with default values, and see the log name `libvirt-sock` and a server reach `ACTIVE`. + +**Acceptance Scenarios**: + +1. **Given** the default URI and a host where only `virtqemud-sock` exists, **When** Lightstack connects, **Then** it dials `virtqemud-sock`. +2. **Given** the default URI and a host where `libvirt-sock` exists and `virtqemud-sock` does not, **When** Lightstack connects, **Then** it dials `libvirt-sock` — as today. +3. **Given** a modular-daemon host with no workaround and a chart install with default values, **When** a server is created, **Then** it boots to `ACTIVE` with a guest behind it. +4. **Given** that release with `dataRetentionPolicy: delete`, **When** it is uninstalled, **Then** the purge Job connects the same way and destroys the guest. + +--- + +### User Story 3 — The first guest boots on a cold, socket-activated libvirt (Priority: P1) + +An operator starts Lightstack on a host whose `virtqemud` is socket-activated and not yet running — after a reboot, with no warm-up. The first capabilities read takes longer than a second and fails. Lightstack keeps trying. The pod reports Ready only once guests can start, and from then on guest starts work — no restart, no warm-up step. The log says when guests became startable. + +**Why this priority**: Today a cold daemon at startup disables guest starts until the pod restarts, while the pod reports Ready. + +**Independent Test**: Unit: a fake libvirt whose first capabilities read times out and whose second succeeds; the next guest start succeeds without a restart, and its per-instance directory belongs to QEMU's user. Readiness: not-ready in real mode before the preparation succeeds, ready after, still ready through a later libvirt outage; stub mode unchanged. Host: stop the socket-activated `virtqemud` and clear its QEMU capabilities cache, then start Lightstack; the log shows the failed first read and the later success, the pod becomes Ready, and a server reaches `ACTIVE`. + +**Acceptance Scenarios**: + +1. **Given** a first capabilities read that times out, **When** libvirt later answers, **Then** the preparation succeeds without a restart, and a log line says so, naming the directory and QEMU's user. +2. **Given** the preparation has not yet succeeded, **When** a guest start arrives, **Then** it makes one attempt within the 1-second bound — or, if an attempt is already running, takes that one's result — before going on, and is refused with the existing message naming the directory and the cause only if that attempt fails. +3. **Given** a per-instance directory created while QEMU's user was unknown, **When** that guest starts after the preparation has succeeded, **Then** the directory belongs to QEMU's user and the guest's console log can be written. +4. **Given** a guest whose per-instance directory is root-owned — left by an earlier release — **When** it is started, hard-rebooted, resumed or self-healed, **Then** the directory is made QEMU's before QEMU launches, and the guest runs. +5. **Given** a host with socket-activated `virtqemud` and no warm-up, **When** Lightstack starts, **Then** the pod becomes Ready, and the first server created boots to `ACTIVE`. +6. **Given** the preparation has succeeded, **When** libvirt restarts and Lightstack reconnects, **Then** the preparation runs again, and a redo that times out does not stop guest starts. +7. **Given** real mode and a preparation that has not yet succeeded, **When** readiness is asked, **Then** it answers not-ready with a reason naming the hypervisor, and the pod is not Ready. +8. **Given** the preparation has succeeded once, **When** libvirt later goes away or a redo fails, **Then** readiness stays ready. +9. **Given** stub mode, **When** readiness is asked, **Then** it depends on the database and the listeners only, as today. + +--- + +### User Story 4 — Point the chart at a specific libvirt socket (Priority: P2) + +An operator whose libvirt socket lives elsewhere — another directory, or one specific daemon's socket — sets `realMode.libvirtSocketPath`. The pods mount its directory, as today, and now the Lightstack pod and the purge Job also dial that socket. + +**Why this priority**: The value looks like the fix and does nothing today. With User Story 2 most hosts no longer need it, but a non-standard layout still does. + +**Independent Test**: helm-unittest: with a non-default value, the rendered config read by the Lightstack pod and by the purge Job names that socket as `libvirt_uri`; with the default value the rendered `libvirt_uri` is unchanged. + +**Acceptance Scenarios**: + +1. **Given** `realMode.libvirtSocketPath: /run/libvirt/virtqemud-sock`, **When** the chart renders, **Then** the rendered config's `libvirt_uri` is that path and the pods mount `/run/libvirt`. +2. **Given** the default value, **When** the chart renders, **Then** `libvirt_uri` is `qemu:///system`, as today. +3. **Given** a configured socket, **When** the binary connects, **Then** it dials exactly that path, whatever else exists. + +--- + +### User Story 5 — A libvirt setting Lightstack cannot honour is refused at startup (Priority: P3) + +An operator writes a `libvirt_uri` Lightstack does not support: a remote URI, a session URI, another driver. Real-mode Lightstack refuses to start and names the value and the forms it accepts, instead of dialing the string as a file and carrying on without a hypervisor. A local URI that names its socket (`qemu+unix:///system?socket=`) works. + +**Why this priority**: A configuration error, not an outage. Today it ends in a misleading "no such file or directory" and a real mode with no hypervisor, and the documentation advertises forms that never worked. + +**Independent Test**: Unit tests over the accepted and refused forms. A startup test with an unsupported URI in real mode exits non-zero with the message; the same URI in stub mode starts. + +**Acceptance Scenarios**: + +1. **Given** `libvirt_uri: "qemu+unix:///system?socket=/run/libvirt/virtqemud-sock"`, **When** Lightstack connects, **Then** it dials that path. +2. **Given** an unsupported URI in real mode, **When** Lightstack or `lightstack cleanup` starts, **Then** it exits non-zero naming the value and the accepted forms. +3. **Given** the same URI in stub mode, **When** Lightstack starts, **Then** it starts — stub mode never dials libvirt. + +--- + +### User Story 6 — Real mode never reports a server it did not create (Priority: P1) + +An operator starts real-mode Lightstack while libvirt cannot be reached — its socket missing, the daemon down, the node still booting. Today Lightstack carries on with no hypervisor and answers `server create` with a server that turns `ACTIVE` with no guest behind it. Instead, guest operations fail as they do while a lost connection is being re-established, the pod is not Ready, and Lightstack keeps trying to connect as it does after a lost connection. Once libvirt is reachable, it prepares, becomes Ready, and guests start normally — no restart. + +**Why this priority**: It is what turns Problems 2 and 3 from refusals into wrong answers: an orchestrator, Terraform or Horizon sees a server that does not exist. + +**Independent Test**: Unit: a manager whose first dial fails and a later one succeeds; guest operations fail while unconnected, as during a lost connection, and succeed after, with no restart. API: in real mode with libvirt unreachable, `server create` ends in `ERROR`, never `ACTIVE`, and a server action answers the generic `500`. Host: start Lightstack with libvirt's sockets stopped, then start them; the log shows the failed first attempt, the later connection and the socket dialed, and a server reaches `ACTIVE` with no restart. + +**Acceptance Scenarios**: + +1. **Given** real mode and libvirt unreachable at startup, **When** a server is created on any path that still reaches the API — Docker Compose, a bare-metal install, the pod's own address — **Then** it is accepted and ends in `ERROR`, never `ACTIVE`, with a `fault` that says why in a curated message; the log names libvirt as the cause. +2. **Given** that state, **When** a server action that does not start a guest is requested — stop, delete, console, volume attach — **Then** it answers `500` with the message its handler gives a lost connection — "failed to perform " for stop, the handler's own generic message for delete, console and volume attach — and the log names libvirt. +3. **Given** that state, **When** readiness is asked, **Then** it answers not-ready with a reason naming the hypervisor. +4. **Given** that state, **When** libvirt becomes reachable, **Then** Lightstack connects without a restart, prepares the console socket directory, becomes Ready, and the next server boots to `ACTIVE`. +5. **Given** the default URI and neither socket present at startup, **When** one appears later, **Then** Lightstack finds it on a later attempt. + +--- + +### Edge Cases + +- **Real mode without host networking**: the pod already uses cluster DNS; nothing is rendered (FR-002). +- **No cluster DNS configured**: for a `ClusterFirstWithHostNet` pod, Kubernetes falls back to the node's resolver and records a `MissingClusterDNS` event; behaviour is as today. +- **Names only the node's resolver can answer**: the node's `/etc/hosts` still applies to host-networked pods. Other names now go through the cluster DNS, which forwards to the node's upstream resolvers by default (CoreDNS in k3s does). A split-horizon name the cluster DNS cannot forward stops resolving — the cost of the standard policy. +- **Upgrading an existing release**: the pod spec gains `dnsPolicy`, so `helm upgrade` replaces the pod (Recreate in real mode), as any pod-spec change does. +- **Both sockets present**: `virtqemud-sock` is chosen. Where `libvirt-sock` belongs to `virtproxyd`, both reach the same daemon. Where it belongs to a monolithic `libvirtd` running beside `virtqemud`, two daemons manage QEMU — a host misconfiguration that libvirt's migration steps avoid by stopping the monolithic daemon first. Lightstack's guests then live in `virtqemud`. +- **Neither socket exists**: the connection fails and the error names both paths (FR-010). +- **Mixed multi-node fleet**: each node's pod makes its own choice, so monolithic and modular nodes run with the same default value. +- **`secrets.existingSecret`**: the chart renders no config, so the operator's own config must name the socket the pods mount. The chart cannot check a Secret it does not render (FR-005). +- **A non-absolute `realMode.libvirtSocketPath`**: refused by the schema (FR-006). It would mount, and now dial, a path relative to the pod's working directory. +- **The explicit default**: through the chart, `libvirt-sock` at its default path cannot be forced over `virtqemud-sock` — the default value means "choose". It is what the binary picks whenever `virtqemud-sock` is absent; the binary's own config can force it with a bare path (FR-008). +- **A guest start while the preparation is pending**: one attempt within the 1-second bound, or the result of the attempt already running — never both; if it fails, the start is refused as today, and a later start succeeds once libvirt answers. +- **A per-instance directory that is not QEMU's** — made while QEMU's user was unknown, left root-owned by an earlier release, or owned by a QEMU user that has since changed: it is made QEMU's immediately before its guest starts, on every path, self-heal included (FR-014). +- **A capabilities call that outlives its bound**: it finishes in the background, as the existing bound already allows, and a later attempt gets the warm answer. No second read is sent while it is pending: an attempt made meanwhile fails at once, so a daemon that never answers holds one call, not one per attempt. +- **libvirt restarts after a successful preparation**: the preparation is redone on reconnect; a redo that times out leaves guest starts working (FR-015), and readiness stays ready (FR-017). +- **A preparation that never succeeds** — libvirt reports no DAC label for QEMU, the directory cannot be chowned, libvirt never comes up: the pod is never Ready. The startup probe restarts it once its budget is spent (`probes.startup`, 30 × 5 s by default) and it ends in CrashLoopBackOff, with the cause at the top of each start's log; `helm install --wait` fails. Before this change the same pod was Ready and refused every guest start. +- **A single node that is not yet Ready**: the Service has no endpoint, so all six APIs — Keystone included — are unreachable through it until the preparation succeeds. The listeners still answer on the pod's own address (the node's, with host networking). +- **Multi-node**: only the pod whose node cannot prepare leaves the Service; the others keep serving the shared API. +- **libvirt unreachable during an uninstall**: the purge makes one attempt, bounded at 1 s, and fails closed (FR-020a); the operator brings libvirt back and uninstalls again. With socket activation the socket exists while the daemon is stopped: the attempt starts the daemon, and if the daemon has not answered within the bound, a re-run finds it warm. +- **Docker Compose and bare metal**: nothing gates on readiness there; `/readyz` reports the same state, and guest operations behave as FR-016 and FR-019 say. +- **Stub mode**: no host networking, no libvirt, no change to readiness; nothing here changes it. + +## Requirements *(mandatory)* + +### Functional Requirements + +**Cluster DNS under host networking (Problem 1)** + +- **FR-001**: Every pod the chart runs with host networking MUST resolve in-cluster Service names. When `realMode.enabled` and `realMode.hostNetwork` are both `true`, the single-node Deployment and the multi-node DaemonSet (through the shared pod spec) and the teardown purge Job MUST carry `dnsPolicy: ClusterFirstWithHostNet`, as the console proxy already does. +- **FR-002**: Without host networking — stub mode, or real mode with `realMode.hostNetwork: false` — the chart MUST render no `dnsPolicy`, exactly as today. +- **FR-003**: helm-unittest cases MUST cover FR-001 for all three workloads and FR-002 for both cases without host networking. +- **FR-003a**: The chart README and the `realMode.hostNetwork` comment in `values.yaml` and description in `values.schema.json` MUST say that host networking renders `dnsPolicy: ClusterFirstWithHostNet`, so the Lightstack pod and the purge Job resolve in-cluster Service names. + +**The libvirt socket (Problem 2)** + +- **FR-004**: When `realMode.libvirtSocketPath` differs from its default (`/var/run/libvirt/libvirt-sock`), the chart MUST render it as `nova.libvirt_uri` in the configuration it renders, which both the Lightstack pod and the purge Job read, so the socket mounted and the socket dialed are the same. With the default value the rendered `libvirt_uri` MUST stay `qemu:///system`. No new chart value. +- **FR-005**: The documentation of `realMode.libvirtSocketPath` (the `values.yaml` comment, the schema description, the chart README) MUST say that a non-default value is also the socket dialed and that the default lets the binary choose (FR-007). Because the chart renders no configuration with `secrets.existingSecret`, it MUST also say that the operator's own config must then name the same socket. +- **FR-006**: The chart schema MUST require `realMode.libvirtSocketPath` to be an absolute path, as it requires `realMode.hostStoragePath`. +- **FR-007**: For `qemu:///system` — and for an empty `libvirt_uri` and `qemu+unix:///system` without a socket parameter, which mean the same — the binary MUST dial the modular QEMU daemon's socket, `/var/run/libvirt/virtqemud-sock`, when it exists, and `/var/run/libvirt/libvirt-sock` otherwise. The choice MUST be made at every connection attempt, reconnects included, so a socket that appears or disappears while Lightstack runs is followed without a restart. +- **FR-008**: The binary MUST accept, and dial exactly, an explicit socket: a `socket=` parameter on `qemu:///system` or `qemu+unix:///system`, or a bare absolute path (today's behaviour, kept). An explicit socket MUST win over the automatic choice. +- **FR-009**: In real mode, any other `libvirt_uri` — another driver, a session URI, a remote transport or host, a relative path, a parameter other than `socket` — MUST stop every entry point that loads the configuration — the server, `lightstack cleanup` and `lightstack console-tokens` — at configuration load, with an error naming the value and the accepted forms. Stub mode never dials libvirt and MUST NOT check the value. The code comment MUST describe what the code does. +- **FR-010**: A failed connection MUST name the socket path or paths it tried, and a successful one MUST log the socket it dialed. +- **FR-011**: Unit tests MUST cover the resolution: the default with only `virtqemud-sock` → that socket; the default with `libvirt-sock` and no `virtqemud-sock` → `libvirt-sock`; the default with both → `virtqemud-sock`; a configured socket (parameter or bare path) → that path, whatever exists; each refused form → the startup error. +- **FR-012**: On a host running the monolithic `libvirtd`, behaviour MUST be unchanged with default values: Lightstack dials `libvirt-sock`. + +**The console socket directory preparation (Problem 3)** + +- **FR-013**: A failed preparation MUST NOT be final. Lightstack MUST attempt it again until it succeeds — in the background with backoff from the moment it first fails, and on demand when a guest start finds it not done. A guest start that finds an attempt already running MUST take that attempt's result instead of starting a second. Every attempt MUST keep the existing 1-second bound on its libvirt call (Constitution: fail-fast), so no guest start waits longer than one bound. +- **FR-014**: Immediately before every guest start, Lightstack MUST ensure the guest's per-instance directory exists and belongs to the user QEMU runs as, as the preparation last recorded it. The step MUST be idempotent, so it also repairs a directory created while that user was unknown, one left root-owned by an earlier release, and one whose owner changed when a reconnect redo found a different user (FR-015). The on-demand preparation attempt (FR-013) MUST come before this step, which needs the user it records. A start whose directory cannot be made QEMU's MUST NOT launch the guest. +- **FR-015**: After each libvirt reconnect the preparation MUST run again, so a change of QEMU's user made while libvirt was down is picked up. A redo that fails MUST NOT withdraw a success already in effect. +- **FR-016**: While the preparation has not succeeded, a guest start that cannot complete it MUST be refused, after no more than one 1-second attempt — its own or the one it joined — with the existing message naming the directory and the cause. Every other service MUST keep answering: the pod is not Ready meanwhile (FR-017), but its listeners stay up. +- **FR-017**: In real mode Lightstack MUST report not-ready until the preparation has succeeded once. From then on nothing about libvirt MUST affect readiness again — not an outage, not a reconnect, not a failed redo — so SPEC-013 FR-001 keeps holding for outages; this amends it for the one-time preparation only. The answer MUST come from the recorded state, never from a libvirt call, and its reason MUST name the hypervisor without error detail, as the database reason does. Stub mode's readiness MUST NOT change. +- **FR-017a**: The log MUST show the first failure with its cause (as today), a line when the preparation succeeds after having failed — naming the directory and QEMU's user — and retries at debug level only, so a long outage does not flood the log. An attempt that fails only for want of a libvirt connection is not that first failure: the connection loop already reports the outage, so it is logged at debug level. +- **FR-018**: A unit test MUST show that a first capabilities read that times out, followed by one that succeeds, lets the next guest start succeed without a restart, with its per-instance directory owned by QEMU's user. Tests MUST also show that each kind of guest start — create, start, hard reboot, resume, self-heal — makes a root-owned per-instance directory QEMU's before the guest launches. +- **FR-018a**: Readiness tests MUST show: in real mode, not-ready before the preparation succeeds, with a reason naming the hypervisor; ready after it; still ready after a later loss of libvirt; and stub mode unchanged. + +**libvirt unreachable at startup (User Story 6)** + +- **FR-019**: In real mode Lightstack MUST NOT report a server it did not create. Until its first libvirt connection it MUST behave as it does while a lost connection is being re-established (SPEC-025): a server create is accepted and ends in `ERROR`, never `ACTIVE`, with a `fault` — every real-mode create that ends `ERROR` records one, its message curated as the network-allocation fault's is, the detail in the log; a server action answers the `500` its handler already gives a lost connection — "failed to perform " for lifecycle actions, the handler's own generic message for delete, console and volume attach or detach — the cause in the log; a guest start meets FR-016's refusal first, since the preparation cannot have succeeded. Nothing takes the stub-mode path, and no new HTTP status or action message is introduced. +- **FR-019a**: A server create that names neither an `imageRef` nor a `block_device_mapping_v2` entry MUST be refused with `400` "Missing imageRef attribute" before anything is created, in stub and real mode, as upstream refuses it (`nova/api/openstack/compute/servers.py`, `_image_from_req_data`). +- **FR-019b**: An action that answers `202` and only then asks libvirt — suspend, resume, shelve — MUST end the server in `ERROR` when libvirt fails it, with the cause in the log, so the record never claims a state the guest is not in. +- **FR-020**: When the server's first connection fails, it MUST keep dialing in the background with the same backoff it uses after a lost connection, making the socket choice of FR-007 on every attempt. Once connected, it MUST run the preparation (FR-013); the pod becomes Ready when that succeeds (FR-017), and guests MUST start without a restart. +- **FR-020a**: `lightstack cleanup` MUST keep SPEC-010's fail-closed contract: one connection attempt — dial and handshake bounded at 1 second, as Article III requires — with the socket choice of FR-007; if it fails, the purge exits non-zero at once, naming the socket or sockets it tried, and the uninstall stops. It MUST NOT dial in the background or retry. Tests MUST show it failing without waiting when the socket is absent, and within the bound when the daemon does not answer. + +**Validation and documentation** + +- **FR-021**: Host validation through the chart on k3s MUST confirm: + - (a) real mode with host networking and a Service-name PostgreSQL URL becomes Ready, and a delete-policy uninstall completes; + - (b) on a host with only modular daemons — no `libvirtd`, no `virtproxyd`, no symlink — Lightstack connects with default values, a VM boots to `ACTIVE`, and a delete-policy uninstall destroys it; + - (c) the cold path, forced: with `virtqemud` stopped and its QEMU capabilities cache cleared before Lightstack starts, the log shows the failed first read and the later success, the pod becomes Ready, and the first guest boots to `ACTIVE` with no restart. If a fast host answers even the cold read within 1 s, the step records that the retry went unexercised, and FR-018's unit test remains the proof; + - (d) the unreachable path, forced: with libvirt's sockets stopped when Lightstack starts, a create ends in `ERROR` and the pod is not Ready; once the sockets are started, the log shows the failed first attempt, the later connection and the socket dialed, the pod becomes Ready, and a VM boots to `ACTIVE` with no restart; + - (e) the monolithic path: a default install against monolithic `libvirtd` — the same host switched over will do — logs `libvirt-sock` as the socket dialed and boots a VM to `ACTIVE`. +- **FR-022**: The libvirt documentation MUST match the result: `docs/CONFIGURATION.md` lists the accepted `libvirt_uri` forms and the automatic choice instead of the session and remote forms it lists today; the `qemu+tcp://` example in `docs/SCALING.md` goes; the `realMode.libvirtSocketPath` documentation follows FR-005; the operator docs that check for `libvirt-sock` (`docs/TROUBLESHOOTING.md`, `docs/REAL_LIBVIRT_MODE.md`, `docs/REAL_MODE_TESTING.md`, `docs/HORIZON_INTEGRATION.md`) also name `virtqemud-sock`. +- **FR-023**: Every description of the readiness inputs MUST add the real-mode condition of FR-017: `charts/lightstack/CONTRACT.md` (§2), the chart README's health table, `templates/NOTES.txt`, the `probes` comments in `values.yaml` and descriptions in `values.schema.json`, and `docs/blueprint.md`. +- **FR-024**: `docs/UPGRADING.md` MUST state what an existing real-mode release meets: + - an unsupported `libvirt_uri` now stops startup; + - a pod that cannot prepare the console socket directory is no longer Ready — its startup probe restarts it — where before it was Ready and refused every guest start; + - with host networking, the pods now resolve through the cluster DNS, so a name only the node's own resolver can answer must be reachable through it; + - the host workarounds can be removed: the `libvirt-sock` symlink, enabling `virtproxyd.socket`, and warming libvirt before Lightstack starts; + - the uninstall purge makes one attempt to reach libvirt, bounded at 1 second (FR-020a), where before it waited on the handshake without a bound: a socket-activated daemon that had exited may miss it, and the uninstall is then run again. + +### Key Entities + +- **libvirt connection setting** (`nova.libvirt_uri`): how the binary finds libvirt. Accepted: the system URI (automatic socket choice), the same with an explicit socket, a bare absolute socket path. Anything else stops real-mode startup. +- **Socket choice**: the socket the binary dials for the system URI, decided at each connection attempt — `virtqemud-sock` when present, otherwise `libvirt-sock`. +- **Chart socket value** (`realMode.libvirtSocketPath`): the socket whose directory the pods mount. A non-default value is also the socket dialed; the default lets the binary choose. +- **Guest start**: any operation that launches a guest's QEMU process — a create (image or volume boot), a start, a hard reboot, a resume, and a self-heal re-creation. Each needs the preparation done (FR-013, FR-016) and its per-instance directory owned by QEMU's user (FR-014). +- **Console socket directory preparation**: either not yet succeeded, or succeeded, with the user QEMU runs as. No cause is stored: a refused start carries the cause of the attempt it made or joined. Retried until it succeeds, redone after each reconnect, and required before any guest starts. In real mode, readiness waits for its first success, and only for that. +- **Host-networked workload**: the Lightstack Deployment or DaemonSet pod and the teardown purge Job, when real mode runs with host networking. The console proxy already resolves cluster names; the quiesce Job does not run on the host network. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: In real mode with host networking, Lightstack and its purge Job reach every in-cluster dependency by Service name: an install whose database URL names a Service becomes Ready, and a delete-policy uninstall completes, with no cluster IP or node address in any dependency's URL. +- **SC-002**: On a host running only libvirt's modular daemons, a chart install with default values boots a VM to `ACTIVE` on the first attempt, and a delete-policy uninstall removes it — with zero host-side workarounds: no proxy daemon, no symlink, no warm-up. +- **SC-003**: On a host running the monolithic daemon, a default install dials `libvirt-sock` and boots guests as before; its default render differs only by the DNS policy line under host networking. +- **SC-004**: After a startup at which libvirt answered slowly, the pod becomes Ready and guest starts work without a restart: the first guest created boots to `ACTIVE`. Zero restarts, zero warm-up steps. +- **SC-004a**: In real mode a Ready pod has a prepared console socket directory: zero cases of a pod that is Ready while every guest start is refused with `console socket directory not ready`. +- **SC-005**: A non-default socket set in the chart is the socket both the Lightstack pod and the purge Job dial. +- **SC-006**: Every unsupported `libvirt_uri` stops real-mode startup with a message naming the value and the accepted forms; none is dialed as a file. +- **SC-007**: In real mode, no server created while libvirt is unreachable is ever reported `ACTIVE`, and every one that ends `ERROR` shows a `fault` saying why. +- **SC-008**: Every acceptance criterion of the issue has an automated test or a host-validation step, and the existing unit, contract and chart suites stay green on both database backends. + +## Assumptions + +- The pods affected by Problem 1 are the Lightstack workload (Deployment or DaemonSet) and the purge Job. The console proxy already sets the policy; the quiesce Job runs `kubectl` without host networking. +- `ClusterFirstWithHostNet` is the standard Kubernetes policy for this case. No chart value for the DNS policy or DNS config is added. +- The default `realMode.libvirtSocketPath` keeps its value, and keeps meaning "mount `/var/run/libvirt`"; it now also means "let the binary choose". Only a different value is passed through (FR-004). +- libvirt's run directory is `/var/run/libvirt`; distributions that use `/run` link `/var/run` to it, and the mounted directory is the same either way. +- Lightstack needs only the domain and connect APIs (`Domain*`, `ConnectGetCapabilities`, `ConnectGetLibVersion`), all served by the QEMU daemon, so `virtqemud-sock` alone is sufficient — no `virtproxyd`, no other modular daemon. +- The background retry reuses the existing reconnect backoff (1 s, doubling to 5 s). The 1-second bound per libvirt call is unchanged. +- The startup probe's existing budget (30 × 5 s, about 150 s) is the grace for a slow libvirt at boot. No new chart value; an operator with a slower host raises `probes.startup.failureThreshold`. +- SPEC-013's own documents are not edited; the amendment of its FR-001 is recorded here (Clarifications, FR-017). +- An unsupported `libvirt_uri` is a configuration error, so it is refused when the configuration loads, like the other real-mode checks there (`internal/common/config.go`), not degraded around. +- No database schema change, no migration, no new Go module, no new chart value. + +## Out of Scope + +- Remote libvirt (TCP, TLS, SSH) and session URIs: refused with a clear message, not implemented. +- A chart value for an arbitrary DNS policy or DNS config. +- Guest DNS: dnsmasq serves DHCP only, and guests get the subnet's nameservers. +- Readiness that follows libvirt outages: once the preparation has succeeded, SPEC-013 FR-001 holds, and an outage does not unready the pod. +- Reporting compute's hypervisor or service state as `down` (both are hard-coded `up` today). +- Host firewall (firewalld, nftables) and SELinux interplay with real mode. + +## Dependencies + +- Kubernetes `dnsPolicy: ClusterFirstWithHostNet` (https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy). +- libvirt's modular daemons and their socket names, `/var/run/libvirt/virt${DRIVER}d-sock` (https://libvirt.org/daemons.html), and the `socket` URI parameter (https://libvirt.org/uri.html). +- The existing libvirt reconnect loop (`keepConnected`) and the 1-second bound (`consoleReadTimeout`) in `pkg/hypervisor`. +- The chart's helm-unittest harness; host validation needs k3s and a host running only libvirt's modular daemons, switchable to monolithic `libvirtd` for FR-021 (e). diff --git a/docs/specs/031-realmode-dns-libvirt-fixes/tasks.md b/docs/specs/031-realmode-dns-libvirt-fixes/tasks.md new file mode 100644 index 00000000..33bee63b --- /dev/null +++ b/docs/specs/031-realmode-dns-libvirt-fixes/tasks.md @@ -0,0 +1,473 @@ +--- + +description: "Task list for SPEC-031 — real-mode cluster DNS, libvirt socket choice, retried console-dir preparation" +--- + +# Tasks: Real mode — cluster DNS under host networking, the libvirt socket, and a capabilities read that is never retried + +**Feature**: `031-realmode-dns-libvirt-fixes` (cloud-storage#630) +**Input**: Design documents from `docs/specs/031-realmode-dns-libvirt-fixes/` +**Prerequisites**: plan.md, spec.md (required); research.md (R1–R13), data-model.md, contracts/ (4 files), quickstart.md (present). Constitution **v1.5.0** (ratified 2026-11-06) — the plan's gate is open. + +**Tests**: INCLUDED, following the constitution's Development Workflow: +1. contract tests with OpenStack clients where an API outcome changes (T016); +2. the test strategy approved by the user before any test is written (T002); +3. every test seen failing first (RED) — a failing compile counts; +4. then the code. + +The tests below implement the strategy approved in T002; a test that departs from it goes back to the user. A test marked **lock** passes before and after: it guards a behaviour this feature must not break. + +**Organization**: Grouped by the spec's user stories. Priority order is US1, US2, US3, US6 (all P1), then US4 (P2), then US5 (P3). Among the P1s, **US6 comes before US3**: US3's background retry runs inside the connection loop US6 builds. + +**Terms**: a *launch* is a *guest start* as the spec's Key Entities define it — create (image or volume boot), start, hard reboot, resume, self-heal. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: can run in parallel — a different file, no dependency on an incomplete task +- **[Story]**: US1–US6 for story-phase tasks; Setup, Foundational and Polish carry none +- Every task names an exact file path. Line numbers are hints from `d81f081`, the `main` commit this branch is based on — find the code by the function named. + +## Path conventions + +Single Go project at the repo root (`cmd/`, `internal/`, `pkg/`, `test/`), the Helm chart under `charts/lightstack/`, docs under `docs/`. No `src/` layout (plan.md § Project Structure). + +**Test databases** (CLAUDE.md): +- PostgreSQL: `make db-up`, then `LIGHTSTACK_TEST_DB_URL='postgres://lightstack:secret@localhost:5432/lightstack?sslmode=disable'` +- SQLite: `LIGHTSTACK_TEST_DB_URL='sqlite:///tmp/ls-test/t.db'` +- Tests that read the seeded project can race across packages; use `go test -p 1` if they flake. + +--- + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: a green baseline that SC-008's "the existing suites stay green" is measured against, and the approved test strategy. + +- [X] T001 Record the baseline in `docs/specs/031-realmode-dns-libvirt-fixes/baseline-gates.md`. Run `make build`, `make lint`, `go test ./...` on PostgreSQL and on SQLite (both URLs above), `helm unittest charts/lightstack`, and `make test-contract` against a running stub Lightstack. Write down pass/fail/skip counts per package and any failure that predates this branch. Fix nothing here; a pre-existing failure is recorded, not hidden. +- [X] T002 Get the test strategy approved before any test is written (constitution Development Workflow, step 2). Review research.md R10 and every test task in this file with the user, and record the approval — date, and any change asked for — in `docs/specs/031-realmode-dns-libvirt-fixes/baseline-gates.md`. No later task starts before it. + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: the `libvirt_uri` grammar (R3) and the per-attempt dialer (R4), which US2, US4, US5 and US6 all build on, plus the fake-libvirtd knobs US2, US3 and US6 test with. + +**⚠️ CRITICAL**: no Go story work starts before T006 is green. US1 is chart-only and may run in parallel with this phase. + +- [X] T003 [P] Write `TestParseLibvirtURI` in `pkg/hypervisor/libvirt_uri_test.go` (new): a table over `ParseLibvirtURI(string) (LibvirtTarget, error)`, per `contracts/libvirt-connection.md` C-URI-1/C-URI-2. + - **Automatic** (empty socket): `""`, `qemu:///system`, `qemu+unix:///system`. + - **Explicit**: `qemu:///system?socket=/run/x/s` and `qemu+unix:///system?socket=/run/x/s` → `/run/x/s`; `/run/libvirt/virtqemud-sock` → itself. + - **Refused**, each error containing the value and the word `accepted`: `qemu:///session`, `qemu:///embed?root=/x`, `lxc:///system`, `qemu://host/system`, `qemu+tcp://host/system`, `qemu+ssh://u@h/system`, `qemu+tls://h/system`, `qemu+libssh2://h/system`, `qemu+ext:///system`, `relative/sock`, `qemu:///system?socket=relative`, `qemu:///system?socket=/a&socket=/b`, `qemu:///system?mode=direct`, `qemu:///system#frag`, `qemu:///system/`. + + RED: the function does not exist (FR-008, FR-009, FR-011). +- [X] T004 [P] Write `TestSocketDialer` in `pkg/hypervisor/libvirt_dialer_test.go` (new). Run real `net.Listen("unix", …)` listeners named `virtqemud-sock` / `libvirt-sock` in a `t.TempDir()` used as the run directory. Subtests: + - only `virtqemud-sock` → `Dial` connects to it, and the dialer reports it as the socket dialed; + - only `libvirt-sock` → that one; + - both → `virtqemud-sock`; + - neither → an error naming **both** paths; + - neither, then `virtqemud-sock` created → the **next** `Dial` connects (the choice is per attempt); + - `libvirt-sock` a symlink to `virtqemud-sock` → resolves; + - an explicit target → dials exactly that path although both automatic sockets exist. + + RED: the dialer does not exist (FR-007, FR-010, FR-011). +- [X] T005 [P] Extend the fake libvirtd in `pkg/hypervisor/fake_libvirtd_test.go`. Add fields: + - `uuid [16]byte` — used by `fakeDomain`, which must also answer `procDomainDefineXML` (it answers nothing today); + - `capsDelay time.Duration` — applied once, to the first `ConnectGetCapabilities` reply across all connections; + - `silentConns int` — the first N connections are read but never answered. + + Add two helpers: + - `serveFakeOnSocket(t, path string, f *fakeLibvirtd)`, which listens on a real unix socket at `path`, serves each accepted connection with `f.serve`, and closes the listener in `t.Cleanup`; + - `waitForWithin(t, what string, d time.Duration, cond func() bool)` beside `waitFor`. `waitFor`'s deadline is 2 s, too short for a 1.5 s first reply plus a retry; the new connection and retry tests use 10 s. + + Existing tests must still pass (`go test ./pkg/hypervisor/...`). +- [X] T006 Implement `pkg/hypervisor/libvirt_uri.go` (new). It makes T003 and T004 green. + - **Parser**: `type LibvirtTarget struct{ Socket string }` (empty `Socket` = automatic) and `ParseLibvirtURI(uri string) (LibvirtTarget, error)`, parsed with `net/url`, exactly the accepted and refused forms of T003. One error format: `nova.libvirt_uri %q is not supported; accepted: qemu:///system or qemu+unix:///system (chooses virtqemud-sock, else libvirt-sock), either with ?socket=, or an absolute socket path`. + - **Run directory**: a package variable `libvirtRunDir = "/var/run/libvirt"`, which tests override. + - **Dialer**: an unexported `socketDialer` implementing go-libvirt's `socket.Dialer` (`Dial() (net.Conn, error)`). On every `Dial`, an explicit target dials its path; an automatic target `os.Stat`s `virtqemud-sock`, then `libvirt-sock` under the run directory, and dials the first that exists. `net.DialTimeout("unix", …, timeout)`. It records the socket dialed (mutex-guarded accessor) and keeps the last returned `net.Conn`, so a caller can close it to abandon a hung handshake (R5). Neither socket present → an error naming both. + - PR review (2026-11-16): the attempt keeps the connection, not the dialer. `dialWithin` hands go-libvirt a `dialAttempt`, which closes its connection when the attempt is given up — and one that opens only afterwards, whose handshake would otherwise wait forever on a daemon that never answers. `socketDialer` keeps only the socket it dialed. + +**Checkpoint**: `go test ./pkg/hypervisor/...` green; the parser and dialer are ready. + +--- + +## Phase 3: User Story 1 — Reach in-cluster services by name in real mode (Priority: P1) 🎯 MVP + +**Goal**: host-networked Lightstack pods and the purge Job resolve `..svc` (FR-001, FR-002, FR-003a). + +**Independent Test**: `helm unittest charts/lightstack` passes the new DNS-policy suite. `helm template … --set realMode.enabled=true --set realMode.hostNetwork=true` shows `dnsPolicy: ClusterFirstWithHostNet` beside `hostNetwork: true` (quickstart §1). On k3s, a Service-name PostgreSQL URL becomes Ready (T044 a). + +### Tests for User Story 1 (write first, confirm RED) + +- [X] T007 [P] [US1] Write `charts/lightstack/tests/dns_policy_test.yaml` (new) for FR-001, FR-002 and FR-003, per `contracts/chart-render.md` C-DNS-1 and C-DNS-2. Set `image.repository`, `secrets.jwtSecret`, `secrets.adminPassword` and a PostgreSQL `database.url` as the other suites do. + - With `realMode.enabled: true`, `realMode.hostNetwork: true`: `equal` `spec.template.spec.dnsPolicy` = `ClusterFirstWithHostNet` for `templates/deployment.yaml`, for `templates/daemonset.yaml` (`deploymentMode: multi-node`), and for `templates/teardown-purge-job.yaml` (`dataRetentionPolicy: delete`). + - `notExists` `spec.template.spec.dnsPolicy` for stub mode with `realMode.hostNetwork: true` (Deployment), and for real mode with `realMode.hostNetwork: false` (Deployment, DaemonSet, purge Job). + + RED: the positive cases fail. + +### Implementation for User Story 1 + +- [X] T008 [US1] FR-001 in `charts/lightstack/templates/_helpers.tpl`, `lightstack.podSpec` (~line 411): inside the existing `{{- if .Values.realMode.enabled }}` block that renders `hostNetwork`, render `dnsPolicy: ClusterFirstWithHostNet` when `.Values.realMode.hostNetwork` is true (R1). +- [X] T009 [P] [US1] FR-001 in `charts/lightstack/templates/teardown-purge-job.yaml` (~line 36): add `dnsPolicy: ClusterFirstWithHostNet` inside the existing `{{- if and .Values.realMode.enabled .Values.realMode.hostNetwork }}` block, after `hostNetwork: true`. +- [X] T010 [P] [US1] Document FR-003a in `charts/lightstack/values.yaml`, `charts/lightstack/values.schema.json` and `charts/lightstack/README.md`: say that host networking renders `dnsPolicy: ClusterFirstWithHostNet`, so the Lightstack pod and the purge Job resolve in-cluster Service names. + - `values.yaml`: the `realMode.hostNetwork` comment (~line 140); + - `values.schema.json`: the `realMode.hostNetwork` description (~line 226); + - `README.md`: the `realMode.privileged` / `hostNetwork` row (~line 137). + +**Checkpoint**: `helm unittest charts/lightstack` green; US1 shippable on its own. + +--- + +## Phase 4: User Story 2 — Run real mode on a host with only libvirt's modular daemons (Priority: P1) + +**Goal**: with default values, Lightstack dials `virtqemud-sock` where only the modular daemons run and `libvirt-sock` where `libvirtd` runs, logs which, and bounds each connection attempt at 1 s (FR-007, FR-010, FR-012; R5). + +**Independent Test**: `go test ./pkg/hypervisor/... -run 'TestNewVMManager'` green. On the openSUSE host, a default chart install logs `/var/run/libvirt/virtqemud-sock` and boots a VM (T044 b); switched to `libvirtd`, it logs `libvirt-sock` (T044 e). + +### Tests for User Story 2 (write first, confirm RED) + +- [X] T011 [P] [US2] Write `pkg/hypervisor/libvirt_connect_test.go` (new). Override `libvirtRunDir` with a `t.TempDir()` (restore in `t.Cleanup`) and serve the fake with `serveFakeOnSocket` (T005). + - `TestNewVMManagerSocketChoice`: + - only `virtqemud-sock` → `NewVMManager("qemu:///system", "real")` connects and reports `virtqemud-sock` as dialed; + - only `libvirt-sock` → connects to it (FR-012); + - neither → the error names both paths. + + In the success cases, capture the zerolog global logger into a buffer (restore it after) and assert the INFO line names the socket (FR-010). + - `TestNewVMManagerBoundsTheAttempt`: a fake with `silentConns: 1` on `virtqemud-sock` → `NewVMManager` returns an error naming the socket within 2 s (bound 1 s). Run it in a goroutine and fail after 5 s, because today it hangs (FR-020a, R5). + + RED: `NewVMManager` dials the fixed `/var/run/libvirt/libvirt-sock` and never times out the handshake. + +### Implementation for User Story 2 + +- [X] T012 [US2] In `pkg/hypervisor/libvirt.go`, `connectLibvirt` (~lines 97-113): + - parse `m.libvirtURI` with `ParseLibvirtURI` and build the T006 `socketDialer`; + - add `dialWithin(d *socketDialer, timeout time.Duration) (*libvirt.Libvirt, error)` with `connectAttemptTimeout = time.Second`. It runs the existing `dial(d)` in a goroutine; on timeout it closes the dialer's last `net.Conn`, waits for the goroutine, and returns an error naming the socket; + - `NewVMManager`'s one attempt uses it (C1 of the analysis, R5); + - delete the verbatim-path fallback and its wrong "strip scheme" comment; + - on success, log INFO naming the socket dialed; a failure's error names the socket or sockets tried (FR-010). + + `NewVMManager` keeps its contract — one synchronous attempt, error on failure — which `internal/teardown` relies on (FR-020a). Makes T011 green. + +**Checkpoint**: T011 green; modular-daemon hosts connect with default values, and no connection attempt waits longer than 1 s. + +--- + +## Phase 5: User Story 6 — Real mode never reports a server it did not create (Priority: P1) + +**Goal**: an unreachable libvirt is an outage, never stub (Constitution IV v1.5.0). +- The server's hypervisor starts unconnected and dials in the background — each attempt ≤ 1 s, pauses 1 → 5 s. +- A create ends `ERROR` with a `fault`, never `ACTIVE`; every real-mode create that ends `ERROR` records a fault. + +(FR-019, FR-020, FR-020a.) + +**Independent Test**: `go test ./pkg/hypervisor/... -run 'TestStartVMManager'`, `go test ./internal/nova/... -run 'TestNoHypervisor|TestInitHypervisor'` (both backends) and `go test ./internal/teardown/...` green. Against a real-mode Lightstack with no libvirt socket (quickstart §2), the gated contract case T016 passes. + +### Tests for User Story 6 (write first, confirm RED) + +- [X] T013 [P] [US6] Write `pkg/hypervisor/libvirt_start_test.go` (new), with `libvirtRunDir` overridden, the fake on a real socket (T005), and `waitForWithin(…, 10*time.Second, …)`: + - `TestStartVMManagerConnectsWhenSocketAppears`: no socket at start → `StartVMManager("qemu:///system", "real")`, the state path in a temporary directory, returns at once with no error, and a call fails "not connected to libvirt"; then `serveFakeOnSocket(virtqemud-sock)` → the manager connects with no restart (FR-020; US6 scenarios 4–5). + - `TestStartVMManagerAbandonsSilentHandshake`: `silentConns: 1` → the first attempt is abandoned after ≈1 s, its connection closed, and the second connects (R5). + - `TestStartVMManagerCloseStopsDialing`: `Close` with no socket present → no dial afterwards (as `TestVMManagerCloseDoesNotReconnect`). + + RED: `StartVMManager` does not exist. +- [X] T014 [P] [US6] Write FR-020a's tests in `internal/teardown/teardown_test.go`: + - `TestCleanupFailsAtOnceWithoutLibvirtSocket` (**lock**): a `common.Config` with `Nova.LibvirtMode: "real"` and `Nova.LibvirtURI: "/missing/libvirt-sock"` → `Run` returns an error naming that path in well under 1 s, and nothing dials afterwards. + - `TestCleanupBoundsASilentDaemon`: a plain unix listener at the configured path that accepts and never answers → `Run` returns an error within 2 s. Guard with a 5 s timeout: RED today, because it hangs. +- [X] T015 [P] [US6] Write `internal/nova/no_hypervisor_test.go` (new; database-backed via `requireDB`, run on both backends, following `boot_failure_test.go`'s set-up) for FR-019. `internal/nova` reaches neither the fake libvirtd nor the run directory, so the unconnected manager comes from an **explicit socket path that does not exist**: `hypervisor.StartVMManager("/missing-sock", "real", "/console-sockets")`. + - `TestNoHypervisorCreateEndsError`: + - (a) with that manager, `paths.SetBase(t.TempDir())` (restored after; no `t.Parallel`), and a boot from an **existing volume** — seeded available, bootable and unattached in the test project, as `internal/nova/boot_from_volume_test.go` does — so the create reaches `CreateVM`. An image boot would instead stage the disk with `qemu-img` and prepare its directory first, and fail there with a different fault. → `POST /v2.1/servers` answers `202`; the server reaches `ERROR`, never `ACTIVE`, and `GET /v2.1/servers/{id}` carries `fault.code` 500 and `fault.message` "the hypervisor is not ready to start guests"; + - (b) real mode with no manager at all (`NewService(testPool, "", "real", nil)`, no `InitHypervisor`) → `ERROR` with the same fault, not `ACTIVE` (C-GUEST-1, R9). + - `TestNoHypervisorActionAnswersGeneric500`: (c) `os-stop` on an instance seeded `ACTIVE` with a `libvirt_domain_id` — so the state precondition passes and no `409` masks the answer — with the unconnected manager → `500` whose message contains `failed to perform` (C-GUEST-2). + - `TestCreateFault` (no database): `createFault` maps an error wrapping `hypervisor.ErrConsoleSocketDirNotReady` to "the hypervisor is not ready to start guests" and any other error to "the hypervisor could not create the guest" (C-GUEST-1). + - `TestInitHypervisorRealModeAlwaysHasAManager`: `NewService(testPool, "/missing-sock", "real", nil).InitHypervisor()` → no error and a non-nil manager; with `qemu+tcp://h/system` → an error. This is the guarantee R9's other nil-manager branches rely on. + - Review (2026-11-16): `TestCreateFaultNamesTheFailedStep` — the root-disk staging fault, the per-instance directory fault on an image and a volume boot, and the image-availability fault (C-GUEST-1); and, in `internal/nova/boot_source_test.go`, `TestCreateWithoutBootSourceIsRefused` for FR-019a in both modes. + + RED: (b) reports `ACTIVE` with no fault today, and the rest does not compile. +- [X] T016 [P] [US6] Write the contract test `test/contract/nova/no_hypervisor_test.go` (new, package `nova_test`) — Development Workflow step 1, with an OpenStack client (FR-019). + - `TestNoHypervisorCreateEndsErrorWithFault` skips unless `LIGHTSTACK_CONTRACT_NO_HYPERVISOR=1`, meaning the Lightstack under test runs real mode with an absent libvirt socket (quickstart §2). + - It uses `contract.SkipIfLightstackNotRunning`, `contract.SetupComputeV2Client` and the image fixture in `test/contract/nova/image_fixture_test.go`, creates a server with gophercloud, and polls `servers.Get` for up to 60 s. It fails on `ACTIVE` and requires `ERROR` with a non-empty `Fault.Message`. + + RED against that server today: the server goes `ACTIVE`. + +### Implementation for User Story 6 + +- [X] T017 [US6] In `pkg/hypervisor/libvirt.go`, add `StartVMManager(uri, mode string) (*VMManager, error)` and one connection loop that replaces `connect`, `keepConnected` and `redial` (~lines 129-194). + - **`StartVMManager`, real mode**: parse the URI (errors only for refused forms), keep `m.conn` nil, start the loop, return. + - **Each loop attempt**: `dialWithin` (T012, 1 s). + - **Between failures**: pause `reconnectBackoffMin` → `reconnectBackoffMax`. Log the first failure at WARN, naming the sockets tried; later ones at DEBUG. + - **On success**: `m.conn.Store`, log INFO with the socket, and call the post-connect hook. For now the hook makes one preparation attempt (`prepareShared(ctx)`, on `paths.ConsoleSocketDir()`, the directory the domain XML names) per connection, as today's startup does; T025 makes it retry. Then wait on `Disconnected()` or `done`, and repeat. + - **`NewVMManager`**: keeps its one synchronous, bounded attempt (T012), then runs the same loop already connected, so reconnects behave as today. + - **Tests**: keep a seam taking a `socket.Dialer` and backoff bounds, and update `newConnectedManager`/`newConnectedManagerWith` in `pkg/hypervisor/fake_libvirtd_test.go` to it. `Close` keeps its `done`/`stopped`/`closeWait` semantics. + - Review (2026-11-16): the connection code takes a `libvirtDialer` — `Dial`, `Dialed`, `abandon` (`abandon` later moved to the attempt, see T006) — instead of probing the dialer for optional methods, and the dialer records its socket before it dials, so a timeout always names it. The connection, UUID and lookup steps every libvirt call opened with are one `lookupDomain`, with one `errNotConnected`. + + Makes T013 green; `libvirt_reconnect_test.go` stays green. +- [X] T018 [US6] FR-019 in `internal/nova/handlers.go`: + - `InitHypervisor()` (~line 206) calls `hypervisor.StartVMManager(svc.libvirtURI, svc.libvirtMode)` instead of `NewVMManager`. + - Add `createFault(err error) string`, curated as `allocationFault` is: `errors.Is(err, hypervisor.ErrConsoleSocketDirNotReady)` → "the hypervisor is not ready to start guests"; anything else → "the hypervisor could not create the guest". The detail stays in the log line the caller already writes. + - Route **every** fault-less `ERROR` write in `CreateServer`'s goroutine through `svc.markInstanceError(instanceID, )`. That is the six inline `UPDATE … status = 'ERROR'` writes, on `d81f081`: + - panic recovery (~1107) → "the server could not be created"; + - boot-from-volume directory (~1260) → "the per-instance directory could not be prepared", the message ~1320 already uses; + - invalid `imageRef` (~1290) → "the image reference is not a valid image ID"; + - image base (~1310) → "the image could not be made available on this host"; + - overlay staging (~1326) → "the root disk could not be staged"; + - `CreateVM` (~1358) → `createFault(err)`. + - In the no-manager branch (~lines 1390-1398), add `else if svc.libvirtMode == "real"` → `svc.markInstanceError(instanceID, "the hypervisor is not ready to start guests")`. Only the stub path keeps the immediate `ACTIVE`. + - Review (2026-11-16), FR-019a: before anything else, refuse a create that names neither `imageRef` nor `block_device_mapping_v2` with `400` "Missing imageRef attribute". The unit tests, contract tests and shell scripts that created servers without a boot source now name one. + - Review (2026-11-16), FR-019b: suspend, resume and shelve route a libvirt failure after their `202` through one `markActionFailed` — the cause in the log, the server in `ERROR`. Shelve used to drop the failure; suspend did not log it. Test: `TestAsyncActionThatFailsInLibvirtEndsError`. + + Makes T015 green, and T016 green against a real-mode server with no socket. +- [X] T019 [US6] In `cmd/lightstack/main.go` (~lines 152-166): + - keep calling `novaService.InitHypervisor()` — the manager prepares `paths.ConsoleSocketDir()` itself; + - in real mode an error is fatal (`log.Fatalf`): only a refused URI can cause one now (R9). Stub mode is unchanged; + - remove the one-shot `novaService.PrepareConsoleSocketDir(...)` call and its ERROR line — the manager prepares after every connection (T017) and logs its own failures. + +**Checkpoint**: T013–T016 green; with libvirt absent, `server create` ends `ERROR` with a fault. + +--- + +## Phase 6: User Story 3 — The first guest boots on a cold, socket-activated libvirt (Priority: P1) + +**Goal**: +- the preparation retries until it succeeds, after every connection, and on demand from a guest start; +- concurrent callers share one in-flight attempt; +- the first success is sticky; +- every guest start first makes the instance's directory QEMU's; +- real-mode readiness waits for the first preparation. + +(FR-013–FR-018a.) + +**Independent Test**: `go test ./pkg/hypervisor/... ./internal/nova/... ./cmd/lightstack/...` green, including the retry, single-flight, stickiness, launch-ownership and readiness tests. On the host, the forced cold path logs a failed first read, then success, and the first server boots `ACTIVE` with no restart (T044 c). + +### Tests for User Story 3 (write first, confirm RED) + +- [X] T020 [P] [US3] Extend `pkg/hypervisor/console_dir_test.go`, using `waitForWithin(…, 10*time.Second, …)` for every wait: + - `TestPreparationRetriesUntilLibvirtAnswers`: fake with `capsDelay: 1500*time.Millisecond` and a DAC label for `otherOwner(t)`, under `StartVMManager` → `Ready` answers `ErrConsoleSocketDirNotReady` after the first attempt, and nil within the deadline, with no restart; the directory owned by that uid:gid. Capture the log and assert one ERROR for the first failure and one INFO naming the directory and uid:gid on success (FR-013, FR-017a, FR-018). + - `TestStartJoinsInFlightAttempt`: with `capsDelay` past the 1 s bound and a background attempt in flight, a concurrent `ensurePrepared` returns that attempt's result within 1.5 s — not after a second attempt of its own — and the fake records one `ConnectGetCapabilities` for both (FR-013, FR-016; Article III v1.5.0). + - `TestPreparedSurvivesDisconnect`: after success `Ready` answers nil, and after `pipeDialer.drop()` it stays nil while unconnected and across the reconnect (FR-015, FR-017, FR-018a). + - `TestRedoAfterReconnect`: the label changes between connections → the owner is replaced; a redo whose caps read fails keeps the earlier owner and `prepared` (FR-015). + - Adapt `TestStartsRefusedUntilConsoleDirReady` (~line 181): a start whose on-demand attempt fails is refused with `ErrConsoleSocketDirNotReady` naming the directory and the cause (FR-016). + + RED: `Ready` does not exist, and nothing retries or shares. +- [X] T021 [P] [US3] Write `pkg/hypervisor/launch_ownership_test.go` (new) for FR-014 and FR-018. Use `paths.SetBase(t.TempDir())` (restored after; no `t.Parallel`, since `pkg/paths` holds global state), a fake whose caps carry `otherOwner(t)`'s uid:gid and whose `uuid` is fixed, and a per-instance directory `InstanceDir()` created beforehand with the test's own owner. + - For each of `CreateVM` (domain from `DomainDefineXML`), `StartVM`, `RebootVM(…, hard=true)` and `ResumeVM` (each passed the same uuid): after the call the directory belongs to `otherOwner`, and the fake recorded `procDomainCreate`. + - With the manager unprepared, the call's single on-demand attempt prepares it first (FR-013). + - `TestGuestStartRefusedWhenInstanceDirCannotBePrepared`: with a regular file where the instances directory belongs, each of the four fails without `procDomainCreate` or `procDomainDestroy`, and the create undefines the domain it defined (FR-014, last sentence; C-GUEST-3). + + RED: the ownership step does not exist. +- [X] T022 [P] [US3] Write `TestHypervisorReady` in `internal/nova/hypervisor_ready_test.go` (new, no database) for FR-017 and FR-018a. `internal/nova` cannot produce a *prepared* manager — the fake libvirtd is test-only in `pkg/hypervisor`, where T020 covers prepared and sticky — so this test covers only: + - a real-mode service with no manager → `HypervisorReady(ctx)` returns an error; + - a real-mode service whose manager comes from `hypervisor.StartVMManager("/missing-sock", "real")` → an error. + + RED: the method does not exist. +- [X] T023 [P] [US3] Write `TestReadinessChecks` in `cmd/lightstack/readiness_checks_test.go` (new). `readinessChecks(pool, novaService, "real")` includes a check named `hypervisor`; with `"stub"` it does not. A `health.Checker` built from a failing `hypervisor` check answers `/readyz` with `503` and reason `hypervisor unavailable`, and with the database failing too `database,hypervisor unavailable` (C-READY-3, FR-018a). RED: `readinessChecks` does not exist. + +### Implementation for User Story 3 + +- [X] T024 [US3] Rework `pkg/hypervisor/console_dir.go` (R6). + - **State**: replace the `consoleDir atomic.Pointer[consoleDirState]` usage with `prepared atomic.Bool` (sticky), the directory, and a `singleflight.Group` (`golang.org/x/sync/singleflight`, already in `go.mod`; run `go mod tidy` so it turns direct — no new module). Then check that the `go.mod`/`go.sum` diff is only that move, and revert anything else `go mod tidy` touched. + - **`prepareShared(ctx)`**: one attempt on `paths.ConsoleSocketDir()` through `group.Do(…)`. A success sets `prepared`, replaces `qemuOwner` and chowns the directory as today; a failure changes nothing in effect once prepared — no cause is stored, a refusal carries the cause of the attempt it made or joined. + - **`ensurePrepared(ctx)`**: `nil` if prepared; else `prepareShared`, which joins the attempt in flight or starts one, and returns `nil` or `ErrConsoleSocketDirNotReady` naming the directory and cause. Never a wait followed by a second attempt. + - **Exports**: `Ready(context.Context) error` — `ErrConsoleSocketDirNotReady` until the first success, nil from then on. + - **Logging (FR-017a)**: first failure ERROR with the cause; success after a failure INFO with the directory and uid:gid; a redo after a reconnect DEBUG, or INFO when the owner changed; retries DEBUG. + - Remove `consoleDirReady` once T026 no longer calls it. + - Review (2026-11-16): an attempt that fails for want of a connection logs at DEBUG and leaves the ERROR line for the first real failure (FR-017a; `TestPreparationErrorLineNotSpentOnMissingConnection`). `ensureOwnedDir` refuses a file that is not a directory where the directory belongs, where it used to chown it (`TestPreparationFailsWhenTheDirectoryCannotBeMade`). +- [X] T025 [US3] In `pkg/hypervisor/libvirt.go`, change the post-connect hook (T017) to retry the preparation — through the same single-flight call — until it succeeds, pausing 1 → 5 s between attempts, after **every** connection even when already prepared (FR-015). Stop when the connection drops (the next connection restarts it) or on `Close`. Makes T020's retry, join and redo tests green. +- [X] T026 [US3] In `pkg/hypervisor/libvirt.go`, in each of the four guest-start functions — `createVMReal` (~221), `hardRebootVMReal` (~370, no context: use `context.Background()`), `startVMReal` (~532) and `resumeVMReal` (~784): + - replace `m.consoleDirReady()` with `m.ensurePrepared(ctx)`; + - call `m.PrepareInstanceDir()` immediately before `conn.DomainCreate(domain)` — the defined domain's UUID in `createVMReal` (move the UUID formatting above `DomainCreate`; on failure undefine the domain, as the existing error path does), the `vmUUID` parameter elsewhere. + - Review (2026-11-16): in `hardRebootVMReal` the step runs once the guest is found — after the id check and the lookup — and still before the power-off, so a refused reboot leaves no directory behind (`TestHardRebootRefusalCreatesNoInstanceDir`). + + In `pkg/hypervisor/disk.go`, rewrite `PrepareInstanceDir`'s comment (~lines 33-59): the "never gets used" premise no longer holds, and the step before every guest start makes the directory QEMU's (R7). Makes T021 green. +- [X] T027 [US3] In `internal/nova/handlers.go`, add `HypervisorReady(ctx context.Context) error`, which delegates to `svc.vmManager.Ready(ctx)`: `nil` once the console socket directory has been prepared; an error otherwise, including real mode with a nil manager. Delete the `PrepareConsoleSocketDir` wrapper (~lines 209-217) if nothing calls it any more (grep the tests). Makes T022 green. +- [X] T028 [US3] In `cmd/lightstack/main.go`: + - add `readinessChecks(pool database.DB, nova *nova.Service, libvirtMode string) []health.Check`, returning `health.DatabaseCheck(pool)` plus, when real, `health.Check{Name: "hypervisor", Probe: nova.HypervisorReady}`; + - move `healthChecker := health.New(...)` from ~line 109 to after `InitHypervisor`, using it — nothing uses the checker in between (its uses: ~371, 415, 436). + + Makes T023 green. + +**Checkpoint**: on a cold host the pod becomes Ready only once guests can start, and stays Ready through later libvirt outages. + +--- + +## Phase 7: User Story 4 — Point the chart at a specific libvirt socket (Priority: P2) + +**Goal**: a non-default `realMode.libvirtSocketPath` reaches the binary as `libvirt_uri`; the default render is unchanged; the schema refuses a relative path (FR-004, FR-005, FR-006, FR-008). + +**Independent Test**: `helm unittest charts/lightstack` green with the new cases. `helm template … --set realMode.libvirtSocketPath=/run/libvirt/virtqemud-sock` renders `libvirt_uri: "/run/libvirt/virtqemud-sock"` in the config Secret (quickstart §1). + +### Tests for User Story 4 (write first, confirm RED) + +- [X] T029 [P] [US4] Add FR-004's cases to `charts/lightstack/tests/config_render_test.yaml`, per C-SOCK-1 and C-SOCK-2, `matchRegex` on `stringData["lightstack.yaml"]`: + - real mode with `realMode.libvirtSocketPath: /run/libvirt/virtqemud-sock` → `libvirt_uri: "/run/libvirt/virtqemud-sock"`; + - the default → `libvirt_uri: "qemu:///system"`; + - set explicitly to `/var/run/libvirt/libvirt-sock` → `libvirt_uri: "qemu:///system"`. + + RED: the first case fails. +- [X] T030 [P] [US4] Add FR-006's cases to `charts/lightstack/tests/schema_test.yaml`, per C-SOCK-3: `realMode.libvirtSocketPath: libvirt-sock` and `realMode.libvirtSocketPath: ""` each → `failedTemplate` with an `errorPattern` matching `libvirtSocketPath|pattern|schema`. RED: both render today. +- [X] T031 [P] [US4] Add `TestExplicitSocketWins` to `pkg/hypervisor/libvirt_connect_test.go` (**lock** once T012 is in): with both automatic sockets served in the run dir, `NewVMManager("/custom-sock", "real")` and `NewVMManager("qemu+unix:///system?socket=/custom-sock", "real")` each dial the custom socket (US4 scenario 3, FR-008). + +### Implementation for User Story 4 + +- [X] T032 [US4] FR-004 in `charts/lightstack/templates/_helpers.tpl`, `lightstack.config` (~line 253): render `libvirt_uri: {{ .Values.realMode.libvirtSocketPath | quote }}` when it differs from `/var/run/libvirt/libvirt-sock`, else `libvirt_uri: "qemu:///system"`. Keep the default literal in one place, e.g. a `lightstack.libvirtDefaultSocket` define (R2). Makes T029 green. +- [X] T033 [P] [US4] In `charts/lightstack/values.schema.json`, give `realMode.libvirtSocketPath` the property `"pattern": "^/.+"` (FR-006; makes T030 green). Then document FR-005 in its description there, in the `charts/lightstack/values.yaml` comment (~lines 141-143), and in the `charts/lightstack/README.md` row (~line 138): + - a non-default value is also the socket dialed; + - the default lets the binary choose `virtqemud-sock`, else `libvirt-sock`; + - with `secrets.existingSecret` the operator's own config must name the same socket. + +**Checkpoint**: a custom socket in the chart is the socket dialed by the pod and the purge Job. + +--- + +## Phase 8: User Story 5 — A libvirt setting Lightstack cannot honour is refused at startup (Priority: P3) + +**Goal**: an unsupported `nova.libvirt_uri` stops real mode at configuration load in every entry point that loads it; stub mode does not check it; the docs stop advertising forms that never worked (FR-009, FR-022). + +**Independent Test**: `go test ./cmd/lightstack/... -run 'TestUnsupportedLibvirtURI|TestLoadConfigLibvirtURI'` green. `lightstack --config ` exits 1 naming the value and the accepted forms (quickstart §2). + +### Tests for User Story 5 (write first, confirm RED) + +- [X] T034 [P] [US5] Add `TestUnsupportedLibvirtURIRefused` to `cmd/lightstack/startup_refusal_test.go` (FR-009). Use the existing `run` helper with a SQLite URL in a temp dir and `extra: "nova:\n libvirt_mode: real\n libvirt_uri: \"qemu+tcp://h/system\"\n"`. For the `server` (exit 1), `cleanup` (2) and `console-tokens` (2) entry points, check the exit code, plus output containing `qemu+tcp://h/system` and `accepted`. Skip `lightstack-migrate`, which never reads libvirt. RED: the server starts and the run times out. +- [X] T035 [P] [US5] Add `TestLoadConfigLibvirtURI` to `cmd/lightstack/config_test.go`: `loadConfig` on a written config with `libvirt_mode: real` and `libvirt_uri: qemu:///session` returns an error naming the value; with `libvirt_mode: stub` the same value loads; real mode with `qemu+unix:///system?socket=/run/x/s` loads. RED: the first case loads. + +### Implementation for User Story 5 + +- [X] T036 [US5] In `cmd/lightstack/main.go`, `loadConfig` (~lines 492-499): after `common.LoadConfig`, when `cfg.Nova.LibvirtMode == "real"`, call `hypervisor.ParseLibvirtURI(cfg.Nova.LibvirtURI)` and return its error unchanged (R3). Makes T034 and T035 green. +- [X] T037 [P] [US5] Update `docs/CONFIGURATION.md` and `docs/SCALING.md` (FR-022): + - `docs/CONFIGURATION.md` (~lines 425-440): replace the "libvirt_uri options" list (`qemu:///session`, `qemu+tcp://`, `qemu+ssh://`) with the accepted forms and the automatic choice (`virtqemud-sock`, else `libvirt-sock`), and note that anything else stops real-mode startup. + - `docs/SCALING.md` (~line 616): replace `qemu+tcp://compute1/system` with `qemu:///system` and a note that each node dials its own local libvirt. + +**Checkpoint**: refused forms stop startup with one clear message. + +--- + +## Phase 9: Polish & Cross-Cutting Concerns + +**Purpose**: the remaining documentation (FR-022, FR-023, FR-024), the gates, a mutation check of the new tests, the host runbook, and host validation (FR-021). + +- [X] T038 [P] FR-022: name `virtqemud-sock` beside `libvirt-sock` wherever an operator doc checks for the socket — `docs/TROUBLESHOOTING.md` (~196), `docs/REAL_LIBVIRT_MODE.md` (~102, ~577), `docs/REAL_MODE_TESTING.md` (~31, ~56, ~164), `docs/HORIZON_INTEGRATION.md` (~364). +- [X] T039 FR-023, in `charts/lightstack/CONTRACT.md`, `charts/lightstack/README.md`, `charts/lightstack/templates/NOTES.txt`, `charts/lightstack/values.yaml`, `charts/lightstack/values.schema.json` and `docs/blueprint.md`: add the real-mode readiness condition — `/readyz` also waits until the console socket directory has been prepared once, and a host where that never succeeds is restarted by the startup probe — to every place that lists the readiness inputs: + - `CONTRACT.md` §2 (~lines 91-93); + - `README.md`, the health table (~line 27); + - `NOTES.txt` (~lines 22-23); + - `values.yaml`, the `probes` comments (~lines 278-289); + - `values.schema.json`, the probe descriptions (~lines 389, 394); + - `docs/blueprint.md` (~lines 526-530). + + Runs after T010 and T033, which touch the same chart files. +- [X] T040 [P] FR-024: add a section to `docs/UPGRADING.md` with the five notes: + - an unsupported `libvirt_uri` now stops real-mode startup; + - a real-mode pod that cannot prepare the console socket directory is no longer Ready, and is restarted by its startup probe; + - host-networked pods now resolve through the cluster DNS, so a name only the node's own resolver answers must be reachable through it; + - the host workarounds can go: the `libvirt-sock` symlink, enabling `virtproxyd.socket`, and warming libvirt before start; + - the uninstall purge makes one attempt to reach libvirt, bounded at 1 s: a socket-activated daemon that had exited may miss it, and the uninstall is run again. +- [X] T041 Re-run T001's gates and append the results to `docs/specs/031-realmode-dns-libvirt-fixes/baseline-gates.md`, comparing with the baseline (SC-008): + - `make fmt` clean, `make lint`, `go vet ./...`; + - `go test ./...` on PostgreSQL and on SQLite; + - `helm unittest charts/lightstack`; + - `make test-contract` against a running stub Lightstack, once with a PostgreSQL and once with a SQLite `database.url`; + - `LIGHTSTACK_CONTRACT_NO_HYPERVISOR=1 go test ./test/contract/nova/ -run NoHypervisor` against a real-mode Lightstack with no libvirt socket (quickstart §2). + + If `make lint` reports paths from a removed worktree, run `go tool golangci-lint cache clean` and repeat. +- [X] T042 Mutation-check the new tests and record the outcome in `docs/specs/031-realmode-dns-libvirt-fixes/baseline-gates.md`. Revert each fix **off the worktree** — a `git clone --shared` scratch clone or `go test -overlay`, never by editing tracked files in place — and confirm its test fails: + - the `dnsPolicy` line (T007); + - the per-attempt socket choice turned into a one-time choice (T004, T013); + - `dialWithin`'s bound removed (T011, T013, T014); + - the single-flight join replaced by a lock-and-retry (T020); + - `prepared` cleared on a failed redo (T020); + - the ownership step removed from one guest-start function at a time (T021); + - the `ERROR` branch restored to `ACTIVE` and the `markInstanceError` call on `CreateVM` failure restored to the inline write (T015, T016); + - the `loadConfig` check removed (T034, T035). +- [X] T043 Generate the host runbook `docs/specs/031-realmode-dns-libvirt-fixes/HOST_TESTING.md` with `/cl.hosttest docs/specs/031-realmode-dns-libvirt-fixes/`: prerequisites, setup, the FR-021 (a)–(e) steps as `quickstart.md` §3 and research R12 lay them out, and teardown, zsh-paste-safe. +- [X] T044 Run host validation FR-021 (a)–(e) with the T043 runbook — k3s on the openSUSE host, `virtqemud` socket-activated, no `libvirtd`, `virtproxyd` or symlink — and record each step's outcome in `docs/specs/031-realmode-dns-libvirt-fixes/HOST_TESTING.md`. A cold step where the fast host answered within 1 s anyway is recorded as "retry not exercised". Restore the host's daemon setup afterwards. **Done 2026-11-13** on openSUSE Tumbleweed (k3s v1.37.2, Helm 4.3.1, Podman; `LIBVIRT_DAEMON=virtqemud`): (a) and (b) pass in 2.1, (d) in 2.3, (e) in 2.4. (c): 2.2's cold start became Ready and booted the first VM `ACTIVE` with no restart, but the host answered within 1 s, so the retry was not exercised there. 2.2b, added for this, froze the daemon: one WARN `libvirt at /var/run/libvirt/virtqemud-sock did not answer within 1s`, then connected after the thaw with `RESTARTS 0`. A freshly started `libvirtd` in 2.4 also missed its first attempt and recovered by itself. QEMU runs as root on that host (DAC `+0:+0`), so handing the directories to QEMU's user was not exercised there; the unit tests cover it. The runbook no longer has a results section, so the outcome is recorded here. The host's daemon setup was restored by §3. + +--- + +## Dependencies & Execution Order + +### Phase dependencies + +- **Setup (T001 → T002)** → everything; T002, the approval, gates every later task. +- **Foundational (T003–T006)** → every Go story (US2, US6, US3, US4's T031, US5). T003, T004 and T005 run in parallel; T006 needs T003 and T004. +- **US1 (T007–T010)**: chart only; needs T002 alone, so it can run in parallel with Foundational. +- **US2 (T011–T012)**: needs T005 and T006. +- **US6 (T013–T019)**: needs US2's T012 — `dialWithin` and the `connectLibvirt` rewrite the loop builds on. T017 → T018 → T019. +- **US3 (T020–T028)**: needs US6 — the retry runs in its loop, and `StartVMManager` exists. + - T024 → T025 → T026 (T025 and T026 share `libvirt.go`). + - T024 → T027 → T028. +- **US4 (T029–T033)**: the chart tasks need only T002. T031 needs T012. T032 and T033 touch different files. T033 runs after T010, which edits the same `values.yaml`, schema and README. +- **US5 (T034–T037)**: needs T006. T036 needs T034 and T035 written. +- **Polish (T038–T044)**: after the stories. + - T039 after T010 and T033. + - T041 after all code; T042 after T041. + - T043 before T044. T044 is last — it needs a built image and the host. + +### Story completion order + +US1 → US2 → US6 → US3 → US4 → US5, then Polish. US6 precedes US3 by dependency; both are P1. + +### Within each story + +Tests first, seen failing (RED) — or green from the start for a **lock** — then the implementation, in the order given. Each story ends at a checkpoint where its independent test passes. + +--- + +## Parallel Examples + +```bash +# Foundational — three test files at once: +Task: "T003 TestParseLibvirtURI in pkg/hypervisor/libvirt_uri_test.go" +Task: "T004 TestSocketDialer in pkg/hypervisor/libvirt_dialer_test.go" +Task: "T005 fake libvirtd knobs + serveFakeOnSocket + waitForWithin in pkg/hypervisor/fake_libvirtd_test.go" + +# US1 alongside Foundational (chart vs Go), then its three edits at once after T007: +Task: "T008 dnsPolicy in charts/lightstack/templates/_helpers.tpl" +Task: "T009 dnsPolicy in charts/lightstack/templates/teardown-purge-job.yaml" +Task: "T010 FR-003a docs in values.yaml / values.schema.json / README.md" + +# US6 tests together: +Task: "T013 pkg/hypervisor/libvirt_start_test.go" +Task: "T014 internal/teardown/teardown_test.go" +Task: "T015 internal/nova/no_hypervisor_test.go" +Task: "T016 test/contract/nova/no_hypervisor_test.go" + +# US3 tests together: +Task: "T020 pkg/hypervisor/console_dir_test.go" +Task: "T021 pkg/hypervisor/launch_ownership_test.go" +Task: "T022 internal/nova/hypervisor_ready_test.go" +Task: "T023 cmd/lightstack/readiness_checks_test.go" +``` + +Never run these in parallel — the tasks share a file: + +| File | Tasks | +|---|---| +| `pkg/hypervisor/libvirt.go` | T012, T017, T025, T026 | +| `pkg/hypervisor/libvirt_connect_test.go` | T011, T031 | +| `cmd/lightstack/main.go` | T019, T028, T036 | +| `internal/nova/handlers.go` | T018, T027 | +| the chart's `values.yaml` / `values.schema.json` / `README.md` | T010, T033, T039 | + +--- + +## Implementation Strategy + +### MVP first + +**US1 (T001, T002, T007–T010)**: chart-only and shippable alone. It removes the cluster-IP workaround for an in-cluster database, RGW or Redis. + +### Incremental delivery + +1. **Foundational + US2** (T003–T006, T011–T012): modular-daemon hosts connect with default values, each attempt bounded at 1 s. +2. **US6** (T013–T019): no fake servers, a fault on every create `ERROR`; startup never waits on libvirt; reconnects as before. +3. **US3** (T020–T028): a cold libvirt no longer disables guest starts; guest starts fix directory ownership; readiness tells the truth. +4. **US4 + US5** (T029–T037): an explicit socket through the chart; refused URIs stop startup. +5. **Polish** (T038–T044): docs, gates, mutation check, runbook, host validation. + +Each increment leaves the earlier stories' tests green. + +--- + +## Notes + +- 44 tasks: Setup 2, Foundational 4, US1 4, US2 2, US6 7, US3 9, US4 5, US5 4, Polish 7. +- The user commits. Leave every change in the working tree; never `git commit`. +- Mutation checks and experiments run off the worktree — a scratch clone or `go test -overlay` — because the working tree can be committed from at any moment. +- Error, fault and status contracts are pinned in `contracts/` — check a test's expectation there before inventing a message. diff --git a/go.mod b/go.mod index df5cdecb..c0c68660 100644 --- a/go.mod +++ b/go.mod @@ -22,6 +22,7 @@ require ( github.com/stretchr/testify v1.11.1 github.com/vishvananda/netlink v1.3.1 golang.org/x/crypto v0.57.0 + golang.org/x/sync v0.23.0 golang.org/x/sys v0.48.0 gopkg.in/yaml.v3 v3.0.1 modernc.org/sqlite v1.60.1 @@ -83,7 +84,6 @@ require ( go.uber.org/atomic v1.11.0 // indirect golang.org/x/arch v0.22.0 // indirect golang.org/x/net v0.60.0 // indirect - golang.org/x/sync v0.23.0 // indirect golang.org/x/text v0.42.0 // indirect google.golang.org/protobuf v1.36.10 // indirect modernc.org/libc v1.77.1 // indirect diff --git a/internal/health/health.go b/internal/health/health.go index 574108e4..5020965b 100644 --- a/internal/health/health.go +++ b/internal/health/health.go @@ -1,9 +1,10 @@ // Package health provides Lightstack's thin, honest readiness and liveness signals. // // Readiness is gated on the database being reachable AND all HTTP listeners -// being bound; liveness is dependency-free (process up). It also exposes an -// OpenStack/oslo-compatible /healthcheck alias of readiness. The endpoints are -// unauthenticated and served on the identity listener only. See +// being bound — and, in real mode, on the hypervisor having prepared the console +// socket directory once (SPEC-031); liveness is dependency-free (process up). It +// also exposes an OpenStack/oslo-compatible /healthcheck alias of readiness. The +// endpoints are unauthenticated and served on the identity listener only. See // docs/specs/013-readiness-liveness-probes/. package health @@ -36,7 +37,8 @@ type Check struct { Probe func(ctx context.Context) error } -// DatabaseCheck is the sole readiness-gating check (FR-001, Q1). It wraps the +// DatabaseCheck gates readiness in every mode (FR-001, Q1); real mode adds the +// hypervisor check (cmd/lightstack, readinessChecks; SPEC-031 FR-017). It wraps the // existing database.HealthCheck — a Ping on the given pool, i.e. a live query // round-trip (a `SELECT 1` on SQLite), not a TCP-only probe (Q7). func DatabaseCheck(pool database.DB) Check { @@ -67,7 +69,8 @@ type Checker struct { prevReady bool } -// New builds a Checker over the given gating checks (typically DatabaseCheck()). +// New builds a Checker over the given gating checks: DatabaseCheck(), and in real mode +// the hypervisor check. func New(logger zerolog.Logger, checks ...Check) *Checker { return &Checker{checks: checks, timeout: defaultCheckTimeout, logger: logger} } @@ -95,7 +98,7 @@ func (c *Checker) Ready(ctx context.Context) Result { return c.record(Result{Reason: "starting"}) } - // Run the gating checks (today just the database) concurrently, each under + // Run the gating checks concurrently, each under // its own bounded context — a Check.Probe must return when ctx is done. var ( mu sync.Mutex diff --git a/internal/nova/advanced_actions.go b/internal/nova/advanced_actions.go index 919909d0..1f8e95dd 100644 --- a/internal/nova/advanced_actions.go +++ b/internal/nova/advanced_actions.go @@ -93,16 +93,7 @@ func (svc *Service) SuspendInstance(c *gin.Context, instanceID, projectID string defer cancel() if err := svc.vmManager.SuspendVM(ctx, libvirtDomainID); err != nil { - // On failure, revert to ERROR state. If the rollback - // itself fails, log it — the instance row will then - // disagree with libvirt and require operator attention. - dbCtx, dbCancel := context.WithTimeout(svc.ctx, 5*time.Second) - defer dbCancel() - if _, dbErr := svc.pool.Exec(dbCtx, - "UPDATE instances SET status = $1, task_state = $2 WHERE id = $3", - "ERROR", "", instanceID); dbErr != nil { - log.Error().Err(dbErr).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after suspend failure") - } + svc.markActionFailed(instanceID, "suspend", err) } }() } @@ -166,20 +157,7 @@ func (svc *Service) ResumeInstance(c *gin.Context, instanceID, projectID string) defer cancel() if err := svc.vmManager.ResumeVM(ctx, libvirtDomainID); err != nil { - // The cause has to reach the log: this runs after the 202, so it is the - // only place it can name, e.g., a console socket directory that is not - // ready (FR-017a). - log.Error().Err(err).Str("instance_id", instanceID).Msg("resume failed in libvirt") - // On failure, revert to ERROR state. If the rollback - // itself fails, log it — the instance row will then - // disagree with libvirt and require operator attention. - dbCtx, dbCancel := context.WithTimeout(svc.ctx, 5*time.Second) - defer dbCancel() - if _, dbErr := svc.pool.Exec(dbCtx, - "UPDATE instances SET status = $1, task_state = $2 WHERE id = $3", - "ERROR", "", instanceID); dbErr != nil { - log.Error().Err(dbErr).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after resume failure") - } + svc.markActionFailed(instanceID, "resume", err) } }() } @@ -187,6 +165,22 @@ func (svc *Service) ResumeInstance(c *gin.Context, instanceID, projectID string) c.Status(http.StatusAccepted) } +// markActionFailed records an action that libvirt failed after its 202 — suspend, resume, +// shelve: the cause in the log, the only place it can still be named (e.g. a console socket +// directory that is not ready, FR-017a), and the server in ERROR, so the record never claims a +// state the guest is not in. A failed rollback is logged too: the record then disagrees with +// libvirt and needs an operator. +func (svc *Service) markActionFailed(instanceID, action string, err error) { + log.Error().Err(err).Str("instance_id", instanceID).Msg(action + " failed in libvirt") + ctx, cancel := context.WithTimeout(svc.ctx, 5*time.Second) + defer cancel() + if _, dbErr := svc.pool.Exec(ctx, + "UPDATE instances SET status = $1, task_state = $2 WHERE id = $3", + "ERROR", "", instanceID); dbErr != nil { + log.Error().Err(dbErr).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after " + action + " failure") + } +} + // Shelve shelves an instance (shutdown and save disk image) func (svc *Service) ShelveInstance(c *gin.Context, instanceID, projectID string) { @@ -262,8 +256,10 @@ func (svc *Service) ShelveInstance(c *gin.Context, instanceID, projectID string) ctx, cancel := context.WithTimeout(svc.ctx, grace+common.LifecycleResponseSlack) defer cancel() - // Stop VM - _ = svc.vmManager.StopVM(ctx, libvirtDomainID, grace) + // A guest that could not be stopped is not shelved. + if err := svc.vmManager.StopVM(ctx, libvirtDomainID, grace); err != nil { + svc.markActionFailed(instanceID, "shelve", err) + } // In real mode, would also snapshot disk and delete instance }() } diff --git a/internal/nova/boot_source_test.go b/internal/nova/boot_source_test.go new file mode 100644 index 00000000..accea33f --- /dev/null +++ b/internal/nova/boot_source_test.go @@ -0,0 +1,46 @@ +package nova + +import ( + "net/http" + "strings" + "testing" + + "github.com/gin-gonic/gin" +) + +// A create must name what it boots from. Upstream refuses one that carries neither an imageRef +// nor a block device mapping with 400 "Missing imageRef attribute", before it looks anything up +// (nova/api/openstack/compute/servers.py, _image_from_req_data). Lightstack accepted it, in +// either mode, and left a server that could only end ERROR. +func TestCreateWithoutBootSourceIsRefused(t *testing.T) { + requireDB(t) + routers := []struct { + mode string + r *gin.Engine + }{ + {"stub", newNovaTestRouter()}, + {"real", newNoHypervisorService(t, nil)}, + } + bodies := []struct{ name, fields string }{ + {"no imageRef", ``}, + {"empty imageRef", `,"imageRef":""`}, + {"empty mapping list", `,"block_device_mapping_v2":[]`}, + } + for _, rt := range routers { + for i, b := range bodies { + t.Run(rt.mode+", "+b.name, func(t *testing.T) { + name := "nosource-" + rt.mode + "-" + string(rune('a'+i)) + cleanupServerNamed(t, name) + body := `{"server":{"name":"` + name + `","flavorRef":"m1.small"` + b.fields + `}}` + + code, resp := createServerAs(t, rt.r, seededProjectID, seededUserID, body) + if code != http.StatusBadRequest || !strings.Contains(resp, "Missing imageRef attribute") { + t.Fatalf("create: %d %s; want 400 \"Missing imageRef attribute\"", code, resp) + } + if n := instanceCount(t, name, seededProjectID); n != 0 { + t.Errorf("%d server(s) created despite the refusal", n) + } + }) + } + } +} diff --git a/internal/nova/boot_volume_test.go b/internal/nova/boot_volume_test.go index 8e8c7fff..7798a274 100644 --- a/internal/nova/boot_volume_test.go +++ b/internal/nova/boot_volume_test.go @@ -122,17 +122,23 @@ func countServersNamed(t *testing.T, name string) int { } // seededImageID seeds an active image the seeded project owns and removes it when the test -// ends. Always its own, and private: taking whatever active image the database held raced -// every package that creates or deletes images — another project's private image cannot be -// booted, one deleted mid-test is gone — and a public image leaks into other tests' exact -// listings. +// ends. func seededImageID(t *testing.T) string { + t.Helper() + return seedImageFor(t, seededProjectID) +} + +// seedImageFor seeds an active image projectID owns and removes it when the test ends. Always +// its own, and private: taking whatever active image the database held raced every package +// that creates or deletes images — another project's private image cannot be booted, one +// deleted mid-test is gone — and a public image leaks into other tests' exact listings. +func seedImageFor(t *testing.T, projectID string) string { t.Helper() var id string if err := testPool.QueryRow(t.Context(), `INSERT INTO images (name, project_id, status, visibility, disk_format, container_format) VALUES ('fr025-img', $1, 'active', 'private', 'qcow2', 'bare') RETURNING id::text`, - seededProjectID).Scan(&id); err != nil { + projectID).Scan(&id); err != nil { t.Fatalf("seeding an image: %v", err) } t.Cleanup(func() { diff --git a/internal/nova/console.go b/internal/nova/console.go index fbdb8627..8767b211 100644 --- a/internal/nova/console.go +++ b/internal/nova/console.go @@ -223,9 +223,10 @@ func (svc *Service) revokeConsoleTokens(c *gin.Context, instanceID, reason strin return true } -// errNoHypervisor is real mode without a hypervisor: InitHypervisor failed at startup -// (libvirt unreachable) and Lightstack carried on. An outage, not stub mode. -var errNoHypervisor = errors.New("hypervisor not initialized: libvirt was unreachable when Lightstack started") +// errNoHypervisor is a compute service with no hypervisor wired in. A real-mode server always +// has one — it stops when InitHypervisor fails (SPEC-031 FR-019) — so only a service built +// without InitHypervisor, as tests build them, gets here. Never stub behaviour. +var errNoHypervisor = errors.New("no hypervisor wired in") // consoleSocket reads the guest's live console socket from the hypervisor, with the // domain ID of the run it serves. diff --git a/internal/nova/console_test.go b/internal/nova/console_test.go index 18115fb7..82501e60 100644 --- a/internal/nova/console_test.go +++ b/internal/nova/console_test.go @@ -155,9 +155,9 @@ func TestConsoleStubModeReturns501(t *testing.T) { } } -// Real mode whose hypervisor never came up — InitHypervisor failed at startup and main -// carried on with no manager — is an outage, not stub mode: 500, and no claim that -// there is no guest to display. The token sweep gets the same answer. +// Real mode with no hypervisor wired in — what main used to carry on with after a failed +// InitHypervisor; it now stops instead — is an outage, not stub mode: 500, and no claim +// that there is no guest to display. The token sweep gets the same answer. func TestConsoleWithoutAHypervisorIsAnOutage(t *testing.T) { requireDB(t) r, _ := newConsoleTestRouter(t, "real") // no manager, as after a failed InitHypervisor diff --git a/internal/nova/handlers.go b/internal/nova/handlers.go index 1a5d878c..d5c01d2d 100644 --- a/internal/nova/handlers.go +++ b/internal/nova/handlers.go @@ -202,9 +202,12 @@ func (svc *Service) SetCinderService(cinder CinderService) { svc.cinderSvc = cinder } -// InitHypervisor initializes the hypervisor connection +// InitHypervisor starts the hypervisor. In real mode the connection to libvirt is made in the +// background (hypervisor.StartVMManager), so a manager always exists, connected or not, and +// the console socket directory is prepared after every connection. Only a libvirt_uri +// Lightstack cannot honour fails it. func (svc *Service) InitHypervisor() error { - vmManager, err := hypervisor.NewVMManager(svc.libvirtURI, svc.libvirtMode) + vmManager, err := hypervisor.StartVMManager(svc.libvirtURI, svc.libvirtMode) if err != nil { return fmt.Errorf("failed to initialize hypervisor: %w", err) } @@ -212,14 +215,15 @@ func (svc *Service) InitHypervisor() error { return nil } -// PrepareConsoleSocketDir makes the console socket directory QEMU's (SPEC-021 -// FR-017a). Until it has, the hypervisor refuses every guest start, because each domain -// names its console socket in that directory and libvirt does not prepare it. -func (svc *Service) PrepareConsoleSocketDir(ctx context.Context, dir string) error { +// HypervisorReady is real-mode readiness's hypervisor check (SPEC-031 FR-017): nil once the +// console socket directory has been prepared, which the hypervisor does after it first reaches +// libvirt, and nil from then on, through later libvirt outages (FR-018a). It reads recorded +// state and makes no libvirt call. +func (svc *Service) HypervisorReady(ctx context.Context) error { if svc.vmManager == nil { return errNoHypervisor } - return svc.vmManager.PrepareConsoleSocketDir(ctx, dir) + return svc.vmManager.Ready(ctx) } // startRefusal is the 500 for a guest start the hypervisor refused. A console socket @@ -644,6 +648,23 @@ func (svc *Service) removeBootVolume(volumeID string) { } } +// The faults a create records for the hypervisor's part, curated as allocationFault's are: +// the tenant reads them, and the hypervisor's own error stays in the log line. +const ( + faultHypervisorNotReady = "the hypervisor is not ready to start guests" + faultHypervisorCreate = "the hypervisor could not create the guest" +) + +// createFault is the fault a create records when the hypervisor refused or failed it. A +// console socket directory that is not ready — which is also how a missing libvirt connection +// surfaces before the first preparation — is "not ready"; anything else is a failed create. +func createFault(err error) string { + if errors.Is(err, hypervisor.ErrConsoleSocketDirNotReady) { + return faultHypervisorNotReady + } + return faultHypervisorCreate +} + // markInstanceError flags an instance ERROR (best-effort), used from the // VM-creation goroutine's failure paths. Uses svc.ctx so it still runs even if // the request context was cancelled. @@ -687,6 +708,16 @@ func (svc *Service) CreateServer(c *gin.Context) { return } + // A create must name what it boots from. Upstream refuses one that carries neither an + // imageRef nor a block device mapping before it looks anything up + // (nova/api/openstack/compute/servers.py, _image_from_req_data). Accepting it only left a + // server that could never boot. Lightstack reads block_device_mapping_v2 only. + if req.Server.ImageRef == "" && len(req.Server.BlockDeviceMappingV2) == 0 { + logger.Warn().Msg("refusing server create: neither imageRef nor a block device mapping") + common.SendError(c, common.NewBadRequestError("Missing imageRef attribute")) + return + } + // Adopt the image from a root block_device_mapping_v2 (source_type=image) when the // caller sent no imageRef — Horizon's launch wizard does this by default. Done once, // here, so every downstream reader (the instances row, the libvirt boot path, the @@ -1104,11 +1135,7 @@ func (svc *Service) CreateServer(c *gin.Context) { defer func() { if r := recover(); r != nil { log.Error().Interface("panic", r).Str("instance_id", instanceID).Msg("PANIC in VM creation goroutine") - if _, err := svc.pool.Exec(context.Background(), - "UPDATE instances SET status = $1, updated_at = $2 WHERE id = $3", - "ERROR", time.Now(), instanceID); err != nil { - log.Error().Err(err).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after VM-creation panic") - } + svc.markInstanceError(instanceID, "the server could not be created") } }() @@ -1257,11 +1284,7 @@ func (svc *Service) CreateServer(c *gin.Context) { // fails to start the domain (SPEC-009 WS-5). if err := svc.vmManager.PrepareInstanceDir(instanceID); err != nil { log.Error().Err(err).Str("instance_id", instanceID).Msg("boot-from-volume: failed to create instance dir") - if _, dbErr := svc.pool.Exec(svc.ctx, - "UPDATE instances SET status = $1, updated_at = $2 WHERE id = $3", - "ERROR", time.Now(), instanceID); dbErr != nil { - log.Error().Err(dbErr).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after instance-dir failure") - } + svc.markInstanceError(instanceID, "the per-instance directory could not be prepared") return } // Record the attachment first, then mark the volume in-use. Both must @@ -1287,11 +1310,7 @@ func (svc *Service) CreateServer(c *gin.Context) { parsedImageUUID, err := uuid.Parse(req.Server.ImageRef) if err != nil { log.Error().Err(err).Str("image_ref", req.Server.ImageRef).Str("instance_id", instanceID).Msg("Invalid imageRef — not a valid UUID") - if _, dbErr := svc.pool.Exec(svc.ctx, - "UPDATE instances SET status = $1, updated_at = $2 WHERE id = $3", - "ERROR", time.Now(), instanceID); dbErr != nil { - log.Error().Err(dbErr).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after invalid imageRef") - } + svc.markInstanceError(instanceID, "the image reference is not a valid image ID") return } imageUUID := parsedImageUUID.String() @@ -1307,11 +1326,7 @@ func (svc *Service) CreateServer(c *gin.Context) { "SELECT COALESCE(checksum, '') FROM images WHERE id = $1", imageUUID).Scan(&checksum) if _, ferr := svc.imageStore.EnsureLocalBase(svc.ctx, imageUUID, checksum); ferr != nil { log.Error().Err(ferr).Str("instance_id", instanceID).Str("image_id", imageUUID).Msg("Failed to materialize image base for boot") - if _, dbErr := svc.pool.Exec(svc.ctx, - "UPDATE instances SET status = $1, updated_at = $2 WHERE id = $3", - "ERROR", time.Now(), instanceID); dbErr != nil { - log.Error().Err(dbErr).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after image fetch failure") - } + svc.markInstanceError(instanceID, "the image could not be made available on this host") return } } @@ -1323,11 +1338,7 @@ func (svc *Service) CreateServer(c *gin.Context) { overlay, stageErr := hypervisor.StageOverlay(instanceID, storage.LocalImagePath(imageUUID)) if stageErr != nil { log.Error().Err(stageErr).Str("instance_id", instanceID).Str("image_id", imageUUID).Msg("Failed to stage per-instance disk overlay") - if _, dbErr := svc.pool.Exec(svc.ctx, - "UPDATE instances SET status = $1, updated_at = $2 WHERE id = $3", - "ERROR", time.Now(), instanceID); dbErr != nil { - log.Error().Err(dbErr).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after overlay staging failure") - } + svc.markInstanceError(instanceID, "the root disk could not be staged") return } imagePath = overlay @@ -1354,14 +1365,7 @@ func (svc *Service) CreateServer(c *gin.Context) { if err != nil { log.Error().Err(err).Str("instance_id", instanceID).Msg("Failed to create VM via libvirt") - // Update instance status to ERROR - dbCtx, dbCancel := context.WithTimeout(svc.ctx, 5*time.Second) - defer dbCancel() - if _, dbErr := svc.pool.Exec(dbCtx, - "UPDATE instances SET status = $1, updated_at = $2 WHERE id = $3", - "ERROR", time.Now(), instanceID); dbErr != nil { - log.Error().Err(dbErr).Str("instance_id", instanceID).Msg("failed to mark instance ERROR after libvirt create failure") - } + svc.markInstanceError(instanceID, createFault(err)) return } @@ -1387,6 +1391,11 @@ func (svc *Service) CreateServer(c *gin.Context) { log.Error().Err(err).Str("instance_id", instanceID).Str("libvirt_uuid", libvirtUUID).Msg("failed to mark instance ACTIVE after libvirt create") } }() + } else if svc.libvirtMode == "real" { + // Real mode never fakes a guest (Constitution IV): with no hypervisor wired in, the + // create fails as it does when libvirt cannot be reached (SPEC-031 FR-019). + logger.Error().Str("instance_id", instanceID).Msg("no hypervisor wired in; the server cannot be created") + svc.markInstanceError(instanceID, faultHypervisorNotReady) } else { logger.Debug().Msg("Stub mode: skipping libvirt VM creation") // In stub mode, immediately set instance to ACTIVE so Terraform polling succeeds diff --git a/internal/nova/hypervisor_ready_test.go b/internal/nova/hypervisor_ready_test.go new file mode 100644 index 00000000..58b3fe7f --- /dev/null +++ b/internal/nova/hypervisor_ready_test.go @@ -0,0 +1,30 @@ +package nova + +import ( + "context" + "path/filepath" + "testing" + + "github.com/cobaltcore-dev/lightstack/pkg/hypervisor" +) + +// SPEC-031 FR-017, FR-018a: real-mode readiness waits for the hypervisor. This package cannot +// produce a prepared manager — the fake libvirtd lives in pkg/hypervisor's tests, which cover +// the prepared and sticky cases (VMManager.Ready) — so it pins the cases it can reach. +func TestHypervisorReady(t *testing.T) { + svc := NewService(nil, "", "real", nil) + if err := svc.HypervisorReady(context.Background()); err == nil { + t.Error("ready with no hypervisor wired in") + } + + dir := t.TempDir() + mgr, err := hypervisor.StartVMManager(filepath.Join(dir, "missing-sock"), "real") + if err != nil { + t.Fatalf("StartVMManager: %v", err) + } + t.Cleanup(mgr.Close) + svc.vmManager = mgr + if err := svc.HypervisorReady(context.Background()); err == nil { + t.Error("ready while libvirt cannot be reached") + } +} diff --git a/internal/nova/no_hypervisor_test.go b/internal/nova/no_hypervisor_test.go new file mode 100644 index 00000000..563ef8fe --- /dev/null +++ b/internal/nova/no_hypervisor_test.go @@ -0,0 +1,347 @@ +package nova + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + + "github.com/cobaltcore-dev/lightstack/pkg/hypervisor" + "github.com/cobaltcore-dev/lightstack/pkg/paths" + "github.com/cobaltcore-dev/lightstack/pkg/storage" +) + +// SPEC-031 FR-019: real mode never reports a server it did not create. Without a libvirt +// connection a create used to take the stub branch and turn ACTIVE with no guest behind it. + +const notReadyFault = "the hypervisor is not ready to start guests" + +// newNoHypervisorService builds a real-mode compute service whose hypervisor is mgr (nil: none +// wired in), with a router like newBootService's. +func newNoHypervisorService(t *testing.T, mgr *hypervisor.VMManager) *gin.Engine { + t.Helper() + _, r := newNoHypervisorServiceAndRouter(t, mgr) + return r +} + +// newNoHypervisorServiceAndRouter is newNoHypervisorService that also hands back the service, +// for a test that wires something more into it. +func newNoHypervisorServiceAndRouter(t *testing.T, mgr *hypervisor.VMManager) (*Service, *gin.Engine) { + t.Helper() + gin.SetMode(gin.TestMode) + svc := NewService(testPool, "", "real", nil) + if mgr != nil { + svc.vmManager = mgr + } + t.Cleanup(svc.Shutdown) + + r := gin.New() + r.Use(func(c *gin.Context) { + if p := c.GetHeader("X-Test-Project"); p != "" { + c.Set("project_id", p) + } + if u := c.GetHeader("X-Test-User"); u != "" { + c.Set("user_id", u) + } + if roles := c.GetHeader("X-Test-Roles"); roles != "" { + c.Set("roles", strings.Split(roles, ",")) + } + }) + svc.RegisterRoutes(&r.RouterGroup) + return svc, r +} + +// unconnectedManager is the hypervisor of a real-mode server whose libvirt socket does not +// exist. The socket is explicit: this package reaches neither the fake libvirtd nor the +// automatic choice's run directory. +func unconnectedManager(t *testing.T) *hypervisor.VMManager { + t.Helper() + dir := t.TempDir() + m, err := hypervisor.StartVMManager(filepath.Join(dir, "missing-sock"), "real") + if err != nil { + t.Fatalf("StartVMManager: %v", err) + } + t.Cleanup(m.Close) + return m +} + +// useTempStatePath keeps per-instance directories out of /var/lib/lightstack for the test. +func useTempStatePath(t *testing.T) { + t.Helper() + prev := paths.Base() + paths.SetBase(t.TempDir()) + t.Cleanup(func() { paths.SetBase(prev) }) +} + +// seedBootableVolume inserts an available, bootable, unattached volume in the seeded project. +// Booting from it reaches the hypervisor without staging an image disk with qemu-img first. +func seedBootableVolume(t *testing.T, name string) string { + t.Helper() + id := uuid.NewString() + if _, err := testPool.Exec(t.Context(), + `INSERT INTO volumes (id, name, project_id, size_gb, status, bootable) + VALUES ($1, $2, $3, 1, 'available', true)`, id, name, seededProjectID); err != nil { + t.Fatalf("seeding volume: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM volume_attachments WHERE volume_id = $1`, id) + _, _ = testPool.Exec(context.Background(), `DELETE FROM volumes WHERE id = $1`, id) + }) + return id +} + +// createFromVolume posts a server booting from volumeID and returns the new server's id. +func createFromVolume(t *testing.T, r *gin.Engine, name, volumeID string) string { + t.Helper() + cleanupServerNamed(t, name) + body := fmt.Sprintf(`{"server":{"name":%q,"flavorRef":"00000000-0000-0000-0000-000000000011",`+ + `"block_device_mapping_v2":[{"uuid":%q,"boot_index":0,"source_type":"volume","destination_type":"volume"}]}}`, + name, volumeID) + code, resp := doReq(r, http.MethodPost, "/v2.1/servers", body, testAuthHdr(seededProjectID, seededUserID)) + if code != http.StatusAccepted { + t.Fatalf("create: %d %s; want 202", code, resp) + } + return createdServerID(t, resp) +} + +// createFromImage posts a server booting from imageID and returns the new server's id. +func createFromImage(t *testing.T, r *gin.Engine, name, imageID string) string { + t.Helper() + cleanupServerNamed(t, name) + body := fmt.Sprintf(`{"server":{"name":%q,"flavorRef":"00000000-0000-0000-0000-000000000011","imageRef":%q}}`, + name, imageID) + code, resp := doReq(r, http.MethodPost, "/v2.1/servers", body, testAuthHdr(seededProjectID, seededUserID)) + if code != http.StatusAccepted { + t.Fatalf("create: %d %s; want 202", code, resp) + } + return createdServerID(t, resp) +} + +// createWithoutRoot posts a server whose only block device mapping is volumeID at boot index 1: +// a mapping, so the create is not refused for naming no boot source, but no root device. +func createWithoutRoot(t *testing.T, r *gin.Engine, name, volumeID string) string { + t.Helper() + cleanupServerNamed(t, name) + body := fmt.Sprintf(`{"server":{"name":%q,"flavorRef":"00000000-0000-0000-0000-000000000011",`+ + `"block_device_mapping_v2":[{"uuid":%q,"boot_index":1,"source_type":"volume","destination_type":"volume"}]}}`, + name, volumeID) + code, resp := doReq(r, http.MethodPost, "/v2.1/servers", body, testAuthHdr(seededProjectID, seededUserID)) + if code != http.StatusAccepted { + t.Fatalf("create: %d %s; want 202", code, resp) + } + return createdServerID(t, resp) +} + +// serverFault reads the fault a server reports. +func serverFault(t *testing.T, r *gin.Engine, id string) (code int, message string) { + t.Helper() + status, resp := doReq(r, http.MethodGet, "/v2.1/servers/"+id, "", testAuthHdr(seededProjectID, seededUserID)) + if status != http.StatusOK { + t.Fatalf("show %s: %d %s", id, status, resp) + } + var body struct { + Server struct { + Fault *struct { + Code int `json:"code"` + Message string `json:"message"` + } `json:"fault"` + } `json:"server"` + } + if err := json.Unmarshal([]byte(resp), &body); err != nil { + t.Fatalf("decoding show: %v", err) + } + if body.Server.Fault == nil { + t.Fatalf("server %s reports no fault; body %s", id, resp) + } + return body.Server.Fault.Code, body.Server.Fault.Message +} + +func TestNoHypervisorCreateEndsError(t *testing.T) { + requireDB(t) + useTempStatePath(t) + + t.Run("unconnected manager", func(t *testing.T) { + r := newNoHypervisorService(t, unconnectedManager(t)) + const name = "spec031-unconnected" + id := createFromVolume(t, r, name, seedBootableVolume(t, name)) + if got := awaitInstanceStatus(t, name, "ERROR"); got != "ERROR" { + t.Fatalf("status %q; want ERROR, never ACTIVE", got) + } + if code, msg := serverFault(t, r, id); code != http.StatusInternalServerError || msg != notReadyFault { + t.Errorf("fault %d %q; want 500 %q", code, msg, notReadyFault) + } + }) + + t.Run("no manager wired in", func(t *testing.T) { + r := newNoHypervisorService(t, nil) + const name = "spec031-no-manager" + id := createFromVolume(t, r, name, seedBootableVolume(t, name)) + if got := awaitInstanceStatus(t, name, "ERROR"); got != "ERROR" { + t.Fatalf("status %q; want ERROR, never ACTIVE", got) + } + if code, msg := serverFault(t, r, id); code != http.StatusInternalServerError || msg != notReadyFault { + t.Errorf("fault %d %q; want 500 %q", code, msg, notReadyFault) + } + }) +} + +// SPEC-031 FR-019, contracts/guest-operations.md C-GUEST-1: a real-mode create that fails before +// the hypervisor is asked also ends ERROR with a fault, and the fault names the step that failed. +func TestCreateFaultNamesTheFailedStep(t *testing.T) { + requireDB(t) + for i, tc := range []struct { + name string + noInstances bool // a regular file where the instances directory belongs + imageStore bool // a local image store, which does not hold the image + fromVolume bool + noRoot bool // mappings that name no root device: a volume at boot index 1 only + want string + }{ + {name: "root disk staging", want: "the root disk could not be staged"}, + {name: "instance directory, volume boot", noInstances: true, fromVolume: true, + want: "the per-instance directory could not be prepared"}, + {name: "instance directory, image boot", noInstances: true, + want: "the per-instance directory could not be prepared"}, + {name: "image not on this host", imageStore: true, + want: "the image could not be made available on this host"}, + // Still reachable: the mappings carry no root device Lightstack boots from (spec, review + // clarification; upstream refuses this with 400 "Block Device Mapping is Invalid"). + {name: "no root device in the mappings", noRoot: true, + want: "the image reference is not a valid image ID"}, + } { + t.Run(tc.name, func(t *testing.T) { + useTempStatePath(t) + svc, r := newNoHypervisorServiceAndRouter(t, unconnectedManager(t)) + if tc.noInstances { + if err := os.WriteFile(paths.InstancesDir(), nil, 0o600); err != nil { + t.Fatal(err) + } + } + if tc.imageStore { + store, err := storage.NewImageStore("local", "images", "", storage.S3Options{}) + if err != nil { + t.Fatalf("NewImageStore: %v", err) + } + svc.SetImageStore(store) + } + + name := fmt.Sprintf("spec031-fault-%d", i) + var id string + switch { + case tc.noRoot: + id = createWithoutRoot(t, r, name, seedBootableVolume(t, name)) + case tc.fromVolume: + id = createFromVolume(t, r, name, seedBootableVolume(t, name)) + default: + id = createFromImage(t, r, name, seededImageID(t)) + } + if got := awaitInstanceStatus(t, name, "ERROR"); got != "ERROR" { + t.Fatalf("status %q; want ERROR", got) + } + if code, msg := serverFault(t, r, id); code != http.StatusInternalServerError || msg != tc.want { + t.Errorf("fault %d %q; want 500 %q", code, msg, tc.want) + } + }) + } +} + +// A server action without a libvirt connection answers as during a lost connection. +func TestNoHypervisorActionAnswersGeneric500(t *testing.T) { + requireDB(t) + r := newNoHypervisorService(t, unconnectedManager(t)) + + var id string + if err := testPool.QueryRow(t.Context(), + `INSERT INTO instances (name, project_id, user_id, flavor_id, status, power_state, libvirt_domain_id) + VALUES ('spec031-stop', $1, $2, (SELECT id FROM flavors WHERE name = 'm1.small'), 'ACTIVE', 1, $3) + RETURNING id::text`, + seededProjectID, seededUserID, uuid.NewString(), + ).Scan(&id); err != nil { + t.Fatalf("seeding instance: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM instances WHERE id = $1`, id) + }) + + code, resp := doReq(r, http.MethodPost, "/v2.1/servers/"+id+"/action", `{"os-stop":null}`, + testAuthHdr(seededProjectID, seededUserID)) + if code != http.StatusInternalServerError || !strings.Contains(resp, "failed to perform") { + t.Errorf("os-stop: %d %s; want 500 \"failed to perform ...\"", code, resp) + } +} + +// The guarantee the remaining nil-manager branches rely on: in real mode InitHypervisor always +// leaves a manager, connected or not; only a refused URI fails it. +func TestInitHypervisorRealModeAlwaysHasAManager(t *testing.T) { + dir := t.TempDir() + svc := NewService(nil, filepath.Join(dir, "missing-sock"), "real", nil) + if err := svc.InitHypervisor(); err != nil { + t.Fatalf("InitHypervisor with no libvirt socket: %v; want an unconnected manager", err) + } + if svc.vmManager == nil { + t.Fatal("real mode left without a hypervisor manager") + } + svc.vmManager.Close() + + refused := NewService(nil, "qemu+tcp://h/system", "real", nil) + if err := refused.InitHypervisor(); err == nil { + t.Error("InitHypervisor accepted qemu+tcp://h/system") + } +} + +// createFault's messages are curated: the hypervisor's own error stays in the log. +func TestCreateFault(t *testing.T) { + if got := createFault(fmt.Errorf("start: %w", hypervisor.ErrConsoleSocketDirNotReady)); got != notReadyFault { + t.Errorf("createFault(console dir not ready) = %q; want %q", got, notReadyFault) + } + if got, want := createFault(errors.New("libvirt: boom")), "the hypervisor could not create the guest"; got != want { + t.Errorf("createFault(other) = %q; want %q", got, want) + } +} + +// An action that answers 202 and only then asks libvirt — suspend, resume, shelve — ends ERROR +// when libvirt fails it, so the record never claims a state the guest is not in. Shelve used to +// drop the stop's failure and read SHELVED over a guest that ran on. +func TestAsyncActionThatFailsInLibvirtEndsError(t *testing.T) { + requireDB(t) + for _, tc := range []struct{ action, from string }{ + {"suspend", "ACTIVE"}, + {"resume", "SUSPENDED"}, + {"shelve", "ACTIVE"}, + } { + t.Run(tc.action, func(t *testing.T) { + r := newNoHypervisorService(t, unconnectedManager(t)) + name := "spec031-async-" + tc.action + var id string + if err := testPool.QueryRow(t.Context(), + `INSERT INTO instances (name, project_id, user_id, flavor_id, status, power_state, libvirt_domain_id) + VALUES ($1, $2, $3, (SELECT id FROM flavors WHERE name = 'm1.small'), $4, 1, $5) + RETURNING id::text`, + name, seededProjectID, seededUserID, tc.from, uuid.NewString(), + ).Scan(&id); err != nil { + t.Fatalf("seeding instance: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM instance_snapshots WHERE instance_id = $1`, id) + _, _ = testPool.Exec(context.Background(), `DELETE FROM instances WHERE id = $1`, id) + }) + + code, resp := doReq(r, http.MethodPost, "/v2.1/servers/"+id+"/action", `{"`+tc.action+`": null}`, + testAuthHdr(seededProjectID, seededUserID)) + if code != http.StatusAccepted { + t.Fatalf("%s: %d %s; want 202", tc.action, code, resp) + } + if got := awaitInstanceStatus(t, name, "ERROR"); got != "ERROR" { + t.Errorf("status %q after a %s libvirt could not perform; want ERROR", got, tc.action) + } + }) + } +} diff --git a/internal/nova/server_detail_keyname_test.go b/internal/nova/server_detail_keyname_test.go index 51c5ecc7..ee8e39d3 100644 --- a/internal/nova/server_detail_keyname_test.go +++ b/internal/nova/server_detail_keyname_test.go @@ -16,7 +16,7 @@ func TestServerDetail_KeyName_RoundTrip(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-kn","flavorRef":"m1.small","key_name":"spec017-key"}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-kn","flavorRef":"m1.small","key_name":"spec017-key"}}`)) if code != http.StatusAccepted { t.Fatalf("create: code=%d body=%s; want 202", code, body) } @@ -65,7 +65,7 @@ func TestServerDetail_KeyName_NullWhenAbsent(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-nokey","flavorRef":"m1.small"}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-nokey","flavorRef":"m1.small"}}`)) if code != http.StatusAccepted { t.Fatalf("create: code=%d body=%s; want 202", code, body) } @@ -94,7 +94,7 @@ func TestCreateServer_UnknownKeyName_400(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-badkey","flavorRef":"m1.small","key_name":"spec017-nope"}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-badkey","flavorRef":"m1.small","key_name":"spec017-nope"}}`)) if code != http.StatusBadRequest { t.Fatalf("create with unknown key_name: code=%d body=%s; want 400", code, body) } diff --git a/internal/nova/server_detail_secgroups_test.go b/internal/nova/server_detail_secgroups_test.go index 57e7ec19..e02d683a 100644 --- a/internal/nova/server_detail_secgroups_test.go +++ b/internal/nova/server_detail_secgroups_test.go @@ -23,7 +23,7 @@ func TestServerDetail_SecurityGroups_RoundTrip(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-sg1","flavorRef":"m1.small","security_groups":[{"name":"default"}]}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-sg1","flavorRef":"m1.small","security_groups":[{"name":"default"}]}}`)) if code != http.StatusAccepted { t.Fatalf("create: code=%d body=%s; want 202", code, body) } @@ -62,7 +62,7 @@ func TestServerDetail_SecurityGroups_DefaultWhenNone(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-sgnone","flavorRef":"m1.small"}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-sgnone","flavorRef":"m1.small"}}`)) if code != http.StatusAccepted { t.Fatalf("create: code=%d body=%s", code, body) } @@ -85,7 +85,7 @@ func TestServerDetail_SecurityGroups_Multiple(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-sgmulti","flavorRef":"m1.small","security_groups":[{"name":"default"},{"name":"spec017-sg2"}]}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-sgmulti","flavorRef":"m1.small","security_groups":[{"name":"default"},{"name":"spec017-sg2"}]}}`)) if code != http.StatusAccepted { t.Fatalf("create: code=%d body=%s", code, body) } @@ -108,7 +108,7 @@ func TestServerDetail_SecurityGroups_OrphanDropped(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-sgorph","flavorRef":"m1.small","security_groups":[{"name":"spec017-sgorphan"}]}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-sgorph","flavorRef":"m1.small","security_groups":[{"name":"spec017-sgorphan"}]}}`)) if code != http.StatusAccepted { t.Fatalf("create: code=%d body=%s", code, body) } @@ -139,7 +139,7 @@ func TestCreateServer_UnknownSecurityGroup_400(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-sgbad","flavorRef":"m1.small","security_groups":[{"name":"spec017-nope"}]}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-sgbad","flavorRef":"m1.small","security_groups":[{"name":"spec017-nope"}]}}`)) if code != http.StatusBadRequest { t.Fatalf("create with unknown SG: code=%d body=%s; want 400", code, body) } @@ -161,7 +161,7 @@ func TestCreateServer_EnsureDefault_FreshProject(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, freshProjectID, seededUserID, - `{"server":{"name":"spec017-fresh","flavorRef":"m1.small","security_groups":[{"name":"default"}]}}`) + withImage(t, freshProjectID, `{"server":{"name":"spec017-fresh","flavorRef":"m1.small","security_groups":[{"name":"default"}]}}`)) if code != http.StatusAccepted { t.Fatalf("create in fresh project: code=%d body=%s; want 202 (ensure default)", code, body) } @@ -193,7 +193,7 @@ func TestCreateServer_ImplicitDefault_FreshProject(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, freshProjectID, seededUserID, - `{"server":{"name":"spec017-implicit","flavorRef":"m1.small"}}`) + withImage(t, freshProjectID, `{"server":{"name":"spec017-implicit","flavorRef":"m1.small"}}`)) if code != http.StatusAccepted { t.Fatalf("implicit create in fresh project: code=%d body=%s; want 202", code, body) } @@ -222,7 +222,7 @@ func TestCreateServer_EmptySecurityGroupName_400(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-sgempty","flavorRef":"m1.small","security_groups":[{"name":""}]}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-sgempty","flavorRef":"m1.small","security_groups":[{"name":""}]}}`)) if code != http.StatusBadRequest { t.Fatalf("create with empty SG name: code=%d body=%s; want 400", code, body) } @@ -238,7 +238,7 @@ func TestCreateServer_NamelessSecurityGroupItem_Ignored(t *testing.T) { r := newNovaTestRouter() code, body := createServerAs(t, r, seededProjectID, seededUserID, - `{"server":{"name":"spec017-sgnameless","flavorRef":"m1.small","security_groups":[{}]}}`) + withImage(t, seededProjectID, `{"server":{"name":"spec017-sgnameless","flavorRef":"m1.small","security_groups":[{}]}}`)) if code != http.StatusAccepted { t.Fatalf("create with nameless SG item: code=%d body=%s; want 202", code, body) } diff --git a/internal/nova/server_image_field_test.go b/internal/nova/server_image_field_test.go index 512d1eb4..12893fbe 100644 --- a/internal/nova/server_image_field_test.go +++ b/internal/nova/server_image_field_test.go @@ -76,8 +76,9 @@ func TestServerResponsesCarryImage(t *testing.T) { } } -// createImageFieldServer creates a server in the seeded project, booted from imageRef -// when it is not empty, and returns its id. The row is removed when the test ends. +// createImageFieldServer creates a server in the seeded project and returns its id: booted +// from imageRef when it is not empty, else from a volume — the boot source that leaves a +// server without an image. The row is removed when the test ends. func createImageFieldServer(t *testing.T, r *gin.Engine, imageRef string) string { t.Helper() name := "imgfield-" + uuid.New().String()[:8] @@ -85,6 +86,11 @@ func createImageFieldServer(t *testing.T, r *gin.Engine, imageRef string) string server := map[string]any{"name": name, "flavorRef": "m1.small"} if imageRef != "" { server["imageRef"] = imageRef + } else { + server["block_device_mapping_v2"] = []map[string]any{{ + "uuid": seedBootableVolume(t, name), "boot_index": 0, + "source_type": "volume", "destination_type": "volume", + }} } req, err := json.Marshal(map[string]any{"server": server}) if err != nil { diff --git a/internal/nova/spec017_helpers_test.go b/internal/nova/spec017_helpers_test.go index e6368ccf..a5568ff2 100644 --- a/internal/nova/spec017_helpers_test.go +++ b/internal/nova/spec017_helpers_test.go @@ -6,6 +6,7 @@ import ( "fmt" "net/http" "slices" + "strings" "testing" "github.com/gin-gonic/gin" @@ -126,6 +127,17 @@ func createServerAs(t *testing.T, r *gin.Engine, projectID, userID, bodyJSON str return doReq(r, http.MethodPost, "/v2.1/servers", bodyJSON, testAuthHdr(projectID, userID)) } +// withImage gives a server-create body the boot source every create must name: an imageRef to +// an image seeded for projectID and removed when the test ends. +func withImage(t *testing.T, projectID, bodyJSON string) string { + t.Helper() + const head = `{"server":{` + if !strings.HasPrefix(bodyJSON, head) { + t.Fatalf("withImage: the body does not start with %s: %s", head, bodyJSON) + } + return fmt.Sprintf(`%s"imageRef":%q,%s`, head, seedImageFor(t, projectID), bodyJSON[len(head):]) +} + // instanceCount returns how many instances exist with the given name in a project. func instanceCount(t *testing.T, name, projectID string) int { t.Helper() diff --git a/internal/nova/start_refusal_test.go b/internal/nova/start_refusal_test.go index d63c3ba4..61d18a27 100644 --- a/internal/nova/start_refusal_test.go +++ b/internal/nova/start_refusal_test.go @@ -30,11 +30,3 @@ func TestStartRefusal(t *testing.T) { t.Errorf("message = %q; a libvirt error must keep the generic message", other.Message) } } - -// With no hypervisor at all there is nothing to prepare; saying so beats a nil deref. -func TestPrepareConsoleSocketDirWithoutAHypervisor(t *testing.T) { - svc := NewService(testPool, "", "real", nil) - if err := svc.PrepareConsoleSocketDir(t.Context(), t.TempDir()); !errors.Is(err, errNoHypervisor) { - t.Errorf("err = %v; want errNoHypervisor", err) - } -} diff --git a/internal/teardown/teardown_test.go b/internal/teardown/teardown_test.go index b98c502a..fcaed166 100644 --- a/internal/teardown/teardown_test.go +++ b/internal/teardown/teardown_test.go @@ -3,12 +3,16 @@ package teardown import ( "context" "errors" + "io" + "net" "os" "path/filepath" "reflect" "strings" "testing" + "time" + "github.com/cobaltcore-dev/lightstack/internal/common" "github.com/cobaltcore-dev/lightstack/pkg/paths" ) @@ -349,3 +353,70 @@ func TestPurgeKeepsBackups(t *testing.T) { t.Errorf("the purge removed a backup: %v", err) } } + +// runWithLibvirtAt runs the purge in real mode against the libvirt socket at path, failing the +// test if it has not returned within 5 seconds. +func runWithLibvirtAt(t *testing.T, path string) (time.Duration, error) { + t.Helper() + cfg := &common.Config{} + cfg.Nova.LibvirtMode = "real" + cfg.Nova.LibvirtURI = path + done := make(chan error, 1) + start := time.Now() + go func() { done <- Run(context.Background(), cfg, Options{DryRun: true}) }() + select { + case err := <-done: + return time.Since(start), err + case <-time.After(5 * time.Second): + t.Fatal("the purge is still waiting on libvirt after 5s") + return 0, nil + } +} + +// SPEC-031 FR-020a: the purge makes one connection attempt and fails closed — at once when the +// socket is missing, within the 1-second bound when nothing answers — and never waits for +// libvirt to come back. +func TestCleanupFailsAtOnceWithoutLibvirtSocket(t *testing.T) { + missing := filepath.Join(t.TempDir(), "missing", "libvirt-sock") + elapsed, err := runWithLibvirtAt(t, missing) + if err == nil { + t.Fatal("the purge succeeded with no libvirt socket") + } + if !strings.Contains(err.Error(), missing) { + t.Errorf("error %q does not name %s", err, missing) + } + if elapsed > time.Second { + t.Errorf("failed after %s; want at once", elapsed) + } +} + +func TestCleanupBoundsASilentDaemon(t *testing.T) { + dir, err := os.MkdirTemp("", "lsock") // short: a unix socket path must fit sun_path + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.RemoveAll(dir) }) + sock := filepath.Join(dir, "libvirt-sock") + l, err := net.Listen("unix", sock) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = l.Close() }) + go func() { // accept and never answer + for { + c, err := l.Accept() + if err != nil { + return + } + go func() { _, _ = io.Copy(io.Discard, c) }() + } + }() + + elapsed, err := runWithLibvirtAt(t, sock) + if err == nil { + t.Fatal("the purge succeeded against a daemon that never answers") + } + if elapsed > 2*time.Second { + t.Errorf("failed after %s; want within about 1s", elapsed) + } +} diff --git a/pkg/hypervisor/console_dir.go b/pkg/hypervisor/console_dir.go index a2bfc7d9..fda60447 100644 --- a/pkg/hypervisor/console_dir.go +++ b/pkg/hypervisor/console_dir.go @@ -11,6 +11,7 @@ import ( "strings" "github.com/cobaltcore-dev/lightstack/pkg/paths" + "github.com/rs/zerolog/log" ) // ErrConsoleSocketDirNotReady refuses a guest start while the console socket directory @@ -20,13 +21,6 @@ import ( // libvirt, so starts are refused before libvirt is asked (FR-017a). var ErrConsoleSocketDirNotReady = errors.New("console socket directory not ready") -// consoleDirState is the outcome of PrepareConsoleSocketDir: the directory, and why it -// is not ready (nil once it is). -type consoleDirState struct { - dir string - err error -} - // dacOwner is the uid:gid libvirt runs QEMU as (its DAC base label). Cached on the // VMManager so per-instance directories can be chowned to it without re-reading libvirt // capabilities on every guest create. @@ -34,45 +28,117 @@ type dacOwner struct { uid, gid int } -// PrepareConsoleSocketDir makes dir the guests' console socket directory: owned by the -// uid:gid libvirt runs QEMU as, which it reports in its capabilities, with mode 0750. -// QEMU can then create and remove its socket there, the kvm group can read it, and -// nobody else gets in. The outcome is recorded, and every start is refused until it -// succeeds. Called at real-mode startup, right after connecting. -func (m *VMManager) PrepareConsoleSocketDir(ctx context.Context, dir string) error { - err := m.prepareConsoleSocketDir(ctx, dir) - m.consoleDir.Store(&consoleDirState{dir: dir, err: err}) +// prepareShared makes the console socket directory — the one every domain names its console +// socket in (paths.ConsoleSocketDir) — the guests': owned by the uid:gid libvirt runs QEMU as, +// which it reports in its capabilities, with mode 0750. QEMU can then create and remove its +// socket there, the kvm group can read it, and nobody else gets in. +// +// It runs one attempt, or joins the one in flight (single-flight). The first success is never +// withdrawn: a later attempt that fails changes nothing in effect, and one that succeeds +// replaces QEMU's user (SPEC-031 FR-015). Guest starts are refused until one has succeeded. +func (m *VMManager) prepareShared(ctx context.Context) error { + _, err, _ := m.prepGroup.Do("console-socket-dir", func() (any, error) { + return nil, m.prepareConsoleSocketDir(ctx, paths.ConsoleSocketDir()) + }) return err } +// prepareConsoleSocketDir is one attempt, logged (SPEC-031 FR-017a): the first failure at ERROR +// with its cause, later ones at DEBUG, the first success at INFO naming the directory and QEMU's +// user, a redo at DEBUG unless QEMU's user changed. func (m *VMManager) prepareConsoleSocketDir(ctx context.Context, dir string) error { + uid, gid, err := m.attemptPrepare(ctx, dir) + if err != nil { + m.logPrepFailure(dir, err) + return err + } + prev := m.qemuOwner.Swap(&dacOwner{uid: uid, gid: gid}) + first := !m.prepared.Swap(true) + logPrepSuccess(dir, fmt.Sprintf("%d:%d", uid, gid), first, prev == nil || prev.uid != uid || prev.gid != gid) + return nil +} + +// logPrepFailure logs the first failure before any success at ERROR, every other at DEBUG. An +// attempt that failed for want of a connection is not that failure: the connection loop reports +// the outage itself, and spending the ERROR line on it would leave the cause that keeps guests +// from starting once libvirt answers at DEBUG. +func (m *VMManager) logPrepFailure(dir string, err error) { + if !errors.Is(err, errNotConnected) && !m.prepared.Load() && m.prepFailureLogged.CompareAndSwap(false, true) { + log.Error().Err(err).Str("dir", dir). + Msg("console socket directory not ready; guest starts are refused until it is") + return + } + log.Debug().Err(err).Str("dir", dir).Msg("console socket directory preparation failed") +} + +// logPrepSuccess logs the first success and a change of QEMU's user at INFO, a plain redo at +// DEBUG. +func logPrepSuccess(dir, owner string, first, ownerChanged bool) { + switch { + case first: + log.Info().Str("dir", dir).Str("qemu_user", owner).Msg("console socket directory ready; guests can start") + case ownerChanged: + log.Info().Str("dir", dir).Str("qemu_user", owner).Msg("console socket directory prepared again for QEMU's new user") + default: + log.Debug().Str("dir", dir).Str("qemu_user", owner).Msg("console socket directory checked again after a reconnect") + } +} + +// errCapsReadPending fails a preparation attempt while an earlier attempt's capabilities read +// is still unanswered. +var errCapsReadPending = errors.New("libvirt has not answered the previous capabilities read") + +// attemptPrepare reads QEMU's user from libvirt's capabilities, under the 1-second bound, and +// makes dir that user's. It records no outcome: the caller records a success. +// +// One read at a time. A read that misses its bound runs on until libvirt answers or the +// connection drops (withDeadline), so an attempt made meanwhile fails at once instead of sending +// another: a daemon that hangs holds one pending call, not one per retry. +func (m *VMManager) attemptPrepare(ctx context.Context, dir string) (uid, gid int, err error) { conn := m.conn.Load() if conn == nil { - return errors.New("not connected to libvirt") + return 0, 0, errNotConnected } - caps, err := withDeadline(ctx, conn.ConnectGetCapabilities) - if err != nil { - return fmt.Errorf("reading libvirt capabilities: %w", err) + if !m.capsReading.CompareAndSwap(false, true) { + return 0, 0, errCapsReadPending } - uid, gid, err := qemuDACOwner(caps) + caps, err := withDeadline(ctx, func() (string, error) { + defer m.capsReading.Store(false) + return conn.ConnectGetCapabilities() + }) if err != nil { - return err + return 0, 0, fmt.Errorf("reading libvirt capabilities: %w", err) + } + if uid, gid, err = qemuDACOwner(caps); err != nil { + return 0, 0, err } - // Cache QEMU's owner so each guest's per-instance directory can be chowned to it on - // create (PrepareInstanceDir), without re-reading capabilities every time. - m.qemuOwner.Store(&dacOwner{uid: uid, gid: gid}) - return ensureOwnedDir(dir, uid, gid) + if err := ensureOwnedDir(dir, uid, gid); err != nil { + return 0, 0, err + } + return uid, gid, nil } -// consoleDirReady reports whether guests may start: nil once PrepareConsoleSocketDir has -// succeeded, otherwise ErrConsoleSocketDirNotReady naming the directory and the cause. -func (m *VMManager) consoleDirReady() error { - s := m.consoleDir.Load() - if s == nil { - return fmt.Errorf("%w: %s has not been prepared", ErrConsoleSocketDirNotReady, paths.ConsoleSocketDir()) +// ensurePrepared lets a guest start go ahead once the console socket directory is QEMU's. If +// it is not yet, the start makes one attempt — or takes the result of the one in flight — +// bounded at 1 second, and is refused with ErrConsoleSocketDirNotReady naming the directory +// and the cause when that fails (SPEC-031 FR-013, FR-016). +func (m *VMManager) ensurePrepared(ctx context.Context) error { + if m.prepared.Load() { + return nil + } + if err := m.prepareShared(ctx); err != nil { + return fmt.Errorf("%w: %s: %v", ErrConsoleSocketDirNotReady, paths.ConsoleSocketDir(), err) } - if s.err != nil { - return fmt.Errorf("%w: %s: %v", ErrConsoleSocketDirNotReady, s.dir, s.err) + return nil +} + +// Ready is real-mode readiness's hypervisor probe (SPEC-031 FR-017): ErrConsoleSocketDirNotReady +// until the console socket directory has been prepared once, nil from then on — through later +// libvirt outages too, since the first success is never withdrawn. It reads recorded state and +// makes no libvirt call. +func (m *VMManager) Ready(context.Context) error { + if !m.prepared.Load() { + return ErrConsoleSocketDirNotReady } return nil } @@ -129,12 +195,14 @@ func qemuDACOwner(capsXML string) (uid, gid int, err error) { // ensureOwnedDir makes dir exist, owned by uid:gid, with mode 0750. Missing parents are // created 0755, not 0750: QEMU must still traverse the storage root to reach its disks. -// The mode is set explicitly because MkdirAll leaves an existing directory's alone. +// The mode is set explicitly because MkdirAll leaves an existing directory's alone. A file +// that is not a directory, where dir belongs, is refused (MkdirAll fails on it) rather than +// chowned in the directory's place. func ensureOwnedDir(dir string, uid, gid int) error { if err := os.MkdirAll(filepath.Dir(dir), 0o755); err != nil { return fmt.Errorf("creating %s: %w", filepath.Dir(dir), err) } - if err := os.Mkdir(dir, 0o750); err != nil && !errors.Is(err, os.ErrExist) { + if err := os.MkdirAll(dir, 0o750); err != nil { return fmt.Errorf("creating %s: %w", dir, err) } if err := os.Chown(dir, uid, gid); err != nil { @@ -153,8 +221,9 @@ func ensureOwnedDir(dir string, uid, gid int) error { // // The goroutine may outlive the call if libvirt is wedged, but it ends with the wedge: // go-libvirt closes every pending call's channel when the socket drops, and a recovered -// libvirtd answers them. Their number is bounded too: console issuance times out as -// well, so no new tokens appear, and the tokens the sweep checks expire within the TTL. +// libvirtd answers them. Their number is bounded too: the console socket directory's +// preparation keeps one read pending at most (attemptPrepare), console issuance times out +// as well, so no new tokens appear, and the tokens the sweep checks expire within the TTL. func withDeadline[T any](ctx context.Context, call func() (T, error)) (T, error) { ctx, cancel := context.WithTimeout(ctx, consoleReadTimeout) defer cancel() diff --git a/pkg/hypervisor/console_dir_test.go b/pkg/hypervisor/console_dir_test.go index 6be7fdd8..1f8ddbea 100644 --- a/pkg/hypervisor/console_dir_test.go +++ b/pkg/hypervisor/console_dir_test.go @@ -1,13 +1,18 @@ package hypervisor import ( + "context" "errors" "fmt" "os" "path/filepath" + "slices" "strings" "syscall" "testing" + "time" + + "github.com/cobaltcore-dev/lightstack/pkg/paths" ) // capsWith is a capabilities document carrying the given secmodels, shaped like the @@ -125,19 +130,20 @@ func TestEnsureOwnedDir(t *testing.T) { }) } -// PrepareConsoleSocketDir reads QEMU's owner from libvirt and records the outcome: -// a prepared directory lets starts through, a failure keeps them refused and says why. +// The preparation reads QEMU's owner from libvirt and records the outcome: a prepared +// directory lets starts through, a failure keeps them refused and says why. func TestPrepareConsoleSocketDir(t *testing.T) { label := fmt.Sprintf("+%d:+%d", os.Getuid(), os.Getgid()) t.Run("prepared", func(t *testing.T) { + useTempBase(t) m := newConnectedManagerWith(t, &fakeLibvirtd{caps: capsWith(apparmorModel + dacModel(label))}) - dir := filepath.Join(t.TempDir(), "console-sockets") - if err := m.PrepareConsoleSocketDir(t.Context(), dir); err != nil { - t.Fatalf("PrepareConsoleSocketDir: %v", err) + dir := paths.ConsoleSocketDir() + if err := m.prepareShared(t.Context()); err != nil { + t.Fatalf("prepareShared: %v", err) } - if err := m.consoleDirReady(); err != nil { - t.Errorf("consoleDirReady after a successful prepare: %v", err) + if err := m.ensurePrepared(t.Context()); err != nil { + t.Errorf("ensurePrepared after a successful prepare: %v", err) } if _, _, mode := statOwnerMode(t, dir); mode != 0o750 { t.Errorf("mode = %o; want 750", mode) @@ -149,10 +155,11 @@ func TestPrepareConsoleSocketDir(t *testing.T) { t.Run("prepared for someone else", func(t *testing.T) { uid, gid := otherOwner(t) qemuLabel := fmt.Sprintf("+%d:+%d", uid, gid) + useTempBase(t) m := newConnectedManagerWith(t, &fakeLibvirtd{caps: capsWith(apparmorModel + dacModel(qemuLabel))}) - dir := filepath.Join(t.TempDir(), "console-sockets") - if err := m.PrepareConsoleSocketDir(t.Context(), dir); err != nil { - t.Fatalf("PrepareConsoleSocketDir: %v", err) + dir := paths.ConsoleSocketDir() + if err := m.prepareShared(t.Context()); err != nil { + t.Fatalf("prepareShared: %v", err) } if gotUID, gotGID, _ := statOwnerMode(t, dir); gotUID != uid || gotGID != gid { t.Errorf("owner = %d:%d; want %d:%d", gotUID, gotGID, uid, gid) @@ -160,14 +167,16 @@ func TestPrepareConsoleSocketDir(t *testing.T) { }) t.Run("no owner reported", func(t *testing.T) { + useTempBase(t) m := newConnectedManagerWith(t, &fakeLibvirtd{caps: capsWith(apparmorModel)}) - dir := filepath.Join(t.TempDir(), "console-sockets") - if err := m.PrepareConsoleSocketDir(t.Context(), dir); err == nil { - t.Fatal("PrepareConsoleSocketDir succeeded without a DAC label to read the owner from") + m.prepared.Store(false) // never prepared + dir := paths.ConsoleSocketDir() + if err := m.prepareShared(t.Context()); err == nil { + t.Fatal("prepareShared succeeded without a DAC label to read the owner from") } - err := m.consoleDirReady() + err := m.ensurePrepared(t.Context()) if !errors.Is(err, ErrConsoleSocketDirNotReady) { - t.Fatalf("consoleDirReady = %v; want ErrConsoleSocketDirNotReady", err) + t.Fatalf("ensurePrepared = %v; want ErrConsoleSocketDirNotReady", err) } if !strings.Contains(err.Error(), dir) { t.Errorf("error %q does not name the directory", err) @@ -180,24 +189,21 @@ func TestPrepareConsoleSocketDir(t *testing.T) { // and a hard reboot never powers a guest off that it then could not start (FR-017a). func TestStartsRefusedUntilConsoleDirReady(t *testing.T) { const id = "11111111-2222-3333-4444-555555555555" - for _, tc := range []struct { - name string - start func(m *VMManager) error - }{ - {"create", func(m *VMManager) error { _, err := m.CreateVM(t.Context(), ""); return err }}, - {"start", func(m *VMManager) error { return m.StartVM(t.Context(), id) }}, - {"hard reboot", func(m *VMManager) error { return m.RebootVM(t.Context(), id, true) }}, - {"resume", func(m *VMManager) error { return m.ResumeVM(t.Context(), id) }}, - } { + for _, tc := range guestStarts(id) { t.Run(tc.name, func(t *testing.T) { - f := &fakeLibvirtd{active: true} + useTempBase(t) + f := &fakeLibvirtd{active: true} // no capabilities: the on-demand attempt fails too m := newConnectedManagerWith(t, f) - m.consoleDir.Store(nil) // never prepared + m.prepared.Store(false) // never prepared + dir := paths.ConsoleSocketDir() err := tc.start(m) if !errors.Is(err, ErrConsoleSocketDirNotReady) { t.Fatalf("err = %v; want ErrConsoleSocketDirNotReady", err) } + if !strings.Contains(err.Error(), dir) || !strings.Contains(err.Error(), "capabilities") { + t.Errorf("err %q; want it to name the directory and the cause (FR-016)", err) + } for _, p := range f.domainCalls() { if p == procDomainCreate || p == procDomainDefineXML || p == procDomainDestroy { t.Errorf("libvirt was asked procedure %d before the refusal", p) @@ -206,3 +212,266 @@ func TestStartsRefusedUntilConsoleDirReady(t *testing.T) { }) } } + +// isReady adapts the readiness probe to waitFor: true once Ready answers nil. +func isReady(m *VMManager) func() bool { + return func() bool { return m.Ready(context.Background()) == nil } +} + +// dacFor is a capabilities document naming uid:gid as the user QEMU runs KVM guests as. +func dacFor(uid, gid int) string { + return capsWith(apparmorModel + dacModel(fmt.Sprintf("+%d:+%d", uid, gid))) +} + +// countLines counts the log lines holding every one of parts. +func countLines(log string, parts ...string) int { + n := 0 + for _, line := range strings.Split(log, "\n") { + all := line != "" + for _, p := range parts { + all = all && strings.Contains(line, p) + } + if all { + n++ + } + } + return n +} + +// SPEC-031 FR-013, FR-017a, FR-018: a first capabilities read held past the 1-second bound — +// a socket-activated daemon probing QEMU on its first call — no longer disables guest starts +// until a restart. The preparation is retried until it succeeds, and the log says when. +func TestPreparationRetriesUntilLibvirtAnswers(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + logs := captureLog(t) + uid, gid := otherOwner(t) + const id = "11111111-2222-3333-4444-555555555555" + domainUUID, err := parseUUID(id) + if err != nil { + t.Fatal(err) + } + f := &fakeLibvirtd{caps: dacFor(uid, gid), capsDelay: 1500 * time.Millisecond, uuid: domainUUID, stopped: true} + serveFakeOnSocket(t, filepath.Join(dir, "virtqemud-sock"), f) + useTempBase(t) + consoleDir := paths.ConsoleSocketDir() + + m, err := StartVMManager("qemu:///system", "real") + if err != nil { + t.Fatalf("StartVMManager: %v", err) + } + t.Cleanup(m.Close) + if err := m.Ready(t.Context()); !errors.Is(err, ErrConsoleSocketDirNotReady) { + t.Fatalf("Ready before libvirt answered: %v; want ErrConsoleSocketDirNotReady", err) + } + waitForWithin(t, "the preparation to succeed", 10*time.Second, isReady(m)) + + if gotUID, gotGID, _ := statOwnerMode(t, consoleDir); gotUID != uid || gotGID != gid { + t.Errorf("directory owner %d:%d; want %d:%d", gotUID, gotGID, uid, gid) + } + out := logs.String() + if n := countLines(out, `"level":"error"`, "console socket directory"); n != 1 { + t.Errorf("%d ERROR lines about the console socket directory; want exactly one, for the first failure:\n%s", n, out) + } + if countLines(out, `"level":"info"`, "console socket directory ready", consoleDir, fmt.Sprintf("%d:%d", uid, gid)) != 1 { + t.Errorf("no INFO line naming the directory and QEMU's user on success:\n%s", out) + } + + // The next guest start goes through, with no restart, into a directory QEMU owns (FR-018). + if err := m.StartVM(context.Background(), id); err != nil { + t.Fatalf("the next guest start: %v", err) + } + if gotUID, gotGID, _ := statOwnerMode(t, InstanceDir(id)); gotUID != uid || gotGID != gid { + t.Errorf("instance directory owner %d:%d; want QEMU's %d:%d", gotUID, gotGID, uid, gid) + } + if !slices.Contains(f.domainCalls(), procDomainCreate) { + t.Errorf("the guest was never started: calls %v", f.domainCalls()) + } +} + +// SPEC-031 FR-013, FR-016 and the v1.5.0 exception: a guest start that finds an attempt in +// flight takes that attempt's result — one bounded wait — instead of waiting and then making +// an attempt of its own. +func TestStartJoinsInFlightAttempt(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + f := &fakeLibvirtd{caps: dacFor(os.Getuid(), os.Getgid()), capsDelay: 3 * time.Second} + serveFakeOnSocket(t, filepath.Join(dir, "virtqemud-sock"), f) + useTempBase(t) + + m, err := StartVMManager("qemu:///system", "real") + if err != nil { + t.Fatalf("StartVMManager: %v", err) + } + t.Cleanup(m.Close) + waitForWithin(t, "the loop's first attempt to be in flight", 10*time.Second, func() bool { return f.capsCalls() == 1 }) + + start := time.Now() + err = m.ensurePrepared(context.Background()) + elapsed := time.Since(start) + if err == nil { + t.Fatal("prepared while the only capabilities read was still held") + } + if elapsed > 1500*time.Millisecond { + t.Errorf("the start waited %s; want at most one 1-second bound", elapsed) + } + if n := f.capsCalls(); n != 1 { + t.Errorf("%d capabilities reads; want the start to share the one in flight", n) + } +} + +// A capabilities read libvirt never answers, on a connection it keeps open, runs on past its +// bound until libvirt answers or the connection drops. The attempts made meanwhile must not send +// reads of their own: a daemon that hangs would otherwise collect one more pending call, and one +// more goroutine, with every retry for as long as it hangs. The read is released with the +// connection, and the next connection reads again. +func TestUnansweredCapabilitiesReadIsNotRepeated(t *testing.T) { + useTempBase(t) + logs := captureLog(t) + f := &fakeLibvirtd{caps: dacFor(os.Getuid(), os.Getgid()), capsSilent: true} + d := &pipeDialer{fake: f} + m := &VMManager{mode: "real", prepares: true, stubVMs: make(map[string]*stubVM)} + if err := m.connect(d, time.Millisecond, 5*time.Millisecond); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(m.Close) + + waitForWithin(t, "three failed attempts", 10*time.Second, func() bool { + return countLines(logs.String(), "console socket directory") >= 3 + }) + if n := f.capsCalls(); n != 1 { + t.Errorf("%d capabilities reads across three attempts; want only the one libvirt has not answered", n) + } + + before := f.capsCalls() + d.drop() + waitForWithin(t, "a read on the next connection", 10*time.Second, func() bool { return f.capsCalls() > before }) +} + +// SPEC-031 FR-015, FR-017, FR-018a: the first success is never withdrawn — not by a lost +// connection, not by a redo that fails — so the readiness probe is ready after it and stays ready +// through a later loss of libvirt; and a redo that succeeds picks up a changed QEMU user. +func TestPreparedSurvivesDisconnect(t *testing.T) { + useTempBase(t) + d := &pipeDialer{fake: &fakeLibvirtd{caps: dacFor(os.Getuid(), os.Getgid())}} + m := &VMManager{mode: "real", prepares: true, stubVMs: make(map[string]*stubVM)} + if err := m.connect(d, time.Millisecond, 5*time.Millisecond); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(m.Close) + waitForWithin(t, "the preparation", 10*time.Second, isReady(m)) + + d.drop() + if err := m.Ready(t.Context()); err != nil { + t.Fatalf("a lost connection withdrew the preparation: %v", err) + } + waitForWithin(t, "the reconnect", 10*time.Second, func() bool { return d.dialCount() >= 2 }) + if err := m.Ready(t.Context()); err != nil { + t.Fatalf("prepared lost across the reconnect: %v", err) + } +} + +func TestRedoAfterReconnect(t *testing.T) { + uid, gid := otherOwner(t) + f := &fakeLibvirtd{caps: dacFor(os.Getuid(), os.Getgid())} + useTempBase(t) + d := &pipeDialer{fake: f} + m := &VMManager{mode: "real", prepares: true, stubVMs: make(map[string]*stubVM)} + dir := paths.ConsoleSocketDir() + if err := m.connect(d, time.Millisecond, 5*time.Millisecond); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(m.Close) + waitForWithin(t, "the preparation", 10*time.Second, isReady(m)) + + // libvirt comes back running QEMU as another user: the redo picks it up. + f.setCaps(dacFor(uid, gid)) + d.drop() + waitForWithin(t, "the redo to replace QEMU's user", 10*time.Second, func() bool { + o := m.qemuOwner.Load() + return o != nil && o.uid == uid && o.gid == gid + }) + waitForWithin(t, "the directory to follow", 10*time.Second, func() bool { + gotUID, gotGID, _ := statOwnerMode(t, dir) + return gotUID == uid && gotGID == gid + }) + + // A redo that fails changes nothing in effect. + before := f.capsCalls() + f.setCaps(capsWith(apparmorModel)) // no DAC label: the redo cannot succeed + d.drop() + waitForWithin(t, "a failed redo", 10*time.Second, func() bool { return f.capsCalls() > before }) + if err := m.Ready(t.Context()); err != nil { + t.Errorf("a failed redo withdrew the preparation: %v", err) + } + if o := m.qemuOwner.Load(); o == nil || o.uid != uid || o.gid != gid { + t.Errorf("a failed redo replaced QEMU's user: %+v", o) + } +} + +// SPEC-031 FR-017a: the one ERROR line belongs to the preparation's first real failure. A guest +// start while libvirt is unreachable fails its attempt for want of a connection, which the +// connection loop reports itself. It must not use the line up, or the cause that keeps guests +// from starting once libvirt answers — here, no DAC label — would be logged at DEBUG only. +func TestPreparationErrorLineNotSpentOnMissingConnection(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + useTempBase(t) + logs := captureLog(t) + + m, err := StartVMManager("qemu:///system", "real") + if err != nil { + t.Fatalf("StartVMManager: %v", err) + } + t.Cleanup(m.Close) + if err := m.ensurePrepared(context.Background()); !errors.Is(err, ErrConsoleSocketDirNotReady) { + t.Fatalf("a start before libvirt is reachable: %v; want ErrConsoleSocketDirNotReady", err) + } + + f := &fakeLibvirtd{caps: capsWith(apparmorModel)} // no DAC label: the preparation cannot succeed + serveFakeOnSocket(t, filepath.Join(dir, "virtqemud-sock"), f) + waitForWithin(t, "an ERROR line naming the missing DAC label", 10*time.Second, func() bool { + return countLines(logs.String(), `"level":"error"`, "DAC base label") == 1 + }) + if n := countLines(logs.String(), `"level":"error"`, "not connected to libvirt"); n != 0 { + t.Errorf("%d ERROR line(s) for the missing connection; the connection loop reports that:\n%s", n, logs.String()) + } +} + +// A preparation whose directory cannot be made QEMU's fails and leaves guests refused: the state +// root is a regular file, or a regular file sits where the directory belongs. The second used to +// pass — the existing "directory" was taken as made, and the file was chowned instead. +func TestPreparationFailsWhenTheDirectoryCannotBeMade(t *testing.T) { + for _, tc := range []struct { + name string + block func(t *testing.T) // puts a regular file in the way + }{ + {"the state root is a file", func(t *testing.T) { + if err := os.Remove(paths.Base()); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(paths.Base(), nil, 0o600); err != nil { + t.Fatal(err) + } + }}, + {"a file where the directory belongs", func(t *testing.T) { + if err := os.WriteFile(paths.ConsoleSocketDir(), nil, 0o600); err != nil { + t.Fatal(err) + } + }}, + } { + t.Run(tc.name, func(t *testing.T) { + useTempBase(t) + tc.block(t) + m := newConnectedManagerWith(t, &fakeLibvirtd{caps: dacFor(os.Getuid(), os.Getgid())}) + m.prepared.Store(false) // never prepared + + if err := m.prepareShared(t.Context()); err == nil { + t.Fatal("prepared a directory that could not be made") + } + if err := m.Ready(t.Context()); !errors.Is(err, ErrConsoleSocketDirNotReady) { + t.Errorf("Ready = %v; want ErrConsoleSocketDirNotReady", err) + } + }) + } +} diff --git a/pkg/hypervisor/console_endpoint.go b/pkg/hypervisor/console_endpoint.go index b23cbd55..749a24f0 100644 --- a/pkg/hypervisor/console_endpoint.go +++ b/pkg/hypervisor/console_endpoint.go @@ -14,8 +14,8 @@ import ( // consoleReadTimeout bounds the libvirt console readback. Constitution III caps // external dependencies at 1 second, and go-libvirt's generated calls take no -// context — the dialer's timeout only covers connect, which happens once at -// startup — so the caller has to impose this itself (SPEC-021 FR-007). +// context — connectAttemptTimeout bounds only the connection attempts, not the calls +// made on an open connection — so the caller has to impose this itself (SPEC-021 FR-007). const consoleReadTimeout = time.Second // Console readback failures, each mapping to a distinct API response so the @@ -75,7 +75,7 @@ func (m *VMManager) GetConsoleSocket(ctx context.Context, vmUUID string) (string } conn := m.conn.Load() if conn == nil { - return "", 0, errors.New("not connected to libvirt") + return "", 0, errNotConnected } console, err := withDeadline(ctx, func() (domainConsole, error) { return fetchDomainConsole(conn, vmUUID) }) diff --git a/pkg/hypervisor/disk.go b/pkg/hypervisor/disk.go index e5a776fd..34574cfa 100644 --- a/pkg/hypervisor/disk.go +++ b/pkg/hypervisor/disk.go @@ -31,13 +31,19 @@ func EnsureInstanceDir(instanceID string) error { } // PrepareInstanceDir creates the per-instance directory and, in real mode, makes it owned -// by the user libvirt runs QEMU as (cached from capabilities at startup). QEMU opens each -// guest's serial console.log — the chardev — itself, so the directory must be -// writable by that user. Older libvirt opened that log as root in the daemon, which masked -// the gap; the modular virtqemud does not, so a root-owned directory makes the domain fail -// to start ("console.log: Permission denied"). Lightstack keeps write access to stage the -// overlay because root is exempt from the directory's mode. Stub mode — or real mode before -// capabilities have been read — just creates the directory. Idempotent. +// by the user libvirt runs QEMU as (read from its capabilities by the console socket +// directory's preparation). QEMU opens each guest's serial console.log — the +// chardev — itself, so the directory must be writable by that user. Older libvirt opened +// that log as root in the daemon, which masked the gap; the modular virtqemud does not, so a +// root-owned directory makes the domain fail to start ("console.log: Permission denied"). +// Lightstack keeps write access to stage the overlay because root is exempt from the +// directory's mode. Stub mode — or real mode before QEMU's user is known — just creates the +// directory. Idempotent. +// +// Every guest start calls it immediately before QEMU launches, once the preparation has +// succeeded (SPEC-031 FR-014): a directory created while QEMU's user was unknown, left +// root-owned by an earlier release, or owned by a QEMU user that has since changed is made +// QEMU's then. func (m *VMManager) PrepareInstanceDir(instanceID string) error { dir := InstanceDir(instanceID) if err := os.MkdirAll(dir, 0o755); err != nil { @@ -48,9 +54,9 @@ func (m *VMManager) PrepareInstanceDir(instanceID string) error { } owner := m.qemuOwner.Load() if owner == nil { - // Capabilities were never read (startup prep failed); guest starts are already - // refused by consoleDirReady, so creating the directory without the chown is - // harmless — it never gets used. + // QEMU's user is not known yet. The create paths call this before a guest start, which + // prepares first and then calls it again; nothing launches into this directory before + // it is QEMU's. return nil } if err := os.Chown(dir, owner.uid, owner.gid); err != nil { diff --git a/pkg/hypervisor/disk_test.go b/pkg/hypervisor/disk_test.go index 783ec611..adbd825f 100644 --- a/pkg/hypervisor/disk_test.go +++ b/pkg/hypervisor/disk_test.go @@ -43,7 +43,7 @@ func TestPrepareInstanceDirOwnsDirInRealMode(t *testing.T) { } // The owner PrepareInstanceDir gives the per-instance directory is the one libvirt -// reports for QEMU in its capabilities, read once at startup by PrepareConsoleSocketDir. +// reports for QEMU in its capabilities, read by the console socket directory's preparation. // Without that hand-over the directory keeps this process's owner, and with an // unprivileged QEMU every guest fails to start on its console.log. func TestPrepareInstanceDirTakesOwnerFromCapabilities(t *testing.T) { @@ -54,8 +54,8 @@ func TestPrepareInstanceDirTakesOwnerFromCapabilities(t *testing.T) { uid, gid := otherOwner(t) label := fmt.Sprintf("+%d:+%d", uid, gid) m := newConnectedManagerWith(t, &fakeLibvirtd{caps: capsWith(dacModel(label))}) - if err := m.PrepareConsoleSocketDir(t.Context(), paths.ConsoleSocketDir()); err != nil { - t.Fatalf("PrepareConsoleSocketDir: %v", err) + if err := m.prepareShared(t.Context()); err != nil { + t.Fatalf("prepareShared: %v", err) } if err := m.PrepareInstanceDir("iid"); err != nil { diff --git a/pkg/hypervisor/fake_libvirtd_test.go b/pkg/hypervisor/fake_libvirtd_test.go index 8a1db098..07f1684d 100644 --- a/pkg/hypervisor/fake_libvirtd_test.go +++ b/pkg/hypervisor/fake_libvirtd_test.go @@ -4,11 +4,14 @@ import ( "encoding/binary" "io" "net" + "os" "sync" "testing" "time" "github.com/digitalocean/go-libvirt/socket" + + "github.com/cobaltcore-dev/lightstack/pkg/paths" ) // Remote-protocol procedures (libvirt's remote_protocol.x) the fake libvirtd answers @@ -21,6 +24,7 @@ const ( procDomainDestroy = 12 procDomainLookupByUUID = 24 procDomainReboot = 27 + procDomainUndefine = 35 procAuthList = 66 procDomainIsActive = 150 procConnectGetLibVersion = 157 @@ -35,17 +39,62 @@ const ( // actually asked libvirt to do. A call it has no payload for gets an empty successful // reply. type fakeLibvirtd struct { - active bool // what DomainIsActive reports; set before serving - stopped bool // DomainLookupByUUID reports ID -1, as libvirt does for a domain not running - caps string // what ConnectGetCapabilities reports; set before serving - xml string // what DomainGetXMLDesc reports; set before serving + active bool // what DomainIsActive reports; set before serving + stopped bool // DomainLookupByUUID reports ID -1, as libvirt does for a domain not running + absent bool // DomainLookupByUUID answers "domain not found", as libvirt does for a guest it lacks + caps string // what ConnectGetCapabilities reports; set before serving + xml string // what DomainGetXMLDesc reports; set before serving + uuid [16]byte // the one domain's UUID, in DomainDefineXML's and DomainLookupByUUID's replies + + // capsDelay holds back the first ConnectGetCapabilities reply, across all connections, + // as a socket-activated daemon probing QEMU on its first call does. Later calls on the + // same connection queue behind it, as they would on the daemon. + capsDelay time.Duration + // capsSilent leaves every ConnectGetCapabilities call unanswered, on every connection, + // while the fake goes on reading and answering the rest: a daemon whose capabilities read + // hangs with the connection open. + capsSilent bool + // silentConns is how many connections, the first ones, are read but never answered: a + // daemon that accepted the socket and then hangs. + silentConns int - mu sync.Mutex - calls []uint32 + mu sync.Mutex + calls []uint32 + conns int + capsDelayed bool +} + +// setCaps changes what ConnectGetCapabilities reports from the next call on: libvirt's +// configuration changing between connections, as across a daemon restart. +func (f *fakeLibvirtd) setCaps(caps string) { + f.mu.Lock() + defer f.mu.Unlock() + f.caps = caps +} + +// capsCalls is how many ConnectGetCapabilities calls the fake has received. +func (f *fakeLibvirtd) capsCalls() int { + f.mu.Lock() + defer f.mu.Unlock() + n := 0 + for _, p := range f.calls { + if p == procConnectGetCapabilities { + n++ + } + } + return n } func (f *fakeLibvirtd) serve(conn net.Conn) { defer conn.Close() + f.mu.Lock() + f.conns++ + silent := f.conns <= f.silentConns + f.mu.Unlock() + if silent { + _, _ = io.Copy(io.Discard, conn) + return + } for { // A packet is its length (these 4 bytes included), a 24-byte header — // program, version, procedure, type, serial, status — and a payload. @@ -59,14 +108,30 @@ func (f *fakeLibvirtd) serve(conn net.Conn) { } proc := binary.BigEndian.Uint32(head[12:16]) f.record(proc) + if proc == procConnectGetCapabilities && f.capsSilent { + continue + } + if proc == procConnectGetCapabilities { + f.mu.Lock() + delay := f.capsDelay + if f.capsDelayed { + delay = 0 + } + f.capsDelayed = true + f.mu.Unlock() + time.Sleep(delay) + } - payload := f.payload(proc) + status, payload := uint32(socket.StatusOK), f.payload(proc) + if proc == procDomainLookupByUUID && f.absent { + status, payload = socket.StatusError, noDomainError() + } reply := make([]byte, len(head)+len(payload)) binary.BigEndian.PutUint32(reply[0:4], uint32(len(reply))) copy(reply[4:16], head[4:16]) // program, version, procedure binary.BigEndian.PutUint32(reply[16:20], socket.Reply) copy(reply[20:24], head[20:24]) // the call's serial - binary.BigEndian.PutUint32(reply[24:28], socket.StatusOK) + binary.BigEndian.PutUint32(reply[24:28], status) copy(reply[len(head):], payload) if _, err := conn.Write(reply); err != nil { return @@ -82,10 +147,14 @@ func (f *fakeLibvirtd) payload(proc uint32) []byte { return make([]byte, 8) case procDomainLookupByUUID: if f.stopped { - return fakeDomain(-1) + return f.domain(-1) } - return fakeDomain(1) + return f.domain(1) + case procDomainDefineXML: + return f.domain(-1) // defined, not yet running case procConnectGetCapabilities: + f.mu.Lock() + defer f.mu.Unlock() return xdrString(f.caps) case procDomainGetXMLDesc: return xdrString(f.xml) @@ -99,6 +168,16 @@ func (f *fakeLibvirtd) payload(proc uint32) []byte { return nil } +// noDomainError is the remote_error libvirt replies with for a guest it does not have: code +// VIR_ERR_NO_DOMAIN, the QEMU driver's error domain, a message, and the error level. +func noDomainError() []byte { + b := binary.BigEndian.AppendUint32(nil, 42) // VIR_ERR_NO_DOMAIN + b = binary.BigEndian.AppendUint32(b, 10) // VIR_FROM_QEMU + b = binary.BigEndian.AppendUint32(b, 1) // the optional message is present + b = append(b, xdrString("Domain not found: no domain with matching uuid")...) + return binary.BigEndian.AppendUint32(b, 2) // VIR_ERR_ERROR +} + // xdrString is s as an XDR string: its length, the bytes, then padding to four. func xdrString(s string) []byte { b := binary.BigEndian.AppendUint32(nil, uint32(len(s))) @@ -106,14 +185,14 @@ func xdrString(s string) []byte { return append(b, make([]byte, (4-len(s)%4)%4)...) } -// fakeDomain is the XDR encoding of the one domain the fake knows: its name as an XDR -// string, padded to four bytes, then a 16-byte UUID and the given id. -func fakeDomain(id int32) []byte { +// domain is the XDR encoding of the one domain the fake knows: its name as an XDR +// string, padded to four bytes, then its 16-byte UUID and the given id. +func (f *fakeLibvirtd) domain(id int32) []byte { name := "instance-fake" b := binary.BigEndian.AppendUint32(nil, uint32(len(name))) b = append(b, name...) b = append(b, make([]byte, (4-len(name)%4)%4)...) - b = append(b, make([]byte, 16)...) + b = append(b, f.uuid[:]...) return binary.BigEndian.AppendUint32(b, uint32(id)) } @@ -131,7 +210,8 @@ func (f *fakeLibvirtd) domainCalls() []uint32 { var out []uint32 for _, p := range f.calls { switch p { - case procDomainCreate, procDomainDefineXML, procDomainDestroy, procDomainLookupByUUID, procDomainReboot, procDomainIsActive: + case procDomainCreate, procDomainDefineXML, procDomainDestroy, procDomainUndefine, + procDomainLookupByUUID, procDomainReboot, procDomainIsActive: out = append(out, p) } } @@ -169,6 +249,9 @@ func (d *pipeDialer) drop() { _ = d.server.Close() } +// Dialed names the in-memory pipe, where a socket dialer names its socket. +func (d *pipeDialer) Dialed() string { return "pipe" } + func (d *pipeDialer) dialCount() int { d.mu.Lock() defer d.mu.Unlock() @@ -178,24 +261,77 @@ func (d *pipeDialer) dialCount() int { // waitFor polls cond until it holds, failing the test after two seconds. func waitFor(t *testing.T, what string, cond func() bool) { t.Helper() - deadline := time.Now().Add(2 * time.Second) + waitForWithin(t, what, 2*time.Second, cond) +} + +// waitForWithin polls cond until it holds, failing the test after d. The connection and +// preparation retries need more than waitFor's two seconds: a first reply held past the +// 1-second bound, then a retry, already takes about that long. +func waitForWithin(t *testing.T, what string, d time.Duration, cond func() bool) { + t.Helper() + deadline := time.Now().Add(d) for !cond() { if time.Now().After(deadline) { - t.Fatalf("timed out waiting for %s", what) + t.Fatalf("timed out after %s waiting for %s", d, what) } time.Sleep(time.Millisecond) } } +// socketDir is a fresh directory with a short path. A unix socket path must fit the +// 108-byte sun_path, which t.TempDir's name-derived paths can exceed in a subtest. +func socketDir(t *testing.T) string { + t.Helper() + dir, err := os.MkdirTemp("", "lsock") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.RemoveAll(dir) }) + return dir +} + +// serveFakeOnSocket serves f on a real unix socket at path until the test ends: libvirt's +// socket as a daemon's socket unit provides it. +func serveFakeOnSocket(t *testing.T, path string, f *fakeLibvirtd) { + t.Helper() + l, err := net.Listen("unix", path) + if err != nil { + t.Fatalf("listening on %s: %v", path, err) + } + t.Cleanup(func() { _ = l.Close() }) + go func() { + for { + c, err := l.Accept() + if err != nil { + return + } + go f.serve(c) + } + }() +} + +// useTempBase points the state path at a directory of the test's own until it ends. A guest +// start makes the instance's directory QEMU's before it launches (SPEC-031 FR-014), which +// needs somewhere writable to create it. +func useTempBase(t *testing.T) { + t.Helper() + prev := paths.Base() + paths.SetBase(t.TempDir()) + t.Cleanup(func() { paths.SetBase(prev) }) +} + func newConnectedManager(t *testing.T, d *pipeDialer) *VMManager { t.Helper() m := &VMManager{mode: "real", stubVMs: make(map[string]*stubVM)} if err := m.connect(d, time.Millisecond, 5*time.Millisecond); err != nil { t.Fatalf("connect: %v", err) } - // As main would have at startup: starts are refused until the console socket + // Stop the connection loop with the test: a loop left running keeps dialing, preparing and + // logging into the next test, whose log capture then races with it. + t.Cleanup(m.Close) + // As the connection loop would have: starts are refused until the console socket // directory is QEMU's (FR-017a), which these tests are not about. - m.consoleDir.Store(&consoleDirState{dir: t.TempDir()}) + m.prepared.Store(true) return m } diff --git a/pkg/hypervisor/launch_ownership_test.go b/pkg/hypervisor/launch_ownership_test.go new file mode 100644 index 00000000..4f0f053b --- /dev/null +++ b/pkg/hypervisor/launch_ownership_test.go @@ -0,0 +1,133 @@ +package hypervisor + +import ( + "context" + "errors" + "os" + "slices" + "testing" + + "github.com/cobaltcore-dev/lightstack/pkg/paths" +) + +// guestStart is one way a guest's QEMU process is launched. +type guestStart struct { + name string + start func(m *VMManager) error +} + +// guestStarts are the four of them, for the domain id. Each reaches DomainCreate in a function +// of its own; create also serves self-heal. +func guestStarts(id string) []guestStart { + return []guestStart{ + {"create", func(m *VMManager) error { _, err := m.CreateVM(context.Background(), ""); return err }}, + {"start", func(m *VMManager) error { return m.StartVM(context.Background(), id) }}, + {"hard reboot", func(m *VMManager) error { return m.RebootVM(context.Background(), id, true) }}, + {"resume", func(m *VMManager) error { return m.ResumeVM(context.Background(), id) }}, + } +} + +// SPEC-031 FR-014, FR-018: immediately before every guest start the instance's directory is +// made QEMU's, so QEMU can write the guest's console.log. Before, only the create paths did +// it, and only when QEMU's user was already known: self-heal skipped it, and a directory made +// while the user was unknown, or by an earlier release, stayed root-owned. +func TestGuestStartMakesTheInstanceDirQEMUs(t *testing.T) { + uid, gid := otherOwner(t) + useTempBase(t) + + const id = "11111111-2222-3333-4444-555555555555" + domainUUID, err := parseUUID(id) + if err != nil { + t.Fatal(err) + } + for _, tc := range guestStarts(id) { + t.Run(tc.name, func(t *testing.T) { + dir := InstanceDir(id) + _ = os.RemoveAll(dir) + if err := os.MkdirAll(dir, 0o755); err != nil { // the test's own owner: not QEMU's + t.Fatal(err) + } + f := &fakeLibvirtd{caps: dacFor(uid, gid), uuid: domainUUID, stopped: true} + m := newConnectedManagerWith(t, f) + // Unprepared: the start's single on-demand attempt prepares it first (FR-013). + m.prepared.Store(false) + + if err := tc.start(m); err != nil { + t.Fatalf("%s: %v", tc.name, err) + } + if gotUID, gotGID, _ := statOwnerMode(t, dir); gotUID != uid || gotGID != gid { + t.Errorf("instance directory owner %d:%d; want QEMU's %d:%d", gotUID, gotGID, uid, gid) + } + if !slices.Contains(f.domainCalls(), procDomainCreate) { + t.Errorf("the guest was never started: calls %v", f.domainCalls()) + } + }) + } +} + +// SPEC-031 FR-014, last sentence: a start whose instance directory cannot be made QEMU's does +// not launch the guest. A create undefines the domain it defined, and a hard reboot fails before +// it powers the guest off, so the failure never leaves a guest off and unable to start. +func TestGuestStartRefusedWhenInstanceDirCannotBePrepared(t *testing.T) { + const id = "11111111-2222-3333-4444-555555555555" + domainUUID, err := parseUUID(id) + if err != nil { + t.Fatal(err) + } + for _, tc := range guestStarts(id) { + t.Run(tc.name, func(t *testing.T) { + useTempBase(t) + // A regular file where the instances directory belongs: creating a directory under + // it fails for every user, root included. + if err := os.WriteFile(paths.InstancesDir(), nil, 0o600); err != nil { + t.Fatal(err) + } + f := &fakeLibvirtd{uuid: domainUUID, stopped: true, active: true} + m := newConnectedManagerWith(t, f) + + if err := tc.start(m); err == nil { + t.Fatalf("%s succeeded with an instance directory that cannot be prepared", tc.name) + } + calls := f.domainCalls() + if slices.Contains(calls, procDomainCreate) { + t.Errorf("the guest was launched: calls %v", calls) + } + if slices.Contains(calls, procDomainDestroy) { + t.Errorf("the guest was powered off before the refusal: calls %v", calls) + } + if tc.name == "create" && !slices.Contains(calls, procDomainUndefine) { + t.Errorf("the defined domain was left behind: calls %v", calls) + } + }) + } +} + +// A hard reboot that is refused — an id that is not a UUID, a guest libvirt does not have — +// leaves no instance directory behind: the directory is made QEMU's only for a guest that +// exists, and still before it is powered off (SPEC-031 FR-014). +func TestHardRebootRefusalCreatesNoInstanceDir(t *testing.T) { + for _, tc := range []struct { + name, id string + fake *fakeLibvirtd + want error // nil: any error + }{ + {"an id that is not a UUID", "not-a-uuid", &fakeLibvirtd{}, nil}, + {"a guest libvirt does not have", "11111111-2222-3333-4444-555555555555", &fakeLibvirtd{absent: true}, ErrDomainNotFound}, + } { + t.Run(tc.name, func(t *testing.T) { + useTempBase(t) + m := newConnectedManagerWith(t, tc.fake) + + err := m.RebootVM(context.Background(), tc.id, true) + if err == nil { + t.Fatal("the hard reboot succeeded") + } + if tc.want != nil && !errors.Is(err, tc.want) { + t.Errorf("err = %v; want %v", err, tc.want) + } + if _, statErr := os.Stat(InstanceDir(tc.id)); !errors.Is(statErr, os.ErrNotExist) { + t.Errorf("the refused reboot left %s behind (stat: %v)", InstanceDir(tc.id), statErr) + } + }) + } +} diff --git a/pkg/hypervisor/libvirt.go b/pkg/hypervisor/libvirt.go index 70073de5..db447329 100644 --- a/pkg/hypervisor/libvirt.go +++ b/pkg/hypervisor/libvirt.go @@ -4,15 +4,16 @@ import ( "context" "errors" "fmt" + "net" "sync" "sync/atomic" "time" libvirt "github.com/digitalocean/go-libvirt" "github.com/digitalocean/go-libvirt/socket" - "github.com/digitalocean/go-libvirt/socket/dialers" "github.com/google/uuid" "github.com/rs/zerolog/log" + "golang.org/x/sync/singleflight" ) // ErrDomainNotFound reports that the hypervisor has no guest with the given id. @@ -26,6 +27,11 @@ import ( // Compare with errors.Is; both the stub and real paths wrap it. var ErrDomainNotFound = errors.New("hypervisor: domain not found") +// errNotConnected is a libvirt call made while there is no connection: before the first one +// opens, or while a lost one is re-established. Such a call fails at once; the connection loop +// reports the outage itself (stayConnected, redial). +var errNotConnected = errors.New("not connected to libvirt") + // VMState represents the state of a virtual machine type VMState string @@ -45,23 +51,40 @@ type VMManager struct { libvirtURI string mode string // "stub" or "real" // The current libvirt connection: replaced, never reopened in place, when it - // drops (see keepConnected). + // drops (see stayConnected). Nil until the first connection opens. conn atomic.Pointer[libvirt.Libvirt] mu sync.Mutex stubVMs map[string]*stubVM // For stub mode - // The reconnect loop (real mode only): closing done stops it, and stopped is + // prepares is whether the connection loop prepares the console socket directory after + // every connection. The server's manager does (StartVMManager); a one-shot manager + // (NewVMManager, for the uninstall purge) starts no guests and prepares nothing. + prepares bool + + // The connection loop (real mode only): closing done stops it, and stopped is // closed once it has returned. done chan struct{} stopped chan struct{} closeOnce sync.Once - // consoleDir is whether the console socket directory is QEMU's yet (SPEC-021 - // FR-017a); nil until PrepareConsoleSocketDir has run. Every real start needs it. - consoleDir atomic.Pointer[consoleDirState] - - // qemuOwner is the uid:gid libvirt runs QEMU as (its DAC base label), cached when - // PrepareConsoleSocketDir reads capabilities at startup. Per-instance directories are + // prepared is whether the console socket directory has been made QEMU's (SPEC-021 + // FR-017a) at least once. Set by the first success and never cleared (SPEC-031 + // FR-015): every real guest start needs it, and real-mode readiness waits for it. + prepared atomic.Bool + // prepFailureLogged is whether a failed preparation has been logged at ERROR. Later + // failures go to DEBUG, so a long outage does not flood the log (SPEC-031 FR-017a). + prepFailureLogged atomic.Bool + // prepGroup lets a caller arriving while a preparation attempt runs take that attempt's + // result instead of following it with one of its own: one bounded wait, never two + // (Constitution Article III, v1.5.0; SPEC-031 FR-013, FR-016). + prepGroup singleflight.Group + // capsReading is set while a capabilities read is unanswered. A read that misses its + // 1-second bound runs on until libvirt answers or the connection drops; preparation + // attempts made meanwhile fail at once rather than send another (attemptPrepare). + capsReading atomic.Bool + + // qemuOwner is the uid:gid libvirt runs QEMU as (its DAC base label), cached by each + // successful preparation of the console socket directory. Per-instance directories are // chowned to it so QEMU can create each guest's serial console.log (the // chardev it opens itself); nil until capabilities have been read. qemuOwner atomic.Pointer[dacOwner] @@ -94,22 +117,42 @@ func NewVMManager(libvirtURI, mode string) (*VMManager, error) { return mgr, nil } -// connectLibvirt establishes a connection to libvirt -func (m *VMManager) connectLibvirt() error { - socketPath := "/var/run/libvirt/libvirt-sock" - // Allow override via configured URI: strip scheme to get socket path - if m.libvirtURI != "" && m.libvirtURI != "qemu:///system" { - // For custom URIs (e.g. qemu+unix:///path), use the path component - // For remote URIs use TCP — not supported in this implementation - if len(m.libvirtURI) > 0 { - socketPath = m.libvirtURI - } +// StartVMManager is NewVMManager for the server. In real mode it returns at once, with the +// connection to libvirt still to be made: a background loop dials — each attempt bounded at +// connectAttemptTimeout, the pause between failures growing from reconnectBackoffMin to +// reconnectBackoffMax — and after every connection, the first and each reconnect, prepares +// the console socket directory for guests (prepareShared). Startup never waits on libvirt; +// until it answers, every libvirt call fails as it does during a lost connection. +// +// A libvirt_uri Lightstack cannot honour is the one error (ParseLibvirtURI). Stub mode reads +// no URI and starts nothing. +func StartVMManager(libvirtURI, mode string) (*VMManager, error) { + mgr := &VMManager{ + libvirtURI: libvirtURI, + mode: mode, + stubVMs: make(map[string]*stubVM), + prepares: true, + } + if mode != "real" { + return mgr, nil + } + target, err := ParseLibvirtURI(libvirtURI) + if err != nil { + return nil, err } + mgr.start(newSocketDialer(target, libvirtRunDir, connectAttemptTimeout), reconnectBackoffMin, reconnectBackoffMax) + return mgr, nil +} - return m.connect(dialers.NewLocal( - dialers.WithSocket(socketPath), - dialers.WithLocalTimeout(2*time.Second), - ), reconnectBackoffMin, reconnectBackoffMax) +// connectLibvirt opens the connection nova.libvirt_uri names (ParseLibvirtURI): the socket +// it gives, or the automatic choice between the modular QEMU daemon's socket and libvirtd's, +// made again on every reconnect. +func (m *VMManager) connectLibvirt() error { + target, err := ParseLibvirtURI(m.libvirtURI) + if err != nil { + return err + } + return m.connect(newSocketDialer(target, libvirtRunDir, connectAttemptTimeout), reconnectBackoffMin, reconnectBackoffMax) } const ( @@ -119,76 +162,220 @@ const ( reconnectBackoffMin = time.Second reconnectBackoffMax = 5 * time.Second - // How long Close waits for the reconnect loop, which may be inside a Connect to an + // How long Close waits for the connection loop, which may be inside a Connect to an // unresponsive libvirtd. Leaving it behind beats hanging shutdown. closeWait = 5 * time.Second + + // connectAttemptTimeout bounds one connection attempt, dial and handshake together + // (Constitution Article III: libvirt connection failures, 1 second). + connectAttemptTimeout = time.Second ) -// connect opens the libvirt connection through dialer and keeps it open. -func (m *VMManager) connect(dialer socket.Dialer, minWait, maxWait time.Duration) error { - l, err := dial(dialer) +// libvirtDialer is what the connection code needs of a dialer: go-libvirt's socket.Dialer, and +// the socket it dialed last, for log lines and errors. +type libvirtDialer interface { + socket.Dialer + Dialed() string +} + +// connect opens the libvirt connection through dialer once, bounded, then keeps it open as +// StartVMManager's loop does. This is NewVMManager's contract, for a caller that cannot go on +// without libvirt: the uninstall purge fails closed when this fails. +func (m *VMManager) connect(dialer libvirtDialer, minWait, maxWait time.Duration) error { + l, err := dialWithin(dialer, connectAttemptTimeout) if err != nil { return err } - + log.Info().Str("socket", dialer.Dialed()).Msg("connected to libvirt") m.conn.Store(l) + m.start(dialer, minWait, maxWait) + return nil +} + +// start runs the connection loop in the background until Close. +func (m *VMManager) start(dialer libvirtDialer, minWait, maxWait time.Duration) { m.done = make(chan struct{}) m.stopped = make(chan struct{}) - go m.keepConnected(dialer, minWait, maxWait) - return nil + go m.stayConnected(dialer, minWait, maxWait) } // dial opens a new libvirt connection through dialer. func dial(dialer socket.Dialer) (*libvirt.Libvirt, error) { l := libvirt.NewWithDialer(dialer) if err := l.Connect(); err != nil { - return nil, fmt.Errorf("failed to connect to libvirt: %w", err) + return nil, err } return l, nil } -// keepConnected replaces the libvirt connection whenever it drops. A libvirtd restart — -// a package upgrade, a crash — closes it, and nothing else would ever open it again: -// the pod stays up, /livez checks only the database, and every libvirt call fails -// until Lightstack itself restarts. +// dialResult is what one dial attempt ends with. +type dialResult struct { + l *libvirt.Libvirt + err error +} + +// dialWithin is dial bounded by timeout. go-libvirt puts no deadline on its handshake, so a +// daemon that accepts the connection and never answers would hold the attempt forever. +func dialWithin(dialer libvirtDialer, timeout time.Duration) (*libvirt.Libvirt, error) { + attempt := &dialAttempt{dialer: dialer} + ch := make(chan dialResult, 1) + go func() { + l, err := dial(attempt) + ch <- dialResult{l, err} + }() + select { + case r := <-ch: + return r.l, r.err + case <-time.After(timeout): + } + abandonDial(attempt, ch) + return nil, fmt.Errorf("libvirt at %s did not answer within %s", dialer.Dialed(), timeout) +} + +// abandonDial gives up on an attempt that outlived its bound (dialAttempt.abandon), and +// disconnects a connection whose handshake completed just as the bound ran out, since the +// caller no longer takes it. +func abandonDial(attempt *dialAttempt, ch <-chan dialResult) { + attempt.abandon() + go func() { + if r := <-ch; r.err == nil { + _ = r.l.Disconnect() + } + }() +} + +// dialAttempt is one bounded attempt's view of a dialer. It keeps the connection the attempt +// opened, so that giving the attempt up closes it. Giving up is recorded on the attempt, not the +// dialer, so a connection that opens only afterwards is closed as well: nothing else would ever +// close it, and with a daemon that never answers, its handshake would wait forever. +type dialAttempt struct { + dialer socket.Dialer + + mu sync.Mutex + conn net.Conn + abandoned bool +} + +// Dial opens the attempt's connection, or closes it again if the attempt was given up meanwhile. +func (a *dialAttempt) Dial() (net.Conn, error) { + conn, err := a.dialer.Dial() + if err != nil { + return nil, err + } + a.mu.Lock() + defer a.mu.Unlock() + if a.abandoned { + _ = conn.Close() + return nil, errors.New("connection attempt abandoned") + } + a.conn = conn + return conn, nil +} + +// abandon gives the attempt up. It closes the connection the attempt opened — go-libvirt has no +// deadline on its handshake, and closing the connection under it is what makes a hung handshake +// return — and has Dial close one that opens later. +func (a *dialAttempt) abandon() { + a.mu.Lock() + defer a.mu.Unlock() + a.abandoned = true + if a.conn != nil { + _ = a.conn.Close() + } +} + +// stayConnected keeps a libvirt connection open for as long as the manager runs: it dials +// until a connection opens, prepares the console socket directory for it, waits for it to +// drop, and starts again. A libvirtd restart — a package upgrade, a crash — closes the +// connection, and nothing else would ever open it again: the pod stays up, /livez checks only +// the database, and every libvirt call fails until Lightstack itself restarts. // -// It runs in the background so that no caller waits on a dial: while the connection -// is down, calls fail fast as before, and they succeed again within one backoff step -// of libvirtd returning. A replacement is published only once it is open. Reopening -// the dropped connection in place would race every call still holding it: go-libvirt -// reads a socket's connection state without the lock its Connect writes it under. -func (m *VMManager) keepConnected(dialer socket.Dialer, minWait, maxWait time.Duration) { - defer close(m.stopped) +// It runs in the background so that no caller waits on a dial: while the connection is down, +// calls fail fast, and they succeed again within one backoff step of libvirtd returning. A +// replacement is published only once it is open. Reopening the dropped connection in place +// would race every call still holding it: go-libvirt reads a socket's connection state without +// the lock its Connect writes it under. +func (m *VMManager) stayConnected(dialer libvirtDialer, minWait, maxWait time.Duration) { + var prep sync.WaitGroup + defer func() { + prep.Wait() + close(m.stopped) + }() + firstWait := time.Duration(0) // the first connection is attempted at once for { - select { - case <-m.done: - return - case <-m.conn.Load().Disconnected(): + if conn := m.conn.Load(); conn != nil { + if !m.untilLost(conn, &prep, minWait, maxWait) { + return + } + firstWait = minWait } - log.Warn().Msg("libvirt connection lost; reconnecting") - l, ok := m.redial(dialer, minWait, maxWait) + l, ok := m.redial(dialer, firstWait, minWait, maxWait) if !ok { return } m.conn.Store(l) - log.Info().Msg("libvirt connection re-established") + log.Info().Str("socket", dialer.Dialed()).Msg("connected to libvirt") } } -// redial dials with capped exponential backoff until a connection opens. It reports -// false when the manager is closed first. -func (m *VMManager) redial(dialer socket.Dialer, minWait, maxWait time.Duration) (*libvirt.Libvirt, bool) { +// untilLost prepares the console socket directory for conn in the background, when this +// manager prepares, and waits for conn to drop. It reports false when the manager closes first. +func (m *VMManager) untilLost(conn *libvirt.Libvirt, prep *sync.WaitGroup, minWait, maxWait time.Duration) bool { + if m.prepares { + prep.Go(func() { m.afterConnect(conn, minWait, maxWait) }) + } + select { + case <-m.done: + return false + case <-conn.Disconnected(): + } + log.Warn().Msg("libvirt connection lost; reconnecting") + return true +} + +// afterConnect prepares the console socket directory for a new connection — guests cannot +// start until it is QEMU's (SPEC-021 FR-017a) — retrying, the pause doubling from minWait up +// to maxWait, until an attempt succeeds. It runs after every connection, even when an earlier +// success is in effect, so a change of QEMU's user made while libvirt was down is picked up +// (SPEC-031 FR-015). It stops when the connection drops, since the next one starts it again, +// or when the manager closes. Each attempt logs its own outcome (prepareConsoleSocketDir). +func (m *VMManager) afterConnect(conn *libvirt.Libvirt, minWait, maxWait time.Duration) { for wait := minWait; ; wait = min(2*wait, maxWait) { + if m.prepareShared(context.Background()) == nil { + return + } + select { + case <-m.done: + return + case <-conn.Disconnected(): + return + case <-time.After(wait): + } + } +} + +// redial dials until a connection opens: first after firstWait, then with the pause doubling +// from minWait up to maxWait. Each attempt is bounded (dialWithin). The first failure is +// logged at WARN, naming the socket or sockets tried; later ones at DEBUG. It reports false +// when the manager is closed first. +func (m *VMManager) redial(dialer libvirtDialer, firstWait, minWait, maxWait time.Duration) (*libvirt.Libvirt, bool) { + wait, next := firstWait, min(max(2*firstWait, minWait), maxWait) + for failures := 0; ; failures++ { select { case <-m.done: return nil, false case <-time.After(wait): } - l, err := dial(dialer) + l, err := dialWithin(dialer, connectAttemptTimeout) if err == nil { return l, true } - log.Debug().Err(err).Dur("next_attempt_in", min(2*wait, maxWait)).Msg("libvirt reconnect failed") + if failures == 0 { + log.Warn().Err(err).Dur("next_attempt_in", next).Msg("libvirt connection attempt failed; retrying in the background") + } else { + log.Debug().Err(err).Dur("next_attempt_in", next).Msg("libvirt connection attempt failed") + } + wait, next = next, min(2*next, maxWait) } } @@ -221,12 +408,12 @@ func (m *VMManager) createVMStub(xml string) (string, error) { func (m *VMManager) createVMReal(ctx context.Context, xml string) (string, error) { // The domain names its console socket in a directory libvirt does not prepare; // refuse here rather than fail inside libvirt (SPEC-021 FR-017a). - if err := m.consoleDirReady(); err != nil { + if err := m.ensurePrepared(ctx); err != nil { return "", err } conn := m.conn.Load() if conn == nil { - return "", fmt.Errorf("not connected to libvirt") + return "", errNotConnected } // Define the domain @@ -235,6 +422,18 @@ func (m *VMManager) createVMReal(ctx context.Context, xml string) (string, error return "", fmt.Errorf("failed to define domain: %w", err) } + // Get the UUID string (domain.UUID is already a [16]byte) + vmUUID := fmt.Sprintf("%08x-%04x-%04x-%04x-%012x", + domain.UUID[0:4], domain.UUID[4:6], domain.UUID[6:8], domain.UUID[8:10], domain.UUID[10:16]) + + // QEMU opens the guest's console.log in the instance's directory itself, so the directory + // must be QEMU's before the guest starts (SPEC-031 FR-014). The domain UUID is the + // instance ID. + if err := m.PrepareInstanceDir(vmUUID); err != nil { + _ = conn.DomainUndefine(domain) + return "", err + } + // Start the domain if err := conn.DomainCreate(domain); err != nil { // Try to undefine on failure @@ -242,10 +441,6 @@ func (m *VMManager) createVMReal(ctx context.Context, xml string) (string, error return "", fmt.Errorf("failed to start domain: %w", err) } - // Get the UUID string (domain.UUID is already a [16]byte) - vmUUID := fmt.Sprintf("%08x-%04x-%04x-%04x-%012x", - domain.UUID[0:4], domain.UUID[4:6], domain.UUID[6:8], domain.UUID[8:10], domain.UUID[10:16]) - return vmUUID, nil } @@ -272,21 +467,9 @@ func (m *VMManager) deleteVMStub(vmUUID string) error { // deleteVMReal deletes a real VM via libvirt func (m *VMManager) deleteVMReal(ctx context.Context, vmUUID string) error { - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - // Parse UUID - uuidBytes, err := parseUUID(vmUUID) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return fmt.Errorf("invalid UUID: %w", err) - } - - // Lookup domain - domain, err := conn.DomainLookupByUUID(uuidBytes) - if err != nil { - return wrapLookupErr(err) + return err } // Get domain state @@ -340,19 +523,9 @@ func (m *VMManager) rebootVMStub(vmUUID string) error { // rebootVMReal reboots a real VM via libvirt func (m *VMManager) rebootVMReal(ctx context.Context, vmUUID string) error { - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) - if err != nil { - return fmt.Errorf("invalid UUID: %w", err) - } - - domain, err := conn.DomainLookupByUUID(uuidBytes) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return wrapLookupErr(err) + return err } if err := conn.DomainReboot(domain, 0); err != nil { @@ -370,22 +543,18 @@ func (m *VMManager) rebootVMReal(ctx context.Context, vmUUID string) error { func (m *VMManager) hardRebootVMReal(vmUUID string) error { // The domain names its console socket in a directory libvirt does not prepare; // refuse here rather than fail inside libvirt (SPEC-021 FR-017a). - if err := m.consoleDirReady(); err != nil { + if err := m.ensurePrepared(context.Background()); err != nil { return err } - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return fmt.Errorf("invalid UUID: %w", err) + return err } - - domain, err := conn.DomainLookupByUUID(uuidBytes) - if err != nil { - return wrapLookupErr(err) + // The instance's directory is made QEMU's once the guest is known to exist, and before it + // is powered off, so a failure here never leaves it off and unable to start (SPEC-031 + // FR-014). + if err := m.PrepareInstanceDir(vmUUID); err != nil { + return err } // Active, not merely running: a paused or suspended guest has a qemu process too, @@ -452,19 +621,9 @@ func (m *VMManager) stopVMStub(vmUUID string) error { // shape as upstream's libvirt driver, which gives the guest shutdown_timeout seconds // and then powers it off. func (m *VMManager) stopVMReal(ctx context.Context, vmUUID string, grace time.Duration) error { - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) - if err != nil { - return fmt.Errorf("invalid UUID: %w", err) - } - - domain, err := conn.DomainLookupByUUID(uuidBytes) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return wrapLookupErr(err) + return err } readState := func() (VMState, error) { @@ -532,22 +691,12 @@ func (m *VMManager) startVMStub(vmUUID string) error { func (m *VMManager) startVMReal(ctx context.Context, vmUUID string) error { // The domain names its console socket in a directory libvirt does not prepare; // refuse here rather than fail inside libvirt (SPEC-021 FR-017a). - if err := m.consoleDirReady(); err != nil { + if err := m.ensurePrepared(ctx); err != nil { return err } - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) - if err != nil { - return fmt.Errorf("invalid UUID: %w", err) - } - - domain, err := conn.DomainLookupByUUID(uuidBytes) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return wrapLookupErr(err) + return err } // Already up: DomainCreate errors on an active domain, and the requested end state @@ -559,6 +708,10 @@ func (m *VMManager) startVMReal(ctx context.Context, vmUUID string) error { return nil } + // QEMU opens the guest's console.log in the instance's directory itself (SPEC-031 FR-014). + if err := m.PrepareInstanceDir(vmUUID); err != nil { + return err + } if err := conn.DomainCreate(domain); err != nil { return fmt.Errorf("failed to start domain: %w", err) } @@ -589,19 +742,9 @@ func (m *VMManager) getVMStateStub(vmUUID string) (string, int, error) { // getVMStateReal returns real VM state via libvirt func (m *VMManager) getVMStateReal(ctx context.Context, vmUUID string) (string, int, error) { - conn := m.conn.Load() - if conn == nil { - return string(VMStateNoState), 0, fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) - if err != nil { - return string(VMStateNoState), 0, fmt.Errorf("invalid UUID: %w", err) - } - - domain, err := conn.DomainLookupByUUID(uuidBytes) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return string(VMStateNoState), 0, wrapLookupErr(err) + return string(VMStateNoState), 0, err } state, _, err := conn.DomainGetState(domain, 0) @@ -638,19 +781,9 @@ func (m *VMManager) attachDeviceStub(vmUUID, deviceXML string) error { // attachDeviceReal attaches a device to a real VM via libvirt func (m *VMManager) attachDeviceReal(ctx context.Context, vmUUID, deviceXML string) error { - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return fmt.Errorf("invalid UUID: %w", err) - } - - domain, err := conn.DomainLookupByUUID(uuidBytes) - if err != nil { - return wrapLookupErr(err) + return err } // Attach device (libvirt flags: 0 for default, 1 for persistent, 2 for live) @@ -685,19 +818,9 @@ func (m *VMManager) detachDeviceStub(vmUUID, deviceXML string) error { // detachDeviceReal detaches a device from a real VM via libvirt func (m *VMManager) detachDeviceReal(ctx context.Context, vmUUID, deviceXML string) error { - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return fmt.Errorf("invalid UUID: %w", err) - } - - domain, err := conn.DomainLookupByUUID(uuidBytes) - if err != nil { - return wrapLookupErr(err) + return err } // Detach device (libvirt flags: 0 for default, 1 for persistent, 2 for live) @@ -734,19 +857,9 @@ func (m *VMManager) suspendVMStub(vmUUID string) error { // suspendVMReal suspends a real VM via libvirt func (m *VMManager) suspendVMReal(ctx context.Context, vmUUID string) error { - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) - if err != nil { - return fmt.Errorf("invalid UUID: %w", err) - } - - domain, err := conn.DomainLookupByUUID(uuidBytes) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return wrapLookupErr(err) + return err } // Suspend domain (saves to disk) @@ -784,25 +897,19 @@ func (m *VMManager) resumeVMStub(vmUUID string) error { func (m *VMManager) resumeVMReal(ctx context.Context, vmUUID string) error { // The domain names its console socket in a directory libvirt does not prepare; // refuse here rather than fail inside libvirt (SPEC-021 FR-017a). - if err := m.consoleDirReady(); err != nil { + if err := m.ensurePrepared(ctx); err != nil { return err } - conn := m.conn.Load() - if conn == nil { - return fmt.Errorf("not connected to libvirt") - } - - uuidBytes, err := parseUUID(vmUUID) + conn, domain, err := m.lookupDomain(vmUUID) if err != nil { - return fmt.Errorf("invalid UUID: %w", err) + return err } - domain, err := conn.DomainLookupByUUID(uuidBytes) - if err != nil { - return wrapLookupErr(err) + // Resume domain (starts from managed save). QEMU opens the guest's console.log in the + // instance's directory itself (SPEC-031 FR-014). + if err := m.PrepareInstanceDir(vmUUID); err != nil { + return err } - - // Resume domain (starts from managed save) if err := conn.DomainCreate(domain); err != nil { return fmt.Errorf("failed to resume domain: %w", err) } @@ -810,7 +917,7 @@ func (m *VMManager) resumeVMReal(ctx context.Context, vmUUID string) error { return nil } -// Close stops the reconnect loop, then closes the libvirt connection. In that order: a +// Close stops the connection loop, then closes the libvirt connection. In that order: a // loop still running would take the deliberate disconnect for a lost connection and // dial again. Safe to call more than once. func (m *VMManager) Close() { @@ -828,6 +935,25 @@ func (m *VMManager) Close() { }) } +// lookupDomain finds the guest vmUUID names on the current connection. It fails with +// errNotConnected without one, "invalid UUID" for an id that is not one, and ErrDomainNotFound +// for a guest libvirt does not have (wrapLookupErr). +func (m *VMManager) lookupDomain(vmUUID string) (*libvirt.Libvirt, libvirt.Domain, error) { + conn := m.conn.Load() + if conn == nil { + return nil, libvirt.Domain{}, errNotConnected + } + uuidBytes, err := parseUUID(vmUUID) + if err != nil { + return nil, libvirt.Domain{}, fmt.Errorf("invalid UUID: %w", err) + } + domain, err := conn.DomainLookupByUUID(uuidBytes) + if err != nil { + return nil, libvirt.Domain{}, wrapLookupErr(err) + } + return conn, domain, nil +} + // wrapLookupErr classifies a domain lookup failure. // // libvirt distinguishes "no such domain" (ERR_NO_DOMAIN) from every other RPC diff --git a/pkg/hypervisor/libvirt_connect_test.go b/pkg/hypervisor/libvirt_connect_test.go new file mode 100644 index 00000000..bedd2089 --- /dev/null +++ b/pkg/hypervisor/libvirt_connect_test.go @@ -0,0 +1,165 @@ +package hypervisor + +import ( + "bytes" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "github.com/rs/zerolog" + "github.com/rs/zerolog/log" +) + +// syncBuffer is a bytes.Buffer safe for the logger's writes from other goroutines. +type syncBuffer struct { + mu sync.Mutex + buf bytes.Buffer +} + +func (b *syncBuffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.Write(p) +} + +func (b *syncBuffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.String() +} + +// captureLog sends the global logger to a buffer for the rest of the test. +func captureLog(t *testing.T) *syncBuffer { + t.Helper() + buf := &syncBuffer{} + prev := log.Logger + log.Logger = zerolog.New(buf) + t.Cleanup(func() { log.Logger = prev }) + return buf +} + +// useRunDir points the automatic socket choice at dir for the rest of the test. +func useRunDir(t *testing.T, dir string) { + t.Helper() + prev := libvirtRunDir + libvirtRunDir = dir + t.Cleanup(func() { libvirtRunDir = prev }) +} + +// With the default URI, Lightstack dials the modular QEMU daemon's socket where only the +// modular daemons run, libvirtd's where it runs, and names the socket it used. +func TestNewVMManagerSocketChoice(t *testing.T) { + for _, sock := range []string{"virtqemud-sock", "libvirt-sock"} { + t.Run("only "+sock, func(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + logs := captureLog(t) + serveFakeOnSocket(t, filepath.Join(dir, sock), &fakeLibvirtd{}) + + m, err := NewVMManager("qemu:///system", "real") + if err != nil { + t.Fatalf("NewVMManager: %v", err) + } + t.Cleanup(m.Close) + if _, err := m.conn.Load().ConnectGetLibVersion(); err != nil { + t.Fatalf("libvirt does not answer: %v", err) + } + if out := logs.String(); !strings.Contains(out, "connected to libvirt") || !strings.Contains(out, filepath.Join(dir, sock)) { + t.Errorf("log does not name the socket dialed (%s):\n%s", filepath.Join(dir, sock), out) + } + }) + } + + t.Run("neither", func(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + _, err := NewVMManager("qemu:///system", "real") + if err == nil { + t.Fatal("NewVMManager succeeded with no socket present") + } + for _, sock := range []string{"virtqemud-sock", "libvirt-sock"} { + if !strings.Contains(err.Error(), filepath.Join(dir, sock)) { + t.Errorf("error %q does not name %s", err, filepath.Join(dir, sock)) + } + } + }) +} + +// A daemon that accepts the connection and never answers must not hold the attempt: it is +// bounded at 1 second, dial and handshake together (Constitution Article III). +func TestNewVMManagerBoundsTheAttempt(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + sock := filepath.Join(dir, "virtqemud-sock") + serveFakeOnSocket(t, sock, &fakeLibvirtd{silentConns: 1}) + + done := make(chan error, 1) + start := time.Now() + go func() { + m, err := NewVMManager("qemu:///system", "real") + if err == nil { + m.Close() + } + done <- err + }() + select { + case err := <-done: + if err == nil { + t.Fatal("NewVMManager succeeded against a daemon that never answers") + } + if elapsed := time.Since(start); elapsed > 2*time.Second { + t.Errorf("gave up after %s; want about 1s", elapsed) + } + if !strings.Contains(err.Error(), sock) { + t.Errorf("error %q does not name %s", err, sock) + } + case <-time.After(5 * time.Second): + t.Fatal("NewVMManager still waiting after 5s on a daemon that never answers") + } +} + +// A socket named in libvirt_uri is the one dialed, even where both automatic sockets exist +// (SPEC-031 FR-008): the automatic choice is only for the default URI. +func TestExplicitSocketWins(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + automatic := []*fakeLibvirtd{{}, {}} + serveFakeOnSocket(t, filepath.Join(dir, "virtqemud-sock"), automatic[0]) + serveFakeOnSocket(t, filepath.Join(dir, "libvirt-sock"), automatic[1]) + custom := filepath.Join(dir, "custom-sock") + explicit := &fakeLibvirtd{} + serveFakeOnSocket(t, custom, explicit) + + conns := func(f *fakeLibvirtd) int { + f.mu.Lock() + defer f.mu.Unlock() + return f.conns + } + for _, uri := range []string{custom, "qemu+unix:///system?socket=" + custom} { + m, err := NewVMManager(uri, "real") + if err != nil { + t.Fatalf("NewVMManager(%q): %v", uri, err) + } + m.Close() + } + if got := conns(explicit); got != 2 { + t.Errorf("the named socket took %d connection(s); want one per URI, 2", got) + } + for i, f := range automatic { + if got := conns(f); got != 0 { + t.Errorf("automatic socket %d was dialed %d time(s) with a socket named", i, got) + } + } +} + +// NewVMManager, the uninstall purge's one attempt, refuses a libvirt_uri Lightstack cannot +// honour as configuration load does, naming it and the accepted forms, without dialing anything. +func TestNewVMManagerRefusesUnsupportedURI(t *testing.T) { + const uri = "qemu+tcp://h/system" + _, err := NewVMManager(uri, "real") + if err == nil || !strings.Contains(err.Error(), uri) || !strings.Contains(err.Error(), "accepted") { + t.Fatalf("NewVMManager(%q) = %v; want a refusal naming the value and the accepted forms", uri, err) + } +} diff --git a/pkg/hypervisor/libvirt_dialer_test.go b/pkg/hypervisor/libvirt_dialer_test.go new file mode 100644 index 00000000..f4017611 --- /dev/null +++ b/pkg/hypervisor/libvirt_dialer_test.go @@ -0,0 +1,192 @@ +package hypervisor + +import ( + "net" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// listenUnix opens a unix listener at path for the rest of the test. Nothing accepts: a +// unix connect completes once the connection is queued, which is all a dial needs. +func listenUnix(t *testing.T, path string) { + t.Helper() + l, err := net.Listen("unix", path) + if err != nil { + t.Fatalf("listening on %s: %v", path, err) + } + t.Cleanup(func() { _ = l.Close() }) +} + +// The automatic socket choice — the modular QEMU daemon's socket when it exists, the +// monolithic or proxy socket otherwise — made afresh on every dial, so a socket that +// appears after Lightstack started is found on the next attempt. +func TestSocketDialer(t *testing.T) { + dial := func(t *testing.T, d *socketDialer) error { + t.Helper() + c, err := d.Dial() + if err == nil { + _ = c.Close() + } + return err + } + auto := LibvirtTarget{} + + t.Run("only virtqemud-sock", func(t *testing.T) { + dir := socketDir(t) + listenUnix(t, filepath.Join(dir, "virtqemud-sock")) + d := newSocketDialer(auto, dir, time.Second) + if err := dial(t, d); err != nil { + t.Fatalf("Dial: %v", err) + } + if got, want := d.Dialed(), filepath.Join(dir, "virtqemud-sock"); got != want { + t.Errorf("dialed %s; want %s", got, want) + } + }) + + t.Run("only libvirt-sock", func(t *testing.T) { + dir := socketDir(t) + listenUnix(t, filepath.Join(dir, "libvirt-sock")) + d := newSocketDialer(auto, dir, time.Second) + if err := dial(t, d); err != nil { + t.Fatalf("Dial: %v", err) + } + if got, want := d.Dialed(), filepath.Join(dir, "libvirt-sock"); got != want { + t.Errorf("dialed %s; want %s", got, want) + } + }) + + t.Run("both prefers virtqemud-sock", func(t *testing.T) { + dir := socketDir(t) + listenUnix(t, filepath.Join(dir, "virtqemud-sock")) + listenUnix(t, filepath.Join(dir, "libvirt-sock")) + d := newSocketDialer(auto, dir, time.Second) + if err := dial(t, d); err != nil { + t.Fatalf("Dial: %v", err) + } + if got, want := d.Dialed(), filepath.Join(dir, "virtqemud-sock"); got != want { + t.Errorf("dialed %s; want %s", got, want) + } + }) + + t.Run("neither names both", func(t *testing.T) { + dir := socketDir(t) + err := dial(t, newSocketDialer(auto, dir, time.Second)) + if err == nil { + t.Fatal("Dial succeeded with no socket present") + } + for _, name := range []string{"virtqemud-sock", "libvirt-sock"} { + if !strings.Contains(err.Error(), filepath.Join(dir, name)) { + t.Errorf("error %q does not name %s", err, filepath.Join(dir, name)) + } + } + }) + + t.Run("a socket that appears later is found on the next dial", func(t *testing.T) { + dir := socketDir(t) + d := newSocketDialer(auto, dir, time.Second) + if err := dial(t, d); err == nil { + t.Fatal("first Dial succeeded with no socket present") + } + listenUnix(t, filepath.Join(dir, "virtqemud-sock")) + if err := dial(t, d); err != nil { + t.Fatalf("second Dial: %v", err) + } + if got, want := d.Dialed(), filepath.Join(dir, "virtqemud-sock"); got != want { + t.Errorf("dialed %s; want %s", got, want) + } + }) + + t.Run("symlinks are followed", func(t *testing.T) { + dir, elsewhere := socketDir(t), socketDir(t) + // libvirt-sock linking to a live socket elsewhere — the host workaround — is used. + listenUnix(t, filepath.Join(elsewhere, "real-sock")) + if err := os.Symlink(filepath.Join(elsewhere, "real-sock"), filepath.Join(dir, "libvirt-sock")); err != nil { + t.Fatal(err) + } + // A dangling virtqemud-sock link is not a socket that exists. + if err := os.Symlink(filepath.Join(elsewhere, "gone"), filepath.Join(dir, "virtqemud-sock")); err != nil { + t.Fatal(err) + } + d := newSocketDialer(auto, dir, time.Second) + if err := dial(t, d); err != nil { + t.Fatalf("Dial: %v", err) + } + if got, want := d.Dialed(), filepath.Join(dir, "libvirt-sock"); got != want { + t.Errorf("dialed %s; want %s", got, want) + } + }) + + t.Run("an explicit socket is dialed whatever else exists", func(t *testing.T) { + dir, elsewhere := socketDir(t), socketDir(t) + listenUnix(t, filepath.Join(dir, "virtqemud-sock")) + listenUnix(t, filepath.Join(dir, "libvirt-sock")) + custom := filepath.Join(elsewhere, "custom-sock") + listenUnix(t, custom) + d := newSocketDialer(LibvirtTarget{Socket: custom}, dir, time.Second) + if err := dial(t, d); err != nil { + t.Fatalf("Dial: %v", err) + } + if got := d.Dialed(); got != custom { + t.Errorf("dialed %s; want %s", got, custom) + } + }) + + t.Run("a missing explicit socket is named", func(t *testing.T) { + custom := filepath.Join(socketDir(t), "custom-sock") + err := dial(t, newSocketDialer(LibvirtTarget{Socket: custom}, socketDir(t), time.Second)) + if err == nil || !strings.Contains(err.Error(), custom) { + t.Errorf("Dial error %v; want one naming %s", err, custom) + } + }) +} + +// delayedDialer connects to a fake libvirtd after delay, over a fresh pipe. A delay past the +// attempt's bound makes a connection that does not exist until the attempt has been given up on. +type delayedDialer struct { + delay time.Duration + fake *fakeLibvirtd + served chan struct{} // closed once the fake stops serving the connection +} + +func (d *delayedDialer) Dial() (net.Conn, error) { + time.Sleep(d.delay) + client, server := net.Pipe() + go func() { + d.fake.serve(server) + close(d.served) + }() + return client, nil +} + +func (d *delayedDialer) Dialed() string { return "delayed-sock" } + +// An attempt that outlives its bound is given up, and its connection closed, not leaked: the +// caller has moved on, and nothing else would ever close it. That holds for a connection whose +// handshake libvirt never answers, and for one that opens only after the attempt was given up on, +// whether libvirt then answers its handshake or not. +func TestDialWithinClosesAnAbandonedConnection(t *testing.T) { + for _, tc := range []struct { + name string + delay time.Duration + fake *fakeLibvirtd + }{ + {"opened in time, the handshake never answered", 0, &fakeLibvirtd{silentConns: 1}}, + {"opened too late, the handshake answered", 300 * time.Millisecond, &fakeLibvirtd{}}, + {"opened too late, the handshake never answered", 300 * time.Millisecond, &fakeLibvirtd{silentConns: 1}}, + } { + t.Run(tc.name, func(t *testing.T) { + d := &delayedDialer{delay: tc.delay, fake: tc.fake, served: make(chan struct{})} + if _, err := dialWithin(d, 100*time.Millisecond); err == nil || !strings.Contains(err.Error(), "delayed-sock") { + t.Fatalf("dialWithin = %v; want a timeout naming the dialer's socket", err) + } + select { + case <-d.served: + case <-time.After(5 * time.Second): + t.Fatal("the abandoned attempt's connection is still open") + } + }) + } +} diff --git a/pkg/hypervisor/libvirt_reboot_test.go b/pkg/hypervisor/libvirt_reboot_test.go index 6caba788..dc58a2c9 100644 --- a/pkg/hypervisor/libvirt_reboot_test.go +++ b/pkg/hypervisor/libvirt_reboot_test.go @@ -9,6 +9,7 @@ const rebootTestUUID = "11111111-2222-3333-4444-555555555555" func rebootAgainst(t *testing.T, f *fakeLibvirtd, hard bool) []uint32 { t.Helper() + useTempBase(t) m := newConnectedManager(t, &pipeDialer{fake: f}) defer m.Close() if err := m.RebootVM(t.Context(), rebootTestUUID, hard); err != nil { diff --git a/pkg/hypervisor/libvirt_reconnect_test.go b/pkg/hypervisor/libvirt_reconnect_test.go index 25cff91f..69527b4a 100644 --- a/pkg/hypervisor/libvirt_reconnect_test.go +++ b/pkg/hypervisor/libvirt_reconnect_test.go @@ -22,7 +22,7 @@ func TestVMManagerReconnectsWhenLibvirtdGoesAway(t *testing.T) { }) } -// Close is a deliberate disconnect. The reconnect loop has to stop before it, or it +// Close is a deliberate disconnect. The connection loop has to stop before it, or it // would treat the shutdown as a lost connection and dial again. func TestVMManagerCloseDoesNotReconnect(t *testing.T) { d := &pipeDialer{} diff --git a/pkg/hypervisor/libvirt_start_test.go b/pkg/hypervisor/libvirt_start_test.go new file mode 100644 index 00000000..2e6c7447 --- /dev/null +++ b/pkg/hypervisor/libvirt_start_test.go @@ -0,0 +1,143 @@ +package hypervisor + +import ( + "context" + "errors" + "path/filepath" + "strings" + "testing" + "time" +) + +// The server's hypervisor starts unconnected and dials in the background: startup never waits +// on libvirt, a socket that appears later is connected without a restart, and until then a +// call fails as it does during a lost connection — never with stub behaviour. +func TestStartVMManagerConnectsWhenSocketAppears(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + useTempBase(t) // the loop prepares the console socket directory once connected + logs := captureLog(t) + + start := time.Now() + m, err := StartVMManager("qemu:///system", "real") + if err != nil { + t.Fatalf("StartVMManager with no libvirt socket: %v; want an unconnected manager", err) + } + t.Cleanup(m.Close) + if elapsed := time.Since(start); elapsed > 500*time.Millisecond { + t.Errorf("StartVMManager took %s; startup must not wait on libvirt", elapsed) + } + if err := m.DeleteVM(context.Background(), "00000000-0000-0000-0000-000000000001"); err == nil || !strings.Contains(err.Error(), "not connected") { + t.Errorf("DeleteVM while unconnected: %v; want a not-connected error", err) + } + // The socket appears only once an attempt has failed without it, so the connection below + // is a later attempt's, not a first dial that happened to run after the socket appeared. + waitForWithin(t, "the first attempt to fail", 5*time.Second, func() bool { + return strings.Contains(logs.String(), "libvirt connection attempt failed") + }) + + serveFakeOnSocket(t, filepath.Join(dir, "virtqemud-sock"), &fakeLibvirtd{}) + waitForWithin(t, "the connection once the socket appears", 10*time.Second, func() bool { + conn := m.conn.Load() + if conn == nil { + return false + } + _, err := conn.ConnectGetLibVersion() + return err == nil + }) +} + +// A daemon that accepts the connection and never answers holds one attempt for 1 second at +// most; the loop abandons it and the next attempt connects. +func TestStartVMManagerAbandonsSilentHandshake(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + useTempBase(t) // the loop prepares the console socket directory once connected + f := &fakeLibvirtd{silentConns: 1} + serveFakeOnSocket(t, filepath.Join(dir, "virtqemud-sock"), f) + + m, err := StartVMManager("qemu:///system", "real") + if err != nil { + t.Fatalf("StartVMManager: %v", err) + } + t.Cleanup(m.Close) + waitForWithin(t, "a connection after the silent one", 10*time.Second, func() bool { + return m.conn.Load() != nil + }) + f.mu.Lock() + conns := f.conns + f.mu.Unlock() + if conns < 2 { + t.Errorf("connected after %d connection(s); want the silent one abandoned and a second made", conns) + } +} + +// Close stops the background dialing: no connection is made afterwards, even once a socket +// appears. +func TestStartVMManagerCloseStopsDialing(t *testing.T) { + dir := socketDir(t) + useRunDir(t, dir) + m, err := StartVMManager("qemu:///system", "real") + if err != nil { + t.Fatalf("StartVMManager: %v", err) + } + time.Sleep(50 * time.Millisecond) + m.Close() + m.Close() // idempotent, as for a connected manager + + select { + case <-m.stopped: + default: + t.Fatal("the connection loop is still running after Close") + } + f := &fakeLibvirtd{} + serveFakeOnSocket(t, filepath.Join(dir, "virtqemud-sock"), f) + time.Sleep(1200 * time.Millisecond) // longer than the first backoff step + f.mu.Lock() + conns := f.conns + f.mu.Unlock() + if conns != 0 { + t.Errorf("dialed %d time(s) after Close; want none", conns) + } +} + +// StartVMManager refuses a libvirt_uri Lightstack cannot honour, as configuration load does. +func TestStartVMManagerRefusesUnsupportedURI(t *testing.T) { + if _, err := StartVMManager("qemu+tcp://h/system", "real"); err == nil { + t.Fatal("StartVMManager accepted qemu+tcp://h/system") + } + // Stub mode reads no URI. + m, err := StartVMManager("qemu+tcp://h/system", "stub") + if err != nil { + t.Fatalf("StartVMManager in stub mode: %v", err) + } + m.Close() +} + +// Until a connection opens — and while a lost one is re-established — every libvirt call fails +// at once with the not-connected error, and nothing answers as stub mode would (SPEC-031 +// FR-019). The manager here is already prepared, so the guest starts get past their refusal and +// meet the missing connection too. +func TestEveryCallFailsWithoutAConnection(t *testing.T) { + m := &VMManager{mode: "real", stubVMs: make(map[string]*stubVM)} + m.prepared.Store(true) + ctx := context.Background() + const id = "11111111-2222-3333-4444-555555555555" + for name, call := range map[string]func() error{ + "create": func() error { _, err := m.CreateVM(ctx, ""); return err }, + "delete": func() error { return m.DeleteVM(ctx, id) }, + "reboot": func() error { return m.RebootVM(ctx, id, false) }, + "hard reboot": func() error { return m.RebootVM(ctx, id, true) }, + "stop": func() error { return m.StopVM(ctx, id, 0) }, + "start": func() error { return m.StartVM(ctx, id) }, + "state": func() error { _, _, err := m.GetVMState(ctx, id); return err }, + "attach": func() error { return m.AttachDevice(ctx, id, "") }, + "detach": func() error { return m.DetachDevice(ctx, id, "") }, + "suspend": func() error { return m.SuspendVM(ctx, id) }, + "resume": func() error { return m.ResumeVM(ctx, id) }, + } { + if err := call(); !errors.Is(err, errNotConnected) { + t.Errorf("%s without a connection: %v; want the not-connected error", name, err) + } + } +} diff --git a/pkg/hypervisor/libvirt_uri.go b/pkg/hypervisor/libvirt_uri.go new file mode 100644 index 00000000..6ab148c9 --- /dev/null +++ b/pkg/hypervisor/libvirt_uri.go @@ -0,0 +1,136 @@ +package hypervisor + +import ( + "fmt" + "net" + "net/url" + "os" + "path/filepath" + "strings" + "sync" + "time" +) + +// LibvirtTarget is where nova.libvirt_uri says to connect: an explicit socket, or, with +// Socket empty, the automatic choice between the modular QEMU daemon's socket and the +// monolithic (or proxy) daemon's socket. +type LibvirtTarget struct { + Socket string +} + +// libvirtRunDir is the directory libvirt's daemons put their sockets in. Distributions that +// use /run link /var/run to it. A variable so tests can point it at a directory of their own. +var libvirtRunDir = "/var/run/libvirt" + +const ( + // virtqemudSocket exists only where libvirt's modular QEMU daemon runs. + virtqemudSocket = "virtqemud-sock" + // libvirtSocket is the monolithic libvirtd's socket, or virtproxyd's on a host that runs + // the modular daemons behind the proxy. Either way it reaches the QEMU driver. + libvirtSocket = "libvirt-sock" + + acceptedLibvirtURIs = "qemu:///system or qemu+unix:///system (chooses virtqemud-sock, else libvirt-sock), " + + "either with ?socket=, or an absolute socket path" +) + +// ParseLibvirtURI reads nova.libvirt_uri. It accepts only what Lightstack can honour: the +// local QEMU system URI, which leaves the socket to the automatic choice, the same URI naming +// its socket, or a bare absolute socket path. Anything else — another driver, a session URI, a +// remote transport or host, a relative path, any other parameter — is refused with one +// message naming the value and the accepted forms, instead of being dialed as a file. +func ParseLibvirtURI(uri string) (LibvirtTarget, error) { + if uri == "" { + return LibvirtTarget{}, nil + } + if strings.HasPrefix(uri, "/") { + return LibvirtTarget{Socket: uri}, nil + } + if u, err := url.Parse(uri); err == nil && isLocalSystemURI(u) { + if socket, ok := socketParam(u.RawQuery); ok { + return LibvirtTarget{Socket: socket}, nil + } + } + return LibvirtTarget{}, fmt.Errorf("nova.libvirt_uri %q is not supported; accepted: %s", uri, acceptedLibvirtURIs) +} + +// isLocalSystemURI reports whether u is qemu:///system or qemu+unix:///system: the QEMU +// driver's system instance on this host, with no user, host or fragment. +func isLocalSystemURI(u *url.URL) bool { + return (u.Scheme == "qemu" || u.Scheme == "qemu+unix") && + u.Opaque == "" && u.User == nil && u.Host == "" && u.Path == "/system" && u.Fragment == "" +} + +// socketParam reads the one parameter a libvirt_uri may carry, socket=, once. +// No parameter gives "": the automatic choice. Any other parameter, or a socket that is +// repeated or not absolute, is not ok. +func socketParam(rawQuery string) (socket string, ok bool) { + query, err := url.ParseQuery(rawQuery) + if err != nil { + return "", false + } + for key, values := range query { + if key != "socket" || len(values) != 1 || !filepath.IsAbs(values[0]) { + return "", false + } + socket = values[0] + } + return socket, true +} + +// socketDialer connects to libvirt's socket for a target. It implements go-libvirt's +// socket.Dialer, which is asked for a connection on every connect and reconnect, so the +// automatic choice is made afresh each time: a daemon whose socket appears after Lightstack +// started is found on the next attempt. +// +// It remembers the socket of its latest attempt, for the log line and errors naming it. Giving up +// on an attempt is the attempt's business (dialAttempt), not the dialer's. +type socketDialer struct { + target LibvirtTarget + runDir string + timeout time.Duration + + mu sync.Mutex + dialed string +} + +func newSocketDialer(target LibvirtTarget, runDir string, timeout time.Duration) *socketDialer { + return &socketDialer{target: target, runDir: runDir, timeout: timeout} +} + +// Dial connects to the target's socket: the explicit one, or the modular QEMU daemon's +// socket when it exists and the monolithic one otherwise. "Exists" follows symlinks. +func (d *socketDialer) Dial() (net.Conn, error) { + path, err := d.choose() + if err != nil { + return nil, err + } + d.mu.Lock() + d.dialed = path // recorded first, so an attempt abandoned mid-dial still names its socket + d.mu.Unlock() + conn, err := net.DialTimeout("unix", path, d.timeout) + if err != nil { + return nil, fmt.Errorf("dialing libvirt at %s: %w", path, err) + } + return conn, nil +} + +func (d *socketDialer) choose() (string, error) { + if d.target.Socket != "" { + return d.target.Socket, nil + } + modular := filepath.Join(d.runDir, virtqemudSocket) + monolithic := filepath.Join(d.runDir, libvirtSocket) + for _, path := range []string{modular, monolithic} { + if _, err := os.Stat(path); err == nil { + return path, nil + } + } + return "", fmt.Errorf("no libvirt socket: neither %s nor %s exists", modular, monolithic) +} + +// Dialed is the socket the latest Dial chose, or "" before any. +func (d *socketDialer) Dialed() string { + d.mu.Lock() + defer d.mu.Unlock() + return d.dialed +} diff --git a/pkg/hypervisor/libvirt_uri_test.go b/pkg/hypervisor/libvirt_uri_test.go new file mode 100644 index 00000000..285b0b51 --- /dev/null +++ b/pkg/hypervisor/libvirt_uri_test.go @@ -0,0 +1,65 @@ +package hypervisor + +import ( + "fmt" + "strings" + "testing" +) + +// The forms of nova.libvirt_uri the binary accepts, and where each one dials. Any other +// value used to be dialed verbatim as a socket path, so the session and remote URIs the +// configuration docs listed only ever failed as "no such file or directory". +func TestParseLibvirtURI(t *testing.T) { + accepted := []struct { + uri, socket string // socket "" = the automatic choice + }{ + {"", ""}, + {"qemu:///system", ""}, + {"qemu+unix:///system", ""}, + {"qemu:///system?socket=/run/x/s", "/run/x/s"}, + {"qemu+unix:///system?socket=/run/x/s", "/run/x/s"}, + {"/run/libvirt/virtqemud-sock", "/run/libvirt/virtqemud-sock"}, + } + for _, tc := range accepted { + got, err := ParseLibvirtURI(tc.uri) + if err != nil { + t.Errorf("ParseLibvirtURI(%q): %v; want accepted", tc.uri, err) + continue + } + if got.Socket != tc.socket { + t.Errorf("ParseLibvirtURI(%q).Socket = %q; want %q", tc.uri, got.Socket, tc.socket) + } + } + + refused := []string{ + "qemu:///session", + "qemu:///embed?root=/x", + "lxc:///system", + "qemu://host/system", + "qemu+tcp://host/system", + "qemu+ssh://u@h/system", + "qemu+tls://h/system", + "qemu+libssh2://h/system", + "qemu+ext:///system", + "relative/sock", + "qemu:///system?socket=relative", + "qemu:///system?socket=", + "qemu:///system?socket=/a&socket=/b", + "qemu:///system?mode=direct", + "qemu:///system#frag", + "qemu:///system/", + "qemu:///system%zz", // not a URL + "qemu:///system?socket=%zz", // a query that does not parse + } + for _, uri := range refused { + _, err := ParseLibvirtURI(uri) + if err == nil { + t.Errorf("ParseLibvirtURI(%q) accepted; want refused", uri) + continue + } + // One message for every refusal: the value, then the forms that work. + if msg := err.Error(); !strings.Contains(msg, fmt.Sprintf("%q", uri)) || !strings.Contains(msg, "accepted") { + t.Errorf("ParseLibvirtURI(%q) error %q; want it to name the value and the accepted forms", uri, msg) + } + } +} diff --git a/test/advanced_actions_test.sh b/test/advanced_actions_test.sh index 0a8641c7..46667c9e 100755 --- a/test/advanced_actions_test.sh +++ b/test/advanced_actions_test.sh @@ -17,6 +17,24 @@ TOKEN=$(curl -i -s -X POST "http://localhost:35357/v3/auth/tokens" \ echo "✓ Got auth token" +# Create the image the instance boots from: a create must name its boot source. +IMAGE_ID=$(curl -s -X POST "http://localhost:9292/v2/images" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name": "test-advanced-actions-image", "disk_format": "raw", "container_format": "bare"}' | jq -r '.id') + +if [ -z "$IMAGE_ID" ] || [ "$IMAGE_ID" == "null" ]; then + echo "✗ Failed to create image" + exit 1 +fi + +curl -s -o /dev/null -X PUT "http://localhost:9292/v2/images/$IMAGE_ID/file" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/octet-stream" \ + --data-binary "lightstack test image" + +echo "✓ Created image: $IMAGE_ID" + # Create instance INSTANCE_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -H "X-Auth-Token: $TOKEN" \ @@ -24,7 +42,8 @@ INSTANCE_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -d '{ "server": { "name": "test-advanced-actions-vm", - "flavorRef": "00000000-0000-0000-0000-000000000010" + "flavorRef": "00000000-0000-0000-0000-000000000010", + "imageRef": "'"$IMAGE_ID"'" } }' | jq -r '.server.id') @@ -143,6 +162,9 @@ echo "Cleaning up..." curl -s -X DELETE "http://localhost:8774/v2.1/servers/$INSTANCE_ID" \ -H "X-Auth-Token: $TOKEN" > /dev/null +curl -s -X DELETE "http://localhost:9292/v2/images/$IMAGE_ID" \ + -H "X-Auth-Token: $TOKEN" > /dev/null + echo "✓ Cleanup complete" echo "" echo "========================================" diff --git a/test/console_test.sh b/test/console_test.sh index e7195ec2..0138fa41 100755 --- a/test/console_test.sh +++ b/test/console_test.sh @@ -12,15 +12,16 @@ # resolve, stay with their instance and are revoked. # # Usage: ./test/console_test.sh -# Env: KEYSTONE_URL, NOVA_URL +# Env: KEYSTONE_URL, NOVA_URL, GLANCE_URL set -euo pipefail KEYSTONE_URL="${KEYSTONE_URL:-http://localhost:35357}" NOVA_URL="${NOVA_URL:-http://localhost:8774}" +GLANCE_URL="${GLANCE_URL:-http://localhost:9292}" -# The instance below has no image, which real mode boots to ERROR with no libvirt -# domain, so in real mode it could never get a console. +# The instance below boots from a few placeholder bytes, not a disk image. Real mode +# cannot turn that into a running guest, so in real mode it could never get a console. if [ "${LIGHTSTACK_MODE:-stub}" != "stub" ]; then echo "console_test.sh is stub-only; for real mode run ./test/console_access_test.sh" >&2 exit 2 @@ -51,11 +52,25 @@ echo "✓ authenticated" # Installed before the instance exists so an early failure still cleans up; both the # rm and the DELETE tolerate a half-finished run. TMP=$(mktemp -d) -trap 'rm -rf "$TMP"; [ -n "${INSTANCE_ID:-}" ] && curl -s -X DELETE "$NOVA_URL/v2.1/servers/$INSTANCE_ID" -H "X-Auth-Token: $TOKEN" >/dev/null 2>&1 || true' EXIT +trap 'rm -rf "$TMP"; [ -n "${INSTANCE_ID:-}" ] && curl -s -X DELETE "$NOVA_URL/v2.1/servers/$INSTANCE_ID" -H "X-Auth-Token: $TOKEN" >/dev/null 2>&1 || true; [ -n "${IMAGE_ID:-}" ] && curl -s -X DELETE "$GLANCE_URL/v2/images/$IMAGE_ID" -H "X-Auth-Token: $TOKEN" >/dev/null 2>&1 || true' EXIT + +# The image the instance boots from: a create must name its boot source. +IMAGE_ID=$(curl -s -X POST "$GLANCE_URL/v2/images" \ + -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" \ + -d '{"name":"spec021-console-image","disk_format":"raw","container_format":"bare"}' \ + | jq -r '.id') + +if [ -z "$IMAGE_ID" ] || [ "$IMAGE_ID" = "null" ]; then + echo "✗ failed to create the test image" + exit 1 +fi +curl -s -o /dev/null -X PUT "$GLANCE_URL/v2/images/$IMAGE_ID/file" \ + -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/octet-stream" \ + --data-binary "lightstack test image" INSTANCE_ID=$(curl -s -X POST "$NOVA_URL/v2.1/servers" \ -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" \ - -d '{"server":{"name":"spec021-console-vm","flavorRef":"00000000-0000-0000-0000-000000000010"}}' \ + -d '{"server":{"name":"spec021-console-vm","flavorRef":"00000000-0000-0000-0000-000000000010","imageRef":"'"$IMAGE_ID"'"}}' \ | jq -r '.server.id') if [ -z "$INSTANCE_ID" ] || [ "$INSTANCE_ID" = "null" ]; then diff --git a/test/contract/nova/no_hypervisor_test.go b/test/contract/nova/no_hypervisor_test.go new file mode 100644 index 00000000..5459e7d5 --- /dev/null +++ b/test/contract/nova/no_hypervisor_test.go @@ -0,0 +1,44 @@ +package nova_test + +import ( + "os" + "testing" + "time" + + "github.com/gophercloud/gophercloud/openstack/compute/v2/servers" + "github.com/stretchr/testify/require" +) + +// SPEC-031 FR-019, through an OpenStack client: on a real-mode Lightstack whose libvirt socket +// does not exist, a create is accepted, ends ERROR with a fault saying why, and never turns +// ACTIVE. It used to turn ACTIVE with no guest behind it. +// +// Gated: the suite's usual target is a stub-mode Lightstack, where ACTIVE is right. Run it +// against the server docs/specs/031-realmode-dns-libvirt-fixes/quickstart.md §2 describes, with +// LIGHTSTACK_CONTRACT_NO_HYPERVISOR=1. +func TestNoHypervisorCreateEndsErrorWithFault(t *testing.T) { + if os.Getenv("LIGHTSTACK_CONTRACT_NO_HYPERVISOR") != "1" { + t.Skip("set LIGHTSTACK_CONTRACT_NO_HYPERVISOR=1 against a real-mode Lightstack with no libvirt socket") + } + client := setupNovaClient(t) + + server, err := servers.Create(client, servers.CreateOpts{ + Name: "spec031-no-hypervisor", + FlavorRef: "00000000-0000-0000-0000-000000000010", + ImageRef: testImageID(t), + }).Extract() + require.NoError(t, err) + t.Cleanup(func() { _ = servers.Delete(client, server.ID).ExtractErr() }) + + var s *servers.Server + for deadline := time.Now().Add(60 * time.Second); time.Now().Before(deadline); time.Sleep(time.Second) { + s, err = servers.Get(client, server.ID).Extract() + require.NoError(t, err) + require.NotEqual(t, "ACTIVE", s.Status, "a server with no hypervisor behind it must never turn ACTIVE") + if s.Status == "ERROR" { + break + } + } + require.Equal(t, "ERROR", s.Status, "the create must end in ERROR") + require.NotEmpty(t, s.Fault.Message, "an ERROR server must say why in its fault") +} diff --git a/test/contract/nova/rbac_test.go b/test/contract/nova/rbac_test.go index df2ae9b8..025385b7 100644 --- a/test/contract/nova/rbac_test.go +++ b/test/contract/nova/rbac_test.go @@ -93,6 +93,7 @@ func TestAdminOnlyResetState_Contract(t *testing.T) { // The seeded m1.tiny, by id — the same reference every other server // action test uses. "1" and a name are not resolvable here. FlavorRef: "00000000-0000-0000-0000-000000000010", + ImageRef: testImageID(t), }).Extract() require.NoError(t, err, "Failed to create test server") defer servers.Delete(adminClient, server.ID) @@ -168,6 +169,7 @@ func TestAdminOnlyEvacuate_Contract(t *testing.T) { server, err := servers.Create(adminClient, servers.CreateOpts{ Name: "evacuate-test-server", FlavorRef: "00000000-0000-0000-0000-000000000010", + ImageRef: testImageID(t), }).Extract() require.NoError(t, err) defer servers.Delete(adminClient, server.ID) diff --git a/test/interface_attach_test.sh b/test/interface_attach_test.sh index 59f1ffe9..68412cc8 100755 --- a/test/interface_attach_test.sh +++ b/test/interface_attach_test.sh @@ -51,6 +51,24 @@ SUBNET_ID=$(curl -s -X POST "http://localhost:9696/v2.0/subnets" \ echo "✓ Created subnet: $SUBNET_ID" +# Create the image the instance boots from: a create must name its boot source. +IMAGE_ID=$(curl -s -X POST "http://localhost:9292/v2/images" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name": "test-interface-image", "disk_format": "raw", "container_format": "bare"}' | jq -r '.id') + +if [ -z "$IMAGE_ID" ] || [ "$IMAGE_ID" == "null" ]; then + echo "✗ Failed to create image" + exit 1 +fi + +curl -s -o /dev/null -X PUT "http://localhost:9292/v2/images/$IMAGE_ID/file" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/octet-stream" \ + --data-binary "lightstack test image" + +echo "✓ Created image: $IMAGE_ID" + # Create instance INSTANCE_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -H "X-Auth-Token: $TOKEN" \ @@ -58,7 +76,8 @@ INSTANCE_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -d '{ "server": { "name": "test-interface-vm", - "flavorRef": "00000000-0000-0000-0000-000000000010" + "flavorRef": "00000000-0000-0000-0000-000000000010", + "imageRef": "'"$IMAGE_ID"'" } }' | jq -r '.server.id') @@ -215,6 +234,9 @@ echo "Cleaning up..." curl -s -X DELETE "http://localhost:8774/v2.1/servers/$INSTANCE_ID" \ -H "X-Auth-Token: $TOKEN" > /dev/null +curl -s -X DELETE "http://localhost:9292/v2/images/$IMAGE_ID" \ + -H "X-Auth-Token: $TOKEN" > /dev/null + curl -s -X DELETE "http://localhost:9696/v2.0/networks/$NETWORK_ID" \ -H "X-Auth-Token: $TOKEN" > /dev/null diff --git a/test/metadata_test.sh b/test/metadata_test.sh index 241e2dd9..db9ab65c 100755 --- a/test/metadata_test.sh +++ b/test/metadata_test.sh @@ -14,6 +14,24 @@ TOKEN=$(curl -i -s -X POST "http://localhost:35357/v3/auth/tokens" \ echo "✓ Got auth token" +# Create the image the instance boots from: a create must name its boot source. +IMAGE_ID=$(curl -s -X POST "http://localhost:9292/v2/images" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name": "test-metadata-image", "disk_format": "raw", "container_format": "bare"}' | jq -r '.id') + +if [ -z "$IMAGE_ID" ] || [ "$IMAGE_ID" == "null" ]; then + echo "✗ Failed to create image" + exit 1 +fi + +curl -s -o /dev/null -X PUT "http://localhost:9292/v2/images/$IMAGE_ID/file" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/octet-stream" \ + --data-binary "lightstack test image" + +echo "✓ Created image: $IMAGE_ID" + # Create instance INSTANCE_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -H "X-Auth-Token: $TOKEN" \ @@ -21,7 +39,8 @@ INSTANCE_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -d '{ "server": { "name": "test-metadata-vm", - "flavorRef": "00000000-0000-0000-0000-000000000010" + "flavorRef": "00000000-0000-0000-0000-000000000010", + "imageRef": "'"$IMAGE_ID"'" } }' | jq -r '.server.id') @@ -122,6 +141,9 @@ echo "Cleaning up..." curl -s -X DELETE "http://localhost:8774/v2.1/servers/$INSTANCE_ID" \ -H "X-Auth-Token: $TOKEN" > /dev/null +curl -s -X DELETE "http://localhost:9292/v2/images/$IMAGE_ID" \ + -H "X-Auth-Token: $TOKEN" > /dev/null + echo "✓ Cleanup complete" echo "" echo "========================================" diff --git a/test/multi_tenant_test.sh b/test/multi_tenant_test.sh index ed469f3b..c016c22c 100755 --- a/test/multi_tenant_test.sh +++ b/test/multi_tenant_test.sh @@ -71,11 +71,29 @@ list_count() { echo "$body" | sed '$d' | jq -r "$filter" 2>/dev/null || echo "unparseable body" } +# boot_image creates a private, active image for the token's project and prints its id. +# Every server create below names one: a create must say what it boots from, and a +# private image per project keeps the isolation checks free of anything shared. +boot_image() { + local token="$1" name="$2" id + id=$(curl -s -X POST "http://localhost:9292/v2/images" \ + -H "X-Auth-Token: $token" \ + -H "Content-Type: application/json" \ + -d '{"name":"'$name'","visibility":"private","container_format":"bare","disk_format":"raw"}' \ + | jq -r '.id') + curl -s -o /dev/null -X PUT "http://localhost:9292/v2/images/$id/file" \ + -H "X-Auth-Token: $token" \ + -H "Content-Type: application/octet-stream" \ + --data-binary "lightstack test image" + echo "$id" +} + # Resources the teardown removes. Populated as setup progresses so an abort part-way # through still cleans up what it managed to create. PROJECT1_ID=""; PROJECT2_ID=""; USER1_ID=""; USER2_ID="" USER1_TOKEN=""; USER2_TOKEN=""; MEMBER_ROLE_ID="" SERVER1_ID=""; SERVER2_ID=""; NET1_ID=""; NET2_ID=""; SUBNET2_ID=""; VOL1_ID=""; IMAGE1_ID="" +BOOT_IMAGE1_ID=""; BOOT_IMAGE2_ID="" SHARED_NET_ID=""; SHARED_INTRUDER_ID="" # Teardown runs however the suite exits — pass, failed check, or aborted setup — and @@ -103,9 +121,11 @@ teardown() { [ -n "$VOL1_ID" ] && [ "$VOL1_ID" != "null" ] && curl -s -o /dev/null -X DELETE "http://localhost:8776/v3/$PROJECT1_ID/volumes/$VOL1_ID" -H "X-Auth-Token: $USER1_TOKEN" 2>/dev/null || true [ -n "$NET1_ID" ] && [ "$NET1_ID" != "null" ] && curl -s -o /dev/null -X DELETE "http://localhost:9696/v2.0/networks/$NET1_ID" -H "X-Auth-Token: $USER1_TOKEN" 2>/dev/null || true [ -n "$IMAGE1_ID" ] && [ "$IMAGE1_ID" != "null" ] && curl -s -o /dev/null -X DELETE "http://localhost:9292/v2/images/$IMAGE1_ID" -H "X-Auth-Token: $USER1_TOKEN" 2>/dev/null || true + [ -n "$BOOT_IMAGE1_ID" ] && [ "$BOOT_IMAGE1_ID" != "null" ] && curl -s -o /dev/null -X DELETE "http://localhost:9292/v2/images/$BOOT_IMAGE1_ID" -H "X-Auth-Token: $USER1_TOKEN" 2>/dev/null || true fi if [ -n "$USER2_TOKEN" ] && [ "$USER2_TOKEN" != "null" ]; then [ -n "$SERVER2_ID" ] && [ "$SERVER2_ID" != "null" ] && curl -s -o /dev/null -X DELETE "http://localhost:8774/v2.1/servers/$SERVER2_ID" -H "X-Auth-Token: $USER2_TOKEN" 2>/dev/null || true + [ -n "$BOOT_IMAGE2_ID" ] && [ "$BOOT_IMAGE2_ID" != "null" ] && curl -s -o /dev/null -X DELETE "http://localhost:9292/v2/images/$BOOT_IMAGE2_ID" -H "X-Auth-Token: $USER2_TOKEN" 2>/dev/null || true # Subnet, then network (deleting the network would also delete the subnet). [ -n "$SUBNET2_ID" ] && [ "$SUBNET2_ID" != "null" ] && curl -s -o /dev/null -X DELETE "http://localhost:9696/v2.0/subnets/$SUBNET2_ID" -H "X-Auth-Token: $USER2_TOKEN" 2>/dev/null || true [ -n "$NET2_ID" ] && [ "$NET2_ID" != "null" ] && curl -s -o /dev/null -X DELETE "http://localhost:9696/v2.0/networks/$NET2_ID" -H "X-Auth-Token: $USER2_TOKEN" 2>/dev/null || true @@ -247,13 +267,19 @@ for pair in "test-user-1:$USER1_TOKEN" "test-user-2:$USER2_TOKEN"; do fi done +# The image each project's servers boot from. +BOOT_IMAGE1_ID=$(boot_image "$USER1_TOKEN" "project1-boot-image") +BOOT_IMAGE2_ID=$(boot_image "$USER2_TOKEN" "project2-boot-image") +require_value "a boot image for project 1" "$BOOT_IMAGE1_ID" +require_value "a boot image for project 2" "$BOOT_IMAGE2_ID" + # Test 1: Server isolation - User1 creates server, User2 cannot see it log_test "Server isolation between projects" SERVER1_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -H "X-Auth-Token: $USER1_TOKEN" \ -H "Content-Type: application/json" \ - -d '{"server":{"name":"project1-server","flavorRef":"m1.tiny"}}' \ + -d '{"server":{"name":"project1-server","flavorRef":"m1.tiny","imageRef":"'$BOOT_IMAGE1_ID'"}}' \ | jq -r '.server.id') if [ -n "$SERVER1_ID" ] && [ "$SERVER1_ID" != "null" ]; then @@ -474,7 +500,7 @@ for i in {1..3}; do curl -s -X POST "http://localhost:8774/v2.1/servers" \ -H "X-Auth-Token: $USER1_TOKEN" \ -H "Content-Type: application/json" \ - -d '{"server":{"name":"project1-server-'$i'","flavorRef":"m1.tiny"}}' > /dev/null + -d '{"server":{"name":"project1-server-'$i'","flavorRef":"m1.tiny","imageRef":"'$BOOT_IMAGE1_ID'"}}' > /dev/null done # Check User1's usage affects their quota, not User2's @@ -497,7 +523,7 @@ log_test "Cannot reference cross-project resources" SERVER2_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -H "X-Auth-Token: $USER2_TOKEN" \ -H "Content-Type: application/json" \ - -d '{"server":{"name":"project2-server","flavorRef":"m1.tiny"}}' \ + -d '{"server":{"name":"project2-server","flavorRef":"m1.tiny","imageRef":"'$BOOT_IMAGE2_ID'"}}' \ | jq -r '.server.id') if [ -n "$SERVER2_ID" ] && [ "$SERVER2_ID" != "null" ] && [ -n "$VOL1_ID" ] && [ "$VOL1_ID" != "null" ]; then diff --git a/test/performance_test.sh b/test/performance_test.sh index 8d3ea9e3..644d0133 100755 --- a/test/performance_test.sh +++ b/test/performance_test.sh @@ -35,6 +35,22 @@ fi echo "Token obtained" echo "" +# The image the test servers boot from: a create must name its boot source. +IMAGE_ID=$(curl -s -X POST "http://localhost:9292/v2/images" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name":"perf-test-image","disk_format":"raw","container_format":"bare"}' \ + | jq -r '.id') + +if [ -z "$IMAGE_ID" ] || [ "$IMAGE_ID" == "null" ]; then + echo -e "${RED}Failed to create the test image${NC}" + exit 1 +fi +curl -s -o /dev/null -X PUT "http://localhost:9292/v2/images/$IMAGE_ID/file" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/octet-stream" \ + --data-binary "lightstack test image" + # Test 1: Token issue performance echo -e "${CYAN}=== Test 1: Token Issue Performance ===${NC}" echo "Measuring token creation latency..." @@ -72,7 +88,7 @@ for i in {1..20}; do SERVER_ID=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ -H "X-Auth-Token: $TOKEN" \ -H "Content-Type: application/json" \ - -d '{"server":{"name":"perf-test-server-'$i'","flavorRef":"m1.tiny"}}' \ + -d '{"server":{"name":"perf-test-server-'$i'","flavorRef":"m1.tiny","imageRef":"'$IMAGE_ID'"}}' \ | jq -r '.server.id') if [ -n "$SERVER_ID" ] && [ "$SERVER_ID" != "null" ]; then SERVER_IDS+=($SERVER_ID) @@ -115,7 +131,7 @@ for i in $(seq 1 $CONCURRENT_REQUESTS); do curl -s -X POST "http://localhost:8774/v2.1/servers" \ -H "X-Auth-Token: $TOKEN" \ -H "Content-Type: application/json" \ - -d '{"server":{"name":"concurrent-test-'$i'","flavorRef":"m1.tiny"}}' > /dev/null + -d '{"server":{"name":"concurrent-test-'$i'","flavorRef":"m1.tiny","imageRef":"'$IMAGE_ID'"}}' > /dev/null ) & PIDS+=($!) done @@ -267,6 +283,10 @@ curl -s -X GET "http://localhost:8774/v2.1/servers" \ fi done +# Delete the test image +curl -s -o /dev/null -X DELETE "http://localhost:9292/v2/images/$IMAGE_ID" \ + -H "X-Auth-Token: $TOKEN" 2>/dev/null || true + # Delete networks curl -s -X GET "http://localhost:9696/v2.0/networks" \ -H "X-Auth-Token: $TOKEN" | jq -r '.networks[].id' | while read nid; do diff --git a/test/volume_attach_test.sh b/test/volume_attach_test.sh index ed02f7ed..b71e5054 100755 --- a/test/volume_attach_test.sh +++ b/test/volume_attach_test.sh @@ -57,6 +57,11 @@ cleanup() { -H "X-Auth-Token: $TOKEN" > /dev/null 2>&1 || true fi + if [ -n "$IMAGE_ID" ]; then + curl -s -X DELETE "http://localhost:9292/v2/images/$IMAGE_ID" \ + -H "X-Auth-Token: $TOKEN" > /dev/null 2>&1 || true + fi + if [ -n "$VOLUME_ID" ]; then curl -s -X DELETE "http://localhost:8776/v3/$PROJECT_ID/volumes/$VOLUME_ID" \ -H "X-Auth-Token: $TOKEN" > /dev/null 2>&1 || true @@ -171,6 +176,24 @@ fi log_success "Using flavor: $FLAVOR_ID" +# Create the image the instance boots from: a create must name its boot source. +IMAGE_ID=$(curl -s -X POST "http://localhost:9292/v2/images" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name": "test-volume-image", "disk_format": "raw", "container_format": "bare"}' | jq -r '.id') + +if [ -z "$IMAGE_ID" ] || [ "$IMAGE_ID" == "null" ]; then + log_error "Failed to create image" + exit 1 +fi + +curl -s -o /dev/null -X PUT "http://localhost:9292/v2/images/$IMAGE_ID/file" \ + -H "X-Auth-Token: $TOKEN" \ + -H "Content-Type: application/octet-stream" \ + --data-binary "lightstack test image" + +log_info "Using image: $IMAGE_ID" + # Create instance (without volume) log_test "Create test instance" INSTANCE_RESPONSE=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ @@ -180,6 +203,7 @@ INSTANCE_RESPONSE=$(curl -s -X POST "http://localhost:8774/v2.1/servers" \ \"server\": { \"name\": \"test-volume-instance\", \"flavorRef\": \"$FLAVOR_ID\", + \"imageRef\": \"$IMAGE_ID\", \"networks\": [{\"uuid\": \"$NETWORK_ID\"}] } }")