Skip to content

fix(real-mode): cluster DNS, libvirt socket choice, console dir retry - #116

Open
uw-sc wants to merge 1 commit into
cobaltcore-dev:mainfrom
uw-sc:bugfix/630-real-mode-cluster-dns-under-hostnetwork-the-libvirt-socket-path-and-a-one-shot-capabilities-read
Open

uw-sc wants to merge 1 commit into
cobaltcore-dev:mainfrom
uw-sc:bugfix/630-real-mode-cluster-dns-under-hostnetwork-the-libvirt-socket-path-and-a-one-shot-capabilities-read

Conversation

@uw-sc

@uw-sc uw-sc commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Real mode through the chart needed host workarounds on k3s with host networking, and on hosts that run
libvirt's modular daemons (virtqemud, socket-activated, no libvirtd). Three defects were behind them.
This fixes all three, and stops real mode from reporting success when libvirt is not there.

Closes cobaltcore-dev/cloud-storage#630

Spec, plan, contracts and host runbook: docs/specs/031-realmode-dns-libvirt-fixes/.

What's in it

Cluster DNS under host networking. With realMode.hostNetwork: true, the Lightstack pod and the
uninstall purge Job resolved through the node, so in-cluster Service names — an operator-managed
PostgreSQL, RGW, Redis — never resolved. The shared pod spec (Deployment and multi-node DaemonSet) and the
purge Job now render dnsPolicy: ClusterFirstWithHostNet, as the console proxy already did. Without host
networking nothing changes.

The libvirt socket. realMode.libvirtSocketPath decided only what was mounted; the binary always
dialed libvirt-sock.

  • A non-default realMode.libvirtSocketPath is rendered as libvirt_uri for the pod and the purge Job.
    The default render is unchanged. The schema now requires an absolute path.
  • With qemu:///system, the default, each connection attempt dials /var/run/libvirt/virtqemud-sock
    where it exists, else libvirt-sock. Modular hosts need no symlink and no virtproxyd.
  • libvirt_uri accepts qemu:///system or qemu+unix:///system, either with ?socket=<absolute path>,
    or an absolute socket path. The server, lightstack cleanup and lightstack console-tokens refuse
    anything else at configuration load, naming the value.

The console socket directory. It was prepared once at startup, under a 1 s bound. A cold
socket-activated daemon missed that read, and every guest start was refused until a restart while
/readyz stayed green.

  • The preparation now runs after every connection and is retried in the background until it succeeds.
    A guest start that finds it missing makes one attempt within 1 s, or joins the one in flight.
  • In real mode /readyz answers 503 hypervisor unavailable until the first success. Later libvirt
    outages do not affect readiness.
  • The first failure is logged at ERROR with its cause, the first success at INFO with the directory and
    QEMU's uid:gid.
  • <state_path>/instances/<id> is made QEMU's immediately before every launch: create, start, hard
    reboot, resume, self-heal.

libvirt unreachable. Real mode used to serve stub answers when libvirt was unreachable at startup: a
create ended ACTIVE with no VM.

  • The server now connects in the background, 1 s per attempt, and never falls back to stub. A create
    ends ERROR with a fault. Every real-mode create that ends ERROR now records one.
  • lightstack cleanup, the uninstall purge, makes one attempt bounded at 1 s and fails closed.

Server create and actions.

  • A create that names neither imageRef nor block_device_mapping_v2 is refused with 400 "Missing
    imageRef attribute", in stub and real mode, as upstream Nova does. It used to answer 202 and end
    ERROR.
  • A shelve libvirt cannot perform ends ERROR; it used to read SHELVED while the guest ran on.
    Suspend, resume and shelve log the cause.

Constitution v1.5.0. Article III gains a bounded exception for re-establishing a dependency in the
background. Article IV forbids a runtime fallback to stub.

No schema change, no migration, no new module: golang.org/x/sync becomes a direct requirement for
singleflight. Seven test/*.sh scripts that created servers without a boot source now create an image
and boot from it.

Operator notes

docs/UPGRADING.md has the details. In short:

  • An unsupported libvirt_uri — qemu:///session, qemu+tcp://, qemu+ssh://, a relative path — now
    stops startup. It used to be dialed as a file name and fail as a missing socket.
  • A real-mode pod whose host's libvirt never answers is no longer Ready. The startup probe (30 × 5 s by
    default) restarts it into CrashLoopBackOff, and helm install --wait fails. Compose and bare metal
    are not gated.
  • With host networking, a name only the node's resolver answers (an /etc/hosts entry, a node-only zone)
    must now be reachable through the cluster DNS.
  • The host workarounds can go: the libvirt-sock symlink, enabling virtproxyd.socket, and warming
    libvirt with virsh capabilities.
  • A delete-policy helm uninstall gives libvirt one second, once. A socket-activated daemon that had
    exited may miss it, and the uninstall aborts; run it again.
  • With secrets.existingSecret the chart renders no config: your nova.libvirt_uri must name the socket
    realMode.libvirtSocketPath mounts.

Testing

On the tip; the gate record is in baseline-gates.md.

  • gofmt, go vet and go mod tidy -diff clean; make lint 0 issues.
  • Unit tests with -race: 19 packages, 0 failures, on PostgreSQL and on SQLite.
  • helm unittest: charts/lightstack 25 suites, 181 tests; charts/horizon 3 tests.
  • Contract suite on SQLite: 353 passed, 0 failed, 13 skipped. PostgreSQL not re-run on the tip.
  • The gated no-hypervisor contract case: a create ends ERROR with a fault; /readyz answers 503.
  • Changed-line coverage against main: 230 of 238 executable lines.
  • Every new test was seen failing first, against the old code or a mutation applied with
    go test -overlay.
  • The seven test/*.sh scripts were not run; CI runs none of them.

Host validation per HOST_TESTING.md (FR-021) on 2026-11-13, on an earlier revision of this branch:
openSUSE Tumbleweed, k3s v1.37.2, socket-activated virtqemud, no libvirtd, virtproxyd or symlink.

  • (a) Host networking, PostgreSQL behind a Service name: Ready; a delete-policy uninstall completes
  • (b) Modular daemons only, default values: connects, a VM boots ACTIVE, the uninstall destroys it
  • (c) Cold libvirt, daemon frozen past the 1 s bound: one WARN, Ready after the thaw, RESTARTS 0,
    the first VM ACTIVE
  • (d) libvirt unreachable at start: a create ends ERROR, the pod is not Ready; once the sockets
    start, it connects and a VM boots ACTIVE with no restart
  • (e) Monolithic libvirtd: dials libvirt-sock, a VM boots ACTIVE

QEMU runs as root on that host, so handing directories to an unprivileged QEMU user is proven by the unit
tests only.

Not in this PR

  • A create whose block_device_mapping_v2 names no root volume or image still answers 202 and ends
    ERROR with "the image reference is not a valid image ID". Matching upstream there needs its own
    ticket.

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Timed-out capabilities calls keep running after their single-flight entries are removed, allowing retries to accumulate unbounded blocked calls and goroutines during a libvirt outage.

1 open issue
What changed in this PR

This PR improves Lightstack’s real-mode operation on host-networked Kubernetes deployments and modular-libvirt hosts. It removes manual socket and startup workarounds and prevents unavailable libvirt from producing fake successful VM creates.

Changes:

  • Correct host-network DNS policy and honor configured libvirt socket paths.
  • Retry hypervisor preparation, gate initial readiness, and report create/action failures.
  • Add regression coverage, update boot-source fixtures, and document operator-facing changes.
File Description
test/​volume_attach_test.sh Adds boot-image setup and cleanup.
test/​performance_test.sh Supplies boot images for server creates.
test/​multi_tenant_test.sh Creates private boot images per project.
test/​metadata_test.sh Adds an explicit boot-image fixture.
test/​interface_attach_test.sh Adds an explicit boot-image fixture.
test/​contract/​nova/​rbac_test.go Supplies images for action-test servers.
test/​contract/​nova/​no_hypervisor_test.go Checks failed creates without a hypervisor.
test/​console_test.sh Adds image setup, endpoint configuration, and cleanup.
test/​advanced_actions_test.sh Adds an explicit boot-image fixture.
pkg/​hypervisor/​libvirt_uri.go Implements URI validation and socket selection.
pkg/​hypervisor/​libvirt_uri_test.go Tests accepted and refused URI forms.
pkg/​hypervisor/​libvirt_start_test.go Tests background connection and disconnected operations.
pkg/​hypervisor/​libvirt_reconnect_test.go Updates connection-loop terminology in a comment.
pkg/​hypervisor/​libvirt_reboot_test.go Uses temporary state for reboot tests.
pkg/​hypervisor/​libvirt_dialer_test.go Tests socket selection and late-connection cleanup.
pkg/​hypervisor/​libvirt_connect_test.go Tests connection bounds and explicit sockets.
pkg/​hypervisor/​launch_ownership_test.go Tests directory ownership before guest launches.
pkg/​hypervisor/​fake_libvirtd_test.go Extends libvirt fixtures for failure scenarios.
pkg/​hypervisor/​disk.go Clarifies ownership timing in comments.
pkg/​hypervisor/​disk_test.go Adapts ownership tests to shared preparation.
pkg/​hypervisor/​console_endpoint.go Reuses the shared disconnected error.
pkg/​hypervisor/​console_dir.go Adds shared preparation and sticky readiness state.
internal/​teardown/​teardown_test.go Tests cleanup against missing or silent libvirt.
internal/​nova/​start_refusal_test.go Removes the obsolete direct-preparation test.
internal/​nova/​spec017_helpers_test.go Adds a boot-image request helper.
internal/​nova/​server_image_field_test.go Uses volume boot for imageless servers.
internal/​nova/​server_detail_secgroups_test.go Supplies boot images for security-group tests.
internal/​nova/​server_detail_keyname_test.go Supplies boot images for key-name tests.
internal/​nova/​hypervisor_ready_test.go Tests readiness before hypervisor preparation.
internal/​nova/​handlers.go Wires background startup, boot validation, and faults.
internal/​nova/​console.go Updates the missing-hypervisor error and explanation.
internal/​nova/​console_test.go Updates comments describing initialization failures.
internal/​nova/​boot_volume_test.go Generalizes project-owned image fixtures.
internal/​nova/​boot_source_test.go Tests rejection of missing boot sources.
internal/​nova/​advanced_actions.go Records and logs background action failures.
internal/​health/​health.go Documents the additional real-mode readiness gate.
go.mod Makes existing x/sync dependency direct.
docs/​UPGRADING.md Explains compatibility and operational changes.
docs/​TROUBLESHOOTING.md Documents both libvirt socket names.
docs/​specs/​031-realmode-dns-libvirt-fixes/​quickstart.md Lists automated and host verification steps.
docs/​specs/​031-realmode-dns-libvirt-fixes/​plan.md Records implementation design and constraints.
docs/​specs/​031-realmode-dns-libvirt-fixes/​data-model.md Defines configuration and in-process state.
docs/​specs/​031-realmode-dns-libvirt-fixes/​contracts/​readiness.md Defines initial and sticky readiness behavior.
docs/​specs/​031-realmode-dns-libvirt-fixes/​contracts/​libvirt-connection.md Defines socket, timeout, and retry contracts.
docs/​specs/​031-realmode-dns-libvirt-fixes/​contracts/​guest-operations.md Defines guest failures and launch ownership.
docs/​specs/​031-realmode-dns-libvirt-fixes/​contracts/​chart-render.md Defines DNS and socket rendering expectations.
docs/​specs/​031-realmode-dns-libvirt-fixes/​checklists/​requirements.md Records requirement validation and clarifications.
docs/​specs/​031-realmode-dns-libvirt-fixes/​baseline-gates.md Records reported test and mutation-check results.
docs/​SCALING.md Uses local libvirt in configuration examples.
docs/​REAL_MODE_TESTING.md Describes background startup and outage behavior.
docs/​REAL_LIBVIRT_MODE.md Updates socket checks and permission guidance.
docs/​HOST_TESTING_OPENSUSE.md Removes obsolete symlink and warm-up workarounds.
docs/​HORIZON_INTEGRATION.md Replaces permissive chmod advice with mount checks.
docs/​CONFIGURATION.md Documents supported local libvirt URI forms.
docs/​blueprint.md Updates the real-mode readiness description.
cmd/​lightstack/​startup_refusal_test.go Tests unsupported URI refusal across commands.
cmd/​lightstack/​readiness_checks_test.go Tests readiness wiring and response reasons.
cmd/​lightstack/​main.go Validates URIs and wires hypervisor readiness.
cmd/​lightstack/​config_test.go Tests mode-dependent URI validation.
CLAUDE.md Updates mode-detection and runtime guidance.
charts/​lightstack/​values.yaml Explains DNS, socket, and probe behavior.
charts/​lightstack/​values.schema.json Requires absolute sockets and updates descriptions.
charts/​lightstack/​tests/​schema_test.yaml Tests invalid socket-path rejection.
charts/​lightstack/​tests/​dns_policy_test.yaml Tests conditional DNS policy across workloads.
charts/​lightstack/​tests/​config_render_test.yaml Tests default and custom socket configuration.
charts/​lightstack/​templates/​teardown-purge-job.yaml Enables cluster DNS under host networking.
charts/​lightstack/​templates/​NOTES.txt Explains real-mode readiness during installation.
charts/​lightstack/​templates/​_helpers.tpl Renders socket configuration and host-network DNS.
charts/​lightstack/​README.md Documents chart behavior and cleanup timeouts.
charts/​lightstack/​CONTRACT.md Updates the real-mode readiness contract.
CHANGELOG.md Summarizes real-mode and server-operation corrections.
.specify/​memory/​constitution.md Permits bounded retries and forbids runtime stub fallback.

Review effort: Balanced

Comment thread pkg/hypervisor/console_dir.go Resolved
@uw-sc
uw-sc force-pushed the bugfix/630-real-mode-cluster-dns-under-hostnetwork-the-libvirt-socket-path-and-a-one-shot-capabilities-read branch from 744d230 to d7ad8e0 Compare November 16, 2026 12:35
@uw-sc
uw-sc requested a balanced review from Copilot November 16, 2026 12:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A timeout can race with connection publication, leaving a silent-daemon handshake running indefinitely and leaking sockets and goroutines as retries continue.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread pkg/hypervisor/libvirt_uri.go Resolved
@uw-sc uw-sc self-assigned this Nov 17, 2026
Real mode with host networking on k3s, and on hosts that run libvirt's
modular daemons, needed host workarounds. Three defects were behind
them (cloud-storage#630):

- Host-networked pods resolved through the node, so in-cluster Service
  names never resolved. The pod spec and the purge Job now render
  dnsPolicy: ClusterFirstWithHostNet.
- realMode.libvirtSocketPath only chose what was mounted; the binary
  always dialed libvirt-sock. The chart now hands a non-default socket
  to the binary, which for qemu:///system dials virtqemud-sock where it
  exists and libvirt-sock otherwise. A libvirt_uri it cannot honour
  stops real-mode startup.
- The console socket directory was prepared once at startup and never
  again. The preparation is now retried in the background and on
  demand, and real-mode readiness waits for its first success.

Real mode also stops faking state. With libvirt unreachable the server
keeps connecting in the background and a create ends ERROR with a
fault; a shelve libvirt cannot perform ends ERROR; and a create that
names no boot source is refused with 400, as upstream does.
Constitution v1.5.0 records the background reconnection and forbids a
runtime fallback to stub.

Assisted-by: Claude:claude-fable-5-2
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
@uw-sc
uw-sc force-pushed the bugfix/630-real-mode-cluster-dns-under-hostnetwork-the-libvirt-socket-path-and-a-one-shot-capabilities-read branch from d7ad8e0 to a5fa37a Compare November 17, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants