Repository navigation
fix(real-mode): cluster DNS, libvirt socket choice, console dir retry - #116
Open
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Timed-out capabilities calls keep running after their single-flight entries are removed, allowing retries to accumulate unbounded blocked calls and goroutines during a libvirt outage.
1 open issue
What changed in this PR
This PR improves Lightstack’s real-mode operation on host-networked Kubernetes deployments and modular-libvirt hosts. It removes manual socket and startup workarounds and prevents unavailable libvirt from producing fake successful VM creates.
Changes:
- Correct host-network DNS policy and honor configured libvirt socket paths.
- Retry hypervisor preparation, gate initial readiness, and report create/action failures.
- Add regression coverage, update boot-source fixtures, and document operator-facing changes.
| File | Description |
|---|---|
test/volume_attach_test.sh |
Adds boot-image setup and cleanup. |
test/performance_test.sh |
Supplies boot images for server creates. |
test/multi_tenant_test.sh |
Creates private boot images per project. |
test/metadata_test.sh |
Adds an explicit boot-image fixture. |
test/interface_attach_test.sh |
Adds an explicit boot-image fixture. |
test/contract/nova/rbac_test.go |
Supplies images for action-test servers. |
test/contract/nova/no_hypervisor_test.go |
Checks failed creates without a hypervisor. |
test/console_test.sh |
Adds image setup, endpoint configuration, and cleanup. |
test/advanced_actions_test.sh |
Adds an explicit boot-image fixture. |
pkg/hypervisor/libvirt_uri.go |
Implements URI validation and socket selection. |
pkg/hypervisor/libvirt_uri_test.go |
Tests accepted and refused URI forms. |
pkg/hypervisor/libvirt_start_test.go |
Tests background connection and disconnected operations. |
pkg/hypervisor/libvirt_reconnect_test.go |
Updates connection-loop terminology in a comment. |
pkg/hypervisor/libvirt_reboot_test.go |
Uses temporary state for reboot tests. |
pkg/hypervisor/libvirt_dialer_test.go |
Tests socket selection and late-connection cleanup. |
pkg/hypervisor/libvirt_connect_test.go |
Tests connection bounds and explicit sockets. |
pkg/hypervisor/launch_ownership_test.go |
Tests directory ownership before guest launches. |
pkg/hypervisor/fake_libvirtd_test.go |
Extends libvirt fixtures for failure scenarios. |
pkg/hypervisor/disk.go |
Clarifies ownership timing in comments. |
pkg/hypervisor/disk_test.go |
Adapts ownership tests to shared preparation. |
pkg/hypervisor/console_endpoint.go |
Reuses the shared disconnected error. |
pkg/hypervisor/console_dir.go |
Adds shared preparation and sticky readiness state. |
internal/teardown/teardown_test.go |
Tests cleanup against missing or silent libvirt. |
internal/nova/start_refusal_test.go |
Removes the obsolete direct-preparation test. |
internal/nova/spec017_helpers_test.go |
Adds a boot-image request helper. |
internal/nova/server_image_field_test.go |
Uses volume boot for imageless servers. |
internal/nova/server_detail_secgroups_test.go |
Supplies boot images for security-group tests. |
internal/nova/server_detail_keyname_test.go |
Supplies boot images for key-name tests. |
internal/nova/hypervisor_ready_test.go |
Tests readiness before hypervisor preparation. |
internal/nova/handlers.go |
Wires background startup, boot validation, and faults. |
internal/nova/console.go |
Updates the missing-hypervisor error and explanation. |
internal/nova/console_test.go |
Updates comments describing initialization failures. |
internal/nova/boot_volume_test.go |
Generalizes project-owned image fixtures. |
internal/nova/boot_source_test.go |
Tests rejection of missing boot sources. |
internal/nova/advanced_actions.go |
Records and logs background action failures. |
internal/health/health.go |
Documents the additional real-mode readiness gate. |
go.mod |
Makes existing x/sync dependency direct. |
docs/UPGRADING.md |
Explains compatibility and operational changes. |
docs/TROUBLESHOOTING.md |
Documents both libvirt socket names. |
docs/specs/031-realmode-dns-libvirt-fixes/quickstart.md |
Lists automated and host verification steps. |
docs/specs/031-realmode-dns-libvirt-fixes/plan.md |
Records implementation design and constraints. |
docs/specs/031-realmode-dns-libvirt-fixes/data-model.md |
Defines configuration and in-process state. |
docs/specs/031-realmode-dns-libvirt-fixes/contracts/readiness.md |
Defines initial and sticky readiness behavior. |
docs/specs/031-realmode-dns-libvirt-fixes/contracts/libvirt-connection.md |
Defines socket, timeout, and retry contracts. |
docs/specs/031-realmode-dns-libvirt-fixes/contracts/guest-operations.md |
Defines guest failures and launch ownership. |
docs/specs/031-realmode-dns-libvirt-fixes/contracts/chart-render.md |
Defines DNS and socket rendering expectations. |
docs/specs/031-realmode-dns-libvirt-fixes/checklists/requirements.md |
Records requirement validation and clarifications. |
docs/specs/031-realmode-dns-libvirt-fixes/baseline-gates.md |
Records reported test and mutation-check results. |
docs/SCALING.md |
Uses local libvirt in configuration examples. |
docs/REAL_MODE_TESTING.md |
Describes background startup and outage behavior. |
docs/REAL_LIBVIRT_MODE.md |
Updates socket checks and permission guidance. |
docs/HOST_TESTING_OPENSUSE.md |
Removes obsolete symlink and warm-up workarounds. |
docs/HORIZON_INTEGRATION.md |
Replaces permissive chmod advice with mount checks. |
docs/CONFIGURATION.md |
Documents supported local libvirt URI forms. |
docs/blueprint.md |
Updates the real-mode readiness description. |
cmd/lightstack/startup_refusal_test.go |
Tests unsupported URI refusal across commands. |
cmd/lightstack/readiness_checks_test.go |
Tests readiness wiring and response reasons. |
cmd/lightstack/main.go |
Validates URIs and wires hypervisor readiness. |
cmd/lightstack/config_test.go |
Tests mode-dependent URI validation. |
CLAUDE.md |
Updates mode-detection and runtime guidance. |
charts/lightstack/values.yaml |
Explains DNS, socket, and probe behavior. |
charts/lightstack/values.schema.json |
Requires absolute sockets and updates descriptions. |
charts/lightstack/tests/schema_test.yaml |
Tests invalid socket-path rejection. |
charts/lightstack/tests/dns_policy_test.yaml |
Tests conditional DNS policy across workloads. |
charts/lightstack/tests/config_render_test.yaml |
Tests default and custom socket configuration. |
charts/lightstack/templates/teardown-purge-job.yaml |
Enables cluster DNS under host networking. |
charts/lightstack/templates/NOTES.txt |
Explains real-mode readiness during installation. |
charts/lightstack/templates/_helpers.tpl |
Renders socket configuration and host-network DNS. |
charts/lightstack/README.md |
Documents chart behavior and cleanup timeouts. |
charts/lightstack/CONTRACT.md |
Updates the real-mode readiness contract. |
CHANGELOG.md |
Summarizes real-mode and server-operation corrections. |
.specify/memory/constitution.md |
Permits bounded retries and forbids runtime stub fallback. |
Review effort: Balanced
uw-sc
force-pushed
the
bugfix/630-real-mode-cluster-dns-under-hostnetwork-the-libvirt-socket-path-and-a-one-shot-capabilities-read
branch
from
November 16, 2026 12:35
744d230 to
d7ad8e0
Compare
Real mode with host networking on k3s, and on hosts that run libvirt's modular daemons, needed host workarounds. Three defects were behind them (cloud-storage#630): - Host-networked pods resolved through the node, so in-cluster Service names never resolved. The pod spec and the purge Job now render dnsPolicy: ClusterFirstWithHostNet. - realMode.libvirtSocketPath only chose what was mounted; the binary always dialed libvirt-sock. The chart now hands a non-default socket to the binary, which for qemu:///system dials virtqemud-sock where it exists and libvirt-sock otherwise. A libvirt_uri it cannot honour stops real-mode startup. - The console socket directory was prepared once at startup and never again. The preparation is now retried in the background and on demand, and real-mode readiness waits for its first success. Real mode also stops faking state. With libvirt unreachable the server keeps connecting in the background and a create ends ERROR with a fault; a shelve libvirt cannot perform ends ERROR; and a create that names no boot source is refused with 400, as upstream does. Constitution v1.5.0 records the background reconnection and forbids a runtime fallback to stub. Assisted-by: Claude:claude-fable-5-2 Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
uw-sc
force-pushed
the
bugfix/630-real-mode-cluster-dns-under-hostnetwork-the-libvirt-socket-path-and-a-one-shot-capabilities-read
branch
from
November 17, 2026 10:27
d7ad8e0 to
a5fa37a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Real mode through the chart needed host workarounds on k3s with host networking, and on hosts that run
libvirt's modular daemons (
virtqemud, socket-activated, nolibvirtd). Three defects were behind them.This fixes all three, and stops real mode from reporting success when libvirt is not there.
Closes cobaltcore-dev/cloud-storage#630
Spec, plan, contracts and host runbook:
docs/specs/031-realmode-dns-libvirt-fixes/.What's in it
Cluster DNS under host networking. With
realMode.hostNetwork: true, the Lightstack pod and theuninstall purge Job resolved through the node, so in-cluster Service names — an operator-managed
PostgreSQL, RGW, Redis — never resolved. The shared pod spec (Deployment and multi-node DaemonSet) and the
purge Job now render
dnsPolicy: ClusterFirstWithHostNet, as the console proxy already did. Without hostnetworking nothing changes.
The libvirt socket.
realMode.libvirtSocketPathdecided only what was mounted; the binary alwaysdialed
libvirt-sock.realMode.libvirtSocketPathis rendered aslibvirt_urifor the pod and the purge Job.The default render is unchanged. The schema now requires an absolute path.
qemu:///system, the default, each connection attempt dials/var/run/libvirt/virtqemud-sockwhere it exists, else
libvirt-sock. Modular hosts need no symlink and novirtproxyd.libvirt_uriacceptsqemu:///systemorqemu+unix:///system, either with?socket=<absolute path>,or an absolute socket path. The server,
lightstack cleanupandlightstack console-tokensrefuseanything else at configuration load, naming the value.
The console socket directory. It was prepared once at startup, under a 1 s bound. A cold
socket-activated daemon missed that read, and every guest start was refused until a restart while
/readyzstayed green.A guest start that finds it missing makes one attempt within 1 s, or joins the one in flight.
/readyzanswers503 hypervisor unavailableuntil the first success. Later libvirtoutages do not affect readiness.
QEMU's uid:gid.
<state_path>/instances/<id>is made QEMU's immediately before every launch: create, start, hardreboot, resume, self-heal.
libvirt unreachable. Real mode used to serve stub answers when libvirt was unreachable at startup: a
create ended
ACTIVEwith no VM.ends
ERRORwith a fault. Every real-mode create that endsERRORnow records one.lightstack cleanup, the uninstall purge, makes one attempt bounded at 1 s and fails closed.Server create and actions.
imageRefnorblock_device_mapping_v2is refused with400"MissingimageRef attribute", in stub and real mode, as upstream Nova does. It used to answer
202and endERROR.ERROR; it used to readSHELVEDwhile the guest ran on.Suspend, resume and shelve log the cause.
Constitution v1.5.0. Article III gains a bounded exception for re-establishing a dependency in the
background. Article IV forbids a runtime fallback to stub.
No schema change, no migration, no new module:
golang.org/x/syncbecomes a direct requirement forsingleflight. Seventest/*.shscripts that created servers without a boot source now create an imageand boot from it.
Operator notes
docs/UPGRADING.mdhas the details. In short:libvirt_uri—qemu:///session,qemu+tcp://,qemu+ssh://, a relative path — nowstops startup. It used to be dialed as a file name and fail as a missing socket.
default) restarts it into
CrashLoopBackOff, andhelm install --waitfails. Compose and bare metalare not gated.
/etc/hostsentry, a node-only zone)must now be reachable through the cluster DNS.
libvirt-socksymlink, enablingvirtproxyd.socket, and warminglibvirt with
virsh capabilities.helm uninstallgives libvirt one second, once. A socket-activated daemon that hadexited may miss it, and the uninstall aborts; run it again.
secrets.existingSecretthe chart renders no config: yournova.libvirt_urimust name the socketrealMode.libvirtSocketPathmounts.Testing
On the tip; the gate record is in
baseline-gates.md.gofmt,go vetandgo mod tidy -diffclean;make lint0 issues.-race: 19 packages, 0 failures, on PostgreSQL and on SQLite.helm unittest:charts/lightstack25 suites, 181 tests;charts/horizon3 tests.ERRORwith a fault;/readyzanswers503.main: 230 of 238 executable lines.go test -overlay.test/*.shscripts were not run; CI runs none of them.Host validation per
HOST_TESTING.md(FR-021) on 2026-11-13, on an earlier revision of this branch:openSUSE Tumbleweed, k3s v1.37.2, socket-activated
virtqemud, nolibvirtd,virtproxydor symlink.ACTIVE, the uninstall destroys itRESTARTS 0,the first VM
ACTIVEERROR, the pod is not Ready; once the socketsstart, it connects and a VM boots
ACTIVEwith no restartlibvirtd: dialslibvirt-sock, a VM bootsACTIVEQEMU runs as root on that host, so handing directories to an unprivileged QEMU user is proven by the unit
tests only.
Not in this PR
block_device_mapping_v2names no root volume or image still answers202and endsERRORwith "the image reference is not a valid image ID". Matching upstream there needs its ownticket.
🤖 Generated with Claude Code