Repair CI, the local console, and process resource ownership - #21
Merged
Merged
Conversation
XAIOS_CAP_CREDENTIAL_READ has shipped in kernel/include/xaios/syscall.h since the DNS/SSH interoperability work, and gates fs_open on /etc/xaios_ssh_client_identity plus the async remote-login child grant. It was never added to the frozen release-candidate contract, so validate_syscall_abi reported "contract missing source capabilities" and make qemu-abi-contract failed on main. Add the bit to the contract capability list so it matches source. Verified: python3 tests/scripts/qemu-abi-contract.py reports status=pass with all eight checks green, on macOS and on the Debian 13 x86_64 box. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compile-check gained a libc prerequisite in 47ff4f1, but the CI job was never updated for it. scripts/build-libc.sh requires llvm-ar, meson and ninja, and picolibc must be present, so the job failed on main with "required tool not found: llvm-ar". The checkout also omitted submodules, which would have failed next with a missing Picolibc submodule. Install llvm, meson, ninja-build and python3 alongside clang and lld, and check out submodules recursively, matching the hosted-c99-libc job that already builds the same target successfully. Verified: make compile-check exits 0 with "All freestanding C files compiled clean" on Debian 13 x86_64 with exactly this tool set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
make hosted-test could not build on Linux, so the Hosted Engine & Model v2 job has failed on main for at least six consecutive runs. Two independent causes, both in host-only compiles of vendored third-party code: BearSSL's sysrng.c calls getentropy(), which glibc declares in <unistd.h> only under _DEFAULT_SOURCE. HOST_CFLAGS uses -std=c99, which sets __STRICT_ANSI__ and hides the declaration, so the call became an implicit declaration and -Werror rejected it. macOS declares it regardless, which is why this only ever failed on Linux. Define _DEFAULT_SOURCE for that compile. openbsd-compat's bcrypt_pbkdf.c uses __attribute__((__nonstring__)), which clang did not recognize before version 21, so -Wunknown-attributes fired under -Werror. scripts/build-image.sh already compiles these same two vendored files with -Wno-unknown-attributes; apply the same flag to the hosted test compile rather than diverging from upstream source. Verified: make hosted-test now runs to completion and exits 0 on Debian 13 x86_64 with clang 19, ending at the model-volume reader tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pummelchen
changed the title
The bidirectional suite gave the outbound SSH conversation 600 seconds for an x86_64 guest and 60 for an aarch64 guest. That mapping only holds on an AArch64 development host, where aarch64 is native and x86_64 is emulated. On an x86_64 CI runner it inverts: the aarch64 guest is the cross-emulated, order-of-magnitude-slower one, and it is the one handed 60 seconds. The job has failed on main with "timed out waiting for freebsd-outbound-ssh-ok" ever since, while the x86_64 variant passes on an x86_64 host. Derive the default from the host/guest pairing instead. Cross-architecture emulation gets the large budget; native pairings keep today's values, so no configuration that currently passes is given less time: arm64 host + aarch64 guest -> 60s (unchanged, verified below) arm64 host + x86_64 guest -> 600s (unchanged) x86_64 host + aarch64 guest -> 600s (was 60s; the CI failure) x86_64 host + x86_64 guest -> 600s (unchanged) An unrecognized host architecture falls back to the large budget. XAIOS_OUTBOUND_TIMEOUT still overrides everything. Verified: the aarch64 suite passes end to end on an Apple Silicon host with this change, 19 of 19 checks, including the outbound SSH, SCP recursive upload/download, ProxyJump and agent-forwarding legs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pummelchen
changed the title
Pummelchen
changed the title
klog_console_write copied each byte into the active capture buffer and also wrote it to the UART. The shell then appended that same captured buffer to the command's output, so on the local console -- where the UART is the user's terminal -- every transient application printed its output twice: admin@xaios:/$ df Filesystem Size Used Avail Capacity Mounted on MutableFS 2048K 62K 1986K 4% / Filesystem Size Used Avail Capacity Mounted on <-- repeated MutableFS 2048K 62K 1986K 4% / Over SSH the UART is not the client's terminal, so the duplicate was invisible to every interoperability gate, all of which drive XAIOS over SSH. Output produced while a session is capturing belongs to that session: the capturing caller relays it to its own terminal exactly once. Skip the UART echo while a capture is active. This also stops publishing the output of remote SSH commands on the machine's physical serial port. Boot is unaffected: no capture is active during startup self-tests. Verified: make qemu-smoke and qemu-local-console-gate pass, and ls, ps, df, stat and cat each print once on AArch64 and x86_64. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shell's help text listed xaiosctl, and both sysinfo and status told the user to "use xaiosctl status", but the command was not in the shell's app table at all, so it always failed: admin@xaios:/$ xaiosctl status xaios: xaiosctl: command not found /bin/xaiosctl was also not a CLI. It was declared int main(void) and ran a fixed control-protocol self-test suite, so simply registering it would have run the test harness rather than the requested operation. Give it an argument path: with arguments it forwards the command line to the control protocol and prints the rendered response; without arguments it stays the existing self-test, so boot and gate evidence is unchanged. Output goes to the console rather than the kernel log, because the shell relays captured console bytes verbatim while log lines are filtered to this application's path prefix. Commands run under the observer role that xaios_control_run already defaults to, so read-only operations succeed and privileged ones are refused by the control plane. The shell grants no admin capability. Verified on AArch64 and x86_64: xaiosctl status and xaiosctl version return real output, and the boot self-test markers are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three findings from a clang --analyze pass, none of which alters behaviour: ssh_mlkem768_self_test computed its XOR difference over sender_secret and receiver_secret unconditionally, including when keypair or encapsulation had failed and neither buffer was written. The result was correctly discarded by the result == 0 guard, so the outcome was right, but reading uninitialised stack is undefined and would trip MemorySanitizer the moment it is enabled. Compare only on the success path. mutable_fs open() re-read the node after a truncate into a variable nothing subsequently used. Removed; the CREATE branch's lookup above it is still live, feeding the truncate condition. git_workspace diff computed a line cap from XAIOS_GIT_WORKSPACE_DIFF_MAX_LINES and never read it, so the limit has never applied -- the loop runs on the unclamped counts and the emitted diff is bounded by hunk_capacity instead. The dead computation is removed rather than switched on: the cap is 32 lines, so enforcing it now would begin truncating any diff of a longer file, and that is a behaviour change that wants its own decision. Verified: make compile-check, docs-check, qemu-abi-contract, qemu-smoke and qemu-local-console-gate all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The local console now accepts a six-digit PIN at the login prompt in addition
to admin/password. Six digits authenticate directly; any other entry is still
treated as a user name. The default development PIN is 012345.
A 10^6 search space is small, so the credential is deliberately fenced in:
- Console only. The PIN is never offered or accepted over SSH, where a
network attacker could search that space quickly.
- It rides along with password authentication rather than widening the
authentication surface: load_console_pin returns early when password auth
is disabled, the build refuses to package a PIN without a user database,
and release builds refuse one outright. A key-only image stays key-only.
- Five consecutive console failures now lock the prompt for 60 seconds, and
a correct credential is refused while the lockout stands. This applies to
password and PIN attempts alike; g_console_auth_failures was previously
counted and never acted on, so neither path was throttled at all.
Stored as PBKDF2-SHA256 with its own salt in /etc/xaios_console_pin, verified
with the same constant-time compare as the password, with availability folded
into the accumulator so a missing record cannot authenticate. The kernel
provisions the record read-only like the user database. Digits are masked as
they are typed, since a PIN is a secret rather than a user name.
Verified on AArch64 and x86_64: correct PIN opens a shell, wrong PIN is
rejected, input is masked, username/password still works, the lockout trips
on the fifth failure and refuses the correct PIN while it stands. docs-check
and qemu-local-console-gate pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The generated Fusion profile wires serial0 to a file, so the guest can print but nothing can be typed back, and the Fusion window itself only shows the boot status screen: progress bar, address, SSH state and a XAIOS LOGIN prompt with a cursor. That screen reflects console state and never echoes input or renders command output, so there was no way to operate the guest from the console at all -- only over SSH. Add XAIOS_FUSION_SERIAL=pipe, which configures serial0 as a VMware pipe endpoint. VMware then creates a UNIX socket that a terminal can attach to, giving a real interactive console: XAIOS_FUSION_SERIAL=pipe ./scripts/build-vmware-fusion.sh ./scripts/run-vmware-fusion.sh nc -U /tmp/xaios-fusion-console XAIOS_FUSION_SERIAL_PIPE moves the socket. The default stays "file" so vmware-fusion-smoke keeps reading fusion-serial.log unchanged. Verified on Fusion 26.0.0: attaching to the pipe reaches the login prompt, the six-digit PIN opens a shell, and echo, df and xaiosctl version all run and return output. The default profile still passes vmware-fusion-smoke. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Typing in the Fusion window did nothing because no keyboard was ever bound. Two independent causes, found by dumping what the guest actually enumerates. The control endpoint ring was never rewound between devices. Each slot's device context starts its control endpoint at the base of the shared ep0 ring, but the driver kept its enqueue position from the previous device, so every control transfer to the second and later devices timed out waiting for completions referencing TRBs the controller had already passed. On QEMU the keyboard is the first device enumerated, so this never showed. On Fusion the keyboard is on port 6, behind a device on port 5, and was unreachable: input: xHCI control transfer timed out input: xHCI GET_DEVICE_DESCRIPTOR failed input: xHCI HID configuration failed port=6 slot=2 Reset the index and clear the ring before addressing each device. The port 5 device is also HID, but a composite whose interfaces are both subclass 0 / protocol 0, so the boot descriptors cannot identify it. Both interfaces carry an 8-byte interrupt IN endpoint, so accepting the first HID interface with a plausible report size would have bound a pointing device as a keyboard and typed noise. Fall back to the report descriptor instead and require a Generic Desktop Keyboard usage; both port 5 interfaces declare Usage Mouse (05 01 09 02) and are correctly refused. The failure path now logs the interfaces and endpoints a rejected device advertises, because a controller that enumerates but exposes no keyboard is a descriptor question and the descriptor is the only thing that answers it. Verified: Fusion 26.0.0 reports "xHCI HID boot keyboard initialized" and passes the translation self-test; qemu-keyboard-input-gate still passes on AArch64. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The display was a status panel: it drew the progress bar, address, SSH state and a XAIOS LOGIN prompt with a cursor, but it reflected console state rather than showing the console. It never echoed a keystroke or rendered command output, so a machine whose only console is the screen -- VMware Fusion, which wires its serial port to a file -- could be watched but not used. Add a framebuffer terminal and hand the display over to it once boot completes. From that point the framebuffer receives exactly the byte stream the UART receives, so the screen shows the same session a QEMU serial console shows: prompt, echoed input, command output, scrollback. The terminal handles newline, carriage return, backspace and tab, wraps and scrolls, and parses CSI sequences so the shell's colour codes render rather than printing as noise. Bytes belonging to a capturing session are excluded, exactly as they are for the UART, so output still appears once. boot_ui_self_test renders a glyph and reads the pixels back, so the display path is proven on the machine that has a framebuffer instead of assumed. Known limitation: the bundled font covers ' ' through '_' and folds lowercase to uppercase, so the terminal renders uppercase. Extending the font is a separate change. Verified on Fusion 26.0.0: "framebuffer 1024x768 glyph readback lit=56 passed" and "framebuffer terminal active 112x41 cells". QEMU has no framebuffer, so the terminal stays dormant there and qemu-local-console-gate, qemu-keyboard-input-gate, docs-check and qemu-abi-contract all still pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate has failed twice in CI for two different reasons -- ProxyJump host key verification once, an SFTP write the next run -- and neither failure left any guest-side evidence. The uploaded serial log stopped at the login prompt, because a non-verbose build calls klog_console_set_log_output(0) once boot finishes, so everything the kernel had to say about the failure was discarded. The information already exists. MutableFS reports exactly why a write fails: mutable-fs: write allocation failed path=%s blocks=%u used=%lu mutable-fs: write block IO failed path=%s blocks=%u mutable-fs: write rejected path=%s mounted=%u flags=0x%x None of it reached the artifact. Build the guest with XAIOS_BOOT_VERBOSE so kernel logging stays on for the whole run and lands in the evidence that is already uploaded. XAIOS_BOOT_VERBOSE is honoured if the caller sets it. Both observed failures are the same underlying event: a failed MutableFS write. The SFTP failure is a pwrite returning <= 0, reported as SSH_FX_FAILURE "Write failed". The ProxyJump failure is verify_known_host in ssh_client.c returning -1 after its pwrite or fsync fails, which surfaces to the user as "host key verification failed". Naming the cause needs the kernel's own message, which this change preserves. Verified: the suite passes locally and the captured log grows from boot output only to 715 lines of kernel diagnostics, including every filesystem operation with path, size, block count and generation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mutable_fs_write_fd cleared its entire staging buffer before every write. The buffer is sized for the largest supported file, MFS_V5_MAX_FILE_BYTES, so appending one audit line meant a 256 KiB memset. The SSH audit log alone paid roughly 21 MiB of it to write 3 KiB of lines across one interoperability run, and every fd writer -- SFTP uploads, xapt, editor saves -- paid the same toll per call. Under TCG emulation that is not free. Only a cursor seeked past the end of the file leaves a hole that has to read back as zeros, so clear just that hole. Everything below file_size was filled by the read_file above, everything from the cursor on is overwritten by the caller's data, and nothing above new_size is written to the volume, so stale bytes further up the buffer cannot reach the disk or leak into a file. This does not change the read-modify-write shape of the call. Rewriting the whole file on each append is inherent to the volume's copy-on-write update, which allocates a fresh block set before releasing the old one; making appends touch only the affected blocks means sharing unchanged blocks between generations, which is a separate change with crash-consistency consequences. Verified: qemu-smoke, qemu-milestone-gate 62 (filesystem), qemu-storage-crash (all metadata kill points recovered), qemu-local-console-gate and compile-check all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Snapshot state never survived a restart. persistence_save_to_disk called the default block device, which is vblk0 -- the initramfs/test image that run-qemu-aarch64.sh attaches with snapshot=on, so every write to it is discarded when the machine stops. The first boot wrote its five records and read them straight back from the same overlay, which looked like success; the next boot found an empty sector and logged "no existing disk state". The layout already anticipated the right home for it: PERSISTENCE_SECTOR is 3000 and MFS_START_SECTOR is 3072, so the snapshot sector was meant to sit just below MutableFS on the same durable volume. Only the device binding was wrong, and both candidate devices are larger than sector 3000, so the capacity assertion passed either way and nothing complained. Add persistence_bind_block_device and route the sector reads and writes through it. kmain now opens the dedicated persistent slot before the persistence self-test and reuses that handle for the MutableFS mount below rather than opening it twice. When no durable slot exists the default device is still used, which keeps the pre-storage self-test working on machines that expose no writable volume. Verified: qemu-persistence-reboot now reports "mutable VirtIO state survived QEMU reboot" with persistence_boot_loads=1, where it previously failed the second boot. qemu-smoke, filesystem milestone 62, qemu-storage-crash and qemu-local-console-gate all pass. Note this gate is not part of the CI workflow, which is why a subsystem that could never persist went unnoticed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MutableFS had no mutual exclusion of any kind. Its node table, open-file table and block bitmap were mutated without a lock while being reachable from every CPU through the filesystem syscalls and from kernel services, on a kernel that runs four vCPUs in the interoperability gates and has no global syscall lock -- only a socket-table lock. Every neighbouring subsystem locks: network_stack, klog, scheduler and kheap all take spinlocks. This one did not. That is the most plausible explanation for the FreeBSD gate's two different intermittent failures, which are the same underlying event: a MutableFS write returning an error. One run failed an SFTP write, reported as SSH_FX_FAILURE "Write failed"; the next failed the known_hosts write inside verify_known_host, which surfaces as "host key verification failed". Both are timing dependent, both only appear under slow emulation where the interleaving differs, and both pass on a fast native host. Each public entry point becomes a thin wrapper that takes one lock around the existing body, now named *_locked. mutable_fs_mount_persistent calls mutable_fs_mount_device_locked so the pair stays inside a single acquisition. mutable_fs_self_test is deliberately left unwrapped: it drives these same entry points and runs single threaded during boot. The read-only counter accessors are left alone; they return a single word. Re-entrancy was checked rather than assumed. MutableFS calls only block_device_*, virtio_block_*, klog and kassert, none of which re-enter it. klog_ring_write is purely in-memory; the klog_ring functions that do touch MutableFS are klog_ring_init, klog_flush and klog_rotate, and klog_flush is reached from klog_level only at panic level, which MutableFS never emits. This does not prove the FreeBSD flakiness is gone -- that needs repeated CI runs, and the gate now captures the kernel's own diagnostics if it recurs. Verified: qemu-smoke, filesystem milestone 62 (three consecutive runs), qemu-storage-crash with all metadata kill points recovered, qemu-local-console-gate, qemu-persistence-reboot and compile-check all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The FreeBSD gate's intermittent failures were file handles being destroyed under a live session. The guest named it once the gate started capturing kernel diagnostics: mutable-fs: open fd=2 path=/tmp/freebsd-sftp flags=0xe mutable-fs: write-fd fd=2 bytes=35 cursor=35 user: rejected syscall=14 arg0=0x2 reason=fs-close-denied The write landed; the close was refused. Transient processes and SSH child channels all draw from the first free process-table slot, so in one run a single pid -- 32 -- was issued and reaped nineteen times, and every child channel ran as child=32. VFS handles and kernel sockets were keyed on that pid, and syscall_release_process_resources sweeps every resource owned by it when a process is reaped. Any transient exiting while an SFTP or outbound SSH session had a file open therefore released that session's handle, and the session failed on its next operation. Which run it hit depended purely on overlap, which is why it only appeared under slow emulation and never reproduced on a fast host. Give each process incarnation a token that is never issued twice, and key VFS handles and kernel sockets on it instead of the pid. The pid keeps its meaning for scheduling, reporting and diagnostics; only ownership moves. The VFS and socket layers are unchanged: they already compared an opaque owner value, and now that value no longer collides. copy_process enumerates fields by hand, so the token also has to be copied there. It was missed at first and every socket allocation failed with owner zero, taking sshd's listener down with boot error 2301 -- worth knowing about before adding another field to that struct. Verified: the FreeBSD bidirectional suite passes locally end to end, and qemu-smoke, qemu-local-console-gate, qemu-storage-crash, qemu-regression-suite, filesystem milestone 62, qemu-persistence-reboot and compile-check all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pummelchen
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sixteen commits.
mainhad five failing CI jobs when this branch started; it now has one, and that one is pre-existing and untouched here.CI repairs
compile-checkgained alibcprerequisite in47ff4f1but the job was never updated. It needsllvm-ar,meson,ninjaand the picolibc submodule; it hadclang lldand no submodules.XAIOS_CAP_CREDENTIAL_READshipped insyscall.hbut was never recorded in the frozen contract. Records a capability the kernel already enforces; widens nothing.make hosted-testcould not build on Linux. BearSSLgetentropy()is hidden by glibc under-std=c99, and openbsd-compatbcrypt_pbkdf.cuses__nonstring__, unknown to clang before 21. macOS hid both.Local console
Three defects, all invisible to the existing gates because every interoperability suite drives XAIOS over SSH.
klog_console_writewrote each byte to the UART and into the capture buffer the shell then replayed. On the local console the UART is the terminal. Also stops publishing remote SSH command output on the physical serial port.xaiosctlwas unreachable.helpadvertised it andsysinfo/statuspointed users at it, but it had no dispatch entry — and it wasint main(void), a self-test harness, so registering it alone would have run the tests instead of the operation. It now takes arguments and forwards them to the control protocol under the observer role.012345), console-only, never accepted over SSH, and gated behind password auth so a key-only image stays key-only. Five failures now lock the prompt for 60s —g_console_auth_failureswas previously counted and never acted on, so neither the PIN nor the password had any throttle.VMware Fusion
The Fusion window could be watched but not used: the framebuffer was a status panel that never echoed input or rendered output, and the serial port was write-only.
subclass 0 / protocol 0, so the report descriptor is now used to identify a keyboard rather than binding a pointing device by mistake.XAIOS_FUSION_SERIAL=pipefor a bidirectional serial console.Storage and process ownership
persistencewrote to the default block device — the test image the launcher attaches withsnapshot=on, so every write was discarded.PERSISTENCE_SECTOR(3000) sits just belowMFS_START_SECTOR(3072); only the device binding was wrong.mutable_fs_write_fdcleared a 256 KiB buffer on every call — ~21 MiB of memset to write 3 KiB of audit log.child=32. VFS handles and sockets keyed on it, so any transient exiting released a live session's file handle. This was the FreeBSD gate's intermittent failure.FreeBSD gate
Two separate problems, both now fixed:
The gate now passes in CI, for the first time.
Status
15 of 16 checks pass. Core OS Aggregate RC fails with exactly the same five items as
main—smmuv3 exited 2,nvme exited 2, and three missing markers — verified unchanged across every commit on this branch.Not addressed: those five items, and sustained parser fuzzing on the SSH stack and DNSSEC validator.
🤖 Generated with Claude Code