Skip to content

Preserve terminal OOM evidence and native exit status - #7

Merged
luke-lombardi merged 3 commits into
masterfrom
ll/terminal-oom-metrics
Oct 21, 2026
Merged

luke-lombardi merged 3 commits into
masterfrom
ll/terminal-oom-metrics

Conversation

@luke-lombardi

@luke-lombardi luke-lombardi commented Oct 21, 2026 •

Copy link
Copy Markdown

Preserve guest OOM counters and init exit status in final metrics so fast exits retain their real cause. Counters reset on restore.

Validated native tests and 15 staging suites covering exits/signals, OOMs, sandbox survival, logging, functions/endpoints, cooldown, and checkpoint compatibility.

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One real issue: the /task/init_exit_status callback calls l.k.GlobalInit().ExitStatus() unconditionally, but globalInit is nil until the root process is created, so any metrics snapshot taken between RegisterExitMetrics() and CreateProcess (metric server scrape / MetricsExport RPC while the sandbox is created but not started) will nil-deref inside ExitStatus and panic the sentry. Guard it (return 0 if GlobalInit() == nil). Non-blocking: the value is 0 while init is still running, indistinguishable from a clean exit, so consumers should only trust it from the final metrics file; and both OOM counters share the same Description, worth making guest_init_oom_kills say it's init-only. OOM counting in killMemoryOOMVictim and the tests look right.

@luke-lombardi

Copy link
Copy Markdown
Author

Fixed in 3bd9d11: guard absent init and give the init-only OOM counter its own description. Added a real Metrics.Export RPC before container start; it fails without the guard and passes with it. Native OOM/final-exit tests pass. Beta9 reads this gauge only from the final metrics file after wait.

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3bd9d11 fixes the nil-init deref (returns 0 before GlobalInit exists), gives guest_init_oom_kills its own description, and the pre-start Metrics.Export in the test covers the crash path. With beta9 only reading init_exit_status from the final metrics file, the 0-while-running ambiguity is fine. LGTM.

@luke-lombardi
luke-lombardi merged commit 205db7f into master Oct 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants