Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions oxa/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

75 changes: 75 additions & 0 deletions oxa/docs/lead-facts-store.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Runner facts storage

`oxa_work::events::Facts` is the L1 storage module for #629. The runner remains the sole task
state writer. These modules never open its task files, invoke a model or start a background
process. The lead consumes the public read APIs rather than logs or private runner records.

The runner supplies exactly `<protected repository root>/facts`, outside writable workspace
and Git binds. `Facts::initialize(root)` explicitly creates a fresh store; `Facts::open(root)`
opens/replays an existing one. Missing, corrupt, unsafe or partial initialization is an error,
never an empty situation. Directories are private (0700), files private (0600), and files and
path components are checked without following symlinks. It opens only its fixed storage leaves.

## Append after a durable checkpoint

Call `append(Checkpoint { source, revision }, at_ms, fact)` only after the corresponding
observation is durable. Stop before another action if append fails. The runner's source is
`runner`; its checkpoint revision is monotonic per repository. A short repository checkpoint
lock serializes allocation, durable task rename, callback and acknowledgement. The runner
recovers a pending task/revision before allocating again and does not re-emit older acknowledged
task records. Reserved checkpoint revisions may have gaps; event sequence numbers do not.

Other producers use their own stable emitter domain and monotonic revisions, not a fresh source
per task. The bounded source table holds the latest observation for up to 16 emitter domains;
it retains no unbounded per-task tombstones. Retrying the latest source/revision with the same
fact returns its original event and timestamp, even after journal rotation. Different facts
under that identity and stale revisions are errors. Sequence exhaustion is an error.

Each append validates the projected situation before reserving the event. It then saves a
pending observation in `facts.json`, appends and syncs the journal line, commits the source
cursor and atomically replaces `situation.json`. Atomic replacements sync the file and directory.
Recovery replays that reserved observation across a torn final line, interrupted rotation,
lost acknowledgement or lagging situation. Complete malformed lines, unreserved torn tails,
conflicting contents and unrecoverable gaps fail closed.

`TaskStarted` supplies the initial preparing location (task, host label, branch and worktree).
Later `TaskTransition` observations update its step/session; `finished` removes that running
attempt. A transition without a known running location is an error. Worker completion, local
acceptance, publication, external acceptance and merge stay distinct. Only `PrMerged` enters
the landed list. Blocks and asks persist until explicit `Unblocked` and `AskClosed` facts;
updating a PR head cannot silently clear either.

## Bounded reads

`situation()` returns a validated `oxa_proto::work::Snapshot` with its exact `as_of_seq`.
It is capped at 64 KiB and 128 total rows. Running work, local acceptance awaiting publication,
published work, blocks and asks are never silently dropped to fit. Capacity failure stops the
next action. Landed/incident history retains at most 16 rows each and may shrink to fit;
`history_truncated` records that loss. An accepted/published pair is retired only by the matching
head's merge observation. Standing revision/goal remain unknown until the standing integration.

`events_after(after, limit)` takes an exclusive repository cursor and a limit from 1 to 64.
The two journals (`events.jsonl` and `events.previous.jsonl`) rotate at 256 KiB each. Pages
report `first_available`, `next` and an explicit `gap`; a consumer seeing a gap reloads the
situation rather than inventing old transitions. A cursor ahead of the journal is an error.

`incidents::recent(&facts)` returns the bounded incident projection from that same situation;
the journal is its durable source, not a competing incident/task-state writer. Reads recover
only derived facts metadata/projections under the facts lock and never alter task state.

## Stall incidents

`incidents::detect_stall(status_bytes, expected_session_id, expected_pid, now_ms)` consumes a
bounded status snapshot supplied by the runner. It validates identity, phase, progress, timeout
and deadline; stale invocation identity or malformed data is an error. It returns a typed
`Stall` only when the conservative deadline has elapsed. Sessions without an active deadline and finished sessions do not
stall. The caller may append `Stall::incident(task)` through its durable event boundary.

Stalls are incidents only: this module never wraps, cancels, kills or polls a worker. The
runner passes `--stall-timeout 900` for the decided 15-minute threshold and separately owns
the per-task wall-time limit and escalation. No second watchdog is introduced.

The runner owns `oxa work situation`, `oxa work events --after N`, checkpoint adapters and
action wiring. The lead owns the status/ask cards. The common schema is described in
[work-facts.md](work-facts.md); storage/recovery tests are in
[events/tests.rs](../oxa-work/src/events/tests.rs).
9 changes: 9 additions & 0 deletions oxa/docs/work-facts.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,15 @@ external acceptance bind a full PR head. A head-independent `Subject::PullReques
available for blocks, asks and future standing holds. Task, host, repository and inbox subjects
also exist, including tasks that do not yet have a PR.

`TaskTransition { task, step, session_id, reason }` reports every durable runner step:
`preparing → worker → acceptance → repair/recheck → saving → finished`. Each observation gets
its own checkpoint revision; worker completion and local acceptance may have separate revisions.
`TaskStep` is closed, `session_id` is optional until known, and `StepReason` is a closed code
such as `checks_failed`, `session_unavailable` or `deadline`, not diagnostic prose. A finished
step requires a reason and says only that this local attempt ended. It does not imply acceptance,
publication or merge. These observations are routine; a separate block/ask/incident carries any
attention condition. Running situation rows expose their current step and cannot be `finished`.

`Snapshot` is a materialized read view: `as_of_seq`, observation time, standing revision, optional
goal, running tasks (host, branch, worktree and optional session), locally accepted work,
publications, landed PRs, blocks, asks and incidents. It is not authoritative task state.
Expand Down
29 changes: 27 additions & 2 deletions oxa/oxa-proto/src/work.rs
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,18 @@ choice!(WorkerOutcome {
Done => "done", NothingToDo => "nothing_to_do", Blocked => "blocked",
Failed => "failed", Interrupted => "interrupted",
});
choice!(TaskStep {
Preparing => "preparing", Worker => "worker", Acceptance => "acceptance",
Repair => "repair", Recheck => "recheck", Saving => "saving", Finished => "finished",
});
choice!(StepReason {
Resumed => "resumed", Prepared => "prepared", WorkerDone => "worker_done",
WorkerNothingToDo => "worker_nothing_to_do", WorkerBlocked => "worker_blocked",
WorkerFailed => "worker_failed", ChecksPassed => "checks_passed", ChecksFailed => "checks_failed",
RepairDone => "repair_done", RepairFailed => "repair_failed", Saved => "saved", SaveFailed => "save_failed",
Interrupted => "interrupted", Deadline => "deadline", SessionUnavailable => "session_unavailable",
SetupFailed => "setup_failed",
});
choice!(EvidenceKind {
LocalCheck => "local_check", Gate => "gate", Review => "review", Queue => "queue",
Status => "status", Standing => "standing",
Expand Down Expand Up @@ -309,14 +321,25 @@ record!(
| { atom(&s.id, 256) }
);
record!(Running {
task: String, host: String, branch: String, worktree: String, session_id: Option<String>,
task: String, host: String, branch: String, worktree: String, session_id: Option<String>, step: TaskStep,
} |s| {
atom(&s.task, 128)?;
atom(&s.host, 128)?;
require(!s.branch.starts_with('-') && s.branch.len() <= 256, "invalid task branch")?;
if let Some(id) = &s.session_id { atom(id, 128)?; }
require(s.step != TaskStep::Finished, "finished task cannot be running")?;
Ok(())
});
record!(
/// One durable runner step, with no task-state contents or implied publication/landing.
/// Finished means this local attempt ended; only the separate acceptance/merge facts
/// provide those respective claims. A terminal step requires an explicit fixed reason.
TaskTransition { task: String, step: TaskStep, session_id: Option<String>, reason: Option<StepReason> } |s| {
atom(&s.task, 128)?;
if let Some(id) = &s.session_id { atom(id, 128)?; }
require(s.step != TaskStep::Finished || s.reason.is_some(), "finished step requires a reason")
}
);
record!(WorkerFinished {
task: String, session_id: Option<String>, outcome: WorkerOutcome, error_class: Option<String>,
} |s| {
Expand Down Expand Up @@ -393,7 +416,7 @@ record!(
}
);
record!(AskClosed { id: String } | s | { atom(&s.id, 128) });
record!(Incident { id: String, subject: Subject, kind: IncidentKind, evidence: Option<Evidence> } |s| { atom(&s.id, 128) });
record!(Incident { id: String, subject: Subject, kind: IncidentKind, evidence: Option<Evidence> } |s| { atom(&s.id, 256) });
record!(
Mechanical {
subject: Subject,
Expand All @@ -406,13 +429,15 @@ tagged!(
/// No variant grants permission or updates runner state. Only PrMerged is evidence of landing.
Fact {
TaskStarted(Running) => "task_started",
TaskTransition(TaskTransition) => "task_transition",
WorkerFinished(WorkerFinished) => "worker_finished",
LocalAcceptance(LocalAcceptance) => "local_acceptance",
Published(Publication) => "published",
ExternalAcceptance(ExternalAcceptance) => "external_acceptance",
Enqueued(PullRequest) => "enqueued",
PrMerged(Landed) => "pr_merged",
Blocked(Blocked) => "blocked",
Unblocked(Subject) => "unblocked",
AskOpened(Ask) => "ask_opened",
AskClosed(AskClosed) => "ask_closed",
Incident(Incident) => "incident",
Expand Down
Loading
Loading