From d3a344f9be143abc6d60f3dd7bb7a1f7cf6a4851 Mon Sep 17 00:00:00 2001
From: ualtinok <94532+ualtinok@users.noreply.github.com>
Date: Mon, 9 Nov 2026 23:56:01 +0100
Subject: [PATCH] WORKER.md: keep a heartbeat on stdout in build-server jobs

Two worker jobs today went silent for over 15 minutes because their output
went only to a job-local file; nobody, including the build-server owner, can
see inside a running job, so silence couldn't be told apart from a hang.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
---
 .cortexkit/WORKER.md                                      | 1 +
 .../slice-descriptors/land-worker-heartbeat.json          | 8 ++++++++
 2 files changed, 9 insertions(+)
 create mode 100644 benchmarks/aft-search/slice-descriptors/land-worker-heartbeat.json

diff --git a/.cortexkit/WORKER.md b/.cortexkit/WORKER.md
index 583e23be61..cf0ba1f6c3 100644
--- a/.cortexkit/WORKER.md
+++ b/.cortexkit/WORKER.md
@@ -27,6 +27,7 @@ not need it; production and release builds must never enable it.
 - **Commit first, then submit, in separate steps.** The job snapshots the checkout when it is accepted. A commit sent in the same parallel step can land after the snapshot. Print `git log -1` as the job's first line so the output shows what ran.
 - **Keep gate scripts out of ignored paths.** The job copies tracked and untracked files, but not ignored ones, so a script under `.cortexkit/` or `target/` is missing on the server.
 - **Write long output to a file inside the job and print a summary.** A job that streams tens of megabytes back can lose its connection and come back with an unknown result.
+- **Keep a heartbeat on stdout.** When output goes to a file, also print a short progress line at least once a minute (for example `tail -f` of that file in the background, or a phase line per step). Nobody can see inside a running job, the build-server owner included, so a silent job can't be told apart from a hung one. Give each long step inside the job its own time limit that prints which step it was in when it fires.
 - **Build a scratch copy of the source with its own target folder.** Linux jobs share the checkout's warm build folder (`CARGO_TARGET_DIR=/run/motor/target`). A copy built there, for example a red-first proof under `$TMPDIR`, leaves its outputs where later jobs in the real checkout reuse them as current, so a gate can pass on old code. Give the copy `CARGO_TARGET_DIR=$TMPDIR/copy-target` or `--target-dir`. If it already happened, `cargo clean -p <crates>` repairs it.
 
 Mandatory when you touch them:
diff --git a/benchmarks/aft-search/slice-descriptors/land-worker-heartbeat.json b/benchmarks/aft-search/slice-descriptors/land-worker-heartbeat.json
new file mode 100644
index 0000000000..ebd574711e
--- /dev/null
+++ b/benchmarks/aft-search/slice-descriptors/land-worker-heartbeat.json
@@ -0,0 +1,8 @@
+{
+  "targeted_mechanism": "none",
+  "kind": "harness",
+  "slice_class": "non_ranking",
+  "fixtures": [
+    "harness-goldens"
+  ]
+}