[build-tools] Add guest telemetry for Simulator profiling

[gwdp]

Why

Duo profiling showed high guest CPU during recording, but app-only metrics could not attribute Simulator background work or establish collector coverage.

How

Add opt-in guest-wide telemetry (EAS_DEVICE_RUN_GUEST_METRICS=1) to the existing Simulator metrics lifecycle. Record actual CPU/RAM, process CPU deltas and parentage, memory pressure/compression/swap, available GPU registry data, sampling errors and terminal health in a separate artifact. Request command timeouts, cap file size, and expose probe-specific timestamps; keep collection best effort.

The artifact workflow compiles the PR head with source commit and checksum for hosted experiments. Physical host GPU/AVE utilization and co-tenant mapping remain outside this collector.

Test Plan

Codex ran 9 focused tests, standalone strict TypeScript compilation, changed-source lint/format checks, and a local Mac smoke. Node 22/24 CI tests, typecheck, lint, formatting, and GraphQL checks passed at ca0ff85. The timestamp follow-up is bc73c71; Node 24 and GraphQL checks passed, with Node 22 coverage still running.

Two hosted Duo runs used the CI-built collector artifact from ca0ff85 (5 vCPU / 10 GiB each): 786 samples and six recording off/on pairs, with no sampling errors in the comparison windows. Run A reproduced the accessibility stall and captured the collapsed-tab fallback in the runner stack. Detailed measurements are in the PR comment.

Final data review found process timestamps coupled to sibling probes. Codex fixed that in bc73c71 and validated its fresh CI artifact in a separate hosted check: 25 samples, no errors, and 11,874 matching CPU-delta recomputations. The full workload matrix was not rerun after the fix; the original recording-window estimates use interval weighting.

Cold collection in the workload runs was incomplete: 19 / 6 samples had errors, with maximum gaps of 49.9 / 24.0 seconds despite requested two-second command timeouts. Keep this opt-in; cold process attribution and strict probe deadlines need further work. Two earlier packaging failures were excluded. The full deployed-worker upload integration and physical-host GPU counters were not tested. Local package-wide tests hit unrelated/dependency failures; clean CI passed.

Authorship and LLM use

Gabe requested a draft and measurements from its artifacts. Codex implemented and tested this draft, corrected experiment packaging after failed attempts, and analyzed hosted telemetry and stacks. Final data review found timestamp coupling; Codex fixed it and checked the rebuilt artifact on a hosted VM. Human inspection and review pending.

[gwdp]

Codex implemented this guest telemetry draft and its tests under Gabe’s direction. Human inspection and review are pending. I am running hosted profiling with the PR-built collector artifact and will add the measured results and limitations here. — Codex

[codecov]

Codecov Report

❌ Patch coverage is 27.81955% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.03%. Comparing base (f949f34) to head (bc73c71).

Files with missing lines	Patch %	Lines
...uild-tools/src/steps/utils/guestMetricsRecorder.ts	18.81%	88 Missing and 7 partials ⚠️
...ools/src/steps/functions/collectServeSimMetrics.ts	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4685      +/-   ##
==========================================
- Coverage   75.15%   75.03%   -0.12%     
==========================================
  Files        1051     1052       +1     
  Lines       51856    51988     +132     
  Branches    11318    11342      +24     
==========================================
+ Hits        38969    39005      +36     
- Misses      11896    11985      +89     
- Partials      991      998       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gwdp]

Hosted profiling results using this PR's CI artifact from ca0ff85 (artifact build). Collector SHA-256: 39bbc0e16e5dc16e0fae1f39ee43886615fc0f1184e282496cb185088d5a1e4b; the hosted artifacts record the same hash and source commit.

• Run A and Run B both received 5 vCPU / 10 GiB on M4 Pro Virtual, iOS 27.1. Same euxy 1.2.0 fixture, serve-sim 0.10.0, agent-device 0.21.28 and Argent 0.29.0. Both panel H.264 readers remained connected without curl errors. Playback snapshots verified the Stop button. There was no browser WebRTC client in this experiment.
• Each VM ran three 25-second recording off/on pairs after a 90-second warmup; means use seconds 5–23. Capture/recording process-tree CPU was higher by 0.032 / 0.026 cores on average in runs A / B (six-pair mean 0.029 cores, about 2.9% of one core). This excludes external system rendering and physical-host work. Whole-guest window means were 3.11–4.33 busy cores; recording-on minus off varied from −0.38 to +0.91 cores, so whole-guest CPU cannot be attributed to recording.
• Background work remained high. During the second recording-on window, guest macOS corespotlightd used 0.73 / 0.47 cores; diagnostics, backboardd, SimMetalHost and XProtectRemediator processes were also active. About 0.5–0.6 cores remained unattributed on average during recording trials. Polling does not prove what triggered those processes.
• Run A's warmed agent snapshots averaged 0.50 s. The inner sheet took 9.70 s, then returned RUNNER_BUSY twice after 5.74 / 6.28 s. Recording had stopped. Direct runner sampling caught recursiveTreeSnapshotAcquisition → appendCollapsedTabFallbackNodes / collapsedTabCandidateNode and XCTest queries; app sampling showed AX traversal. After the queries, the app averaged 0.95 cores. Its sampled physical footprint was 218.2M; runner 57.4M. This is much stronger evidence for the accessibility fallback hot path than for recording as the cause of this stall.
• Run B's warmed snapshots averaged 0.55 s and the sheet recovered (2.96 → 1.21 → 1.17 s). Warmed Argent describe averaged 0.28 s in both runs, excluding the first cold call. No sustained Argent stall was reproduced.
• Compressor occupancy reached 1705 / 1997 MiB during recording trials. Run A reached 298 MiB swap in warmup and retained 154–202 MiB in the recording trials; Run B had no swap. Both valid runs had 10 GiB, so this does not establish whether 10 GiB jobs are worse than 20 GiB jobs.
• Recorder encode means were 34.1–39.4 ms, maximum 184.2 ms across six recordings; no encoder failures or writer backpressure. Encoding is asynchronous, so mean encode latency is not directly an FPS limit. GPU registry peak in-use shared memory was 1167 / 1027 MiB, allocated 1533 / 1524 MiB. Physical GPU/AVE utilization and host co-tenant contention remain unmeasured.
• Collector health: 380 / 406 samples; terminal status stopped in both. Recording comparison windows had no errors and complete accepted coverage. Cold collection had 19 / 6 errored samples and maximum gaps of 49.9 / 24.0 s, despite requested two-second command timeouts. Cold CPU/process attribution is incomplete; strict probe deadlines need hardening before enabling this by default. Two earlier attempts failed before Simulator boot because the experiment archive omitted the nested artifact folder; they are excluded from measurements.

Both workflows completed and their owned Simulators were shut down. Raw telemetry, phase/command timings, recording encoder logs and direct process samples are in the workflow artifacts. This validates the standalone collector built by the PR; the deployed-worker upload path still needs a worker containing the change.

— Codex

[github-actions]

✅ Thank you for adding a changeset!

[gwdp]

Final data review found that a slow sibling probe could shift the timestamp used for process CPU deltas. Fixed in bc73c71: record each process probe's own completion timestamp, expose separate aggregate/process timestamps, request SIGKILL on command timeout, and retain error codes/signals. The docs now explicitly acknowledge that OS stalls can exceed the requested timeout.

The six-pair workload results above remain pinned to ca0ff85. Recording windows were fast and error-free; the final comparison analysis uses interval-weighted process CPU deltas (means remain 0.032 / 0.026 cores). Cold attribution is excluded. The full Duo matrix has not been rerun after timestamp hardening.

The fresh CI artifact from bc73c71 passed a separate hosted collector check: 25 samples, no errors, maximum gap 1.011 s, terminal status stopped, and 11,874 per-process CPU-delta recomputations matching the recorded values exactly. Artifact SHA-256: c7bbbf305a84b7f1fb9fc7e51c97f57938f793b81654a3b75ed61ec4fa970c7b.

This short check received 5 vCPU / 20 GiB; it did not run the Duo workload and cannot serve as a RAM-size performance control. One mistaken smoke launch without the artifact was canceled and excluded. The successful smoke workflow is complete. Nine focused tests, strict standalone compilation and changed-source lint passed again after the fix. Current-head CI is still running.

— Codex

[gwdp]

Closing at Gabe’s direction: eas-cli changes were outside the intended scope. No worker deployment was performed; the experiments ran standalone collector artifacts on existing hosted workers. — Codex