feat(cvs) silent mode for pssh exec, reusable LogPoller + ConsoleSpinner, adapt jaxmaxtext training log polling

[solaiys]

Summary

Extracts the JAX MaxText per-node log-polling loop into a reusable, framework-agnostic CVS library (cvs/lib/utils/) so any suite (training, inference, benchmarks) can tail per-node logs, stream/scan output, detect completion, and keep the console alive — without re-implementing the loop. jaxmaxtext is then re-pointed at the shared utility.

Motivation

Every suite that launches a long-running containerized workload and tails per-node logs needs the same machinery: drain new lines every few seconds, surface them, watch for failure signatures, detect completion, and animate a spinner in between. This logic previously lived inside jaxmaxtext_training_lib.py. Pulling it out removes ~135 net lines from that suite and makes the loop testable and reusable.

What's in this PR (3 commits)

1. fix(parallel): silent mode for pssh exec

Makes print_console=False actually silence all orchestrator chatter, not just bulk output:

• phandle._process_output now gates the #---# / Host == <ip> == banners behind print_console.
• multiprocess_phandle logs the command echo (cmd = ...) at debug instead of info when print_console=False.
• Adds unit tests asserting banners/echo are suppressed when quiet and present when not.

2. feat(utils): reusable LogPoller + ConsoleSpinner

New shared utilities under cvs/lib/utils/ with tests and a README:

• LogPoller — orchestrator-driven poll loop: per-node tail with a cursor (each line reaches the console/log exactly once), one followed stream_node, ordered case-insensitive error_patterns scanning, completion via complete_pattern + complete_policy (all/any/node0) or an is_complete callable, and a scan_chunk callable escape hatch. Raises on error signature or timeout; returns on completion.
• ConsoleSpinner — one-line in-place busy indicator on /dev/tty (visible even under pytest's fd-level capture, never written to --log-file); falls back to a periodic heartbeat off-TTY (CI/nohup).

New LogPoller options included here:

• ignore_error_patterns={name: regex} (default None) — an error match whose offending line also matches one of these is treated as benign and skipped. Per-line matching means a benign line cannot mask a real error elsewhere in the same chunk.
• silent_poll=True (default) — keeps the per-node tail/grep commands off the console; pass silent_poll=False to echo them (print_console=True) when debugging the poll loop itself.

3. refactor(jaxmaxtext): drive the training poll loop via LogPoller

Deletes the bespoke spinner/poll/scan/drain code from jaxmaxtext_training_lib.py and delegates to LogPoller, merging NaN/Inf + always-on fatal + config-driven signatures into a single ordered error_patterns dict. Tests updated accordingly (spinner/drain/is_complete tests moved to the utils suite).

Testing

• cvs/lib/utils/unittests/ — 22 LogPoller + 5 ConsoleSpinner tests.
• Updated parallel and jaxmaxtext unit tests.
• make fmt-check, make lint (ruff + pylint E1205/E1206), and the affected unit suites all pass.

Backward compatibility

No behavior change for existing callers: jaxmaxtext keeps the defaults (ignore_error_patterns=None, silent_poll=True), which reproduce the prior quiet, first-match-wins behavior.

[atnair-amd]

has the baremetal path (as well as the container path) been tested? if so, can you link a ticket with the runs

[atnair-amd]

design: The extraction is useful for JAX MaxText, but the current API generalizes JAX's specific topology rather than the broader CVS workload model.

LogPoller assumes:

1. log_paths[i] belongs to orch.hosts[i];
2. every host has exactly one log;
3. completion is a regex found in those logs; and
4. the orchestrator supports exec_cmd_list.

Those assumptions fit JAX, where every node writes one training.log and every node must emit the final-step marker. Other suites have materially different layouts:

• vLLM: server logs exist per rank, but the benchmark client log exists only on the head. Using one path per orch.hosts would either poll nonexistent client logs on workers or execute redundant reads of the head log.
• SGLang disaggregated: hosts have roles such as prefill, decode, router, and benchmark. A host may own multiple logs, while a benchmark-only host has no server log. Readiness also requires specific role coordinators—not simply all, any, or node0.
• Megatron/Primus: the last node is currently authoritative. node0 is wrong, any could accept the wrong node, and all changes existing behavior.
• ATOM: native server readiness comes from an HTTP /health probe, while benchmark completion may come from a result artifact. A log regex is not the authoritative completion signal.
• xDiT: execution is foreground through Pssh/LocalPssh and completion is the command's exit status. There may be no remote log file to poll.
• Baremetal: LogPoller documents baremetal support, but BaremetalOrchestrator does not implement the required exec_cmd_list method.

There are also reliability problems in the generic contract:

• Partial-line loss: if the poller reads Traceback (most recent before the writer finishes the line, splitlines() advances the cursor. When call last) is appended, the next tail -n +N skips the completed line and the traceback pattern is never detected.
• Transport failures advance the file cursor: the parallel layer can return timeout or connection-exception text as ordinary output. The poller counts those lines even though they did not come from the file, potentially skipping real lines on the next successful poll.
• The timeout is not bounded: poll() checks its deadline, but the exec_cmd_list calls receive no timeout. A hung SSH operation can therefore block beyond timeout_s.
• Stale completion markers: completion uses grep over the entire file. Suites that append to existing logs, such as Primus, can immediately match a marker from a previous run.
• Late failures: observing a completion marker does not prove the process exited successfully. A shutdown traceback written after the final drain can be missed.

I think there are three reasonable scopes for this PR:

1. Keep the poller private to JAX MaxText, extracting it only to simplify that suite without presenting its topology as a general CVS contract.
2. Keep it shared but describe it narrowly as a per-node container-log poller for homogeneous workloads, without claiming general baremetal/training/inference support.
3. Make it genuinely reusable by introducing explicit targets—host, path, role, label, and completion group—plus structured execution results, bounded command timeouts, byte-safe cursors, and pluggable completion probes for log markers, process exit, HTTP health, and result artifacts.

The pssh quiet-mode fix remains useful independently; this concern is about the public LogPoller contract and its claim of suite-wide generality.

[amd-droy]

lgtm. thanks Saravanan.

[solaiys]

Thanks @atnair-amd — this is a fair critique, and you're right that the original README over-claimed a CVS-wide contract that the implementation doesn't actually honor. Rather than redesign the API in this PR (your option 3), we've taken option 2: keep LogPoller shared but scope it honestly to what it really models, and fixed the cheap reliability gaps. Done in 46f54e82 (on top of e97007b2).

Scope, documented explicitly (README):

• LogPoller models one homogeneous topology: exactly one container log per orch.hosts[i], drained via exec_cmd_list, completion via a log marker under all/any/node0 — or a caller-supplied is_complete.
• Non-goals called out: multi-log/role topologies (SGLang prefill/decode/router/bench), head-only client logs (some vLLM benchmark layouts), and foreground/exit-status workloads with no remote log (xDiT). Those should keep their own control flow, not bend this one.
• The is_complete / scan_chunk / callable-log_paths escape hatches are documented as the way a suite with HTTP-health readiness (ATOM), a "last node wins" rule (Megatron/Primus), or a marker file plugs in without the built-in grep/policy.

Baremetal (also answering your testing question): you're correct — BaremetalOrchestrator implements exec() but not exec_cmd_list, so LogPoller can't drive baremetal today. It has only been exercised on the container orchestrator (via JAX MaxText); there is no baremetal run to link. I've removed the false "both baremetal and container orchestrators satisfy this" line and the README now states container-only, baremetal pending an exec_cmd_list implementation.

Reliability items:

• Partial-line loss — fixed in e97007b2: the cursor now advances only on complete \n-terminated records, and a trailing fragment is re-read on the next drain (with a regression test using real GNU tail).
• Unbounded command timeout — fixed: each exec_cmd_list now gets cmd_timeout_s (default 60s), so a hung SSH/tail can't block past the overall timeout_s.
• Stale completion markers — documented as a caller precondition (fresh/rotated per-run logs, or a run-scoped is_complete). JAX MaxText writes fresh per-run logs, so it's unaffected.
• Completion ≠ clean exit / late failures — documented; poll() already drains + scans every node once more after the completion check (catching a shutdown traceback or last-step NaN in that window). A failure written after that final drain is acknowledged as not observable by a log poller.
• Transport-failure text advancing the cursor — acknowledged as a real gap; robustly distinguishing orchestrator error text from file content needs structured exec_cmd_list results (part of your option-3 direction). Tracking it as a follow-up rather than a partial guard here.

The pssh quiet-mode fix stands on its own as you noted. If you'd still prefer option 1 (keep it private under training/), I'm happy to move it — but with the scope now stated honestly and the escape hatches documented, option 2 keeps the reuse that already works for marker/callable-completion suites without pretending to be universal.

Follow up: We will discuss with other framework owners and decide whether to improve on top this implementation to make it generic OR keep it training specific.

[atnair-amd]

lgtm Saravanan, thanks!