Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 44 additions & 9 deletions docs/development/dashboard-counters.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,30 @@ SparkRing's `/opt/sparkring/toolchain/toolchain.py serve` wrapper and
`python3 -m vllm.entrypoints.cli.main serve` supply the same vLLM launch facts.
Literal defaults are used only for recognized server launches without a
configuration-file argument. Shell variables are not expanded. Unknown
GPU containers remain visible with engine `unknown`; absent launch facts
GPU containers remain visible with engine `unknown` in the observation row;
the page omits a launch fact it could not read rather than naming it, and a
GPU compute job shows its container and image alone. Absent launch facts
cannot establish a cross-Spark group. Bridged Docker port translation and
launches discoverable only through environment/configuration files are not
inferred.

A bare `flare run` process is recognized from its GPU PID and bounded argv.
The parser accepts `flare` and `flare-<hex>` binaries, rejects `flare agent`
and `flare orbit`, and publishes only rank, rendezvous, HTTP bind/port and
the served alias. Flare's documented defaults apply: `--host 127.0.0.1`,
`--port 8000`, rank zero when `--rank` is absent and no HTTP API on peers.
Package paths, publisher keys, trust stores and API-key paths are not
published. Ranks sharing a rendezvous form one group; a process without one
is a one-Spark group. A missing rank zero is named **head Spark not
reporting** rather than counted as a complete smaller group, including when
only one peer rank reports. `flare run` has no world-size option; the
prepared package learns it at rendezvous, so the expected count is the
highest observed rank plus one and ranks above that highest one are not
detected as missing. A non-IP `--host` publishes no API facts rather than
being rewritten as loopback. The same parser runs for a container's
entrypoint, so a containerized Flare server is found when its launch is in
`Path`/`Args`.

The recovery timer writes an allowlisted projection to
`/var/lib/sparkring/dashboard/nodes.json` (`sparkring-dashboard-nodes/v1`).
It reads all setup-recorded SSH targets through its existing client and
Expand Down Expand Up @@ -90,21 +109,37 @@ Unmanaged models use the same polling, bounded history, rate calculation,
graph controls and integer token-rate formatting as managed models.
SGLang's generation/prompt counters, first-token histogram sum/count,
running/waiting gauges and token-usage gauge map to the shared numeric
contract. Other absent counters remain unknown. Metric names follow the
contract. Flare's generated/prefill token counters, active/pending gauges,
KV page tokens, draft counters, prefix-cache counters and preemptions map to
the same contract; uptime supplies the up-time line, but Flare publishes no
start time, so a restart does not begin a new history run. Other absent
counters remain unknown. Metric names follow the
[SGLang metrics reference](https://docs.sglang.io/docs/references/production_metrics)
and its [collector](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/observability/metrics_collector.py).
Launch defaults follow the [vLLM serve reference](https://docs.vllm.ai/en/stable/cli/serve/).
Launch defaults follow the [vLLM serve reference](https://docs.vllm.ai/en/stable/cli/serve/)
and Flare's `flare run --help`.

The HTTP client sends `Accept: text/plain` only for `/metrics`; model-list
and runtime-status reads request JSON. The Flare parser also handles JSON
metric objects and Helios snapshots through the same series-to-short-name
mapping. Snapshot KV page counts are multiplied by `page_tokens`; missing
or nonnumeric fields remain unknown. A response with no usable Flare
series publishes `metrics_problem` and displays **Metrics unavailable**
instead of silently leaving the card's figures blank. The allowed hosts,
exact read paths, response bounds and redirect refusal are unchanged.

The agent unit preserves its shared `/run/sparkring` directory across
stop/start, including package upgrades. Operator quiet files and holder
records retain their contents and expiry; reboot still clears runtime state.

Offline coverage is in `test_dashboard_unmanaged.py`, `test_dashboard_ring_observation.py` and
`test_dashboard_charts.py`: both engines, one through eleven Sparks,
non-contiguous membership, mixed parallel dimensions, incomplete groups,
stale/stopped records, ownership, bounds, secret exclusion, restricted reads,
quiet mode and in-place browser updates. These tests do not qualify live
Docker, SSH latency, GPU serving or recovery on Spark hardware.
Offline coverage is in `test_dashboard_unmanaged.py`, `test_flare_dashboard.py`,
`test_dashboard_ring_observation.py` and `test_dashboard_charts.py`: both
engines, Flare's Orbit/manual/default launch shapes, one through eleven
Sparks, non-contiguous membership, mixed parallel dimensions, incomplete
groups, missing heads, stale/stopped records, ownership, bounds, secret
exclusion, restricted reads, quiet mode and in-place browser updates. These
tests do not qualify live Docker, SSH latency, GPU serving or recovery on
Spark hardware.

The initial history response contains raw `counters` and optional
`plugin_counters` envelopes for replay at their `observed_at` time.
Expand Down
46 changes: 38 additions & 8 deletions docs/operations/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,17 +209,37 @@ frozen view of up to 6 hours. Precision, counter sources and cost are in

### Models installed outside SparkRing

The page also shows vLLM and SGLang containers that the host agents find
running. Their cards say **Not installed by SparkRing: no automatic recovery,
no install record** and show the Sparks, engine, image, served name and
parallel sizes. Containers that share a head Spark and rendezvous port form
one group; a group with missing ranks shows **incomplete: N of M Sparks
reporting**. Only rank 0's reported API port is read, with no port scan;
SGLang needs `--enable-metrics` for rates. **Thinking** reads **unknown**
unless the launch or runtime status gives explicit chat defaults. The
The page also shows vLLM and SGLang containers, and bare `flare run`
processes, that the host agents find running. Their cards say **Not
installed by SparkRing: no automatic recovery, no install record** and show
the Sparks, engine, image or process, served name and parallel sizes. A
fact the page could not read is left out rather than named, and a GPU
compute job shows its container and image alone.
Containers and Flare ranks that share a head Spark and rendezvous port form
one group; a Flare process without a rendezvous is a one-Spark group. A
group with missing ranks shows **incomplete: N of M Sparks reporting**, and
a Flare group whose rank zero is absent says **head Spark not reporting**.
Flare's argv carries no world size, so M is inferred from the highest
observed rank: ranks above the highest one seen are not detected as
missing. Only rank 0's reported API port is read, with no port scan; SGLang needs
`--enable-metrics` for rates. **Thinking** reads **unknown** unless the
launch or runtime status gives explicit chat defaults. The
[observation contract](../development/dashboard-counters.md#unmanaged-model-observations)
states the bounds and freshness rules.

A Flare rank is recognized from the GPU process's bounded argv, whichever
way it was started: Orbit, a manual `flare run`, a systemd unit or a shell
wrapper. Rank, `--rendezvous`, `--host`, `--port` and `--served-model-name`
are published; package paths, trust stores, API-key paths and the rest of
argv stay private. The card reads the head's `/metrics` and `/v1/models` and
uses Flare's own counters for tokens, requests, KV use, draft acceptance and
prefix hits. Two limits follow from Flare's defaults: a launch bound to
`127.0.0.1` (Flare's default) has no live counters from another Spark, and
a launch configured only through environment variables or an opaque
container entrypoint cannot be identified. The host-agent cache updates
about once a minute, so a starting or stopped process may take that long to
appear or leave the page.

### Other endpoints

Models served outside the installer, such as Local Inference Lab's Flare
Expand Down Expand Up @@ -257,6 +277,16 @@ context length from `/v1/models` and its state. The state follows the
ring's rules: metrics answering is **Serving**, a refused connection **No
model running**, no answer **Offline**, an HTTP error **Not ready**.

The collector requests Prometheus text (`Accept: text/plain`) for `/metrics`
and JSON (`Accept: application/json`) for `/v1/models` and the permitted
SparkRing runtime-status read. Flare's counters also accept a JSON metrics
response, mapping recognized series or Helios snapshot fields to the same
figures; fields absent from JSON remain unknown. A successful response
without recognized numeric Flare metrics keeps the endpoint's **Serving**
state but shows **Metrics unavailable** and an explanation on its card.
The warning disappears after a usable metrics response. This applies to
both listed endpoints and Flare processes discovered by the host agents.

### Terminal view

The same status in a terminal on Node A, for example over SSH:
Expand Down
48 changes: 39 additions & 9 deletions runtime/host/dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -434,8 +434,8 @@ <h2>Events</h2>
const PALETTE = ["var(--g1)", "var(--g2)", "var(--g3)", "var(--g4)"];
const JOB_COLOR = "var(--muted)";
const gcolor = (i) => PALETTE[(i || 0) % PALETTE.length];
const memberColor = m => m.unmanaged || m.gpu_job ? JOB_COLOR : m.external && m.kind === "flare" ? "var(--flare)" : gcolor(m.arc?.color);
const isFlare = m => !!(m && m.external && m.kind === "flare");
const isFlare = m => !!(m && ((m.external && m.kind === "flare") || (m.unmanaged && (m.flare || m.launch?.engine === "flare"))));
const memberColor = m => isFlare(m) ? "var(--flare)" : m.unmanaged || m.gpu_job ? JOB_COLOR : gcolor(m.arc?.color);
const FLAME_PATH = "M12 2c.5 4.2-5 6.6-5 11.4a5 5 0 0 0 10 0c0-2.5-1.3-4.1-2.3-5.2-.2 1.7-1.1 2.8-2 2.8-1.2 0-1.8-1.2-1.3-2.6C12.8 6.6 12.3 4.2 12 2z";
const FLAME = `<svg class="flame" viewBox="0 0 24 24" aria-hidden="true"><path d="${FLAME_PATH}"/></svg>`;
// Folded cards, by member id; kept in this browser only.
Expand Down Expand Up @@ -1240,14 +1240,40 @@ <h2>Events</h2>
}

function unmanagedFacts(s) {
const f = s.launch || {};
const f = s.launch || {}, flare = isFlare(s);
// A fact the page could not read is left out, never named, as the Spark details are:
// a GPU compute job shows the container's identity alone, and a Flare process lists
// only the rank, rendezvous, served name and API its launch declared.
const known = v => has(v) && v !== "" && String(v).toLowerCase() !== "unknown";
const tell = (...parts) => parts.filter(Boolean).join(" · ");
const rendezvous = c => !known(c.master_addr) ? (known(c.master_port) ? "master port " + esc(c.master_port) : "")
: "master " + esc(c.master_addr) + (known(c.master_port) ? ":" + esc(c.master_port) : "");
const api = c => c.headless ? "headless" : !known(c.api_port) ? ""
: known(c.api_bind) ? "API " + esc(c.api_bind) + ":" + esc(c.api_port) : "API port " + esc(c.api_port);
const rank = c => known(c.node_rank) ? "node rank " + esc(c.node_rank) : "";
const lines = c => {
const identity = tell(`position ${c.position}: ${esc(c.name)}`,
flare ? "flare process" : known(c.image) ? esc(c.image) : "",
known(c.image_id) ? esc(c.image_id) : "");
if (flare) {
return [identity, tell("engine flare", known(c.served_model) ? "served name " + esc(c.served_model) : "", rank(c)),
tell(rendezvous(c), api(c))];
}
return s.gpu_job ? [identity] : [identity,
tell(known(c.engine) ? esc(c.engine) : "", known(c.served_model) ? "served name " + esc(c.served_model) : "",
rank(c), known(c.nnodes) ? `${esc(c.nnodes)} node${c.nnodes === 1 ? "" : "s"}` : ""),
tell(esc(parallelism(c)), rendezvous(c), api(c))];
};
const summary = s.gpu_job ? "" : flare
? tell(esc(s.layout.label), "flare process", known(f.served_model) ? esc(f.served_model) : "")
: tell(esc(s.layout.label), known(f.engine) ? esc(f.engine) : "",
known(f.image_id) ? "image " + esc(f.image_id) : "");
return `<p class="down-note">${esc(s.ownership_label)}</p>
${s.group_note ? `<p class="down-note">${esc(s.group_note)}</p>` : ""}
${s.gpu_job ? "" : `<p>${esc(s.layout.label)} · ${esc(f.engine || "unknown")} · image ${esc(f.image_id || "unknown")}</p>`}
<details class="launch-facts"><summary>Container launch facts and install records</summary>
${(s.containers || []).map(c => `<p>position ${c.position}: ${esc(c.name)} · ${esc(c.image)} · ${esc(c.image_id || "unknown")}${s.gpu_job ? "" : `<br>
${esc(c.engine || "unknown")} · served name ${esc(c.served_model || "unknown")} · node rank ${c.node_rank ?? "?"} · ${c.nnodes ?? "?"} nodes<br>
${esc(parallelism(c))} · master ${esc(c.master_addr || "unknown")}:${c.master_port ?? "?"} · ${c.headless ? "headless" : "API port " + (c.api_port ?? "unknown")}`}</p>`).join("")}
${s.api_local ? `<p class="down-note">The API is bound to loopback on the head Spark, so this page cannot read live counters. Start it with <code>--host 0.0.0.0</code> to show them.</p>` : ""}
${summary ? `<p>${summary}</p>` : ""}
<details class="launch-facts"><summary>${flare ? "Process" : "Container"} launch facts and install records</summary>
${(s.containers || []).map(c => `<p>${lines(c).filter(Boolean).join("<br>")}</p>`).join("")}
${(s.stopped_records || []).map(r => `<p>Stopped install record: ${esc(r.id)} · ${esc(r.model)} · ${esc(positionText(r.positions))}</p>`).join("")}
</details>`;
}
Expand Down Expand Up @@ -1296,15 +1322,17 @@ <h2>Events</h2>
const thinking = s.thinking ? `<div class="thinking muted">Thinking: ${esc(s.thinking)}</div>` : "";
const modelText = s.name === s.model && !s.gpu_job ? (serving ? "" : `<span class="model muted">Last served: same model</span>`) : `<span class="model">${model}</span>`;
const open = !folded.has(s.id);
return `<article class="card st-${esc(st)}${serving ? "" : " inactive"}${gc ? " ringed" : ""}${s.unmanaged || s.gpu_job ? " outline" : ""}${open ? "" : " collapsed"}" data-member="${esc(s.id)}"${gc ? ` style="--gc:${gc}"` : ""}>
return `<article class="card st-${esc(st)}${serving ? "" : " inactive"}${gc ? " ringed" : ""}${s.unmanaged || s.gpu_job ? " outline" : ""}${isFlare(s) ? " flare" : ""}${open ? "" : " collapsed"}" data-member="${esc(s.id)}"${gc ? ` style="--gc:${gc}"` : ""}>
<div class="head">
${foldButton(s.id, s.name)}<span class="name"><span class="${arc ? "run" : ""}">${esc(s.name)}</span></span>
${serving ? modelText + (sizes ? `<span class="tag">${esc(sizes)}</span>` : "") : ""}
${tags.join("")}
<span class="state"><span class="pill p-${esc(st)}">${esc(s.startup ? startupLabel(s) : s.state_label)}</span><span class="since" data-age="${s.state_since || 0}" data-prefix="for ">${esc(since)}</span></span>
${s.metrics_problem ? `<span class="tag warn" title="${esc(s.metrics_problem)}">Metrics unavailable</span>` : ""}
${!serving ? modelText : ""}${endpoint}${start}${recoveryChip}
</div>${open ? `
${startupDetails(s)}
${s.metrics_problem ? `<p class="down-note" role="status">${esc(s.metrics_problem)}</p>` : ""}
${serving ? thinking + tiles(s) + `<div class="secondary muted">${one(s.live?.steps_ps)} steps/s · ${acceptancePct(s.live?.acceptance)} draft acceptance${has(s.live?.accepted_per_step) ? ` · ${acceptedStep(s.live.accepted_per_step)} accepted tok/step` : ""} · ${sampleSource(s, now)}</div>` + charts(s) + sparks(s) : ""}
<details class="card-details"><summary>Details</summary><div class="detail-body">
<div class="sub">${sub}</div>
Expand Down Expand Up @@ -1347,9 +1375,11 @@ <h2>Events</h2>
${serving ? `<span class="model">${model}</span>` : ""}
<span class="tag${flare ? " flare" : ""}">${flare ? FLAME : ""}${esc(kind)}</span>${where}
<span class="state"><span class="pill p-${esc(st)}">${esc(e.state_label || "Checking")}</span><span class="since" data-age="${e.state_since || 0}">${e.state_since ? "for " + dur(now - e.state_since) : ""}</span></span>
${e.metrics_problem ? `<span class="tag warn" title="${esc(e.metrics_problem)}">Metrics unavailable</span>` : ""}
${!serving ? `<span class="model">${model}</span>` : ""}${endpoint}
</div>${open ? `
<div class="sub">${spark}${esc(kind)} at <span class="mono">${esc(e.host)}:${esc(e.port)}</span>${has(e.max_model_len) ? ` · ${int(e.max_model_len)}-token context` : ""}</div>
${e.metrics_problem ? `<p class="down-note" role="status">${esc(e.metrics_problem)}</p>` : ""}
${tiles}${line}
<details class="card-details"><summary>Details</summary><div class="detail-body">
<div class="sub">${esc(kind)} at ${esc(e.host)}:${esc(e.port)}${e.root ? ` · ${esc(e.root)}` : ""}${e.metrics_url ? ` · metrics ${esc(e.metrics_url)}` : ""}</div>
Expand Down
Loading
Loading