Repository navigation
bot: read-only Telegram / Lark training-ops bot on the dashboard API - #339
Merged
Merged
Conversation
python -m yeto.bot polls the dashboard's /api/view over HTTP (never ssh, never the training processes), answers chat commands and pushes tiered alerts. Configuration is environment variables only. - Channels behind one interface (send / poll_commands / answer): Telegram Bot API with getUpdates long polling, the Lark self-built app bot (tenant_access_token cached and refreshed before expiry, im/v1 messages as interactive cards, im/v1/images for charts, commands and card.action.trigger over the lark-oapi long connection when installed) and the Lark group webhook (signed, send-only). Each channel is enabled when its configuration is complete; Markdown/card sends fall back to plain text; long messages are truncated. - Alerts: P0 (red, mentions the owners, re-notified every 30 min until acked: stale island, loss NaN/Inf, HBM peak at the refs limit, syncer outer step not advancing past median round + 2 x grace, OOM / checkpoint-failure / disk hooks), P1 (orange, 1 h cooldown per subject: loss > mean + 3 sd, grad_norm, missed rounds, HBM warn, host memory, syncer RSS, rejected / timed-out pushes, eval worse than the previous point) and P2 (grey: checkpoints, new eval point, joins, finished islands). Transitions only, one merged card per poll with Ack / Details / Open dashboard buttons; /ack, /mute, /unmute and quiet hours (P0 only). - Progress card every 2 h and a daily digest with loss / eval charts (matplotlib when installed, text sparklines otherwise). - Commands: /status /island /eval /merge /res /ckpt /alerts /ack /details /mute /unmute /help /start; only the registered report chat may command or press buttons. Users are authorized by numeric id (Telegram from.id, Lark open_id), never by display name. Registering the report chat with /start, and anything that changes alerting (/ack, /mute, /unmute, decision presses), needs a configured allowlist; read-only commands do not. Commands given through the local --command CLI are trusted. - Decision-request cards record the pressed option in an append-only JSONL audit log and execute nothing. Text that is not a command goes to an optional natural-language handler with read-only tools. - Alerts held back by mute, quiet hours or a cooldown are delivered when the hold ends if their condition is still active. - View rows are read by the column names the dashboard sends; eval references are every vs_<name> comparison; the HBM warning line derives from the HBM limit. - YETO_BOT_DRY_RUN=1 prints every request body instead of calling out; bot and webhook tokens are redacted from it and from HTTP errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michaellchung
force-pushed
the
pr/ops-bot
branch
from
November 11, 2026 00:54
fb98042 to
34f9978
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #337 (stacked on
pr/dashboard-sft: the bot reads thesftsection of/api/view).Summary
python -m yeto.bot: a read-only Telegram / Lark (Feishu) training-ops bot. It polls the dashboard's/api/viewover HTTP, answers chat commands and pushes tiered alerts; it never connects to nodes or training processes.Changes
lark-oapilong connection when installed) and the Lark group webhook (signed, send-only). Each channel is enabled when its configuration is complete; configuration is environment variables only./status /island /eval /merge /res /ckpt /alerts /ack /details /mute /unmute /help /start; progress card every 2 h and a daily digest (matplotlib charts when installed, text sparklines otherwise).from.id, Larkopen_id), never by display name; registering the chat and anything that changes alerting (/ack,/mute,/unmute, decision presses) require a configured allowlist. Decision cards only record the pressed option in an append-only audit log and execute nothing.YETO_BOT_DRY_RUN=1prints request bodies instead of calling out; bot and webhook tokens are redacted there and in HTTP errors.Testing
pytest tests/test_bot_app.py tests/test_bot_channels.py tests/test_bot_monitor.py tests/test_dashboard_sft.py: 99 passed, 2 skipped (lark_oapi,matplotlibnot installed). Channels are tested against fake HTTP; nothing calls Telegram or Lark.scripts/precommit_check.py: clean (severity namesP0/P1/P2give tag warnings only).Run locally on macOS arm64, Python 3.11 with the CI install (
torch==2.8.0CPU,-r pyproject.toml --extra test),PYTHONPATH=. python -m pytest ... --timeout=600. Optional stacks that are not installed (Miles, harbor, lark-oapi, matplotlib) skip as on CI.Notes
TELEGRAM_ALLOWED_USERStakes numeric user ids; a non-numeric entry fails at startup rather than silently locking everyone out.cols/merge_cols).P0/P1/P2severity names trigger the pre-commit tag warning (warning only).🤖 Generated with Claude Code