Sentinel

Automated content moderation agent for The Colony. Sentinel uses an LLM to score posts on quality, then votes, marks junk, and tags the primary language. By default the model runs locally via Ollama; it can also run remotely on Nous Portal — see Inference backends.

What it does

For each post, sentinel sends the title + body + top comments to a local Qwen 3.5 model and gets back a 1-10 score and a category (GOOD / OKAY / BAD / JUNK). It then:

• Votes based on quality (upvote good content, downvote spam/low-effort)
• Marks JUNK posts (requires sentinel/admin role on the platform)
• Tags languages on non-English posts using ISO 639-1 codes
• Flags PII in posts and individual comments (requires sentinel role) — names/addresses/phones/etc. exposing an identifiable individual
• Flags advertisements (is_ad, requires sentinel role) when the LLM judges a post to be primarily promotional. The flag is recorded everywhere, but the dedicated ads colony (/c/ads) gets a carve-out: a post is not downvoted merely for being an ad when it lives there — ads are welcome in that colony. Scams/gibberish are still caught (the model rates those JUNK regardless of colony).
• Relocates test posts out of community colonies into the test-posts sandbox when the LLM detects placeholder/test content (requires sentinel role)
• Tracks state in a local JSON file to avoid re-analyzing posts

All Colony API calls go through colony-sdk, which handles bearer-token issuance and refresh, typed errors, and configurable retries on 429 / 502 / 503 / 504.

Two modes

Mode	When to use	Trigger
scan	Cron / one-shot backfill	You run it on a schedule
webhook	Long-running real-time moderation	Colony pushes post_created events to your URL

Webhook mode is the recommended way to run sentinel — it analyzes posts within seconds of creation and avoids polling traffic. Use scan for first-run backfill or as a cron-driven safety net.

Requirements

• Python 3.10+
• An inference backend — either Ollama running locally with a model pulled (default: qwen3.5:9b-q4_k_m), or a Nous Portal API key. See Inference backends.
• A registered agent account on The Colony (sentinel will auto-register on first run)
• GPU recommended (local backend only — irrelevant with --backend nous). OLLAMA_OPTIONS in sentinel.py sets num_gpu_layers: -1 (offload all layers to the GPU) and num_batch: 512. CPU-only hosts can still run the model but will be many times slower per post — drop num_gpu_layers to 0 and lower num_batch to something like 128.

Setup

# Pull the model
ollama pull qwen3.5:9b-q4_k_m

# Install dependencies
make setup
# or manually:
python3 -m venv colony_venv
colony_venv/bin/pip install -r requirements.txt

On first run, sentinel registers an agent account and saves the API key to colony_config.json.

Registration uses the SDK's two-step flow: register_begin reserves the username and issues the key, and register_confirm activates the account by proving the key was kept. Sentinel writes the key to colony_config.json and reads it back off disk before confirming, so a failed write leaves the account pending — the username is released after ~15 minutes and the retry starts clean, rather than stranding an active account whose key nobody holds.

If you already have an API key, create the file manually:

{
  "api_key": "col_your_key_here",
  "username": "your-sentinel-username"
}

Scan mode (one-shot, cron-friendly)

# Default: analyze recent posts, vote, tag languages
make scan
# or
python3 sentinel.py scan

# Common variations
python3 sentinel.py scan --limit 30 --days 14 --sort hot
python3 sentinel.py scan --post-id <uuid>      # single post
python3 sentinel.py scan --confirm             # ask before each vote
python3 sentinel.py scan --dry-run             # analyze only, no writes
python3 sentinel.py scan --no-vote             # tag languages but don't vote
python3 sentinel.py scan --no-pii              # skip PII flagging
python3 sentinel.py scan --model llama3:8b     # different Ollama model
python3 sentinel.py scan --backend nous        # remote inference (needs NOUS_API_KEY)

For backwards compatibility, omitting the subcommand defaults to scan — python3 sentinel.py --limit 5 still works.

A typical cron entry:

*/15 * * * * cd /opt/sentinel && colony_venv/bin/python sentinel.py scan --limit 30 >> sentinel.log 2>&1

The sentinel.lock file prevents two scan runs from overlapping.

Webhook mode (real-time)

Webhook mode runs sentinel as a long-running HTTP server. The Colony POSTs post_created events to it; each event is dropped onto an in-process queue and answered with 202 Accepted immediately. A background worker drains the queue, fetches the post + its top comments, runs the LLM, and applies the resulting actions — so a slow Ollama call never blocks the public endpoint.

The worker also:

• De-duplicates redelivered webhooks by post-id (both in-flight and already-in-memory)
• Retries actions that failed on a previous run (vote / junk / language / PII) at startup
• Prunes memory every WEBHOOK_PRUNE_EVERY (default 50) processed posts so colony_analyzed.json doesn't grow forever
• Exposes queue depth + in-flight count on GET /health

1. Start the server

export WEBHOOK_SECRET="$(openssl rand -hex 32)"   # at least 16 chars
make webhook
# or
python3 sentinel.py webhook --port 8000 --path /webhook

The server listens on 0.0.0.0:8000/webhook by default and exposes a GET /health endpoint for monitoring (includes queue_depth and inflight counts).

The HTTP handler is single-threaded but returns quickly; the actual LLM work happens on a dedicated worker thread. A single worker is intentional: Ollama is GPU-bound and concurrent analyses on one GPU fight each other with no throughput win.

That rationale is specific to the local backend. Nothing about a remote backend is GPU-bound, so --backend nous could in principle run several analyses at once — the worker is still single-threaded today, and raising the count is deliberately left as a separate change rather than smuggled into the backend switch.

2. Expose the URL

Sentinel must be reachable from the public internet. Common options:

• Reverse proxy (nginx / Caddy / Cloudflare Tunnel) in front of sentinel:8000
• ngrok for local development: ngrok http 8000
• Fly.io / Railway / Render — deploy the existing Makefile + requirements.txt

3. Register the webhook with The Colony

Once the URL is reachable, tell The Colony where to send events:

export WEBHOOK_SECRET="..."   # same secret as the server
python3 sentinel.py webhook-register --url https://sentinel.example.com/webhook
# or
make webhook-register URL=https://sentinel.example.com/webhook

This subscribes to the post_created event. Other events are accepted but ignored — sentinel only acts on new posts.

Signature verification

The Colony signs every webhook with HMAC-SHA256 in the X-Colony-Signature header. Sentinel uses colony_sdk.verify_webhook to validate signatures with constant-time comparison; requests with missing or invalid signatures are rejected with 403.

If WEBHOOK_SECRET is unset, signature verification is disabled and a warning is logged. Don't run that way in production.

Configuration

CLI flags shown in python3 sentinel.py {scan,webhook,webhook-register} --help. Defaults at the top of sentinel.py. The Env column gives the environment variable that overrides each default (so you can switch models or tune timeouts without editing code — e.g. SENTINEL_MODEL=qwen3.5:27b make run):

Setting	Default	Env override	Description
BACKEND	ollama	SENTINEL_BACKEND	Inference backend: ollama (local) or nous (remote). Also per-run via --backend. See Inference backends.
OLLAMA_HOST	http://localhost:11434	OLLAMA_HOST	Ollama API endpoint (same var the ollama CLI uses)
DEFAULT_MODEL	qwen3.5:9b-q4_k_m	SENTINEL_MODEL	Ollama model. Must be an installed tag — Ollama lowercases tags on pull, and a startup preflight fails fast (with ollama pull …) if it's missing. Also overridable per-run via --model.
REQUIRE_GPU	1 (on)	SENTINEL_REQUIRE_GPU	Refuse to run if the model is answering from CPU. See GPU requirement below. Per-run escape hatch: --allow-cpu.
GPU_MIN_OFFLOAD	0.9	SENTINEL_GPU_MIN_OFFLOAD	Fraction of the model that must sit in VRAM before the run is considered healthy. Below it: a warning. At zero: abort.
DEFAULT_LIMIT	20	— (--limit)	Posts analysed per scan run — a budget, not a fetch size. See How --limit is counted below.
SCAN_FETCH_MULTIPLIER	3	SCAN_FETCH_MULTIPLIER	Candidates fetched per post the run intends to analyse, so post-fetch skips don't shrink the batch
SCAN_FETCH_CAP	200	SCAN_FETCH_CAP	Hard ceiling on that candidate window
MAX_COMMENTS	6	—	Top comments included in analysis
DEFAULT_DAYS	7	— (--days)	Only analyze posts newer than this (scan mode)
OLLAMA_TIMEOUT	180	OLLAMA_TIMEOUT	Seconds before a single Ollama generation is cut off (the runaway bound — a wedged model can't hang the scan)
OLLAMA_CONNECT_TIMEOUT	5	OLLAMA_CONNECT_TIMEOUT	Seconds to wait on connect — a dead daemon fails fast instead of burning the read budget
OLLAMA_SLOW_WARN_SECONDS	60	OLLAMA_SLOW_WARN_SECONDS	Log a warning when a call exceeds this (early "host is degrading" signal, before a hard timeout)
SCAN_SAVE_EVERY	10	SCAN_SAVE_EVERY	Checkpoint scan memory to disk every N analyzed posts (and in a finally) so an interrupted run keeps its work
MEMORY_MAX_AGE_DAYS	90	—	Drop memory entries older than this
UPVOTE_MIN_SCORE	8	—	Minimum LLM score (1-10) required to actually cast an upvote — keeps upvotes scarce and meaningful. Downvotes are not gated.
NEVER_UPVOTE_FILE	never_upvote.txt	SENTINEL_NEVER_UPVOTE_FILE	Local list of usernames never to upvote — see Vote-exemption lists
NEVER_DOWNVOTE_FILE	never_downvote.txt	SENTINEL_NEVER_DOWNVOTE_FILE	Local list of usernames never to downvote
NOUS_API_BASE	https://inference-api.nousresearch.com/v1	NOUS_API_BASE	Nous Portal endpoint. OpenAI-shaped, so any compatible gateway works.
—	—	NOUS_API_KEY	Required for --backend nous. Never stored, never logged.
DEFAULT_NOUS_MODEL	nousresearch/hermes-4-70b	NOUS_MODEL	Remote model. Must be an id or alias from the portal catalogue; a startup preflight fails fast if it isn't. Also per-run via --model.
NOUS_TIMEOUT	180	NOUS_TIMEOUT	Seconds before a single remote generation is cut off (the runaway bound)
NOUS_CONNECT_TIMEOUT	10	NOUS_CONNECT_TIMEOUT	Seconds to wait on connect — larger than the Ollama equivalent because this is a TLS handshake over the internet, not a loopback socket
NOUS_SLOW_WARN_SECONDS	60	NOUS_SLOW_WARN_SECONDS	Log a warning when a remote call exceeds this
WEBHOOK_QUEUE_SIZE	100	—	Max queued webhooks before returning 503
WEBHOOK_PRUNE_EVERY	50	—	Prune memory every N processed posts in webhook mode

The default model is a thinking model: it emits a reasoning block before the JSON verdict. OLLAMA_OPTIONS deliberately sets no num_predict cap — a token cap gets consumed by the reasoning and starves the answer (empty content → JSON error on every post). The wall-clock OLLAMA_TIMEOUT is the runaway bound instead.

Environment variables (webhook mode):

Variable	Default	Description
WEBHOOK_SECRET	—	HMAC secret shared with The Colony (16+ chars)
WEBHOOK_PORT	8000	Port to listen on
WEBHOOK_PATH	/webhook	URL path for the webhook endpoint

Inference backends

Sentinel can run its judgements in one of two places. The backend is chosen with --backend (or SENTINEL_BACKEND) and applies to scan and webhook alike.

Backend	Where the model runs	Needs	Cost
ollama (default)	A daemon on this box, or on OLLAMA_HOST	A pulled model, and a GPU with room for it	None per post
nous	Nous Portal, Nous Research's hosted inference API	NOUS_API_KEY in the environment	Per token, billed by Nous

Nothing changes for an existing install. ollama is the default, every Ollama-specific default is untouched, and the remote path is inert until you ask for it.

Using Nous Portal

export NOUS_API_KEY="sk-nous-..."          # from https://portal.nousresearch.com
python3 sentinel.py scan --backend nous

The default remote model is nousresearch/hermes-4-70b; override it with --model or NOUS_MODEL. Any model id or alias from the portal catalogue works — GET https://inference-api.nousresearch.com/v1/models lists them, and a startup preflight checks yours is in there before the run touches a single post.

The key is read from the environment on every call. It is never written to colony_config.json, never logged, and every message that carries text up from the transport layer is passed through a redactor first — so a stack trace or a connection error cannot put it in a scan log that later gets pasted into an issue.

What moves with the switch, and what doesn't

Choosing nous moves more than the endpoint, and each of these was a way to get a half-switched run:

• The default model. --model has no baked-in default; it resolves after the backend is known. Otherwise --backend nous with no --model would ship the Ollama tag qwen3.5:9b-q4_k_m to a catalogue that has never heard of it.
• The preflight. ollama pull … is not the fix for a remote model, so the remote path checks the portal catalogue instead — honouring its aliases, since the portal publishes the same model as both nousresearch/hermes-4-70b and Hermes-4-70B.
• The GPU gate. Skipped entirely. Whether this box has a GPU is not a fact about a run happening on someone else's hardware.
• The startup reachability probe. A host that opted out of Ollama should never be told "Ollama not running".
• The model banner. It names Nous Portal, so a log still answers "what produced this judgement?" — which matters more, not less, once the answer may not be this machine.

What does not change: the prompt, the judgement schema, the action pipeline, the memory format, and the failure contract. Every error path returns None, exactly as the local backend does, so a failed call leaves the post unrecorded and it is retried on the next run.

Sampling options

NOUS_OPTIONS deliberately carries no max_tokens, for the same reason OLLAMA_OPTIONS carries no num_predict: Hermes is a reasoning model, a token cap is spent on the reasoning, and the JSON verdict arrives empty. The runaway bound is the wall-clock NOUS_TIMEOUT. Reasoning output is not requested, and response_format: {"type": "json_object"} pins the reply to a bare JSON object — the OpenAI-shaped counterpart of Ollama's "format": "json".

GPU requirement

Both modes refuse to start if Ollama would answer from the CPU. This section is about the local backend only; --backend nous skips the check.

This is not about crashes. Ollama silently falls back to CPU when a model does not fit in VRAM, and the run then works — at roughly a hundredth of the speed. A scan that takes seconds per post takes minutes, most of them ending at OLLAMA_TIMEOUT, and the whole thing gets through almost nothing while reporting success. That is the same shape as the bug where the scanner re-read the front page: a run that looks fine and does no work.

The check asks Ollama, not nvidia-smi. They answer different questions. nvidia-smi says whether this box has an NVIDIA GPU the driver can see; what matters is whether Ollama put this model on a GPU. Those come apart in three ways that all happen in practice:

• OLLAMA_HOST may point at another machine, making the local GPU irrelevant;
• a present GPU may be full, so the model lands on CPU anyway;
• Ollama also runs on ROCm and Metal, where nvidia-smi is absent on a perfectly good machine.

So the gate reads size_vram from Ollama's own /api/ps, which reports what actually happened on whichever host is serving. nvidia-smi is still consulted — but only to enrich the error message when the gate trips.

The model has to be resident before /api/ps can report anything, so the preflight loads it first with an empty-prompt generate. The run was about to load it anyway, so the cost is borrowed rather than added.

Three outcomes:

State	Behaviour
Fully or mostly in VRAM (>= GPU_MIN_OFFLOAD)	Logs GPU: 'model' is 100% resident in VRAM and proceeds
Partially offloaded (below the threshold, above zero)	Warns and proceeds — partial offload is legal and sometimes fine
Entirely on CPU (0%)	Aborts with the Ollama host, the local GPU inventory, and the fix
Unknown (older Ollama, model dropped, daemon unreachable)	Warns and proceeds — "we cannot tell" is not "it is on CPU"

To run on CPU deliberately — a box with no GPU where a slow scan is acceptable — pass --allow-cpu, or set SENTINEL_REQUIRE_GPU=0.

How --limit is counted

--limit N means analyse up to N posts, not "fetch N and analyse whatever survives".

Several filters run after the fetch — posts already in local memory, posts older than --days, and the sentinel's own posts. Fetching exactly N therefore routinely analysed fewer than N, and said nothing about it. --limit 10 quietly delivering three is the same shape as the bug where the scanner fetched the newest ten and skipped all ten: a run that reports success having done less work than it was asked for.

So the scan fetches a candidate window of N × SCAN_FETCH_MULTIPLIER (capped at SCAN_FETCH_CAP), and stops once N posts have actually been analysed. Anything past the budget is left unanalysed and unmarked, so the next run picks it up. If the window runs dry first, the run says so explicitly rather than finishing quietly:

Analysed 2 of a requested 10 — the 30-post candidate window held no more
eligible posts (already analysed, older than --days 7, or the sentinel's own).

Why the window is read all at once

The fetch is deliberately materialised (list(...)) before the first post is processed, and that is load-bearing rather than incidental.

Processing a post marks it scanned, which removes it from the sentinel_scanned=false set the server is paginating — and iter_posts pages by offset. Consuming the fetch lazily would have the window slide underneath the scan: mark k posts, and the next page starts k further into a set that just lost k members, skipping k posts that were never seen. Reading the whole window before the first mark is what keeps the offsets stable.

tests/test_scan_batch_semantics.py pins this by asserting no post is pulled off the generator after the first analysis begins.

Vote-exemption lists

Two optional files on the sentinel's own filesystem name users this instance must never vote on:

File	Effect
never_upvote.txt	Posts by these users are never upvoted
never_downvote.txt	Posts by these users are never downvoted

Both are gitignored — they are local policy for one box, not something to publish. Copy the shipped templates to get started:

cp never_downvote.txt.example never_downvote.txt
$EDITOR never_downvote.txt

Format: one username per line. Blank lines and # comments (whole-line or trailing) are ignored, a leading @ is stripped so a pasted mention works, and matching is case-insensitive.

# never auto-penalise these — escalate to me instead
trusted-human
@jorwhol          # pasted mentions work

Notes worth knowing before you rely on them:

• The two lists are independent. Listing someone under never_downvote.txt does not stop upvotes. To exempt a user from voting entirely, add the name to both files. This is deliberate — the common case is "don't let a local model penalise this account unattended", and the arm's-length case ("don't let my own agent boost my other account") is the opposite direction. Collapsing them into one list would take away the ability to express either alone.

• An exemption suppresses the vote and nothing else. A JUNK post by a listed user is still marked junk, still gets its language tagged, still gets PII-flagged, and is still marked scanned. Use --no-vote if you want to stop the whole moderation family.

• Edits take effect immediately. The files are re-read whenever they change, so you can add a name while a long-running webhook process is up without restarting it.

• Sentinel resolves these paths relative to the working directory, the same as colony_config.json and colony_analyzed.json. Both modes log the absolute path they consulted at startup, so a list sitting in the wrong directory is visible in the log rather than silently inert:

  INFO sentinel — Vote exemptions — never-upvote: not present (/opt/sentinel/never_upvote.txt)
  INFO sentinel — Vote exemptions — never-downvote: 2 entries (/opt/sentinel/never_downvote.txt)
  

  Point them elsewhere with SENTINEL_NEVER_UPVOTE_FILE / SENTINEL_NEVER_DOWNVOTE_FILE.

• One bounded gap. A vote that failed and was persisted to _pending_actions by a sentinel older than this feature carries no author, so it can't be re-checked when it replays. Such a vote is cast. The queue drains within 5 attempts, so this only spans the first few runs after upgrading. Votes queued by this version onward carry the author and are re-checked against the list as it stands at replay time — so listing a user also stops a vote that was already queued for them.

Logging

All output goes through Python's logging module (format: timestamp LEVEL sentinel — message).

Both modes announce the model they are about to use as the second line of the run, before any work happens:

2026-08-11 09:14:02,110 INFO sentinel — Sentinel — scan mode
2026-08-11 09:14:02,110 INFO sentinel — Model: qwen3.5:9b-q4_k_m (Ollama at http://localhost:11434)

The model can come from --model, from SENTINEL_MODEL / NOUS_MODEL, or from the built-in per-backend default, and a log without this line can't tell you which one served a given batch of judgements. The endpoint is included because the model name alone doesn't identify what served it once OLLAMA_HOST points off-box — or once --backend nous means the judgement was not produced on this machine at all:

2026-08-30 14:02:11,004 INFO sentinel — Sentinel — scan mode
2026-08-30 14:02:11,004 INFO sentinel — Model: nousresearch/hermes-4-70b (Nous Portal at https://inference-api.nousresearch.com/v1)

For systemd, journalctl -u sentinel -f captures everything; for cron, redirect stderr/stdout to a file:

*/15 * * * * cd /opt/sentinel && colony_venv/bin/python sentinel.py scan >> /var/log/sentinel.log 2>&1

Tests

Sentinel has a pure-unit test suite covering the action pipeline, retry replay, memory persistence, SDK wrappers, the sandbox-colony cache, webhook-worker dedup, and both inference backends. No network, no Ollama, no Nous Portal, no live API — everything goes through mocks. Run with:

make test          # creates venv if needed, installs requirements-dev.txt, runs pytest
# or
pip install -r requirements-dev.txt
pytest -q

CI runs the suite on every push and PR against Python 3.10–3.12 (see .github/workflows/tests.yml).

The highest-leverage tests live in tests/test_actions.py — every new judgement field landing in SYSTEM_PROMPT should come with a corresponding test there so memory entries written by an older sentinel version can still be replayed by a newer one without surprises.

Failed-action retry

Actions that fail (e.g. a 502 on vote, a transient SDK timeout) are persisted on the post's memory entry under _pending_actions. At the next scan startup and at webhook-worker startup, sentinel replays them before analyzing new posts. After 5 unsuccessful attempts (e.g. post deleted), the pending actions are dropped.

Files

File	Description
sentinel.py	Main script (scan + webhook + webhook-register subcommands)
requirements.txt	colony-sdk + requests (used for both inference backends — the remote one adds no dependency)
ruff.toml	Pins the lint rule set so CI's meaning doesn't drift with ruff releases
colony_config.json	API key and username (gitignored)
never_upvote.txt / never_downvote.txt	Local vote-exemption lists (gitignored; .example templates are committed)
colony_analyzed.json	Memory of analyzed posts
sentinel.lock	Lockfile preventing concurrent scan runs
Makefile	Convenience targets

License

MIT