[will-exaforce]

Summary

• Adds two static "lure" rules, on by default, that flag install lines piping a download from a raw IP address or a paste site into a shell, or base64-decoding a payload into an interpreter. This is the delivery style of ClawHavoc, the one real campaign in our corpus: +41 malware caught on the full benchmark, and no benign verdict changed.
• Adds an opt-in block threshold (SKILLSPECTOR_EXA_BLOCK_THRESHOLD=49; the default stays at upstream's 51). It passed its benchmark test (+0.87 pt recall, −0.40 pt precision), but at real-world malware rates it blocks several benign skills per extra catch, so it is not the default.
• Adds benchmark tooling for same-day, replicated comparisons (arms, screen, replay, health, evalset, rulelab, report extras). It is what let us reject five other attempts (P1–P4, P7) cheaply. Their code is rolled back and their records are kept.
• Outcome at the shipped defaults, against the original run: recall 0.709 → 0.725, precision 0.928 → 0.930. Only the lure rules' +0.0064 recall is ours. The rest is drift in the model service, so the gain is small; the tooling and the record of what doesn't work are most of this PR's value.

Details

Start here. benchmark/results/recall-improvement-program.md is the entry point: a summary, each plan's verdict and why, lessons, and next steps ranked by expected value. About 55k of the ~65k added lines are data in benchmark/results/ (evaluation sets, the dev/holdout split, screen manifests, per-plan reports). Apart from the write-up, reviewers can skip that folder.

Classifier (src/skillspector/exaforce/). Runtime patches only; no upstream file is edited.

• _lure_rules.py:107: the EXA-LU1 / EXA-LU4 regexes, severity HIGH at confidence 0.9, with the paste-host list at :80. Every gap is bounded and long lines are scanned in overlapping windows, so crafted input can't make the scan slow or hide the lure by padding the line. SKILLSPECTOR_EXA_LURE_RULES=off disables them.
• _score_patches.py:40: the threshold flag. Unset means 51, and the upstream result passes through unchanged. At any other value the score is clamped (:71) so that the CLI exit code, MCP safe_to_install and the recommendation agree.
• _analyzer_registry.py:71, _fork_context.py:30, _patchlib.py:100: plumbing the lure rules need.
  • The registry adds a fork-owned analyzer node.
  • The fork context exempts fork findings from upstream's "looks like a code example" drop; otherwise an attacker could append "e.g." to a lure line to have it ignored.
  • intercept_module_callable wraps an upstream function with signature drift checks.

Tests. Upstream tests that the patches invert by design are never edited. They are marked strict-xfail in the root conftest.py and listed in docs/superpowers/EXPECTED_TEST_FAILURES.md. The five test_report entries apply only while the threshold is 50 or below.

Benchmark (benchmark/). P0's tooling, rulelab (offline match counts for candidate static rules), replay threshold support, a worker-count default, a dropped-LLM-call count per scan (the basis of health), and a fix that stores a non-string explanation as JSON instead of aborting the run (79176a33).

Known limits of the lure rules. EXA-LU1 does not exclude private IP ranges and does include gist URLs, so a homelab-style curl http://192.168.1.10/install.sh | bash is flagged HIGH. None of the units they fire on are in the holdout split, so there is no out-of-sample measurement. The case for them is that the patterns are generic and match a real campaign.

Plans: docs/superpowers/plans/2026-10-29-recall-p0-measurement.md … -p7-model-split.md; spec docs/superpowers/specs/2026-10-29-recall-improvement-program.md.
Tests: 2,046 passed, 9 xfailed (uv run pytest -m "not integration and not provider" tests/); 2,041 passed, 14 xfailed with SKILLSPECTOR_EXA_BLOCK_THRESHOLD=49; benchmark 214 passed (cd benchmark && uv run pytest).

Implementation Plan

Each plan pre-registered its candidates and decision rule before any run. Claims are same-day comparisons on the same code tree, replicated, with arms that differ only in env flags. A rejected plan's code was rolled back in a revert commit and its record kept.

plan	change	verdict	code
pre-P0	--workers default and dropped-call counts (9e8c48cb); why upstream 2.13.1 is on hold (c3cc13f2)	n/a	kept
P0	measurement tooling, baseline-v2 run	n/a	kept
P1	semantic-finding gate, doc-finding cap	rejected (false-positive cost on dev)	rolled back 5a0bbb04
P2	text from PDF/office files, hidden-text rules	no verdict (1 replicate)	rolled back 68032655
P3	skill-context header, per-skill instruction-flow analyzer	rejected (false-positive cost, run health)	rolled back 1b950d70
P4	necessity audit (SSD-5, EXA-NE1)	rejected (false-positive cost, run health)	rolled back b9354821
P5	lure rules EXA-LU1, EXA-LU4	accepted	on by default
P6	block threshold 51 → 49, score clamp	accepted; made opt-in at PR review (525a426f)	shipped, opt-in
P7	stronger model for the SSD slot	rejected (too little gain)	records rolled back cb417b57

Main lesson for whoever comes next: about 90% of the malware the scanner misses gets no high-severity finding from the model under any of four models. The limit is detection, not the scoring cap.