Skip to content

process(routing, mdl-agent): mdl helper loads MDL rules per statement type at Stage 5 (lazy-load PR 2b) - #572

Merged
MendixMau merged 7 commits into
masterfrom
claude/loving-gates-hauc1n-lazy-pr2b
Oct 23, 2026
Merged

MendixMau merged 7 commits into
masterfrom
claude/loving-gates-hauc1n-lazy-pr2b

Conversation

@MendixMau

@MendixMau MendixMau commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Direct lane

Classification: DISCUSS. The maintainer approved this as option C of the lazy-load plan (decision #16 = B: run more page reps before opening).

What changed and why (one paragraph):
At Stage 5 the mdl helper used to preload 14 baseline files (29,388 routed words), whatever the script wrote. Now it gets a smaller core and runs bin/mdl-rules.sh <kinds> (added in PR 2a, #571) to load the rules for the statement types each script writes. The STOP rows, gotchas and preflight sections still reach it in full, but only for the types the script writes. Three files move to on demand: learned-microflow-patterns.md, microflow-preflight.md and ui-preflight-pages.md. Five files go on demand for the mdl role only (mdl?) and stay as they were for every other role: learned-mdl-preflight.md, design-spacing.md, learned-detection-gaps.md, security-is-not-a-later-script.md and retesting-learned-rules.md. A new baseline row mdl-rules (Stages 5 and 6) takes their place. The mdl stub gains step 4a and the report line "STOP rows for kinds K: n/n checked". No skill text changed.

Also in this PR:

  • gate-check build-ready item 2 fix. The rendered table no longer lists ui-preflight-pages.md, so a project synced after this change would have flipped to FAIL. bin/gate-check.sh now accepts bin/mdl-rules.sh as well as ui-preflight-pages. A project synced before the change still passes on its old table. Fixture: tests/wave2/test-mdl-rules.sh T13 covers both layouts.
  • Behaviour scorer fixes, found by the A/B below:
    • tests/behaviour/score.sh mfpre now accepts :, an em/en dash or - after microflow-preflight. Captured fixture tests/behaviour/fixtures/L-mf-r3.md, asserted in tests/wave2/test-behaviour-score.sh.
  • CLAUDE.md: the always-on line no longer lists learned-microflow-patterns.md.
  • A/B results: tests/behaviour/results/2026-10-23-lazy-pr2b.tsv.
  • Independent-review fixes (1f84c6d):
    • agents/mdl-agent.md step 4a and the report line name <toolkit>/bin/mdl-rules.sh --for (a consuming project's bin/ does not carry it).
    • project-bin/context-pack.sh "Rules" list now names bin/mdl-rules.sh <kinds> under the toolkit root (or <toolkit>/ when the root is not found) with the step's kinds (page / logic → microflow / domain → entity), read ALL of it. After this PR, microflow-preflight, security-is-not-a-later-script, design-spacing, learned-detection-gaps and retesting-learned-rules reach a pack-fed helper only through mdl-rules. tests/wave2/test-context-pack.sh asserts it for a page step and a logic step.
    • mdl-rules routing row is stage 5,6, so Stage 6 mdl keeps detection-gaps and retesting: mdl Stage 6 6,129 → 9,476 words (below 10,093).
  • Re-review fix (4fa05df): CHANGELOG states the per-script saving by kind and what the A/B covered.

Impact

Row Answer
Process No new human steps or gates. The mdl helper runs bin/mdl-rules.sh per script (stub step 4a); a pack-fed helper is told to by the context pack. Gate-check item 2 accepts either routing name, so no verdict flips to FAIL.
Dependencies Shared files touched: bin/lib/skill-routing.tsv, bin/lib/skill-routing.sh and their re-render (README.md, ROUTING.md), agents/mdl-agent.md, bin/gate-check.sh, project-bin/context-pack.sh. Depends on PR 2a (#571, merged). New artifact read: bin/mdl-rules.sh output, produced by the mandatory mdl stub step 4a.
Context cost Baseline 72,891 → 63,877 words (budget 80,000). Always-loaded mdl Stage 5: 29,388 → 14,600 words (the 3,347-word core of mdl-rules.sh counted, like the runbook slice). Per script the helper then reads mdl-rules output on top, measured by kind: page only ≈21.4k words in total (≈8k saved), microflow+access ≈22.5k (≈6.9k saved), page+microflow ≈28.1k (≈1.3k saved), all kinds ≈29.5k (≈0 saved). So the per-script saving is ≈0–8k words depending on the kinds a script writes, not the full 14.8k. mdl Stage 6: 9,476 words.
Behaviour A/B run, M = master pack and L = 2b pack + mdl-rules.sh output. Verdict: L no worse than M except crumbs at 3/3 (M 2/15, L 1/15, one rep, same median 2/3; reported per RUNBOOK:67-68). Table below.
Runtime n/a: routing only. mdl-rules.sh runs in under a second per call.

A/B (tests/behaviour/RUNBOOK.md)

Setup

  • One model (claude-opus-5-5) for every rep.
  • A fresh claude -p agent per rep, with Read/Write only, started from an empty cwd with no CLAUDE.md above it.
  • Arms interleaved.
  • Packs rebuilt at 87676f8 (M) and 59bc033 (L). Apart from the header ref they are byte-identical to the packs used in run 1.
  • L packs carry the mdl-rules.sh output for the task's statement types.
  • Every rep read its pack to the end (read_all=1).

What the A/B tested: the L arm was handed mdl-rules output with kinds pre-selected, so the A/B shows the rules suffice when read; it does not test that the agent runs step 4a on its own — the context-pack line (this PR) and PR 3 cover delivery.

Reproducibility: the trap tasks and the trap-18 scorer are not committed, so trap rows are not reproducible from the repo.

Runs

  • Run 1: 5 page reps, 5 mf reps and 8 trap types (trap 18 ×2) per arm.
  • Run 2: 10 more page reps per arm, because run 1 had L page uixref 4/5 against M 5/5.

Re-scoring run 1 with the fixed scorers

  • mfpre: only L-mf-r3 changed, 0→1. Its two checklist lines are written microflow-preflight — … — tier: Guided.
  • Trap 18: only L-trap-18-r1/r2 changed, planted 0→1. Both reps split the work into two scripts, marked with -- FILE n of 2: headers inside the single fenced block the task format requires. The trap scorer now counts per script, meaning a fenced block or one of those headers. It is the session's own scorer and is not committed.
  • No M score moved.
  • Negative control: L-trap-18-r1 with its FILE 2 header deleted still scores planted=0.

Page, 15 reps per arm (median; count of reps at the good value)

metric M L
sections 4/4 15/15 15/15
report 15/15 15/15
wmode (median) 7/6 7/6
quoted (median; reps at 100) 100; 9/15 100; 14/15
folder (median) 1/3 1/3
scan = 0 15/15 15/15
planted (STOP #4) 15/15 15/15
crumbs (median; reps at 3/3) 2/3; 2/15 2/3; 1/15
txtcrumb = 0 15/15 15/15
h1 2/2 15/15 15/15
empty 15/15 15/15
uixref 11/15 (run 1: 5/5, run 2: 6/10) 14/15 (run 1: 4/5, run 2: 10/10)
tool_calls (median) 7 6
tokens (median) 341k 292k

The uixref misses look the same in both arms: the Step 5 checks are posted as NOT RUN, with no named "UI cross-reference" block.

Finding, reported as the decision rule requires: crumbs reached 3/3 in 2/15 M reps against 1/15 L reps. The median is 2/3 in both arms, so this is a one-rep difference.

Microflow, 5 reps per arm

metric M L
sections, report, folder, scan, planted, cinloop, rinloop, pos, vfb all at the good value, 5/5 all at the good value, 5/5
mfpre 5/5 5/5 (4/5 before the scorer fix)
quoted (median) 90 95
wmode (median) 3/3 4/4
tool_calls (median) 7 6

Planted traps (planted / flagged)

trap M L
12 1/1 1/1
16 1/1 1/1
18 (×2) 2/2 2/2 (0/2 before the scorer fix)
1b 1/1 1/1
26 1/1 1/1
29 1/1 1/1
33 1/1 1/1

Field evidence: owed. This PR has only the A/B; a field run on a real project comes with PR 3.

  • No client data anywhere in the diff
  • Size cap. Excluding fixtures, results and CHANGELOG, the diff is within the cap.
  • Test tier (review-fix commit 1f84c6d, scoped, in a worktree): test-mdl-rules 43/0, test-context-pack 59/0, test-baseline-pack 22/0, test-routing-tier 26/0, test-render-routing 5/0, test-runbook-slice 31/0, test-bug12-sync 62/0, test-bug03-gates 47/0; render-routing.sh --check, mdl-rules.sh --check, check-portability.sh clean. Earlier: the mfpre regex was replayed over the 4 committed fixtures (no change; N-mf-r1 stays 0) and over all 20 run-1 page/mf outputs (only L-mf-r3 changed); score.sh on the new fixture prints mfpre=1. Behaviour A/B run (above).
  • Golden input captured: L-mf-r3 is a verbatim agent output.
  • CHANGELOG.md line appended (updated for the context pack, Stage 6 and the per-script saving)

Review: MERGE-AFTER-FIX → fixed (two review rounds; remaining findings were text only, fixed in 4fa05df)


Generated by Claude Code

bin/mdl-rules.sh <kind...> | --for <script.mdl> | --render | --check prints the
always-needed core plus the sections bin/lib/mdl-rule-kinds.tsv maps to each
statement kind (page, microflow, access, workflow, entity, any), located by
file + heading text, and closes with "Rules for kinds K: STOP rows n/n listed".
--for reuses exec.sh --subjects. skills/generated/rules-<kind>.md are the
rendered, drift-checked plain-read copies; CI runs --check. No routing change:
lazy-load PR 2b demotes the mdl preloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
…loaded (lazy-load PR 2b)

Demote learned-microflow-patterns, microflow-preflight and ui-preflight-pages to
on-demand; learned-mdl-preflight, design-spacing, learned-detection-gaps,
security-is-not-a-later-script and retesting-learned-rules to on-demand for mdl
only (mdl?). New baseline row bin/mdl-rules.sh, counted as its core output.
mdl stub gains step 4a + report line "STOP rows for kinds K: n/n checked".
gate-check build-ready item 2 accepts bin/mdl-rules.sh, so synced projects do
not flip to FAIL (test-mdl-rules.sh T7, both layouts).
Baseline 72,891 -> 63,877 words; mdl Stage 5 29,388 -> 14,600.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
Conflicts: bin/mdl-rules.sh and bin/lib/mdl-rule-kinds.tsv take master's
(reviewed) versions, which 2b never changed; 2b's gate fixture moves to
T13 after master's T7-T12; CHANGELOG keeps both lines.
test-mdl-rules 43/0, mdl-rules --check clean, baseline 63,877.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
…flight

The lazy-load 2b A/B scored L-mf-r3 mfpre=0 although it posted two full
checklist lines written 'microflow-preflight — Sales."…" — tier: Guided'.
The pattern now takes ':', an em/en dash or ' - ' after the name. Re-scored
all 20 page/mf outputs of that run: the only change is L-mf-r3 0 -> 1; no M
score moves; the no-pack fixture N-mf-r1 still scores 0. L-mf-r3 is added as
a captured fixture with its own assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
tests/behaviour/results/2026-10-23-lazy-pr2b.tsv: M (master pack) vs L (2b
pack + bin/mdl-rules.sh output), 15 page reps per arm, 5 mf, 8 trap kinds.
L holds: page uixref L 14/15 vs M 11/15; every other metric equal or
better except crumbs at 3/3 (M 2/15, L 1/15, same median 2/3).

CLAUDE.md: the always-on set no longer lists learned-microflow-patterns.md,
which 2b routes on demand; the mdl helper runs bin/mdl-rules.sh instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
…les in context packs, Stage 6

- agents/mdl-agent.md: step 4a's --for run and the report line name
  <toolkit>/bin/mdl-rules.sh (a project's bin/ does not carry it).
- project-bin/context-pack.sh: the "Rules" list names
  `<tk>bin/mdl-rules.sh <kinds>` with the step's kinds (page / microflow /
  entity); microflow-preflight, security-is-not-a-later-script,
  design-spacing, learned-detection-gaps and retesting-learned-rules now
  reach a pack-fed helper only through it. Fixture asserts a page step and
  a logic step.
- skill-routing.tsv: mdl-rules row stages 5,6, so Stage 6 mdl keeps
  detection-gaps and retesting (mdl Stage 6 6,129 -> 9,476 words, below
  10,093). Re-rendered README.md / ROUTING.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
Re-review of #572: the preload figure read as a flat 14.8k saving; give the
measured per-script range and what the A/B covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@MendixMau
MendixMau merged commit b082f4b into master Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants