diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 26a95a21..4d986961 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -33,6 +33,9 @@ jobs: - name: Routing tables in sync with skill-routing.tsv (render-routing.sh --check) run: bin/render-routing.sh --check + - name: MDL rules by kind cover every rule and match skills/generated (mdl-rules.sh --check) + run: bin/mdl-rules.sh --check + - name: No macOS-only assumptions reintroduced (check-portability.sh) run: bin/check-portability.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 571162b7..5235b2b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- new(mdl-rules): **`bin/mdl-rules.sh | --for ` prints the MDL rules one script needs, by the kinds of statement it writes** — lazy-load PR 2a. `bin/lib/mdl-rule-kinds.tsv` maps page / microflow / access / workflow / entity / any to sections located by file and heading text (never line numbers): the STOP rows by their `Kind` column, the kind-specific gotchas, `learned-microflow-patterns.md`, `microflow-preflight.md`, `ui-preflight-pages.md`, `design-spacing.md` §1–3, the detection-gaps register rows by construct, `security-is-not-a-later-script.md` and `retesting-learned-rules.md`. Output is the core (Step 0, the `any` STOP rows, gotchas, final self-check: 3,347 words) plus the kind sections, closed by "Rules for kinds K: STOP rows n/n listed" (page 10,223 words, microflow 10,186, access 4,730, entity 5,183, workflow 3,732), against the 29,388 words the mdl helper preloads at Stage 5 today. `--for` reuses exec.sh's statement classifier (`exec.sh --subjects`; Java actions and mappings count as microflow, demo users as access) and exits 1 if it fails; a heading that is gone fails loudly, never prints empty. `skills/generated/rules-.md` are the same output for harnesses that read files (Copilot, Cursor, Windsurf); `--check` (CI) fails on drift, on any STOP row, gotcha, register row or `##` section that reaches no kind (a section left out on purpose needs an `exclude` row with its reason), and on any register ERE that matches no row by itself, and on an unclosed code fence in a file it reads. No routing change yet: PR 2b demotes the preloads. Behaviour suite: `build-packs.sh --rules ""` expands the script in a pack. Fixture `tests/wave2/test-mdl-rules.sh` — lazy-load plan, option C (maintainer) - process(preflight): **`learned-mdl-preflight.md` moves its STOP-row narrative out of the baseline and gains a `Kind` column (page / microflow / access / workflow / any)** — lazy-load PR 1. The 15 retired rows collapse to one line listing their numbers, the version notes move, and each live row keeps its If/Then text plus one failure clause; the full rows, retired rows and version notes are in `process/baseline-incident-history.md` § PF1, verbatim (`check-moved-text.sh`: 43 of 43 removed lines found). The workflow `DECISION` STOP and the #879 read-first pointer stay in the skill. `Kind` sits at the end of each row so `mdl-scan`'s `| 🔎` markers still match. Baseline 73,946 → 72,891 words (−1,055). — maintainer (Option C lazy-loading plan) - fix(progress-report): **`progress-report.py` no longer crashes with `TypeError: '<' not supported between instances of 'dict' and 'dict'` when two exec'd scripts share a leading number** (#513). `script_phases()` sorted `(number, phase dict)` pairs, so `10-data.mdl` and `10b-data-flows.mdl` (or one script logged under two paths), both matched to a plan phase, compared the dicts and every `exec.sh` run ended with `WARN: progress-report.html not updated`. It now sorts on (number, script name) only; an unmatched numbered script still takes the phase of the nearest lower-numbered matched one, ties broken by name. Fixture: `test-progress-report.sh` T17 (fails 3/3 on master). — jopterhorst (#513) - new(canvas): **`project-bin/mf-canvas-facts.py` prints how a microflow or nanoflow lies on the Studio Pro canvas: labels on top of each other, lines through objects, note lines leaving the side that faces away, flows that run right-to-left or curl, captions wider than their shape, and decision captions that ask the opposite of their true branch (SENSE).** It prints facts, one line each, and no verdicts; read-only (it reads units through `mxpatch.py`), exit 2 when a flow could not be read. `verify-module.sh` runs it per module as an info rung (1c, log `14-canvas-facts.log`; a missing script, Python or `.mpr`, an exit 2 or a timeout is an INFO row reading NOT MEASURED, never a gate). Fixture `tests/wave2/test-mf-canvas-facts.sh` runs on a model mxcli v0.27.0 wrote (units and `.mpr` captured); `test-verify-module-446.sh` gains V8 for the rung. Calibration is from Studio Pro 10.24 screenshots, not verified on Studio Pro 11. Field evidence, measured 2026-10-21 on one Mendix 10.24 app: 1,588 flows read in one pass; on the 36 flows an mxcli-built branch added or changed in one module, OVERLAP 19, LINE 27, NOTE-SIDE 17, LEFTWARD 3, TURN 4, SENSE 9 before repair — saingge diff --git a/bin/lib/mdl-rule-kinds.tsv b/bin/lib/mdl-rule-kinds.tsv new file mode 100644 index 00000000..204c4214 --- /dev/null +++ b/bin/lib/mdl-rule-kinds.tsv @@ -0,0 +1,74 @@ +# mdl-rule-kinds.tsv — which MDL rules a script needs, by the kind of statement it writes. +# Read by bin/mdl-rules.sh (and nothing else); rendered to skills/generated/rules-.md. +# +# WHY (lazy-load PR 2, 2026-10-23). The mdl helper preloaded every kind-specific skill at Stage 5 +# (29,388 words), though a page script needs none of the microflow rules and a microflow script +# none of the page rules. This table says which sections each statement kind needs, so the helper +# loads them per script instead. The skills keep every word; only who preloads them changes. +# +# Sections are located by FILE plus HEADING TEXT, never by line number (the bin/lib/runbook-span.sh +# precedent): a heading matches when its text, after the leading #s, starts with the heading cell. +# A heading that no longer exists makes bin/mdl-rules.sh fail loudly — it never prints empty. +# +# Format: TAB-separated, four columns. Blank lines and #-comments ignored. +# 1 kind page | microflow | access | workflow | entity | any (any = every script) +# 2 file repo-relative path +# 3 selector file the whole file +# top everything above the file's first "## " heading +# section the heading and everything under it, down to the next heading of its level +# stop the STOP table under that heading, rows filtered by their Kind column +# (the kinds asked for, plus `any`; the retired-numbers row always) +# bullet one bullet under that heading: arg is "::" +# rows the table rows under that heading whose FIRST cell matches an ERE +# (case-insensitive): arg is "::" +# exclude kind "-": a `##` section deliberately reaching no kind (it is not a rule +# for writing MDL); arg is "::" +# 4 arg heading text, or heading::lead / heading::ERE / heading::reason as above; "-" for +# file and top +# +# Denominators (bin/mdl-rules.sh --check): every STOP row carries a known Kind, every bullet under +# a `bullet` heading is selected by some row here, every table row under a `rows` heading is matched +# by some ERE here and every ERE matches some row on its own, every `##` section of a file read in +# parts (no `file` row) is selected or excluded below, and every generated file equals a fresh +# render. A new gotcha, register row or section that no kind selects fails --check, so it cannot +# silently stop reaching the mdl helper. Headings inside code fences are not headings. +# +# kind file selector arg +any skills/learned-mdl-preflight.md top - +any skills/learned-mdl-preflight.md section Step 0 — choose the write mode +any skills/learned-mdl-preflight.md stop The checklist — run before writing a single line of MDL +any skills/learned-mdl-preflight.md bullet MDL gotchas::**Quote / unquote retry rule:** +any skills/learned-mdl-preflight.md bullet MDL gotchas::**Grants are co-located +any skills/learned-mdl-preflight.md bullet MDL gotchas::**`mxcli check` does not catch corruption.** +any skills/learned-mdl-preflight.md bullet MDL gotchas::**mxcli v0.25 — keep the skills' spellings +any skills/learned-mdl-preflight.md bullet MDL gotchas::**`exec` is not `check`.** +page skills/learned-mdl-preflight.md bullet MDL gotchas::**Cross-module association paths +microflow skills/learned-mdl-preflight.md bullet MDL gotchas::**Cross-module association paths +page skills/learned-mdl-preflight.md bullet MDL gotchas::**Changing an existing DataGrid 2 column +page skills/learned-mdl-preflight.md bullet MDL gotchas::**No `elseif` in page/widget expressions** +entity skills/learned-mdl-preflight.md bullet MDL gotchas::**Enumeration values named with reserved words** +access skills/learned-mdl-preflight.md bullet MDL gotchas::**Demo users:** +any skills/learned-mdl-preflight.md section Final self-check +page skills/ui-preflight-pages.md file - +page skills/design-spacing.md top - +page skills/design-spacing.md section 1. The scale +page skills/design-spacing.md section 2. The rhythm hierarchy +page skills/design-spacing.md section 3. The page scaffold +- skills/design-spacing.md exclude 4. Review-side::judging a screenshot at module review (module-review.md 4d), not writing MDL +- skills/design-spacing.md exclude Related::cross-references to other skills, no rule of its own +microflow skills/learned-microflow-patterns.md file - +microflow skills/microflow-preflight.md file - +access skills/learned-microflow-patterns.md section Current User — Two Separate Mechanisms +entity skills/learned-microflow-patterns.md section Association Direction — Reading SHOW ASSOCIATIONS Correctly +access skills/security-is-not-a-later-script.md file - +entity skills/security-is-not-a-later-script.md file - +any skills/learned-detection-gaps.md top - +any skills/learned-detection-gaps.md section The ladder +any skills/learned-detection-gaps.md rows The register::non-idempotent|mxcli < v0\.26 +page skills/learned-detection-gaps.md rows The register::page|widget|datagrid|dg2|dropdownfilter|navigation|dynamictext|contentparams|elseif|bindings|combo box|snippetcall +microflow skills/learned-detection-gaps.md rows The register::java action|getvalue|commit|microflow|on error|create entity|xpath|currentuser|import mapping|get workflows|loop +entity skills/learned-detection-gaps.md rows The register::entity|attribute|enumeration|reserved|calculated by|autocreated +workflow skills/learned-detection-gaps.md rows The register::workflow|parallel split +access skills/learned-detection-gaps.md rows The register::access|currentuser|role +any skills/learned-detection-gaps.md section Operating rules that fall out of this table +any skills/retesting-learned-rules.md file - diff --git a/bin/lib/routing-exempt.tsv b/bin/lib/routing-exempt.tsv index af87f44c..647a976d 100644 --- a/bin/lib/routing-exempt.tsv +++ b/bin/lib/routing-exempt.tsv @@ -20,3 +20,4 @@ skills/checkpoints/ reached via the checkpoint mechanism (conversion-runbook sta skills/journey-map.md tombstone — falsification trial failed 2026-08-22, row removed then; file kept so old links resolve skills/ui-review-loop.md merge tombstone — content lives in skills/module-review.md (routed) skills/module-completion-loop.md merge tombstone — content lives in skills/module-review.md (routed) +skills/generated/ rendered output of bin/mdl-rules.sh (rules-.md, drift-checked by bin/mdl-rules.sh --check) — the plain-read twin of that script's output, reached through bin/mdl-rules.sh's own routing row, never routed one by one diff --git a/bin/mdl-rules.sh b/bin/mdl-rules.sh new file mode 100755 index 00000000..854cc8dd --- /dev/null +++ b/bin/mdl-rules.sh @@ -0,0 +1,408 @@ +#!/usr/bin/env bash +# mdl-rules.sh — print the MDL rules one script needs, by the kinds of statement it writes. +# +# bin/mdl-rules.sh page entity # the core (kind `any`) plus the page and entity rules +# bin/mdl-rules.sh --for # the kinds that script's statements write, then the same +# bin/mdl-rules.sh --render # rewrite skills/generated/rules-.md +# bin/mdl-rules.sh --check # coverage + drift of those files (CI); prints nothing else +# +# Kinds: page, microflow, access, workflow, entity, any. The output is the always-needed core +# (learned-mdl-preflight.md's Step 0, its `any` STOP rows, gotchas and final self-check; the +# detection-gaps ladder; retesting-learned-rules.md), then the sections bin/lib/mdl-rule-kinds.tsv +# maps to each kind asked for, then one closing line: +# Rules for kinds : STOP rows / listed +# The mdl helper reports that count back as "STOP rows for kinds : n/n checked" in its PREFLIGHT. +# +# WHY (lazy-load PR 2, 2026-10-23). The mdl helper preloaded every kind-specific skill at Stage 5 — +# 29,388 words — although a page script needs none of the microflow rules and a microflow script +# none of the page rules. This prints the rules for the statements actually being written, the +# way bin/runbook-slice.sh prints one stage of the runbook and bin/bug-lookup.sh one ledger entry. +# The skills keep every word; only who preloads them changes. Harnesses that cannot run a script +# (Copilot, Cursor, Windsurf) read the same output as a file: skills/generated/rules-.md. +# +# --for reuses exec.sh's statement classifier (`project-bin/exec.sh --subjects`, the one lexer +# exec.sh, affected-tests.sh and the change class share) rather than a second copy of it. Subject +# kinds map: page|snippet|layout → page; microflow|nanoflow|rule|action|mapping → microflow (a Java +# action or an import/export mapping is called from a microflow); access|userrole|role, and a +# CREATE/ALTER/DROP DEMO USER head (exec.sh reports it `unmapped`, or by its login entity) → access; +# workflow → workflow; entity|association|enumeration → entity. Anything else (module, constant, +# structure, other unmapped heads) gets the core only, and the header names it. A classifier that +# fails (exec.sh missing or erroring) exits 1 with nothing on stdout — never the core alone. +# +# Never prints empty: a heading the table names that is gone, a bullet lead or a register ERE that +# selects nothing (each ERE on its own, not only the kind's union), or a STOP count that does not +# add up exits 1 with the reason on stderr and nothing on stdout. Headings inside ``` / ~~~ code +# fences are not headings. Reads skills only: no mxcli, no mxbuild, no project. bash 3.2 + POSIX awk. +# +# Exit: 0 printed / clean · 1 lookup failed, coverage gap or drift · 2 usage (unknown kind, no file) +set -eu +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +TSV="${MXTK_MDL_RULES_TSV:-$ROOT/bin/lib/mdl-rule-kinds.tsv}" +GEN_DIR="$ROOT/skills/generated" +ALL_KINDS="page microflow access workflow entity any" +LC_ALL=C; export LC_ALL + +usage() { sed -n '4,8p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; } +die() { echo "mdl-rules: $*" >&2; exit 1; } +[ -f "$TSV" ] || die "no kind table at $TSV" + +valid_kind() { case " $ALL_KINDS " in *" $1 "*) return 0 ;; esac; return 1; } + +# _extract — the rules text for those kinds (plus any) on stdout, or exit 1. One awk +# program reads the table and every file it names: rows are grouped by (file, selector, heading), +# so two kinds selecting the same gotcha or register row print it once, and a whole-file row +# suppresses that file's section rows. +_extract() { + awk -v want=" any $* " -v root="$ROOT" -v tsv="$TSV" ' + function err(m) { print "mdl-rules: " m > "/dev/stderr"; failed = 1 } + function load(f, p, l, n) { + if (f in N) return N[f] > 0 + p = root "/" f; n = 0 + while ((getline l < p) > 0) { + L[f, ++n] = l + if (l ~ /^[ \t]*(```|~~~)/) { infence = !infence; FE[f, n] = 1 } else FE[f, n] = infence + } + close(p); N[f] = n + if (infence) err("unclosed code fence in " f " (an odd number of ``` / ~~~ lines hides every heading after it)") + infence = 0 + if (n == 0) err("cannot read " f) + return n > 0 + } + # hlevel: the heading level of line j of f, 0 for a non-heading or a line inside a code fence. + function hlevel(f, j, s) { s = L[f, j]; if (FE[f, j] || s !~ /^#+ /) return 0; match(s, /^#+/); return RLENGTH } + function htext(s) { sub(/^#+[ \t]+/, "", s); return s } + # findh: the line number of the first heading in f whose text starts with h, else 0. + function findh(f, h, j) { + for (j = 1; j <= N[f]; j++) if (hlevel(f, j) && index(htext(L[f, j]), h) == 1) return j + return 0 + } + # send: the line after the section starting at s ends (next heading at its level or above). + function send(f, s, lv, j) { + lv = hlevel(f, s) + for (j = s + 1; j <= N[f]; j++) if (hlevel(f, j) && hlevel(f, j) <= lv) return j + return N[f] + 1 + } + function emit(s) { buf = buf s "\n" } + function trim(s) { gsub(/^[ \t]+|[ \t]+$/, "", s); return s } + function src(f, h) { emit(""); emit("> Source: `" f "`" (h == "" ? "" : " § " h)) } + function cells(line, c, n, i) { # split a table row; c[1..n] are the cells, trimmed + n = split(line, c, "|"); for (i = 1; i <= n; i++) c[i] = trim(c[i]); return n + } + BEGIN { + while ((getline row < tsv) > 0) { + if (row ~ /^[ \t]*#/ || row ~ /^[ \t]*$/) continue + nf = split(row, t, "\t") + if (nf < 4) { err("malformed table row: " row); continue } + if (index(want, " " t[1] " ") == 0) continue + f = t[2]; sel = t[3]; arg = t[4]; h = arg; part = "" + if (sel == "bullet" || sel == "rows") { + k = index(arg, "::"); if (k == 0) { err("no :: in " sel " arg: " arg); continue } + h = substr(arg, 1, k - 1); part = substr(arg, k + 2) + } + if (sel == "file" || sel == "top") h = "" + key = f SUBSEP sel SUBSEP h + if (!(key in seen)) { seen[key] = ++ng; gf[ng] = f; gs[ng] = sel; gh[ng] = h } + g = seen[key] + if (sel == "bullet") { nb[g]++; lead[g, nb[g]] = part } + if (sel == "rows") { nr[g]++; rx[g, nr[g]] = part } + if (sel == "file") whole[f] = 1 + } + close(tsv) + stops_listed = 0; stops_total = 0 + for (g = 1; g <= ng; g++) { + f = gf[g]; sel = gs[g]; h = gh[g] + if (sel != "file" && (f in whole)) continue + if (!load(f)) continue + if (sel == "file") { src(f, ""); for (j = 1; j <= N[f]; j++) emit(L[f, j]); continue } + if (sel == "top") { + src(f, "top"); n0 = 0 + for (j = 1; j <= N[f] && hlevel(f, j) != 2; j++) { emit(L[f, j]); n0++ } + if (n0 == 0) err(f ": nothing above its first ## heading") + continue + } + s = findh(f, h) + if (!s) { err(f ": heading not found: \"" h "\" (" sel ") — fix bin/lib/mdl-rule-kinds.tsv"); continue } + e = send(f, s) + if (sel == "section") { + src(f, h); for (j = s; j < e; j++) emit(L[f, j]) + if (e - s < 2) err(f ": section \"" h "\" is empty") + continue + } + if (sel == "bullet") { + src(f, h) + for (j = s; j < e && L[f, j] !~ /^- /; j++) emit(L[f, j]) + for (i = 1; i <= nb[g]; i++) hit[i] = 0 + for (; j < e; j++) { + if (L[f, j] !~ /^- /) continue + on = 0 + for (i = 1; i <= nb[g]; i++) if (index(substr(L[f, j], 3), lead[g, i]) == 1) { on = 1; hit[i]++ } + if (!on) continue + emit(L[f, j]) + for (x = j + 1; x < e && L[f, x] !~ /^- / && L[f, x] !~ /^#/ && L[f, x] !~ /^[ \t]*$/; x++) emit(L[f, x]) + } + for (i = 1; i <= nb[g]; i++) if (!hit[i]) err(f ": no bullet under \"" h "\" starts with " lead[g, i]) + continue + } + if (sel == "rows" || sel == "stop") { + src(f, h); tab = 0 + for (i = 1; i <= nr[g]; i++) rhit[i] = 0 + for (j = s; j < e; j++) { + line = L[f, j] + if (line !~ /^\|/) { if (tab < 3 || sel == "stop") emit(line); if (tab) tab = 3; continue } + tab++ + if (tab <= 2) { emit(line); continue } # header and separator + n = cells(line, c) + if (sel == "rows") { + on = 0 + for (i = 1; i <= nr[g]; i++) if (tolower(c[2]) ~ rx[g, i]) { on = 1; rhit[i]++ } + if (on) emit(line) + } else { + kd = c[n - 1] + if (kd == "") { emit(line); continue } # the retired-numbers row: kept for citations + if (index(want, " " kd " ")) { emit(line); stops_listed++ } + } + } + if (tab == 0) err(f ": no table under \"" h "\"") + if (sel == "rows") for (i = 1; i <= nr[g]; i++) if (!rhit[i]) err(f ": no row under \"" h "\" matches /" rx[g, i] "/") + if (sel == "stop") stops_file = f SUBSEP h + continue + } + err("unknown selector " sel " in " tsv) + } + if (stops_file == "") err("the table has no stop row for kind any — nothing would list the STOP table") + if (failed) exit 1 + printf "%s", buf + printf "STOPS\t%d\n", stops_listed > "/dev/stderr" + }' +} + +# _stop_total — the denominator, counted by a second, independent pass over the STOP +# table: rows whose Kind cell is one of the kinds (or any). The two must agree or the run fails. +_stop_total() { + local f h + f="$(awk -F'\t' '$1 == "any" && $3 == "stop" { print $2; exit }' "$TSV")" + h="$(awk -F'\t' '$1 == "any" && $3 == "stop" { print $4; exit }' "$TSV")" + awk -v want=" any $* " -v h="$h" ' + /^[ \t]*(```|~~~)/ { fe = !fe; next } + fe { next } + index($0, "## " h) == 1 { on = 1; next } + on && /^## / { exit } + on && /^\|/ { n = split($0, c, "|"); k = c[n - 1]; gsub(/^[ \t]+|[ \t]+$/, "", k) + if (k != "" && k != "Kind" && k !~ /^-+$/ && index(want, " " k " ")) t++ } + END { print t + 0 }' "$ROOT/$f" +} + +# print_rules