diff --git a/CHANGELOG.md b/CHANGELOG.md index 1cd9cac9..7f8e7cb7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- fix(bug-lookup): **`bin/bug-lookup.sh` also searches the project's own `bug-logs/**.md` drafts, marked `(project draft: :)`, after the ledger's output.** It searched only the toolkit ledger, so a bug a session had already written down in the project was invisible to the lookup run before the next experiment on the same error; the ledger's output and ranking are unchanged (drafts are ranked in a separate pass), `--full` and `--index` stay ledger-only, and `MXTK_PROJECT_DIR=` turns drafts off. Field: a Mendix 10.24 app with mxcli v0.27.0, 43 files in `bug-logs/`; the query "cannot be spliced" listed 0 project drafts at 393d253 and lists 3 (3 distinct files) now, each pointing at a `## ` line; the project was not written. Pinned by group P of `tests/wave2/test-bug-lookup.sh` (3 of its checks fail on 393d253) on synthetic drafts; the run under /bin/bash 3.2.57 passes, other awks not run. — saingge - fix(harvest-learnings): **section 1 drafted only the first `## ` section of each bug-log file.** The awk splitter's `print buf "\x01"` put a newline after each delimiter, so every later section began with an empty line, read an empty heading and was skipped. It is `printf "%s\x01", buf` now. Field run on a Mendix 10.24 app with mxcli v0.27.0 (43 bug-log files, 54 `## ` sections): `harvest-learnings.sh` at 393d253 drafted 43, the fix drafts 54. Pinned by `tests/wave2/test-harvest-learnings.sh` T5, on a two-file fixture that keeps a captured log's structure with its text replaced (fixed: 7 of 7 sections, 20 of 20 checks; 393d253: 2 of 7, T5 fails). The splitter's delimiter bytes were checked under macOS awk 20200816 only. — saingge - fix(script-owners): **the restatement guard now parses under macOS's default bash 3.2, so `exec.sh` no longer skips it silently** (guard added in #498). `exec.sh` runs `bash script-owners.sh check … 2>/dev/null || true`; under /bin/bash 3.2.57 the file failed to parse (an apostrophe in a comment inside `$(...)`, then `case` patterns without a leading `(`), so the call (and the ledger `record` call) printed nothing and warned nothing. Fixture `test-restatement-guard.sh` under /bin/bash 3.2.57: 16 of 58 pass before, 58 of 58 after; `bin/check-scripts.sh` now also parses every script with /bin/bash when the bash on PATH is a different one. — saingge - fix(facts-lock): **`check` no longer reports "N BRD(s) agree with all 0 frozen fact(s)" when nothing was compared** (#519). A corpus whose BRDs carry identifiers only under keys the harvest does not read (`domainEntities[].name`, `pages[].name`) froze an empty lock, and `check` printed what reads as a pass. With 0 frozen facts, `check` now prints `PENDING facts-lock: 0 identifiers found under any known key in N BRD(s): the lock is empty and check proves nothing (keys read: ...)`, or, when identifiers exist but no lock was built, `PENDING ... run build first; check compared nothing`; `build` prints the same empty-lock line when it froze 0 facts and found no conflicts. Exit codes are unchanged (an exit-3 "nothing measured" and harvesting `name` keys are left for a separate change). Fixture: `test-facts-lock.sh` (the issue's repro: 3 fail on master; 19/19). — AlvinnYeo (#519) diff --git a/bin/bug-lookup.sh b/bin/bug-lookup.sh index 5a38665a..8a21d938 100755 --- a/bin/bug-lookup.sh +++ b/bin/bug-lookup.sh @@ -7,7 +7,7 @@ # bin/bug-lookup.sh '' # ranked: top 3 — header, score, one excerpt line # bin/bug-lookup.sh --full 2 '' # the 2nd ranked entry, in full # bin/bug-lookup.sh --exact "widget name" # literal phrase match, case-insensitive -# bin/bug-lookup.sh --index # one line per entry: id, title, status flags +# bin/bug-lookup.sh --index # one line per entry: id, title, status flags (ledger only) # # WHY. bug-logs/mxcli-bugs.md is far too large to read on every session (`wc -w` it for the # current size; a number written here goes stale). Nobody needs all of it; they need the entry @@ -20,8 +20,18 @@ # term in the header x1.5, an exact BUG id in the header +100. "No confident match" is printed # when the best entry covers < MXTK_BUG_MINCOV (default 0.40) of the query's term weight, or # when the query's CE code is in no entry. Measured on tests/wave2/fixtures/bug-lookup/. -# Exit: 0 entries printed (the "no confident match" banner too: it still lists the nearest 3), -# 1 nothing matches or no query, 2 no ledger or a bad --full rank. +# PROJECT DRAFTS. Run from inside a wired project, it also searches that project's own +# bug-logs/**.md: the drafts a session writes for a bug it hit, one `## ` entry each, which the +# ledger does not carry until someone merges them. Draft hits come AFTER the ledger's output, +# under a "Project drafts" line, each marked `(project draft: :)`. The ledger's +# output and ranking are the same with or without drafts (they are ranked in a separate pass, +# so they never move the ledger's IDF). A ranked draft is shown only when it covers 40% of the +# query's term weight; a ledger hit below that is shown under a banner, a draft is dropped. +# The project is found as every toolkit script finds it (project-bin/_common.sh, mxtk_walk_root +# from $PWD); MXTK_PROJECT_DIR= names it, MXTK_PROJECT_DIR= (empty) turns drafts off. +# --full N and --index are ledger-only. +# Exit: 0 entries printed (the "no confident match" banner too: it still lists the nearest 3; +# a draft hit alone counts), 1 nothing matches or no query, 2 no ledger or a bad --full rank. # Bash 3.2 + POSIX awk (mawk, gawk, BWK awk); no gawk-isms. set -eu ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" @@ -48,13 +58,41 @@ done Q="$*" [ -n "$Q" ] || usage +# --- The project's own drafts: bug-logs/**.md beside the project, never the ledger itself --- +DRAFTS=() +if [ -n "${MXTK_PROJECT_DIR+x}" ]; then PD="$MXTK_PROJECT_DIR" +else PD="$( . "$ROOT/project-bin/_common.sh" >/dev/null 2>&1; mxtk_walk_root "$PWD" 2>/dev/null )" || PD=""; fi +if [ -n "$PD" ] && [ -d "$PD/bug-logs" ]; then + while IFS= read -r _f; do [ "$_f" = "$LOG" ] || DRAFTS+=("$_f"); done \ + < <(find "$PD/bug-logs" -name '*.md' -type f | sort) +fi +# draft_word: the draft entries that contain "$Q" (literal, ASCII case-insensitive), in full up +# to 6 of them, headers only above. Prints nothing and returns 1 when none match. +draft_word() { + [ "${#DRAFTS[@]}" -gt 0 ] || return 1 + BUGQ="$Q" LC_ALL=C awk -v PD="$PD/" ' + function flush( i) { if (cur && index(tolower(body), q)) { n++; H[n] = hdr; W[n] = where; B[n] = body } cur = 0 } + BEGIN { q = tolower(ENVIRON["BUGQ"]) } + FNR == 1 { flush() } + /^## / { flush(); if ($0 ~ /^## Provenance/) next + cur = 1; hdr = $0; body = $0 "\n"; f = FILENAME + if (index(f, PD) == 1) f = substr(f, length(PD) + 1) + where = "(project draft: " f ":" FNR ")"; next } + cur { body = body $0 "\n" } + END { flush(); if (!n) exit 1 + if (n > 6) { print "bug-lookup: project drafts: " n " entries mention \"" ENVIRON["BUGQ"] "\" — headers only:" + for (i = 1; i <= n; i++) print " " substr(H[i], 4) " " W[i]; exit 0 } + print "bug-lookup: project drafts (" n "):" + for (i = 1; i <= n; i++) { print W[i]; printf "%s\n", B[i] } }' "${DRAFTS[@]}" +} + # --- Ranked path: several words, or --full ------------------------------------------------ case "$Q" in *[[:space:]]*) MULTI=1 ;; *) MULTI=0 ;; esac if [ -n "$FULL" ] || { [ "$MULTI" = 1 ] && [ "$EXACT" = 0 ]; }; then TOP="${MXTK_BUG_TOP:-3}"; [ -n "$FULL" ] && [ "$FULL" -gt "$TOP" ] && TOP="$FULL" # The query goes in through ENVIRON, not -v: -v expands backslash escapes (a pasted Windows # path would change). LC_ALL=C: byte semantics, so every awk tokenises and truncates alike. - OUT="$(BUGQ="$Q" LC_ALL=C awk -v TOP="$TOP" -v MINCOV="${MXTK_BUG_MINCOV:-0.40}" -v FULL="${FULL:-0}" ' + RANK_AWK=' function toks(s, out, n, i, a, t, m) { # s -> out[token]=1 ; returns count s = tolower(s); n = 0 while (index(s, "bug-") && match(s, /(bug-[a-z]+[0-9]+|bug-[0-9]+|bug-draft-[a-z0-9-]+)/)) { @@ -77,7 +115,7 @@ BEGIN { nq = toks(ENVIRON["BUGQ"], QT) } /^## / { if ($0 ~ /^## Provenance/) { cur = 0; next } - N++; cur = N; HDR[N] = substr($0, 4); START[N] = NR + N++; cur = N; HDR[N] = substr($0, 4); START[N] = NR; SRC[N] = FILENAME; LNO[N] = FNR delete H; toks(HDR[N], H); for (t in H) HT[N SUBSEP t] = 1 } cur { delete L; toks($0, L) @@ -98,22 +136,37 @@ END { for (k = 1; k <= TOP; k++) { m = 0; mi = 0 for (e = 1; e <= N; e++) if (!(e in USED) && SC[e] > m) { m = SC[e]; mi = e } if (!mi) break; USED[mi] = 1; R[k] = mi } - if (!(1 in R)) { print "bug-lookup: no ledger entry shares a term with the query — likely not a known tool quirk."; exit 1 } + if (!(1 in R)) { if (!DRAFT) print "bug-lookup: no ledger entry shares a term with the query — likely not a known tool quirk."; exit 1 } if (FULL > 0) { if (FULL in R) { print "@" START[R[FULL]]; exit 0 } print "bug-lookup: fewer than " FULL " entries share a term with the query."; exit 1 } cov = (TOT > 0) ? SC[R[1]] / (TOT * 1.5) : 0; if (cov > 1) cov = 1 + if (DRAFT && (UNK != "" || cov < MINCOV)) exit 1 if (UNK != "") printf "bug-lookup: no ledger entry mentions%s — likely not a known tool quirk. Nearest by wording:\n", UNK else if (cov < MINCOV) printf "bug-lookup: no confident match (best covers %.0f%% of query weight) — likely not a known tool quirk. Nearest:\n", cov * 100 for (k = 1; k <= TOP && (k in R); k++) { e = R[k] + if (DRAFT && TOT > 0 && SC[e] / (TOT * 1.5) < MINCOV) break x = EX[e SUBSEP BT[e]]; gsub(/^[ \t>*`-]+/, "", x) if (length(x) > 160) { x = substr(x, 1, 157); sub(/[\300-\377][\200-\277]*$/, "", x); x = x "..." } - printf "%d. [%.1f] %s\n %s\n", k, SC[e], HDR[e], x } - print "(--full N with the same query prints entry N in full)" -}' "$LOG")" || { printf '%s\n' "$OUT"; exit 1; } - case "$OUT" in - @*) awk -v S="${OUT#@}" 'NR >= S { if (NR > S && /^## /) exit; print }' "$LOG" ;; - *) printf '%s\n' "$OUT" ;; - esac + if (DRAFT) { f = SRC[e]; if (index(f, PD) == 1) f = substr(f, length(PD) + 1) + printf "%d. [%.1f] %s (project draft: %s:%d)\n %s\n", k, SC[e], HDR[e], f, LNO[e], x } + else printf "%d. [%.1f] %s\n %s\n", k, SC[e], HDR[e], x } + if (!DRAFT) print "(--full N with the same query prints entry N in full)" +}' + # rank ...: the ledger (draft 0) or the project drafts (draft 1) + rank() { BUGQ="$Q" LC_ALL=C awk -v TOP="$TOP" -v MINCOV="${MXTK_BUG_MINCOV:-0.40}" -v FULL="$1" -v DRAFT="$2" -v PD="$PD/" "$RANK_AWK" "${@:3}"; } + RC=0; OUT="$(rank "${FULL:-0}" 0 "$LOG")" || RC=$? + DOUT="" + if [ -z "$FULL" ] && [ "${#DRAFTS[@]}" -gt 0 ]; then DOUT="$(rank 0 1 "${DRAFTS[@]}")" || DOUT=""; fi + if [ "$RC" -ne 0 ]; then + printf '%s\n' "$OUT" + [ -n "$DOUT" ] || exit 1 + else + case "$OUT" in + @*) awk -v S="${OUT#@}" 'NR >= S { if (NR > S && /^## /) exit; print }' "$LOG" ;; + *) printf '%s\n' "$OUT" ;; + esac + fi + [ -z "$DOUT" ] || printf 'Project drafts (not in the ledger):\n%s\n' "$DOUT" exit 0 fi @@ -134,9 +187,10 @@ for h in "${HDR[@]}"; do if sed -n "${a},${b}p" "$LOG" | grep -qiF -- "$Q"; then matches="$matches $h"; fi done set -- $matches -if [ "$#" -eq 0 ]; then echo "bug-lookup: nothing in the ledger mentions '$Q' — it may be a modelling mistake, not a tool quirk. Try a shorter keyword or the CE code."; exit 1; fi +if [ "$#" -eq 0 ]; then echo "bug-lookup: nothing in the ledger mentions '$Q' — it may be a modelling mistake, not a tool quirk. Try a shorter keyword or the CE code."; draft_word && exit 0; exit 1; fi if [ "$#" -gt 6 ]; then echo "bug-lookup: $# entries mention '$Q' — headers only; narrow the query to read one:" - for h in "$@"; do sed -n "${h}p" "$LOG" | sed 's/^## / /'; done; exit 0 + for h in "$@"; do sed -n "${h}p" "$LOG" | sed 's/^## / /'; done; draft_word || true; exit 0 fi for h in "$@"; do read -r a b < <(entry_range "$h"); sed -n "${a},${b}p" "$LOG"; echo; done +draft_word || true diff --git a/tests/wave2/fixtures/bug-lookup/project-drafts/bug-logs/2026-01-02-order-flow.md b/tests/wave2/fixtures/bug-lookup/project-drafts/bug-logs/2026-01-02-order-flow.md new file mode 100644 index 00000000..1beb6259 --- /dev/null +++ b/tests/wave2/fixtures/bug-lookup/project-drafts/bug-logs/2026-01-02-order-flow.md @@ -0,0 +1,17 @@ +# Draft findings, synthetic (test fixture: invented names and messages, no real project) + +## BUG-DRAFT-rewire-refusal: ALTER MICROFLOW refuses to rewire an activity (2026-01-02) + +Version: mxcli v0.27.0, Mendix 10.24. + +- Expected: `ALTER MICROFLOW Shop.ACT_Order` moves the Approve activity behind the new decision. +- Actual: `Shop.ACT_Order cannot be rewired: no base activity survives the edit`. +- Workaround: drop the flow and create it again in one script. + +## BUG-DRAFT-gallery-lost-ref: a gallery action leaves no catalog edge (2026-01-02) + +Version: mxcli v0.27.0. + +- Expected: `show references to Shop.DS_Orders` lists the page. +- Actual: `no references found` while `describe page Shop.Order_Overview` prints the datasource. +- Workaround: grep the `describe page` output. diff --git a/tests/wave2/fixtures/bug-lookup/project-drafts/bug-logs/2026-01-03-misc.md b/tests/wave2/fixtures/bug-lookup/project-drafts/bug-logs/2026-01-03-misc.md new file mode 100644 index 00000000..987b7008 --- /dev/null +++ b/tests/wave2/fixtures/bug-lookup/project-drafts/bug-logs/2026-01-03-misc.md @@ -0,0 +1,10 @@ +## BUG-DRAFT-frobnicate-limit: the frobnicate command stops at 50 items (2026-01-03) + +Version: mxcli v0.27.0. + +- Actual: `frobnicate limit reached: 50` with no way to raise it. +- Workaround: run it once per folder. + +## Provenance + +The rewire refusal above was seen twice in one week. diff --git a/tests/wave2/test-bug-lookup.sh b/tests/wave2/test-bug-lookup.sh index c2edfbc1..4c26e1a3 100644 --- a/tests/wave2/test-bug-lookup.sh +++ b/tests/wave2/test-bug-lookup.sh @@ -23,6 +23,9 @@ # L one word keeps the full-entry path: BUG-n, CE code, keyword, --index, --exact; --index # prints the same entries under a BSD-sed shim (\t read as t, #488) # F --full N prints the N-th ranked entry in full and stops at the next entry +# P project drafts: a wired project's own bug-logs/*.md are searched after the ledger, marked +# `(project draft: ...)`; the ledger part is byte-identical with drafts off; --full and +# --index stay ledger-only (fixtures/bug-lookup/project-drafts/, synthetic, invented text) # U usage: no query exits 1; --full 0 exits 2; a backslash in the query survives (`\\n` is # not read as a newline: same ranking as the query with the backslashes spaced out) set -uo pipefail @@ -32,6 +35,7 @@ SUBJ="${1:-$TOOLKIT/bin/bug-lookup.sh}" case "$SUBJ" in /*) ;; *) SUBJ="$PWD/$SUBJ" ;; esac FX="$TOOLKIT/tests/wave2/fixtures/bug-lookup" export MXTK_BUG_LOG="$TOOLKIT/bug-logs/mxcli-bugs.md" +export MXTK_PROJECT_DIR="" # drafts off for every group but P, whatever folder the run starts in WORK="$(mktemp -d "${TMPDIR:-/tmp}/buglookup.XXXXXX")" || exit 2 trap 'rm -rf "$WORK"' EXIT @@ -153,6 +157,38 @@ second=$(bash "$SUBJ" "$q" | grep '^2\. \[' | sed 's/^2\. \[[0-9.]*\] //' | cut [ -n "$second" ] && [ "$(bash "$SUBJ" --full 2 "$q" | head -1 | cut -c4-43)" = "$second" ] \ && ok "--full 2: the entry ranked 2nd" || bad "--full 2 is not the 2nd ranked entry" +echo "== P: project drafts" +PX="$FX/project-drafts" +o="$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" rewire)"; rc=$? +[ "$rc" = 0 ] && printf '%s\n' "$o" | grep -q '^(project draft: bug-logs/2026-01-02-order-flow.md:[0-9]*)$' \ + && printf '%s\n' "$o" | grep -q 'Shop.ACT_Order cannot be rewired' \ + && ok "one word: ledger says nothing, the draft entry follows in full, marked, exit 0" || bad "one word: exit $rc: $(printf '%s\n' "$o" | head -3)" +bash "$SUBJ" rewire >/dev/null 2>&1; [ $? = 1 ] && ok "drafts off (MXTK_PROJECT_DIR empty): the same word exits 1" || bad "drafts off: wrong exit" +o="$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" 'Shop.ACT_Order cannot be rewired')"; rc=$? +[ "$rc" = 0 ] && printf '%s\n' "$o" | grep -q '^Project drafts (not in the ledger):$' \ + && printf '%s\n' "$o" | grep -q '^1\. \[[0-9.]*\] BUG-DRAFT-rewire-refusal:.*(project draft: bug-logs/2026-01-02-order-flow.md:[0-9]*)$' \ + && ok "ranked: the best draft is listed first, marked, exit 0 although the ledger has no match" || bad "ranked: exit $rc: $(printf '%s\n' "$o" | head -4)" +! printf '%s\n' "$o" | grep -q 'gallery-lost-ref' && ok "ranked: a draft below the 40% coverage floor is not listed" || bad "ranked: a weak draft was listed" +! printf '%s\n' "$o" | grep -q 'frobnicate' && ok "ranked: the Provenance section is not an entry" || bad "ranked: Provenance listed" +o="$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" provenance 2>&1)"; ! printf '%s\n' "$o" | grep -q 'seen twice' \ + && ok "one word: Provenance text is not searched in drafts" || bad "one word: Provenance text was found" +q='[error] [CE0710] "The main flow cannot join an error flow or end in an error event."' +[ "$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" "$q")" = "$(bash "$SUBJ" "$q")" ] \ + && ok "ranked ledger query no draft matches: output identical to drafts off" || bad "ranked ledger output changed by drafts" +[ "$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" BUG-102)" = "$(bash "$SUBJ" BUG-102)" ] \ + && ok "BUG-102: output identical to drafts off" || bad "BUG-102 output changed by drafts" +[ "$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" --full 1 "$q")" = "$(bash "$SUBJ" --full 1 "$q")" ] \ + && [ "$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" --index)" = "$(bash "$SUBJ" --index)" ] \ + && ok "--full 1 and --index: ledger only, identical to drafts off" || bad "--full/--index changed by drafts" +# Found from the working directory the way a session runs it: inside a wired project. +PRJ="$WORK/proj"; mkdir -p "$PRJ/.claude" "$PRJ/bug-logs" "$PRJ/deep/er"; : > "$PRJ/PROJECT.md" +cp "$PX"/bug-logs/*.md "$PRJ/bug-logs/" +o="$(cd "$PRJ/deep/er" && env -u MXTK_PROJECT_DIR bash "$SUBJ" rewire)"; rc=$? +[ "$rc" = 0 ] && printf '%s\n' "$o" | grep -q '^(project draft: bug-logs/2026-01-02-order-flow.md:' \ + && ok "found from a subfolder of the wired project (no variable set)" || bad "cwd discovery: exit $rc: $(printf '%s\n' "$o" | head -2)" +o="$(cd "$TOOLKIT" && env -u MXTK_PROJECT_DIR bash "$SUBJ" rewire 2>&1)"; rc=$? +[ "$rc" = 1 ] && ok "outside any project (the toolkit itself): no drafts, exit 1" || bad "outside a project: exit $rc: $(printf '%s\n' "$o" | head -2)" + echo "== U: usage" bash "$SUBJ" >/dev/null 2>&1; [ $? = 1 ] && ok "no query: exit 1" || bad "no query: wrong exit" bash "$SUBJ" --full 0 x >/dev/null 2>&1; [ $? = 2 ] && ok "--full 0: exit 2" || bad "--full 0: wrong exit"