fix(bug-lookup): also search the project's own bug-logs drafts (replaces #531)

CHANGELOG.md

@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
 
 ## Unreleased
 
+- fix(bug-lookup): **`bin/bug-lookup.sh` also searches the project's own `bug-logs/**.md` drafts, marked `(project draft: <file>:<line>)`, after the ledger's output.** It searched only the toolkit ledger, so a bug a session had already written down in the project was invisible to the lookup run before the next experiment on the same error; the ledger's output and ranking are unchanged (drafts are ranked in a separate pass), `--full` and `--index` stay ledger-only, and `MXTK_PROJECT_DIR=` turns drafts off. Field: a Mendix 10.24 app with mxcli v0.27.0, 43 files in `bug-logs/`; the query "cannot be spliced" listed 0 project drafts at 393d253 and lists 3 (3 distinct files) now, each pointing at a `## ` line; the project was not written. Pinned by group P of `tests/wave2/test-bug-lookup.sh` (3 of its checks fail on 393d253) on synthetic drafts; the run under /bin/bash 3.2.57 passes, other awks not run. — saingge
 - fix(harvest-learnings): **section 1 drafted only the first `## ` section of each bug-log file.** The awk splitter's `print buf "\x01"` put a newline after each delimiter, so every later section began with an empty line, read an empty heading and was skipped. It is `printf "%s\x01", buf` now. Field run on a Mendix 10.24 app with mxcli v0.27.0 (43 bug-log files, 54 `## ` sections): `harvest-learnings.sh` at 393d253 drafted 43, the fix drafts 54. Pinned by `tests/wave2/test-harvest-learnings.sh` T5, on a two-file fixture that keeps a captured log's structure with its text replaced (fixed: 7 of 7 sections, 20 of 20 checks; 393d253: 2 of 7, T5 fails). The splitter's delimiter bytes were checked under macOS awk 20200816 only. — saingge
 - fix(script-owners): **the restatement guard now parses under macOS's default bash 3.2, so `exec.sh` no longer skips it silently** (guard added in #498). `exec.sh` runs `bash script-owners.sh check … 2>/dev/null || true`; under /bin/bash 3.2.57 the file failed to parse (an apostrophe in a comment inside `$(...)`, then `case` patterns without a leading `(`), so the call (and the ledger `record` call) printed nothing and warned nothing. Fixture `test-restatement-guard.sh` under /bin/bash 3.2.57: 16 of 58 pass before, 58 of 58 after; `bin/check-scripts.sh` now also parses every script with /bin/bash when the bash on PATH is a different one. — saingge
 - fix(facts-lock): **`check` no longer reports "N BRD(s) agree with all 0 frozen fact(s)" when nothing was compared** (#519). A corpus whose BRDs carry identifiers only under keys the harvest does not read (`domainEntities[].name`, `pages[].name`) froze an empty lock, and `check` printed what reads as a pass. With 0 frozen facts, `check` now prints `PENDING facts-lock: 0 identifiers found under any known key in N BRD(s): the lock is empty and check proves nothing (keys read: ...)`, or, when identifiers exist but no lock was built, `PENDING ... run build first; check compared nothing`; `build` prints the same empty-lock line when it froze 0 facts and found no conflicts. Exit codes are unchanged (an exit-3 "nothing measured" and harvesting `name` keys are left for a separate change). Fixture: `test-facts-lock.sh` (the issue's repro: 3 fail on master; 19/19). — AlvinnYeo (#519)

bin/bug-lookup.sh

@@ -7,7 +7,7 @@
 #   bin/bug-lookup.sh '<pasted error message>'    # ranked: top 3 — header, score, one excerpt line
 #   bin/bug-lookup.sh --full 2 '<same message>'   # the 2nd ranked entry, in full
 #   bin/bug-lookup.sh --exact "widget name"       # literal phrase match, case-insensitive
-#   bin/bug-lookup.sh --index                     # one line per entry: id, title, status flags
+#   bin/bug-lookup.sh --index                     # one line per entry: id, title, status flags (ledger only)
 #
 # WHY. bug-logs/mxcli-bugs.md is far too large to read on every session (`wc -w` it for the
 # current size; a number written here goes stale). Nobody needs all of it; they need the entry
@@ -20,8 +20,18 @@
 # term in the header x1.5, an exact BUG id in the header +100. "No confident match" is printed
 # when the best entry covers < MXTK_BUG_MINCOV (default 0.40) of the query's term weight, or
 # when the query's CE code is in no entry. Measured on tests/wave2/fixtures/bug-lookup/.
-# Exit: 0 entries printed (the "no confident match" banner too: it still lists the nearest 3),
-# 1 nothing matches or no query, 2 no ledger or a bad --full rank.
+# PROJECT DRAFTS. Run from inside a wired project, it also searches that project's own
+# bug-logs/**.md: the drafts a session writes for a bug it hit, one `## ` entry each, which the
+# ledger does not carry until someone merges them. Draft hits come AFTER the ledger's output,
+# under a "Project drafts" line, each marked `(project draft: <file>:<line>)`. The ledger's
+# output and ranking are the same with or without drafts (they are ranked in a separate pass,
+# so they never move the ledger's IDF). A ranked draft is shown only when it covers 40% of the
+# query's term weight; a ledger hit below that is shown under a banner, a draft is dropped.
+# The project is found as every toolkit script finds it (project-bin/_common.sh, mxtk_walk_root
+# from $PWD); MXTK_PROJECT_DIR=<dir> names it, MXTK_PROJECT_DIR= (empty) turns drafts off.
+# --full N and --index are ledger-only.
+# Exit: 0 entries printed (the "no confident match" banner too: it still lists the nearest 3;
+# a draft hit alone counts), 1 nothing matches or no query, 2 no ledger or a bad --full rank.
 # Bash 3.2 + POSIX awk (mawk, gawk, BWK awk); no gawk-isms.
 set -eu
 ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
@@ -48,13 +58,41 @@ done
 Q="$*"
 [ -n "$Q" ] || usage
 
+# --- The project's own drafts: bug-logs/**.md beside the project, never the ledger itself ---
+DRAFTS=()
+if [ -n "${MXTK_PROJECT_DIR+x}" ]; then PD="$MXTK_PROJECT_DIR"
+else PD="$( . "$ROOT/project-bin/_common.sh" >/dev/null 2>&1; mxtk_walk_root "$PWD" 2>/dev/null )" || PD=""; fi
+if [ -n "$PD" ] && [ -d "$PD/bug-logs" ]; then
+  while IFS= read -r _f; do [ "$_f" = "$LOG" ] || DRAFTS+=("$_f"); done \
+    < <(find "$PD/bug-logs" -name '*.md' -type f | sort)
+fi
+# draft_word: the draft entries that contain "$Q" (literal, ASCII case-insensitive), in full up
+# to 6 of them, headers only above. Prints nothing and returns 1 when none match.
+draft_word() {
+  [ "${#DRAFTS[@]}" -gt 0 ] || return 1
+  BUGQ="$Q" LC_ALL=C awk -v PD="$PD/" '
+  function flush(   i) { if (cur && index(tolower(body), q)) { n++; H[n] = hdr; W[n] = where; B[n] = body } cur = 0 }
+  BEGIN { q = tolower(ENVIRON["BUGQ"]) }
+  FNR == 1 { flush() }
+  /^## / { flush(); if ($0 ~ /^## Provenance/) next
+           cur = 1; hdr = $0; body = $0 "\n"; f = FILENAME
+           if (index(f, PD) == 1) f = substr(f, length(PD) + 1)
+           where = "(project draft: " f ":" FNR ")"; next }
+  cur { body = body $0 "\n" }
+  END { flush(); if (!n) exit 1
+        if (n > 6) { print "bug-lookup: project drafts: " n " entries mention \"" ENVIRON["BUGQ"] "\" — headers only:"
+                     for (i = 1; i <= n; i++) print "  " substr(H[i], 4) "  " W[i]; exit 0 }
+        print "bug-lookup: project drafts (" n "):"
+        for (i = 1; i <= n; i++) { print W[i]; printf "%s\n", B[i] } }' "${DRAFTS[@]}"
+}
+
 # --- Ranked path: several words, or --full ------------------------------------------------
 case "$Q" in *[[:space:]]*) MULTI=1 ;; *) MULTI=0 ;; esac
 if [ -n "$FULL" ] || { [ "$MULTI" = 1 ] && [ "$EXACT" = 0 ]; }; then
   TOP="${MXTK_BUG_TOP:-3}"; [ -n "$FULL" ] && [ "$FULL" -gt "$TOP" ] && TOP="$FULL"
   # The query goes in through ENVIRON, not -v: -v expands backslash escapes (a pasted Windows
   # path would change). LC_ALL=C: byte semantics, so every awk tokenises and truncates alike.
-  OUT="$(BUGQ="$Q" LC_ALL=C awk -v TOP="$TOP" -v MINCOV="${MXTK_BUG_MINCOV:-0.40}" -v FULL="${FULL:-0}" '
+  RANK_AWK='
 function toks(s, out,   n, i, a, t, m) {        # s -> out[token]=1 ; returns count
   s = tolower(s); n = 0
   while (index(s, "bug-") && match(s, /(bug-[a-z]+[0-9]+|bug-[0-9]+|bug-draft-[a-z0-9-]+)/)) {
@@ -77,7 +115,7 @@ BEGIN {
   nq = toks(ENVIRON["BUGQ"], QT)
 }
 /^## / { if ($0 ~ /^## Provenance/) { cur = 0; next }
-         N++; cur = N; HDR[N] = substr($0, 4); START[N] = NR
+         N++; cur = N; HDR[N] = substr($0, 4); START[N] = NR; SRC[N] = FILENAME; LNO[N] = FNR
          delete H; toks(HDR[N], H); for (t in H) HT[N SUBSEP t] = 1 }
 cur {
   delete L; toks($0, L)
@@ -98,22 +136,37 @@ END {
   for (k = 1; k <= TOP; k++) { m = 0; mi = 0
     for (e = 1; e <= N; e++) if (!(e in USED) && SC[e] > m) { m = SC[e]; mi = e }
     if (!mi) break; USED[mi] = 1; R[k] = mi }
-  if (!(1 in R)) { print "bug-lookup: no ledger entry shares a term with the query — likely not a known tool quirk."; exit 1 }
+  if (!(1 in R)) { if (!DRAFT) print "bug-lookup: no ledger entry shares a term with the query — likely not a known tool quirk."; exit 1 }
   if (FULL > 0) { if (FULL in R) { print "@" START[R[FULL]]; exit 0 }
                   print "bug-lookup: fewer than " FULL " entries share a term with the query."; exit 1 }
   cov = (TOT > 0) ? SC[R[1]] / (TOT * 1.5) : 0; if (cov > 1) cov = 1
+  if (DRAFT && (UNK != "" || cov < MINCOV)) exit 1
   if (UNK != "") printf "bug-lookup: no ledger entry mentions%s — likely not a known tool quirk. Nearest by wording:\n", UNK
   else if (cov < MINCOV) printf "bug-lookup: no confident match (best covers %.0f%% of query weight) — likely not a known tool quirk. Nearest:\n", cov * 100
   for (k = 1; k <= TOP && (k in R); k++) { e = R[k]
+    if (DRAFT && TOT > 0 && SC[e] / (TOT * 1.5) < MINCOV) break
     x = EX[e SUBSEP BT[e]]; gsub(/^[ \t>*`-]+/, "", x)
     if (length(x) > 160) { x = substr(x, 1, 157); sub(/[\300-\377][\200-\277]*$/, "", x); x = x "..." }
-    printf "%d. [%.1f] %s\n     %s\n", k, SC[e], HDR[e], x }
-  print "(--full N with the same query prints entry N in full)"
-}' "$LOG")" || { printf '%s\n' "$OUT"; exit 1; }
-  case "$OUT" in
-    @*) awk -v S="${OUT#@}" 'NR >= S { if (NR > S && /^## /) exit; print }' "$LOG" ;;
-    *)  printf '%s\n' "$OUT" ;;
-  esac
+    if (DRAFT) { f = SRC[e]; if (index(f, PD) == 1) f = substr(f, length(PD) + 1)
+                 printf "%d. [%.1f] %s  (project draft: %s:%d)\n     %s\n", k, SC[e], HDR[e], f, LNO[e], x }
+    else printf "%d. [%.1f] %s\n     %s\n", k, SC[e], HDR[e], x }
+  if (!DRAFT) print "(--full N with the same query prints entry N in full)"
+}'
+  # rank <full-rank> <draft 0|1> <file>...: the ledger (draft 0) or the project drafts (draft 1)
+  rank() { BUGQ="$Q" LC_ALL=C awk -v TOP="$TOP" -v MINCOV="${MXTK_BUG_MINCOV:-0.40}" -v FULL="$1" -v DRAFT="$2" -v PD="$PD/" "$RANK_AWK" "${@:3}"; }
+  RC=0; OUT="$(rank "${FULL:-0}" 0 "$LOG")" || RC=$?
+  DOUT=""
+  if [ -z "$FULL" ] && [ "${#DRAFTS[@]}" -gt 0 ]; then DOUT="$(rank 0 1 "${DRAFTS[@]}")" || DOUT=""; fi
+  if [ "$RC" -ne 0 ]; then
+    printf '%s\n' "$OUT"
+    [ -n "$DOUT" ] || exit 1
+  else
+    case "$OUT" in
+      @*) awk -v S="${OUT#@}" 'NR >= S { if (NR > S && /^## /) exit; print }' "$LOG" ;;
+      *)  printf '%s\n' "$OUT" ;;
+    esac
+  fi
+  [ -z "$DOUT" ] || printf 'Project drafts (not in the ledger):\n%s\n' "$DOUT"
   exit 0
 fi
 
@@ -134,9 +187,10 @@ for h in "${HDR[@]}"; do
   if sed -n "${a},${b}p" "$LOG" | grep -qiF -- "$Q"; then matches="$matches $h"; fi
 done
 set -- $matches
-if [ "$#" -eq 0 ]; then echo "bug-lookup: nothing in the ledger mentions '$Q' — it may be a modelling mistake, not a tool quirk. Try a shorter keyword or the CE code."; exit 1; fi
+if [ "$#" -eq 0 ]; then echo "bug-lookup: nothing in the ledger mentions '$Q' — it may be a modelling mistake, not a tool quirk. Try a shorter keyword or the CE code."; draft_word && exit 0; exit 1; fi
 if [ "$#" -gt 6 ]; then
   echo "bug-lookup: $# entries mention '$Q' — headers only; narrow the query to read one:"
-  for h in "$@"; do sed -n "${h}p" "$LOG" | sed 's/^## /  /'; done; exit 0
+  for h in "$@"; do sed -n "${h}p" "$LOG" | sed 's/^## /  /'; done; draft_word || true; exit 0
 fi
 for h in "$@"; do read -r a b < <(entry_range "$h"); sed -n "${a},${b}p" "$LOG"; echo; done
+draft_word || true

tests/wave2/fixtures/bug-lookup/project-drafts/bug-logs/2026-01-02-order-flow.md

@@ -0,0 +1,17 @@
+# Draft findings, synthetic (test fixture: invented names and messages, no real project)
+
+## BUG-DRAFT-rewire-refusal: ALTER MICROFLOW refuses to rewire an activity (2026-01-02)
+
+Version: mxcli v0.27.0, Mendix 10.24.
+
+- Expected: `ALTER MICROFLOW Shop.ACT_Order` moves the Approve activity behind the new decision.
+- Actual: `Shop.ACT_Order cannot be rewired: no base activity survives the edit`.
+- Workaround: drop the flow and create it again in one script.
+
+## BUG-DRAFT-gallery-lost-ref: a gallery action leaves no catalog edge (2026-01-02)
+
+Version: mxcli v0.27.0.
+
+- Expected: `show references to Shop.DS_Orders` lists the page.
+- Actual: `no references found` while `describe page Shop.Order_Overview` prints the datasource.
+- Workaround: grep the `describe page` output.

tests/wave2/fixtures/bug-lookup/project-drafts/bug-logs/2026-01-03-misc.md

@@ -0,0 +1,10 @@
+## BUG-DRAFT-frobnicate-limit: the frobnicate command stops at 50 items (2026-01-03)
+
+Version: mxcli v0.27.0.
+
+- Actual: `frobnicate limit reached: 50` with no way to raise it.
+- Workaround: run it once per folder.
+
+## Provenance
+
+The rewire refusal above was seen twice in one week.

tests/wave2/test-bug-lookup.sh

@@ -23,6 +23,9 @@
 #   L  one word keeps the full-entry path: BUG-n, CE code, keyword, --index, --exact; --index
 #      prints the same entries under a BSD-sed shim (\t read as t, #488)
 #   F  --full N prints the N-th ranked entry in full and stops at the next entry
+#   P  project drafts: a wired project's own bug-logs/*.md are searched after the ledger, marked
+#      `(project draft: ...)`; the ledger part is byte-identical with drafts off; --full and
+#      --index stay ledger-only (fixtures/bug-lookup/project-drafts/, synthetic, invented text)
 #   U  usage: no query exits 1; --full 0 exits 2; a backslash in the query survives (`\\n` is
 #      not read as a newline: same ranking as the query with the backslashes spaced out)
 set -uo pipefail
@@ -32,6 +35,7 @@ SUBJ="${1:-$TOOLKIT/bin/bug-lookup.sh}"
 case "$SUBJ" in /*) ;; *) SUBJ="$PWD/$SUBJ" ;; esac
 FX="$TOOLKIT/tests/wave2/fixtures/bug-lookup"
 export MXTK_BUG_LOG="$TOOLKIT/bug-logs/mxcli-bugs.md"
+export MXTK_PROJECT_DIR=""   # drafts off for every group but P, whatever folder the run starts in
 WORK="$(mktemp -d "${TMPDIR:-/tmp}/buglookup.XXXXXX")" || exit 2
 trap 'rm -rf "$WORK"' EXIT
 
@@ -153,6 +157,38 @@ second=$(bash "$SUBJ" "$q" | grep '^2\. \[' | sed 's/^2\. \[[0-9.]*\] //' | cut
 [ -n "$second" ] && [ "$(bash "$SUBJ" --full 2 "$q" | head -1 | cut -c4-43)" = "$second" ] \
   && ok "--full 2: the entry ranked 2nd" || bad "--full 2 is not the 2nd ranked entry"
 
+echo "== P: project drafts"
+PX="$FX/project-drafts"
+o="$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" rewire)"; rc=$?
+[ "$rc" = 0 ] && printf '%s\n' "$o" | grep -q '^(project draft: bug-logs/2026-01-02-order-flow.md:[0-9]*)$' \
+  && printf '%s\n' "$o" | grep -q 'Shop.ACT_Order cannot be rewired' \
+  && ok "one word: ledger says nothing, the draft entry follows in full, marked, exit 0" || bad "one word: exit $rc: $(printf '%s\n' "$o" | head -3)"
+bash "$SUBJ" rewire >/dev/null 2>&1; [ $? = 1 ] && ok "drafts off (MXTK_PROJECT_DIR empty): the same word exits 1" || bad "drafts off: wrong exit"
+o="$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" 'Shop.ACT_Order cannot be rewired')"; rc=$?
+[ "$rc" = 0 ] && printf '%s\n' "$o" | grep -q '^Project drafts (not in the ledger):$' \
+  && printf '%s\n' "$o" | grep -q '^1\. \[[0-9.]*\] BUG-DRAFT-rewire-refusal:.*(project draft: bug-logs/2026-01-02-order-flow.md:[0-9]*)$' \
+  && ok "ranked: the best draft is listed first, marked, exit 0 although the ledger has no match" || bad "ranked: exit $rc: $(printf '%s\n' "$o" | head -4)"
+! printf '%s\n' "$o" | grep -q 'gallery-lost-ref' && ok "ranked: a draft below the 40% coverage floor is not listed" || bad "ranked: a weak draft was listed"
+! printf '%s\n' "$o" | grep -q 'frobnicate' && ok "ranked: the Provenance section is not an entry" || bad "ranked: Provenance listed"
+o="$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" provenance 2>&1)"; ! printf '%s\n' "$o" | grep -q 'seen twice' \
+  && ok "one word: Provenance text is not searched in drafts" || bad "one word: Provenance text was found"
+q='[error] [CE0710] "The main flow cannot join an error flow or end in an error event."'
+[ "$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" "$q")" = "$(bash "$SUBJ" "$q")" ] \
+  && ok "ranked ledger query no draft matches: output identical to drafts off" || bad "ranked ledger output changed by drafts"
+[ "$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" BUG-102)" = "$(bash "$SUBJ" BUG-102)" ] \
+  && ok "BUG-102: output identical to drafts off" || bad "BUG-102 output changed by drafts"
+[ "$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" --full 1 "$q")" = "$(bash "$SUBJ" --full 1 "$q")" ] \
+  && [ "$(MXTK_PROJECT_DIR="$PX" bash "$SUBJ" --index)" = "$(bash "$SUBJ" --index)" ] \
+  && ok "--full 1 and --index: ledger only, identical to drafts off" || bad "--full/--index changed by drafts"
+# Found from the working directory the way a session runs it: inside a wired project.
+PRJ="$WORK/proj"; mkdir -p "$PRJ/.claude" "$PRJ/bug-logs" "$PRJ/deep/er"; : > "$PRJ/PROJECT.md"
+cp "$PX"/bug-logs/*.md "$PRJ/bug-logs/"
+o="$(cd "$PRJ/deep/er" && env -u MXTK_PROJECT_DIR bash "$SUBJ" rewire)"; rc=$?
+[ "$rc" = 0 ] && printf '%s\n' "$o" | grep -q '^(project draft: bug-logs/2026-01-02-order-flow.md:' \
+  && ok "found from a subfolder of the wired project (no variable set)" || bad "cwd discovery: exit $rc: $(printf '%s\n' "$o" | head -2)"
+o="$(cd "$TOOLKIT" && env -u MXTK_PROJECT_DIR bash "$SUBJ" rewire 2>&1)"; rc=$?
+[ "$rc" = 1 ] && ok "outside any project (the toolkit itself): no drafts, exit 1" || bad "outside a project: exit $rc: $(printf '%s\n' "$o" | head -2)"
+
 echo "== U: usage"
 bash "$SUBJ" >/dev/null 2>&1; [ $? = 1 ] && ok "no query: exit 1" || bad "no query: wrong exit"
 bash "$SUBJ" --full 0 x >/dev/null 2>&1; [ $? = 2 ] && ok "--full 0: exit 2" || bad "--full 0: wrong exit"