populate: strip hard line breaks by default; text_unwrap joins _utils (#6)

[williamdemeo]

Closes #6.

What changes

• _utils/text_unwrap.py — a pure, total, idempotent reflow of markdown prose onto single lines (two spaces after sentence ends, one otherwise), preserving byte-for-byte everything that is not reflowable prose: headings, rules/setext underlines, tables, blockquotes, fenced code (mermaid/agda), indented code, <!--/-->/generated-region marker lines, and the plan grammar's line-oriented **Labels:** / **Milestone:** / **Repository**: metadata. No I/O, no exceptions — callers use file_ops.
• populate strips by default: the input is unwrapped in memory before parsing, so pushed issue bodies and milestone descriptions reach GitHub without authored hard breaks and the issue page soft-wraps them. --keep-line-breaks pushes verbatim. The run output states the mode either way. The plan file on disk is never rewritten by the transform — the (#N) write-back still edits the authored text, and the post-populate update run is what lands normalized content in the file.
• Normalize-after-populate documented and hinted (adopting the field report from another session): populate now ends real creation runs with the three-step hint — commit the write-backs; run one plain update and commit it as normalization; then update --check is the drift gate — because the first --check on a fresh plan is always stale (update drops **Milestone:** lines, reorders labels to GitHub's order, trims trailing ---). README and both committed skills say the same.

Acceptance property (the review's suggested test, adopted)

Pinned in test_text_unwrap.py: a plan file parses identically before and after unwrapping — same milestones, labels (names and colors), issue ids, titles, and label sets — and lint_plan reports zero errors on the unwrapped text; only bodies and descriptions reflow. Checked against the shared fixture and the shipped worked example (mermaid fence intact).

Deliberately out of scope

The create-only gap the field report flagged — reflowed bodies can't reach issues that already exist on GitHub — is filed as #7 (--sync-bodies) with its design questions, rather than grown into this PR.

Tests

Suite 134 → 143: reflow semantics (9 golden cases + idempotence + degenerate inputs), the parse-identity acceptance property, and three fake-gh integration tests (default strips bodies and milestone descriptions with the mode line printed; --keep-line-breaks pushes verbatim; the next-steps hint appears only after real creations). nix flake check green; ruff/shellcheck clean. Engine VERSION → 0.4.0.

AI-assisted development: Claude Fable 5 (Anthropic)

[Copilot]

Pull request overview

Adds default Markdown prose unwrapping before populate pushes content to GitHub, with opt-out support and normalization guidance.

Changes:

• Adds structure-aware prose unwrapping and tests.
• Integrates --keep-line-breaks into populate.
• Documents the post-populate normalization workflow.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/VERSION	Bumps engine version to 0.4.0.
scripts/_utils/text_unwrap.py	Implements Markdown prose unwrapping.
scripts/gh_project_populate.py	Applies unwrapping and prints workflow guidance.
scripts/tests/test_text_unwrap.py	Tests reflow and parsing invariants.
scripts/tests/test_integration.py	Tests populate line-break modes and hints.
README.md	Documents unwrapping and normalization.
.claude/skills/roadmap-upkeep/SKILL.md	Adds normalization guidance.
.claude/skills/project-setup/SKILL.md	Updates project setup instructions.

Suppressed comments (3)

scripts/_utils/text_unwrap.py:47

• Bold metadata is only treated as a boundary when another paragraph is already open. If valid input omits the optional blank line after **Repository**: or **Description:**, the following prose is joined onto that metadata line; the repository regex then stops matching, and milestone descriptions can disappear. Treat these line-oriented metadata records as structural in every position so unwrapping cannot change parsing.

BOLD_META_RE = re.compile(r"^\s{0,3}\*\*[^*\n]+(?:\*\*\s*:|:\*\*)")

scripts/_utils/text_unwrap.py:58

• Only table rows beginning with | are protected. GitHub-flavored Markdown also permits tables without leading pipes (Name | Value followed by --- | ---); this implementation joins all such rows into one prose line and destroys the table. Table detection needs to preserve the header, delimiter, and data rows for both valid forms.

        or stripped.startswith(("|", ">", "<!--", "-->"))

scripts/_utils/text_unwrap.py:118

• Markdown permits a tab to supply the indentation for an indented code block. Tab-indented code currently falls through to paragraph reflow, which strips its indentation and joins its lines, despite the transform's code-preservation contract.

        if line.startswith("    "):
            out.append(line)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[williamdemeo]

Confirmed, all four locations — and the underlying pattern was under-recognition of structure generally, so I also swept the review's three suppressed findings, which were legitimate instances of the same family. Fixed in 6270d59: (1) this comment — setext underlines now match any =/- run (one character suffices in CommonMark) and thematic breaks match 3+ of the same marker with spaces/tabs between (- - -, * * *); (2) bold plan metadata (**Description:**, **Repository**:, …) is structural in every position, not only when a paragraph is already open — the blankless-**Description:** shape would otherwise have its value joined on and destroyed; (3) pipe-less GFM tables are protected by treating any pipe-bearing line as structural (a false positive costs only an unremoved break, never a changed rendering); (4) tab-indented code blocks are preserved. Six new golden cases plus a parse-identity test over the blankless metadata shapes pin all of it; suite 143 → 144, each case idempotence-checked.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (7)

scripts/_utils/text_unwrap.py:42

• Bare ATX headings are valid Markdown (# through ###### may end immediately), but this pattern requires trailing whitespace. unwrap("#\nbody\n") therefore produces "# body\n", turning an empty heading plus paragraph into a different heading. Accept end-of-line as well, and add this shape to the golden cases.

HEADING_RE = re.compile(r"^\s{0,3}#{1,6}\s")

scripts/gh_project_populate.py:536

• The printed backtick commands are not executable names in this project: the supported interface is make update / make update-check (Makefile:81-85), while the installed binary is gh-project-update. Because this is the post-run action users will copy, print the canonical Make targets used in README.md:99-105 instead of update and update --check.

            print("  2. run a plain `update` once and commit the result — it")
            print("     normalizes the generated regions to the engine's")
            print("     canonical rendering (the first `update --check` on a")
            print("     fresh plan is ALWAYS stale otherwise)")
            print("  3. from then on, `update --check` is the drift gate")

scripts/_utils/text_unwrap.py:41

• Empty list items are not recognized because this requires a non-whitespace character after the marker. For example, *\nfollowing paragraph\n is joined into * following paragraph, moving the paragraph into the list item and changing the Markdown structure. Empty *, +, and ordered markers need to be treated as standalone structure rather than as paragraphs with joinable continuations.

This issue also appears on line 42 of the same file.

LIST_ITEM_RE = re.compile(r"^\s*(?:[-*+]|\d{1,9}[.)])\s+\S")

scripts/_utils/text_unwrap.py:137

• This assumption fails for a list item's second paragraph. In - item\n\n continued\n prose\n, the four-space lines are prose inside the list (list-relative code would require additional indentation), but after the blank line paragraph is empty and both lines are preserved as “code.” Thus the promised list-item reflow leaves these hard breaks intact. Track list context/indentation across blank lines before classifying every fresh four-space line as code.

        # Fresh block position: four-space- or tab-indented lines are
        # code (list continuations never reach here — their paragraph
        # is still open when they arrive).
        if line.startswith(("    ", "\t")):
            out.append(line)

scripts/gh_project_populate.py:523

• This also fires after a partial failed run because issues_created is incremented before later issue failures and before a write-back failure. The hint then tells the user to normalize from incomplete GitHub state; update can overwrite generated regions and drop issue definitions that were not created, and “it now records the issue numbers” is false when persistence failed. Emit these next steps only after a fully successful run; failed runs should retain the existing rerun guidance.

This issue also appears on line 532 of the same file.

        if issues_created:

.claude/skills/roadmap-upkeep/SKILL.md:31

• update --check is not a command exposed by this repository. The same skill identifies the actual target as make update-check at line 19, and Makefile:84 maps that target to the engine's --check option. Use make update-check here so the normalization guidance is actionable.

A freshly populated plan's FIRST `update --check` is always stale —

.claude/skills/project-setup/SKILL.md:52

• update --check is not an available command; this workflow otherwise consistently uses the Make targets, including make update four lines above and make update-check in the roadmap-upkeep skill. Name make update-check here to avoid sending users to a nonexistent executable.

   `update --check` on a freshly populated plan is ALWAYS stale until

[williamdemeo]

Response to the final review's seven suppressed findings (d38fbf4)

Acted on six — all legitimate:

1. Hint after partial runs (populate:523) — the most consequential: issues_created counts successes before later failures, so a partially failed run got advice that is wrong twice over (the file does not yet record every number, and normalizing via update from incomplete GitHub state would drop not-yet-created issue definitions from the plan). The hint now requires a fully successful issues stage (created > 0, failed = 0); failed runs keep the per-issue rerun guidance. Pinned by a fault-injection test.
2. Non-runnable commands in the hint (populate:536) — it now prints the canonical make update / make update-check.
   3–4. Skills said update --check — both now name make update-check.
3. Bare ATX headings (# alone is a valid empty heading) no longer absorb the following paragraph.
4. Bare list markers (*, +, 1. — valid empty items) no longer absorb the following paragraph.

Declined one, by design — the list item's second paragraph (four-space prose after a blank line inside an item). Distinguishing it from an indented code block requires tracking list context and indentation across blank lines — a substantial slice of a CommonMark block parser — for a case whose current behavior errs in the safe direction: an unremoved line break, never a changed rendering. The limitation is now documented in the module docstring and pinned as deliberate by a golden case.

Suite 144 → 145; every new case idempotence-checked.