Skip to content

Canvas test harness, TypeScript IMSCC validator in CI, real-Canvas fixtures, refreshed CLAUDE.md - #2

Merged
daTechGuy merged 1 commit into
mainfrom
tooling-harness-ci-docs
Oct 3, 2026
Merged

daTechGuy merged 1 commit into
mainfrom
tooling-harness-ci-docs

Conversation

@daTechGuy

Copy link
Copy Markdown
Owner

Makes the Canvas-testing knowledge durable and enforces it in CI. Previously it lived in ad-hoc scripts under the gitignored output/.

What's here

1. Canvas test harness — tools/canvas-test/

  • setup-canvas.sh: reproducible, resumable, idempotent dev-Canvas build pinned to exact Canvas + QTIMigrationTool commits, with the three fixes baked in (publish PORT:80; undeclared @instructure/platform-alerts dependency; QTI tool installed before first boot). fixes / smoke subcommands.
  • cct.sh (+ ruby/): import a cartridge into a throwaway course and dump what Canvas actually built; reference / template build a course in Canvas and export it with Canvas's own exporter; list; cleanup (only touches courses ending (cct)). Local or over SSH.
  • build-demo-cartridge.sh, and a README documenting the workflow and every pitfall.

2. Validator + fixtures in CI

  • src/services/export/validateImscc.ts (TypeScript port of the Python prototype, now removed; npm run validate:imscc). Native-format rules catch the "imports cleanly but drops content" class: marker without module_meta.xml, quiz meta without native QTI, wiki_content/ page missing from the manifest (stray page), dangling references, malformed XML, bare &.
  • tests/fixtures/*.imscc: a reference course and an instructor template written by Canvas's own exporter (scrubbed of instance details). tests/imscc-validator.test.ts and tests/canvas-real-fixtures.test.ts run the validator, parser and template exporter against them — locking in the behaviours that were real bugs. CI already runs the whole suite, so these guard every PR.

3. CLAUDE.md rewritten for this fork (LLM/provider layer, the two export modes and the rules that bit us, verification workflow, merge-commit-only and upstream-sync notes).

Notes

  • .gitignore excepts the fixtures from *.imscc; .gitattributes pins *.sh/*.rb to LF (with core.autocrlf=true they'd break on Windows) and marks *.imscc binary. Scripts are committed executable.
  • Verified locally: tsc -b, eslint (0 errors), vitest --coverage (319 pass, 1 skipped — the harness-only generator), vite build. The harness itself was exercised against the live Canvas (list, import of native + template cartridges, reference/template generation, cleanup) and setup-canvas.sh fixes/smoke were run on the existing checkout (no-op, healthy). A full from-scratch setup-canvas.sh run was not repeated.
  • Not done: relaxing the branch-protection review rule (a protection change was blocked for me; see the conversation).

🤖 Generated with Claude Code

…ures, refreshed CLAUDE.md

The knowledge from testing exports against a live Canvas lived only in ad-hoc
scripts under the gitignored output/ folder. Make it durable and enforce it in CI.

Validator
- src/services/export/validateImscc.ts: offline structural validator, ported from
  the Python prototype (which is removed). Native-Canvas-format rules catch the
  failure that "imports cleanly" but drops content: marker without module_meta.xml,
  quiz meta resource without native QTI, wiki_content page not in the manifest
  (stray page), dangling module/organization references, malformed XML, bare "&".
  Never imported by the app, so not bundled. CLI: npm run validate:imscc.
- tests/imscc-validator.test.ts: passes Canvas's own exports (no false positives)
  and our output; fails each corruption with a specific message.
- tests/canvas-real-fixtures.test.ts + tests/fixtures/*.imscc: our parser and
  template exporter run against a reference course and an instructor template
  written by Canvas's OWN exporter (scrubbed of instance details). Locks in the
  behaviours that were bugs (href-less discussion resources, verbatim external
  links, no stray pages). .gitignore excepts these fixtures from "*.imscc".
  CI already runs the whole suite, so these now guard every PR.

Harness: tools/canvas-test/
- setup-canvas.sh: reproducible, resumable, idempotent Canvas build pinned to exact
  Canvas and QTIMigrationTool commits, with the three fixes baked in (publish
  PORT:80, undeclared @instructure/platform-alerts dependency, QTI tool before
  first boot). `fixes` and `smoke` subcommands; verified idempotent on the live VM.
- cct.sh (+ ruby/): import a cartridge into a throwaway course and dump what Canvas
  actually built; build reference/template courses with Canvas's own exporter;
  list/cleanup (only touches courses ending " (cct)"). Local or over SSH.
- build-demo-cartridge.sh (+ gated tests/build-demo-cartridge.test.ts).
- README documenting the workflow and every pitfall.

.gitattributes pins *.sh / *.rb to LF (core.autocrlf would otherwise break them on
Windows checkouts) and marks *.imscc binary. CLAUDE.md rewritten for this fork:
LLM/provider layer, the two export modes and the rules that bit us, verification
workflow, merge-commit-only and upstream-sync notes. README validator reference fixed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@daTechGuy
daTechGuy merged commit 2016236 into main Oct 3, 2026
1 check passed
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 77.53623% with 31 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/services/export/validateImscc.ts 77.53% 8 Missing and 23 partials ⚠️

📢 Thoughts on this report? Let us know!

@daTechGuy
daTechGuy deleted the tooling-harness-ci-docs branch October 4, 2026 01:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants