Why
The 2026-09-29/30 mass-agent push (18 concurrent workers on one develop) exposed how expensive our CI is: every PR re-reads the whole dataset (~190k records / ~292k dump files) no matter how little changed, dump refresh takes ~33 min and re-runs on every rebase, and the automation volume coincided with the TechEngineBot account suspension.
Goal
Make every CI job scale with the size of the change, not the size of the dataset.
Plan
- Unblock – drop the suspended TechEngineBot from
pr-metadata assignees (404s every PR).
- Scope calculator (TechEngine) – changed paths ->
{category, files, full_run_required}; falls back to a full run when code/schema changed.
- Scoped loading –
load_category accepts a file list; brand/soc slug sets for FK checks stay full (small).
- verify-comment –
fetch-depth: 1, read data/_verify/status.json instead of rescoring everything, score once.
- validate – PRs validate changed files (+FK targets); full validation only on develop/main push and nightly.
- dump – per-category and per-path generation, category matrix; move dump refresh off the per-PR critical path (needs a decision on the "dump is the last commit of every data PR" rule).
- deploy-pages – precompute the homepage history timeline so the build no longer needs
fetch-depth: 0.
- Cleanup – stale worktrees/branches (29 worktrees, 111 local branches), stale
dump-refresh/* PRs.
Related: #297 (automation), TechEngine #97-#100.
Why
The 2026-09-29/30 mass-agent push (18 concurrent workers on one
develop) exposed how expensive our CI is: every PR re-reads the whole dataset (~190k records / ~292k dump files) no matter how little changed, dump refresh takes ~33 min and re-runs on every rebase, and the automation volume coincided with the TechEngineBot account suspension.Goal
Make every CI job scale with the size of the change, not the size of the dataset.
Plan
pr-metadataassignees (404s every PR).{category, files, full_run_required}; falls back to a full run when code/schema changed.load_categoryaccepts a file list; brand/soc slug sets for FK checks stay full (small).fetch-depth: 1, readdata/_verify/status.jsoninstead of rescoring everything, score once.fetch-depth: 0.dump-refresh/*PRs.Related: #297 (automation), TechEngine #97-#100.