[andrewdefilippis]

Problem / background

This repository turns the hand-maintained PAX/RTP pages at solotime.info/pax into machine-readable files. A review against the live source found that the data and tooling had drifted:

• Stale data. solotime.info revised the 2024 index on 2023-12-20, ten days after it was scraped. XS/SSR were removed, XU 0.869 was added, EVX went 0.830 → 0.834 and XA 0.842 → 0.844. The repo never picked this up. The same thing happened to 2026: master added it on 2026-01-04, and the page was then revised on 2026-03-20 (CAM-C 0.827 → 0.828, GS 0.793 → 0.809, SS 0.837 → 0.840, XU 0.869 → 0.872). This branch merges master and supersedes those values.
• Lossy scraping. Labels like ST (STS) (2009) and AST(STR) (2025) mark renames and redistributions. The scraper either dropped the annotation, which spliced unrelated classes into one series (STS before and after 2009 are different classes), or kept it in the key (AST(STR)). All errors were swallowed.
• Non-idempotent combined file. rtp_files_generator.py merged with an append strategy, so every re-run duplicated a year and a revised page could not replace one.
• Wrong categories. Regex-derived categories put XP in Xtreme Street and XS in Street, and left 31 classes blank.
• Other issues. YAML year keys were integers while JSON's were strings, rtp_form_generator.py did not run, and nothing was automated.

What is changing

1. Canonical CSV (CSV/):
   • rtp.csv: each index exactly as published, plus its verbatim label.
   • sources.csv: URL, the page's "Last update" date, fetch time and SHA-256 for each year.
   • classes.csv: SCCA name and category per year, with a citation and verified/unverified status.
   • lineage.csv: renamed and derived classes.
2. Typed, strict tooling.
   • Scripts/rtp_model.py parses CSVs into domain types (Parse, don't validate).
   • The scraper fails loudly on anything unexpected, honors the site's 10 s crawl delay, and replaces whole years on revision.
   • The generator rebuilds all outputs from CSV, so it is idempotent.
3. Published files.
   • JSON/<year>.json keeps its shape.
   • New JSON/latest.json / YAML/latest.yaml.
   • YAML year keys are now strings.
   • rtp.json v2 maps each class's years to Index, Name, SoloCategory, ClassificationVerified, plus Predecessors/Successors and per-year source metadata. JSON Schemas are in Schema/.
4. Automation.
   • ci.yml runs the tests and checks that generated files match the CSV.
   • update-rtp.yml runs weekly and opens a PR on any source change. PRs that need a new class classified open as drafts labeled needs-classification.
5. uv. pyproject.toml + hash-locked uv.lock (one-week exclude-newer, no-build) replace requirements.txt. Dependabot covers actions and uv.
6. Docs. README data contract and maintainer notes, plus a CHANGELOG listing the breaking changes.

Why

• Clubs' scoring software needs data that is correct, current and stable to consume.
• The source is edited by hand and revised mid-season, so revisions have to be detected automatically.
• Provenance has to be recorded.
• Class identity has to be explicit, because SCCA reuses codes:
  • 2014: every Stock class became Street-R for one year, while AS–HS were reused for the new Street category.
  • 2025: Street Touring was redistributed. For example, most of STR went to CST, while the ND MX-5 went to AST.

Expected outcome

• Every year 1995–2026 matches solotime.info exactly. This was checked with two independent parsers against all 32 pages.
• Running the generator twice produces no diff, and CI enforces this.
• 1,092 of 1,311 class-years (83%) have SCCA name and category verified against a source. The rest are marked unverified with a reason, and 10 early class-years that no source names are null.
• A source revision shows up as a weekly PR with a diff summary. A layout change fails the job instead of committing bad data.

Breaking changes for consumers (see CHANGELOG.md):

• rtp.json/rtp.yaml have a new shape.
• The 2025 keys AST(STR) etc. become AST etc.
• YAML year keys are strings.

Actions / tests

• uv run pytest: 183 tests covering:
  • the parser (every known label and value quirk, using hand-written fixtures);
  • the model, including CSV injection, Unicode and URL validation;
  • the fetcher (crawl delay, retries, redirects, size and time limits);
  • main() modes, including year rollover;
  • the generator (schemas, idempotency, consistency of the real data);
  • the PR-job patch guard, run against real git with 20+ bypass attempts.
• The CI steps were simulated in a fresh checkout with uv sync --locked.
• Full live re-fetch of 1995–2026. The diff against the old JSON was exactly the expected set listed above.
• actionlint is clean.
• Security audit (security-auditor), repeated until nothing Medium or above remained. The resolutions include:
  • splitting the update workflow by privilege;
  • hash-locked dependencies and a checksum-pinned uv binary;
  • a stdlib-only, index-based patch guard;
  • redirect, size and time limits;
  • CSV formula and markup rejection;
  • sanitizing the PR body.
• Staff-level code review. It approved with comments, and all comments were addressed or documented.

Owner follow-ups after merge (repository settings):

• Settings → Actions → General: enable "Allow GitHub Actions to create and approve pull requests".
• Protect master and require the CI / test check, because bot PRs opened with GITHUB_TOKEN don't trigger CI on their own.
• Run "Update RTP data" once with workflow_dispatch to confirm the end-to-end flow.

References

• Source: https://www.solotime.info/pax/ (archives at rtp<year>.html). robots.txt sets Crawl-delay: 10.
• SCCA National Solo Rules: scca.com/downloads (2024–2025), 2023, 2022. Older editions are cited per row in CSV/classes.csv.
• SCCA, Oct 11, 2013: Street category proposal approved (Stock → Street / Street-R).
• Tennessee Region SCCA: 2025 Street Touring restructure.
• uv: dependency cooldowns / exclude-newer, uv + Dependabot.

🤖 Generated with Claude Code