Skip to content

Stop CI from committing generated files and check they are up to date - #38

Merged
apmoore1 merged 1 commit into
UCREL:masterfrom
apmoore1:ci-pipeline-fix
Oct 6, 2026
Merged

apmoore1 merged 1 commit into
UCREL:masterfrom
apmoore1:ci-pipeline-fix

Conversation

@apmoore1

@apmoore1 apmoore1 commented Oct 6, 2026

Copy link
Copy Markdown
Member

CI previously regenerated the TSV files and lexicon_statistics.md and pushed them back to the repository using a token with write access. That exposed the write token to every dependency and script, interpolated github.ref into a shell command, raced between overlapping runs, and cannot work once master is branch protected.

Contributors now generate these files themselves and commit them in their pull request. CI regenerates them and fails if they differ from what is committed.

CI workflow:

  • Use a read-only token and set persist-credentials: false on checkout
  • Pin actions/checkout and actions/setup-python to full commit SHAs
  • Only run on pushes to master and on pull requests
  • Run txt_to_tsv.py and lexicon_statistics.py before test_all_collections.py so the generated TSV files are the ones that get validated
  • Add a check that fails if any generated file is modified or missing

Documentation:

  • CONTRIBUTING.md: add steps to set up the development environment and run the generation commands before opening a pull request, note that a maintainer can run them if the contributor cannot, and fix broken links (../ paths and lexicon_resources.json -> language_resources.json)
  • README.md: add a top-level Development Environment section covering local (venv) and dev container setup, and update the lexicon statistics and Text to TSV sections to say contributors generate the files and CI checks them
  • .gitignore: ignore .venv/

Fixes #37

CI previously regenerated the TSV files and lexicon_statistics.md and
pushed them back to the repository using a token with write access. That
exposed the write token to every dependency and script, interpolated
github.ref into a shell command, raced between overlapping runs, and
cannot work once master is branch protected.

Contributors now generate these files themselves and commit them in their
pull request. CI regenerates them and fails if they differ from what is
committed.

CI workflow:
- Use a read-only token and set persist-credentials: false on checkout
- Pin actions/checkout and actions/setup-python to full commit SHAs
- Only run on pushes to master and on pull requests
- Run txt_to_tsv.py and lexicon_statistics.py before test_all_collections.py
  so the generated TSV files are the ones that get validated
- Add a check that fails if any generated file is modified or missing

Documentation:
- CONTRIBUTING.md: add steps to set up the development environment and
  run the generation commands before opening a pull request, note that a
  maintainer can run them if the contributor cannot, and fix broken links
  (../ paths and lexicon_resources.json -> language_resources.json)
- README.md: add a top-level Development Environment section covering
  local (venv) and dev container setup, and update the lexicon statistics
  and Text to TSV sections to say contributors generate the files and CI
  checks them
- .gitignore: ignore .venv/

Fixes UCREL#37
@apmoore1
apmoore1 merged commit d2b639e into UCREL:master Oct 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harden the CI workflow: read-only token, pinned actions and no auto-commits

1 participant