Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/workflows/compat-prune-monthly.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Compat prune (monthly)

# The organization's monthly [compat] prune: every CT* package audited together, one PR per
# repository with something to drop, one tracking issue here in CTActions. The logic is the
# reusable compat-prune.yml; see docs/WORKFLOWS.md §3.2.

on:
schedule:
- cron: '0 4 1 * *'
workflow_dispatch:
inputs:
dry_run:
description: 'Only audit and report: no PR, no issue'
type: boolean
default: true

concurrency:
group: compat-prune-monthly
cancel-in-progress: false

jobs:
prune:
uses: ./.github/workflows/compat-prune.yml
with:
# Dry run on every trigger until the GitHub App is set up; then the schedule
# publishes: `github.event_name == 'workflow_dispatch' && inputs.dry_run`.
dry_run: ${{ github.event_name != 'workflow_dispatch' || inputs.dry_run }}
# the scripts of the same commit as this workflow
ctactions_ref: ${{ github.sha }}
secrets:
CT_APP_CLIENT_ID: ${{ secrets.CT_APP_CLIENT_ID }}
CT_APP_PRIVATE_KEY: ${{ secrets.CT_APP_PRIVATE_KEY }}
51 changes: 51 additions & 0 deletions .github/workflows/compat-prune-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Compat prune tests

# Self-test of compat-prune, in CTActions itself: the test suite of scripts/compat-prune on
# every audited Julia version (unit, non-regression and integration tests), then a real
# dry-run audit of the ecosystem through the reusable workflow. Runs when the scripts or the
# workflows change; nothing is written anywhere (dry run).

on:
pull_request:
paths:
- 'scripts/compat-prune/**'
- '.github/workflows/compat-prune*.yml'
push:
branches: [main]
paths:
- 'scripts/compat-prune/**'
- '.github/workflows/compat-prune*.yml'
workflow_dispatch:

concurrency:
group: compat-prune-tests-${{ github.ref }}
cancel-in-progress: true

jobs:
tests:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
strategy:
fail-fast: false
matrix:
version: ['1.10', '1.12', '1.13']
steps:
- uses: actions/checkout@v6
- uses: julia-actions/setup-julia@v2
with:
version: ${{ matrix.version }}
- name: Add the General registry
run: julia --startup-file=no -e 'using Pkg; Pkg.Registry.add("General")'
- name: Tests (with integration)
env:
COMPAT_PRUNE_INTEGRATION: '1'
run: julia --startup-file=no scripts/compat-prune/test/runtests.jl

dry-run:
needs: tests
uses: ./.github/workflows/compat-prune.yml
with:
dry_run: true
ctactions_ref: ${{ github.event.pull_request.head.sha || github.sha }}
185 changes: 185 additions & 0 deletions .github/workflows/compat-prune.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
name: Compat prune

# Finds the [compat] specifiers that no supported Julia version ever selects across the
# control-toolbox packages, proves that dropping them leaves every resolved manifest
# unchanged, and (unless dry_run) opens one PR per affected repository plus one tracking
# issue (in CTActions by default), both updated in place by later runs. The prune is the
# organization's: it is called monthly by `compat-prune-monthly.yml`, here in CTActions, and
# by the self-test `compat-prune-tests.yml`; see docs/WORKFLOWS.md §3.2.
#
# The `audit` job is read-only: it resolves the joint, test and docs environments of every
# package on every Julia version (Pkg's resolver only: nothing is downloaded or installed),
# classifies the multi-specifier entries, validates the prunes, applies them to fresh
# checkouts of the default branches, and uploads the result (`compat-prune` artifact,
# report in the job summary). Scripts and tests: scripts/compat-prune/.
#
# The `publish` job (skipped on dry runs) is the only one that writes, with a token of the
# organization's GitHub App (secrets CT_APP_CLIENT_ID, CT_APP_PRIVATE_KEY; permissions:
# contents, pull requests, issues). It re-applies the prunes to fresh checkouts, then keeps
# one branch `ct/compat-prune` and one PR per repository, and one tracking issue in
# `tracking_repo`: a rerun updates them in place, and closes what has nothing left.

on:
workflow_call:
inputs:
packages:
description: 'JSON list of the audited packages (registered names)'
required: false
type: string
default: '["CTBase","CTParser","CTModels","CTLie","CTFlows","CTDirect","CTSolvers","OptimalControl"]'
julia_versions:
description: 'JSON list of the Julia versions (juliaup channels); the last one is the newest'
required: false
type: string
default: '["1.10","1.12","1.13"]'
dry_run:
description: 'Only audit and report: no PR, no issue'
required: false
type: boolean
default: true
ctactions_ref:
description: 'Ref of CTActions the scripts are checked out from'
required: false
type: string
default: main
tracking_repo:
description: 'Repository of the tracking issue'
required: false
type: string
default: control-toolbox/CTActions
secrets:
CT_APP_CLIENT_ID:
description: 'Client ID of the GitHub App that publishes (not needed for a dry run)'
required: false
CT_APP_PRIVATE_KEY:
description: 'Private key of that GitHub App'
required: false

jobs:
audit:
runs-on: ubuntu-latest
timeout-minutes: 120
permissions:
contents: read
env:
PACKAGES: ${{ inputs.packages }}
JULIA_VERSIONS: ${{ inputs.julia_versions }}
SCRIPTS: ctactions/scripts/compat-prune
OUT: ${{ github.workspace }}/compat-prune
GH_TOKEN: ${{ github.token }}
steps:
- uses: actions/checkout@v6
with:
repository: control-toolbox/CTActions
ref: ${{ inputs.ctactions_ref }}
path: ctactions

- uses: julia-actions/install-juliaup@v3
with:
channel: release

- name: Install the audited Julia versions
run: |
for v in $(jq -r '.[]' <<< "$JULIA_VERSIONS"); do juliaup add "$v"; done
juliaup status

# ct-registry is public: added over HTTPS, no SSH key needed. The registries are
# updated once more by audit.jl, then frozen for the whole audit.
- name: Add the registries
run: |
julia --startup-file=no -e 'using Pkg
Pkg.Registry.add("General")
Pkg.Registry.add(url="https://github.com/control-toolbox/ct-registry.git")'

- name: Test the scripts
run: julia --startup-file=no "$SCRIPTS/test/runtests.jl"

- name: Audit
run: julia --startup-file=no "$SCRIPTS/audit.jl" "$PACKAGES" "$JULIA_VERSIONS" "$OUT"

- name: Apply the prunes to fresh checkouts of the default branches
run: |
"$SCRIPTS/checkout.sh" "$OUT" checkouts
julia --startup-file=no "$SCRIPTS/edit.jl" "$OUT" checkouts
for d in checkouts/*/; do
git -C "$d" diff --src-prefix="a/$(basename "$d")/" --dst-prefix="b/$(basename "$d")/"
done > "$OUT/edits.diff"

- name: Report
run: |
RUN_URL="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \
julia --startup-file=no "$SCRIPTS/report.jl" "$OUT"
{
cat "$OUT/report.md"
echo
echo "<details><summary>Edits on the default branches (diff)</summary>"
echo
echo '```diff'
cat "$OUT/edits.diff"
echo '```'
echo "</details>"
} >> "$GITHUB_STEP_SUMMARY"

- uses: actions/upload-artifact@v7
with:
name: compat-prune
path: compat-prune/
retention-days: 30

publish:
needs: audit
if: ${{ !inputs.dry_run }}
runs-on: ubuntu-latest
timeout-minutes: 30
# Two publications at once would race on the same branches and issue.
concurrency:
group: compat-prune-publish
cancel-in-progress: false
permissions:
contents: read
env:
SCRIPTS: ctactions/scripts/compat-prune
OUT: ${{ github.workspace }}/compat-prune
steps:
- name: Check the App credentials
env:
CLIENT_ID: ${{ secrets.CT_APP_CLIENT_ID }}
PRIVATE_KEY: ${{ secrets.CT_APP_PRIVATE_KEY }}
run: |
if [ -z "$CLIENT_ID" ] || [ -z "$PRIVATE_KEY" ]; then
echo "::error::Publishing needs the secrets CT_APP_CLIENT_ID and CT_APP_PRIVATE_KEY (or run with dry_run: true)."
exit 1
fi

- uses: actions/checkout@v6
with:
repository: control-toolbox/CTActions
ref: ${{ inputs.ctactions_ref }}
path: ctactions

- uses: actions/download-artifact@v8
with:
name: compat-prune
path: compat-prune

- uses: julia-actions/install-juliaup@v3
with:
channel: release

- uses: actions/create-github-app-token@v3
id: app
with:
client-id: ${{ secrets.CT_APP_CLIENT_ID }}
private-key: ${{ secrets.CT_APP_PRIVATE_KEY }}
owner: control-toolbox

- name: Publish
env:
GH_TOKEN: ${{ steps.app.outputs.token }}
TRACKING_REPO: ${{ inputs.tracking_repo }}
run: |
gh auth setup-git
"$SCRIPTS/checkout.sh" "$OUT" checkouts
julia --startup-file=no "$SCRIPTS/edit.jl" "$OUT" checkouts
RUN_URL="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \
julia --startup-file=no "$SCRIPTS/publish.jl" "$OUT" checkouts "$TRACKING_REPO" | tee -a "$GITHUB_STEP_SUMMARY"
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ repository and for org-wide updates — is described in
|---|---|---|
| `occidata-runner-maintenance.yml` | Weekly (Mon 02:30 UTC) | Purge Julia cache, update TeXLive, rotate logs on self-hosted runner |
| `occidata-runner-watchdog.yml` | Daily (03:15 UTC) | Restart, through SSH, the Occidata runner jobs (one per organization) that are neither pending nor running |
| `compat-prune-monthly.yml` | Monthly (1st, 04:00 UTC) | Drop the `[compat]` specifiers no supported Julia version selects, across all CT\* packages: one PR per repository, one tracking issue here (logic: reusable `compat-prune.yml`, scripts and tests in `scripts/compat-prune/`) |

The Occidata watchdog requires these CTActions repository or organization secrets:

Expand Down
86 changes: 83 additions & 3 deletions docs/WORKFLOWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,15 +336,18 @@ jobs:

### 3.2 Maintenance workflows that live *in* `CTActions` (not called)

These run on `CTActions` itself, on a schedule, to keep the **self-hosted runners**
healthy. They are not `workflow_call` reusables.
These run on `CTActions` itself: on a schedule, to keep the **self-hosted runners**
healthy or to act on the whole organization, or on its own PRs, to test the scripts behind
a workflow. No package calls them.

| Workflow (`CTActions`) | Role | Trigger |
| --- | --- | --- |
| `occidata-runner-maintenance.yml` | Purge Julia compile cache, update TeXLive, rotate logs on the `occidata` self-hosted runner | weekly cron, `workflow_dispatch` |
| `occidata-runner-watchdog.yml` | Over SSH, run `scripts/start_runners.sh` (control-toolbox/occidata), which restarts every Occidata runner instance whose SLURM job is neither pending nor running | daily cron, `workflow_dispatch` |
| `compat-prune-monthly.yml` | The organization's `[compat]` prune: runs the reusable `compat-prune.yml` on every CT\* package; one PR per repository with something to drop, one tracking issue in CTActions | monthly cron (1st, 04:00 UTC), `workflow_dispatch` (`dry_run`) |
| `compat-prune-tests.yml` | Self-test of `compat-prune`: the test suite of `scripts/compat-prune` on every audited Julia version, then a real dry-run audit through `compat-prune.yml` | PR or push touching `scripts/compat-prune/` or `compat-prune*.yml`, `workflow_dispatch` |

`occidata` started as the target of this maintenance workflow only; since CTFlows.jl's
`occidata` started as the target of the maintenance workflow only; since CTFlows.jl's
`occidata-runner` job (§3.1, added 2026-08-24) it is also a live GPU CI target for every
package's test matrix that needs one, and (2026-09-02, control-toolbox/OptimalControl.jl#885)
a documentation-build target too — not just a runner kept healthy for its own sake. Two
Expand Down Expand Up @@ -381,6 +384,82 @@ organization. It runs in `cpu` mode: no `$OCCIDATA_GPU_RUN`, so no GPU ever. Ins
their modes and how to add one are documented in control-toolbox/occidata,
`docs/gha-runner-config.md` § 9.

**`[compat]` entries are pruned for the whole organization, once a month.**
`compat-helper.yml` runs with `KeepEntry()`: a new major is added next to the old ones
(`JET = "0.9, 0.11, 0.12"`), and
`ci.yml` tests the newest on CompatHelper PRs (`force_latest_compatible_version=auto`,
except on Julia 1.10, CTActions#93). Old specifiers therefore pile up, and none can be
dropped from inside one package: `JET 0.9` is still what Julia 1.10 selects, and
CTSolvers' `CUDSS 0.7` is what OptimalControl's tests select. The prune is therefore the
organization's, decided for all CT\* packages at once (control-toolbox/OptimalControl.jl#986),
and lives here: `compat-prune-monthly.yml` runs the reusable `compat-prune.yml` (also run
by the self-test) on the 1st of every month, and keeps its tracking issue in CTActions
(input `tracking_repo`). No package carries a caller.

- **What is audited.** The version of each package that the joint resolution of all of
them selects, read from its registered tree (not from a branch), and 17 environments: the
joint one, plus the test environment (`test/Project.toml` or `[targets] test`) and the
docs environment of every package, each on every Julia version of `julia_versions`.
Resolution calls Pkg's resolver only: nothing is downloaded or installed, and one
registry snapshot serves the whole audit (about 4 minutes on a runner).
- **What is pruned.** In an entry with several specifiers, those older than the newest one
in use, that no environment selects on any Julia version, counting only the environments
where the entry applies (where its package is at its audited version, or the very `test/`
or `docs/` environment of the file). Every prune is then simulated, by intersecting the
compat of the environments where it applies, and kept only if all manifests stay
identical. A prune therefore changes nothing that is resolved: the PRs can be merged in
any order and need no release.
- **What is reported.** The tracking issue (hidden marker `<!-- ct-compat-prune -->`)
opens with the record of the audit (environments resolved per Julia version, entries
examined by status, registry snapshot, and, folded, the entries kept or idle with why),
then lists the prunes per repository with their PR, those merged but not yet released, the
*drifts* (the default branch no longer has the audited line: untouched), the prunes
rejected by the simulation, the *withheld* ones (an environment failed to resolve: nothing
is pruned that run), and the *blocked* entries (the newest specifier is never selected,
with the packages that cap it: information only).
- **Idempotence.** One branch `ct/compat-prune` and one PR per repository, one issue: a
rerun rewrites them in place (the branch is pushed only when its content changes), closes
the PR and deletes the branch of a repository with nothing left, and closes the issue
(reopened, never recreated) when nothing is left at all. The edits are re-applied to the
current default branch, one `[compat]` line at a time, never copied from the audit.
- **Dry run.** `dry_run: true` (the default) runs the read-only `audit` job only: report
and diff in the job summary, everything in the `compat-prune` artifact. The `publish`
job writes with a token of the organization's GitHub App (`CT_APP_CLIENT_ID`,
`CT_APP_PRIVATE_KEY`; permissions: contents, pull requests, issues, on the CT\*
repositories) and is the only one that does.

`ct-registry` is added over HTTPS (it is public), so no `SSH_KEY` is needed. The scripts
live in `scripts/compat-prune/` and run locally the same way (juliaup channels):

```bash
julia scripts/compat-prune/audit.jl '["CTBase","CTParser","CTModels","CTLie","CTFlows","CTDirect","CTSolvers","OptimalControl"]' '["1.10","1.12","1.13"]' /tmp/audit
COMPAT_PRUNE_INTEGRATION=1 julia scripts/compat-prune/test/runtests.jl # unit, non-regression, integration
```

The monthly workflow:

```yaml
# .github/workflows/compat-prune-monthly.yml (CTActions)
on:
schedule:
- cron: '0 4 1 * *'
workflow_dispatch:
inputs:
dry_run: { type: boolean, default: true }
jobs:
prune:
uses: ./.github/workflows/compat-prune.yml
with:
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
ctactions_ref: ${{ github.sha }}
secrets:
CT_APP_CLIENT_ID: ${{ secrets.CT_APP_CLIENT_ID }}
CT_APP_PRIVATE_KEY: ${{ secrets.CT_APP_PRIVATE_KEY }}
```

Until the GitHub App is set up, every trigger is a dry run
(`dry_run: ${{ github.event_name != 'workflow_dispatch' || inputs.dry_run }}`).

### 3.3 Non-centralized / per-repo special workflows

| Workflow | Where | Why it isn't centralized |
Expand Down Expand Up @@ -495,6 +574,7 @@ runner is the one maintained by the scheduled `CTActions` maintenance workflows
| `DOCUMENTER_KEY` | Documentation, CompatHelper, TagBot | Deploy docs / sign tags (Documenter deploy key) |
| `CODECOV_TOKEN` → `codecov-secret` | Coverage | Upload coverage to Codecov |
| `PROJECT_TOKEN` → `project_token` | AddToProject | Write to the org project board |
| `CT_APP_CLIENT_ID`, `CT_APP_PRIVATE_KEY` | `compat-prune-monthly.yml` in CTActions (`publish` job) | Token of the organization's GitHub App: push `ct/compat-prune` and open PRs in the CT\* repositories, keep the tracking issue |
| `GITHUB_TOKEN` | (built-in) most | Default repo-scoped token |

Use `secrets: inherit` (as `UpdateReadme` does) only when the reusable workflow needs
Expand Down
Loading
Loading