From 5b07f93d3a31df1613a63b5f7cddcbfbfec05496 Mon Sep 17 00:00:00 2001 From: smmscout-ops Date: Tue, 13 Oct 2026 08:30:55 +0000 Subject: [PATCH] fix(refresh): run the mirror after the source's weekly run, not an hour before it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `refresh.yml` pulled https://smmscout.com/api/panels.json at Monday 03:00 UTC while the source repo (RelayStack/smmscout, weekly-maintenance.yml) is scheduled for Monday 04:00 UTC and publishes its new payload ~15 min into the run. GitHub fires scheduled runs hours late on this account (source starts observed 2026-10-12T11:29Z and 2026-10-05T11:01Z), so the mirror always pulled the build the source was about to supersede: on 2026-10-13 the dataset carried data_built_at 2026-10-12T01:51:34Z while live was 2026-10-12T11:36:05Z, and 59 of 129 published scores differed. In a week with no out-of-band dispatch the mirror is a full cycle (~7 days) behind the live API. - schedule moves to `0 20 * * 1`: ~8 h past the latest observed source start, with margin for the schedule delay. The comment block in the workflow records why the ordering matters so the slot cannot be moved back by accident. - new step "Report which build was mirrored" prints the mirrored `data_built_at` against the committed copy on every run, and emits a `::warning::` when the pull is not newer than what is already committed — the silent-staleness case that hid this bug for weeks (no diff, green run, nothing to notice). - README refreshing section updated (Mondays 03:00 -> 20:00 UTC, with the reason). The pull URL and the dataset shape are unchanged: the job still consumes the live Function-derived payload (with `operational`), and the python writer is untouched (`indent=1`, same CSV header, same derived files). Verified locally: `node scripts/validate.mjs`, `node scripts/stats.mjs` + `git diff --exit-code -- data/stats.json` and `node scripts/verify-consistency.mjs --report` all pass; the new step was rehearsed against the real live payload (notice) and the committed one (warning). Refs kanban t_23be34f6. --- .github/workflows/refresh.yml | 40 ++++++++++++++++++++++++++++++++++- README.md | 6 +++++- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/.github/workflows/refresh.yml b/.github/workflows/refresh.yml index 426bb43..67c1354 100644 --- a/.github/workflows/refresh.yml +++ b/.github/workflows/refresh.yml @@ -1,8 +1,21 @@ name: refresh +# ORDERING MATTERS. This workflow mirrors the *served* payload +# (https://smmscout.com/api/panels.json), which the source repo +# RelayStack/smmscout republishes from its weekly run +# (.github/workflows/weekly-maintenance.yml, `cron: '0 4 * * 1'`, new data +# ~15 min into the run). GitHub fires scheduled runs hours late on this +# account (source starts observed 2026-10-12T11:29Z and 2026-10-05T11:01Z for +# that 04:00 schedule), so a mirror slot only an hour later always pulled the +# payload the source was about to supersede — on 2026-10-13 this dataset +# carried data_built_at 2026-10-12T01:51:34Z while live was +# 2026-10-12T11:36:05Z, and 59 of 129 published scores differed. The mirror +# therefore has to run *after* the source's Monday refresh, with margin for the +# schedule delay. Keep this slot strictly later than the source's cron. on: schedule: - - cron: '0 3 * * 1' + # Mondays 20:00 UTC — ~8 h after the latest observed source run, not 1 h before it. + - cron: '0 20 * * 1' workflow_dispatch: permissions: @@ -52,6 +65,31 @@ jobs: except Exception as e: print('probe snapshot skipped:', e) PY + # A mirror run that fires before the source refreshed re-publishes the build it + # already had: no diff, "no changes", green — the silent staleness that hid the + # ordering bug for weeks. State which build was mirrored on every run, and flag it + # when the pull is not newer than the committed copy. + - name: Report which build was mirrored + run: | + python3 - <<'PY' + import json, subprocess + live = json.load(open('data/panels.json')) + built = live.get('data_built_at') + prev = subprocess.run(['git', 'show', 'HEAD:data/panels.json'], + capture_output=True, text=True) + was = None + if prev.returncode == 0: + try: + was = json.loads(prev.stdout).get('data_built_at') + except Exception: + was = None + if was and built and built <= was: + print('::warning title=Stale mirror pull::mirrored data_built_at %s is not newer than the ' + 'committed %s - the source weekly run may not have landed yet; check this workflow\'s ' + 'schedule against RelayStack/smmscout weekly-maintenance.yml' % (built, was)) + else: + print('::notice::mirrored data_built_at %s (committed copy was %s)' % (built, was)) + PY - name: Validate and derive stats run: | node scripts/validate.mjs diff --git a/README.md b/README.md index 9915420..4530287 100644 --- a/README.md +++ b/README.md @@ -128,7 +128,11 @@ counts and API probes from a single vantage point, dated on every panel page. ## 🔄 Refreshing -Weekly via `.github/workflows/refresh.yml` (Mondays 03:00 UTC). Manual trigger: +Weekly via `.github/workflows/refresh.yml` (Mondays 20:00 UTC — deliberately *after* +[`smmscout`'s weekly run](https://github.com/RelayStack/smmscout/blob/main/.github/workflows/weekly-maintenance.yml), +which is scheduled for 04:00 UTC and, like every scheduled run on this account, starts +hours late; the later slot means the mirror picks up the build the source just published +instead of the one it is about to supersede). Manual trigger: **Actions → Refresh data → Run workflow**. Every push and pull request is checked by `.github/workflows/verify.yml`, so a