🐛 Keep migration prepare moving past hangs and regenerable gaps

[kirkkwang]

Story

🐛 Time out and retry a hung kubectl exec in migration

e77eab4

On prod, kubectl exec sometimes waits forever after the pod-side script is
gone, stalling a sheet's prepare for hours. Lookup, stage and fill calls
now die after POD_TIMEOUT, retry once, then stop the step cleanly. The
import call keeps no timeout since rerunning it would make a second importer.

Assisted by: Opus 5.5

🎁 Write every unmatched identifier to a file

d769108

The lookup stop named only the first five unmatched identifiers, so a
sheet missing 27 looked like it was missing 5. The full list now goes
to tmp/migration/out/-unmatched.txt.

Assisted by: Opus 5.5

🐛 Continue past failed derivatives the import regenerates

42ffc2a

A PDF or intermediate file set whose thumbnail fails to stage arrives
with no thumbnail, so the import regenerates all its derivatives. Its
failed uploads no longer stop the prepare; any other failure still does.

Assisted by: Opus 5.5

🐛 Fail one derivative download, not the whole batch

af29c07

A dropped S3 connection raised out of the pod's download script, so
Fill lost the batch's 200 results and stopped. Each download now
reports its own failure and the batch carries on.

Assisted by: Opus 5.5

🎁 Resume a prepare from where the last run stopped

3480a90

A rerun rechecked every original and derivative, about an hour on a
92k file set sheet. A progress file now records finished steps and how
many file sets staging and fill got through, keyed by target and input
so a changed sheet or destination starts over. --recheck ignores it.

Assisted by: Opus 5.5

🐛 Cap concurrent kubectl execs into legacy

db36159

Seven prepares at once took legacy's web pods down. Each exec into
legacy now holds one of LEGACY_SLOTS (default 2) file locks shared by
every prepare on the machine, and waits for a free one.

Upload URLs are signed after the slot is taken, so a long wait cannot
expire them.

Assisted by: Opus 5.5

Expected Behavior Before Changes

• A kubectl exec that hung during prepare_sheet (lookup, staging, fill) waited forever; on prod it stalled sheets for 1 to 2 hours.
• A lookup stop named only the first 5 unmatched identifiers, so a sheet missing 27 read as missing 5.
• One failed derivative upload stopped the whole prepare, even when the import would regenerate that derivative.
• A dropped connection while filling derivatives failed the whole batch of 200.
• A rerun rechecked every original and derivative from the start.
• Nothing limited how many prepares ran against legacy; seven at once took its web pods down.

Expected Behavior After Changes

• A hung call is killed after POD_TIMEOUT seconds (default 1800), retried once, then stops the step with a clear message. import_sheet is unchanged.
• Every unmatched identifier is written to tmp/migration/out/<sheet>-unmatched.txt.
• Failed uploads on a PDF or intermediate file set whose thumbnail also failed (and is not already in S3) no longer stop the prepare: that file set arrives with no thumbnail, so the import regenerates all its derivatives. Any other failure still stops it.
• A failed download is reported for that file alone; the rest of the batch fills.
• A rerun skips steps already done and resumes staging and fill after the last clean batch (tmp/migration/out/<sheet>-progress.json); --recheck checks everything again.
• At most LEGACY_SLOTS (default 2) execs run against legacy at once across every prepare on the machine; the rest wait and say so.

Screenshots / Video

None: command-line only

Notes

• cd bin/migration && bundle exec rspec: 57 examples pass. The legacy slot wait, the retry, the non-blocking stdin write, the regeneration filter, the stale-thumbnail guard the per-file download rescue and every progress guard were each mutation-tested.
• Killing kubectl does not stop a script still running in the pod; a slow fill batch retried alongside it fails loudly and a rerun reports the file as present.

🤖 Generated with Claude Code

https://claude.ai/code/session_014ZtPeRZ48cbc5NSvJKCDxT

[github-actions]

Test Results

      4 files  ±    0        4 suites  ±0   51s ⏱️ ±0s
209 097 tests +3 940  209 097 ✅ +3 940  0 💤 ±0  0 ❌ ±0 
  1 365 runs  +    5    1 365 ✅ +    5  0 💤 ±0  0 ❌ ±0 

Results for commit db36159. ± Comparison against base commit a3d5f5b.

♻️ This comment has been updated with latest results.