Pause and resume the metadata refresh when Apple throttles

[kasnder]

Why

A single 403 or 429 from Apple breaks the entire refresh run. That was cheap when the cache was fresh, but production currently has 3,358 of 5,082 app_store_cache rows older than 30 days, so a throttled night could refresh a handful of apps and stop — and the next run starts from the same backlog.

(Separately: refresh_attempted_at is NULL for every row in production, i.e. the Railway cron service has never run. That's a deployment step, not a code change — this PR is about making the run survive throttling once it is scheduled.)

What

• A rate-limited request now pauses and retries the same app instead of ending the run.
• The pause budget (--rate-limit-retries=, default 3, METADATA_REFRESH_RATE_LIMIT_RETRIES) belongs to the run, not to each app — a genuinely throttled client still gives up quickly rather than pausing once per remaining selection.
• The pause doubles per pause taken, starting at --rate-limit-backoff-ms= (default 60s, METADATA_REFRESH_RATE_LIMIT_BACKOFF_MS). Apple's Retry-After overrides it when sent, parsed as seconds or as an HTTP date.
• Every pause is capped at 15 minutes, so a cron service never idles for an hour holding an open database connection.
• Flags forward through pnpm metadata-cron; the result now carries pauses.

Behaviour on giving up is unchanged: the run stops, the interrupted app records no failure of its own (it describes the client, not the app), and the stop reason now also reports the pause count.

Tests

node --test — 154 pass. New coverage: resume-after-pause, run-wide budget with doubling, Retry-After precedence and the cap, and seconds/HTTP-date parsing. The two existing 403/429 tests now pass rateLimitRetries: 0 to assert the give-up path.

🤖 Generated with Claude Code

[kasnder]

Added statement_timeout (0bc1f6b) — but not the whole-run deadline, on reflection.

Why the timeout is safe. statement_timeout is a session parameter: it is sent in the cron connection's startup parameters and applies only to queries on that connection. It cannot reach the web service's pool or the analyser's /uploadAnalysis transactions — those keep the server default. It constrains the cron's own queries and nothing else.

Why the cron cannot block uploads either. The refresher runs each statement as its own implicit transaction — there is no BEGIN around the loop — so its single-row UPDATEs hold row locks for milliseconds. Crucially the rate-limit pause happens between queries, so a 15-minute pause holds no row lock; it holds one idle connection, nothing more. The advisory lock it keeps for the whole run is keyed to the refresh job, so it excludes other refresh runs and nothing else. The tables barely overlap in any case: the jobs write app_store_cache, uploads write apps / app_analyses.

Why no deadline. The run is already bounded on every axis — 15s HTTP timeout per request, a LIMIT-ed selection, a pause budget capped at 3 × 15 min, and now a 30s ceiling on each query. Worst case is roughly limit × (delay + 15s) + 45min: ~78 min at limit=100, ~2h at limit=250. A deadline would add a knob without removing a failure mode.

node --test: 155 pass.