[hahn-kev-bot]

better failure state handling, based on a recent incident which took down the server

---

🤖 AI summary

Incident: A single "Low Bandwidth" (HgResume v03 / resumable) client DDoS'd prod by looping pullBundleChunk against an empty repo (sena-3) with stale cached base hashes. The server's isValidBase spun on the empty repo and OOM-killed the container; the client retried every FAIL(400) with no backoff. This closes the server side of that incident so every degenerate or malicious request terminates with a correct, non-retryable-forever response.

All changes are in csharp/src/HgResume.Api (the C# rewrite).

A — empty/missing baseHashes (the live bug). baseHashes is now optional in RestDispatcher (a cache-cleared client re-sends pullBundleChunk with none; requiring it produced a retried FAIL(400)), and HgRunner.MakeBundle treats an empty list as --all. Result: empty repo → NOCHANGE, non-empty repo → full clone. Both terminate.

B — isValidBase → direct per-hash lookup. Replaced the whole-history paging scan with a per-hash hg log -r <hash> existence check (O(k)), hex-validated (^[0-9a-fA-F]{1,40}$) and passed positionally (no revset/shell injection). Independently fixes the empty-repo spin.

C — hg reliability. ProcessRunner now returns exit code and stderr; getRevisions/getBranchTips/isValidBase key on the exit code and fail fast with the stderr on a real hg error, instead of inferring failure from empty stdout (the old uninformative command 'hg log' failed!). The empty-repo -1→0 sentinel is preserved.

D — bounded getRevisions. Each hg log is bounded to O(offset+quantity) via -l, preserving newest-first ordering and offset semantics.

E — concurrent-generation guard + crash recovery. Before MakeBundle, an in-flight generation is detected via the in-process Running registry (and lock-file age) and polled rather than duplicated; a stale .async_run left by a crashed/restarted process is reaped and respawned; respawns are bounded by a genAttempts counter → FAIL.

F — cache GC. New CacheGarbageCollector BackgroundService reaps .bundle/.metadata/.async_run/.incoming older than ~24h (prod leaked 1,258 .async_run locks).

G — chunkSize clamp to a config cap (HGRESUME_CHUNK_SIZE_MAX, default 20 MB).

All new behaviour is env-overridable via ApiConfig with safe defaults.

Test plan

• HgResume.Api.Tests (unit): 17/17 — includes new CacheGarbageCollectorTests (reaps stale artifacts; leaves fresh/unrelated files).
• HgResume.IntegrationTests (Testcontainers, real image):
  • PullFacts 19/19 — includes new regressions: empty-repo first-sync with no baseHashes → NOCHANGE then push succeeds; non-empty repo with no baseHashes → servable full-clone bundle; plus the existing empty-repo-loop and missing-base-across-pages regressions.
  • PushFacts + MiscFacts + ContractFacts 31/31.
  • SendReceiveTests + VerifyHgWorking (end-to-end Chorus client) 4/4.

---

This change is 

[rmunn]

One style nit that doesn't need to be addressed but I wanted to mention, and a few minor changes that should be addressed, like changing the lower bound on the regex repeat count. Then after addressing those minor changes, no need for a re-review; it LGTM and can be merged once those changes are addressed. (Which is why I've approved it now rather than using "Request changes", in case I'm unavailable after the changes are made).