M-CHECKPOINT CHK.4: Frequency displacement checkpoint/resume

[jonathanschultzNU]

Summary

Follow-up to #118 (merged) on the same branch — the previous PR's history
is preserved via rebase onto the post-merge main; this PR only contains
the two new commits below.

Adds concurrency-safe, per-displacement checkpoint/resume for Frequency's
finite-difference IR and Raman loops (M-CHECKPOINT CHK.4, the one item
left open from the Lab 2 findings in #118). Full design writeup lives in
the (private) planning repo's TODO/roadmaps/34-m-checkpoint-calc-restart-roadmap.md.

Geometry-Opt and PES-Scan (#118) both have a natural sequence point to
checkpoint at — an optimizer step, a scan point. The Hessian's 6N
finite-difference displacement loop doesn't, and freq_ir_workers.py/
freq_raman_workers.py can run displacements concurrently via the
opt-in QUANTUI_FREQ_PARALLEL worker pool, so resume has to diff a set
of completed displacement ids, not trim a prefix.

• quantui/checkpoint.py — set-oriented completion API
  (mark_item_done/completed_item_ids/completed_items) alongside the
  existing index-based completed_points(), plus path-only siblings
  (mark_item_done_at/completed_items_at) for a ProcessPoolExecutor
  worker that only has a directory path, not a picklable Checkpoint
  object.
• quantui/freq_displacement_ids.py (new) — deterministic
  (atom_index, axis, sign) → id scheme, the single source of truth
  shared by every path below.
• quantui/freq_calc.py — run_freq_calc() takes checkpoint/
  resume (same convention as Geometry-Opt). Serial and parallel paths
  are unified around one shared dict keyed by displacement id, so resume,
  the parallel pool, and the serial fallback can never disagree about
  what's already done — and the Hessian assembly step only runs once
  every required id is present, regardless of which path produced it.
• quantui/freq_ir_workers.py / quantui/freq_raman_workers.py —
  each worker durably records its own completion (temp-file +
  os.replace) before returning to the parent, so a completed
  displacement survives even if the parent process dies before collecting
  that Future.
• quantui/raman_calc.py — identical treatment under a separate named
  item set (raman_displacements), so its records can't collide with
  IR's (freq_displacements) despite sharing the exact same displacement
  grid.
• quantui/backends/worker.py — _run_frequency now calls
  _begin_worker_checkpoint() and threads checkpoint/resume through, the
  same pattern _run_geometry_opt/_run_pes_scan already used. This is
  the actual SLURM batch path the motivating production case hit.

One pre-existing bug fixed in passing: a parallel-pool failure partway
through used to discard every already-completed displacement and
recompute all of them serially. The shared-dict unification fixes this
for free — the serial fallback now skips anything already done.

Testing

pytest -m "not network": full suite green, 3237 passed / 30 skipped. 37
new tests (test_freq_calc.py, test_raman_calc.py,
test_backends_worker.py) — the load-bearing ones assert run_scf_with_rescue
call counts, not just a plausible-looking answer: a fully-banked resume
does zero displacement recompute, a partial resume recomputes exactly the
missing ids, a corrupted banked item is recomputed rather than trusted,
and resume=False correctly ignores a populated checkpoint. ruff +
black clean.

Authorship

• Claude (Sonnet 5): code edits, review, and conceptual discussion
• Jonathan Schultz: overall vision, planning, review, and orchestration

---

Generated by Claude Code