Make FSDP quorum recovery state-safe

[Skuld7451]

Summary

• allow selected state-dict callbacks to run on the training thread
• add first-quorum ordering so recovery state is sourced consistently
• make FSDP peer healing safe from out-of-phase replica-local collectives

Closes #343

Related work

Draft PR #112 addresses one part of this problem: it waits for the first quorum
before forward can run. This PR has the same ordering requirement when
init_sync=True; unlike #112, it preserves asynchronous startup when
init_sync=False.

The remaining changes in this PR are FSDP-specific and are not covered by #112:

• synchronize replica-local process-group setup before checkpoint transfer;
• assign all local ranks of a recovering replica to the same source replica;
• materialize collective FSDP state on the training thread; and
• isolate state reads from optimizer updates and queued device work.

If #112 lands first, this PR can rebase on it and remove the overlapping
first-quorum wait after the unconditional versus init_sync-gated behavior is
resolved. The FSDP recovery changes above remain necessary independently.

Dependencies

• Requires Use the current accelerator for streams and process-group lifecycle #344 for the accelerator-neutral stream and synchronization helpers
  used by the recovery path.
• Draft PR Disable async quorum for the first quorum sync #112 overlaps only with the first-quorum wait and is related work,
  not a prerequisite. If it lands first, this PR can rebase and remove the
  overlapping wait after the startup policy is resolved.

Testing

python -m pytest -q \
  torchft/manager_test.py \
  torchft/optim_test.py \
  torchft/checkpointing/http_transport_test.py
40 passed, 1 skipped

cargo test test_compute_quorum_results
4 passed

[meta-cla]

Hi @Skuld7451!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!