feat: add distopt AllGather - next step Forward overlap - #217
Open
Chamberlain0w0 wants to merge 1 commit into
Open
feat: add distopt AllGather - next step Forward overlap#217Chamberlain0w0 wants to merge 1 commit into
Chamberlain0w0 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
ZeRO-1/2 在 optimizer 更新本地参数分片后,需要通过 AllGather 恢复完整参数。
原实现会在
DistributedOptimizer::Step()中启动并等待所有参数 AllGather 完成,下一轮 Forward 必须等全部参数同步结束后才能开始。实际上 Forward 只需要保证“当前即将使用的参数”已经同步,因此可以将后续 bucket 的 AllGather 与下一轮 Forward 计算重叠。主要修改
1. 按参数实际使用时机同步 bucket
Non-overlap 路径保持原有行为,在 optimizer step 后完成全部参数 AllGather。
2. 管理跨 iteration 的 AllGather 状态
3. 建立稳定的 Module/Parameter 注册顺序
Forward pre-hook 的 bucket 顺序依赖参数注册顺序,因此为
Module增加与 PyTorch 语义对齐的注册接口:RegisterParameter()RegisterBuffer()RegisterModule()底层仍保留
unordered_map用于按名称查询,同时使用 order vector 保存注册顺序。以下接口改为按注册顺序遍历:
Parameters()NamedParameters()NamedModules()Buffers()StateDict()To()/Apply()同时支持:
Parameters(recurse=false)获取 Module 的直接参数。StateDict()。项目内原先直接写入 registry map 的代码已迁移至
RegisterXXX()。