fix(mori,docker): unblock MORI smoke on containerized RDMA clusters - #452
Closed
speriaswamy-amd wants to merge 5 commits into
Closed
speriaswamy-amd wants to merge 5 commits into
speriaswamy-amd wants to merge 5 commits into
Conversation
…onfig Smoke now passes smoke.steps via maxtext_overrides so polls match the configured step count instead of maxtext_config.steps. Add dtype, weight_dtype, and quantization to the DSv4-284B maxtext_config so training sweeps use BF16 weights and avoid GPU OOM after smoke passes. Signed-off-by: Saravanan Solaiyappan <saravanan.solaiyappan@amd.com>
Point jaxmaxtext READMEs at jaxmaxtext.rst instead of the removed jax.rst after the how-to guide was renamed. Signed-off-by: Saravanan Solaiyappan <saravanan.solaiyappan@amd.com>
* removed the values of few variables that were accidently checked in the code * Addressed the review comment to fix the value of rdma connectivity mode --------- Signed-off-by: Urvashi Tiwari <urtiwari.com>
Document AMD Node Check under Burn-in / Diag and mention it with MI4XX health tests on the concepts page. Co-authored-by: Cursor <cursoragent@cursor.com>
Set memlock ulimit only for MORI container launch (ibv_reg_mr needs it; Docker default 8MB breaks RDMA registration). Export torchlib_dir via LD_LIBRARY_PATH in mori env script (CVS-004). Accept concurrent_put_thread/ imm_thread completion marker as smoke pass.
speriaswamy-amd
force-pushed
the
speriaswamy/mori-docker-validation-fixes
branch
from
September 17, 2026 23:43
108a730 to
4c858a1
Compare
Contributor
Author
|
Replacing with a branch cut from main (previous head was based on release/cvs-0.2.0 and pulled in unrelated release commits). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
docker_lib.py: add--ulimit memlock=-1(Docker defaults to 8MB; breaksibv_reg_mrfor RDMA)mori_lib.py: exportLD_LIBRARY_PATHfromtorchlib_dir(CVS-004)mori_lib.py: accepttest done!forconcurrent_put_thread/concurrent_put_imm_threadsmoke checks (these binaries do not printPASSED)Test plan
concurrent_put_*, shmem)Made with Cursor