Repository navigation
Add optional FP32 storage for trainable adapter parameters - #3299
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
--adapter_model_precision=no_change|fp32to the CLI, environment configuration, and WebUI. FP32 preserves small trainable LoRA/LyCORIS and auxiliary parameter updates while frozen base weights retain their configured precision. The default preserves existing behavior; FP32 rejects the BF16-only optimizer.Upcast before adapter loading and optimizer construction, retain precision through device moves and LyCORIS checkpoint restore, and document the memory tradeoff in every OPTIONS translation. Strengthen MOSS tests to update the dT-MLP before verifying inference, initialization, and checkpoint reloads.
Validation:
.venv/bin/python -m unittest -v -f tests.test_adapter_model_precision tests.test_moss_image— 47 tests passed.