Skip to content

Add optional FP32 storage for trainable adapter parameters - #3299

Merged
bghira merged 1 commit into
mainfrom
feature/adapter-fp32-precision
Oct 28, 2026
Merged

bghira merged 1 commit into
mainfrom
feature/adapter-fp32-precision

Conversation

@bghira

@bghira bghira commented Oct 28, 2026

Copy link
Copy Markdown
Owner

Add --adapter_model_precision=no_change|fp32 to the CLI, environment configuration, and WebUI. FP32 preserves small trainable LoRA/LyCORIS and auxiliary parameter updates while frozen base weights retain their configured precision. The default preserves existing behavior; FP32 rejects the BF16-only optimizer.

Upcast before adapter loading and optimizer construction, retain precision through device moves and LyCORIS checkpoint restore, and document the memory tradeoff in every OPTIONS translation. Strengthen MOSS tests to update the dT-MLP before verifying inference, initialization, and checkpoint reloads.

Validation:

  • .venv/bin/python -m unittest -v -f tests.test_adapter_model_precision tests.test_moss_image — 47 tests passed.
  • Full unittest discovery passed all tests before a fixed-port webhook fixture collision between parallel suites. The webhook fixture and remaining 479 tests passed when rerun serially.
  • All pre-commit hooks passed.

@bghira
bghira merged commit 89e9702 into main Oct 28, 2026
4 checks passed
@bghira
bghira deleted the feature/adapter-fp32-precision branch October 28, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant