Official PyTorch implementation of the core StructLoRA architecture and algorithm for Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation (ACL 2026).
Release scope: The core modules and a tiny-model smoke test are available. Full benchmark recipes, model-specific launch scripts, and pretrained adapters are not included in this release.
StructLoRA keeps LoRA's zero-latency inference property while improving training with two training-only modules:
- Information Bottleneck-guided direction filtering: learns a gate over LoRA rank-one directions and regularizes it with a variational IB-style KL term.
- Graph-based layer coordination: treats adapted layers as graph nodes, passes filtered update signals across adjacent and semantically aligned layers, and materializes coordinated deltas for final merging.
frozen W0 + LoRA factors A/B
|
rank-one directions B[:, i] A[i, :]
|
IB gate m_i keeps task-relevant directions
|
filtered updates become graph node features
|
shallow residual GCN coordinates layer updates
|
final deltas are merged into W0 for inference
configs/structlora_base.yaml Minimal StructLoRA configuration
scripts/smoke_test.py End-to-end tiny model check
src/structlora/filtering.py IB-guided rank-direction gate
src/structlora/layers.py StructLoRA linear adapter
src/structlora/coordinator.py Layer graph and residual GCN coordinator
src/structlora/injection.py Regex-based injection into nn.Linear modules
src/structlora/losses.py IB, sparsity, and drift regularizers
src/structlora/merge.py Zero-cost inference merge utilities
src/structlora/training.py Generic training-step helper
tests/test_structlora_core.py Minimal forward/coordination test
pip install -r requirements.txt
python scripts/smoke_test.pyfrom structlora import (
GraphCoordinator,
StructLoRAConfig,
StructLoRARegularizer,
collect_structlora_layers,
inject_structlora,
merge_structlora,
)
inject_structlora(
model,
target_regex=["q_proj", "v_proj"],
config=StructLoRAConfig(rank=8, alpha=16, dropout=0.05),
)
names, layers = zip(*collect_structlora_layers(model))
coordinator = GraphCoordinator.from_layers(list(layers), hidden_dim=128)
regularizer = StructLoRARegularizer(ib_kl_weight=1e-3, drift_weight=1e-2)
# During training:
coordinator.apply_to_layers(list(layers), list(names))
task_loss = compute_task_loss(model, batch)
reg = regularizer(list(layers))
loss = task_loss + reg["total"]
loss.backward()
# Before deployment:
merge_structlora(list(layers), coordinator, list(names))This release contains the architecture and algorithmic core. Full benchmark recipes, model-specific launch scripts, and pretrained adapters will be released separately.
@misc{xiao2026structlora,
title={Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation},
author={Xiao, Xi and Ma, Chenrui and Zhang, Yunbei and Liu, Chen and Wang, Zhuxuanzi and Li, Yanshu and Zhao, Lin and Hu, Guosheng and Wang, Tianyang and Xu, Hao},
year={2026},
eprint={2603.14228},
archivePrefix={arXiv},
primaryClass={cs.CV}
}