Skip to content

About

StructLoRA (ACL 2026): structured, task-aware low-rank adaptation. Architecture and core training scaffold.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

Repository files navigation

StructLoRA

Official PyTorch implementation of the core StructLoRA architecture and algorithm for Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation (ACL 2026).

Paper · Project page

Release scope: The core modules and a tiny-model smoke test are available. Full benchmark recipes, model-specific launch scripts, and pretrained adapters are not included in this release.

StructLoRA keeps LoRA's zero-latency inference property while improving training with two training-only modules:

  • Information Bottleneck-guided direction filtering: learns a gate over LoRA rank-one directions and regularizes it with a variational IB-style KL term.
  • Graph-based layer coordination: treats adapted layers as graph nodes, passes filtered update signals across adjacent and semantically aligned layers, and materializes coordinated deltas for final merging.

Method Overview

frozen W0 + LoRA factors A/B
        |
rank-one directions B[:, i] A[i, :]
        |
IB gate m_i keeps task-relevant directions
        |
filtered updates become graph node features
        |
shallow residual GCN coordinates layer updates
        |
final deltas are merged into W0 for inference

Repository Layout

configs/structlora_base.yaml       Minimal StructLoRA configuration
scripts/smoke_test.py              End-to-end tiny model check
src/structlora/filtering.py        IB-guided rank-direction gate
src/structlora/layers.py           StructLoRA linear adapter
src/structlora/coordinator.py      Layer graph and residual GCN coordinator
src/structlora/injection.py        Regex-based injection into nn.Linear modules
src/structlora/losses.py           IB, sparsity, and drift regularizers
src/structlora/merge.py            Zero-cost inference merge utilities
src/structlora/training.py         Generic training-step helper
tests/test_structlora_core.py      Minimal forward/coordination test

Quick Start

pip install -r requirements.txt
python scripts/smoke_test.py
from structlora import (
    GraphCoordinator,
    StructLoRAConfig,
    StructLoRARegularizer,
    collect_structlora_layers,
    inject_structlora,
    merge_structlora,
)

inject_structlora(
    model,
    target_regex=["q_proj", "v_proj"],
    config=StructLoRAConfig(rank=8, alpha=16, dropout=0.05),
)
names, layers = zip(*collect_structlora_layers(model))
coordinator = GraphCoordinator.from_layers(list(layers), hidden_dim=128)
regularizer = StructLoRARegularizer(ib_kl_weight=1e-3, drift_weight=1e-2)

# During training:
coordinator.apply_to_layers(list(layers), list(names))
task_loss = compute_task_loss(model, batch)
reg = regularizer(list(layers))
loss = task_loss + reg["total"]
loss.backward()

# Before deployment:
merge_structlora(list(layers), coordinator, list(names))

This release contains the architecture and algorithmic core. Full benchmark recipes, model-specific launch scripts, and pretrained adapters will be released separately.

Citation

@misc{xiao2026structlora,
  title={Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation},
  author={Xiao, Xi and Ma, Chenrui and Zhang, Yunbei and Liu, Chen and Wang, Zhuxuanzi and Li, Yanshu and Zhao, Lin and Hu, Guosheng and Wang, Tianyang and Xu, Hao},
  year={2026},
  eprint={2603.14228},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

About

StructLoRA (ACL 2026): structured, task-aware low-rank adaptation. Architecture and core training scaffold.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages