A graph neural network for counting cellular community motifs in spatial tissue graphs — built as a high-accuracy replacement for the subgraph counting component of TrimNN (Wang et al., 2024).
This project also includes the intestinal tissue dataset and VF2 ground-truth pipeline used for training and evaluation, developed independently to benchmark both TrimNN and SubgraphGNN against exact subgraph isomorphism counts.
| Metric | TrimNN | SubgraphGNN | Improvement |
|---|---|---|---|
| Size-3 Spearman vs VF2 ground truth | 0.456 | 0.816 | +79% |
| Size-4 Spearman vs VF2 ground truth | ~0.00 | 0.706 | — |
| Size-4 binary MCC | ~0.00 | 0.659 | — |
| Calibrated RMSE vs VF2 (size-3) | 1592 | 68 | −96% |
| Top-10 pattern overlap with TrimNN | — | 100% | — |
Evaluated on 2,925 size-3 patterns and 600 size-4 patterns, each scored across 1,000 tissue nodes selected by lowest 2-hop cell-type entropy (bot-entropy selection, empirically +2.8σ over random-500).
Multi-sample generalization — held out 10 intestinal samples across 4 donors never seen during training:
| Sample | Spearman |
|---|---|
| B005_ascending | 0.707 |
| B005_descendingSigmoid | 0.705 |
| B005_duodenum | 0.707 |
| B005_ileum | 0.761 |
| B006_descendingSigmoid | 0.779 |
| B006_descending | 0.771 |
| B008_ascending | 0.744 |
| B008_transverse | 0.757 |
| B010_ascending | 0.783 |
| B011_ascending | 0.786 |
| Average | 0.751 |
VF2 exact subgraph isomorphism used as ground truth throughout. TrimNN baseline for cross-sample comparison: Spearman = 0.456 (B004_ascending, size-3).
All intestinal tissue data used in this project was collected and processed as part of this work to benchmark the TrimNN algorithm against exact subgraph counts.
Scale: 65 tissue samples from 9 donors (B004–B012) spanning 8 gut regions:
- Ascending colon, descending colon, descending sigmoid, duodenum, ileum, mid-jejunum, proximal jejunum, transverse colon
Each sample is a spatial proteomics graph (CODEX/HuBMAP) where nodes are individual cells with cell-type labels and edges connect spatially adjacent cells (Delaunay triangulation). Graphs range from ~5,800 to ~33,000 nodes and 22–25 cell types per sample.
Graph variants: Each sample is processed in two forms:
nps— triangulation without edge pruning (denser graph)ps— pruning of outlier edges (99th percentile length threshold)
Training used 124 samples (all samples minus the B005 donor held out for evaluation). The B004_ascending sample is used as the primary benchmark target.
To evaluate both TrimNN and SubgraphGNN against exact counts, we implemented our own VF2 subgraph isomorphism runner using igraph's get_subisomorphisms_vf2. Three scripts handle different scopes:
Generates per-node VF2 training data for the synthetic demo graph (743 nodes, 8 cell types), sizes 5–9. Produces (pattern, k-hop subgraph, count) triples in the same format as the existing sizes 3–4 training data. Uses canonical growing-kite topologies to match TrimNN's motif family.
Generates whole-graph VF2 occurrence counts for the demo graph, sizes 5–9, using random label sampling. Faster than exhaustive enumeration — samples N random label assignments per size, computes global VF2 count for each. Output matches TrimNN's Predicted_occurrence_sizeN.csv format for direct comparison.
Same random-sampling strategy as gen_vf2_fast.py but applied to real intestinal tissue graphs. Targets sizes 5–7 on 5 representative samples (B004_ascending, B005_ascending, B006_descendingSigmoid, B008_ascending, B010_ascending).
| Metric | Sizes | Coverage |
|---|---|---|
| Exhaustive VF2 occurrence counts | 3, 4 | All 65 intestinal samples |
| VF2 occurrence counts (random-sampled) | 5–9 | Demo graph; 5 intestinal samples (s5–7 only) |
| SubgraphGNN Spearman vs VF2 | 3, 4 | Primary benchmark (B004_ascending) + 10 held-out samples |
| SubgraphGNN Spearman vs VF2 | 3–6 | Demo graph |
| SubgraphGNN binary MCC vs VF2 | 3, 4 | B004_ascending |
| TrimNN predictions for comparison | 3, 4 | B004_ascending (2,925 patterns) |
| TrimNN Spearman vs VF2 baseline | 3, 4 | B004_ascending |
| Metric | Why |
|---|---|
| Exhaustive VF2 for s5–9 on intestinal graphs | igraph VF2 is exponential in pattern size — at 17K–33K nodes it becomes intractable. A size-5 exhaustive run on B004_ascending would take days on CPU. |
| TrimNN predictions for s5–9 on intestinal graphs | TrimNN's motif enumeration step requires ~3 min/sample on an A100 GPU; we have no GPU. A test run estimated ~79 hours for the demo graph (743 nodes) on CPU alone. |
| Per-node VF2 counts for intestinal s5–9 | Would require exhaustive VF2, blocked by above. |
| SubgraphGNN vs TrimNN comparison on s5–9 | Blocked by missing TrimNN baseline. We have SubgraphGNN's Spearman vs sampled VF2 for s5 (demo graph: 0.618 all-patterns / 0.857 nonzero-only) but no TrimNN equivalent. |
The size 3 and 4 coverage is complete and fully benchmarked. For sizes 5+, SubgraphGNN is trained and produces predictions, but the ground-truth pipeline is sampled rather than exhaustive and TrimNN cannot be run for comparison without GPU access.
SubgraphGNN (train_trimnn.py) is a 3-layer message-passing GNN with:
- Bidirectional cross-attention between pattern and graph subgraph embeddings
- Gated pooling for permutation-invariant aggregation
- Size embedding for multi-size pattern generalization (sizes 3–9)
- Soft-BCE loss on rank-normalized VF2 targets — the key breakthrough over MSE
- Jointly trained on 89K synthetic demo samples + 124 real intestinal samples
Model size: 677K parameters (hidden=192, 3 layers).
Pattern graph ──► 3-layer GNN ──► cross-attn ──┐
├──► gated pool ──► sigmoid score
Target subgraph ──► 3-layer GNN ──► cross-attn ──┘
Key finding — soft-BCE loss: replacing MSE regression with soft binary cross-entropy on rank-normalized targets:
# Rank-normalized target: (i+1)/n_nonzero for nonzero patterns, 0.0 for zero
int_loss = F.binary_cross_entropy(int_pred, rank_targets.clamp(0.0, 1.0))This single change simultaneously improved RMSE by 59%, MCC by 89%, and Spearman by 7pp over the MSE baseline — the central result of the project.
| File | Description |
|---|---|
train_trimnn.py |
Model definition (SubgraphGNN) + full training loop |
eval_trimnn_benchmark.py |
Full benchmark vs TrimNN and VF2 on B004_ascending |
eval_multisample.py |
Cross-sample generalization eval (10 held-out intestinal samples) |
eval_allsizes.py |
Demo graph eval across sizes 3–6 |
eval_crosssample.py |
Cross-sample Spearman analysis |
eval_ensemble_benchmark.py |
Ensemble model evaluation |
gen_vf2_sizes59.py |
Per-node VF2 training data for demo graph, sizes 5–9 |
gen_vf2_fast.py |
Whole-graph VF2 counts for demo graph, sizes 5–9 (random-sampled) |
gen_vf2_intestinal59.py |
Whole-graph VF2 counts for intestinal graphs, sizes 5–7 |
time_inference.py |
Inference time profiling |
trimnn_app.py |
Streamlit app for interactive SubgraphGNN inference |
trimnn_model.pt |
Best trained weights (exp63, hidden=192, seed=42, 15000s) |
trimnn_model_BEST_backup.pt |
Earlier best weights backup |
trimnn_benchmark_results.txt |
Detailed benchmark output (early run) |
results.tsv |
Full experiment log (75+ experiments) |
run.log / run_exp*.log |
Training run logs |
# Train from scratch (~4h on 8-core CPU)
python3 train_trimnn.py
# Benchmark against TrimNN on B004_ascending (~35 min, CPU)
python3 eval_trimnn_benchmark.py --model trimnn_model.pt
# Multi-sample generalization eval (~23 min, CPU)
python3 eval_multisample.py --model trimnn_model.pt
# Demo graph across sizes 3-6
python3 eval_allsizes.py --model trimnn_model.pt --sizes 3,4,5,6
# Generate VF2 ground truth for demo graph sizes 5-9
python3 gen_vf2_fast.pyRequires: torch, igraph, numpy, scipy, pandas. No GPU required.
75+ experiments tracked in results.tsv. Key milestones:
- Baseline — MSE loss: Spearman 0.468 vs VF2
- Soft-BCE breakthrough: Spearman jumps to 0.81+, MCC 0.65+, RMSE −59%
- Architecture search: hidden=192, 3-layer GNN, dropout=0.0 confirmed optimal
- Ablations: width 128/192/256 (192 best), depth 3/4 (3 best), LR sweep, ILW=20 optimal
- Eval optimization: bot-1000 entropy node selection (+2.8σ over random-500 baseline)
- Plateau confirmed: 0.811–0.816 across 15+ experiments — ceiling appears data/architecture-limited at current scale
If you use this work, please also cite the original TrimNN paper:
@article{wang2024trimnn,
title={Characterizing cellular community motifs for studying multicellular
topological organization in complex tissues},
author={Wang, Juexin and others},
year={2024}
}Built on the autoresearch autonomous research framework by @karpathy.