Skip to content

Calibration expansion, stage 2: shape lattice, Qwen3.5-35B-A3B class, HybridEP fix, per-table certificates #852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
bradhilton wants to merge 20 commits into
base: trainer-rank-calibration-classes
Choose a base branch
from
Draft

Calibration expansion, stage 2: shape lattice, Qwen3.5-35B-A3B class, HybridEP fix, per-table certificates #852

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
0477b08
Add a shape-lattice calibration recipe for the multi-class campaign
bradhilton Sep 3, 2026
6941499
Record HybridEP rows and buffer capacity per calibration run
bradhilton Sep 3, 2026
fc6c4ff
Build HybridEP in the lattice recipe setup
bradhilton Sep 4, 2026
835ccda
Enable faulthandler in the lattice recipe so native crashes show Pyth…
bradhilton Sep 4, 2026
6eca347
Record a forced layout with no split-rung match as an admission failure
bradhilton Sep 4, 2026
5e58dc2
Size the HybridEP buffer for the busiest rank's planned rows
bradhilton Sep 4, 2026
cb2010e
One manifest and certificate per calibrated table
bradhilton Sep 4, 2026
521c5f7
Add the Qwen3.5-35B-A3B class manifest; let the fitter take exclusion…
bradhilton Sep 4, 2026
4cab443
Certify the Qwen3.5-35B-A3B execution class (gdn-moe-h2048-h200-bf16)
bradhilton Sep 4, 2026
2686332
Use a hidden-3072 example for the unmeasured-geometry fallback test
bradhilton Sep 4, 2026
554cb66
Document the CP-plan blind spot of the layout features
bradhilton Sep 4, 2026
c2ce9df
Raise the lattice calibration pod deadline to 24 h
bradhilton Sep 4, 2026
273631e
Certify the dense attention classes at their passing shapes; withhold…
bradhilton Sep 4, 2026
073f802
Vectorize the CP assignment search's plan evaluator (4-5x faster search)
bradhilton Sep 4, 2026
a2d1a56
Two-stage layout selection: re-rank CP4 shortlists by the CP plan str…
bradhilton Sep 4, 2026
725e4b3
Fitter: refuse a re-ranked-only evidence set for the direct table
bradhilton Sep 4, 2026
b54ead9
Calibrate the Qwen3-30B-A3B class (attention + MoE) where its gates pass
bradhilton Sep 4, 2026
deab9f5
Calibrate the Qwen3.5-27B class (dense GDN, hidden 5,120)
bradhilton Sep 4, 2026
6900b24
Test: HybridEP capacity reservation includes the largest planned rows
bradhilton Sep 5, 2026
8184957
Qwen3-30B-A3B: admit every measured shape after the sixteen-round re-…
bradhilton Sep 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 140 additions & 0 deletions dev/trainer_rank_cost_calibration_certificate_attn-moe-h2048-h200-bf16.json
View file
Open in desktop

Large diffs are not rendered by default.

87 changes: 87 additions & 0 deletions dev/trainer_rank_cost_calibration_certificate_dense-attn-h2048-h200-bf16.json
View file
Open in desktop

Large diffs are not rendered by default.

87 changes: 87 additions & 0 deletions dev/trainer_rank_cost_calibration_certificate_dense-attn-h4096-h200-bf16.json
View file
Open in desktop

Large diffs are not rendered by default.

101 changes: 101 additions & 0 deletions dev/trainer_rank_cost_calibration_certificate_dense-attn-h5120-h200-bf16.json
View file
Open in desktop

Large diffs are not rendered by default.

83 changes: 83 additions & 0 deletions dev/trainer_rank_cost_calibration_certificate_dense-gdn-h5120-h200-bf16.json
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions ...er_rank_cost_calibration_certificate.json → ...on_certificate_dense-h2560-h200-bf16.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@
{"candidates":[{"features":{"max_depth":2,"packed_tokens":19380,"segment_count":15,"segments_below":[0,0,1,4,8,10,14,15]},"label":"depth_one","median_ms":1107.144287109375,"n":8,"spread_pct":4.335775965007102},{"features":{"max_depth":2,"packed_tokens":25012,"segment_count":13,"segments_below":[0,0,0,1,4,7,11,13]},"label":"minimum_effective_span_2049","median_ms":1305.4077758789062,"n":8,"spread_pct":2.64851912545848},{"features":{"max_depth":2,"packed_tokens":20916,"segment_count":14,"segments_below":[0,0,1,2,6,9,13,14]},"label":"minimum_effective_span_513","median_ms":1173.864013671875,"n":8,"spread_pct":3.677009043192731},{"features":{"max_depth":1,"packed_tokens":29108,"segment_count":12,"segments_below":[0,0,0,0,2,5,9,12]},"label":"no_sharing","median_ms":1321.0035400390625,"n":8,"spread_pct":1.2978506719070204}],"cell":"cal-hetero|Qwen/Qwen3.5-4B|L32|tp2|cp1","device":"NVIDIA H200","device_capability":[9,0],"device_memory_class":"hbm-141g","facts":{"cp":1.0,"ep":1.0,"etp":1.0,"gdn_layers":24.0,"layers":32.0,"tp":2.0,"uses_gdn":1.0},"geometry":{"ffn_hidden_size":9216,"gdn_conv_kernel":4,"gdn_key_head_dim":128,"gdn_key_heads":16,"gdn_value_head_dim":128,"gdn_value_heads":32,"hidden_size":2560,"kv_channels":256,"moe_experts":0,"moe_ffn_hidden_size":0,"moe_shared_expert_ffn":0,"moe_topk":0,"num_attention_heads":16,"num_query_groups":4},"hidden_size":2560,"model":"Qwen/Qwen3.5-4B","param_dtype":"torch.bfloat16","requests_sha256":"ff3a1fcb7cc64401a9b7c23a99af2568e823fe915b381223e7fc8723b667a319","shape":[2,1,1,1],"source":"797746b9f425d844419fe609c164c2e4940507f3bbb2f82b815a73740d5abc2d","workload":{"kind":"hetero","seed":7777}}
],
"fit_arguments": {
"exclude_cells": [],
"exclude_cells": ["cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g0","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g1","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g2","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g3","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g4","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g5","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g6","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g7","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g0","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g1","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g2","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g3","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g4","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g5","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g6","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g7"],
"holdout": "cp4|g4",
"objective": "regret",
"require_complete": 8,
Expand All @@ -70,9 +70,9 @@
"integer_table_milli_us": {"attention_token_cp_exchange":27,"gdn_level":3336555,"gdn_level_tp":1194872,"gdn_token_per_rank":594,"level_cp_per_layer":187255,"level_tp_per_layer":684798,"tiny_segment_per_layer":219288,"token_cp_exchange":39,"token_per_rank":2002,"token_tp_collective":221},
"integer_table_sha256": "3624ae7dd985d92fed47b511e058f0e792559ead8919e11585dcb9860f3394ad",
"manifest": {
"excluded": [],
"expected": ["cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g0","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g1","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g2","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g3","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g4","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g5","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g6","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g7","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g0","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g1","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g2","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g3","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g4","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g5","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g6","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g7","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g0","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g1","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g2","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g3","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g4","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g5","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g6","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g7","cal-grpo-g16|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-grpo-g16|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-grpo-g16|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-grpo-g16|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-grpo-g4x4|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-grpo-g4x4|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-grpo-g4x4|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-grpo-g4x4|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-grpo-g8-long|Qwen/Qwen3-4B|L2|tp1|cp1","cal-grpo-g8-long|Qwen/Qwen3-4B|L2|tp1|cp4","cal-grpo-g8-long|Qwen/Qwen3.5-4B|L2|tp1|cp1","cal-grpo-g8-long|Qwen/Qwen3.5-4B|L2|tp1|cp4","cal-grpo-g8|Qwen/Qwen3-4B|L36|tp1|cp1","cal-grpo-g8|Qwen/Qwen3-4B|L36|tp1|cp2","cal-grpo-g8|Qwen/Qwen3-4B|L36|tp1|cp4","cal-grpo-g8|Qwen/Qwen3-4B|L36|tp2|cp1","cal-grpo-g8|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-grpo-g8|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-grpo-g8|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-grpo-g8|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-hetero2|Qwen/Qwen3-4B|L36|tp1|cp1","cal-hetero2|Qwen/Qwen3-4B|L36|tp1|cp4","cal-hetero2|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-hetero2|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-hetero2|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-hetero2|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-hetero3|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-hetero3|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-hetero3|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-hetero3|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-hetero|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-hetero|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-hetero|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-hetero|Qwen/Qwen3.5-4B|L32|tp2|cp1"],
"path": "trainer_rank_cost_calibration_manifest.json"
"excluded": ["cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g0","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g1","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g2","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g3","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g4","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g5","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g6","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g7","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g0","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g1","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g2","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g3","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g4","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g5","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g6","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g7"],
"expected": ["cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g0","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g1","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g2","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g3","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g4","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g5","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g6","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp1|g7","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g0","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g1","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g2","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g3","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g4","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g5","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g6","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp2|g7","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g0","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g1","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g2","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g3","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g4","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g5","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g6","cal-ellavox|Qwen/Qwen3.5-4B|L32|tp1|cp4|g7","cal-grpo-g16|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-grpo-g16|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-grpo-g16|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-grpo-g16|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-grpo-g4x4|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-grpo-g4x4|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-grpo-g4x4|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-grpo-g4x4|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-grpo-g8-long|Qwen/Qwen3-4B|L2|tp1|cp1","cal-grpo-g8-long|Qwen/Qwen3-4B|L2|tp1|cp4","cal-grpo-g8-long|Qwen/Qwen3.5-4B|L2|tp1|cp1","cal-grpo-g8-long|Qwen/Qwen3.5-4B|L2|tp1|cp4","cal-grpo-g8|Qwen/Qwen3-4B|L36|tp1|cp1","cal-grpo-g8|Qwen/Qwen3-4B|L36|tp1|cp2","cal-grpo-g8|Qwen/Qwen3-4B|L36|tp1|cp4","cal-grpo-g8|Qwen/Qwen3-4B|L36|tp2|cp1","cal-grpo-g8|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-grpo-g8|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-grpo-g8|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-grpo-g8|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-hetero2|Qwen/Qwen3-4B|L36|tp1|cp1","cal-hetero2|Qwen/Qwen3-4B|L36|tp1|cp4","cal-hetero2|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-hetero2|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-hetero2|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-hetero2|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-hetero3|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-hetero3|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-hetero3|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-hetero3|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-hetero|Qwen/Qwen3.5-4B|L32|tp1|cp1","cal-hetero|Qwen/Qwen3.5-4B|L32|tp1|cp2","cal-hetero|Qwen/Qwen3.5-4B|L32|tp1|cp4","cal-hetero|Qwen/Qwen3.5-4B|L32|tp2|cp1","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g0","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g1","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g2","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g3","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g4","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g5","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g6","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp1|cp4|g7","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g0","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g1","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g2","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g3","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g4","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g5","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g6","cal-ellavox-qwen3|Qwen/Qwen3-4B|L36|tp2|cp2|g7"],
"path": "trainer_rank_cost_calibration_manifest_dense-h2560-h200-bf16.json"
},
"measured_envelope": {"device_names":["NVIDIA H200"],"hidden_sizes":[2560],"param_dtypes":["torch.bfloat16"]},
"metrics": {
Expand Down
Loading
Loading