Add Code Behind the Layout benchmark: generator-edit agents on SKY130

benchmarks/code_behind_layout/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.py[cod]
+.DS_Store
+.venv/
+/checkpoints/
+/runs/
+/logs/
+/results/
+/tmp/
+/runtime/*
+!/runtime/telescopic_sizing_qualification_20261019/
+!/runtime/e3_fidelity_control_20261019/

benchmarks/code_behind_layout/README.md

@@ -0,0 +1,95 @@
+# The Code Behind the Layout
+
+Code and frozen configurations for *The Code Behind the Layout: AI Agents for Analog Layout Generators*. The study asks a single question: does one shared edit to a gLayout generator's source improve post-extraction performance across a declared family of parameter configurations, while every configuration that passed before still passes? Every candidate is checked on SKY130: GDS generation, DRC, LVS against an independent reference, RC extraction, and SPICE at the tt/ss/ff corners.
+
+Two frontier models act as the editing policy: Claude Opus 5.5 and GPT-6.1. A local Qwen2.5-Coder-32B-Instruct model is fine-tuned under point and family supervision, with SFT and DPO.
+
+## Experiments
+
+| | Question | Entry points |
+| --- | --- | --- |
+| E1 | Which action level works: geometry (G0), primitives (G1) or generator source (G2)? | `scripts/run_e1_scope_suite.py`, `scripts/opus_cohort/run_e1_scope_suite_opus_v2.py` |
+| E2 | Which cheap signal ranks candidates before full evaluation: area, a geometric rule, extracted RC, or the calibrated interval F3? | `scripts/e2_four_source.py` |
+| E3 | Which test rule finds family regressions within a fixed budget: A anchor, R random, B boundary or C risk? | `scripts/run_e3_suite.py`, `scripts/opus_cohort/run_e3_suite_opus.py` |
+| E4 | Point vs. family supervision, SFT vs. DPO (T1–T4, seeds 41/73/109) | `scripts/train_policy.py`, `scripts/run_e4_*.sh`, `scripts/e4_checkpoint_validation.py` |
+| E5 | Physical CPU, GPU and token cost per verified configuration | `scripts/analyze_e5_costs.py` |
+
+The protocol is in [docs/protocol.md](docs/protocol.md), [docs/e1_protocol.md](docs/e1_protocol.md) and [docs/e3_protocol.md](docs/e3_protocol.md). The amendments are in [docs/protocol_amendments.md](docs/protocol_amendments.md). The S0/S1/S2 measurement interface is in [docs/measurement_interface.md](docs/measurement_interface.md).
+
+## Layout
+
+- `src/` contains the libraries: measurement and S0/S1/S2 assessment, trusted physical store, policy adapters, student interface, training, checkpoint selection and analysis.
+- `scripts/` contains the scripts for task preparation, collection, freezes, suite runners, training, analysis and figures.
+- `scripts/opus_cohort/` runs the frozen E1/E3 suites with the Claude Opus 5.5 policy.
+- `configs/` contains the study definition, model configs, task preparations, frozen experiment manifests (`configs/experiments/frozen/`), training configs and the E4 pool declaration.
+- `data/splits/`, `data/manifests/` contain the source-grouped split registry and the collection manifests.
+- `reproducibility/glayout_fixes.patch` contains the gLayout fixes used by the E1–E3 circuits.
+- `runtime/` contains the two qualified flow modules that the E2/E3 code loads. The data bundle below also extracts here.
+- `tests/` contains the unit tests.
+
+## Data bundle
+
+The frozen task packets and the E4 training set are distributed separately (`cbl-data-v1.tar.gz`, about 400 MB unpacked):
+
+```sh
+bash scripts/fetch_data.sh
+```
+
+It extracts the following into this directory:
+
+- `runtime/protocol_freezes/`: the frozen E1, E2 and E3 manifests with their task packets, registries, model configs, qualification and baseline plans. This includes the private evaluator side (references, benches, hidden configurations).
+- `runtime/e4_validation_preparation/`, `runtime/e4_validation_tasks/`: the E4 checkpoint-validation plan, budget and the nine-point validation task.
+- `data/training/e4_textbook_v1/`: the E4 training set (see its README).
+
+## Environment
+
+- **Analysis and unit tests.** Python 3.12:
+
+  ```sh
+  python3 -m pip install -r requirements.txt
+  python3 -m unittest discover -s tests -p 'test_*.py'
+  ```
+
+- **Physical flow.** Runs on Linux with Magic, Netgen, ngspice and SKY130A. Two gLayout trees are used:
+  - For E1–E3: gLayout at upstream commit `5af536b` with `reproducibility/glayout_fixes.patch` applied.
+  - For the E4 textbook collections: the `glayout.sharing` module and the `textbook_sharing` benchmark from ReaLLMASIC/gLayout#127.
+- **Training.** Needs one 80 GB GPU with PyTorch, Transformers, PEFT and bitsandbytes, and the base model `Qwen/Qwen2.5-Coder-32B-Instruct` at revision `381fc969f78efac66bc87ff7ddeadb7e73c218a7`.
+
+The scripts that drive the server from a workstation use these placeholders. Recorded paths in the manifests use the same ones.
+
+| Placeholder | Meaning |
+| --- | --- |
+| `cbl-server` | SSH host alias of the evaluation and training server (`~/.ssh/config`) |
+| `~/.ssh/cbl_server_key` | SSH identity for that host |
+| `/srv/code-behind-layout` | Location of this directory on the server |
+| `GPT_CLI`, `CLAUDE_CLI` | Command-line clients for the two frontier policies (`src/model_adapter.py`, `src/opus_cli_adapter.py`) |
+
+## Reproducing
+
+**E1 and E3.** Run the suite runners against the frozen manifests:
+
+```sh
+python3 scripts/run_e1_scope_suite.py --manifest runtime/protocol_freezes/E1_scope_v2_16k/manifest.json --output runs/E1_gpt
+python3 scripts/opus_cohort/run_e1_scope_suite_opus_v2.py --manifest runtime/protocol_freezes/E1_scope_v2_16k_claude_v2/manifest.json --output runs/E1_opus
+python3 scripts/run_e3_suite.py --part gpt --manifest runtime/protocol_freezes/E3_primary_v2/manifest.json --output runs/E3_gpt
+python3 scripts/opus_cohort/run_e3_suite_opus.py --part gpt --manifest runtime/protocol_freezes/E3_primary_v2_claude/manifest.json --output runs/E3_opus
+```
+
+E3 parts are `gpt` (the manifest's frontier policy under rules A/R/B/C), `parameter`, `signals`, `fidelity` (F3, with `E3_F3_primary`), `local` and `instance`. Each runner checks its freeze's bound implementation files before dispatch and reserves a trial identity before launch. Failures are recorded, never retried in place.
+
+**E2.**
+
+```sh
+python3 scripts/e2_four_source.py run --manifest runtime/protocol_freezes/E2_four_source_primary/manifest.json --output runs/E2 --method {full_oracle,fixed_area,adaptive_geometry,fixed_rc_product,f3_calibrated} [--pool-id <pool>]
+```
+
+**E4.**
+
+```sh
+python3 scripts/train_policy.py --manifest data/training/e4_textbook_v1/training_manifest.json --method T1 --seed 41 --output checkpoints/e4_textbook_v1/T1/seed-41 --validate-only
+bash scripts/run_e4_sft_training.sh data/training/e4_textbook_v1
+bash scripts/run_e4_validation_selection.sh data/training/e4_textbook_v1 runtime/e4_validation_preparation/e4_textbook_v1_validation_plan_v2.json T1 41 runs/e4_validation_v2
+bash scripts/run_e4_dpo_training.sh data/training/e4_textbook_v1 runs/e4_validation_v2/e4_textbook_v1/T1
+```
+
+`--validate-only` checks admission, the split registry and the samples without a GPU. T3/T4 use `training_manifest_dpo.json`. Checkpoints are selected by mean task S2 fraction on the nine validation points, with ties going to the earliest update; training loss is never used for selection. `scripts/assemble_e4_methods_draft.py` and `scripts/freeze_e4_methods.py` freeze the selected models, and `scripts/serve_frozen_policy.py` serves them to the E3 local-model part.

benchmarks/code_behind_layout/configs/analysis/e2_four_source_statistics_v1.json

@@ -0,0 +1,94 @@
+{
+  "status": "FROZEN_E2_ANALYSIS_PLAN",
+  "sealed_at_utc": "2026-10-19T06:27:04.783292+00:00",
+  "activation_evidence": "results/derived/E2_actual_analysis_declaration_pause.json",
+  "activation_observation": "All five original coordinator parents were paused at 2026-10-19T06:25:05.885Z. Each had zero full, PEX and nominal acquisitions. No formal electrical outcomes were available or inspected for this declaration. Resume preserves all original sources, streams, budgets and records.",
+  "analysis_id": "E2_four_source_statistics_v1",
+  "drafted_at_utc": "2026-10-19T06:26:13Z",
+  "applies_to_backend_manifest": "/srv/code-behind-layout/runtime/protocol_freezes/E2_four_source_primary/manifest.json",
+  "scope": "Separate prospective statistics declaration. The original E2 freeze did not declare these rank/subset rules. This file does not amend execution, selection, circuits, budgets, or any original record.",
+  "activation": "Root must seal this declaration and the observed all-five-coordinators full=PEX=nominal=0 pause evidence before resuming. A DRAFT or unbound declaration is exploratory only.",
+  "candidate_denominator": 64,
+  "pool_rules": {
+    "pool_count": 12,
+    "source_families": 4,
+    "sizings_per_source": 3,
+    "generation_attempt_limit": 256,
+    "generation_failures_and_exact_mask_duplicates": "Keep every attempt and actual acquisition cost. Do not replace a later physical or electrical failure after a candidate is accepted.",
+    "incomplete_oracle": "Any unmeasured or unresolved accepted candidate keeps the 64-candidate oracle incomplete. No finite-pool regret or full-pool top-k recall is defined.",
+    "no_feasible_oracle": "Report zero verified feasible and null regret/top-k recall. Do not invent an objective for failed or unknown candidates."
+  },
+  "objective": {
+    "name": "worst_ugb_Hz",
+    "unit": "Hz",
+    "direction": "max",
+    "feasible_selection_requires": "Complete original frozen physical, device-intent, analysis and electrical constraints over tt/ss/ff, with S2=true."
+  },
+  "near_frontier": {
+    "fraction": 0.2,
+    "count": "ceil(fraction * accepted_pool_size); 13 for the declared 64",
+    "selection": "Ascending actual physical_mask_bbox_area_um2, then ascending stream_index",
+    "selection_timing": "Membership uses geometry only, before conditioning on any PEX, nominal or full electrical outcome. Do not select the subset using true UGB or S2.",
+    "report_with": "Broad-pool and near-frontier results, declared/matched/nonrankable/unknown counts, and original accepted candidate denominator."
+  },
+  "numeric_ties": {
+    "prediction_absolute_tolerance": 0.0,
+    "actual_absolute_tolerance": 0.0,
+    "relative_tolerance": 0.0,
+    "primary": "Exact saved finite numeric values; Spearman average ranks and Kendall tau-b exact ties. Ranking cutoffs use ascending stream_index to select exactly k.",
+    "evidence": "results/derived/E2_actual_repeat_precision_evidence.json",
+    "evidence_limit": "Three same-PEX/same-original-TT-deck saved repetitions have identical decks and 9-significant-digit printed AC vectors, with observed maximum difference zero. This does not establish universally zero simulation noise."
+  },
+  "precision_sensitivity": {
+    "method": "Propagate each of the two printed unity-crossing rows' 9-significant-digit half-ULP intervals through the unchanged frozen measurement interpolation. Effective pair reversal uses only pairs with nonoverlapping actual and predictor intervals.",
+    "eligibility": "Only intervals derived from the actual candidate's saved rows and exact frozen measurement rule; an absent/unverified interval yields UNAVAILABLE for this sensitivity, not an invented tolerance.",
+    "primary_results_unchanged": true
+  },
+  "rank_tables": {
+    "same_objective_estimates": [
+      "actually acquired nominal_tt_ugb_Hz",
+      "saved frozen-calibration prediction_ugb_Hz"
+    ],
+    "same_objective_metrics": [
+      "Spearman",
+      "Kendall_tau_b",
+      "effective_pair_reversal_rate",
+      "top_k_recall"
+    ],
+    "descriptive_features": [
+      "physical_mask_bbox_area_um2",
+      "union_rectilinear_proxy_um",
+      "R_sum_times_C_sum_s"
+    ],
+    "descriptive_feature_interpretation": "Negative feature score ranks smaller area/wire/RC ahead. Report rank association and top-k selection recall only; this is not calibrated UGB prediction or same-unit prediction-error/reversal evidence.",
+    "rank_eligibility": "Use matched finite actual UGB and actually acquired predictor/feature values. Full-pool primary tables require a complete oracle; explicitly labeled conditional measured-pair diagnostics may be shown for incomplete pools, with all exclusions counted. Never turn failed or unknown measurements into zero UGB."
+  },
+  "top_k": {
+    "fraction": 0.2,
+    "predicted_count": "ceil(fraction * accepted_pool_size)",
+    "actual_target_count": "min(predicted_count, number of complete S2-feasible candidates)",
+    "actual_target": "Highest worst_ugb_Hz among complete S2-feasible candidates; stream_index breaks exact ties",
+    "recall": "Size of predicted top-k intersect actual target, divided by actual_target_count. All accepted candidates must have the declared score, and the finite-pool oracle must be complete; otherwise unavailable. Known infeasible candidates still consume predicted selection slots."
+  },
+  "same_sized_netlist_prelayout": {
+    "rule": "One constant schematic benchmark per independently fixed N(z). It has no layout ranking or rank correlation. Do not rank the 64 layouts using repeated copies of the same prelayout value, and do not pool different sized netlists to manufacture a layout ranking."
+  },
+  "selection_comparisons": {
+    "finite_pool_oracle": "Complete 64-candidate reference, identified separately from execution of every online strategy.",
+    "offline_replay": "Label any retrospective replay explicitly OFFLINE_ORACLE_REPLAY; account for the actual measurements used to construct it. It is not an online saving measurement.",
+    "real_online": "Use the independently executed method's actual ordered full_evaluations and all actual geometry/PEX/nominal acquisitions. No other method's labels entered its decisions.",
+    "join_requirement": "Same frozen pool/source/N/contract/deck/PDK/tool bindings and accepted stream indices, plus actual retained-GDS mask equality. IDs, bbox summaries or a declaration alone do not prove pool equality.",
+    "regret": "Oracle best feasible worst_ugb_Hz minus best actually delivered verified feasible worst_ugb_Hz, in Hz. Null if oracle incomplete/has no feasible candidate, or the online method has no verified feasible delivery."
+  },
+  "budget_curves": {
+    "query_axis": "Actual full-query attempts in recorded order; retain physical failure and unknown attempts",
+    "time_axis": "Only actual recorded timestamps/costs. Missing timing remains unavailable. Do not infer wall or CPU from allocation, query number or theoretical simulator count.",
+    "costs": "Charge generation failures/duplicates and every actually acquired fidelity, host mask/controller work and queue/deployment work. Nested references to the same physical run are counted once; per-SPICE CPU is a component of trusted CPU, not added twice."
+  },
+  "reporting_units": "Pool results remain separate by source and sized N; 64 layouts and three sizings are not independent circuits. Any cross-pool uncertainty resamples whole source-family clusters and states that there are four source families.",
+  "no_formal_results_read_for_this_draft": true,
+  "new_model_or_EDA_calls": 0,
+  "execution_freeze_id": "E2_four_source_primary",
+  "execution_manifest": "/srv/code-behind-layout/runtime/protocol_freezes/E2_four_source_primary/manifest.json",
+  "pause_evidence": "results/derived/E2_actual_analysis_declaration_pause.json"
+}