Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,17 @@ the 10 hand-labelled dev documents above. Later results are reported as *agreeme
drafting model*, never as accuracy, and the release gates in [`docs/targets.md`](docs/targets.md)
cannot be judged as declared.

| Agreement with model drafts (172 dev records, not accuracy) | Typed-condition micro-F1 | Exact span F1 | False-prerequisite rate |
|---|---:|---:|---:|
| Rules (`rules-2`) | 0.107 [0.058, 0.154] | 0.142 [0.085, 0.195] | 0.330 [0.165, 0.467] |
| CRF (`crf-pilot-2-batch-01`) | 0.128 [0.095, 0.168] | 0.174 [0.134, 0.215] | 0.416 [0.294, 0.493] |
| CRF (`crf-batches-01-02`, + 100 model-drafted `batch-02` records) | 0.145 [0.105, 0.185] | 0.221 [0.171, 0.268] | 0.353 [0.260, 0.416] |

Adding `batch-02` clearly improves exact-span agreement (+0.047, 95% CI [+0.004, +0.086]), but not
typed agreement. On the 10 hand-labelled documents, `crf-batches-01-02` scores 0.370 typed F1
[0.000, 0.640], and the difference from `crf-pilot-2-batch-01` is not clear; see
[`reports/agreement/`](reports/agreement/README.md).

95% project-group bootstrap intervals (1,000 resamples). Full reports, including scores against
the 28 model-made dev records (which measure agreement with the drafting model, not accuracy),
are in [`reports/pilot/`](reports/pilot/README.md). The paired difference from adding `batch-01` (+0.214 typed F1,
Expand Down Expand Up @@ -226,6 +237,8 @@ uv run triggermap predict --model crf --model-dir artifacts/crf-1 --out data/pre
uv run triggermap evaluate annotations/pilot/pilot-1.jsonl data/predictions/crf-1.jsonl --out reports/crf-1.md
uv run triggermap compare annotations/pilot/pilot-2.jsonl reports/pilot/predictions/rules-2.jsonl \
reports/pilot/predictions/crf-pilot-2.jsonl --gold-origin manual # paired bootstrap, B - A
uv run triggermap evaluate annotations/batches/batch-03.jsonl reports/agreement/predictions/crf-batches-01-02.jsonl \
--add-gold annotations/batches/batch-06.jsonl --split dev # agreement with model drafts, not accuracy
```

**Transformers** (once gold labels exist; DistilBERT, DeBERTa-v3-small; GPU recommended, see
Expand Down
4 changes: 3 additions & 1 deletion docs/model_card.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ the review-policy version.
Training uses the train pool of `splits/pool-v1.json` only (see
[`dataset_card.md`](dataset_card.md)). The models trained so far used `pilot-2` (39 in the
train split) and `batch-01` (100). Of these 139 records, 7 are `manual`. The other 132 are
model drafts. `batch-02` (100 more model-drafted train records) is available and not yet used.
model drafts. `crf-batches-01-02` adds `batch-02` (239 records: 7 `manual`, 232 model drafts).
Its dev results, as agreement with the model drafts and as accuracy on the 10 hand-labelled
documents, are in [`reports/agreement/`](../reports/agreement/README.md).

## Preliminary dev results

Expand Down
84 changes: 84 additions & 0 deletions reports/agreement/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Agreement with model drafts (dev split only)

Labels after the pilot are model drafts (owner's decision, 2026-11-05; see
[`docs/targets.md`](../../docs/targets.md)). This report scores three systems in two ways:

- **Agreement:** against the 172 model-drafted dev records of `batch-03` (112) and `batch-06`
(60), all `model_draft_accepted`, guidelines v0.2. These numbers measure how closely a system
reproduces the drafting model's labels. **They are not accuracy.**
- **Accuracy:** against the 10 hand-labelled dev documents of `pilot-2` (`--gold-origin manual`,
17 gold conditions), the only accuracy slice that exists.

The test split was not touched. All intervals are 95% project-group bootstrap intervals, 1,000
resamples, seed 0.

| System | Training records (train split) | Origins |
|---|---|---|
| `rules-2` | none (deterministic rules) | n/a |
| `crf-pilot-2-batch-01` | `pilot-2` + `batch-01`: 139 | 7 `manual`, 132 model drafts |
| `crf-batches-01-02` | `pilot-2` + `batch-01` + `batch-02`: 239 | 7 `manual`, 219 `model_draft_accepted`, 13 `model_draft_revised` |

## Agreement: 172 model-drafted dev documents

155 documents have draft conditions (386 conditions) and 17 have none.

| Metric | `rules-2` | `crf-pilot-2-batch-01` | `crf-batches-01-02` |
|---|---:|---:|---:|
| Typed-condition micro-F1 | 0.107 [0.058, 0.154] | 0.128 [0.095, 0.168] | 0.145 [0.105, 0.185] |
| Exact span F1 | 0.142 [0.085, 0.195] | 0.174 [0.134, 0.215] | 0.221 [0.171, 0.268] |
| Exact span precision | 0.272 [0.182, 0.340] | 0.277 [0.215, 0.372] | 0.332 [0.273, 0.400] |
| Exact span recall | 0.096 [0.055, 0.141] | 0.127 [0.090, 0.162] | 0.166 [0.122, 0.216] |
| Group exact-match F1 | 0.000 | 0.000 | 0.000 |
| Document exact match, no conditions | 0.941 [0.840, 1.000] | 0.941 [0.800, 1.000] | 0.882 [0.714, 1.000] |
| False-prerequisite rate (lower is better) | 0.330 [0.165, 0.467] | 0.416 [0.294, 0.493] | 0.353 [0.260, 0.416] |
| Reversed-meaning rate (lower is better) | 0.060 [0.011, 0.132] | 0.090 [0.035, 0.154] | 0.074 [0.029, 0.131] |

## Accuracy: 10 hand-labelled dev documents

| Metric | `rules-2` | `crf-pilot-2-batch-01` | `crf-batches-01-02` |
|---|---:|---:|---:|
| Typed-condition micro-F1 | 0.095 [0.000, 0.333] | 0.444 [0.133, 0.667] | 0.370 [0.000, 0.640] |
| Exact span precision | 0.250 [0.000, 1.000] | 0.600 [0.250, 1.000] | 0.500 [0.000, 0.818] |
| Exact span recall | 0.059 [0.000, 0.231] | 0.353 [0.077, 0.611] | 0.294 [0.000, 0.571] |
| Group exact-match F1 | 0.000 | 0.500 [0.000, 1.000] | 0.500 [0.000, 1.000] |
| False-prerequisite rate (lower is better) | 0.750 [0.000, 1.000] | 0.300 [0.000, 0.750] | 0.333 [0.000, 1.000] |

## Paired differences (same documents, same resamples)

| Comparison | Slice | Typed F1, B − A | Exact span F1, B − A | Report |
|---|---|---:|---:|---|
| `crf-pilot-2-batch-01` → `crf-batches-01-02` (adding `batch-02`) | agreement | +0.017 [−0.014, +0.047] | **+0.047 [+0.004, +0.086]** | [`compare_agreement_batch-02.md`](compare_agreement_batch-02.md) |
| `rules-2` → `crf-batches-01-02` | agreement | +0.038 [−0.012, +0.097] | **+0.079 [+0.028, +0.138]** | [`compare_agreement_rules.md`](compare_agreement_rules.md) |
| `crf-pilot-2-batch-01` → `crf-batches-01-02` (adding `batch-02`) | accuracy | −0.074 [−0.182, +0.000] | −0.074 [−0.182, +0.000] | [`compare_manual_batch-02.md`](compare_manual_batch-02.md) |

Bold marks a clear difference (the whole interval on one side of zero).

## Reading

- **Agreement is low.** Even the best system reproduces about one in six of the drafting
model's exact spans (span recall 0.166), and typed F1 stays below 0.15. Exact matching is
strict: a span that differs by one word counts as a miss.
- **Adding `batch-02` clearly improves span agreement** (+0.047 exact span F1) but not typed
agreement: the interval for typed F1 includes zero.
- **The CRF clearly beats the rules on span agreement** (+0.079), but not on typed F1 or on the
false-prerequisite rate.
- **On the 10 hand-labelled documents nothing is clear.** Adding `batch-02` moves typed F1 from
0.444 to 0.370, with a paired interval of [−0.182, +0.000]. Ten documents cannot separate that
from chance.
- **No group is matched** in the agreement slice. The drafts contain few groups (9 in
`batch-03`), and no system reproduces one exactly.

None of these numbers is a release result, and agreement is not accuracy.

## Reproduce

```bash
uv run triggermap train --gold annotations/pilot/pilot-2.jsonl annotations/batches/batch-01.jsonl annotations/batches/batch-02.jsonl --out artifacts/crf-batches-01-02 --model-id crf-batches-01-02
uv run triggermap predict --model crf --model-dir artifacts/crf-batches-01-02 --sample annotations/pilot/pilot_manifest.json annotations/batches/batch-03.json annotations/batches/batch-06.json --out reports/agreement/predictions/crf-batches-01-02.jsonl
uv run triggermap evaluate annotations/batches/batch-03.jsonl reports/agreement/predictions/crf-batches-01-02.jsonl --add-gold annotations/batches/batch-06.jsonl --split dev --out reports/agreement/crf-batches-01-02_dev_agreement.md
uv run triggermap evaluate annotations/pilot/pilot-2.jsonl reports/agreement/predictions/crf-batches-01-02.jsonl --split dev --gold-origin manual --out reports/agreement/crf-batches-01-02_dev_manual.md
uv run triggermap compare annotations/batches/batch-03.jsonl reports/agreement/predictions/crf-pilot-2-batch-01.jsonl reports/agreement/predictions/crf-batches-01-02.jsonl --add-gold annotations/batches/batch-06.jsonl --split dev --out reports/agreement/compare_agreement_batch-02.md
```

`rules-2` uses `predict` without `--model`, and `crf-pilot-2-batch-01` uses the bundle trained in
[`reports/training-origin/`](../training-origin/README.md).
25 changes: 25 additions & 0 deletions reports/agreement/compare_agreement_batch-02.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Agreement: adding batch-02 (crf-pilot-2-batch-01 -> crf-batches-01-02)

- Gold: `annotations/batches/batch-03.jsonl`, `annotations/batches/batch-06.jsonl` (dev split of `splits/pool-v1.json`; origin: all; 172 documents)
- Baseline (A): `reports/agreement/predictions/crf-pilot-2-batch-01.jsonl` (crf-pilot-2-batch-01)
- Candidate (B): `reports/agreement/predictions/crf-batches-01-02.jsonl` (crf-batches-01-02)
- Bootstrap: 1000 paired resamples, project clusters, seed 0
- Documents: 172

Differences are B minus A with 95% paired bootstrap intervals. *Clear* means the whole interval lies on one side of zero.

| Metric | A (crf-pilot-2-batch-01) | B (crf-batches-01-02) | B - A | 95% CI | Verdict |
|---|---:|---:|---:|:---:|---|
| Typed condition micro-F1 (span + category + role) | 0.128 | 0.145 | +0.017 | [-0.014, +0.047] | no clear difference |
| Exact span F1 | 0.174 | 0.221 | +0.047 | [+0.004, +0.086] | B clearly better |
| Exact span precision | 0.277 | 0.332 | +0.055 | [-0.026, +0.108] | no clear difference |
| Exact span recall | 0.127 | 0.166 | +0.039 | [+0.006, +0.069] | B clearly better |
| Role macro-F1 (roles with gold support) | 0.089 | 0.115 | +0.026 | [-0.001, +0.059] | no clear difference |
| Category macro-F1 | 0.122 | 0.137 | +0.015 | [-0.007, +0.035] | no clear difference |
| Group exact-match F1 | 0.000 | 0.000 | +0.000 | [+0.000, +0.000] | no clear difference |
| Complete-document exact match, documents with conditions | 0.058 | 0.065 | +0.006 | [-0.015, +0.027] | no clear difference |
| Complete-document exact match, documents without conditions | 0.941 | 0.882 | -0.059 | [-0.160, +0.000] | no clear difference |
| False prerequisite rate (lower is better) | 0.416 | 0.353 | -0.062 | [-0.124, +0.036] | no clear difference |
| Reversed meaning rate (lower is better) | 0.090 | 0.074 | -0.016 | [-0.054, +0.014] | no clear difference |
| Category accuracy on exactly matched spans | 0.857 | 0.797 | -0.060 | [-0.138, -0.003] | B clearly worse |
| Role accuracy on exactly matched spans | 0.878 | 0.859 | -0.018 | [-0.071, +0.064] | no clear difference |
25 changes: 25 additions & 0 deletions reports/agreement/compare_agreement_rules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Agreement: rules-2 -> crf-batches-01-02

- Gold: `annotations/batches/batch-03.jsonl`, `annotations/batches/batch-06.jsonl` (dev split of `splits/pool-v1.json`; origin: all; 172 documents)
- Baseline (A): `reports/agreement/predictions/rules-2.jsonl` (rules-2)
- Candidate (B): `reports/agreement/predictions/crf-batches-01-02.jsonl` (crf-batches-01-02)
- Bootstrap: 1000 paired resamples, project clusters, seed 0
- Documents: 172

Differences are B minus A with 95% paired bootstrap intervals. *Clear* means the whole interval lies on one side of zero.

| Metric | A (rules-2) | B (crf-batches-01-02) | B - A | 95% CI | Verdict |
|---|---:|---:|---:|:---:|---|
| Typed condition micro-F1 (span + category + role) | 0.107 | 0.145 | +0.038 | [-0.012, +0.097] | no clear difference |
| Exact span F1 | 0.142 | 0.221 | +0.079 | [+0.028, +0.138] | B clearly better |
| Exact span precision | 0.272 | 0.332 | +0.060 | [-0.018, +0.171] | no clear difference |
| Exact span recall | 0.096 | 0.166 | +0.070 | [+0.030, +0.113] | B clearly better |
| Role macro-F1 (roles with gold support) | 0.116 | 0.115 | -0.002 | [-0.071, +0.126] | no clear difference |
| Category macro-F1 | 0.098 | 0.137 | +0.039 | [-0.002, +0.082] | no clear difference |
| Group exact-match F1 | 0.000 | 0.000 | +0.000 | [+0.000, +0.000] | no clear difference |
| Complete-document exact match, documents with conditions | 0.019 | 0.065 | +0.045 | [+0.007, +0.084] | B clearly better |
| Complete-document exact match, documents without conditions | 0.941 | 0.882 | -0.059 | [-0.200, +0.000] | no clear difference |
| False prerequisite rate (lower is better) | 0.330 | 0.353 | +0.023 | [-0.091, +0.170] | no clear difference |
| Reversed meaning rate (lower is better) | 0.060 | 0.074 | +0.014 | [-0.047, +0.067] | no clear difference |
| Category accuracy on exactly matched spans | 0.865 | 0.797 | -0.068 | [-0.199, +0.083] | no clear difference |
| Role accuracy on exactly matched spans | 0.892 | 0.859 | -0.033 | [-0.113, +0.092] | no clear difference |
25 changes: 25 additions & 0 deletions reports/agreement/compare_manual_batch-02.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Hand-labelled dev: adding batch-02

- Gold: `annotations/pilot/pilot-2.jsonl` (dev split of `splits/pool-v1.json`; origin: manual; 10 documents)
- Baseline (A): `reports/agreement/predictions/crf-pilot-2-batch-01.jsonl` (crf-pilot-2-batch-01)
- Candidate (B): `reports/agreement/predictions/crf-batches-01-02.jsonl` (crf-batches-01-02)
- Bootstrap: 1000 paired resamples, project clusters, seed 0
- Documents: 10

Differences are B minus A with 95% paired bootstrap intervals. *Clear* means the whole interval lies on one side of zero.

| Metric | A (crf-pilot-2-batch-01) | B (crf-batches-01-02) | B - A | 95% CI | Verdict |
|---|---:|---:|---:|:---:|---|
| Typed condition micro-F1 (span + category + role) | 0.444 | 0.370 | -0.074 | [-0.182, +0.000] | no clear difference |
| Exact span F1 | 0.444 | 0.370 | -0.074 | [-0.182, +0.000] | no clear difference |
| Exact span precision | 0.600 | 0.500 | -0.100 | [-0.500, +0.027] | no clear difference |
| Exact span recall | 0.353 | 0.294 | -0.059 | [-0.200, +0.000] | no clear difference |
| Role macro-F1 (roles with gold support) | 0.231 | 0.200 | -0.031 | [-0.122, +0.000] | no clear difference |
| Category macro-F1 | 0.267 | 0.237 | -0.030 | [-0.133, +0.000] | no clear difference |
| Group exact-match F1 | 0.500 | 0.500 | +0.000 | [+0.000, +0.000] | no clear difference |
| Complete-document exact match, documents with conditions | 0.222 | 0.222 | +0.000 | [+0.000, +0.000] | no clear difference |
| Complete-document exact match, documents without conditions | 1.000 | 1.000 | +0.000 | [+0.000, +0.000] | no clear difference |
| False prerequisite rate (lower is better) | 0.300 | 0.333 | +0.033 | [-0.050, +0.500] | no clear difference |
| Reversed meaning rate (lower is better) | 0.000 | 0.000 | +0.000 | [+0.000, +0.000] | no clear difference |
| Category accuracy on exactly matched spans | 1.000 | 1.000 | +0.000 | [-1.000, +0.000] | no clear difference |
| Role accuracy on exactly matched spans | 1.000 | 1.000 | +0.000 | [-1.000, +0.000] | no clear difference |
57 changes: 57 additions & 0 deletions reports/agreement/crf-batches-01-02_dev_agreement.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Agreement with model drafts: crf-batches-01-02

- Gold: `annotations/batches/batch-03.jsonl`, `annotations/batches/batch-06.jsonl` (dev split of `splits/pool-v1.json`; origin: all; 172 documents)
- Predictions: `reports/agreement/predictions/crf-batches-01-02.jsonl` (crf-batches-01-02)
- Bootstrap: 1000 resamples, project clusters, seed 0
- Documents: 172 (155 with gold conditions, 17 without)

## Headline metrics (95% bootstrap intervals)

| Metric | Estimate | 95% CI |
|---|---:|:---:|
| Typed condition micro-F1 (span + category + role) | 0.145 | [0.105, 0.185] |
| Exact span F1 | 0.221 | [0.171, 0.268] |
| Exact span precision | 0.332 | [0.273, 0.400] |
| Exact span recall | 0.166 | [0.122, 0.216] |
| Role macro-F1 (roles with gold support) | 0.115 | [0.065, 0.160] |
| Category macro-F1 | 0.137 | [0.094, 0.174] |
| Group exact-match F1 | 0.000 | [0.000, 0.000] |
| Complete-document exact match, documents with conditions | 0.065 | [0.030, 0.104] |
| Complete-document exact match, documents without conditions | 0.882 | [0.714, 1.000] |
| False prerequisite rate (lower is better) | 0.353 | [0.260, 0.416] |
| Reversed meaning rate (lower is better) | 0.074 | [0.029, 0.131] |
| Category accuracy on exactly matched spans | 0.797 | [0.667, 0.880] |
| Role accuracy on exactly matched spans | 0.859 | [0.758, 0.976] |

## Per-role results (exact span + role)

| Label | Precision | Recall | F1 | Gold support |
|---|---:|---:|---:|---:|
| REQUIRED | 0.305 | 0.158 | 0.208 | 323 |
| EXCEPTION | 0.000 | 0.000 | 0.000 | 17 |
| NOT_REQUIRED | 0.286 | 0.222 | 0.250 | 18 |
| UNCERTAIN | 0.000 | 0.000 | 0.000 | 28 |

## Per-category results (exact span + category)

| Label | Precision | Recall | F1 | Gold support |
|---|---:|---:|---:|---:|
| CONFIGURATION | 0.095 | 0.056 | 0.070 | 36 |
| USAGE | 0.189 | 0.097 | 0.128 | 186 |
| ACCESS | 0.405 | 0.224 | 0.288 | 134 |
| PLATFORM | 0.333 | 0.033 | 0.061 | 30 |

## Precision versus coverage

| Coverage | Confidence ≥ | Typed precision | Retained recall |
|---:|---:|---:|---:|
| 10% | 0.834 | 0.316 | 0.016 |
| 20% | 0.736 | 0.385 | 0.039 |
| 30% | 0.670 | 0.431 | 0.065 |
| 40% | 0.591 | 0.351 | 0.070 |
| 50% | 0.505 | 0.323 | 0.080 |
| 60% | 0.464 | 0.276 | 0.083 |
| 70% | 0.421 | 0.267 | 0.093 |
| 80% | 0.385 | 0.253 | 0.101 |
| 90% | 0.333 | 0.236 | 0.106 |
| 100% | 0.186 | 0.218 | 0.109 |
57 changes: 57 additions & 0 deletions reports/agreement/crf-batches-01-02_dev_manual.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Hand-labelled dev: crf-batches-01-02

- Gold: `annotations/pilot/pilot-2.jsonl` (dev split of `splits/pool-v1.json`; origin: manual; 10 documents)
- Predictions: `reports/agreement/predictions/crf-batches-01-02.jsonl` (crf-batches-01-02)
- Bootstrap: 1000 resamples, project clusters, seed 0
- Documents: 10 (9 with gold conditions, 1 without)

## Headline metrics (95% bootstrap intervals)

| Metric | Estimate | 95% CI |
|---|---:|:---:|
| Typed condition micro-F1 (span + category + role) | 0.370 | [0.000, 0.640] |
| Exact span F1 | 0.370 | [0.000, 0.640] |
| Exact span precision | 0.500 | [0.000, 0.818] |
| Exact span recall | 0.294 | [0.000, 0.571] |
| Role macro-F1 (roles with gold support) | 0.200 | [0.000, 0.600] |
| Category macro-F1 | 0.237 | [0.000, 0.524] |
| Group exact-match F1 | 0.500 | [0.000, 1.000] |
| Complete-document exact match, documents with conditions | 0.222 | [0.000, 0.455] |
| Complete-document exact match, documents without conditions | 1.000 | [0.000, 1.000] |
| False prerequisite rate (lower is better) | 0.333 | [0.000, 1.000] |
| Reversed meaning rate (lower is better) | 0.000 | [0.000, 0.000] |
| Category accuracy on exactly matched spans | 1.000 | [0.000, 1.000] |
| Role accuracy on exactly matched spans | 1.000 | [0.000, 1.000] |

## Per-role results (exact span + role)

| Label | Precision | Recall | F1 | Gold support |
|---|---:|---:|---:|---:|
| REQUIRED | 0.556 | 0.312 | 0.400 | 16 |
| EXCEPTION | 0.000 | 0.000 | 0.000 | 0 |
| NOT_REQUIRED | 0.000 | 0.000 | 0.000 | 0 |
| UNCERTAIN | 0.000 | 0.000 | 0.000 | 1 |

## Per-category results (exact span + category)

| Label | Precision | Recall | F1 | Gold support |
|---|---:|---:|---:|---:|
| CONFIGURATION | 0.000 | 0.000 | 0.000 | 1 |
| USAGE | 0.429 | 0.333 | 0.375 | 9 |
| ACCESS | 1.000 | 0.400 | 0.571 | 5 |
| PLATFORM | 0.000 | 0.000 | 0.000 | 2 |

## Precision versus coverage

| Coverage | Confidence ≥ | Typed precision | Retained recall |
|---:|---:|---:|---:|
| 10% | 0.905 | 0.000 | 0.000 |
| 20% | 0.871 | 0.500 | 0.059 |
| 30% | 0.815 | 0.667 | 0.118 |
| 40% | 0.791 | 0.750 | 0.176 |
| 50% | 0.755 | 0.800 | 0.235 |
| 60% | 0.666 | 0.667 | 0.235 |
| 70% | 0.644 | 0.571 | 0.235 |
| 80% | 0.589 | 0.500 | 0.235 |
| 90% | 0.441 | 0.444 | 0.235 |
| 100% | 0.393 | 0.500 | 0.294 |
Loading