Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions docs/contracts/project-folder.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ examples, except `environments/cuda-torch`, a second environment not shown.
| `run` | unless `earlier_work.rebuild` is `no` | see the `run` block |
| `splits` | no | object, split name to a number of items (integer ≥ 0) |
| `view` | no | how its product is shown as records; see [Looking inside a product](#looking-inside-a-product) |
| `generation` | no | the data generation its data comes from, 1 to 64 characters of `a-z`, `0-9`, `.`, `_` and `-`, not `mixed`; without it, its inputs' (see [Data generations](#data-generations)) |

A dataset's run may name a `step` instead of a `command`: see
[Steps Tuldok provides](#steps-tuldok-provides).
Expand Down Expand Up @@ -575,6 +576,7 @@ A recipe trains a model; its product is weights.
| `compared_with` | no | at most 50 objects, each `recipe` (reference to another recipe, each once in the list), `explanation` (1 to 2,000 characters) and `excerpt` (optional, at most 2,000 characters); no other field |
| `serve` | no | how Tuldok serves the model: `module_file` (a code path of a `.py` file, required), `function` (default `predict`), `load`, `backend` (`local`, the default and only one), `environment` (default `run.environment`; one of the two is required), `idle_minutes` (1 to 1,440), `start_seconds` (1 to 3,600), `request_seconds` (0.1 to 600), `gpu`, `example` (an object); see [Serving a model](#serving-a-model). Not an input: it never makes the model out of date |
| `sweep` | no | a trial of a sweep (see [`sweep`](#sweep)): `definition` (reference to `sweeps/`), `trial` (its number, 1 or more) and `values` (the settings it was given, by path, at most 16); written by Tuldok when it runs the sweep |
| `generations` | no | only `"mixed"`: it trains on data of more than one generation on purpose (see [Data generations](#data-generations)); a branch and a sweep's trial carry it |

The label `short`, when a recipe has it, is the text of its node on the map
(at most 16 characters are shown). `derived_from` and `compared_with` say how
Expand Down Expand Up @@ -613,6 +615,7 @@ solver.
| Field | Required | Rule |
| --- | --- | --- |
| `run` | unless `earlier_work.rebuild` is `no` | see the `run` block |
| `generation` | no | the data generation of what it makes (a solver, a source), 1 to 64 characters of `a-z`, `0-9`, `.`, `_` and `-`, not `mixed`; without it, its inputs' (see [Data generations](#data-generations)) |

Example, `steps/reference-solver.json`:

Expand Down Expand Up @@ -643,6 +646,8 @@ Its command runs once per subject.
| `subjects` | yes | 0 to 1,000 objects, see below |
| `cases` | no | `{"dataset"?: reference to datasets/, "split"?: non-empty string, "count"?: integer ≥ 0}` |
| `summary` | no | `{"group_by"?: up to 8 distinct ids, "statistics"?: distinct names from mean, median, p10, p90, min, max}` |
| `generation` | no | the data generation its cases come from, 1 to 64 characters of `a-z`, `0-9`, `.`, `_` and `-`, not `mixed`; without it, its cases' data's (see [Data generations](#data-generations)) |
| `generations` | no | only `"mixed"`: its subjects may come from different data generations |

A metric is `{"id", "title", "units"?, "better"}`: `id` matches
`^[a-z][a-z0-9_]{0,63}$` and is unique in the benchmark; `title` is 1 to 200
Expand Down Expand Up @@ -1000,6 +1005,47 @@ pilot-dense-k1,velocity_error_final,cells_per_metre=8,4,0,0.195,0.175,0.129,0.27
group and no group is its own ancestor; a recipe is never derived from or
compared with itself; following `derived_from` never comes back to where it
started; `map.chart` is a chart of the form `progression`.
8. Data generations do not mix unless a recipe or benchmark says so; see
[Data generations](#data-generations).

### Data generations

A **data generation** names the source a project's data comes from: the
solver, simulator, sensor or corpus release that made it (`rheon-reference-v1`,
`rheon-v2`). It keeps data of one generation for the models of that generation,
and says so when they cross. The label is opt-in: a folder that writes none
checks as before, and nothing without a generation is reported.

- A dataset or step may declare `generation`. One that declares none takes the
generation of its **data inputs**: the entries of `run.inputs` that name a
dataset or a step, whatever made it (`freeze`, `select`, `embed`, `annotate`,
`import` with dataset inputs, or the project's own command). Inputs without a
generation do not decide it; a dataset or step none of whose inputs has one has
none.
- **Warning:** a dataset, step, recipe or benchmark that has a generation
(declared, inherited or `mixed`) and takes a data input that has none ("takes
unlabelled data <reference>; label it with the generation it comes from"). An
input without a generation because its own inputs disagree is not named (it
has its error). A project that labels nothing gets no such warning.
- **Error:** a dataset or step whose data inputs have more than one generation
(the message names each input and its generation), and one whose declared
generation differs from an input's. Data has no escape: declaring one does
not settle inputs that disagree.
- A recipe's generation is computed: that of its data inputs. **Error:** a
recipe whose data inputs have more than one generation ("mixes data
generations"), unless it declares `"generations": "mixed"`, a deliberate
cross-generation experiment, which is then a **warning**; its generation is
then `mixed`. A recipe input (a model to start from) is not data: a recipe
derived from, or fine-tuned from, a model of another generation is fine, and
the Map shows it.
- A benchmark's generation is its declared `generation`, else that of its cases'
data (`cases.dataset` and the data inputs of its run). Its cases' data is
checked against the declaration as a dataset's inputs are (**error**).
**Warning:** its subjects' recipes have more than one generation (`mixed`
counts as one), unless it declares `"generations": "mixed"`.
- The reads of the database copy add `generation` where there is one: to a
model of `map`, to a dataset, step, recipe or benchmark of `definition`, and
to its row of `definitions`.

## Checking a folder

Expand Down
66 changes: 66 additions & 0 deletions docs/plans/training-and-curation/spec-generations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Implementation spec: data generations

A small feature beside [the training and curation plan](plan.md). A project can say which **generation** of its source its data comes from (the solver, simulator, sensor or corpus release that made it), and Tuldok keeps the generations apart: a model trains on one generation's data, a dataset made from others keeps their generation, and mixing them is an error unless the definition says it is meant. The label is general, not of one project; it is opt-in, so a project that writes none checks as before.

**Owner, 2026-11-19,** about the lab's fluid models: "I want to make sure that we maintain the separation where our data sets that we currently have continue to be used for the models that we currently have. And then from this point onwards, we will use only the new version of Rheon with new models. That way we don't poison our data."

## 1. Decisions

- **A label, not a kind.** A generation is a name (`rheon-reference-v1`, `rheon-v2`), 1 to 64 characters of `a-z0-9._-`, written on the definitions that bring data in. There is no list of generations to declare first; the names in use are the project's generations. `mixed` is not a generation name (it is the word of the escape below).
- **Declared where data enters, inherited downstream.** A dataset or a step may declare `generation`. A dataset or step that declares none takes the generation of its data inputs (the entries of `run.inputs` that name a dataset or a step), whatever made it: `freeze`, `select`, `embed`, `annotate`, `import` with dataset inputs, or the project's own command. Inputs without a generation do not decide it (they are unlabelled, not another generation), but a dataset, step, recipe or benchmark that has a generation and takes unlabelled data is warned (section 6).
- **Data never mixes silently.** A dataset or step whose labelled inputs have more than one generation is an error naming each input and its generation; declaring a generation does not settle it, because a declared generation that differs from an input's is an error too. There is no escape for data: a cross-generation dataset is made from two that are each one generation, under a recipe that says it mixes.
- **A recipe's generation is computed, never written.** It is the generation of the datasets and steps it trains on (its data inputs). A recipe whose data inputs have more than one generation is an error ("mixes data generations …") unless it declares `"generations": "mixed"`, a deliberate cross-generation experiment, which is then a warning, and its generation is `mixed`. A recipe derived from a parent of another generation is allowed (fine-tuning across generations) and shown on the Map.
- **A benchmark says what it measures.** Its subjects may span generations only when it declares `"generations": "mixed"`; otherwise a warning lists the subjects' generations (a warning, so that projects that label their data later still check). It may declare `"generation"` for its cases; its cases' data (`cases.dataset` and the data inputs of its run) is checked against that as a dataset's inputs are, and without a declaration its generation is that data's.
- **Unlabelled stays unlabelled.** Nothing without a generation and without labelled inputs gets one, and nothing is reported about it. The answers of projects without generations are what they were: the reads add `generation` only where there is one.

## 2. Fields

| Kind | Field | Rule |
| --- | --- | --- |
| `dataset`, `step` | `generation` | optional; the name rule above. Its data inputs' generations must equal it |
| `recipe` | `generations` | optional; only `"mixed"` |
| `benchmark` | `generation` | optional; the name rule. Its cases' data must equal it |
| `benchmark` | `generations` | optional; only `"mixed"`: its subjects may span generations |

The schemas (dataset, step, recipe, benchmark) hold the syntax; `tuldok_project/generations.py` the rules across files, called at the end of `check.check_map`, so the folder's check and the working copy's `set_definition` both report them. A cycle of inputs is the cycle rule's problem; the generation walk stops at it.

## 3. Reads

- `map`: a model has `generation` (its computed generation, `mixed` included) when it has one.
- `definition`: a dataset, step, recipe or benchmark has `generation` when it has one (computed as above).
- `definitions`: a row of a dataset, step, recipe or benchmark has `generation` when it has one.

## 4. The page

- **The Map:** a model's generation is a small chip after its name in the list and in its drawer, and a line of its tooltip. A model whose parent has another generation shows it (the drawer: "from a model of `<generation>`"; the chip's title). A filter group **Generation** in the Map's bar, beside the kinds of change, when any model has one: one chip per generation and one for models without; pressing them filters as the kinds do (faint on the lane map, hidden in the list).
- **The Data tab:** a dataset's card and drawer show its generation as the same chip.
- Everything is drawn where it already is; no node of a live block is rebuilt on a reading for it.

## 5. Tests

`tests/test_tuldok_project_generations.py`: inheritance through steps and datasets of several levels; inputs that disagree, and a declaration that differs; a recipe of one generation, of two (the error), of two declared mixed (the warning, `mixed`); a cross-generation parent (no problem); a benchmark whose subjects span generations (the warning), declared mixed (none), its cases against its declaration; the reserved name and the pattern through the schema; an unlabelled project unchanged (no problem, no `generation` in the reads); the reads of the working copy. `tests/browser_project_workspace_generations.cjs`: the chip in the Map's list and drawer and on the Data tab, and the Generation filter. Regressions: the project suite, the agent tests, `browser_project_workspace.cjs`, the lab's project checked with 0 errors.

## As built

Built on `feat/data-generations` from `main` d8a57a1d.

- **`tuldok_project/generations.py`** (new) holds the rule: `resolve(documents)` (pure, {reference: document}) gives every generation and every problem; `check_generations` is called at the end of `check.check_map`, so the folder's check and `set_definition` report the same; `labelled` feeds the reads. Datasets and steps are walked once each, a cycle of inputs stops the walk.
- **Schemas:** `generation` on dataset, step and benchmark (`^(?!mixed$)[a-z0-9._-]{1,64}$`), `generations` (`const "mixed"`) on recipe and benchmark. Branch from here and a sweep's trial carry a recipe's `generations` with its run (`workspace.draft_branch`, `sweeps.trial_document`).
- **Reads:** `map` (a model's `generation`), `definition` and `definitions` (a dataset's, step's, recipe's or benchmark's), each only where there is one, so the answers of unlabelled projects are byte for byte as before (the workspace tests compare whole answers). `definitions` and `definition` read every document once more for it; `map` adds no query. The operation descriptions and the catalog digest are unchanged.
- **Found while building:** a dataset declaring a generation while its input step has another is an error, as section 1 says; the browser check's second dataset therefore takes its own solver step. `mixed` counts as one generation in a benchmark's subjects, so a benchmark measuring a mixed recipe beside a one-generation recipe warns too.
- **The page:** `tree.js` (`genChip`, the filter in `visibleFor`), `screens.js` (the Generation group in the Map's bar, the chip and "from a model of …" in the drawer), `app.js` (`state.generations`, `toggleGeneration`), `adapter.js` (a model's `generation`), `tabs.js` (the Data tab's card and drawer), `base.css` (`.gen`, dashed `.gen.crosses`). In the list, a name with a chip is cut before the chip; below 600 px wide the list leaves the chip out so the name has the room (the drawer and the filter keep it). The filter is not kept in the address (the kinds are not either).
- **Tests:** `tests/test_tuldok_project_generations.py`, 18 (15 without a database, 3 with). `tests/browser_project_workspace_generations.cjs`, 7 PASS (`--shots <dir>` writes screenshots).

### Runs at the end

Disposable PostgreSQL under `/tmp/claude-1000/gen`, no GPU: the project suite 916 OK (898 before, plus 18); `test_agent_api test_agent_negotiation test_agent_http` 43 OK, 4 skipped; `browser_project_workspace_generations.cjs` 7 PASS; `browser_project_workspace.cjs` 24 PASS; `tuldok_project check` of the lab's `projects/surrogate-lab` 0 errors, 8 warnings (none about generations; the lab writes no generation yet).

### Not done

The lab's definitions are not labelled (that is the owner's edit: a `generation` on its reference solver step or its datasets, `rheon-reference-v1`, and `rheon-v2` on the new ones). No filter or chip on the Train, Benchmarks or Compare tabs; no generation in the plan view or in a model card's text; the editor shows the problems as for any rule but has no control of its own for the field.

## 6. Unlabelled data taken by labelled definitions (added after the first build)

The gap named in "Not done" above: an unlabelled dataset could be mixed into labelled data unnoticed. Now a dataset, step, recipe or benchmark that has a generation (declared, inherited, or `mixed` for a recipe declared so, which still gets it) and takes a data input (a dataset or step, through `run.inputs`, or a benchmark's `cases.dataset`) without one is a **warning** at that input: "Takes unlabelled data <reference>; label it with the generation it comes from." Each input is named once. An input that has no generation because its own inputs disagree is not named (its error says why). The unlabelled input still does not decide a generation, and nothing is reported in a project that labels nothing.

**As built:** `generations.resolve` warns after a definition's generation is known (`unlabelled`), keeping the datasets and steps left without one by a conflict apart. Tests: 5 more in `tests/test_tuldok_project_generations.py` (23 in all: datasets, steps and declared datasets warned; a recipe, a mixed recipe and two benchmarks warned; no warning for a conflicted input; none in an unlabelled project; the fixture's dataset labelled alone warns of its solver), and the folder and working-copy tests now label the solver steps rather than the datasets. Runs: the project suite 921 OK (916 plus 5); `browser_project_workspace_generations.cjs` 7 PASS; the lab's `projects/surrogate-lab` 0 errors, 8 warnings, none about generations (it labels nothing).
3 changes: 2 additions & 1 deletion static/project-workspace/adapter.js
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ window.TuldokProject = (() => {
group: groups.has(m.group) ? m.group : null, kind: kinds.has(m.change) ? m.change : UNSET,
parent: m.derived_from || null, trainedAt: m.trained_at, state: m.state, adopted: m.adopted, card: m.card,
value: m.value, count: m.count, benchmarks: m.benchmarks, unsaved: !!m.unsaved,
generation: m.generation || null, // its data generation, when it has one (spec-generations.md)
// Never built: drawn hollow and dashed (spec-w3 section 3.2).
planned: m.state === 'not-built' && !m.adopted && !m.trained_at
}));
Expand All @@ -186,7 +187,7 @@ window.TuldokProject = (() => {
id: d.reference, title: d.title, short: d.short || null, description: d.description || '',
group: groups.has(d.group) ? d.group : null, kind: kinds.has(d.change) ? d.change : UNSET,
parent: d.parent, trainedAt: null, state: 'draft', adopted: false, card: null,
value: null, count: null, benchmarks: 0, unsaved: false, planned: true, draft: true
value: null, count: null, benchmarks: 0, unsaved: false, planned: true, draft: true, generation: null
});
const byId = new Map(models.map(m => [m.id, m]));
for (const m of models) if (m.parent && !byId.has(m.parent)) m.parent = null;
Expand Down
4 changes: 3 additions & 1 deletion static/project-workspace/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
const POLL_MS = 2000, IDLE_MS = 15000; // how often the page asks again while something builds, and while nothing does
const ACTIVE_RUN = ['queued', 'running'], ACTIVE_REQUEST = ['queued', 'running'];
const $ = id => document.getElementById(id);
const state = { screen: 'map', sel: {}, kinds: null, query: '', zoom: 1, track: false, ui: {}, benchmark: null, cmp: [] };
const state = { screen: 'map', sel: {}, kinds: null, generations: null, query: '', zoom: 1, track: false, ui: {}, benchmark: null, cmp: [] };
const data = { summary: null, map: null, standing: null, needs: null, status: null, readAt: '', polling: false };
let failed = null, conflict = null, pollTimer = null, pollFast = false, states = null, opener = null;
const clock = () => new Date().toLocaleTimeString('en-GB', { hour: '2-digit', minute: '2-digit' });
Expand All @@ -48,6 +48,8 @@
zoom: d => { state.zoom = Math.round(Math.min(1.6, Math.max(0.5, (state.zoom || 1) + d)) * 10) / 10; render(); },
toggleKind: (k, all) => { if (!state.kinds) state.kinds = new Set([k]); else if (state.kinds.has(k)) { state.kinds.delete(k); if (!state.kinds.size) state.kinds = null; } else { state.kinds.add(k); if (state.kinds.size === all.length) state.kinds = null; } render(); },
toggleTrack: () => { state.track = !state.track; render(); },
// The Map's Generation filter (spec-generations.md): as the kinds, '' for the models without one.
toggleGeneration: (g, all) => { if (!state.generations) state.generations = new Set([g]); else if (state.generations.has(g)) { state.generations.delete(g); if (!state.generations.size) state.generations = null; } else { state.generations.add(g); if (state.generations.size === all.length) state.generations = null; } render(); },
// The other tabs: their own settings (state.ui), and the benchmark and compare set, which the address keeps (?benchmark=, ?cmp=).
setUI: (k, v) => { state.ui[k] = v; if (ADDRESSED.includes(k)) address(); render(); },
/** Keeps a setting of the tab that a view has already drawn in place (fields.js's slider): into the address, and nothing drawn again. */
Expand Down
Loading
Loading