diff --git a/docs/contracts/project-folder.md b/docs/contracts/project-folder.md index b6e0551d..0f5461c6 100644 --- a/docs/contracts/project-folder.md +++ b/docs/contracts/project-folder.md @@ -238,6 +238,7 @@ examples, except `environments/cuda-torch`, a second environment not shown. | `run` | unless `earlier_work.rebuild` is `no` | see the `run` block | | `splits` | no | object, split name to a number of items (integer ≥ 0) | | `view` | no | how its product is shown as records; see [Looking inside a product](#looking-inside-a-product) | +| `generation` | no | the data generation its data comes from, 1 to 64 characters of `a-z`, `0-9`, `.`, `_` and `-`, not `mixed`; without it, its inputs' (see [Data generations](#data-generations)) | A dataset's run may name a `step` instead of a `command`: see [Steps Tuldok provides](#steps-tuldok-provides). @@ -575,6 +576,7 @@ A recipe trains a model; its product is weights. | `compared_with` | no | at most 50 objects, each `recipe` (reference to another recipe, each once in the list), `explanation` (1 to 2,000 characters) and `excerpt` (optional, at most 2,000 characters); no other field | | `serve` | no | how Tuldok serves the model: `module_file` (a code path of a `.py` file, required), `function` (default `predict`), `load`, `backend` (`local`, the default and only one), `environment` (default `run.environment`; one of the two is required), `idle_minutes` (1 to 1,440), `start_seconds` (1 to 3,600), `request_seconds` (0.1 to 600), `gpu`, `example` (an object); see [Serving a model](#serving-a-model). Not an input: it never makes the model out of date | | `sweep` | no | a trial of a sweep (see [`sweep`](#sweep)): `definition` (reference to `sweeps/`), `trial` (its number, 1 or more) and `values` (the settings it was given, by path, at most 16); written by Tuldok when it runs the sweep | +| `generations` | no | only `"mixed"`: it trains on data of more than one generation on purpose (see [Data generations](#data-generations)); a branch and a sweep's trial carry it | The label `short`, when a recipe has it, is the text of its node on the map (at most 16 characters are shown). `derived_from` and `compared_with` say how @@ -613,6 +615,7 @@ solver. | Field | Required | Rule | | --- | --- | --- | | `run` | unless `earlier_work.rebuild` is `no` | see the `run` block | +| `generation` | no | the data generation of what it makes (a solver, a source), 1 to 64 characters of `a-z`, `0-9`, `.`, `_` and `-`, not `mixed`; without it, its inputs' (see [Data generations](#data-generations)) | Example, `steps/reference-solver.json`: @@ -643,6 +646,8 @@ Its command runs once per subject. | `subjects` | yes | 0 to 1,000 objects, see below | | `cases` | no | `{"dataset"?: reference to datasets/, "split"?: non-empty string, "count"?: integer ≥ 0}` | | `summary` | no | `{"group_by"?: up to 8 distinct ids, "statistics"?: distinct names from mean, median, p10, p90, min, max}` | +| `generation` | no | the data generation its cases come from, 1 to 64 characters of `a-z`, `0-9`, `.`, `_` and `-`, not `mixed`; without it, its cases' data's (see [Data generations](#data-generations)) | +| `generations` | no | only `"mixed"`: its subjects may come from different data generations | A metric is `{"id", "title", "units"?, "better"}`: `id` matches `^[a-z][a-z0-9_]{0,63}$` and is unique in the benchmark; `title` is 1 to 200 @@ -1000,6 +1005,47 @@ pilot-dense-k1,velocity_error_final,cells_per_metre=8,4,0,0.195,0.175,0.129,0.27 group and no group is its own ancestor; a recipe is never derived from or compared with itself; following `derived_from` never comes back to where it started; `map.chart` is a chart of the form `progression`. +8. Data generations do not mix unless a recipe or benchmark says so; see + [Data generations](#data-generations). + +### Data generations + +A **data generation** names the source a project's data comes from: the +solver, simulator, sensor or corpus release that made it (`rheon-reference-v1`, +`rheon-v2`). It keeps data of one generation for the models of that generation, +and says so when they cross. The label is opt-in: a folder that writes none +checks as before, and nothing without a generation is reported. + +- A dataset or step may declare `generation`. One that declares none takes the + generation of its **data inputs**: the entries of `run.inputs` that name a + dataset or a step, whatever made it (`freeze`, `select`, `embed`, `annotate`, + `import` with dataset inputs, or the project's own command). Inputs without a + generation do not decide it; a dataset or step none of whose inputs has one has + none. +- **Warning:** a dataset, step, recipe or benchmark that has a generation + (declared, inherited or `mixed`) and takes a data input that has none ("takes + unlabelled data ; label it with the generation it comes from"). An + input without a generation because its own inputs disagree is not named (it + has its error). A project that labels nothing gets no such warning. +- **Error:** a dataset or step whose data inputs have more than one generation + (the message names each input and its generation), and one whose declared + generation differs from an input's. Data has no escape: declaring one does + not settle inputs that disagree. +- A recipe's generation is computed: that of its data inputs. **Error:** a + recipe whose data inputs have more than one generation ("mixes data + generations"), unless it declares `"generations": "mixed"`, a deliberate + cross-generation experiment, which is then a **warning**; its generation is + then `mixed`. A recipe input (a model to start from) is not data: a recipe + derived from, or fine-tuned from, a model of another generation is fine, and + the Map shows it. +- A benchmark's generation is its declared `generation`, else that of its cases' + data (`cases.dataset` and the data inputs of its run). Its cases' data is + checked against the declaration as a dataset's inputs are (**error**). + **Warning:** its subjects' recipes have more than one generation (`mixed` + counts as one), unless it declares `"generations": "mixed"`. +- The reads of the database copy add `generation` where there is one: to a + model of `map`, to a dataset, step, recipe or benchmark of `definition`, and + to its row of `definitions`. ## Checking a folder diff --git a/docs/plans/training-and-curation/spec-generations.md b/docs/plans/training-and-curation/spec-generations.md new file mode 100644 index 00000000..d3ba2cfc --- /dev/null +++ b/docs/plans/training-and-curation/spec-generations.md @@ -0,0 +1,66 @@ +# Implementation spec: data generations + +A small feature beside [the training and curation plan](plan.md). A project can say which **generation** of its source its data comes from (the solver, simulator, sensor or corpus release that made it), and Tuldok keeps the generations apart: a model trains on one generation's data, a dataset made from others keeps their generation, and mixing them is an error unless the definition says it is meant. The label is general, not of one project; it is opt-in, so a project that writes none checks as before. + +**Owner, 2026-11-19,** about the lab's fluid models: "I want to make sure that we maintain the separation where our data sets that we currently have continue to be used for the models that we currently have. And then from this point onwards, we will use only the new version of Rheon with new models. That way we don't poison our data." + +## 1. Decisions + +- **A label, not a kind.** A generation is a name (`rheon-reference-v1`, `rheon-v2`), 1 to 64 characters of `a-z0-9._-`, written on the definitions that bring data in. There is no list of generations to declare first; the names in use are the project's generations. `mixed` is not a generation name (it is the word of the escape below). +- **Declared where data enters, inherited downstream.** A dataset or a step may declare `generation`. A dataset or step that declares none takes the generation of its data inputs (the entries of `run.inputs` that name a dataset or a step), whatever made it: `freeze`, `select`, `embed`, `annotate`, `import` with dataset inputs, or the project's own command. Inputs without a generation do not decide it (they are unlabelled, not another generation), but a dataset, step, recipe or benchmark that has a generation and takes unlabelled data is warned (section 6). +- **Data never mixes silently.** A dataset or step whose labelled inputs have more than one generation is an error naming each input and its generation; declaring a generation does not settle it, because a declared generation that differs from an input's is an error too. There is no escape for data: a cross-generation dataset is made from two that are each one generation, under a recipe that says it mixes. +- **A recipe's generation is computed, never written.** It is the generation of the datasets and steps it trains on (its data inputs). A recipe whose data inputs have more than one generation is an error ("mixes data generations …") unless it declares `"generations": "mixed"`, a deliberate cross-generation experiment, which is then a warning, and its generation is `mixed`. A recipe derived from a parent of another generation is allowed (fine-tuning across generations) and shown on the Map. +- **A benchmark says what it measures.** Its subjects may span generations only when it declares `"generations": "mixed"`; otherwise a warning lists the subjects' generations (a warning, so that projects that label their data later still check). It may declare `"generation"` for its cases; its cases' data (`cases.dataset` and the data inputs of its run) is checked against that as a dataset's inputs are, and without a declaration its generation is that data's. +- **Unlabelled stays unlabelled.** Nothing without a generation and without labelled inputs gets one, and nothing is reported about it. The answers of projects without generations are what they were: the reads add `generation` only where there is one. + +## 2. Fields + +| Kind | Field | Rule | +| --- | --- | --- | +| `dataset`, `step` | `generation` | optional; the name rule above. Its data inputs' generations must equal it | +| `recipe` | `generations` | optional; only `"mixed"` | +| `benchmark` | `generation` | optional; the name rule. Its cases' data must equal it | +| `benchmark` | `generations` | optional; only `"mixed"`: its subjects may span generations | + +The schemas (dataset, step, recipe, benchmark) hold the syntax; `tuldok_project/generations.py` the rules across files, called at the end of `check.check_map`, so the folder's check and the working copy's `set_definition` both report them. A cycle of inputs is the cycle rule's problem; the generation walk stops at it. + +## 3. Reads + +- `map`: a model has `generation` (its computed generation, `mixed` included) when it has one. +- `definition`: a dataset, step, recipe or benchmark has `generation` when it has one (computed as above). +- `definitions`: a row of a dataset, step, recipe or benchmark has `generation` when it has one. + +## 4. The page + +- **The Map:** a model's generation is a small chip after its name in the list and in its drawer, and a line of its tooltip. A model whose parent has another generation shows it (the drawer: "from a model of ``"; the chip's title). A filter group **Generation** in the Map's bar, beside the kinds of change, when any model has one: one chip per generation and one for models without; pressing them filters as the kinds do (faint on the lane map, hidden in the list). +- **The Data tab:** a dataset's card and drawer show its generation as the same chip. +- Everything is drawn where it already is; no node of a live block is rebuilt on a reading for it. + +## 5. Tests + +`tests/test_tuldok_project_generations.py`: inheritance through steps and datasets of several levels; inputs that disagree, and a declaration that differs; a recipe of one generation, of two (the error), of two declared mixed (the warning, `mixed`); a cross-generation parent (no problem); a benchmark whose subjects span generations (the warning), declared mixed (none), its cases against its declaration; the reserved name and the pattern through the schema; an unlabelled project unchanged (no problem, no `generation` in the reads); the reads of the working copy. `tests/browser_project_workspace_generations.cjs`: the chip in the Map's list and drawer and on the Data tab, and the Generation filter. Regressions: the project suite, the agent tests, `browser_project_workspace.cjs`, the lab's project checked with 0 errors. + +## As built + +Built on `feat/data-generations` from `main` d8a57a1d. + +- **`tuldok_project/generations.py`** (new) holds the rule: `resolve(documents)` (pure, {reference: document}) gives every generation and every problem; `check_generations` is called at the end of `check.check_map`, so the folder's check and `set_definition` report the same; `labelled` feeds the reads. Datasets and steps are walked once each, a cycle of inputs stops the walk. +- **Schemas:** `generation` on dataset, step and benchmark (`^(?!mixed$)[a-z0-9._-]{1,64}$`), `generations` (`const "mixed"`) on recipe and benchmark. Branch from here and a sweep's trial carry a recipe's `generations` with its run (`workspace.draft_branch`, `sweeps.trial_document`). +- **Reads:** `map` (a model's `generation`), `definition` and `definitions` (a dataset's, step's, recipe's or benchmark's), each only where there is one, so the answers of unlabelled projects are byte for byte as before (the workspace tests compare whole answers). `definitions` and `definition` read every document once more for it; `map` adds no query. The operation descriptions and the catalog digest are unchanged. +- **Found while building:** a dataset declaring a generation while its input step has another is an error, as section 1 says; the browser check's second dataset therefore takes its own solver step. `mixed` counts as one generation in a benchmark's subjects, so a benchmark measuring a mixed recipe beside a one-generation recipe warns too. +- **The page:** `tree.js` (`genChip`, the filter in `visibleFor`), `screens.js` (the Generation group in the Map's bar, the chip and "from a model of …" in the drawer), `app.js` (`state.generations`, `toggleGeneration`), `adapter.js` (a model's `generation`), `tabs.js` (the Data tab's card and drawer), `base.css` (`.gen`, dashed `.gen.crosses`). In the list, a name with a chip is cut before the chip; below 600 px wide the list leaves the chip out so the name has the room (the drawer and the filter keep it). The filter is not kept in the address (the kinds are not either). +- **Tests:** `tests/test_tuldok_project_generations.py`, 18 (15 without a database, 3 with). `tests/browser_project_workspace_generations.cjs`, 7 PASS (`--shots ` writes screenshots). + +### Runs at the end + +Disposable PostgreSQL under `/tmp/claude-1000/gen`, no GPU: the project suite 916 OK (898 before, plus 18); `test_agent_api test_agent_negotiation test_agent_http` 43 OK, 4 skipped; `browser_project_workspace_generations.cjs` 7 PASS; `browser_project_workspace.cjs` 24 PASS; `tuldok_project check` of the lab's `projects/surrogate-lab` 0 errors, 8 warnings (none about generations; the lab writes no generation yet). + +### Not done + +The lab's definitions are not labelled (that is the owner's edit: a `generation` on its reference solver step or its datasets, `rheon-reference-v1`, and `rheon-v2` on the new ones). No filter or chip on the Train, Benchmarks or Compare tabs; no generation in the plan view or in a model card's text; the editor shows the problems as for any rule but has no control of its own for the field. + +## 6. Unlabelled data taken by labelled definitions (added after the first build) + +The gap named in "Not done" above: an unlabelled dataset could be mixed into labelled data unnoticed. Now a dataset, step, recipe or benchmark that has a generation (declared, inherited, or `mixed` for a recipe declared so, which still gets it) and takes a data input (a dataset or step, through `run.inputs`, or a benchmark's `cases.dataset`) without one is a **warning** at that input: "Takes unlabelled data ; label it with the generation it comes from." Each input is named once. An input that has no generation because its own inputs disagree is not named (its error says why). The unlabelled input still does not decide a generation, and nothing is reported in a project that labels nothing. + +**As built:** `generations.resolve` warns after a definition's generation is known (`unlabelled`), keeping the datasets and steps left without one by a conflict apart. Tests: 5 more in `tests/test_tuldok_project_generations.py` (23 in all: datasets, steps and declared datasets warned; a recipe, a mixed recipe and two benchmarks warned; no warning for a conflicted input; none in an unlabelled project; the fixture's dataset labelled alone warns of its solver), and the folder and working-copy tests now label the solver steps rather than the datasets. Runs: the project suite 921 OK (916 plus 5); `browser_project_workspace_generations.cjs` 7 PASS; the lab's `projects/surrogate-lab` 0 errors, 8 warnings, none about generations (it labels nothing). diff --git a/static/project-workspace/adapter.js b/static/project-workspace/adapter.js index c101f813..ea81ac17 100644 --- a/static/project-workspace/adapter.js +++ b/static/project-workspace/adapter.js @@ -178,6 +178,7 @@ window.TuldokProject = (() => { group: groups.has(m.group) ? m.group : null, kind: kinds.has(m.change) ? m.change : UNSET, parent: m.derived_from || null, trainedAt: m.trained_at, state: m.state, adopted: m.adopted, card: m.card, value: m.value, count: m.count, benchmarks: m.benchmarks, unsaved: !!m.unsaved, + generation: m.generation || null, // its data generation, when it has one (spec-generations.md) // Never built: drawn hollow and dashed (spec-w3 section 3.2). planned: m.state === 'not-built' && !m.adopted && !m.trained_at })); @@ -186,7 +187,7 @@ window.TuldokProject = (() => { id: d.reference, title: d.title, short: d.short || null, description: d.description || '', group: groups.has(d.group) ? d.group : null, kind: kinds.has(d.change) ? d.change : UNSET, parent: d.parent, trainedAt: null, state: 'draft', adopted: false, card: null, - value: null, count: null, benchmarks: 0, unsaved: false, planned: true, draft: true + value: null, count: null, benchmarks: 0, unsaved: false, planned: true, draft: true, generation: null }); const byId = new Map(models.map(m => [m.id, m])); for (const m of models) if (m.parent && !byId.has(m.parent)) m.parent = null; diff --git a/static/project-workspace/app.js b/static/project-workspace/app.js index 044658dd..4e85d4da 100644 --- a/static/project-workspace/app.js +++ b/static/project-workspace/app.js @@ -23,7 +23,7 @@ const POLL_MS = 2000, IDLE_MS = 15000; // how often the page asks again while something builds, and while nothing does const ACTIVE_RUN = ['queued', 'running'], ACTIVE_REQUEST = ['queued', 'running']; const $ = id => document.getElementById(id); - const state = { screen: 'map', sel: {}, kinds: null, query: '', zoom: 1, track: false, ui: {}, benchmark: null, cmp: [] }; + const state = { screen: 'map', sel: {}, kinds: null, generations: null, query: '', zoom: 1, track: false, ui: {}, benchmark: null, cmp: [] }; const data = { summary: null, map: null, standing: null, needs: null, status: null, readAt: '', polling: false }; let failed = null, conflict = null, pollTimer = null, pollFast = false, states = null, opener = null; const clock = () => new Date().toLocaleTimeString('en-GB', { hour: '2-digit', minute: '2-digit' }); @@ -48,6 +48,8 @@ zoom: d => { state.zoom = Math.round(Math.min(1.6, Math.max(0.5, (state.zoom || 1) + d)) * 10) / 10; render(); }, toggleKind: (k, all) => { if (!state.kinds) state.kinds = new Set([k]); else if (state.kinds.has(k)) { state.kinds.delete(k); if (!state.kinds.size) state.kinds = null; } else { state.kinds.add(k); if (state.kinds.size === all.length) state.kinds = null; } render(); }, toggleTrack: () => { state.track = !state.track; render(); }, + // The Map's Generation filter (spec-generations.md): as the kinds, '' for the models without one. + toggleGeneration: (g, all) => { if (!state.generations) state.generations = new Set([g]); else if (state.generations.has(g)) { state.generations.delete(g); if (!state.generations.size) state.generations = null; } else { state.generations.add(g); if (state.generations.size === all.length) state.generations = null; } render(); }, // The other tabs: their own settings (state.ui), and the benchmark and compare set, which the address keeps (?benchmark=, ?cmp=). setUI: (k, v) => { state.ui[k] = v; if (ADDRESSED.includes(k)) address(); render(); }, /** Keeps a setting of the tab that a view has already drawn in place (fields.js's slider): into the address, and nothing drawn again. */ diff --git a/static/project-workspace/base.css b/static/project-workspace/base.css index 050a810d..a0f8382f 100644 --- a/static/project-workspace/base.css +++ b/static/project-workspace/base.css @@ -613,6 +613,15 @@ textarea { font: inherit; resize: vertical; } /* The unsaved mark, and "not kept yet" where something typed is not kept. */ .umark { font-size: 11.5px; color: var(--warn); font-weight: 500; white-space: nowrap; } .umark:empty { display: none; } +/* A data generation (spec-generations.md): a small outlined chip; dashed when the model's parent has another. */ +.gen { display: inline-block; vertical-align: 1px; font: 11px/16px var(--mono); padding: 0 6px; border: 1px solid var(--line-strong); border-radius: 999px; color: var(--ink-2); white-space: nowrap; max-width: 18ch; overflow: hidden; text-overflow: ellipsis; } +.gen.crosses { border-style: dashed; } +.tree-list .trow .nm.has-gen { display: flex; align-items: center; min-width: 0; } +.tree-list .trow .nm.has-gen .nmt { min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.tree-list .trow .nm.has-gen > .umark, .tree-list .trow .nm .gen { flex: none; } +.tree-list .trow .nm .gen { margin-left: 8px; } +@media (max-width: 600px) { .tree-list .trow .nm .gen { display: none; } } /* narrow: the name first; the drawer shows the chip */ +.btn.gen-f { font-family: var(--mono); font-size: 11.5px; } .umark.typed { color: var(--accent); } .tree-list .trow .nm .umark, .row .title .umark { margin-left: 8px; } /* The editor. */ diff --git a/static/project-workspace/screens.js b/static/project-workspace/screens.js index 3355683f..ab275048 100644 --- a/static/project-workspace/screens.js +++ b/static/project-workspace/screens.js @@ -88,22 +88,30 @@ window.LabScreens = (() => { const focus = planSel ? planSel.slice(5).split(',')[0] : E ? E.subject(raw) : raw, sel = focus && g.byId.has(focus) ? focus : null; const kinds = P.KINDS.map(k => k.id).concat(g.nodes.some(m => m.kind === P.UNSET) ? [P.UNSET] : []); const withValue = g.nodes.filter(m => m.value != null).length; + const named = [...new Set(g.nodes.map(m => m.generation).filter(Boolean))].sort(); + const gens = named.length ? named.concat(g.nodes.some(m => !m.generation) ? [''] : []) : []; const buildBar = () => { const bar = h('div', 'filters'); const q = h('input'); q.type = 'search'; q.placeholder = 'Find a model'; q.value = state.query || ''; q.setAttribute('aria-label', 'Find a model'); q.addEventListener('input', () => act.setQuery(q.value)); bar.appendChild(q); const grp = h('div', 'grp'); grp.setAttribute('role', 'group'); grp.setAttribute('aria-label', 'Kind of change'); for (const k of kinds) { const b = h('button', 'btn sm chip ' + k, P.kindLabel(k)); b.type = 'button'; b.dataset.kind = k; b.setAttribute('aria-pressed', String(!state.kinds || state.kinds.has(k))); b.addEventListener('click', () => act.toggleKind(k, kinds)); grp.appendChild(b); } bar.appendChild(grp); + // Generation (spec-generations.md): one chip per data generation and one for the models without, when any has one. + if (gens.length) { + const gg = h('div', 'grp'); gg.setAttribute('role', 'group'); gg.setAttribute('aria-label', 'Data generation'); + for (const k of gens) { const b = h('button', 'btn sm gen-f', k || 'no generation'); b.type = 'button'; b.dataset.generation = k; b.dataset.key = 'gen-' + k; b.setAttribute('aria-pressed', String(!state.generations || state.generations.has(k))); b.addEventListener('click', () => act.toggleGeneration(k, gens)); gg.appendChild(b); } + bar.appendChild(gg); + } if (g.track) { const tb = h('button', 'btn sm', 'Accuracy track'); tb.type = 'button'; tb.title = P.trackName(g.track) + ', ' + P.better(g.track); tb.setAttribute('aria-pressed', String(!!state.track)); tb.addEventListener('click', () => act.toggleTrack()); bar.appendChild(tb); } const z = h('div', 'zoomctl'); for (const [t, d, l] of [['−', -0.2, 'Zoom out'], ['+', 0.2, 'Zoom in']]) { const b = h('button', 'btn sm', t); b.type = 'button'; b.setAttribute('aria-label', l); b.addEventListener('click', () => act.zoom(d)); z.appendChild(b); } bar.appendChild(z); bar.appendChild(h('span', 'pop', g.nodes.length + ' models · ' + (g.track ? withValue + ' on the track' : 'no track'))); return bar; }; if (!g.nodes.length) { main.textContent = ''; main._tree = null; main.appendChild(h('p', 'empty', 'This project has no recipes, so there are no models to map.')); aside.textContent = ''; return false; } - const sig = JSON.stringify([state.kinds ? [...state.kinds] : null, state.query || '', state.zoom || 1, !!state.track]); + const sig = JSON.stringify([state.kinds ? [...state.kinds] : null, state.query || '', state.zoom || 1, !!state.track, state.generations ? [...state.generations] : null]); const prev = main._tree; // While a parent is being picked (editor.js), the next model pressed is the parent instead. - const common = { onSelect: id => { if (!(E && E.picked(id, act))) act.select(id); }, kinds: state.kinds, query: state.query }; + const common = { onSelect: id => { if (!(E && E.picked(id, act))) act.select(id); }, kinds: state.kinds, query: state.query, generations: state.generations }; if (prev && prev.sig === sig && prev.shaped === shaped && main.contains(prev.bar)) { // The same tree: only a new selection is drawn (spec-w4 R2, R6). if (sel !== prev.sel) { if (sel) { prev.map.scrollTo(sel, true); prev.list.scrollTo(sel, true); } prev.map.select(sel); prev.list.select(sel); prev.sel = sel; } @@ -159,6 +167,7 @@ window.LabScreens = (() => { const meta = h('div', 'ins-meta'); meta.appendChild(h('span', 'chip ' + m.kind, P.kindLabel(m.kind))); meta.appendChild(modelState(m.state)); meta.appendChild(h('span', 'small faint', m.trainedAt ? 'trained ' + P.day(m.trainedAt) : m.planned ? 'never built' : 'no training date')); meta.appendChild(Tree.mark(m)); + const gc = Tree.genChip(g, m); if (gc) meta.appendChild(gc); hd.appendChild(meta); ins.appendChild(hd); if (E) E.decorate(hd); // Plan build, for a model to build (spec-w3 section 3.1): the plan view, never a request from here. A model @@ -175,6 +184,8 @@ window.LabScreens = (() => { const lin = h('div', 'actions'); const jump = (target, text) => { const b = h('button', 'btn sm ghost'); b.type = 'button'; b.appendChild(icon('branch')); b.appendChild(document.createTextNode(' ' + text)); b.addEventListener('click', () => act.select(target.id)); return b; }; if (m.parent) lin.appendChild(jump(g.byId.get(m.parent), 'From ' + Tree.refName(g.byId.get(m.parent)))); + const pg = m.parent && g.byId.get(m.parent).generation; + if (pg && m.generation && pg !== m.generation) lin.appendChild(h('span', 'small faint gen-from', 'from a model of ' + pg)); else lin.appendChild(h('span', 'small faint', 'No recorded ancestry')); const kids = g.nodes.filter(n => n.parent === m.id); if (kids.length === 1) lin.appendChild(jump(kids[0], Tree.refName(kids[0]) + ' derives from it')); diff --git a/static/project-workspace/tabs.js b/static/project-workspace/tabs.js index 8a249987..31eeebae 100644 --- a/static/project-workspace/tabs.js +++ b/static/project-workspace/tabs.js @@ -703,6 +703,7 @@ card.appendChild(pv); const t = h('div', 't'); t.appendChild(h('div', 'title', d.title)); const meta = h('div', 'meta'); meta.appendChild(h('span', null, outputs.length ? plural(outputs.length, 'output') + ' · ' + bytes(size) : whole ? 'a directory · ' + bytes(size) : 'no files here')); + if (d.generation) { const gc = h('span', 'gen', d.generation); gc.dataset.generation = d.generation; gc.title = 'Data generation ' + d.generation; meta.appendChild(gc); } // spec-generations.md const st = ps.some(p => p.adopted) ? S.stateEl('good', 'adopted') : modelState(d.state); st.classList.add('glyphs'); meta.appendChild(st); t.appendChild(meta); const used = [...(users.get(d.reference) || [])].sort(); t.appendChild(h('div', 'uses', used.length ? 'used by ' + used.length + ': ' + used.map(nameOfRef).join(', ') : 'used by nothing')); @@ -723,7 +724,9 @@ if (E && E.owns(raw, data)) { E.drawer(aside, raw, act, data, 'data'); return true; } drawer(aside, sel, () => Promise.all([P.read('definition', { reference: sel }), P.read('products', { reference: sel })]), (ins, [d, p]) => { const hd = h('header'); hd.appendChild(h('span', 'kicker', d.reference)); const t2 = h('h2', null, d.title); t2.tabIndex = -1; hd.appendChild(t2); - const meta = h('div', 'ins-meta'); meta.appendChild(p.products.some(x => x.adopted) ? S.stateEl('good', 'adopted') : modelState(d.state)); meta.appendChild(Tree.mark({ id: d.reference, unsaved: d.unsaved })); hd.appendChild(meta); ins.appendChild(hd); + const meta = h('div', 'ins-meta'); meta.appendChild(p.products.some(x => x.adopted) ? S.stateEl('good', 'adopted') : modelState(d.state)); meta.appendChild(Tree.mark({ id: d.reference, unsaved: d.unsaved })); + if (d.generation) { const gc = h('span', 'gen', d.generation); gc.dataset.generation = d.generation; gc.title = 'Data generation ' + d.generation; meta.appendChild(gc); } + hd.appendChild(meta); ins.appendChild(hd); // Edit (spec-w3 section 3.2): the editor, in the text view. if (p.products.some(x => x.run_id)) { const look = h('div', 'actions'); look.appendChild(lookInside(d.reference, act)); ins.appendChild(look); } const doc = d.document || {}; diff --git a/static/project-workspace/tree.js b/static/project-workspace/tree.js index a15444e8..cb1ee125 100644 --- a/static/project-workspace/tree.js +++ b/static/project-workspace/tree.js @@ -80,11 +80,20 @@ window.LabTree = (() => { if (!q) return true; q = q.toLowerCase(); return [m.title, m.short, m.description, m.groupTitle, m.id].some(s => s && s.toLowerCase().includes(q)); } - const visibleFor = (kinds, query) => m => (!kinds || kinds.has(m.kind)) && matches(m, query); + // generations: null (every model) or a Set of generation names, '' for a model without one (spec-generations.md). + const visibleFor = (kinds, query, generations) => m => (!kinds || kinds.has(m.kind)) && (!generations || generations.has(m.generation || '')) && matches(m, query); + /** The small chip of a model's data generation, dashed when its parent's is another; null without one. */ + function genChip(graph, m) { + if (!m.generation) return null; + const p = m.parent && graph.byId.get(m.parent), other = p && p.generation && p.generation !== m.generation ? p.generation : null; + const s = h('span', 'gen' + (other ? ' crosses' : ''), m.generation); s.dataset.generation = m.generation; + s.title = 'Data generation ' + m.generation + (other ? '; derived from a model of ' + other : ''); + return s; + } const trackText = (graph, m) => m.value == null ? 'no value' : C.fmtSci(m.value) + P.units(graph.track); // ---- Vertical list ------------------------------------------------------- - function renderList(container, graph, { selected = null, onSelect = () => {}, kinds = null, query = '' } = {}) { + function renderList(container, graph, { selected = null, onSelect = () => {}, kinds = null, query = '', generations = null } = {}) { container.textContent = ''; const laneW = 16, pad = 10, gw = pad * 2 + laneW * Math.max(1, graph.laneCount - 1) + 8, rowH = 30; const shortest = Math.max(3, ...graph.nodes.map(m => (shortOf(m) || '').length)); @@ -93,7 +102,7 @@ window.LabTree = (() => { const ul = h('ul', 'tree-list'); ul.setAttribute('role', 'listbox'); ul.setAttribute('aria-label', 'Models by section, in training order'); const active = t => graph.families.filter(f => f.start < t && t <= f.end); const laneX = l => pad + 4 + l * laneW; - const visible = visibleFor(kinds, query); + const visible = visibleFor(kinds, query, generations); const rows = new Map(); let shown = 0; for (const s of graph.sections) { const list = graph.nodes.slice(s.start, s.end + 1).filter(visible); @@ -124,7 +133,8 @@ window.LabTree = (() => { svg.appendChild(dot); b.appendChild(svg); b.appendChild(h('span', 'ck', shortOf(m) || '—')); // The name, with the mark of an unsaved edit (or, from editor.js, of typing not kept yet). - const nm = h('span', 'nm'); nm.appendChild(document.createTextNode(m.title)); nm.appendChild(mark(m)); b.appendChild(nm); + // With a generation, the name is cut before the chip, so that the chip stays in sight (spec-generations.md). + const nm = h('span', 'nm'), gc = genChip(graph, m); nm.appendChild(gc ? h('span', 'nmt', m.title) : document.createTextNode(m.title)); nm.appendChild(mark(m)); if (gc) { nm.classList.add('has-gen'); nm.appendChild(gc); } b.appendChild(nm); b.appendChild(h('span', 'pu', m.description)); b.appendChild(h('span', 'chip ' + m.kind + (m.draft ? ' draft' : ''), m.draft ? 'draft' : P.kindLabel(m.kind))); const bar = h('span', 'mbar' + (m.value == null ? ' none' : '')); @@ -152,7 +162,7 @@ window.LabTree = (() => { // ---- Horizontal lane map, with the track beneath when asked ---------------- const labelWidth = text => text.length * 7 + 24; // 12px semibold section titles, with room const nodeLabelWidth = text => text.length * 6.4; // 10.5px mono - function renderMap(container, graph, { selected = null, onSelect = () => {}, kinds = null, query = '', zoom = 1, track = false } = {}) { + function renderMap(container, graph, { selected = null, onSelect = () => {}, kinds = null, query = '', zoom = 1, track = false, generations = null } = {}) { container.textContent = ''; const dx = 30, left = 70, top = 44, laneH = 52, plotH = 120; // Each section is as wide as its models or its title, whichever is wider. @@ -168,7 +178,7 @@ window.LabTree = (() => { const x = m => xOf.get(m.id), y = m => top + 20 + m.lane * laneH; const inner = h('div', 'inner'); inner.style.width = W + 'px'; inner.style.height = H + 'px'; inner.style.transform = 'scale(' + zoom + ')'; const svg = el('svg', { width: W, height: H, viewBox: `0 0 ${W} ${H}`, role: 'img', 'aria-label': 'Model lineage: sections left to right in training order, branches in lanes' + (showTrack ? ', with ' + trackLabel + ' beneath' : '') }); - const visible = visibleFor(kinds, query); + const visible = visibleFor(kinds, query, generations); graph.sections.forEach((s, i) => { const x1 = sx[i], width = (sx[i + 1] || cursor) - x1; svg.appendChild(el('line', { x1, y1: 8, x2: x1, y2: H - 8, class: 'band' })); @@ -211,6 +221,7 @@ window.LabTree = (() => { rows.push(graph.track ? { v: m.value == null ? '—' : C.fmtSci(m.value) + P.units(graph.track), l: P.trackName(graph.track) } : { l: 'The project names no track' }); rows.push({ l: m.draft ? 'a draft, not kept yet' : m.trainedAt ? 'trained ' + P.day(m.trainedAt) : m.planned ? 'never built' : 'no training date' }); rows.push({ v: P.kindLabel(m.kind), l: m.parent ? 'from ' + refName(graph.byId.get(m.parent)) : '' }); + if (m.generation) rows.push({ v: m.generation, l: 'data generation' }); const [px, py] = at(ev); tip.show(px, py, rows); }; const nodeEls = new Map(); @@ -280,5 +291,5 @@ window.LabTree = (() => { }; api.select(selected); return api; } - return { build, renderList, renderMap, related, comparisons, matches, nameOf, refName, mark }; + return { build, renderList, renderMap, related, comparisons, matches, nameOf, refName, mark, genChip }; })(); diff --git a/tests/browser_project_workspace_generations.cjs b/tests/browser_project_workspace_generations.cjs new file mode 100644 index 00000000..aa81321d --- /dev/null +++ b/tests/browser_project_workspace_generations.cjs @@ -0,0 +1,182 @@ +'use strict'; +// Data generations on the page (docs/plans/training-and-curation/spec-generations.md section 4) in a headless +// browser, signed in with a credential that may only inspect. Tuldok runs with --project-folder on a copy of the +// minimal fixture whose solver step declares the generation toy-solver-1 (so datasets/waves, stepper-k1 and +// stepper-k2 inherit it), with a second solver step of toy-solver-2, a dataset made with it, and a recipe stepper-k3 +// trained on that dataset and derived from stepper-k2 (a cross-generation fine-tune). +// 1. The Map's list shows each model's generation as a chip after its name (none for early-fit), k3's dashed. +// 2. The Generation filter: a chip per generation and one for models without; pressing toy-solver-2 leaves only +// k3 in the list and makes the other nodes of the lane map faint; pressing it again shows them all. +// 3. k3's drawer shows its chip and "from a model of toy-solver-1". +// 4. The Data tab's cards show the datasets' generations. +// 5. Light and dark at 1440 and 390 wide: the chips shown (the list leaves them out at 390, for the names), no +// sideways overflow of the page; no script error, no failed request. +// node tests/browser_project_workspace_generations.cjs [--shots ] +const assert = require('node:assert/strict'), fs = require('node:fs'), os = require('node:os'), path = require('node:path'); +const { spawn, execFileSync } = require('node:child_process'), { once } = require('node:events'); +const { authenticatedFixture } = require('./auth_helpers.cjs'), { waitForDebugger } = require('./browser_startup.cjs'), { cdpSession } = require('./browser_cdp_session.cjs'); + +const root = path.resolve(__dirname, '..'); +assert.ok(process.env.TULDOK_DATABASE_URL, 'TULDOK_DATABASE_URL must name a disposable PostgreSQL cluster'); +const venv = path.join(root, '.venv', 'bin'); +if (fs.existsSync(path.join(venv, 'python'))) process.env.PATH = venv + path.delimiter + process.env.PATH; // auth_helpers runs python3 +const python = fs.existsSync(path.join(venv, 'python')) ? path.join(venv, 'python') : 'python3'; + +const argv = process.argv.slice(2), shots = argv.includes('--shots') ? path.resolve(argv[argv.indexOf('--shots') + 1]) : null; +if (shots) fs.mkdirSync(shots, { recursive: true }); +const temporary = fs.mkdtempSync(path.join(os.tmpdir(), 'tuldok-generations-browser-')); +const children = [], errors = [], failedRequests = []; +let server, serverErrors = '', origin, auth, socket, session, send, evaluate, owned = null; +const pause = ms => new Promise(resolve => setTimeout(resolve, ms)); +async function until(fn, label, tries = 300) { for (let i = 0; i < tries; i++) { const value = await fn(); if (value) return value; await pause(100); } throw Error('Timed out: ' + label); } +const check = text => console.log('PASS ' + text); +function launch(command, args, options) { const child = spawn(command, args, options); children.push(child); return child; } +async function stop(child) { if (!child || child.exitCode !== null || child.signalCode !== null) return; const exited = once(child, 'exit'); child.kill('SIGTERM'); await Promise.race([exited, pause(10000)]); if (child.exitCode === null && child.signalCode === null) { child.kill('SIGKILL'); await exited; } } +function findBrowser() { + const candidates = [process.env.BROWSER, '/usr/bin/chromium', '/usr/bin/chromium-browser', '/usr/bin/google-chrome']; + const cache = path.join(os.homedir(), '.cache', 'ms-playwright'); + if (fs.existsSync(cache)) for (const name of fs.readdirSync(cache).filter(n => n.startsWith('chromium_headless_shell-')).sort().reverse()) candidates.push(path.join(cache, name, 'chrome-headless-shell-linux64', 'chrome-headless-shell')); + const found = candidates.find(candidate => candidate && fs.existsSync(candidate)); + assert.ok(found, 'No headless browser: set BROWSER'); + return found; +} +/** Drops the project's schema and the catalog's from the disposable database. */ +function dropOwned(folder, catalog) { + const id = JSON.parse(fs.readFileSync(path.join(folder, 'tuldok.json'), 'utf8')).id; + execFileSync(python, ['-c', `import json, os, pathlib, sys, psycopg +from psycopg import sql +from tuldok_project import store +store.drop_project(sys.argv[1], os.environ['TULDOK_DATABASE_URL']) +root = pathlib.Path(sys.argv[2]) +if root.exists(): + with psycopg.connect(os.environ['TULDOK_DATABASE_URL'], autocommit=True) as c: + for p in root.rglob('.tuldok.json'): + v = json.loads(p.read_text()) + name = 'projection_' + v['projection_id'] if v.get('version') == 2 else 'catalog_' + v['dataset_id'] + c.execute(sql.SQL('DROP SCHEMA IF EXISTS {} CASCADE').format(sql.Identifier(name))) +`, id, catalog], { cwd: root, stdio: ['ignore', 'pipe', 'pipe'] }); +} +async function api(name, body = {}) { + const capabilities = await (await auth.fetch(origin + '/api/agent/capabilities')).json(); + const operation = capabilities.operations.find(item => item.id === 'project.' + name), contract = operation.contracts[operation.contracts.length - 1]; + const response = await auth.fetch(origin + '/api/project/' + name, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Tuldok-Operation-Contract': 'project.' + name + '/' + contract.version + '/' + contract.sha256 }, body: JSON.stringify(body) }); + const value = await response.json(); assert.equal(response.status, 200, JSON.stringify(value)); return value; +} +const readJson = file => JSON.parse(fs.readFileSync(file, 'utf8')); +const writeJson = (file, value) => fs.writeFileSync(file, JSON.stringify(value, null, 2) + '\n'); +async function open(theme, width, hash) { + await send('Emulation.setDeviceMetricsOverride', { width, height: 900, deviceScaleFactor: 1, mobile: width < 600 }); + await send('Emulation.setEmulatedMedia', { features: [{ name: 'prefers-color-scheme', value: theme }, { name: 'prefers-reduced-motion', value: 'reduce' }] }); + await send('Page.navigate', { url: origin + '/?theme=' + theme + '#' + hash }); + await until(() => evaluate(`document.readyState === 'complete' && !!document.getElementById('needs-btn') && !document.getElementById('needs-btn').disabled`), 'the project read (' + hash + ')'); + await until(() => evaluate(`(() => { const t = document.getElementById('main').innerText; return t.trim().length > 0 && !t.includes('Reading'); })()`), 'tab ' + hash); +} +const listChips = () => evaluate(`[...document.querySelectorAll('.tree-list .trow')].map(r => { const g = r.querySelector('.nm .gen'); return [r.dataset.id, g ? g.textContent : null, !!(g && g.classList.contains('crosses'))]; })`); +async function shot(name) { if (shots) fs.writeFileSync(path.join(shots, name + '.png'), Buffer.from((await send('Page.captureScreenshot', { format: 'png' })).data, 'base64')); } +const press = selector => evaluate(`(() => { const e = document.querySelector(${JSON.stringify(selector)}); if (!e) return false; e.click(); return true; })()`); + +(async () => { + // ---- The fixture: generations on the minimal project ------------------------------------------------------------- + fs.cpSync(path.join(root, 'tests', 'fixtures', 'tuldok_project', 'minimal'), path.join(temporary, 'fixture'), { recursive: true }); + const folder = path.join(temporary, 'fixture', 'project'); + const project = readJson(path.join(folder, 'tuldok.json')); project.id = require('node:crypto').randomUUID().replace(/-/g, ''); writeJson(path.join(folder, 'tuldok.json'), project); + const solver = readJson(path.join(folder, 'steps', 'solver.json')); + writeJson(path.join(folder, 'steps', 'solver.json'), Object.assign({}, solver, { generation: 'toy-solver-1' })); + writeJson(path.join(folder, 'steps', 'solver-new.json'), Object.assign({}, solver, { title: 'The new toy solver', generation: 'toy-solver-2' })); + const waves = readJson(path.join(folder, 'datasets', 'waves.json')); + writeJson(path.join(folder, 'datasets', 'waves-new.json'), Object.assign({}, waves, { title: 'Wave scenes, new solver', + run: Object.assign({}, waves.run, { inputs: { solver: { from: 'steps/solver-new', output: 'binary' } } }) })); + const k2 = readJson(path.join(folder, 'recipes', 'stepper-k2.json')); + const k3 = Object.assign({}, k2, { title: 'Stepper fine-tuned on the new solver', labels: { short: 'K3' }, derived_from: 'recipes/stepper-k2' }); + delete k3.compared_with; + k3.run = Object.assign({}, k2.run, { inputs: { start: { from: 'recipes/stepper-k2', output: 'weights' }, training: 'datasets/waves-new' } }); + writeJson(path.join(folder, 'recipes', 'stepper-k3.json'), k3); + const checked = JSON.parse(execFileSync(python, ['-m', 'tuldok_project', 'check', folder, '--json'], { cwd: root }).toString()); + assert.equal(checked.errors, 0, JSON.stringify(checked.problems)); + + const home = path.join(temporary, 'home'), catalog = path.join(temporary, 'catalog'); fs.mkdirSync(home); + owned = [folder, catalog]; dropOwned(folder, catalog); + let output = ''; + server = launch(python, ['-u', 'app.py', '--port', '0', '--data', catalog, '--project-folder', folder], { cwd: root, env: { ...process.env, TULDOK_PROJECT_HOME: home }, stdio: ['ignore', 'pipe', 'pipe'] }); + server.stdout.on('data', chunk => { output += chunk; }); server.stderr.on('data', chunk => { serverErrors += chunk; }); + const port = await until(() => { if (server.exitCode !== null) throw Error('Tuldok exited: ' + serverErrors); return output.match(/127\.0\.0\.1:(\d+)/)?.[1]; }, 'Tuldok start-up', 1800); + origin = 'http://127.0.0.1:' + port; + auth = authenticatedFixture(); await auth.register(origin, catalog, root); + const map = await api('map'), byRef = Object.fromEntries(map.models.map(m => [m.reference, m.generation || null])); + assert.deepEqual(byRef, { 'recipes/early-fit': null, 'recipes/stepper-k1': 'toy-solver-1', 'recipes/stepper-k2': 'toy-solver-1', 'recipes/stepper-k3': 'toy-solver-2' }); + check('map answers the generations: ' + JSON.stringify(byRef)); + + const browser = launch(findBrowser(), ['--headless', '--no-sandbox', '--disable-gpu', '--no-first-run', '--hide-scrollbars', '--remote-debugging-port=0', '--window-size=1440,900', '--user-data-dir=' + path.join(temporary, 'browser'), 'about:blank'], { stdio: 'ignore', env: { ...process.env, XDG_CONFIG_HOME: temporary, XDG_CACHE_HOME: temporary } }); + const debugPort = await waitForDebugger(browser, path.join(temporary, 'browser', 'DevToolsActivePort')); + const target = (await (await fetch('http://127.0.0.1:' + debugPort + '/json')).json()).find(item => item.type === 'page'); + socket = new WebSocket(target.webSocketDebuggerUrl); await once(socket, 'open'); + session = cdpSession(socket, { timeoutMs: 60000, onEvent: event => { + if (event.method === 'Runtime.exceptionThrown') errors.push(event.params.exceptionDetails.exception?.description || event.params.exceptionDetails.text); + if (event.method === 'Runtime.consoleAPICalled' && event.params.type === 'error') errors.push(event.params.args.map(a => a.value ?? a.description).join(' ')); + if (event.method === 'Network.loadingFailed' && !event.params.canceled) failedRequests.push({ id: event.params.requestId, error: event.params.errorText }); + if (event.method === 'Network.responseReceived' && event.params.response.status >= 400) failedRequests.push({ url: event.params.response.url, status: event.params.response.status }); + } }); + send = session.send; + evaluate = async expression => { const result = await send('Runtime.evaluate', { expression, awaitPromise: true, returnByValue: true }); if (result.exceptionDetails) throw Error(JSON.stringify(result.exceptionDetails)); return result.result.value; }; + await send('Page.enable'); await send('Runtime.enable'); await send('Network.enable'); + const issued = await auth.fetch(origin + '/api/access/credentials', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ label: 'A read-only browser check', scopes: ['inspect'], ttl_seconds: 3600 }) }); + const inspector = await issued.json(); assert.equal(issued.status, 201, JSON.stringify(inspector)); + assert.notEqual((await send('Network.setCookie', { name: 'tuldok_session', value: inspector.token, url: origin + '/', httpOnly: true, sameSite: 'Strict', path: '/' })).success, false); + + // ---- 1. The chips of the list ---------------------------------------------------------------------------------------- + await open('light', 1440, 'map'); + const chips = await until(async () => { const c = await listChips(); return c.length === 4 ? c : null; }, 'four rows'); + const shown = Object.fromEntries(chips.map(([id, g, crosses]) => [id, [g, crosses]])); + assert.deepEqual(shown, { 'recipes/early-fit': [null, false], 'recipes/stepper-k1': ['toy-solver-1', false], 'recipes/stepper-k2': ['toy-solver-1', false], 'recipes/stepper-k3': ['toy-solver-2', true] }); + check('The list shows each model\'s generation after its name, k3\'s dashed (its parent is of toy-solver-1)'); + + // ---- 2. The Generation filter ------------------------------------------------------------------------------------------ + const filter = await evaluate(`[...document.querySelectorAll('.filters [aria-label="Data generation"] button')].map(b => [b.dataset.generation, b.textContent, b.getAttribute('aria-pressed')])`); + assert.deepEqual(filter, [['toy-solver-1', 'toy-solver-1', 'true'], ['toy-solver-2', 'toy-solver-2', 'true'], ['', 'no generation', 'true']]); + assert.ok(await press('.filters button[data-generation="toy-solver-2"]')); + await until(async () => (await listChips()).length === 1, 'the list filtered'); + assert.deepEqual((await listChips()).map(c => c[0]), ['recipes/stepper-k3']); + const faint = await evaluate(`[...document.querySelectorAll('.tree-map g.node')].map(g => [g.dataset.id, g.classList.contains('dim')])`); + assert.deepEqual(Object.fromEntries(faint), { 'recipes/early-fit': true, 'recipes/stepper-k1': true, 'recipes/stepper-k2': true, 'recipes/stepper-k3': false }); + assert.deepEqual(await evaluate(`[...document.querySelectorAll('.filters [aria-label="Data generation"] button')].map(b => b.getAttribute('aria-pressed'))`), ['false', 'true', 'false']); + assert.ok(await press('.filters button[data-generation="toy-solver-2"]')); + await until(async () => (await listChips()).length === 4, 'the list whole again'); + check('The Generation filter: toy-solver-2 alone leaves k3 in the list and the other nodes faint; pressed again, all four'); + + // ---- 3. The drawer ----------------------------------------------------------------------------------------------------- + await evaluate(`location.hash = '#map/' + encodeURIComponent('recipes/stepper-k3')`); + const drawer = await until(() => evaluate(`(() => { const g = document.querySelector('#aside .ins-meta .gen'), f = document.querySelector('#aside .gen-from'); return g && f ? [g.textContent, f.textContent] : null; })()`), 'the drawer of k3'); + assert.deepEqual(drawer, ['toy-solver-2', 'from a model of toy-solver-1']); + await shot('drawer-k3'); + check('k3\'s drawer: its generation and "from a model of toy-solver-1"'); + + // ---- 4. The Data tab --------------------------------------------------------------------------------------------------- + await evaluate(`location.hash = '#data'`); + const cards = await until(() => evaluate(`(() => { const c = [...document.querySelectorAll('.ds-card')]; return c.length === 2 ? c.map(x => [x.dataset.ref, (x.querySelector('.meta .gen') || {}).textContent || null]) : null; })()`), 'the dataset cards'); + assert.deepEqual(Object.fromEntries(cards), { 'datasets/waves': 'toy-solver-1', 'datasets/waves-new': 'toy-solver-2' }); + check('The Data tab\'s cards show the datasets\' generations'); + + // ---- 5. Themes and widths ----------------------------------------------------------------------------------------------- + for (const theme of ['light', 'dark']) for (const width of [1440, 390]) for (const hash of ['map', 'data']) { + await open(theme, width, hash); await pause(200); + const sizes = await evaluate(`[innerWidth, document.documentElement.scrollWidth, document.body.scrollWidth, [...document.querySelectorAll('#main .gen')].filter(e => e.offsetWidth > 0).length]`); + assert.ok(sizes[1] <= sizes[0] && sizes[2] <= sizes[0], theme + ' ' + width + ' ' + hash + ': sideways overflow ' + JSON.stringify(sizes)); + // Narrow, the list leaves the chips out so that the names have the room (the drawer and the filter keep them). + assert.equal(sizes[3], hash === 'data' ? 2 : width < 600 ? 0 : 3, theme + ' ' + width + ' ' + hash + ': generation chips shown ' + JSON.stringify(sizes)); + await shot(theme + '-' + width + '-' + hash); + } + check('Light and dark at 1440 and 390 wide, Map and Data: the chips shown (the list\'s left out at 390), nothing overflows sideways'); + assert.deepEqual(errors, [], 'Script errors'); assert.deepEqual(failedRequests, [], 'Failed requests'); + check('No script error, no failed request'); +})().catch(async error => { + process.exitCode = 1; + console.error(error); if (serverErrors) console.error('Server:', serverErrors.slice(-4000)); + console.error('Script errors:', JSON.stringify(errors)); console.error('Failed requests:', JSON.stringify(failedRequests)); + try { if (evaluate) console.error('Page:', (await evaluate('document.body.innerText')).slice(0, 2000)); } catch (_) { /* the page is gone */ } +}).finally(async () => { + try { session?.close(); } catch (_) { /* closed */ } + for (const child of children.reverse()) await stop(child); + if (owned) try { dropOwned(...owned); } catch (error) { console.error('Dropping the disposable schemas failed:', error.message); process.exitCode = 1; } + try { execFileSync('chmod', ['-R', 'u+w', temporary]); } catch (_) { /* nothing was made read-only */ } + fs.rmSync(temporary, { recursive: true, force: true }); +}); diff --git a/tests/test_tuldok_project_generations.py b/tests/test_tuldok_project_generations.py new file mode 100644 index 00000000..0d89e10a --- /dev/null +++ b/tests/test_tuldok_project_generations.py @@ -0,0 +1,316 @@ +"""Data generations (docs/plans/training-and-curation/spec-generations.md). + +The rules (generations.resolve) are tested on small documents and through the +folder's checker on a copy of the minimal fixture, without a database. The +working copy's reads (map, definitions, definition) and set_definition run +only when TULDOK_DATABASE_URL names a disposable cluster. +""" +import json +import os +from pathlib import Path +import sys +import unittest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from tuldok_project import generations, workspace # noqa: E402 +from tuldok_project.check import check_folder # noqa: E402 +from tuldok_project.store import ProjectError # noqa: E402 +import test_tuldok_project_check as check_base # noqa: E402 +import test_tuldok_project_store as store_base # noqa: E402 + +DATABASE = os.environ.get('TULDOK_DATABASE_URL') + + +def _step(generation=None): + document = {'kind': 'step', 'run': {'command': ['make']}} + if generation is not None: + document['generation'] = generation + return document + + +def _dataset(generation=None, **inputs): + document = {'kind': 'dataset', 'run': {'command': ['make'], 'inputs': inputs}} + if generation is not None: + document['generation'] = generation + return document + + +def _recipe(*, mixed=False, derived_from=None, **inputs): + document = {'kind': 'recipe', 'run': {'command': ['train'], 'inputs': inputs}} + if mixed: + document['generations'] = 'mixed' + if derived_from: + document['derived_from'] = derived_from + return document + + +def _benchmark(*recipes, generation=None, mixed=False, cases=None): + document = {'kind': 'benchmark', 'run': {'command': ['bench']}, + 'subjects': [{'name': recipe.split('/')[1], 'recipe': recipe} for recipe in recipes]} + if cases: + document['cases'] = {'dataset': cases} + if generation is not None: + document['generation'] = generation + if mixed: + document['generations'] = 'mixed' + return document + + +def _two_sources(): + return {'steps/old': _step('rheon-reference-v1'), 'steps/new': _step('rheon-v2'), + 'datasets/old': _dataset(solver={'from': 'steps/old', 'output': 'bin'}), + 'datasets/new': _dataset(solver={'from': 'steps/new', 'output': 'bin'})} + + +class Resolve(unittest.TestCase): + """generations.resolve on documents written here.""" + + def problems(self, resolved): + return [(reference, severity, pointer) for reference, severity, pointer, _ in resolved.problems] + + def test_inherited_through_steps_and_datasets_of_several_levels(self): + definitions = { + 'steps/solver': _step('rheon-v2'), + 'datasets/raw': _dataset(solver={'from': 'steps/solver', 'output': 'bin'}), + 'datasets/frozen': {'kind': 'dataset', 'run': {'step': 'freeze', 'inputs': {'records': 'datasets/raw'}}}, + 'datasets/embedded': _dataset(records='datasets/frozen'), + 'recipes/fit': _recipe(training='datasets/embedded'), + 'benchmarks/bench': _benchmark('recipes/fit', cases='datasets/frozen'), + } + resolved = generations.resolve(definitions) + self.assertEqual(resolved.problems, []) + self.assertEqual(resolved.generation, {reference: 'rheon-v2' for reference in definitions}) + + def test_inputs_that_disagree_are_an_error_naming_each(self): + definitions = {**_two_sources(), 'datasets/both': _dataset(a='datasets/old', b='datasets/new')} + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [('datasets/both', 'error', '/run/inputs/a')]) + message = resolved.problems[0][3] + self.assertIn('datasets/old (rheon-reference-v1)', message) + self.assertIn('datasets/new (rheon-v2)', message) + self.assertIsNone(resolved.generation['datasets/both']) + + def test_a_declaration_that_differs_from_the_inputs_is_an_error(self): + definitions = {**_two_sources(), 'datasets/relabelled': _dataset('rheon-v2', a='datasets/old')} + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [('datasets/relabelled', 'error', '/generation')]) + self.assertIn('datasets/old (rheon-reference-v1)', resolved.problems[0][3]) + # Declaring does not settle inputs that disagree. + definitions['datasets/relabelled'] = _dataset('rheon-v2', a='datasets/old', b='datasets/new') + self.assertEqual(self.problems(generations.resolve(definitions)), + [('datasets/relabelled', 'error', '/generation')]) + + def test_a_declaration_equal_to_the_inputs_is_fine(self): + definitions = {**_two_sources(), 'datasets/same': _dataset('rheon-v2', a='datasets/new')} + resolved = generations.resolve(definitions) + self.assertEqual(resolved.problems, []) + self.assertEqual(resolved.generation['datasets/same'], 'rheon-v2') + + def test_unlabelled_data_taken_by_a_labelled_dataset_or_step_is_a_warning(self): + definitions = {**_two_sources(), 'datasets/plain': _dataset(), 'steps/plain': _step(), + 'datasets/with-plain': _dataset(a='datasets/new', b='datasets/plain'), + 'datasets/declared': _dataset('rheon-v2', b='datasets/plain', c='datasets/plain'), + 'steps/labelled': {**_step('rheon-v2'), 'run': {'command': ['make'], + 'inputs': {'s': {'from': 'steps/plain', 'output': 'x'}}}}} + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [('datasets/declared', 'warning', '/run/inputs/b'), + ('datasets/with-plain', 'warning', '/run/inputs/b'), + ('steps/labelled', 'warning', '/run/inputs/s/from')]) + self.assertEqual(resolved.problems[1][3], + 'Takes unlabelled data datasets/plain; label it with the generation it comes from.') + # The unlabelled input does not decide the generation; it stays without one. + self.assertEqual(resolved.generation['datasets/with-plain'], 'rheon-v2') + self.assertIsNone(resolved.generation['datasets/plain']) + + def test_unlabelled_data_of_a_recipe_and_a_benchmark_is_a_warning(self): + definitions = {**_two_sources(), 'datasets/plain': _dataset(), + 'recipes/fit': _recipe(training='datasets/new', extra='datasets/plain'), + 'recipes/mixed': _recipe(mixed=True, old='datasets/old', new='datasets/new', + extra='datasets/plain'), + 'benchmarks/bench': _benchmark(cases='datasets/plain', generation='rheon-v2'), + 'benchmarks/inherited': {**_benchmark(cases='datasets/new'), + 'run': {'command': ['b'], 'inputs': {'more': 'datasets/plain'}}}} + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [ + ('recipes/fit', 'warning', '/run/inputs/extra'), + ('recipes/mixed', 'warning', '/generations'), + ('recipes/mixed', 'warning', '/run/inputs/extra'), + ('benchmarks/bench', 'warning', '/cases/dataset'), + ('benchmarks/inherited', 'warning', '/run/inputs/more')]) + + def test_no_warning_for_an_input_without_generation_because_of_its_own_error(self): + definitions = {**_two_sources(), 'datasets/both': _dataset(a='datasets/old', b='datasets/new'), + 'recipes/fit': _recipe(training='datasets/new', also='datasets/both')} + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [('datasets/both', 'error', '/run/inputs/a')]) + + def test_unlabelled_only_has_no_warning(self): + definitions = {'datasets/a': _dataset(), 'datasets/b': _dataset(a='datasets/a'), + 'recipes/r': _recipe(training='datasets/b', more='datasets/a'), + 'benchmarks/x': _benchmark('recipes/r', cases='datasets/a')} + self.assertEqual(generations.resolve(definitions).problems, []) + + def test_a_recipe_on_two_generations_is_an_error(self): + definitions = {**_two_sources(), 'recipes/fit': _recipe(old='datasets/old', new='datasets/new')} + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [('recipes/fit', 'error', '/run/inputs/old')]) + self.assertIn('mixes data generations', resolved.problems[0][3]) + self.assertIn('"generations": "mixed"', resolved.problems[0][3]) + self.assertEqual(resolved.generation['recipes/fit'], 'mixed') + + def test_a_recipe_declared_mixed_is_a_warning(self): + definitions = {**_two_sources(), 'recipes/fit': _recipe(mixed=True, old='datasets/old', new='datasets/new')} + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [('recipes/fit', 'warning', '/generations')]) + self.assertEqual(resolved.generation['recipes/fit'], 'mixed') + + def test_a_recipe_takes_only_its_data_inputs(self): + # A model input (another recipe) and a parent of another generation are fine: fine-tuning across generations. + definitions = {**_two_sources(), 'recipes/old': _recipe(training='datasets/old'), + 'recipes/tuned': _recipe(derived_from='recipes/old', training='datasets/new', + start='recipes/old')} + resolved = generations.resolve(definitions) + self.assertEqual(resolved.problems, []) + self.assertEqual((resolved.generation['recipes/old'], resolved.generation['recipes/tuned']), + ('rheon-reference-v1', 'rheon-v2')) + + def test_a_benchmark_whose_subjects_span_generations_warns_unless_mixed(self): + definitions = {**_two_sources(), 'recipes/a': _recipe(training='datasets/old'), + 'recipes/b': _recipe(training='datasets/new'), 'recipes/plain': _recipe(), + 'benchmarks/bench': _benchmark('recipes/a', 'recipes/b', 'recipes/plain')} + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [('benchmarks/bench', 'warning', '/subjects')]) + self.assertIn('rheon-reference-v1: a; rheon-v2: b', resolved.problems[0][3]) + self.assertIsNone(resolved.generation['benchmarks/bench']) + definitions['benchmarks/bench'] = _benchmark('recipes/a', 'recipes/b', mixed=True) + self.assertEqual(generations.resolve(definitions).problems, []) + + def test_a_benchmark_declares_its_cases(self): + definitions = {**_two_sources(), 'benchmarks/bench': _benchmark(cases='datasets/old')} + self.assertEqual(generations.resolve(definitions).generation['benchmarks/bench'], 'rheon-reference-v1') + definitions['benchmarks/bench'] = _benchmark(cases='datasets/old', generation='rheon-reference-v1') + self.assertEqual(generations.resolve(definitions).problems, []) + definitions['benchmarks/bench'] = _benchmark(cases='datasets/old', generation='rheon-v2') + resolved = generations.resolve(definitions) + self.assertEqual(self.problems(resolved), [('benchmarks/bench', 'error', '/generation')]) + self.assertEqual(resolved.generation['benchmarks/bench'], 'rheon-v2') + definitions['benchmarks/bench'] = _benchmark(generation='rheon-v2') # no cases: the declaration alone + self.assertEqual(generations.resolve(definitions).generation['benchmarks/bench'], 'rheon-v2') + + def test_an_unlabelled_project_has_no_generation_and_no_problem(self): + definitions = {'steps/s': _step(), 'datasets/d': _dataset(s={'from': 'steps/s', 'output': 'x'}), + 'recipes/r': _recipe(training='datasets/d'), 'benchmarks/b': _benchmark('recipes/r')} + resolved = generations.resolve(definitions) + self.assertEqual(resolved.problems, []) + self.assertEqual(set(resolved.generation.values()), {None}) + self.assertEqual(generations.labelled(definitions), {}) + + def test_a_cycle_of_inputs_ends(self): + definitions = {'datasets/a': _dataset('g1', b='datasets/b'), 'datasets/b': _dataset(a='datasets/a')} + resolved = generations.resolve(definitions) + self.assertEqual(resolved.generation['datasets/a'], 'g1') + + +class Folder(check_base.FixtureCopy): + """The rules through check_folder on the minimal fixture (steps/solver -> datasets/waves -> recipes/stepper-k1).""" + + def test_the_unlabelled_fixture_checks_as_before(self): + self.assertEqual(check_folder(self.project).problems, []) + + def test_a_label_on_the_solver_reaches_the_dataset_the_recipe_and_the_benchmark(self): + self.edit('steps/solver.json', '/generation', 'toy-solver-1') + report = check_folder(self.project) + self.assertEqual(report.problems, []) + decoded = {reference: d.document for reference, d in report.definitions.items()} + self.assertEqual(generations.labelled(decoded), { + 'steps/solver': 'toy-solver-1', 'datasets/waves': 'toy-solver-1', + 'recipes/stepper-k1': 'toy-solver-1', 'recipes/stepper-k2': 'toy-solver-1', + 'benchmarks/rollout': 'toy-solver-1'}) + + def test_the_schema_holds_the_name_rule(self): + for value in ('mixed', 'Rheon-V2', '', 'x' * 65, 'a b'): + with self.subTest(value=value): + self.edit('steps/solver.json', '/generation', value) + self.assertProblem(check_folder(self.project), 'steps/solver.json', '/generation') + self.edit('steps/solver.json', '/generation', 'rheon-v2.1_b') + self.edit('recipes/stepper-k1.json', '/generations', 'all') + self.assertProblem(check_folder(self.project), 'recipes/stepper-k1.json', '/generations') + + def test_a_label_on_the_dataset_alone_warns_of_its_unlabelled_solver(self): + self.edit('datasets/waves.json', '/generation', 'rheon-reference-v1') + problem = self.assertProblem(check_folder(self.project), 'datasets/waves.json', '/run/inputs/solver/from', + 'warning') + self.assertIn('Takes unlabelled data steps/solver', problem.message) + + def _second_source(self): + """steps/solver of rheon-reference-v1, and steps/solver-new of rheon-v2 making datasets/waves-new.""" + solver = json.loads((self.project / 'steps' / 'solver.json').read_text(encoding='utf-8')) + self.edit('steps/solver.json', '/generation', 'rheon-reference-v1') + self.write('steps/solver-new.json', json.dumps({**solver, 'generation': 'rheon-v2'}, indent=2) + '\n') + waves = json.loads((self.project / 'datasets' / 'waves.json').read_text(encoding='utf-8')) + waves['run']['inputs']['solver']['from'] = 'steps/solver-new' + self.write('datasets/waves-new.json', json.dumps(waves, indent=2) + '\n') + + def test_a_recipe_mixing_two_generations(self): + self._second_source() + self.edit('recipes/stepper-k1.json', '/run/inputs/newer', 'datasets/waves-new') + report = check_folder(self.project) + self.assertEqual(report.count('error'), 1) + problem = self.assertProblem(report, 'recipes/stepper-k1.json', '/run/inputs/training', alone=False) + self.assertIn('mixes data generations: datasets/waves (rheon-reference-v1), datasets/waves-new (rheon-v2)', + problem.message) + # The benchmark measures k1 (mixed) beside k2 (rheon-reference-v1): a warning, as long as it does not say so. + warning = self.assertProblem(report, 'benchmarks/rollout.json', '/subjects', 'warning', alone=False) + self.assertIn('mixed: stepper-k1; rheon-reference-v1: stepper-k2', warning.message) + self.edit('recipes/stepper-k1.json', '/generations', 'mixed') + self.edit('benchmarks/rollout.json', '/generations', 'mixed') + self.assertProblem(check_folder(self.project), 'recipes/stepper-k1.json', '/generations', 'warning') + + +@unittest.skipUnless(DATABASE, 'TULDOK_DATABASE_URL is not set; database tests need a disposable cluster.') +class WorkingCopy(store_base.StoreCase): + """The reads and set_definition on the minimal fixture.""" + + def test_an_unlabelled_project_reads_without_generation(self): + project = self.open() + self.assertFalse(any('generation' in model for model in workspace.project_map(project)['models'])) + self.assertFalse(any('generation' in row for row in project.definitions())) + self.assertNotIn('generation', project.definition('datasets/waves')) + + def test_the_reads_carry_the_generation(self): + self.write('steps/solver', self.edited('steps/solver', lambda d: d.update(generation='toy-solver-1'))) + project = self.open() + models = {model['reference']: model for model in workspace.project_map(project)['models']} + self.assertEqual(models['recipes/stepper-k1']['generation'], 'toy-solver-1') + self.assertEqual(models['recipes/stepper-k2']['generation'], 'toy-solver-1') + self.assertNotIn('generation', models['recipes/early-fit']) # earlier work with no data input + rows = {row['reference']: row for row in project.definitions()} + self.assertEqual(rows['datasets/waves']['generation'], 'toy-solver-1') + self.assertNotIn('generation', rows['environments/lab-cpu']) + self.assertEqual(project.definition('benchmarks/rollout')['generation'], 'toy-solver-1') + self.assertEqual(project.definition('steps/solver')['generation'], 'toy-solver-1') + + def test_set_definition_refuses_a_mixing_recipe_and_keeps_a_mixed_one(self): + self.write('datasets/waves', self.edited('datasets/waves', lambda d: d.update(generation='rheon-reference-v1'))) + project = self.open() + newer = self.edited('datasets/waves', lambda d: d.update(generation='rheon-v2')) + project.set_definition('datasets/waves-new', newer) + + def both(document, mixed=False): + document['run']['inputs']['newer'] = 'datasets/waves-new' + if mixed: + document['generations'] = 'mixed' + + with self.assertRaises(ProjectError) as caught: + project.set_definition('recipes/stepper-k1', self.edited('recipes/stepper-k1', both)) + self.assertEqual([(p.path, p.pointer) for p in caught.exception.problems], + [('recipes/stepper-k1.json', '/run/inputs/training')]) + project.set_definition('recipes/stepper-k1', self.edited('recipes/stepper-k1', lambda d: both(d, True))) + models = {model['reference']: model for model in workspace.project_map(project)['models']} + self.assertEqual(models['recipes/stepper-k1']['generation'], 'mixed') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/test_tuldok_project_sweeps.py b/tests/test_tuldok_project_sweeps.py index 6a6a4484..8273beb4 100644 --- a/tests/test_tuldok_project_sweeps.py +++ b/tests/test_tuldok_project_sweeps.py @@ -574,7 +574,9 @@ def refused(self, served, name, arguments): return refused.exception.code def test_minutes_stop_a_sweep(self): - self.write('recipes/base', recipe({**self.SETTINGS, 'steps': 30, 'sleep': 0.1}, group='small')) + # Each trial would take 60 s, far past the 3.6 s budget, so the budget always ends inside a trial (on a slow + # runner, trials of about the budget's length could finish just before it ran out, leaving none stopped). + self.write('recipes/base', recipe({**self.SETTINGS, 'steps': 600, 'sleep': 0.1}, group='small')) self.commit('Slower.') self.keep('sweeps/m', sweep(budget={'trials': 10, 'minutes': 0.06})) served = self.service() diff --git a/tuldok_project/check.py b/tuldok_project/check.py index c4c97b16..fc93c324 100644 --- a/tuldok_project/check.py +++ b/tuldok_project/check.py @@ -9,8 +9,8 @@ (in definitions and in the code_paths of tuldok.json), the map's fields (spec-w1.md section 1): groups, a recipe's group, derived_from and compared_with, and map.chart, each annotations/.jsonl against its -task (spec-d4.md section 2; the rules of a line are annotations.py's), and -the settings, inputs and tasks of a dataset whose run names a step Tuldok +task (spec-d4.md section 2; the rules of a line are annotations.py's), the +data generations (generations.py, spec-generations.md), and the settings, inputs and tasks of a dataset whose run names a step Tuldok provides (section 6; generate is spec-d5.md section 3). Problems are collected, never raised, so one bad file never hides another. """ @@ -20,7 +20,7 @@ import os import re -from . import annotations, documents, folder as folder_reader, graph, results, steps +from . import annotations, documents, folder as folder_reader, generations, graph, results, steps from .documents import Problem _SEVERITY_ORDER = {'error': 0, 'warning': 1} @@ -769,6 +769,7 @@ def check_map(project, definitions, add): 'name a chart of the form progression.') from . import sweeps # here: sweeps imports store, which imports this module sweeps.check_sweeps(definitions, add) + generations.check_generations(definitions, add) # spec-generations.md def _check_results(files, definitions, unsound, add): diff --git a/tuldok_project/generations.py b/tuldok_project/generations.py new file mode 100644 index 00000000..23ae35c3 --- /dev/null +++ b/tuldok_project/generations.py @@ -0,0 +1,193 @@ +"""Data generations (docs/plans/training-and-curation/spec-generations.md): which source generation data comes from. + +A dataset or step may declare `generation`; one that declares none takes the +generation of its data inputs (the entries of run.inputs naming a dataset or a +step). A recipe's generation is computed from its data inputs; a benchmark's +is declared or its cases' data. resolve() is pure: it takes {reference: +decoded document} (documents need not be valid) and returns every computed +generation and every problem, so the checker and the working copy's reads +share one rule. Inputs without a generation do not decide one, but a labelled +definition that takes one is warned; nothing without a labelled input gets a +generation. +""" +from dataclasses import dataclass, field + +from . import documents, graph + +MIXED = 'mixed' # a recipe's or a benchmark's word for crossing generations on purpose; never a generation name +_DATA = ('datasets', 'steps') + + +@dataclass +class Resolved: + """generation: {reference: name or None} for datasets, steps, recipes and benchmarks. + + problems: (reference, severity, pointer, message) tuples. + """ + generation: dict = field(default_factory=dict) + problems: list = field(default_factory=list) + + +def _directory(reference): + parsed = documents.parse_reference(reference) if isinstance(reference, str) else None + return parsed[0] if parsed else None + + +def _declared(document, key='generation'): + value = document.get(key) if isinstance(document, dict) else None + return value if isinstance(value, str) else None + + +def _data_inputs(document, definitions): + """[(pointer, reference)] of the run's inputs that name a dataset or step present in definitions.""" + found = [] + for use in graph.uses(document): + if use.role.startswith('input:') and _directory(use.target) in _DATA and use.target in definitions: + found.append((use.pointer, use.target)) + return found + + +def _listed(pairs): + return ', '.join(f'{reference} ({generation})' for reference, generation in pairs) + + +def resolve(definitions): + """Every generation and every generation problem of a project (spec-generations.md sections 1 and 2).""" + resolved = Resolved() + data = {} + walking = set() + conflicted = set() # datasets and steps without a generation because their inputs disagree (an error already) + + def unlabelled(reference, inputs): + """A warning per data input of a labelled definition that has no generation of its own (section 1).""" + seen = set() + for pointer, target in inputs: + if target in seen or data.get(target) is not None or target in conflicted: + continue + seen.add(target) + resolved.problems.append((reference, 'warning', pointer, + f'Takes unlabelled data {target}; label it with the generation it comes from.')) + + def of_data(reference): + """The generation of a dataset or step (None: unlabelled, or its inputs disagree), walked once each.""" + if reference in data: + return data[reference] + if reference in walking: # a cycle of inputs: the cycle rule reports it + return None + walking.add(reference) + document = definitions[reference] + inputs = _data_inputs(document, definitions) + labelled = [(pointer, target, of_data(target)) for pointer, target in inputs] + walking.discard(reference) + labelled = [(pointer, target, value) for pointer, target, value in labelled if value is not None] + declared = _declared(document) + names = sorted({value for _, _, value in labelled}) + if declared is not None: + wrong = [(target, value) for _, target, value in labelled if value != declared] + if wrong and declared != MIXED: + resolved.problems.append((reference, 'error', '/generation', + f'This declares the data generation {declared!r} but its inputs come from ' + f'another: {_listed(wrong)}; make it from data of {declared!r} only, or ' + 'change generation.')) + value = declared if declared != MIXED else None + elif len(names) > 1: + resolved.problems.append((reference, 'error', labelled[0][0], + f'Its inputs come from different data generations: ' + f'{_listed((t, v) for _, t, v in labelled)}; make it from one generation.')) + value = None + conflicted.add(reference) + else: + value = names[0] if names else None + data[reference] = value + if value is not None: + unlabelled(reference, inputs) + return value + + for reference in sorted(definitions): + directory = _directory(reference) + if directory in _DATA: + resolved.generation[reference] = of_data(reference) + + for reference in sorted(definitions): + document, directory = definitions[reference], _directory(reference) + if directory != 'recipes' or not isinstance(document, dict): + continue + inputs = _data_inputs(document, definitions) + labelled = [(pointer, target, data.get(target)) for pointer, target in inputs] + labelled = [item for item in labelled if item[2] is not None] + names = sorted({value for _, _, value in labelled}) + mixed = document.get('generations') == MIXED + if len(names) > 1: + listed = _listed((t, v) for _, t, v in labelled) + if mixed: + resolved.problems.append((reference, 'warning', '/generations', + f'This recipe mixes data generations on purpose: {listed}.')) + else: + resolved.problems.append((reference, 'error', labelled[0][0], + f'This recipe mixes data generations: {listed}; train on one generation, ' + 'or declare "generations": "mixed" for a deliberate cross-generation ' + 'experiment.')) + resolved.generation[reference] = MIXED + else: + resolved.generation[reference] = names[0] if names else None + if resolved.generation[reference] is not None: + unlabelled(reference, inputs) + + for reference in sorted(definitions): + document, directory = definitions[reference], _directory(reference) + if directory != 'benchmarks' or not isinstance(document, dict): + continue + cases, seen, inputs = [], set(), [] + for use in graph.uses(document): + if (use.role == 'cases' or use.role.startswith('input:')) and use.target in data: + inputs.append((use.pointer, use.target)) + if data[use.target] is not None and use.target not in seen: + seen.add(use.target) + cases.append((use.pointer, use.target, data[use.target])) + declared = _declared(document) + names = sorted({value for _, _, value in cases}) + if declared is not None: + wrong = [(target, value) for _, target, value in cases if value != declared] + if wrong: + resolved.problems.append((reference, 'error', '/generation', + f'This benchmark declares its cases of the data generation {declared!r} ' + f'but its data comes from another: {_listed(wrong)}; fix generation or ' + 'the cases.')) + value = declared + elif len(names) > 1: + resolved.problems.append((reference, 'error', cases[0][0], + f'Its cases come from different data generations: ' + f'{_listed((t, v) for _, t, v in cases)}; measure on one generation.')) + value = None + else: + value = names[0] if names else None + resolved.generation[reference] = value + if value is not None: + unlabelled(reference, inputs) + subjects = document.get('subjects') if isinstance(document.get('subjects'), list) else [] + spanned = {} + for subject in subjects: + recipe = subject.get('recipe') if isinstance(subject, dict) else None + generation = resolved.generation.get(recipe) if isinstance(recipe, str) else None + if generation is not None: + spanned.setdefault(generation, []).append(subject.get('name') or recipe) + if len(spanned) > 1 and document.get('generations') != MIXED: + listed = '; '.join(f'{name}: {", ".join(str(s) for s in names)}' for name, names in sorted(spanned.items())) + resolved.problems.append((reference, 'warning', '/subjects', + f'The subjects come from different data generations ({listed}); measure one ' + 'generation, or declare "generations": "mixed" if comparing across them is ' + 'meant.')) + return resolved + + +def check_generations(definitions, add): + """Report the generation rules for {reference: folder.Definition} (called by check.check_map).""" + decoded = {reference: d.document for reference, d in definitions.items() if isinstance(d.document, dict)} + for reference, severity, pointer, message in resolve(decoded).problems: + add(severity, definitions[reference].path, pointer, message) + + +def labelled(documents_by_reference): + """{reference: generation} of the datasets, steps, recipes and benchmarks that have one.""" + return {reference: value for reference, value in resolve(documents_by_reference).generation.items() + if value is not None} diff --git a/tuldok_project/schemas/benchmark.schema.json b/tuldok_project/schemas/benchmark.schema.json index 5a85600d..48b7d962 100644 --- a/tuldok_project/schemas/benchmark.schema.json +++ b/tuldok_project/schemas/benchmark.schema.json @@ -157,6 +157,15 @@ } } } + }, + "generation": { + "type": "string", + "pattern": "^(?!mixed$)[a-z0-9._-]{1,64}$", + "description": "a data generation: 1 to 64 characters of a-z, 0-9, '.', '_' and '-', not \"mixed\"" + }, + "generations": { + "const": "mixed", + "description": "\"mixed\": a deliberate crossing of data generations" } }, "if": { diff --git a/tuldok_project/schemas/dataset.schema.json b/tuldok_project/schemas/dataset.schema.json index 86082e3b..cfa4cfe3 100644 --- a/tuldok_project/schemas/dataset.schema.json +++ b/tuldok_project/schemas/dataset.schema.json @@ -116,6 +116,11 @@ ] } ] + }, + "generation": { + "type": "string", + "pattern": "^(?!mixed$)[a-z0-9._-]{1,64}$", + "description": "a data generation: 1 to 64 characters of a-z, 0-9, '.', '_' and '-', not \"mixed\"" } }, "if": { diff --git a/tuldok_project/schemas/recipe.schema.json b/tuldok_project/schemas/recipe.schema.json index a8cce58a..ce52d9a5 100644 --- a/tuldok_project/schemas/recipe.schema.json +++ b/tuldok_project/schemas/recipe.schema.json @@ -178,6 +178,10 @@ "type": "object" } } + }, + "generations": { + "const": "mixed", + "description": "\"mixed\": a deliberate crossing of data generations" } }, "if": { diff --git a/tuldok_project/schemas/step.schema.json b/tuldok_project/schemas/step.schema.json index 9ffb0cd0..c9801661 100644 --- a/tuldok_project/schemas/step.schema.json +++ b/tuldok_project/schemas/step.schema.json @@ -49,6 +49,11 @@ "description": "Only a dataset's run may name a step Tuldok provides; give a command." } } + }, + "generation": { + "type": "string", + "pattern": "^(?!mixed$)[a-z0-9._-]{1,64}$", + "description": "a data generation: 1 to 64 characters of a-z, 0-9, '.', '_' and '-', not \"mixed\"" } }, "if": { diff --git a/tuldok_project/store.py b/tuldok_project/store.py index 5e07d148..60dd0fdc 100644 --- a/tuldok_project/store.py +++ b/tuldok_project/store.py @@ -51,7 +51,7 @@ from metadata import connection_info from . import (activity, annotation_rows, annotations, build_requests, builds, check, documents, environments, - folder as folder_reader, graph, result_rows, results) + folder as folder_reader, generations, graph, result_rows, results) from .documents import Problem PROJECT_ID = re.compile(r'^[a-f0-9]{32}$') @@ -673,16 +673,23 @@ def summary(self): 'code_root': state['code_root'], 'kinds': dict(sorted(rows)), 'states': dict(sorted(states.items()))} def definitions(self, kind=None): - """One row per definition, sorted by reference: reference, kind, title, state, unsaved, removed.""" + """One row per definition, sorted by reference: reference, kind, title, state, unsaved, removed, and + generation for a dataset, step, recipe or benchmark that has one (spec-generations.md).""" states = self._states() query = f'SELECT {_REFERENCE_SQL}, kind, title, unsaved, removed FROM {{schema}}.definitions' rows = (self._execute(query + ' WHERE kind = %s', (kind,)) if kind is not None else self._execute(query)).fetchall() - return [{'reference': reference, 'kind': row_kind, 'title': title, 'state': states[reference], - 'unsaved': unsaved, 'removed': removed} - for reference, row_kind, title, unsaved, removed in sorted(rows)] + labelled = generations.labelled(self._documents(removed=False)) + found = [{'reference': reference, 'kind': row_kind, 'title': title, 'state': states[reference], + 'unsaved': unsaved, 'removed': removed} + for reference, row_kind, title, unsaved, removed in sorted(rows)] + for row in found: # only where there is one (spec-generations.md section 3) + if row['reference'] in labelled: + row['generation'] = labelled[row['reference']] + return found def definition(self, reference): - """The text, document and state of one definition, what it uses and what uses it (with roles).""" + """The text, document and state of one definition, what it uses and what uses it (with roles), and its + data generation when it has one (spec-generations.md).""" directory, name = _split(reference) row = self._execute('SELECT kind, title, text, document, unsaved, removed FROM {schema}.definitions ' 'WHERE directory = %s AND name = %s', (directory, name)).fetchone() @@ -693,10 +700,15 @@ def definition(self, reference): (reference,)).fetchall() used_by = self._execute('SELECT from_ref, role FROM {schema}.edges WHERE to_ref = %s ORDER BY from_ref, role', (reference,)).fetchall() - return {'reference': reference, 'kind': kind, 'title': title, 'text': text, 'document': document, - 'state': self._states()[reference], 'unsaved': unsaved, 'removed': removed, - 'uses': [{'reference': target, 'role': role} for target, role in uses], - 'used_by': [{'reference': source, 'role': role} for source, role in used_by]} + found = {'reference': reference, 'kind': kind, 'title': title, 'text': text, 'document': document, + 'state': self._states()[reference], 'unsaved': unsaved, 'removed': removed, + 'uses': [{'reference': target, 'role': role} for target, role in uses], + 'used_by': [{'reference': source, 'role': role} for source, role in used_by]} + if kind in ('dataset', 'step', 'recipe', 'benchmark') and not removed: # spec-generations.md section 3 + generation = generations.labelled(self._documents(removed=False)).get(reference) + if generation is not None: + found['generation'] = generation + return found def graph(self): """The definitions not removed, with kind and state, and the edges between them.""" diff --git a/tuldok_project/sweeps.py b/tuldok_project/sweeps.py index e98888c9..9f4ac9c9 100644 --- a/tuldok_project/sweeps.py +++ b/tuldok_project/sweeps.py @@ -343,7 +343,7 @@ def trial_document(sweep_reference, sweep, recipe_reference, recipe, number, val document = {'format': documents.DEFINITION_FORMAT, 'kind': 'recipe', 'title': f'{sweep["title"]} · trial {number}'[:200], 'description': f'Trial {number} of {sweep_reference}: {shown}.'[:4000]} - for key in ('model', 'group'): + for key in ('model', 'group', 'generations'): # generations: the run's inputs come along (spec-generations.md) if key in recipe: document[key] = recipe[key] document['labels'] = {'short': f'T{number}'} diff --git a/tuldok_project/workspace.py b/tuldok_project/workspace.py index 62d36c35..8febe336 100644 --- a/tuldok_project/workspace.py +++ b/tuldok_project/workspace.py @@ -25,7 +25,7 @@ from psycopg import sql -from . import builds, charts, documents, environments, folder as folder_reader, graph, result_rows, results +from . import builds, charts, documents, environments, folder as folder_reader, generations, graph, result_rows, results from .store import ProjectError, _REFERENCE_SQL NOTE_LIMIT = 1024 * 1024 @@ -151,6 +151,7 @@ def project_map(project): with_rows.setdefault(recipe, set()).add(benchmark) finished = _product_finish_times(project) + labelled = generations.labelled(documents_by_reference) recipes = sorted(reference for reference, item in definitions.items() if item['kind'] == 'recipe') models = [] @@ -168,6 +169,8 @@ def project_map(project): 'card': _text(document.get('card')), 'value': value, 'count': count, 'benchmarks': len(with_rows.get(reference, ())), }) + if labelled.get(reference) is not None: # only where there is one (spec-generations.md section 3) + models[-1]['generation'] = labelled[reference] models.sort(key=lambda model: (model['trained_at'] is None, model['trained_at'] or '', model['reference'])) recipe_documents = {reference: definitions[reference]['document'] for reference in recipes} @@ -317,7 +320,7 @@ def _settings_object(code_root, path): def draft_branch(project, recipe, title, *, name=None, change=None, description=None): """{reference, text, parent, inlined, problems}: a new recipe branched from recipe, stored nowhere (spec-w3.md). - The text carries the parent's model, group and run, its kind of change + The text carries the parent's model, group, generations and run, its kind of change unless change is given, and derived_from naming the parent; not its card, labels, earlier_work or comparisons. A run whose settings_file is a JSON object under the code root carries it as settings, and inlined names the @@ -350,7 +353,7 @@ def draft_branch(project, recipe, title, *, name=None, change=None, description= document = {'format': parent['format'], 'kind': 'recipe', 'title': title} if description is not None: document['description'] = description - for field in ('model', 'group'): + for field in ('model', 'group', 'generations'): # generations: the run's inputs come along if field in parent: document[field] = parent[field] if change is not None or 'change' in parent: