Automatic reduction is a batch reduced as its files arrive

docs/developer/README.md

@@ -60,7 +60,7 @@ What the framework adds to the plain loop:
 ## Components
 
 ```
-  controller: a notebook, an application, the trigger loop   decides what to run
+  controller: a notebook, an application, automatic reduction   decides what to run
         │  client: submit, push, output, release, ...
         ▼
   workspace ─── file source: files and their metadata
@@ -128,7 +128,7 @@ An output of a record of the bound spec, one the workspace runs with the same th
 An output of a record of another spec, such as an older version or the same version with changed models, is read as stored, an array as it is and anything else as a `DataGroup`, and a new request may not reference it ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)).
 
 **Controller.** The workspace does not decide what to run.
-A *controller* is client code that decides over time what to submit, push, and release, such as the cells of a notebook, the callbacks of an application's screen, or the trigger loop of automatic reduction.
+A *controller* is client code that decides over time what to submit, push, and release, such as the cells of a notebook, the callbacks of an application's screen, or the loop of automatic reduction.
 A controller is never part of the workspace, and calls it only through a client:
 
 ```python
@@ -265,19 +265,20 @@ total = client.freeze(volume)                               # the final sum, as
 
 ## Automatic reduction
 
-A *rule* calls an application's function for each new file as it arrives; the function pairs runs as the technique needs and submits what the file needs ([automatic-reduction.md](automatic-reduction.md)):
+Automatic reduction is a batch of `ess.apps.batch` reduced as its files arrive ([automatic-reduction.md](automatic-reduction.md)).
+The application's source pairs runs as the technique needs, and a row runs once for each new file at its trigger column:
 
 ```python
-def reduce_transmission(client: Client, run: FileRef) -> Record:
-    return client.submit(TRANSMISSION, params={'run': run, 'empty_beam': file(run=60336)},
-                         label='transmission', persist=True)
+def transmission_runs(files: Sequence[FileInfo]) -> list[Row]:
+    return [Row(str(f.metadata['run']), {'run': f.ref})
+            for f in files if f.metadata.get('role') == 'transmission']
 
-rule = Rule('transmission', reduce_transmission, selector=Selector(role='transmission'),
-            label='transmission', at='run')
-TriggerLoop(client, rules=[rule]).run()
+batch = Batch(client, TRANSMISSION, columns=('run',), label='transmission',
+              source=transmission_runs, persist=True)
+run([Automatic(batch, at='run', settings=lambda: {'empty_beam': file(run=60336)})])
 ```
 
-The loop reads which files a rule has handled from the changes, so a step reads only the records that changed since the last one.
+The loop reads from the changes which files the records of each row hold, so a restarted loop reduces no row again.
 
 ## Provenance and publication
 

docs/developer/adr/0002-a-value-lives-while-something-keeps-it.md

@@ -10,7 +10,7 @@ History holds records, not values ([ADR 0004](0004-history-is-append-only-lists.
 Values take memory, so something must decide when they go, and which are written.
 
 - **Two deployments.** Interactive work runs in the user's process (`local()`), on a VISA desktop or a laptop, and ending the process frees everything. The service runs batch and automatic reduction, and work spread over a cluster, for hours or days. The service must bound leaks and peak memory by design, not rely on users freeing memory ([users](../../requirements/users.md)).
-- **Unattended work.** The trigger loop submits through one client and never releases anything. A batch client closes, and users read its outputs the next morning.
+- **Unattended work.** The automatic reduction submits through one client and never releases anything. A batch client closes, and users read its outputs the next morning.
 - **Several connections.** A user works on one proposal through several clients at once, such as a notebook next to an app, or a web page and the same page reloaded.
 - **Writing costs.** A batch of 500 would write story S3's masked counts, an output meant for inspecting a reduction, once per record, 500 times. A volume that a web UI only shows cuts of (B4) is hundreds of GB. The framework cannot tell intermediate from final outputs.
 
@@ -43,7 +43,7 @@ So whether it is refused depends only on what the workspace keeps, not on how fa
 
 **Nothing is written unless persisted**, since the framework cannot tell which outputs are worth writing (Context).
 A client chooses per output, with `persist=` at submission or at `client.freeze`.
-The store is then the only keeper, as batch reduction and the trigger loop need.
+The store is then the only keeper, as batch reduction and automatic reduction need.
 A failed write fails the record, so a controller sees it like any other failed record.
 The store keys a value by record ID and output name, so a record holds no file name and a write changes no record.
 A persisted output stays readable after an upgrade removes its record's spec version, since the record stays in history, and is read as plain scipp data ([ADR 0009](0009-a-record-of-another-spec-stays-readable.md)).

docs/developer/adr/0004-history-is-append-only-lists.md

@@ -13,7 +13,7 @@ So history only grows until it is dropped.
 
 The stories need four things from history:
 
-- Records outlive the process of the workspace (B6, H2, H3). The trigger loop knows which files a rule has handled only from the records under the rule's label.
+- Records outlive the process of the workspace (B6, H2, H3). The automatic reduction knows which rows it has reduced only from the records under the batch's label.
 - A submission is stored whole or not at all.
 - The order of the records under a label, and of the pushes into an accumulator, is kept.
 - Persisted values are found again.
@@ -52,15 +52,15 @@ History is five lists, each only appended to:
 
 - **History defined as one event log, with records and labels as views**, indexes rebuilt from the log. The lists hold the same history. The log adds one order across all lists, and outside storage only the change feed needs an order, of the records alone. It would tie a hosted workspace to a log.
 - **A history held without a person kept as long as the proposal.** That may be forever, a second record of what ran next to SciCat.
-- **Each record dropped by age, keeping the older records that a kept record reads.** The trigger loop would reduce again every file whose records were dropped. It would need a memory of its own, such as a set of handled files per rule.
-- **A proposal's history dropped a fixed time after its last record.** A trigger loop that sees no new file for that long loses its rule's records and then reduces every file again. A client still in use would lose the records it reads.
+- **Each record dropped by age, keeping the older records that a kept record reads.** The automatic reduction would reduce again every row whose records were dropped. It would need a memory of its own, such as a set of reduced files per batch.
+- **A proposal's history dropped a fixed time after its last record.** An automatic reduction that sees no new file for that long loses its records and then reduces every row again. A client still in use would lose the records it reads.
 - **The record of a freeze stored with every row, in storage that shares repeated rows.** Sharing saves bytes, but every layer above the store still handles each row twice, once in its push and once in the record.
 - **Output values kept with their records.** This stores one output per state, and makes a catalogue of outputs next to SciCat, which is a non-goal.
 
 ## Consequences
 
 - Dropping a proposal's history leaves no dangling reference. No other proposal reads its records (system story G5), and an idle proposal gets no call and has no pending record.
-- A running trigger loop calls its client while it waits for changes, so its proposal is not idle. A loop started for a proposal whose history was dropped reduces its files again.
+- A running automatic reduction calls its client at each step, so its proposal is not idle. A loop started for a proposal whose history was dropped reduces its rows again.
 - A proposal that is never idle, such as one whose automatic reduction runs all year, keeps all its history.
 - A client whose process ended makes no call, so it does not keep its proposal from being idle.
 - A workspace that restarts does not know when its proposal was last called, so it counts idle time from its start.

docs/developer/adr/0007-packages-split-by-dependencies.md

@@ -77,8 +77,8 @@ essspec  <-  essapps  <-  frontends (over HTTP)
 
 - **Names follow the ess monorepo**: distribution `essX`, package `ess.X`, one level under `ess`. A second distribution cannot install into a regular package (one with an `__init__.py`) such as `ess.apps`, and a namespace package (one without, which may span distributions) could not export `local` and the rest. So `ess.apps.panel` exists only as part of the `essapps` distribution.
 - **`essspec` (`ess.spec`) is everything a workflow author writes against.** The vocabulary (`WorkflowSpec`, references, the functions that read field types, parameter models), the binding protocol (`Binding`, `StageBinding`, `AccumulatorBinding`, `Driver`), and the bindings `DriverBinding` and `function_binding` need only pydantic. A client imports `ess.spec` for the specs, and the bindings cost it nothing more. `ess.spec.conversions` needs scipp (extra `[scipp]`), and `ess.spec.pipeline` with `PipelineBinding` needs sciline (extra `[sciline]`). It is developed in this repository while it changes, and nothing outside this repository depends on a version of it. Once a second consumer, such as esslivedata or essnmx, is to adopt it, it moves to scipp/ess as a distribution of its own and is published from there.
-- **`essapps` (`ess.apps`) is what notebooks and apps use**: records, the client, the workspace and its log, files, the controllers (rules, the trigger loop), the batch toolbox `ess.apps.batch`, and fakes for tests. The core depends on essspec and pydantic, not on essreduce or sciline. The subpackages behind extras change in step with the core, so they are released with it. Importing one without its extra raises an `ImportError` that names the extra. A local workspace serves a web frontend on VISA or a laptop through `ess.apps.http.server`, as the service does. The trigger loop is in the core, and the function of a rule in the application that knows the technique; `ess.apps.service` runs the loop as a daemon from a configuration file.
-- **`essinstruments` (`ess.instruments`) holds one subpackage per instrument, and is released on its own schedule.** A fix to LoKI's specs ships without whatever changed in the core since its last release, and a core release ships without unreleased instrument changes. It depends on essspec, not on essapps. Each instrument has two modules: `specs.py` with the specs, and `bindings.py` with their bindings, which need the instrument's workflow package, behind an extra such as `[loki]`, so that an install for LoKI does not install the workflow packages of every other instrument. The two are separate because a spec may move into its workflow package without its binding, and a type checker checks the binding against the spec. `ess.instruments.loki` holds the specs and bindings of LoKI's reductions; `ess.loki`, from esssans, holds the workflow they bind to. A spec can move into its workflow package only after essspec has moved to scipp/ess, and only once its binding is a plain `PipelineBinding`, with no facility defaults or adapters. Deployment configuration, such as which automatic-reduction rules run and data paths, is not here; it belongs to the service's configuration.
+- **`essapps` (`ess.apps`) is what notebooks and apps use**: records, the client, the workspace and its log, files, the batch toolbox `ess.apps.batch` with its loop of automatic reduction, and fakes for tests. The core depends on essspec and pydantic, not on essreduce or sciline. The subpackages behind extras change in step with the core, so they are released with it. Importing one without its extra raises an `ImportError` that names the extra. A local workspace serves a web frontend on VISA or a laptop through `ess.apps.http.server`, as the service does. The loop of automatic reduction is in `ess.apps.batch`, and the source of rows in the application that knows the technique; `ess.apps.service` runs the loop as a daemon from a configuration file.
+- **`essinstruments` (`ess.instruments`) holds one subpackage per instrument, and is released on its own schedule.** A fix to LoKI's specs ships without whatever changed in the core since its last release, and a core release ships without unreleased instrument changes. It depends on essspec, not on essapps. Each instrument has two modules: `specs.py` with the specs, and `bindings.py` with their bindings, which need the instrument's workflow package, behind an extra such as `[loki]`, so that an install for LoKI does not install the workflow packages of every other instrument. The two are separate because a spec may move into its workflow package without its binding, and a type checker checks the binding against the spec. `ess.instruments.loki` holds the specs and bindings of LoKI's reductions; `ess.loki`, from esssans, holds the workflow they bind to. A spec can move into its workflow package only after essspec has moved to scipp/ess, and only once its binding is a plain `PipelineBinding`, with no facility defaults or adapters. Deployment configuration, such as which batches automatic reduction runs and data paths, is not here; it belongs to the service's configuration.
 - **A Python client that imports spec modules installs the instrument's workflow package.** A local workspace, a notebook, and a Panel app on VISA or a laptop run the bindings, so they have it anyway. A notebook against the service wants scipp and plotting for the outputs, so it runs in an ESS environment that has the workflow packages too. A client without workflow packages, such as a web frontend or the command-line tool with `--url`, reads `SerializedWorkflowSpec`: the JSON form of a spec, which names each data type as a string. A spec module imports scipp for the types of data: an array is a field of a group, typed by its scipp class, such as `intensity: sc.DataArray` of the group `IofQ`, so that a type checker checks a binding against the spec, and the workspace checks by the group's class that an output fits a field typed `Referenced[IofQ]`.
 - **`essapps` finds bindings through the entry-point group `ess.apps.workflows`.** An entry point is a name that a distribution registers under a group in its `pyproject.toml` (see the example below); `importlib.metadata.entry_points(group=...)` lists the entries of all installed distributions. Each entry names a function that returns a list of bindings, each of which names its spec. `local()` without `bind=`, the command-line tool, and the service read the group, so a distribution registers its workflows without importing the framework. Listing the specs this way imports the bindings; a client of the service gets the list from the service instead. An entry registers whether or not its extra is installed: an entry whose workflow package is missing is skipped, and `essapps specs` lists its specs as unavailable, naming the extra; any other import error fails the load. Two entries that register the same spec name and version fail the load, naming both entries, since picking one would leave records ambiguous about what ran. This is decided and not implemented.
 - **The command-line tool is a client in `essapps`**: `ess.apps.cli`, with the script `essapps`. It builds its options from `SerializedWorkflowSpec`, so it runs any registered spec in its own process, and with `--url` any spec of the service without importing a workflow package. Each parameter is one option, `--<parameter>`. A file field takes a file's identity or a path, `--sample run:loki/4711`. A group takes its fields as JSON typed from the params schema. A union of groups takes one alternative's fields, with its tag, as JSON typed from the params schema. A parameter `Referenced[T]` takes a record's output, such as `--centre r17.centre`. A union with a member `Referenced[...]` takes either, told apart by the argument: a JSON object, a file, or a record's output. A list parameter that is not a table takes a JSON list of its elements, typed from the params schema. A table takes a JSON or CSV file with one row per line. The tool needs only argparse, and `--url` needs `[remote]`. Writing outputs to files uses `ess.apps.writers`, which the service uses too ([ADR 0002](0002-a-value-lives-while-something-keeps-it.md)). A tool with instrument-specific arguments, such as essnmx's reducer, stays in its workflow package.

docs/developer/adr/0010-a-file-is-identified-by-its-catalogue-identifier.md

@@ -9,7 +9,7 @@
 A person names a file as `file(run=60339)`, and the file source resolves the name to the file's *identity*, which the record keeps and provenance lists.
 With paths as identities:
 
-- A moved folder changes every identity, so a trigger loop reduces its runs again.
+- A moved folder changes every identity, so the automatic reduction reduces its runs again.
 - The same run seen through two file sources has two identities.
 
 The catalogue is SciCat.
@@ -40,7 +40,7 @@ A person still writes `file(run=60339)`.
 - A moved folder, and a second file source, give a raw file the same identity.
 - Provenance lists raw files as the catalogue names them.
 - A file source reads the identifier from a raw file, or gets it from the catalogue.
-- A file first listed without its catalogue identifier and later with it changes identity: a trigger loop reduces it again, and the records made before name an identity the file source no longer knows.
+- A file first listed without its catalogue identifier and later with it changes identity: the automatic reduction reduces it again, and the records made before name an identity the file source no longer knows.
 - Identities are in the log, so changing them changes the log's format ([ADR 0004](0004-history-is-append-only-lists.md#decision)).
 
 ## Open