[SimonHeybrock]

Step 4 of #184, first part: automatic reduction and batch reduction are one model. ess.apps.rules (Rule, RuleStatus, TriggerLoop) is removed. Automatic reduction is a Batch that ess.apps.batch.auto.Automatic reduces as its files arrive.

auto = Automatic(iofq_batch(client, label='auto/iofq'), at='sample', settings=lambda: layers('iofq'))
auto.step()      # scans, and reduces each row with a new file at the trigger column; records by key
auto.refused     # why each row with a new file did not run, by key
run([auto], interval=10.0)

Why

Rules predate the batch. A rule's submit(client, file) did three things in one function per file: pair the file with its partners, decide whether the file was handled, and submit. The batch has the first as a row source (a function from the files to rows, tested on plain data) and the third as Batch.reduce. So a technique wrote its pairing twice, once as a row source and once as a rule function, and automatic reduction had none of what the batch gives: a state per row that says why a run waits, a stage for what the rows share, and the views.

Which rows run

A person and the automatic reduction differ in one predicate over the same join of rows with the records under the batch's label:

A row runs if	no result	failed or cancelled	a setting or another cell changed	a new file at the trigger column
a view, "reduce remaining"	yes	yes	yes (stale)	yes
essapps-batch reduce --remaining	yes	yes	no	no
Automatic	yes	no	no	yes

Automatic reads which files the records of each row hold at the trigger column from the proposal's changes, so a restart needs nothing on disk. Records of any spec and any submitter count. A new label has no records, so it reduces from the first file.

What behaves differently from rules

• A chain (a sample row that references the output of the transmission row) is two batches, the second with a source that reads the state of the first. It was one function call.
• A row with a new file whose record is still running waits, and is submitted when that record finishes. Rules submitted a second record next to the running one.
• One refused row does not hold back the others: Automatic drops the rows a refusal names and submits the rest. Batch.reduce stays all or none for a person.
• A step that raises (the source, the settings, or the submission) loses nothing: what runs is worked out from the records at every step, and the refused rows only suppress a retry until their cells, their records, or the settings change.
• A cell at the trigger column must hold the identity of its file, as a source gets it in FileInfo.ref. A cell that names a file, such as file(run=1), would never match what its record holds, and a client has no call to resolve the name. step raises ValueError for such a row after its first record, and at every later step before it submits. This takes the place of the check that a rule's record settles its file.

Changes to the batch

• Batch.state(settings=None, keys=None) works out the state of chosen rows, so that reduce does not work out every row. It still reads every record under the label.
• state.results does not filter by submitter: one folder or one service holds one history.
• An unknown record in a cell affects the state of its own row only.

Cost

The requirement, from Simon: some instruments will write a file every few seconds, so 10 000 to 100 000 files per proposal. One step with one new file, toy workflow, fake file source, on a shared machine:

runs reduced	best of ten	mean of ten	of which listing the files
3 000	15 ms	30 ms	5 ms
30 000	214 ms	401 ms	82 ms
100 000	822 ms	1.4 s	324 ms

A step is linear in the number of files: it lists them, runs the source over them, and checks each row against the files its records hold. TriggerLoop was linear too (272 ms at 100 000), since it listed all files at each step. A step costs the same for one new file or fifty, so the interval of run sets the load. The mean is about twice the best because of Python's garbage collector.

docs/developer/automatic-reduction.md lists what is open and not built: a file source that reports what is new since a position; two ways to make a step cost less than linear (a group per file that the source's author declares, or a source that makes one row by queries on an index that records what each row asked for); a range in Selector to start a new label at a given run; running old rows again for a new spec version; reading the records of chosen rows only. It also says why the history is not cut off by age or count to bound the cost.

Known limits

• A view reads Batch.state, which counts results of the batch's spec version only, while Automatic counts any record under the label. After a new spec version a view would show old rows as not reduced while Automatic holds them. Related to What a batch is: how its definition is tied to its records #191.
• Batch keeps each submitted request in memory for "stale" (What a batch is: how its definition is tied to its records #191), so a long-running Automatic holds one request per row.
• A source that reads another batch's state() works out every row of that batch at each step.
• Not measured: a folder workspace, a restart at 100 000 records, memory. run has no test.

Review

A fresh reviewer probed the branch and mutated the code. Fixed from it: a step that raised lost its rows until a restart; a trigger cell that named a file was submitted at every step; cost statements that were false; state(keys=...) differing from state() for a row; documented behaviours with no test that failed when they broke.

Remaining in step 4 of #184: one batch presenter, and one plot contract for technique packages (#149).

🤖 Generated with Claude Code