Tracking: settle disk formats, deployments, and where reductions run #184

A review of the design on 2026-11-12 (core, performance, spec interface, applications, requirements coverage) found the core mechanism sound: records, the history log, the two-phase check, the change feed, specs as two pydantic models, and the binding protocols. The open points are where the system meets something outside one Python process. This issue tracks the steps that settle them, in order. #155 stays the map of components.

Steps

• 0. Stopgaps and small fixes. A format number in the log, so that a folder written in another format is refused with a message. A stored group that lacks a required field is refused when read. Client.wait asks only about records that changed. Merged in Log format number, stored groups checked on read, and a linear Client.wait #185. Left out: refusing a pipeline parameter that no output depends on, since PipelineBinding accepts an unused blank on purpose; to be decided.
• 0. Component map. Tracking: the components and interfaces of essapps, and how far each has got #155 brought up to date.
• 1. One drawing per deployment. Which process holds the application, the client, and the workspace, for a notebook, a technique application with a batch page, and automatic reduction: docs/developer/deployments.md, merged in docs: say which process holds what in each deployment #186. Decided: application logic is Python that holds a client in the workspace's process, and a screen never calls a workspace; one automatic reduction per instrument with a workspace per proposal; a value passes between workspaces as a file; a process split, if needed, goes between workspace and workers first.
• 2. Where reductions run: a measured spike. Measured on real LoKI reductions (numbers in a comment below). The workspace's threads do not speed up a workflow that is parallel inside, and multiply its memory; the stage gives the gain; worker processes give 1.4 to 1.9 times on plain requests and lose to the in-process stage. Decided: one workspace thread by default, settable per application, since the best value depends on the instrument. Worker processes are deferred until a deployment needs their isolation. Done from it: the LoKI bindings build the esssans workflow once (perf: build the LoKI workflow once per process #187); one workspace thread by default (perf: one workspace thread by default #188).
• 3. ADR: what is durable. Decided in docs: ADRs on spec identity and file identity; log format, store, and retention stated #192: a record names its spec by name, version, and digest, and a record of another spec stays readable as stored data (ADR 0009); a file is identified by its catalogue identifier, or else by its path (ADR 0010); a log reads only files of its own format, and a history is dropped only where no person holds it (ADR 0004); the store is an interface and a file format belongs to each stored value (ADR 0002). Implemented: the digest of a spec and the check for technique packages (feat: a digest of a spec's interface, and a check that a changed interface gets a new version #193), the digest in the log and reading a record of another spec (feat: a record of another spec stays readable as stored data #194), file identity (feat: a file is identified by its catalogue identifier, or else by its path relative to the folder #195). All merged. Moved out: what a batch is and how its definition is tied to its records (What a batch is: how its definition is tied to its records #191). Done ahead of the ADR: a record that was running when its workspace ended is failed at the next start, not run again (feat: fail a record that was running when its workspace ended #189).
• 4. One model for batch and automatic reduction. Decided: one model. Automatic reduction is a batch reduced as its files arrive, and a person and the automatic reduction differ in one predicate (which rows run); Rule and TriggerLoop are removed (Automatic reduction is a batch reduced as its files arrive #196, open). A step is linear in the number of files; what would make it cost less is listed as open in docs/developer/automatic-reduction.md. Remaining: one batch presenter; one plot contract for technique packages (Plots and slicing in applications #149); Automatic reduction as an application #151 rewritten around the batch.
• 5. A second technique. BIFROST is the candidate, since it exercises a large volume and an accumulator with a real driver (An accumulating PipelineBinding built on StreamProcessor #40).

Steps 1 and 2 come before 3, because they decide what must be durable and where values live.

The open points behind the steps

Assumptions made (Simon, 2026-11-12)

• How a file says what role a run plays is not settled at ESS. Applications assume a rule and let the user override it, as the batch table does.
• The automatic-reduction service may create derived datasets in SciCat.
• Only automatic reduction must run as a service. Batch reduction may run as an application that a user starts on a VISA machine, and for a first release automatic reduction may too (Simon, 2026-11-13).
• Automatic reduction has a base configuration per instrument, adjusted per proposal or within a proposal.
• How services are deployed at DMSC is not settled.

Not in scope now

connect() and everything hosted (#27), the cluster executor, the choice of widget library, detector banks (#166), the serialized spec schema (#88, #118), technique export formats beyond the writer interface (#176).