Write the architecture, emulator and Python bindings chapters

docs/architecture/index.md

@@ -1,69 +1,121 @@
 # Architecture
 
-:::{note}
-Expands in M3 with the thread model, the frame path and the instrument plugin contract in full. The
-layering below is accurate now.
-:::
-
-The C++ reference here is narrative and links to source rather than reproducing signatures.
+How the pieces fit together, and why acquisition is kept separate from output. The C++ reference
+here is narrative and links to source rather than reproducing signatures.
 
 ## Layers
 
 ```
-        client (socket, socksend, or Python module)
+        client (socket, camerad-socksend, or the Python module)
                           |
-   Camera::Server ........|...... command dispatch, ports, logging
+   Camera::Server .......................  one TCP port, command dispatch, logging
                           |
-   Camera::Interface .....|...... the camera, as commands
-        |         |
-        |         +------ Camera::ExposureMode ..... how one exposure runs
+   Camera::Interface ....................  the camera, expressed as commands
+        |            |              |
+        |            |              +-----  FrameOutput ....  FITS file, shared memory
+        |            |
+        |            +--------------------  Camera::ExposureMode ....  how one exposure runs
         |
-   Camera::Controller ....|...... the wire to the hardware
+   Camera::Controller ...................  the wire to the hardware
                           |
               Archon (TCP) or ARC (PCIe)
 ```
 
 `Camera::Server` ({source}`camerad/camera_server.h`)
-: Owns the ports and the command dispatch loop. Parses a command line, calls the matching
-  `Interface` method, and formats the reply. Knows nothing about detectors.
+: Owns the listening socket and the command dispatch chain. Parses a command line, calls the
+  matching `Interface` method, appends `DONE` or `ERROR`, writes the reply back. Knows nothing about
+  detectors. Each client connection is served on its own thread.
 
 `Camera::Interface` ({source}`camerad/camera_interface.h`)
-: The abstract camera. Declares one pure virtual per command: `expose`, `exptime`, `bias`, `bin`,
-  `load_firmware`, `power` and the rest. `ArchonInterface` and `AstroCamInterface` implement it for
-  the two controller families, and an instrument module subclasses one of those.
+: The abstract camera. Declares one pure virtual per command, and owns the config, the
+  `Camera::Information` for the current exposure, and the list of frame outputs.
+  `ArchonInterface` and `AstroCamInterface` implement it for the two controller families; an
+  instrument module subclasses one of those.
 
 `Camera::Controller`
-: The transport to the hardware, separate from `Interface` so that the command semantics and the
-  wire protocol can vary independently.
+: The transport to the hardware, kept separate from `Interface` so command semantics and wire
+  protocol vary independently.
 
 `Camera::ExposureMode` ({source}`camerad/exposure_modes.h`)
-: One way of running an exposure, as a producer/consumer pair: an acquisition thread pulls frames
-  from the controller, a processing thread turns them into images. A single `expose` command means
-  different things for a slow CCD readout and a continuously reading H2RG, and this is where that
-  difference lives. `Interface::select_expose_mode()` picks one.
+: One way of running an exposure. `ExposureModeTemplate<InterfaceType>` is the base each concrete
+  mode derives from; the template parameter gives a mode typed access to its interface. Each mode
+  owns its own `Camera::Information` for processed and unprocessed images, and an `ImageProcessor`
+  for deinterlacing.
+
+A single `expose` means different things for a slow CCD readout and a continuously reading H2RG,
+which is exactly what the exposure mode abstracts.
+
+## Acquisition: producer and consumer
+
+An exposure mode is a producer/consumer pair over a bounded queue:
+
+`image_acquisition_thread()`
+: The producer. Pulls frames off the controller as fast as the hardware delivers them and enqueues
+  them.
+
+`image_processing_thread()`
+: The consumer. Dequeues a frame, runs it through the deinterlacer, and hands the result to the
+  frame outputs.
+
+The base class carries the synchronization (`queue_mutex`, `queue_cv`, and the
+`is_producer_finished` / `is_producer_error` / `is_consumer_error` flags), so a concrete mode
+implements only the parts that differ. In the tracking camera module, for instance, a shared base
+owns the queue and the consumer and each subclass implements only the producer.
+
+Two lifetimes exist. For a counted exposure the consumer is one-shot: `do_expose()` spawns it,
+and it terminates once the producer is finished and the queue is drained. In freerun, one producer
+and one consumer run for the whole session and `expose` returns immediately, leaving them going
+until an abort or a producer error.
+
+## The frame path
+
+A processed frame is fanned out synchronously to every configured output by
+`Interface::dispatch_frame()`, which simply calls `write()` on each in turn. When the whole exposure
+command finishes, `end_exposure()` tells each output, so a multi-frame output such as a data cube
+can finalize its file.
+
+Outputs are built once at startup by `Interface::configure_frame_outputs()`, called from
+{source}`camerad/camerad.cpp` for every instrument. It is deliberately not virtual, so no derived
+class can silently skip wiring its outputs.
+
+### Why outputs never block
+
+`dispatch_frame()` runs on the consumer thread, so anything slow inside an output would stall
+acquisition. Each output is therefore required to return promptly, and absorbs the mismatch itself.
+
+The FITS writer is the clearest case ({source}`utils/fits_writer.h`): `write()` copies the pixels
+into a bounded queue and returns, never touching CCfits. A dedicated worker thread drains that queue
+to disk. When the queue is full the oldest frame is dropped. Disk is slower than acquisition can be,
+and the design chooses to lose a frame rather than apply backpressure to the detector.
+
+This is also why `FrameOutput::status()` is documented as a snapshot and never a barrier: letting a
+caller wait on an output would serialize acquisition behind disk I/O, reintroducing the coupling the
+queue exists to remove.
+
+The shared-memory output makes the opposite trade ({source}`utils/shared_memory_writer.cpp`): it
+copies the frame into the stream buffer and calls `ImageStreamIO_UpdateIm`, which posts the
+semaphores. Nothing it does depends on a reader, so a reader that cannot keep up misses frames but
+never delays the writer. Anything that must be woken per frame should attach there rather than watch
+for files.
+
+See [frame output keys](../configuration/frame-outputs.md) for configuring them.
 
 ## Instrument modules
 
-An instrument is a separate git repository, checked out as a submodule under
-`camerad/Instruments/<name>` and selected with `-DINSTRUMENT=`. It contributes four things:
+An instrument is a separate repository, checked out as a submodule under
+`camerad/Instruments/<name>` and selected with `-DINSTRUMENT=`. It contributes:
 
-1. A `<name>.cmake` fragment setting `INSTRUMENT_SOURCES`, which is the whole of the build
-   integration.
-2. An `Interface` subclass, deriving from `ArchonInterface` or `AstroCamInterface`, overriding what
-   the detector needs and adding instrument commands via `instrument_cmd()` and
-   `is_instrument_command()`.
-3. Its own `ExposureMode` implementations, if the stock ones do not fit.
+1. A `<name>.cmake` fragment setting `INSTRUMENT_SOURCES`. That is the whole build integration.
+2. An `Interface` subclass deriving from `ArchonInterface` or `AstroCamInterface`, overriding what
+   the detector needs and adding commands through `instrument_cmd()` and `is_instrument_command()`.
+3. Its own `ExposureMode` implementations, where the stock ones do not fit.
 4. Optionally a FITS header dictionary and shipped `.cfg` and `.acf` files.
 
-An interface factory function ties it together, so the core builds against the base class and never
-names a concrete instrument.
+An interface factory ties it together, so the core builds against the base class and never names a
+concrete instrument.
 
-See [instruments](../instruments/index.md) for the four that exist, and
-{source}`camerad/Instruments/hispec_tracking_camera` for the most complete example.
+Instrument commands are checked *before* the base command chain, so an instrument can add commands
+and also override a base one of the same name.
 
-## Frame path
-
-Acquisition is deliberately decoupled from output. The exposure mode's processing thread publishes a
-completed frame to every configured [frame output](../configuration/frame-outputs.md); each output
-owns its own queue and thread. Nothing an output does can block acquisition, which is why the FITS
-writer drops frames instead of applying backpressure.
+See [instruments](../instruments/index.md) for the modules that exist, and
+{source}`camerad/Instruments/hispec_tracking_camera` for the most complete example.

docs/emulator/index.md

@@ -1,11 +1,7 @@
 # Emulator
 
-`camerad-emulator` stands in for an Archon controller so the server, an instrument module and a
-client can all be exercised with no hardware. CI uses it for the end-to-end tests.
-
-:::{note}
-Expands in M3 with the emulated command list and the limits of the emulation.
-:::
+`camerad-emulator` impersonates an Archon controller over TCP, so the server, an instrument module
+and a client can all be exercised with no hardware. CI uses it for the end-to-end tests.
 
 There is no ARC emulator. `-DINTERFACE_TYPE=AstroCam` therefore builds none, and the default
 `-DINTERFACE_TYPE=Archon` builds this one regardless of which `CONTROLLER` was selected.
@@ -16,24 +12,60 @@ There is no ARC emulator. `-DINTERFACE_TYPE=AstroCam` therefore builds none, and
 camerad-emulator <file.cfg> -i <instrument>
 ```
 
-It reads `EMULATOR_PORT` and `EMULATOR_SYSTEM` from the same `.cfg` the server uses, so the only
-thing that makes a config point at the emulator rather than hardware is `ARCHON_IP` and
-`ARCHON_PORT`. `-i generic` suits the shipped test configs.
+It reads its keys from the same `.cfg` the server uses, so the only thing separating a test rig from
+real hardware is where `ARCHON_IP` and `ARCHON_PORT` point. `-i generic` suits the shipped test
+configs.
+
+```{eval-rst}
+.. camerad-config-keys:: Emulator
+   :widths: 30 70
+```
 
 `EMULATOR_SYSTEM` names a `.system` file describing the module complement to report, which is what
-lets the emulator answer `SYSTEM` convincingly for a given instrument.
+lets the emulator answer `SYSTEM` convincingly for a given instrument. `config/demo/demo.system` is
+the shipped example.
+
+## What it emulates
+
+The emulator answers the Archon command set the server actually uses:
+
+Configuration
+: `WCONFIG`, `RCONFIG`, `CLEARCONFIG`, `APPLYALL`, `APPLYMOD`, `APPLYDIO`
+
+Parameters
+: `LOADPARAM`, `PREPPARAM`, `FASTLOADPARAM`, `FASTPREPPARAM`
 
-## What it models
+Timing and power
+: `RESETTIMING`, `HOLDTIMING`, `RELEASETIMING`, `POWERON`, `POWEROFF`
 
-The emulator answers the Archon command set the server uses: configuration load, parameter writes,
-frame status, and pixel delivery on the timing the exposure implies.
+Status and data
+: `STATUS`, `SYSTEM`, `FRAME`, `TIMER`, `FETCH`, `LOCK`, `POLLON`, `POLLOFF`
 
-Frames carry synthetic pixel data by default, so it validates plumbing, geometry, timing and headers
-rather than image quality. Point `EMULATOR_DATADIR` at a directory of real frames to have it serve
+It tracks configuration memory and parameters across `WCONFIG` and `LOADPARAM`, so a timing script
+loaded by `load` reads back the way the server expects, and it delivers pixels on timing derived
+from the exposure and readout parameters rather than instantly.
+
+## Pixel data
+
+Frames carry synthetic pixel data by default, which validates plumbing, geometry, timing and headers
+rather than image quality. Setting `EMULATOR_DATADIR` to a directory of real frames makes it serve
 those instead, which is what makes it useful for exercising downstream processing.
 
-Sources are in {source}`emulator`, with the per-detector frame sources alongside
-(`generic.h`, `nirc2.h`).
+Per-detector frame sources live alongside the emulator ({source}`emulator/generic.h`,
+{source}`emulator/nirc2.h`) and are selected by `-i`. They also parse their own keys from the
+config, including `READOUT_TIME` and the pixel and row timings that set how fast a readout appears
+to proceed.
+
+:::{note}
+Those detector-description keys are a separate namespace from the server configuration keys in the
+[configuration reference](../configuration/index.md), even though they are read from the same file.
+:::
+
+## Limits
+
+The emulator models the command and data protocol, not the controller. It does not reproduce
+electrical behaviour, real detector noise, or the failure modes of a misconfigured ACF. A timing
+script that is wrong in a way the Archon would reject may still appear to work here.
 
 ## In CI
 
@@ -43,5 +75,11 @@ Two workflow jobs run against it on every push and pull request:
   validates the written header and reads the shared-memory segment back.
 - A Python module test that drives the same exposure through the `camera_interface` bindings.
 
-Both are in `.github/workflows/emulator-integration.yml` and are the closest thing to a
-regression suite for the acquisition path.
+Both are in `.github/workflows/emulator-integration.yml` and are the closest thing the project has
+to a regression suite for the acquisition path.
+
+:::{tip}
+The frame outputs job is also the most complete worked example of a full setup: it builds with
+shared memory enabled, starts the emulator, starts `camerad` against it, drives an exposure with
+`camerad-socksend`, then checks both outputs. Read it when a local setup misbehaves.
+:::