Give RAW captures their own output stream and gate reads on RAWENABLE

.github/workflows/emulator-integration.yml

@@ -84,6 +84,35 @@ jobs:
         python3 python/tests/fits_header_check.py \
           /tmp/ci_fits_test/ci_frame_outputs_*.fits --exptime 1.5
 
+        image_size() {
+          bin/camerad-shm-reader ci_frame_outputs_shm /tmp/ci_milk_shm \
+            | grep -o 'size=[0-9]*x[0-9]*' | head -1
+        }
+        before=$(image_size)
+
+        # read must refuse to fetch until capture is actually enabled
+        send "raw read" | grep -q "ERROR" || { echo "raw read ran with RAWENABLE=0"; exit 1; }
+
+        # The Archon captures raw alongside the frame, so enabling it has to
+        # precede the exposure it should appear in
+        send "raw set RAWENABLE 1"
+        resp=$(send "expose 1")
+        echo "expose -> $resp"
+        echo "$resp" | grep -q "DONE" || exit 1
+
+        resp=$(send "raw read")
+        echo "raw read -> $resp"
+        echo "$resp" | grep -q "DONE" || exit 1
+        sleep 1
+
+        # RAW has its own geometry, so it has to land in its own outputs rather
+        # than collide with the image file or resize the image's SHM stream
+        bin/camerad-shm-reader ci_frame_outputs_shm_raw /tmp/ci_milk_shm
+        ls /tmp/ci_fits_test/ci_frame_outputs_*_raw.fits
+
+        after=$(image_size)
+        test "$before" = "$after" || { echo "image stream changed: $before -> $after"; exit 1; }
+
         pkill -f 'bin/camerad' || true
         pkill -f 'bin/camerad-emulator' || true
 

README.md

@@ -237,6 +237,10 @@ Logging follows `LOG_STDERR` from the `.cfg`; pass `log_to_stderr=` to override
 
 Every instrument publishes each acquired frame to one or more outputs, configured entirely via `.cfg` file keys (`Camera::Interface::configure_frame_outputs()` builds them from `Camera::apply_config_overrides()`, called once at startup for every instrument, not just HISPEC). Both outputs are independent; either, both, or neither can be enabled per instrument.
 
+### Streams
+
+Every frame carries an optional stream name so that one exposure can deliver outputs of different geometry without them colliding. The primary image leaves it empty; an Archon `raw read` sets it to `raw`. FITS appends the name to the filename (`image_00000123_raw.fits`) and shared memory appends it to the segment name (`camera_raw`), so a RAW capture neither overwrites the image file nor resizes the image stream.
+
 ### FITS
 
 Writes one FITS file per frame asynchronously (a queue plus a dedicated writer thread, so the readout thread never blocks on disk I/O).
@@ -261,7 +265,7 @@ Publishes each frame as an [ImageStreamIO](https://github.com/milk-org/ImageStre
 | `SHM_RING_BUFFER_SIZE`  | `4`        | Depth of ImageStreamIO's internal history ring buffer (`CBsize`); the live frame a real-time reader sees is separate from this |
 | `SHM_DIR`               | (unset)    | Base directory ImageStreamIO writes into. If unset, ImageStreamIO falls back to its own default resolution (`MILK_SHM_DIR` env var, then `/milk/shm`). If set, it must already exist and be writable. |
 
-Frame geometry (width/height/pixel depth) isn't a config key: it's fixed for an ImageStreamIO stream's whole life, so the writer (re)creates the stream automatically whenever it sees the geometry change from what's currently allocated.
+Frame geometry (width/height/pixel depth) isn't a config key: it's fixed for an ImageStreamIO stream's whole life, so the writer (re)creates a stream automatically whenever it sees the geometry change from what's currently allocated for that stream.
 
 Two readers ship with the repo. `camerad-shm-reader` prints geometry, keywords and pixel statistics once, for diagnostics. `python/examples/shm_read_frames.py` is a sample streaming consumer: it blocks on the stream's semaphore and reports every frame as it arrives, flagging any it missed.
 

camerad/archon_controller.cpp

@@ -1791,6 +1791,7 @@ namespace Camera {
       this->get_configmap_value("BIGBUF", mode->bigbuf);
       this->get_configmap_value("FRAMEMODE", mode->geometry.framemode);
       this->get_configmap_value("RAWENABLE", mode->rawenable);
+      this->get_configmap_value("RAWENABLE", this->rawinfo.enable);
       this->get_configmap_value("RAWSEL", this->rawinfo.adchan);
       this->get_configmap_value("RAWSAMPLES", this->rawinfo.samples);
       this->get_configmap_value("RAWSTARTLINE", this->rawinfo.startline);
@@ -2344,8 +2345,9 @@ namespace Camera {
     geom.samples         = static_cast<uint32_t>(this->rawinfo.samples);
     geom.blocks_per_line = static_cast<uint32_t>(this->frameinfo.bufrawblocks[index]);
     geom.lines           = static_cast<uint32_t>(this->frameinfo.bufrawlines[index]);
+    geom.from_config     = (geom.blocks_per_line == 0 || geom.lines == 0);
 
-    if (geom.blocks_per_line == 0 || geom.lines == 0) {
+    if (geom.from_config) {
       geom.blocks_per_line =
         (static_cast<size_t>(geom.samples) * sizeof(uint16_t) + BLOCK_LEN - 1) / BLOCK_LEN;
       const int span = this->rawinfo.endline - this->rawinfo.startline + 1;
@@ -2430,6 +2432,7 @@ namespace Camera {
     }
 
     // refresh cached geometry from the now-updated configmap
+    this->get_configmap_value("RAWENABLE", this->rawinfo.enable);
     this->get_configmap_value("RAWSEL", this->rawinfo.adchan);
     this->get_configmap_value("RAWSAMPLES", this->rawinfo.samples);
     this->get_configmap_value("RAWSTARTLINE", this->rawinfo.startline);
@@ -2441,6 +2444,35 @@ namespace Camera {
   /***** Camera::ArchonController::set_raw_config **************************/
 
 
+  /***** Camera::ArchonController::raw_frame_keys **************************/
+  /**
+   * @brief      build the provenance keys describing a RAW capture
+   * @details    An AD channel and an ADM channel arrive as an identical uint16
+   *             block sampled at different rates, so the samples cannot be
+   *             interpreted without RAWSEL and the installed module types.
+   * @return     shared_ptr suitable for FrameMetadata::frame_keys
+   */
+  std::shared_ptr<const Common::FitsKeys> ArchonController::raw_frame_keys() const {
+    auto keys = std::make_shared<Common::FitsKeys>();
+    keys->addkey("RAWSEL", this->rawinfo.adchan, "Archon RAWSEL, channel captured");
+
+    // RAWSEL's channels per slot is 4 in the manual but 18 in the Archon GUI,
+    // so report every candidate slot rather than derive one from an unknown stride
+    for (int slot = AD_SLOT_FIRST;
+         slot <= AD_SLOT_LAST && slot <= static_cast<int>(this->modtype.size()); ++slot) {
+      keys->addkey("RAWMOD"+std::to_string(slot), this->modtype[slot-1],
+                   "MODn_TYPE of slot "+std::to_string(slot));
+    }
+
+    keys->addkey("RAWSAMP",  static_cast<int>(this->rawinfo.samples),    "RAWSAMPLES");
+    keys->addkey("RAWSLINE", static_cast<int>(this->rawinfo.startline),  "RAWSTARTLINE");
+    keys->addkey("RAWELINE", static_cast<int>(this->rawinfo.endline),    "RAWENDLINE");
+    keys->addkey("RAWSPIX",  static_cast<int>(this->rawinfo.startpixel), "RAWSTARTPIXEL");
+    return keys;
+  }
+  /***** Camera::ArchonController::raw_frame_keys **************************/
+
+
   /***** Camera::ArchonController::read_raw *******************************/
   /**
    * @brief      retrieve RAW (pre-CDS) data from the newest frame buffer
@@ -2452,6 +2484,14 @@ namespace Camera {
   long ArchonController::read_raw(std::string &retstring) {
     const std::string function("Camera::ArchonController::read_raw");
 
+    // Without this the fetch returns whatever sits at the raw offset, since a
+    // controller with capture off is indistinguishable from one reporting nothing
+    if (this->rawinfo.enable == 0) {
+      logwrite(function, "ERROR RAW capture is disabled");
+      retstring = "RAW capture is disabled; set it with \"raw set RAWENABLE 1\"";
+      return ERROR;
+    }
+
     long error = this->get_frame_status();
     if (error != NO_ERROR) {
       logwrite(function, "ERROR getting frame status");
@@ -2466,6 +2506,13 @@ namespace Camera {
       return ERROR;
     }
 
+    // Raw is captured with the frame, so a buffer filled before RAWENABLE was set
+    // holds none and reports zero, leaving only the config keys to size the fetch
+    if (geom.from_config) {
+      logwrite(function, "WARNING controller reports no raw data in this buffer; "
+                         "using configured geometry");
+    }
+
     const size_t fetch_bytes = static_cast<size_t>(geom.blocks_per_line) * geom.lines * BLOCK_LEN;
     std::shared_ptr<char[]> raw_buffer(new char[fetch_bytes]);
     char* bufptr = raw_buffer.get();
@@ -2496,6 +2543,8 @@ namespace Camera {
     meta.width           = geom.samples;
     meta.height          = geom.lines;
     meta.bytes_per_pixel = sizeof(uint16_t);
+    meta.stream          = RAW_STREAM;
+    meta.frame_keys      = this->raw_frame_keys();
     this->interface->dispatch_frame(reinterpret_cast<const char*>(samples.data()),
                                     payload_bytes, meta);
 

camerad/archon_controller.h

@@ -47,6 +47,13 @@ constexpr int MODTYPE_ADLN    = 15;
 constexpr int MODTYPE_UNKNOWN = 16;
 constexpr int MODTYPE_ADM     = 17;
 
+// AD and ADM modules are restricted to slots 5-8, the range RAWSEL addresses
+constexpr int AD_SLOT_FIRST = 5;
+constexpr int AD_SLOT_LAST  = 8;
+
+// Stream label keeping a pre-CDS RAW capture in outputs of its own
+const std::string RAW_STREAM = "raw";
+
 /**
  * Archon commands
  */
@@ -253,6 +260,7 @@ namespace Camera {
 
       /** @brief Archon RAW (pre-CDS) capture configuration, mirrors ACF keywords */
       struct rawinfo_t {
+        int      enable{0};      // RAWENABLE: whether the controller captures raw at all
         int      adchan{0};      // RAWSEL: AD channel captured
         uint16_t samples{0};     // RAWSAMPLES: 16-bit samples per line
         uint16_t startline{0};   // RAWSTARTLINE
@@ -377,13 +385,15 @@ namespace Camera {
         uint32_t samples;          // valid 16-bit samples per line (RAWSAMPLES)
         uint32_t blocks_per_line;  // 1024-byte blocks per line, padded per Archon
         uint32_t lines;            // number of raw lines (RAWENDLINE-RAWSTARTLINE+1)
+        bool from_config{false};   // controller reported nothing, so this is inferred
       };
 
       static bool is_raw_config_key(const std::string &key);
       raw_geometry_t raw_geometry() const;
       uint32_t raw_frame_bytes() const;   // padded, size-aware byte count for a RAW fetch
       long set_raw_config(const std::string &args, std::string &retstring);
       long get_raw_config(std::string &retstring);
+      std::shared_ptr<const Common::FitsKeys> raw_frame_keys() const;
       long read_raw(std::string &retstring);
 
 

camerad/archon_interface.cpp

@@ -1002,6 +1002,7 @@ namespace Camera {
       retstring.append( "  set <KEY> <VAL> ..  set RAW keyword(s) then apply\n" );
       retstring.append( "  read                retrieve RAW data in-band as 16-bit samples\n" );
       retstring.append( "  Keys: RAWENABLE RAWSEL RAWSTARTLINE RAWENDLINE RAWSTARTPIXEL RAWSAMPLES\n" );
+      retstring.append( "  RAWENABLE must be set before the exposure the raw data comes from\n" );
       return HELP;
     }
 

docs/commands/controller.md

@@ -27,10 +27,10 @@ confirming that a reply came back. The commands most worth knowing:
 The server itself parses `FRAME`, `STATUS` and `SYSTEM` replies for its own bookkeeping, so use
 `native` for inspection rather than as a control path.
 
-Beyond `native`, the Archon-specific commands are `raw` for configuration memory, `getp` and `setp`
-for parameters, `inreg` for a VCPU input register, `loadtiming` and `readacf` for loading, `mode`
-for camera modes, `autofetch_mode`, and `heater` and `sensor` for the thermal modules. The
-[heater and sensor](#heater-and-sensor) syntax is below.
+Beyond `native`, the Archon-specific commands are `raw` for pre-CDS sample capture, `getp` and
+`setp` for parameters, `inreg` for a VCPU input register, `loadtiming` and `readacf` for loading,
+`mode` for camera modes, `autofetch_mode`, and `heater` and `sensor` for the thermal modules. The
+[raw samples](#raw-samples) and [heater and sensor](#heater-and-sensor) syntax is below.
 
 ## ARC (AstroCam)
 
@@ -45,6 +45,92 @@ and returns success, so the command replies `DONE` while doing nothing.
 Passing three-letter DSP commands through to an ARC controller is therefore not currently possible
 from the command interface.
 
+## Raw samples
+
+The Archon can capture a window of unprocessed ADC samples from a single channel alongside the
+normal CDS frame, which is how a detector engineer inspects the output waveform for settling times,
+sampling windows, reset level stability and clock feed through. `raw` configures and retrieves it.
+
+```
+raw [ config | set <KEY> <VAL> [...] | read ]
+```
+
+| Form | Effect |
+|---|---|
+| `raw config` | Report the six RAW keywords |
+| `raw set <KEY> <VAL> ...` | Write the keyword(s) to configuration memory, then apply |
+| `raw read` | Fetch the raw region of the newest buffer and dispatch it as a frame |
+
+The keywords are `RAWENABLE`, `RAWSEL`, `RAWSTARTLINE`, `RAWENDLINE`, `RAWSTARTPIXEL` and
+`RAWSAMPLES`. `RAWSAMPLES` is rounded up to a whole 1024-byte block per line.
+
+:::{warning}
+`RAWENABLE` takes effect at capture time, not at read time. The controller acquires raw samples
+while it fills the frame buffer, so enabling it after an exposure cannot populate that buffer. Set
+it before the exposure the samples should come from.
+
+`raw set` can only write keywords that already exist in the loaded ACF, so an ACF that omits them
+cannot be driven from the command interface at all.
+:::
+
+`raw read` refuses while `RAWENABLE` is 0. A controller with capture disabled reports zero raw
+blocks and lines, which is indistinguishable from one that reports nothing, so the fetch would
+otherwise return whatever happens to sit at the raw offset and label it as raw data.
+
+The result is dispatched on its own stream, named `raw`, so it never collides with the image. The
+FITS writer gives it a separate file and the shared-memory writer a separate segment. See
+[frame output keys](../configuration/frame-outputs.md).
+
+### Interpreting the samples
+
+An AD channel and an ADM channel both arrive as an identical block of `uint16`, but they are not
+sampled the same way:
+
+AD
+: 16 bits at 100 MHz, so consecutive samples are 10 ns apart.
+
+ADM
+: 18 bits at 12.5 MHz, truncated to 16 bits, with each value repeated eight times and dithered so
+  that averaging the eight recovers the original 18-bit sample. The effective period is 80 ns.
+
+Nothing in the data itself distinguishes the two, so `raw read` records the provenance in the
+header instead.
+
+| Keyword | Meaning |
+|---|---|
+| `RAWSEL` | The channel selector in force for the capture |
+| `RAWMOD5` to `RAWMOD8` | `MODn_TYPE` of each slot `RAWSEL` can address |
+| `RAWSAMP` | `RAWSAMPLES` |
+| `RAWSLINE`, `RAWELINE` | `RAWSTARTLINE`, `RAWENDLINE` |
+| `RAWSPIX` | `RAWSTARTPIXEL` |
+
+Every candidate slot is reported rather than the one slot `RAWSEL` selects, because the mapping
+from `RAWSEL` to a slot is not reliably known. See below.
+
+### Module types and the RAWSEL range
+
+The Archon manual is incomplete here and partly stale, and the GUI source distributed with the
+controller is the better reference. `archongui/src/archon.h` carries the full module type list,
+where the manual stops at `16+: Unknown`:
+
+| Type | Module |
+|---|---|
+| 6 | Atlas |
+| 16 | DriverX |
+| 17 | ADM |
+| 18 | Unknown, the sentinel |
+
+`archongui/src/archongui.cpp` then builds its Raw Channel Select control with
+`for (i = 1; i <= 72; i++)` and stores the zero-based index as `RAWSEL`, so the selector spans 0 to
+71. That is 18 channels across each of slots 5 to 8, matching the ADM channel layout. The manual
+instead documents `RAWSEL` as 0 to 15, four per slot, described purely in terms of AD modules.
+
+:::{note}
+The two strides disagree, and on a chassis with a mix of AD and ADM modules they resolve the same
+`RAWSEL` to different slots. `camerad` therefore derives nothing from `RAWSEL` and records it
+verbatim beside the type of every slot it could refer to, leaving the reader to resolve it.
+:::
+
 ## Heater and sensor
 
 For Archon **Heater** and **HeaterX** modules. Both require firmware to be loaded and a

docs/fits/index.md

@@ -9,12 +9,16 @@ in [frame output keys](../configuration/frame-outputs.md).
 The writer builds each name itself:
 
 ```
-<FITS_OUTPUT_DIR>/<FITS_BASENAME>_<frame number>.fits
+<FITS_OUTPUT_DIR>/<FITS_BASENAME>_<frame number>[_<stream>].fits
 ```
 
 The frame number is zero-padded to eight digits. If that path already exists the writer appends
 `_1`, `_2` and so on until it finds a free name, so a file is never silently overwritten.
 
+The stream is omitted for the primary image and present for anything published alongside it, such
+as the `raw` stream from [raw samples](../commands/controller.md#raw-samples). It comes last so a
+frame and its companions sort together.
+
 `FITS_AUTODIR` puts all of this inside a `YYYYMMDD` subdirectory of `FITS_OUTPUT_DIR`.
 
 :::{warning}