Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/source/aind/core_services.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Core Services

The interactions between many of these services are illustrated in the [AIND software diagrams](./diagrams.md).
The interactions between many of these services are illustrated in the [AIND software diagrams](../diagrams/index.md).

**aind-data-transfer-service**

Expand Down
11 changes: 0 additions & 11 deletions docs/source/aind/diagrams.md

This file was deleted.

13 changes: 13 additions & 0 deletions docs/source/diagrams/data_acquisition.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Data Acquisition

:::{note}
This diagram is a **proposal** for the instrumentation software architecture — a
planned design rather than a system currently in production.
:::

This diagram presents a proposed software architecture for AIND instrumentation
during data acquisition. It outlines how the acquisition software components are
intended to interact to control the instrument and capture data during an
experiment.

![SIPE instrument diagram](mid_level/sipe_instrument.drawio.svg)
28 changes: 28 additions & 0 deletions docs/source/diagrams/data_staging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Data Staging

The local data center hosts a variety of services that stage and process data on
its way to the cloud:

`aind-data-transfer-service`
: FastAPI service that accepts and validates data upload requests, retrieves the
associated metadata, and dispatches compression and transfer jobs to the workflow
manager for execution on the HPC cluster.

`aind-airflow-service`
: Apache Airflow–based workflow manager that schedules, submits, and monitors the
data processing and upload jobs.

`slurm`
: HPC cluster for processing and upload tasks.

`aind-metadata-service`
: REST API that compiles and returns `aind-data-schema` metadata, aggregating and
mapping records from multiple upstream data providers.

`VAST`
: Shared storage.

Container registry
: Docker images used to run processing jobs.

![Local data center flow](mid_level/local_data_center_flow.drawio.svg)
35 changes: 35 additions & 0 deletions docs/source/diagrams/data_storage_processing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Data Storage and Processing

## Code Ocean pipeline

Raw data lands in S3 as a single data asset that carries its `aind-data-schema`
metadata, with all of a session's modalities associated with one object.
Processing pipelines are modality-specific — each pipeline processes a single
modality. A pipeline outputs an NWB file along with `aind-data-schema` metadata,
including processing metrics and quality control artifacts and metrics. Once the
outputs have been QC'd, they can be combined into a final NWB file with its
associated metadata.

Each pipeline wraps modality- and platform-specific libraries that handle the
underlying data processing, quality control, and NWB packaging.

![Code Ocean pipeline diagram](mid_level/codeocean_pipeline_diagram.svg)



## Quality control

Each pipeline produces quality control artifacts and metrics alongside its
processed outputs, captured in the asset's `aind-data-schema` metadata. These are
surfaced through the [QC portal](https://github.com/AllenNeuralDynamics/aind-qc-portal),
a web application for viewing and annotating quality control metadata for AIND
data assets. The portal pulls QC metadata from the document database and displays
the corresponding reference figures from the Code Ocean data assets.

Reviewers use the portal to evaluate metrics marked `PENDING`, supporting AIND's
two-step quality control process: first assessing whether an asset's data is
suitable for analysis, then evaluating individual components (such as neurons)
within the asset for usability.

![QC diagram](mid_level/QC.drawio.svg)

40 changes: 40 additions & 0 deletions docs/source/diagrams/dynamic_foraging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Dynamic foraging diagrams

These diagrams outline the architecture of the Dynamic Foraging platform.

## Lab Management System

Dynamic Foraging relies on the lab management system for session and subject
planning. It also uses the system's water restriction tracking to support
behavior training.

![Dynamic Foraging lab management](dynamic_foraging/mid_level/lab_management_df.svg)

## Data Acquisition

![Dynamic Foraging acquisition architecture](dynamic_foraging/dynamic_foraging_architecture.svg)

## Data Storage and Processing

### Mid-level Processing Pipeline

Data from Dynamic Foraging is processed by two pipelines — a behavior pipeline
and a fiber pipeline. Each runs independently and produces a modality-specific
NWB file when it completes successfully.

![Dynamic Foraging processing pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg)

#### Fiber Pipeline

![Dynamic Foraging fiber pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg)

#### Behavior Pipeline

![Dynamic Foraging behavior pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg)

### Low-level Processing Pipeline Library

This pipeline processes raw Harp hardware signals into the trials table, packages
the data into NWB, and runs QC on both the raw and processed data.

![Dynamic Foraging low-level processing](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg)
Loading