diff --git a/docs/source/aind/core_services.md b/docs/source/aind/core_services.md index a5e222f..97c80ac 100644 --- a/docs/source/aind/core_services.md +++ b/docs/source/aind/core_services.md @@ -1,6 +1,6 @@ # Core Services -The interactions between many of these services are illustrated in the [AIND software diagrams](./diagrams.md). +The interactions between many of these services are illustrated in the [AIND software diagrams](../diagrams/index.md). **aind-data-transfer-service** diff --git a/docs/source/aind/diagrams.md b/docs/source/aind/diagrams.md deleted file mode 100644 index a3f1b84..0000000 --- a/docs/source/aind/diagrams.md +++ /dev/null @@ -1,11 +0,0 @@ -# AIND Software and Systems Diagrams - -This page contains diagrams illustrating the interactions between AIND software and systems, including core services, data storage, and compute resources. These diagrams are intended to provide a high-level overview of how different components fit together, and will be updated periodically as our software and systems evolve. - -**New diagrams coming soon for future plans and low-level architecture.** - -## High-level architecture - -![High-level data flow](../diagrams/high_level/general_data_flow.drawio.svg) - -![AIND Software Overview](../_static/aind-software-overview.png) \ No newline at end of file diff --git a/docs/source/diagrams/data_acquisition.md b/docs/source/diagrams/data_acquisition.md new file mode 100644 index 0000000..2a900e0 --- /dev/null +++ b/docs/source/diagrams/data_acquisition.md @@ -0,0 +1,13 @@ +# Data Acquisition + +:::{note} +This diagram is a **proposal** for the instrumentation software architecture — a +planned design rather than a system currently in production. +::: + +This diagram presents a proposed software architecture for AIND instrumentation +during data acquisition. It outlines how the acquisition software components are +intended to interact to control the instrument and capture data during an +experiment. + +![SIPE instrument diagram](mid_level/sipe_instrument.drawio.svg) diff --git a/docs/source/diagrams/data_staging.md b/docs/source/diagrams/data_staging.md new file mode 100644 index 0000000..c4addfb --- /dev/null +++ b/docs/source/diagrams/data_staging.md @@ -0,0 +1,28 @@ +# Data Staging + +The local data center hosts a variety of services that stage and process data on +its way to the cloud: + +`aind-data-transfer-service` +: FastAPI service that accepts and validates data upload requests, retrieves the +associated metadata, and dispatches compression and transfer jobs to the workflow +manager for execution on the HPC cluster. + +`aind-airflow-service` +: Apache Airflow–based workflow manager that schedules, submits, and monitors the +data processing and upload jobs. + +`slurm` +: HPC cluster for processing and upload tasks. + +`aind-metadata-service` +: REST API that compiles and returns `aind-data-schema` metadata, aggregating and +mapping records from multiple upstream data providers. + +`VAST` +: Shared storage. + +Container registry +: Docker images used to run processing jobs. + +![Local data center flow](mid_level/local_data_center_flow.drawio.svg) diff --git a/docs/source/diagrams/data_storage_processing.md b/docs/source/diagrams/data_storage_processing.md new file mode 100644 index 0000000..0818c2d --- /dev/null +++ b/docs/source/diagrams/data_storage_processing.md @@ -0,0 +1,35 @@ +# Data Storage and Processing + +## Code Ocean pipeline + +Raw data lands in S3 as a single data asset that carries its `aind-data-schema` +metadata, with all of a session's modalities associated with one object. +Processing pipelines are modality-specific — each pipeline processes a single +modality. A pipeline outputs an NWB file along with `aind-data-schema` metadata, +including processing metrics and quality control artifacts and metrics. Once the +outputs have been QC'd, they can be combined into a final NWB file with its +associated metadata. + +Each pipeline wraps modality- and platform-specific libraries that handle the +underlying data processing, quality control, and NWB packaging. + +![Code Ocean pipeline diagram](mid_level/codeocean_pipeline_diagram.svg) + + + +## Quality control + +Each pipeline produces quality control artifacts and metrics alongside its +processed outputs, captured in the asset's `aind-data-schema` metadata. These are +surfaced through the [QC portal](https://github.com/AllenNeuralDynamics/aind-qc-portal), +a web application for viewing and annotating quality control metadata for AIND +data assets. The portal pulls QC metadata from the document database and displays +the corresponding reference figures from the Code Ocean data assets. + +Reviewers use the portal to evaluate metrics marked `PENDING`, supporting AIND's +two-step quality control process: first assessing whether an asset's data is +suitable for analysis, then evaluating individual components (such as neurons) +within the asset for usability. + +![QC diagram](mid_level/QC.drawio.svg) + diff --git a/docs/source/diagrams/dynamic_foraging.md b/docs/source/diagrams/dynamic_foraging.md new file mode 100644 index 0000000..77b86c0 --- /dev/null +++ b/docs/source/diagrams/dynamic_foraging.md @@ -0,0 +1,40 @@ +# Dynamic foraging diagrams + +These diagrams outline the architecture of the Dynamic Foraging platform. + +## Lab Management System + +Dynamic Foraging relies on the lab management system for session and subject +planning. It also uses the system's water restriction tracking to support +behavior training. + +![Dynamic Foraging lab management](dynamic_foraging/mid_level/lab_management_df.svg) + +## Data Acquisition + +![Dynamic Foraging acquisition architecture](dynamic_foraging/dynamic_foraging_architecture.svg) + +## Data Storage and Processing + +### Mid-level Processing Pipeline + +Data from Dynamic Foraging is processed by two pipelines — a behavior pipeline +and a fiber pipeline. Each runs independently and produces a modality-specific +NWB file when it completes successfully. + +![Dynamic Foraging processing pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg) + +#### Fiber Pipeline + +![Dynamic Foraging fiber pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg) + +#### Behavior Pipeline + +![Dynamic Foraging behavior pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg) + +### Low-level Processing Pipeline Library + +This pipeline processes raw Harp hardware signals into the trials table, packages +the data into NWB, and runs QC on both the raw and processed data. + +![Dynamic Foraging low-level processing](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio index 3aa17bf..c201bf2 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio @@ -1,11 +1,23 @@ - + + + + + + + + + + + + + - + @@ -25,38 +37,38 @@ - - + + - + - + - + - + - + - + - + - + @@ -79,10 +91,10 @@ - + - + @@ -116,8 +128,8 @@ - - + + @@ -144,19 +156,19 @@ - + - + - + - + - + @@ -168,41 +180,41 @@ - + - + - - + + - + - - + + - + - + - - + + - + @@ -371,79 +383,79 @@ - + - + - + - - + + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - - - + + + - - + + - - + + - - + + diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg index 1af56ca..3bf257e 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg @@ -1,4 +1,4 @@ -
Fiber Pipeline
DocDB
S3: aind-open-data
DF Behavior Pipeline
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Fib
Behavior
Behavior-videos
DocDB
S3: aind-open-data
 \ No newline at end of file +
Fiber Pipeline
DocDB
S3: aind-open-data
DF Behavior Pipeline
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Raw Data
Fib
Behavior
Behavior-videos
aind-data-schema 
metadata
Fib
Behavior
Behavior-videos
NWB File
processing.json
Transform Raw Fib Streams into NWB File
QC Raw Data
Aggregate Quality Control Data
Fiber Pipeline
DocDB
S3: aind-open-data
DF Behavior Pipeline
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File 
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
Compute DF / F and Package into NWB File
quality_control.json
Artifacts
NWB File
quality_control.json
 \ No newline at end of file diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg index 3a6830c..a958e26 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg @@ -1,4 +1,4 @@ -
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File 
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
 \ No newline at end of file +
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File 
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
 \ No newline at end of file diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_closeup.svg b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_closeup.svg index 3449416..fca1b2b 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_closeup.svg +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_closeup.svg @@ -1,4 +1,4 @@ -
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
Fib
Behavior
Behavior-videos
aind-data-schema 
metadata
NWB File
processing.json
Transform Raw Fib Streams into NWB File
QC Raw Data
Aggregate Quality Control Data
Fiber Pipeline
DF Behavior Pipeline
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File with 
aind-data-schema 
metadata
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
Compute DF / F and Package into NWB File
quality_control.json
Artifacts
NWB File
quality_control.json
 \ No newline at end of file +
Fiber Pipeline
DocDB
S3: aind-open-data
DF Behavior Pipeline
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Fib
Behavior
Behavior-videos
DocDB
S3: aind-open-data
 \ No newline at end of file diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg index 51d8847..c00452b 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg @@ -1,4 +1,4 @@ -
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
Fib
Behavior
Behavior-videos
aind-data-schema 
metadata
NWB File
processing.json
Transform Raw Fib Streams into NWB File
QC Raw Data
Aggregate Quality Control Data
Compute DF / F and Package into NWB File
quality_control.json
Artifacts
NWB File
quality_control.json
 \ No newline at end of file +
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
Fib
Behavior
Behavior-videos
aind-data-schema 
metadata
NWB File
processing.json
Transform Raw Fib Streams into NWB File
QC Raw Data
Aggregate Quality Control Data
Compute DF / F and Package into NWB File
quality_control.json
Artifacts
NWB File
quality_control.json
 \ No newline at end of file diff --git a/docs/source/diagrams/index.md b/docs/source/diagrams/index.md new file mode 100644 index 0000000..d609c5e --- /dev/null +++ b/docs/source/diagrams/index.md @@ -0,0 +1,30 @@ +# Software and Systems Diagrams + +This section contains diagrams illustrating the interactions between AIND software and systems, including core services, data storage, and compute resources. They are organized around the main stages of the data-flow pipeline — follow a stage to drill into its detailed diagrams. These pages are updated periodically as our software and systems evolve. + +## General data flow + +This diagram illustrates a bird's-eye view of the data-flow pipeline. Four main +components encompass how data are generated and move through the system: + +1. [Lab Management System](lab_management.md) +2. [Data Acquisition](data_acquisition.md) +3. [Data Staging](data_staging.md) +4. [Data Storage and Processing](data_storage_processing.md) + +![High-level data flow](high_level/general_data_flow.drawio.svg) + +## Platform-specific + +- [Dynamic foraging](dynamic_foraging.md) + +```{toctree} +:maxdepth: 1 +:hidden: + +lab_management +data_acquisition +data_staging +data_storage_processing +dynamic_foraging +``` diff --git a/docs/source/diagrams/lab_management.md b/docs/source/diagrams/lab_management.md new file mode 100644 index 0000000..506b238 --- /dev/null +++ b/docs/source/diagrams/lab_management.md @@ -0,0 +1,3 @@ +# Lab Management System + +![Lab management diagram](mid_level/lab_management.svg) diff --git a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.drawio b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.drawio index a0b3067..a54b694 100644 --- a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.drawio +++ b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.drawio @@ -1,11 +1,20 @@ - + + + + + + + + + + - + @@ -28,41 +37,41 @@ - - + + - + - + - + - + - + - + - + - + - + @@ -91,10 +100,10 @@ - + - + @@ -143,8 +152,8 @@ - - + + @@ -177,19 +186,19 @@ - + - + - + - + - + @@ -201,106 +210,106 @@ - + - + - - + + - - + + - - + + - + - - + + - + - - + + - + - + - - + + - + - - + + - + - - + + - + - - + + - + - + - + - + - - + + diff --git a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg index 797e1ba..b5119f4 100644 --- a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg +++ b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg @@ -1,4 +1,4 @@ -
Pipeline Modality 1
DocDB
S3: aind-open-data
Pipeline Modality 2
Pipeline Modality 3
NWB File
aind-data-schema metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
DocDB
S3: aind-open-data
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File with aind-data-schema metadata
Plots and metrics
NWB File
aind-data-schema metadata
Plots and metrics
DocDB
S3: aind-open-data
NWB File
aind-data-schema metadata
Plots and metrics
NWB File
aind-data-schema metadata
Plots and metrics
NWB File
aind-data-schema metadata
Plots and metrics
aind-data-schema metadata
Raw Data
modality1
modality2%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20parent%3D%221%22%20style%3D%22html%3D1%3BverticalLabelPosition%3Dbottom%3Balign%3Dcenter%3BlabelBackgroundColor%3D%23ffffff%3BverticalAlign%3Dtop%3BstrokeWidth%3D2%3BstrokeColor%3D%230080F0%3Bshadow%3D0%3Bdashed%3D0%3Bshape%3Dmxgraph.ios7.icons.folder%3B%22%20value%3D%22modality1%22%20vertex%3D%221%22%3E%3CmxGeometry%20height%3D%22130%22%20width%3D%22200%22%20x%3D%22-550%22%20y%3D%22-100%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
modality3
aind-data-schema metadata
modality1
modality2%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20parent%3D%221%22%20style%3D%22html%3D1%3BverticalLabelPosition%3Dbottom%3Balign%3Dcenter%3BlabelBackgroundColor%3D%23ffffff%3BverticalAlign%3Dtop%3BstrokeWidth%3D2%3BstrokeColor%3D%230080F0%3Bshadow%3D0%3Bdashed%3D0%3Bshape%3Dmxgraph.ios7.icons.folder%3B%22%20value%3D%22modality1%22%20vertex%3D%221%22%3E%3CmxGeometry%20height%3D%22130%22%20width%3D%22200%22%20x%3D%22-550%22%20y%3D%22-100%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
modality3
NWB File
processing.json
Tranform and Process
QC NWB FFile
Metadata Manager
+
+
+
Pipeline N
 \ No newline at end of file +
Pipeline Modality 1
DocDB
S3: aind-open-data
Pipeline Modality 2
Pipeline Modality 3
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File with 
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
modality 1
modality 2
modality 3
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
Pipeline N
DocDB
S3: aind-open-data
 \ No newline at end of file diff --git a/docs/source/index.md b/docs/source/index.md index 0698c73..c98a8cc 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -32,10 +32,10 @@ Follow these links to request access to: ## I want to learn about... -[Data organization](policies_practices/data_organization.md), [data governance](policies_practices/data_governance.md), [software practices](policies_practices/software_practices.md), or [visualize how our software and systems interact](aind/diagrams.md). +[Data organization](policies_practices/data_organization.md), [data governance](policies_practices/data_governance.md), [software practices](policies_practices/software_practices.md), or [visualize how our software and systems interact](diagrams/index.md). ```{toctree} -:maxdepth: 2 +:maxdepth: 1 :hidden: :caption: Acquire, upload & process @@ -48,21 +48,20 @@ acquire_upload/process_data ``` ```{toctree} -:maxdepth: 2 +:maxdepth: 1 :hidden: :caption: Explore, QC & analyze explore_analyze/quality_control explore_analyze/find_data explore_analyze/analyze_data -explore_analyze/create_processing_metadata explore_analyze/outreach ``` ```{toctree} -:maxdepth: 2 +:maxdepth: 1 :hidden: :caption: Policies & practices @@ -77,11 +76,12 @@ policies_practices/docs ``` ```{toctree} -:maxdepth: 2 +:maxdepth: 1 :hidden: -:caption: AIND Resources +:caption: Resources aind/core_services +diagrams/index aind/project_proposals.md aind/teams.md policies_practices/rse_resources diff --git a/docs/source/process_data/version_processes.md b/docs/source/process_data/version_processes.md new file mode 100644 index 0000000..e69de29