From 0fa4ed4c681e6087a14628d3487a93c759ba23b7 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Fri, 26 Jun 2026 14:16:58 -0700 Subject: [PATCH 01/15] feat: build documentation for diagrams --- docs/source/aind/core_services.md | 2 +- docs/source/aind/diagrams.md | 11 ----------- docs/source/index.md | 20 ++++++++++++++------ 3 files changed, 15 insertions(+), 18 deletions(-) delete mode 100644 docs/source/aind/diagrams.md diff --git a/docs/source/aind/core_services.md b/docs/source/aind/core_services.md index a5e222f..97c80ac 100644 --- a/docs/source/aind/core_services.md +++ b/docs/source/aind/core_services.md @@ -1,6 +1,6 @@ # Core Services -The interactions between many of these services are illustrated in the [AIND software diagrams](./diagrams.md). +The interactions between many of these services are illustrated in the [AIND software diagrams](../diagrams/index.md). **aind-data-transfer-service** diff --git a/docs/source/aind/diagrams.md b/docs/source/aind/diagrams.md deleted file mode 100644 index a3f1b84..0000000 --- a/docs/source/aind/diagrams.md +++ /dev/null @@ -1,11 +0,0 @@ -# AIND Software and Systems Diagrams - -This page contains diagrams illustrating the interactions between AIND software and systems, including core services, data storage, and compute resources. These diagrams are intended to provide a high-level overview of how different components fit together, and will be updated periodically as our software and systems evolve. - -**New diagrams coming soon for future plans and low-level architecture.** - -## High-level architecture - -![High-level data flow](../diagrams/high_level/general_data_flow.drawio.svg) - -![AIND Software Overview](../_static/aind-software-overview.png) \ No newline at end of file diff --git a/docs/source/index.md b/docs/source/index.md index 0698c73..4b9b48c 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -32,10 +32,10 @@ Follow these links to request access to: ## I want to learn about... -[Data organization](policies_practices/data_organization.md), [data governance](policies_practices/data_governance.md), [software practices](policies_practices/software_practices.md), or [visualize how our software and systems interact](aind/diagrams.md). +[Data organization](policies_practices/data_organization.md), [data governance](policies_practices/data_governance.md), [software practices](policies_practices/software_practices.md), or [visualize how our software and systems interact](diagrams/index.md). ```{toctree} -:maxdepth: 2 +:maxdepth: 1 :hidden: :caption: Acquire, upload & process @@ -48,21 +48,20 @@ acquire_upload/process_data ``` ```{toctree} -:maxdepth: 2 +:maxdepth: 1 :hidden: :caption: Explore, QC & analyze explore_analyze/quality_control explore_analyze/find_data explore_analyze/analyze_data -explore_analyze/create_processing_metadata explore_analyze/outreach ``` ```{toctree} -:maxdepth: 2 +:maxdepth: 1 :hidden: :caption: Policies & practices @@ -77,7 +76,16 @@ policies_practices/docs ``` ```{toctree} -:maxdepth: 2 +:maxdepth: 1 +:hidden: +:caption: Diagrams + +diagrams/index + +``` + +```{toctree} +:maxdepth: 1 :hidden: :caption: AIND Resources From 4f17e4dcb88fd6ddca2a944b9f440b8637e35cdf Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Fri, 26 Jun 2026 14:43:55 -0700 Subject: [PATCH 02/15] chore: fix the header for the diagrams contents --- docs/source/diagrams/index.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 docs/source/diagrams/index.md diff --git a/docs/source/diagrams/index.md b/docs/source/diagrams/index.md new file mode 100644 index 0000000..ecc96d0 --- /dev/null +++ b/docs/source/diagrams/index.md @@ -0,0 +1,14 @@ +# Software and Systems Diagrams + +This section contains diagrams illustrating the interactions between AIND software and systems, including core services, data storage, and compute resources. They are intended to provide a high-level overview of how different components fit together, and are organized by altitude — from high-level overviews down to low-level service architecture — plus platform-specific diagrams. These pages are updated periodically as our software and systems evolve. + +[![Software Overview](../_static/aind-software-overview.png)](../_static/aind-software-overview.pdf) + +```{toctree} +:maxdepth: 1 + +high_level +mid_level +low_level +dynamic_foraging +``` From 860562b7d282d1e33dc17cca4e8760282f707fa9 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Fri, 26 Jun 2026 15:10:43 -0700 Subject: [PATCH 03/15] fix: add diagram sub-pages so diagrams render in the docs The diagrams/index.md toctree referenced high_level, mid_level, low_level, and dynamic_foraging, but those pages were never tracked, so the published build showed the landing header with no diagrams. Co-Authored-By: Claude Opus 4.8 --- docs/source/diagrams/dynamic_foraging.md | 11 +++++++ docs/source/diagrams/high_level.md | 7 +++++ docs/source/diagrams/low_level.md | 38 ++++++++++++++++++++++++ docs/source/diagrams/mid_level.md | 20 +++++++++++++ 4 files changed, 76 insertions(+) create mode 100644 docs/source/diagrams/dynamic_foraging.md create mode 100644 docs/source/diagrams/high_level.md create mode 100644 docs/source/diagrams/low_level.md create mode 100644 docs/source/diagrams/mid_level.md diff --git a/docs/source/diagrams/dynamic_foraging.md b/docs/source/diagrams/dynamic_foraging.md new file mode 100644 index 0000000..4b40a0e --- /dev/null +++ b/docs/source/diagrams/dynamic_foraging.md @@ -0,0 +1,11 @@ +# Dynamic foraging diagrams + +Diagrams specific to the dynamic foraging platform. + +## Architecture + +![Dynamic foraging architecture](dynamic_foraging/dynamic_foraging_architecture.svg) + +## Low-level processing + +![Dynamic foraging low-level processing](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) diff --git a/docs/source/diagrams/high_level.md b/docs/source/diagrams/high_level.md new file mode 100644 index 0000000..10e18d3 --- /dev/null +++ b/docs/source/diagrams/high_level.md @@ -0,0 +1,7 @@ +# High-level diagrams + +High-level overviews of how data flows through AIND systems. + +## General data flow + +![High-level data flow](high_level/general_data_flow.drawio.svg) diff --git a/docs/source/diagrams/low_level.md b/docs/source/diagrams/low_level.md new file mode 100644 index 0000000..0d4d0e2 --- /dev/null +++ b/docs/source/diagrams/low_level.md @@ -0,0 +1,38 @@ +# Low-level diagrams + +Service-level architecture diagrams showing the internals of individual AIND +services and their APIs. + +## Asset registration API + +![Asset registration API](low_level/asset_registration_api.svg) + +## Data asset indexer + +![Data asset indexer](low_level/data_asset_indexer.svg) + +## Data schema to DocDB + +![Data schema to DocDB](low_level/data_schema_to_docdb.svg) + +## DocDB API + +![DocDB API](low_level/docdb_api.svg) + +## Redshift client + +![Redshift client](low_level/redshift_client.svg) + +## Metadata service + +### Overview + +![Metadata service overview](low_level/aind-metadata-service/metadata_service_overview_diagram.drawio.svg) + +### Kubernetes deployment + +![Metadata service Kubernetes deployment](low_level/aind-metadata-service/aind-metadata-service-k8s.drawio.svg) + +### Procedures extractor + +![Metadata service procedures extractor](low_level/aind-metadata-service/metadata_service_procedures_extractor_diagram.drawio.svg) diff --git a/docs/source/diagrams/mid_level.md b/docs/source/diagrams/mid_level.md new file mode 100644 index 0000000..dc2d502 --- /dev/null +++ b/docs/source/diagrams/mid_level.md @@ -0,0 +1,20 @@ +# Mid-level diagrams + +Diagrams of the major pipelines and data flows, sitting between the high-level +overview and the low-level service architecture. + +## Code Ocean pipeline + +![Code Ocean pipeline diagram](mid_level/codeocean_pipeline_diagram.svg) + +## Quality control + +![QC diagram](mid_level/QC.drawio.svg) + +## Local data center flow + +![Local data center flow](mid_level/local_data_center_flow.drawio.svg) + +## SIPE instrument + +![SIPE instrument diagram](mid_level/sipe_instrument.drawio.svg) From ffdf59f199c688be1dae8f4fd0956d507f242225 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 14:29:03 -0700 Subject: [PATCH 04/15] fix: add descriptions and links --- docs/source/diagrams/high_level.md | 8 ++++++++ docs/source/diagrams/mid_level.md | 12 ++++++++++++ 2 files changed, 20 insertions(+) diff --git a/docs/source/diagrams/high_level.md b/docs/source/diagrams/high_level.md index 10e18d3..45d91db 100644 --- a/docs/source/diagrams/high_level.md +++ b/docs/source/diagrams/high_level.md @@ -4,4 +4,12 @@ High-level overviews of how data flows through AIND systems. ## General data flow +This diagram illustrates a bird's-eye view of the data-flow pipeline. Four main +components encompass how data are generated and move through the system: + +1. [Lab Management System](mid_level.md#lab-management) +2. [Data Acquisition](mid_level.md#sipe-instrument) +3. [Data Staging](mid_level.md#local-data-center-flow) +4. [Data Storage and Processing](mid_level.md#code-ocean-pipeline) + ![High-level data flow](high_level/general_data_flow.drawio.svg) diff --git a/docs/source/diagrams/mid_level.md b/docs/source/diagrams/mid_level.md index dc2d502..e19d351 100644 --- a/docs/source/diagrams/mid_level.md +++ b/docs/source/diagrams/mid_level.md @@ -7,6 +7,18 @@ overview and the low-level service architecture. ![Code Ocean pipeline diagram](mid_level/codeocean_pipeline_diagram.svg) +### Pipeline overview + +![Pipeline overview](mid_level/pipeline_overview.svg) + +### Pipeline close-up + +![Pipeline close-up](mid_level/pipeline_close_up.svg) + +## Lab management + +![Lab management diagram](mid_level/lab_management.svg) + ## Quality control ![QC diagram](mid_level/QC.drawio.svg) From 767911cb3832e0c668e308042a6746ab44d88e5f Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 15:39:46 -0700 Subject: [PATCH 05/15] chore: rearrange diagram layout --- docs/source/diagrams/data_acquisition.md | 3 ++ docs/source/diagrams/data_staging.md | 3 ++ .../diagrams/data_storage_processing.md | 54 +++++++++++++++++++ docs/source/diagrams/dynamic_foraging.md | 10 ++-- docs/source/diagrams/high_level.md | 15 ------ docs/source/diagrams/index.md | 26 +++++++-- docs/source/diagrams/lab_management.md | 3 ++ docs/source/diagrams/low_level.md | 38 ------------- docs/source/diagrams/mid_level.md | 32 ----------- docs/source/process_data/version_processes.md | 0 10 files changed, 90 insertions(+), 94 deletions(-) create mode 100644 docs/source/diagrams/data_acquisition.md create mode 100644 docs/source/diagrams/data_staging.md create mode 100644 docs/source/diagrams/data_storage_processing.md delete mode 100644 docs/source/diagrams/high_level.md create mode 100644 docs/source/diagrams/lab_management.md delete mode 100644 docs/source/diagrams/low_level.md delete mode 100644 docs/source/diagrams/mid_level.md create mode 100644 docs/source/process_data/version_processes.md diff --git a/docs/source/diagrams/data_acquisition.md b/docs/source/diagrams/data_acquisition.md new file mode 100644 index 0000000..74747e1 --- /dev/null +++ b/docs/source/diagrams/data_acquisition.md @@ -0,0 +1,3 @@ +# Data Acquisition + +![SIPE instrument diagram](mid_level/sipe_instrument.drawio.svg) diff --git a/docs/source/diagrams/data_staging.md b/docs/source/diagrams/data_staging.md new file mode 100644 index 0000000..720a4bc --- /dev/null +++ b/docs/source/diagrams/data_staging.md @@ -0,0 +1,3 @@ +# Data Staging + +![Local data center flow](mid_level/local_data_center_flow.drawio.svg) diff --git a/docs/source/diagrams/data_storage_processing.md b/docs/source/diagrams/data_storage_processing.md new file mode 100644 index 0000000..88983bf --- /dev/null +++ b/docs/source/diagrams/data_storage_processing.md @@ -0,0 +1,54 @@ +# Data Storage and Processing + +## Code Ocean pipeline + +![Code Ocean pipeline diagram](mid_level/codeocean_pipeline_diagram.svg) + +### Pipeline overview + +![Pipeline overview](mid_level/pipeline_overview.svg) + +### Pipeline close-up + +![Pipeline close-up](mid_level/pipeline_close_up.svg) + +## Quality control + +![QC diagram](mid_level/QC.drawio.svg) + +## Service internals + +Lower-level diagrams of the backend services that store, index, and serve data +and metadata. + +### Asset registration API + +![Asset registration API](low_level/asset_registration_api.svg) + +### Data asset indexer + +![Data asset indexer](low_level/data_asset_indexer.svg) + +### Data schema to DocDB + +![Data schema to DocDB](low_level/data_schema_to_docdb.svg) + +### DocDB API + +![DocDB API](low_level/docdb_api.svg) + +### Redshift client + +![Redshift client](low_level/redshift_client.svg) + +### Metadata service + +![Metadata service overview](low_level/aind-metadata-service/metadata_service_overview_diagram.drawio.svg) + +#### Kubernetes deployment + +![Metadata service Kubernetes deployment](low_level/aind-metadata-service/aind-metadata-service-k8s.drawio.svg) + +#### Procedures extractor + +![Metadata service procedures extractor](low_level/aind-metadata-service/metadata_service_procedures_extractor_diagram.drawio.svg) diff --git a/docs/source/diagrams/dynamic_foraging.md b/docs/source/diagrams/dynamic_foraging.md index 4b40a0e..4c7c5b7 100644 --- a/docs/source/diagrams/dynamic_foraging.md +++ b/docs/source/diagrams/dynamic_foraging.md @@ -1,11 +1,11 @@ # Dynamic foraging diagrams -Diagrams specific to the dynamic foraging platform. +These diagrams outline the architecture of the Dynamic Foraging behavior platform -## Architecture +## Data Acquisition -![Dynamic foraging architecture](dynamic_foraging/dynamic_foraging_architecture.svg) +![Mid-level](dynamic_foraging/dynamic_foraging_architecture.svg) -## Low-level processing +## Data Storage and Processing -![Dynamic foraging low-level processing](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) +![Low-level Processing Pipeline](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) diff --git a/docs/source/diagrams/high_level.md b/docs/source/diagrams/high_level.md deleted file mode 100644 index 45d91db..0000000 --- a/docs/source/diagrams/high_level.md +++ /dev/null @@ -1,15 +0,0 @@ -# High-level diagrams - -High-level overviews of how data flows through AIND systems. - -## General data flow - -This diagram illustrates a bird's-eye view of the data-flow pipeline. Four main -components encompass how data are generated and move through the system: - -1. [Lab Management System](mid_level.md#lab-management) -2. [Data Acquisition](mid_level.md#sipe-instrument) -3. [Data Staging](mid_level.md#local-data-center-flow) -4. [Data Storage and Processing](mid_level.md#code-ocean-pipeline) - -![High-level data flow](high_level/general_data_flow.drawio.svg) diff --git a/docs/source/diagrams/index.md b/docs/source/diagrams/index.md index ecc96d0..6269b02 100644 --- a/docs/source/diagrams/index.md +++ b/docs/source/diagrams/index.md @@ -1,14 +1,32 @@ # Software and Systems Diagrams -This section contains diagrams illustrating the interactions between AIND software and systems, including core services, data storage, and compute resources. They are intended to provide a high-level overview of how different components fit together, and are organized by altitude — from high-level overviews down to low-level service architecture — plus platform-specific diagrams. These pages are updated periodically as our software and systems evolve. +This section contains diagrams illustrating the interactions between AIND software and systems, including core services, data storage, and compute resources. They are organized around the main stages of the data-flow pipeline — follow a stage to drill into its detailed diagrams. These pages are updated periodically as our software and systems evolve. + +## General data flow + +This diagram illustrates a bird's-eye view of the data-flow pipeline. Four main +components encompass how data are generated and move through the system: + +1. [Lab Management System](lab_management.md) +2. [Data Acquisition](data_acquisition.md) +3. [Data Staging](data_staging.md) +4. [Data Storage and Processing](data_storage_processing.md) + +![High-level data flow](high_level/general_data_flow.drawio.svg) + +## Platform-specific + +- [Dynamic foraging](dynamic_foraging.md) [![Software Overview](../_static/aind-software-overview.png)](../_static/aind-software-overview.pdf) ```{toctree} :maxdepth: 1 +:hidden: -high_level -mid_level -low_level +lab_management +data_acquisition +data_staging +data_storage_processing dynamic_foraging ``` diff --git a/docs/source/diagrams/lab_management.md b/docs/source/diagrams/lab_management.md new file mode 100644 index 0000000..506b238 --- /dev/null +++ b/docs/source/diagrams/lab_management.md @@ -0,0 +1,3 @@ +# Lab Management System + +![Lab management diagram](mid_level/lab_management.svg) diff --git a/docs/source/diagrams/low_level.md b/docs/source/diagrams/low_level.md deleted file mode 100644 index 0d4d0e2..0000000 --- a/docs/source/diagrams/low_level.md +++ /dev/null @@ -1,38 +0,0 @@ -# Low-level diagrams - -Service-level architecture diagrams showing the internals of individual AIND -services and their APIs. - -## Asset registration API - -![Asset registration API](low_level/asset_registration_api.svg) - -## Data asset indexer - -![Data asset indexer](low_level/data_asset_indexer.svg) - -## Data schema to DocDB - -![Data schema to DocDB](low_level/data_schema_to_docdb.svg) - -## DocDB API - -![DocDB API](low_level/docdb_api.svg) - -## Redshift client - -![Redshift client](low_level/redshift_client.svg) - -## Metadata service - -### Overview - -![Metadata service overview](low_level/aind-metadata-service/metadata_service_overview_diagram.drawio.svg) - -### Kubernetes deployment - -![Metadata service Kubernetes deployment](low_level/aind-metadata-service/aind-metadata-service-k8s.drawio.svg) - -### Procedures extractor - -![Metadata service procedures extractor](low_level/aind-metadata-service/metadata_service_procedures_extractor_diagram.drawio.svg) diff --git a/docs/source/diagrams/mid_level.md b/docs/source/diagrams/mid_level.md deleted file mode 100644 index e19d351..0000000 --- a/docs/source/diagrams/mid_level.md +++ /dev/null @@ -1,32 +0,0 @@ -# Mid-level diagrams - -Diagrams of the major pipelines and data flows, sitting between the high-level -overview and the low-level service architecture. - -## Code Ocean pipeline - -![Code Ocean pipeline diagram](mid_level/codeocean_pipeline_diagram.svg) - -### Pipeline overview - -![Pipeline overview](mid_level/pipeline_overview.svg) - -### Pipeline close-up - -![Pipeline close-up](mid_level/pipeline_close_up.svg) - -## Lab management - -![Lab management diagram](mid_level/lab_management.svg) - -## Quality control - -![QC diagram](mid_level/QC.drawio.svg) - -## Local data center flow - -![Local data center flow](mid_level/local_data_center_flow.drawio.svg) - -## SIPE instrument - -![SIPE instrument diagram](mid_level/sipe_instrument.drawio.svg) diff --git a/docs/source/process_data/version_processes.md b/docs/source/process_data/version_processes.md new file mode 100644 index 0000000..e69de29 From b33696a6ccabcaac79b908240ba607975abe86ea Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 18:46:27 -0700 Subject: [PATCH 06/15] chore: add all links to DF diagrams --- docs/source/diagrams/dynamic_foraging.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/source/diagrams/dynamic_foraging.md b/docs/source/diagrams/dynamic_foraging.md index 4c7c5b7..82d6b0c 100644 --- a/docs/source/diagrams/dynamic_foraging.md +++ b/docs/source/diagrams/dynamic_foraging.md @@ -1,6 +1,10 @@ # Dynamic foraging diagrams -These diagrams outline the architecture of the Dynamic Foraging behavior platform +These diagrams outline the architecture of the Dynamic Foraging behavior platform. + +## Lab Management System + +![Mid-level Processing Pipeline Library](dynamic_foraging/mid_level/lab_management_df.svg) ## Data Acquisition @@ -8,4 +12,6 @@ These diagrams outline the architecture of the Dynamic Foraging behavior platfor ## Data Storage and Processing -![Low-level Processing Pipeline](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) +![Mid-level Processing Pipeline Library](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg) + +![Low-level Processing Pipeline Library](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) From 04781df498fd35219a76fd8556aef80eff3d9531 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 18:51:40 -0700 Subject: [PATCH 07/15] chore: add titles --- docs/source/diagrams/dynamic_foraging.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/diagrams/dynamic_foraging.md b/docs/source/diagrams/dynamic_foraging.md index 82d6b0c..41ec5b9 100644 --- a/docs/source/diagrams/dynamic_foraging.md +++ b/docs/source/diagrams/dynamic_foraging.md @@ -12,6 +12,8 @@ These diagrams outline the architecture of the Dynamic Foraging behavior platfor ## Data Storage and Processing -![Mid-level Processing Pipeline Library](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg) +### Mid-level Processing Pipeline +![Mid-level Processing Pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg) +### Low-level Processing Pipeline Library ![Low-level Processing Pipeline Library](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) From ecfb0a6825967a12aa6aa4e7ba16a708c978e9e8 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 19:03:43 -0700 Subject: [PATCH 08/15] chore: added more text and fixed edits in wording --- docs/source/diagrams/dynamic_foraging.md | 31 ++++++++++++++++++++---- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/docs/source/diagrams/dynamic_foraging.md b/docs/source/diagrams/dynamic_foraging.md index 41ec5b9..77b86c0 100644 --- a/docs/source/diagrams/dynamic_foraging.md +++ b/docs/source/diagrams/dynamic_foraging.md @@ -1,19 +1,40 @@ # Dynamic foraging diagrams -These diagrams outline the architecture of the Dynamic Foraging behavior platform. +These diagrams outline the architecture of the Dynamic Foraging platform. ## Lab Management System -![Mid-level Processing Pipeline Library](dynamic_foraging/mid_level/lab_management_df.svg) +Dynamic Foraging relies on the lab management system for session and subject +planning. It also uses the system's water restriction tracking to support +behavior training. + +![Dynamic Foraging lab management](dynamic_foraging/mid_level/lab_management_df.svg) ## Data Acquisition -![Mid-level](dynamic_foraging/dynamic_foraging_architecture.svg) +![Dynamic Foraging acquisition architecture](dynamic_foraging/dynamic_foraging_architecture.svg) ## Data Storage and Processing ### Mid-level Processing Pipeline -![Mid-level Processing Pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg) + +Data from Dynamic Foraging is processed by two pipelines — a behavior pipeline +and a fiber pipeline. Each runs independently and produces a modality-specific +NWB file when it completes successfully. + +![Dynamic Foraging processing pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg) + +#### Fiber Pipeline + +![Dynamic Foraging fiber pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg) + +#### Behavior Pipeline + +![Dynamic Foraging behavior pipeline](dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg) ### Low-level Processing Pipeline Library -![Low-level Processing Pipeline Library](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) + +This pipeline processes raw Harp hardware signals into the trials table, packages +the data into NWB, and runs QC on both the raw and processed data. + +![Dynamic Foraging low-level processing](dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg) From 806feed78cedc8dfbe9ae0cff7cd5d7ce8de7e96 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 19:48:50 -0700 Subject: [PATCH 09/15] chore: add descriptions for processing and qc --- docs/source/diagrams/data_acquisition.md | 10 +++ docs/source/diagrams/data_staging.md | 25 ++++++++ .../diagrams/data_storage_processing.md | 61 +++++++------------ 3 files changed, 56 insertions(+), 40 deletions(-) diff --git a/docs/source/diagrams/data_acquisition.md b/docs/source/diagrams/data_acquisition.md index 74747e1..2a900e0 100644 --- a/docs/source/diagrams/data_acquisition.md +++ b/docs/source/diagrams/data_acquisition.md @@ -1,3 +1,13 @@ # Data Acquisition +:::{note} +This diagram is a **proposal** for the instrumentation software architecture — a +planned design rather than a system currently in production. +::: + +This diagram presents a proposed software architecture for AIND instrumentation +during data acquisition. It outlines how the acquisition software components are +intended to interact to control the instrument and capture data during an +experiment. + ![SIPE instrument diagram](mid_level/sipe_instrument.drawio.svg) diff --git a/docs/source/diagrams/data_staging.md b/docs/source/diagrams/data_staging.md index 720a4bc..c4addfb 100644 --- a/docs/source/diagrams/data_staging.md +++ b/docs/source/diagrams/data_staging.md @@ -1,3 +1,28 @@ # Data Staging +The local data center hosts a variety of services that stage and process data on +its way to the cloud: + +`aind-data-transfer-service` +: FastAPI service that accepts and validates data upload requests, retrieves the +associated metadata, and dispatches compression and transfer jobs to the workflow +manager for execution on the HPC cluster. + +`aind-airflow-service` +: Apache Airflow–based workflow manager that schedules, submits, and monitors the +data processing and upload jobs. + +`slurm` +: HPC cluster for processing and upload tasks. + +`aind-metadata-service` +: REST API that compiles and returns `aind-data-schema` metadata, aggregating and +mapping records from multiple upstream data providers. + +`VAST` +: Shared storage. + +Container registry +: Docker images used to run processing jobs. + ![Local data center flow](mid_level/local_data_center_flow.drawio.svg) diff --git a/docs/source/diagrams/data_storage_processing.md b/docs/source/diagrams/data_storage_processing.md index 88983bf..0818c2d 100644 --- a/docs/source/diagrams/data_storage_processing.md +++ b/docs/source/diagrams/data_storage_processing.md @@ -2,53 +2,34 @@ ## Code Ocean pipeline -![Code Ocean pipeline diagram](mid_level/codeocean_pipeline_diagram.svg) +Raw data lands in S3 as a single data asset that carries its `aind-data-schema` +metadata, with all of a session's modalities associated with one object. +Processing pipelines are modality-specific — each pipeline processes a single +modality. A pipeline outputs an NWB file along with `aind-data-schema` metadata, +including processing metrics and quality control artifacts and metrics. Once the +outputs have been QC'd, they can be combined into a final NWB file with its +associated metadata. -### Pipeline overview +Each pipeline wraps modality- and platform-specific libraries that handle the +underlying data processing, quality control, and NWB packaging. -![Pipeline overview](mid_level/pipeline_overview.svg) +![Code Ocean pipeline diagram](mid_level/codeocean_pipeline_diagram.svg) -### Pipeline close-up -![Pipeline close-up](mid_level/pipeline_close_up.svg) ## Quality control -![QC diagram](mid_level/QC.drawio.svg) - -## Service internals - -Lower-level diagrams of the backend services that store, index, and serve data -and metadata. - -### Asset registration API - -![Asset registration API](low_level/asset_registration_api.svg) - -### Data asset indexer - -![Data asset indexer](low_level/data_asset_indexer.svg) - -### Data schema to DocDB +Each pipeline produces quality control artifacts and metrics alongside its +processed outputs, captured in the asset's `aind-data-schema` metadata. These are +surfaced through the [QC portal](https://github.com/AllenNeuralDynamics/aind-qc-portal), +a web application for viewing and annotating quality control metadata for AIND +data assets. The portal pulls QC metadata from the document database and displays +the corresponding reference figures from the Code Ocean data assets. -![Data schema to DocDB](low_level/data_schema_to_docdb.svg) +Reviewers use the portal to evaluate metrics marked `PENDING`, supporting AIND's +two-step quality control process: first assessing whether an asset's data is +suitable for analysis, then evaluating individual components (such as neurons) +within the asset for usability. -### DocDB API - -![DocDB API](low_level/docdb_api.svg) - -### Redshift client - -![Redshift client](low_level/redshift_client.svg) - -### Metadata service - -![Metadata service overview](low_level/aind-metadata-service/metadata_service_overview_diagram.drawio.svg) - -#### Kubernetes deployment - -![Metadata service Kubernetes deployment](low_level/aind-metadata-service/aind-metadata-service-k8s.drawio.svg) - -#### Procedures extractor +![QC diagram](mid_level/QC.drawio.svg) -![Metadata service procedures extractor](low_level/aind-metadata-service/metadata_service_procedures_extractor_diagram.drawio.svg) From 0263df47b032e10cca55cfe65d4004ec3c708737 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 19:53:25 -0700 Subject: [PATCH 10/15] chore: move diagrams to resources --- docs/source/index.md | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/docs/source/index.md b/docs/source/index.md index 4b9b48c..c98a8cc 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -78,18 +78,10 @@ policies_practices/docs ```{toctree} :maxdepth: 1 :hidden: -:caption: Diagrams - -diagrams/index - -``` - -```{toctree} -:maxdepth: 1 -:hidden: -:caption: AIND Resources +:caption: Resources aind/core_services +diagrams/index aind/project_proposals.md aind/teams.md policies_practices/rse_resources From 6fe67df8e5e3b4293ed91cdd04db11ee022540fe Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 19:57:12 -0700 Subject: [PATCH 11/15] chore: remove diagram from platform specific label --- docs/source/diagrams/index.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/source/diagrams/index.md b/docs/source/diagrams/index.md index 6269b02..d609c5e 100644 --- a/docs/source/diagrams/index.md +++ b/docs/source/diagrams/index.md @@ -18,8 +18,6 @@ components encompass how data are generated and move through the system: - [Dynamic foraging](dynamic_foraging.md) -[![Software Overview](../_static/aind-software-overview.png)](../_static/aind-software-overview.pdf) - ```{toctree} :maxdepth: 1 :hidden: From b822db811a9fe23a1562a894fdb17f188ec4ba2b Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 20:11:25 -0700 Subject: [PATCH 12/15] chore: save background as not transparent --- docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg index 797e1ba..86c40ea 100644 --- a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg +++ b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg @@ -1,4 +1,4 @@ -
Pipeline Modality 1
DocDB
S3: aind-open-data
Pipeline Modality 2
Pipeline Modality 3
NWB File
aind-data-schema metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
DocDB
S3: aind-open-data
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File with aind-data-schema metadata
Plots and metrics
NWB File
aind-data-schema metadata
Plots and metrics
DocDB
S3: aind-open-data
NWB File
aind-data-schema metadata
Plots and metrics
NWB File
aind-data-schema metadata
Plots and metrics
NWB File
aind-data-schema metadata
Plots and metrics
aind-data-schema metadata
Raw Data
modality1
modality2%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20parent%3D%221%22%20style%3D%22html%3D1%3BverticalLabelPosition%3Dbottom%3Balign%3Dcenter%3BlabelBackgroundColor%3D%23ffffff%3BverticalAlign%3Dtop%3BstrokeWidth%3D2%3BstrokeColor%3D%230080F0%3Bshadow%3D0%3Bdashed%3D0%3Bshape%3Dmxgraph.ios7.icons.folder%3B%22%20value%3D%22modality1%22%20vertex%3D%221%22%3E%3CmxGeometry%20height%3D%22130%22%20width%3D%22200%22%20x%3D%22-550%22%20y%3D%22-100%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
modality3
aind-data-schema metadata
modality1
modality2%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20parent%3D%221%22%20style%3D%22html%3D1%3BverticalLabelPosition%3Dbottom%3Balign%3Dcenter%3BlabelBackgroundColor%3D%23ffffff%3BverticalAlign%3Dtop%3BstrokeWidth%3D2%3BstrokeColor%3D%230080F0%3Bshadow%3D0%3Bdashed%3D0%3Bshape%3Dmxgraph.ios7.icons.folder%3B%22%20value%3D%22modality1%22%20vertex%3D%221%22%3E%3CmxGeometry%20height%3D%22130%22%20width%3D%22200%22%20x%3D%22-550%22%20y%3D%22-100%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
modality3
NWB File
processing.json
Tranform and Process
QC NWB FFile
Metadata Manager
+
+
+
Pipeline N
 \ No newline at end of file +
Pipeline Modality 1
DocDB
S3: aind-open-data
Pipeline Modality 2
Pipeline Modality 3
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File with 
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
modality 1
modality 2
modality 3
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
Pipeline N
DocDB
S3: aind-open-data
 \ No newline at end of file From f634dd9101a5956ab2226cb5ac33b7a139324e25 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 20:20:03 -0700 Subject: [PATCH 13/15] chore: add background to general pipeline diagram --- .../codeocean_pipeline_diagram.drawio | 119 ++++++++++-------- .../mid_level/codeocean_pipeline_diagram.svg | 2 +- 2 files changed, 65 insertions(+), 56 deletions(-) diff --git a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.drawio b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.drawio index a0b3067..a54b694 100644 --- a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.drawio +++ b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.drawio @@ -1,11 +1,20 @@ - + + + + + + + + + + - + @@ -28,41 +37,41 @@ - - + + - + - + - + - + - + - + - + - + - + @@ -91,10 +100,10 @@ - + - + @@ -143,8 +152,8 @@ - - + + @@ -177,19 +186,19 @@ - + - + - + - + - + @@ -201,106 +210,106 @@ - + - + - - + + - - + + - - + + - + - - + + - + - - + + - + - + - - + + - + - - + + - + - - + + - + - - + + - + - + - + - + - - + + diff --git a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg index 86c40ea..b5119f4 100644 --- a/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg +++ b/docs/source/diagrams/mid_level/codeocean_pipeline_diagram.svg @@ -1,4 +1,4 @@ -
Pipeline Modality 1
DocDB
S3: aind-open-data
Pipeline Modality 2
Pipeline Modality 3
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File with 
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
modality 1
modality 2
modality 3
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
Pipeline N
DocDB
S3: aind-open-data
 \ No newline at end of file +
Pipeline Modality 1
DocDB
S3: aind-open-data
Pipeline Modality 2
Pipeline Modality 3
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File with 
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
modality 1
modality 2
modality 3
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
Pipeline N
DocDB
S3: aind-open-data
 \ No newline at end of file From 03fa31733c089a75ff49eb0dec5da6403bfa36ab Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 20:25:02 -0700 Subject: [PATCH 14/15] chore: add background to the df pipeline diagram --- .../codeocean_pipeline_diagram_DF.drawio | 234 +++++++++--------- .../codeocean_pipeline_diagram_DF.svg | 2 +- 2 files changed, 124 insertions(+), 112 deletions(-) diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio index 3aa17bf..cc49c11 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio @@ -1,11 +1,23 @@ - + + + + + + + + + + + + + - + @@ -25,38 +37,38 @@ - - + + - + - + - + - + - + - + - + - + @@ -79,10 +91,10 @@ - + - + @@ -116,8 +128,8 @@ - - + + @@ -144,19 +156,19 @@ - + - + - + - + - + @@ -168,41 +180,41 @@ - + - + - - + + - + - - + + - + - + - - + + - + @@ -217,233 +229,233 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - + - + - + - + - + - - + + - - + + - - + + - + - - + + - + - - + + - + - + - - + + - + - - + + - + - - + + - + - - + + - + - + - + - + - + - + - - + + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - - - + + + - - + + - - + + - - + + diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg index 1af56ca..3bf257e 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.svg @@ -1,4 +1,4 @@ -
Fiber Pipeline
DocDB
S3: aind-open-data
DF Behavior Pipeline
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Fib
Behavior
Behavior-videos
DocDB
S3: aind-open-data
 \ No newline at end of file +
Fiber Pipeline
DocDB
S3: aind-open-data
DF Behavior Pipeline
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Raw Data
Fib
Behavior
Behavior-videos
aind-data-schema 
metadata
Fib
Behavior
Behavior-videos
NWB File
processing.json
Transform Raw Fib Streams into NWB File
QC Raw Data
Aggregate Quality Control Data
Fiber Pipeline
DocDB
S3: aind-open-data
DF Behavior Pipeline
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File 
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
Compute DF / F and Package into NWB File
quality_control.json
Artifacts
NWB File
quality_control.json
 \ No newline at end of file From 42e7bdb2b9cf20c71174f15bb7850ce154119b42 Mon Sep 17 00:00:00 2001 From: Arielle Leon Date: Mon, 29 Jun 2026 20:29:06 -0700 Subject: [PATCH 15/15] chore: add backgrounds to the dynamic foraging pipelines --- .../codeocean_pipeline_diagram_DF.drawio | 102 +++++++++--------- ...codeocean_pipeline_diagram_DF_behavior.svg | 2 +- .../codeocean_pipeline_diagram_DF_closeup.svg | 2 +- .../codeocean_pipeline_diagram_DF_fiber.svg | 2 +- 4 files changed, 54 insertions(+), 54 deletions(-) diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio index cc49c11..c201bf2 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF.drawio @@ -5,7 +5,7 @@ - + @@ -229,158 +229,158 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - + - + - + - + - + - - + + - - + + - - + + - + - - + + - + - - + + - + - + - - + + - + - - + + - + - - + + - + - - + + - + - + - + diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg index 3a6830c..a958e26 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_behavior.svg @@ -1,4 +1,4 @@ -
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File 
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
 \ No newline at end of file +
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File 
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
 \ No newline at end of file diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_closeup.svg b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_closeup.svg index 3449416..fca1b2b 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_closeup.svg +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_closeup.svg @@ -1,4 +1,4 @@ -
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
Fib
Behavior
Behavior-videos
aind-data-schema 
metadata
NWB File
processing.json
Transform Raw Fib Streams into NWB File
QC Raw Data
Aggregate Quality Control Data
Fiber Pipeline
DF Behavior Pipeline
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File with 
aind-data-schema 
metadata
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
modality 1
modality 2
modality 3
aind-data-schema 
metadata
NWB File
processing.json
Tranform and Process
QC NWB File
Metadata Manager
+
+
+
Compute DF / F and Package into NWB File
quality_control.json
Artifacts
NWB File
quality_control.json
 \ No newline at end of file +
Fiber Pipeline
DocDB
S3: aind-open-data
DF Behavior Pipeline
NWB File
aind-data-schema 
metadata
aind-data-schema metadata:
  • procedures.json
  • subject.json
  • data_description.json
  • processing.json
  • data_description.json
  • acquisition.json
  • instrument.json
Raw Data
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
NWB File
aind-data-schema 
metadata
Plots and metrics
aind-data-schema 
metadata
Fib
Behavior
Behavior-videos
DocDB
S3: aind-open-data
 \ No newline at end of file diff --git a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg index 51d8847..c00452b 100644 --- a/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg +++ b/docs/source/diagrams/dynamic_foraging/mid_level/codeocean_pipeline_diagram_DF_fiber.svg @@ -1,4 +1,4 @@ -
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
Fib
Behavior
Behavior-videos
aind-data-schema 
metadata
NWB File
processing.json
Transform Raw Fib Streams into NWB File
QC Raw Data
Aggregate Quality Control Data
Compute DF / F and Package into NWB File
quality_control.json
Artifacts
NWB File
quality_control.json
 \ No newline at end of file +
DocDB
S3: aind-open-data
quality_control.json
Artifacts
data_description.json
NWB File
aind-data-schema 
metadata
Plots and metrics
DocDB
S3: aind-open-data
Raw Data
Fib
Behavior
Behavior-videos
aind-data-schema 
metadata
NWB File
processing.json
Transform Raw Fib Streams into NWB File
QC Raw Data
Aggregate Quality Control Data
Compute DF / F and Package into NWB File
quality_control.json
Artifacts
NWB File
quality_control.json
 \ No newline at end of file