feat: build documentation for diagrams

[arielleleon]

• adds readthedocs for the diagrams library

---

📚 Documentation preview 📚: https://scicomp-docs--183.org.readthedocs.build/en/183/

[dbirman]

I think something needs to be tweaked, the built version (in the PR comment link) for me doesn't seem to show any of the diagrams?

[arielleleon]

@dbirman - Try now

[dbirman]

Currently these seem mostly like reference material, since they are disconnected from the how-to guides where we have most of our docs content. I think that's okay, but I feel like they would be much more useful embedded alongside the relevant content related to each diagram. They don't need to be in-line, they could appear as hover tooltips or something, or link from that content to a page with each of the diagrams.

At a minimum it would be good to provide a text description of each diagram, and probably use the glossary system that Tom set up for definitions of systems that a user might not know in advance.

I think also in an effort to reduce clutter on the left sidebar we should put this under the AIND Resources section, and probably just rename that "Resources". I will revisit the sidebar in the future though to reduce the number of things there, so not necessary to do that now.

[arielleleon]

Currently these seem mostly like reference material, since they are disconnected from the how-to guides where we have most of our docs content. I think that's okay, but I feel like they would be much more useful embedded alongside the relevant content related to each diagram. They don't need to be in-line, they could appear as hover tooltips or something, or link from that content to a page with each of the diagrams.

At a minimum it would be good to provide a text description of each diagram, and probably use the glossary system that Tom set up for definitions of systems that a user might not know in advance.

I think also in an effort to reduce clutter on the left sidebar we should put this under the AIND Resources section, and probably just rename that "Resources". I will revisit the sidebar in the future though to reduce the number of things there, so not necessary to do that now.

Yeah - this fits. I will add some documentation and put them under the Resources section

[patricklatimer]

Looking at the built docs, it looks like the transparent images aren't working correctly for dark/light mode.

[dougollerenshaw]

Two things I'd like to see fleshed out:

1. All three of the Data Acquisition, Data Staging and Data Storage and Processing diagrams should be more explicit about how simultaneous multi-modal acquisitions are set up. Some things that should be clear from diagrams:

• How are acquisition and instrument files named when two modalities are running simultaneously?
• At what point in the flow are multi-modal instrument and acquisition JSON files merged?
• What specific process handles the merge (there should probably be a block that specifically covers this. I think it's the GatherMetadataJob inside the Metadata Mapper, but I'm not completely sure)

2. The distinction between creating valid metadata on the rigs vs extractor/mapper pattern should be shown explicitly in the diagrams. Not sure which one this belongs in. But this has been a consistent source of confusion, so we should make a clear distinction between:

• Generating aind-schema compliant acquisition metadata at the rig (no further action needed)
• Generating non-compliant extracted metadata at the rig, which is followed by a validation step against the model in the extractor repo, then a call to the metadata mapper to actually generate the aind-schema compliant metadata.

[dougollerenshaw]

Minor, but important. The text in the Data Storage and Processing diagrams is unreadably small in the pre-merge rendered version (https://scicomp-docs--183.org.readthedocs.build/en/183/diagrams/data_storage_processing.html)

[arielleleon]

Two things I'd like to see fleshed out:

1. All three of the Data Acquisition, Data Staging and Data Storage and Processing diagrams should be more explicit about how simultaneous multi-modal acquisitions are set up. Some things that should be clear from diagrams:

• How are acquisition and instrument files named when two modalities are running simultaneously?
• At what point in the flow are multi-modal instrument and acquisition JSON files merged?
• What specific process handles the merge (there should probably be a block that specifically covers this. I think it's the GatherMetadataJob inside the Metadata Mapper, but I'm not completely sure)

2. The distinction between creating valid metadata on the rigs vs extractor/mapper pattern should be shown explicitly in the diagrams. Not sure which one this belongs in. But this has been a consistent source of confusion, so we should make a clear distinction between:

• Generating aind-schema compliant acquisition metadata at the rig (no further action needed)
• Generating non-compliant extracted metadata at the rig, which is followed by a validation step against the model in the extractor repo, then a call to the metadata mapper to actually generate the aind-schema compliant metadata.

@dougollerenshaw - A lot of these will be generated and added to the RTD through this issue #92 . @jtyoung84 , @patricklatimer and I will continue to iterate on the documentation as we resolve issues from the previously referenced issue