Document Pipeline Standards for multi-backend support

[seanmcculloch]

Broad ticket spawned by discussion on 9/8 regarding alignment on use of CO

[seanmcculloch]

slides: https://alleninstitute-my.sharepoint.com/:p:/g/personal/alessio_buccino_alleninstitute_org/IQDZAsNEr1FESZlY4oVo-66lAc5fZrGDw88Jen5pV-UdhYs?e=rzMtDc

[arielleleon]

Thanks, everyone, for meeting! Here are my notes

What to include in the documentation
• Document pipeline and process structure
○ High level diagram on configs, structure and deployment (github, ghcr, code ocean)
• Rely only on GHCR
• Use semantic versioning (still need to hash details)
○ Use feat -> dev -> main with manual main push to trigger release
○ Upstream features need to bubble up to pipeline changelog (do we write this manually)?
• Standard logging in all pipelines (I don't have notes on this discussion)
• Testing nf-test is what pytest is for python (check connections) as well as automatic integration tests
• Process testing unit testing + integration testing in github ci/cd
• How we log parameters and automated building of the app panel
• Web app to generate and validate user parameters
• Tease out general vs pipeline specific features to align on central documentation
Pipeline specific docs are accessible from central documentation