Kubernetes deployment of KernelCI API/Pipeline

[nuclearcat]

Description

This task is responsible for deploying the KernelCI API and Pipeline.
As development approaching pre-production stage, we need to have a way to deploy the KernelCI API and Pipeline in a Kubernetes cluster.
Based on prior experience and taking in account design of new API/Pipeline, we have decided to use Kubernetes as the deployment platform.

Components

• API consists of several components - redis container, API backend, MongoDB Atlas SaaS.
• Pipeline consists of several components - monitor, generic scheduler, scheduler-k8s, scheduler-lava, tarball, timeout(separate daemons for mode: timeout, closing, holdoff), trigger
• Two ingress controllers, each for its own namespace
• Single instance of ClusterIssuer for ACME integration (cert-manager)

Requirements

• Deploy KernelCI API and Pipeline in a Kubernetes cluster over two separate namespaces, with ACME integration, with at least two ingress endpoints - API REST interface, LAVA callback endpoint.
• Allow to configure API and Pipeline components via k8s ConfigMap and Secrets.
• Allow to rollout at least weekly updates of API and Pipeline components with minimal downtime with option to rollback to previous version in case of failure.

Action items

• Create a Kubernetes cluster
• Create a namespace for API, with ingress controller, ClusterIssuer, MongoDB Atlas SaaS, redis container, API backend, for a moment "by hand"
• Create a second namespace for Pipeline, with second ingress controller, using same ClusterIssuer, with all components, for a moment "by hand"
• Make sure all components are working as expected
• Create unified deploy scripts in kernelci-deploy that use manifest files from kube directories in kernelci-api and kernelci-pipeline
• Create a script to update API and Pipeline components in a rolling fashion
• Create a script to simulate rollback of API and Pipeline components

Additional action items due discovered incomplete features

• Create Dockerfile for API backend
• Create Dockerfile for Pipeline components
• Create Dockerfile for separate lava-callback service
• Create script to build versioned Docker images for API and Pipeline components after testing on staging

Additional actions

• Share experience and possible issues related to Kubernetes deployment

[nuclearcat]

Work in progress PRs:
kernelci/kernelci-api#393
kernelci/kernelci-pipeline#354

[nuclearcat]

Closing as completed. The maintained repositories now contain the deployment artifacts requested here: production Dockerfiles and AKS manifests in kernelci-pipeline, plus kubernetes/api-pipeline-deploy.sh, kubernetes/api-production-update.sh, and maintained production playbooks in kernelci-deploy. API and Pipeline are actively deployed with tagged rollouts, so this initial deployment-buildout issue has reached its intended outcome.