feat: add the sre factory

Makefile

@@ -16,7 +16,7 @@ KORA_CRDS_CHART ?= $(KORA_CHARTS)/kora-crds --version $(KORA_VERSION)
 SCHEMAS := .schemas
 
 # The kustomizations make validate builds, one per example repository.
-KUSTOMIZATIONS := single-tenant/platform single-tenant/tenant multi-tenant/platform multi-tenant/engineering multi-tenant/sre factories/autofix/platform factories/autofix/factory
+KUSTOMIZATIONS := single-tenant/platform single-tenant/tenant multi-tenant/platform multi-tenant/engineering multi-tenant/sre factories/autofix/platform factories/autofix/factory factories/sre/platform factories/sre/factory factories/sre/demo
 
 KUBECONFORM := kubeconform -strict -summary \
 	-schema-location default \

README.md

@@ -19,6 +19,7 @@ An example is a set of Kubernetes manifests that declare tenants, models, agents
 *   [`single-tenant`](single-tenant): One tenant with an agent that summarises pull requests.
 *   [`multi-tenant`](multi-tenant): An engineering team and an SRE team, each in a separate tenant and repository.
 *   [`factories/autofix`](factories/autofix): A factory of four agents that finds, verifies and fixes problems in a GitHub repository and hands a person the pull request.
+*   [`factories/sre`](factories/sre): A factory that triages alerts in the cluster, sweeps its costs with OpenCost and proposes fixes as pull requests.
 
 Each example splits its manifests by owner:
 

factories/sre/README.md

@@ -0,0 +1,74 @@
+# SRE factory
+
+The SRE factory is an example of a lights-off incident response loop in runkora. When an alert fires, `incident-triage` investigates the cluster and records the incident in GitHub, and `remediator` opens a pull request with the fix. A person reviews and merges it. When the alert resolves, `incident-triage` writes up the incident and closes it. Once a week, `cost-sweeper` reviews what the cluster costs and files an issue with savings, which `remediator` turns into a pull request when a manifest change applies.
+
+## How the factory works
+
+1.  **Triage:** An alert fires and `incident-triage` starts. Alerts from the same group join the same run. It reads the pods, logs, events and rollouts in the alert's namespace with `kubectl`.
+2.  **Record:** It opens an `incident` issue in the deploy repository with the cause, the evidence and the impact. When a manifest change fixes the cause, it adds the `remediate` label.
+3.  **Fix:** `remediator` makes the change and opens a pull request. The incident stays open until its alert resolves.
+4.  **Respond:** Comments on the incident go to `incident-triage`. Reviews and comments on the pull request go to `remediator`.
+5.  **Resolve:** A person merges the fix. When the alert resolves, `incident-triage` writes a summary and closes the incident.
+6.  **Sweep:** Every Monday, `cost-sweeper` reads the cluster's costs from OpenCost and opens a `cost` issue with savings. It adds `remediate` when a manifest change applies.
+
+The remediator can push to the deploy repository, and pushing also allows merging. Protect your default branch with a required approving review. The factory cannot approve its own pull requests.
+
+## What each agent can reach
+
+Each agent is set up with an example configuration:
+
+| Agent | Model | Cluster access | GitHub actions | Monthly budget |
+| --- | --- | --- | --- | --- |
+| `incident-triage` | `anthropic/claude-haiku-5-5` | `workloads:read`, `config:read`, `networking:read`, `cluster:read` | `read`, `issues:write` | $30 |
+| `remediator` | `anthropic/claude-haiku-5-5` | None | `read`, `contents:write`, `pulls:write`, `issues:write` | $20 |
+| `cost-sweeper` | `anthropic/claude-haiku-5-5` | OpenCost `get_allocation_costs`, `get_asset_costs`, `get_efficiency` | `read`, `issues:write` | $10 |
+
+`incident-triage` reads the cluster through a kubernetes connection with a read-only ServiceAccount. It cannot read Secrets. `cost-sweeper` calls OpenCost's MCP server through a relay in the tenant namespace.
+
+## Adapt the agents to your cluster
+
+The prompts are general. Teach each agent about your cluster: your runbooks, how your services depend on each other and where the deploy repository keeps each workload.
+
+## Set it up
+
+1.  Create a GitHub repository for your deploy manifests and incidents, and push `demo/` to it.
+2.  Create a GitHub App with read and write access to contents, issues and pull requests, install it on the repository and generate a private key.
+3.  In `factory/shared/github-connection.yaml`, set `app_id`, `installation_id` and `owner`.
+4.  In every file in `factory/triggers/`, replace `example-org/example-deploy` with your repository and `example-sre[bot]` with your App's bot user `<app-slug>[bot]`.
+5.  On the quickstart, set `GITHUB_APP_PRIVATE_KEY_FILE` in `quickstart/.env`, uncomment `quickstart/providers/openrouter.yaml` and run:
+
+    ```bash
+    make credentials providers example EXAMPLE=factories/sre
+    ```
+
+    `make example` lists anything missing first. It installs OpenCost, connects the factory to the quickstart's API server and deploys the broken `checkout` service from `demo/`.
+6.  Run `make tunnel` and add a webhook to your repository with the URL and secret it prints. Select the issues, issue comments and pull request reviews events. See [Receive webhooks locally](../../quickstart/README.md#receive-webhooks-locally).
+
+Sign in as `admin@oncall.local.runkora.io` to watch the runs. `sre@oncall.local.runkora.io` can see the agents, triggers and runs.
+
+## Fire an alert on the quickstart
+
+The quickstart has no Alertmanager. `make alert` sends the alert it would send for the crash-looping `checkout` pod:
+
+```bash
+make alert
+```
+
+Nothing deploys the repository to the quickstart. After you merge the fix, apply it yourself and resolve the alert:
+
+```bash
+kubectl --context k3d-kora-quickstart apply -k <your clone of the deploy repository>
+make alert STATUS=resolved
+```
+
+## Run it on your own cluster
+
+1.  Copy `platform/` and `factory/` into the repositories your GitOps tool reads.
+2.  In `platform/tenants/oncall.yaml`, replace the `oidc` block with your identity provider's client.
+3.  In `platform/models/`, point each route at an `InferenceProvider` you registered.
+4.  In `factory/shared/cluster-connection.yaml`, set `baseHost` to your API server and `caCert` to its CA certificate.
+5.  Create a `ClusterSecretStore` named `example-secret-store`. Store the App's private key under `oncall-github-app`, the GitHub webhook secret under `oncall-github-webhook` and the Alertmanager token under `oncall-alertmanager-webhook`.
+6.  Point an Alertmanager webhook receiver at the alertmanager source's URL, with the token as its bearer.
+7.  Install OpenCost with `opencost.mcp.enabled: true`. `quickstart/values/opencost.yaml` shows the values for a cluster without Prometheus.
+
+See [Declarative resources](https://github.com/runkora/kora/blob/main/docs/guides/declarative-resources.md), [Register a connection](https://github.com/runkora/kora/blob/main/docs/guides/register-connection.md) and [Create a trigger](https://github.com/runkora/kora/blob/main/docs/guides/create-a-trigger.md) for the resources the factory uses.

factories/sre/demo/kustomization.yaml

@@ -0,0 +1,7 @@
+# A deploy repository with one broken workload for the factory to fix. Push
+# this folder to the repository the factory keeps its incidents in. On the
+# quickstart, make example applies it to the cluster.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - shop/checkout.yaml

factories/sre/demo/shop/checkout.yaml

@@ -0,0 +1,61 @@
+# The checkout service. It starts only when MODE is live. The ConfigMap sets
+# MODE to lvie, so the pod crash-loops and logs why.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: shop
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: checkout
+  namespace: shop
+data:
+  MODE: lvie
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: checkout
+  namespace: shop
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: checkout
+  template:
+    metadata:
+      labels:
+        app: checkout
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: checkout
+          image: busybox:1.37
+          command:
+            - sh
+            - -c
+            - |
+              if [ "$MODE" != live ]; then
+                echo "checkout: unknown MODE '$MODE', expected live" >&2
+                exit 1
+              fi
+              echo "checkout: serving"
+              exec sleep infinity
+          envFrom:
+            - configMapRef:
+                name: checkout
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 32Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: [ALL]

factories/sre/factory/agents/cost-sweeper.yaml

@@ -0,0 +1,38 @@
+apiVersion: runkora.io/v1alpha1
+kind: Agent
+metadata:
+  name: cost-sweeper
+spec:
+  displayName: Cost sweeper
+  description: Reviews the cluster's costs each week with OpenCost and files an issue with rightsizing recommendations and their savings.
+  identityMode: autonomous
+  model:
+    family: anthropic
+    name: claude-haiku-5-5
+  systemPrompt: |
+    You review what the cluster costs with the OpenCost tools and report in
+    the GitHub repository you are given.
+
+    1.  Get the allocation costs by namespace and by workload for the last
+        seven days, and the efficiency of each workload.
+    2.  Find workloads whose CPU or memory requests far exceed their usage,
+        and workloads that sit idle.
+    3.  Search the repository's open issues labelled cost. Do not repeat a
+        recommendation an open issue already makes.
+    4.  Open an issue titled "[cost] Weekly sweep <date>" with the label cost.
+        State the total cost, the five most expensive workloads and each
+        recommendation with its estimated monthly saving.
+    5.  If a workload's manifest is in the repository, state the file and the
+        change to its requests, and add the label remediate.
+  environments:
+    - name: ops
+      default: true
+  mcpTools:
+    - server: opencost
+      tools: [get_allocation_costs, get_asset_costs, get_efficiency]
+  connectors:
+    - connection: github
+      actions: [read, issues:write]
+  budget:
+    limitUSD: 10
+    resetCadence: monthly

factories/sre/factory/agents/incident-triage.yaml

@@ -0,0 +1,44 @@
+apiVersion: runkora.io/v1alpha1
+kind: Agent
+metadata:
+  name: incident-triage
+spec:
+  displayName: Incident triage
+  description: Investigates firing alerts in the cluster, keeps the incident issue up to date, answers comments on incident issues and closes the incident when the alert resolves.
+  identityMode: autonomous
+  model:
+    family: anthropic
+    name: claude-haiku-5-5
+  systemPrompt: |
+    You are on call for the Kubernetes cluster you reach with kubectl. You keep
+    the incident record as issues in the GitHub repository you are given.
+
+    For a firing alert:
+    1.  Search the repository's open issues labelled incident for the alert
+        group in the prompt. If one exists, comment what changed and stop.
+    2.  Investigate with kubectl in the alert's namespace: the pods, their
+        logs and events, and the latest rollouts. Only read.
+    3.  Open an issue titled "[incident] <alertname> in <namespace>" with the
+        label incident. In the body, state the alert group, the likely cause,
+        the evidence and the impact.
+    4.  If a change to a file in the repository fixes the cause, state the
+        file and the change, and add the label remediate.
+
+    For a resolved alert:
+    1.  Find the open incident issue for the alert group.
+    2.  Comment a summary with the timeline, the cause, the fix and any
+        follow-ups, then close the issue.
+
+    For a comment on an incident issue, read the issue and answer it.
+    Investigate again when the comment asks you to.
+  environments:
+    - name: ops
+      default: true
+  connectors:
+    - connection: cluster
+      actions: [workloads:read, config:read, networking:read, cluster:read]
+    - connection: github
+      actions: [read, issues:write]
+  budget:
+    limitUSD: 30
+    resetCadence: monthly

factories/sre/factory/agents/remediator.yaml

@@ -0,0 +1,36 @@
+apiVersion: runkora.io/v1alpha1
+kind: Agent
+metadata:
+  name: remediator
+spec:
+  displayName: Remediator
+  description: Makes the change an incident or cost issue recommends in a pull request to the deploy repository, and answers review and comments on its own pull requests.
+  identityMode: autonomous
+  model:
+    family: anthropic
+    name: claude-haiku-5-5
+  systemPrompt: |
+    You make the change an issue recommends in the GitHub repository that
+    deploys the workload. The issue is an incident or a cost sweep.
+
+    For an issue labelled remediate:
+    1.  Read the issue and the evidence it gives.
+    2.  Clone the repository, make the smallest change that does what the
+        issue recommends and push a branch named remediate/<issue>.
+    3.  Open a pull request that states what you changed and why. For an
+        incident, start the body with "Refs #<issue>". The incident stays
+        open until its alert resolves. For a cost issue, start the body with
+        "Fixes #<issue>".
+    4.  Comment on the issue with a link to the pull request.
+
+    For a review or a comment on your pull request, read every review and
+    comment, address them and push to the same branch. Do not merge.
+  environments:
+    - name: ops
+      default: true
+  connectors:
+    - connection: github
+      actions: [read, contents:write, pulls:write, issues:write]
+  budget:
+    limitUSD: 20
+    resetCadence: monthly

factories/sre/factory/kustomization.yaml

@@ -0,0 +1,32 @@
+# The SRE factory: a tenant team's repository that triages alerts, sweeps the
+# cluster's costs, keeps the record in GitHub issues and proposes fixes to the
+# deploy repository. Point your GitOps tool at this folder, or apply it with
+# kubectl apply -k.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+# The factory's tenant namespace. The resource files leave namespace out.
+namespace: kora-tenant-oncall
+commonAnnotations:
+  # The console shows this value as "Managed by" on each declared item.
+  runkora.io/repository: github.com/runkora/examples
+# Applies the resources in the order below. The OpenCost MCPServer must exist
+# before its relay pod.
+sortOptions:
+  order: fifo
+resources:
+  - shared/cluster-connection.yaml
+  - shared/github-connection.yaml
+  - shared/github-source.yaml
+  - shared/alertmanager-source.yaml
+  - shared/opencost-mcp.yaml
+  - shared/ops-environment.yaml
+  - shared/sre-role.yaml
+  - agents/incident-triage.yaml
+  - agents/remediator.yaml
+  - agents/cost-sweeper.yaml
+  - triggers/triage-alert.yaml
+  - triggers/resolve-alert.yaml
+  - triggers/remediate.yaml
+  - triggers/incident-comment.yaml
+  - triggers/remediator-on-review.yaml
+  - triggers/sweep-weekly.yaml

factories/sre/factory/shared/alertmanager-source.yaml

@@ -0,0 +1,35 @@
+# Alertmanager's webhook receiver. Point an Alertmanager webhook_configs entry
+# at the API host plus status.webhookPath, with the secret stored under the
+# remote key below as its bearer token. It posts alert.firing and
+# alert.resolved events.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: alertmanager-webhook
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: example-secret-store
+  target:
+    name: alertmanager-webhook
+    template:
+      metadata:
+        labels:
+          # kora reads only Secrets that carry this label.
+          runkora.io/trigger-source: "true"
+  data:
+    - secretKey: secret
+      remoteRef:
+        key: oncall-alertmanager-webhook
+        property: secret
+---
+apiVersion: runkora.io/v1alpha1
+kind: TriggerSource
+metadata:
+  name: alertmanager
+spec:
+  provider: alertmanager
+  signingSecretRef:
+    name: alertmanager-webhook
+    key: secret

factories/sre/factory/shared/cluster-connection.yaml

@@ -0,0 +1,36 @@
+# The factory's read-only access to the cluster its alerts come from. The
+# egress gateway adds the ServiceAccount's token to each kubectl request from an
+# agent's sandbox. The ClusterRole in ../../platform/cluster-access.yaml sets
+# what the token can read.
+#
+# On your own cluster, set baseHost to the API server's hostname and caCert to
+# the cluster's CA certificate, such as from aws eks describe-cluster. On the
+# quickstart, make example fills in caCert from the cluster. See
+# https://github.com/runkora/kora/blob/main/docs/guides/register-connection.md.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: cluster-reader
+---
+# The cluster fills in the ServiceAccount's token. No token lives in git.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cluster-reader-token
+  annotations:
+    kubernetes.io/service-account.name: cluster-reader
+type: kubernetes.io/service-account-token
+---
+apiVersion: runkora.io/v1alpha1
+kind: Connection
+metadata:
+  name: cluster
+spec:
+  adapter: kubernetes
+  baseHost: k3d-kora-quickstart-server-0:6443
+  secretRef: cluster-reader-token
+  secretKey: token
+  caCert: |
+    -----BEGIN CERTIFICATE-----
+    <the cluster's CA certificate>
+    -----END CERTIFICATE-----