diff --git a/Makefile b/Makefile index 68dff34..4eeb75e 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ KORA_CRDS_CHART ?= $(KORA_CHARTS)/kora-crds --version $(KORA_VERSION) SCHEMAS := .schemas # The kustomizations make validate builds, one per example repository. -KUSTOMIZATIONS := single-tenant/platform single-tenant/tenant multi-tenant/platform multi-tenant/engineering multi-tenant/sre factories/autofix/platform factories/autofix/factory +KUSTOMIZATIONS := single-tenant/platform single-tenant/tenant multi-tenant/platform multi-tenant/engineering multi-tenant/sre factories/autofix/platform factories/autofix/factory factories/sre/platform factories/sre/factory factories/sre/demo KUBECONFORM := kubeconform -strict -summary \ -schema-location default \ diff --git a/README.md b/README.md index 6700f3d..8949b69 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ An example is a set of Kubernetes manifests that declare tenants, models, agents * [`single-tenant`](single-tenant): One tenant with an agent that summarises pull requests. * [`multi-tenant`](multi-tenant): An engineering team and an SRE team, each in a separate tenant and repository. * [`factories/autofix`](factories/autofix): A factory of four agents that finds, verifies and fixes problems in a GitHub repository and hands a person the pull request. +* [`factories/sre`](factories/sre): A factory that triages alerts in the cluster, sweeps its costs with OpenCost and proposes fixes as pull requests. Each example splits its manifests by owner: diff --git a/factories/sre/README.md b/factories/sre/README.md new file mode 100644 index 0000000..e9013f5 --- /dev/null +++ b/factories/sre/README.md @@ -0,0 +1,74 @@ +# SRE factory + +The SRE factory is an example of a lights-off incident response loop in runkora. When an alert fires, `incident-triage` investigates the cluster and records the incident in GitHub, and `remediator` opens a pull request with the fix. A person reviews and merges it. When the alert resolves, `incident-triage` writes up the incident and closes it. Once a week, `cost-sweeper` reviews what the cluster costs and files an issue with savings, which `remediator` turns into a pull request when a manifest change applies. + +## How the factory works + +1. **Triage:** An alert fires and `incident-triage` starts. Alerts from the same group join the same run. It reads the pods, logs, events and rollouts in the alert's namespace with `kubectl`. +2. **Record:** It opens an `incident` issue in the deploy repository with the cause, the evidence and the impact. When a manifest change fixes the cause, it adds the `remediate` label. +3. **Fix:** `remediator` makes the change and opens a pull request. The incident stays open until its alert resolves. +4. **Respond:** Comments on the incident go to `incident-triage`. Reviews and comments on the pull request go to `remediator`. +5. **Resolve:** A person merges the fix. When the alert resolves, `incident-triage` writes a summary and closes the incident. +6. **Sweep:** Every Monday, `cost-sweeper` reads the cluster's costs from OpenCost and opens a `cost` issue with savings. It adds `remediate` when a manifest change applies. + +The remediator can push to the deploy repository, and pushing also allows merging. Protect your default branch with a required approving review. The factory cannot approve its own pull requests. + +## What each agent can reach + +Each agent is set up with an example configuration: + +| Agent | Model | Cluster access | GitHub actions | Monthly budget | +| --- | --- | --- | --- | --- | +| `incident-triage` | `anthropic/claude-haiku-5-5` | `workloads:read`, `config:read`, `networking:read`, `cluster:read` | `read`, `issues:write` | $30 | +| `remediator` | `anthropic/claude-haiku-5-5` | None | `read`, `contents:write`, `pulls:write`, `issues:write` | $20 | +| `cost-sweeper` | `anthropic/claude-haiku-5-5` | OpenCost `get_allocation_costs`, `get_asset_costs`, `get_efficiency` | `read`, `issues:write` | $10 | + +`incident-triage` reads the cluster through a kubernetes connection with a read-only ServiceAccount. It cannot read Secrets. `cost-sweeper` calls OpenCost's MCP server through a relay in the tenant namespace. + +## Adapt the agents to your cluster + +The prompts are general. Teach each agent about your cluster: your runbooks, how your services depend on each other and where the deploy repository keeps each workload. + +## Set it up + +1. Create a GitHub repository for your deploy manifests and incidents, and push `demo/` to it. +2. Create a GitHub App with read and write access to contents, issues and pull requests, install it on the repository and generate a private key. +3. In `factory/shared/github-connection.yaml`, set `app_id`, `installation_id` and `owner`. +4. In every file in `factory/triggers/`, replace `example-org/example-deploy` with your repository and `example-sre[bot]` with your App's bot user `[bot]`. +5. On the quickstart, set `GITHUB_APP_PRIVATE_KEY_FILE` in `quickstart/.env`, uncomment `quickstart/providers/openrouter.yaml` and run: + + ```bash + make credentials providers example EXAMPLE=factories/sre + ``` + + `make example` lists anything missing first. It installs OpenCost, connects the factory to the quickstart's API server and deploys the broken `checkout` service from `demo/`. +6. Run `make tunnel` and add a webhook to your repository with the URL and secret it prints. Select the issues, issue comments and pull request reviews events. See [Receive webhooks locally](../../quickstart/README.md#receive-webhooks-locally). + +Sign in as `admin@oncall.local.runkora.io` to watch the runs. `sre@oncall.local.runkora.io` can see the agents, triggers and runs. + +## Fire an alert on the quickstart + +The quickstart has no Alertmanager. `make alert` sends the alert it would send for the crash-looping `checkout` pod: + +```bash +make alert +``` + +Nothing deploys the repository to the quickstart. After you merge the fix, apply it yourself and resolve the alert: + +```bash +kubectl --context k3d-kora-quickstart apply -k +make alert STATUS=resolved +``` + +## Run it on your own cluster + +1. Copy `platform/` and `factory/` into the repositories your GitOps tool reads. +2. In `platform/tenants/oncall.yaml`, replace the `oidc` block with your identity provider's client. +3. In `platform/models/`, point each route at an `InferenceProvider` you registered. +4. In `factory/shared/cluster-connection.yaml`, set `baseHost` to your API server and `caCert` to its CA certificate. +5. Create a `ClusterSecretStore` named `example-secret-store`. Store the App's private key under `oncall-github-app`, the GitHub webhook secret under `oncall-github-webhook` and the Alertmanager token under `oncall-alertmanager-webhook`. +6. Point an Alertmanager webhook receiver at the alertmanager source's URL, with the token as its bearer. +7. Install OpenCost with `opencost.mcp.enabled: true`. `quickstart/values/opencost.yaml` shows the values for a cluster without Prometheus. + +See [Declarative resources](https://github.com/runkora/kora/blob/main/docs/guides/declarative-resources.md), [Register a connection](https://github.com/runkora/kora/blob/main/docs/guides/register-connection.md) and [Create a trigger](https://github.com/runkora/kora/blob/main/docs/guides/create-a-trigger.md) for the resources the factory uses. diff --git a/factories/sre/demo/kustomization.yaml b/factories/sre/demo/kustomization.yaml new file mode 100644 index 0000000..1815bd5 --- /dev/null +++ b/factories/sre/demo/kustomization.yaml @@ -0,0 +1,7 @@ +# A deploy repository with one broken workload for the factory to fix. Push +# this folder to the repository the factory keeps its incidents in. On the +# quickstart, make example applies it to the cluster. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - shop/checkout.yaml diff --git a/factories/sre/demo/shop/checkout.yaml b/factories/sre/demo/shop/checkout.yaml new file mode 100644 index 0000000..0790f16 --- /dev/null +++ b/factories/sre/demo/shop/checkout.yaml @@ -0,0 +1,61 @@ +# The checkout service. It starts only when MODE is live. The ConfigMap sets +# MODE to lvie, so the pod crash-loops and logs why. +apiVersion: v1 +kind: Namespace +metadata: + name: shop +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: checkout + namespace: shop +data: + MODE: lvie +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: checkout + namespace: shop +spec: + replicas: 1 + selector: + matchLabels: + app: checkout + template: + metadata: + labels: + app: checkout + spec: + securityContext: + runAsNonRoot: true + runAsUser: 65534 + seccompProfile: + type: RuntimeDefault + containers: + - name: checkout + image: busybox:1.37 + command: + - sh + - -c + - | + if [ "$MODE" != live ]; then + echo "checkout: unknown MODE '$MODE', expected live" >&2 + exit 1 + fi + echo "checkout: serving" + exec sleep infinity + envFrom: + - configMapRef: + name: checkout + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + memory: 32Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] diff --git a/factories/sre/factory/agents/cost-sweeper.yaml b/factories/sre/factory/agents/cost-sweeper.yaml new file mode 100644 index 0000000..5f4153e --- /dev/null +++ b/factories/sre/factory/agents/cost-sweeper.yaml @@ -0,0 +1,38 @@ +apiVersion: runkora.io/v1alpha1 +kind: Agent +metadata: + name: cost-sweeper +spec: + displayName: Cost sweeper + description: Reviews the cluster's costs each week with OpenCost and files an issue with rightsizing recommendations and their savings. + identityMode: autonomous + model: + family: anthropic + name: claude-haiku-5-5 + systemPrompt: | + You review what the cluster costs with the OpenCost tools and report in + the GitHub repository you are given. + + 1. Get the allocation costs by namespace and by workload for the last + seven days, and the efficiency of each workload. + 2. Find workloads whose CPU or memory requests far exceed their usage, + and workloads that sit idle. + 3. Search the repository's open issues labelled cost. Do not repeat a + recommendation an open issue already makes. + 4. Open an issue titled "[cost] Weekly sweep " with the label cost. + State the total cost, the five most expensive workloads and each + recommendation with its estimated monthly saving. + 5. If a workload's manifest is in the repository, state the file and the + change to its requests, and add the label remediate. + environments: + - name: ops + default: true + mcpTools: + - server: opencost + tools: [get_allocation_costs, get_asset_costs, get_efficiency] + connectors: + - connection: github + actions: [read, issues:write] + budget: + limitUSD: 10 + resetCadence: monthly diff --git a/factories/sre/factory/agents/incident-triage.yaml b/factories/sre/factory/agents/incident-triage.yaml new file mode 100644 index 0000000..925c9d7 --- /dev/null +++ b/factories/sre/factory/agents/incident-triage.yaml @@ -0,0 +1,44 @@ +apiVersion: runkora.io/v1alpha1 +kind: Agent +metadata: + name: incident-triage +spec: + displayName: Incident triage + description: Investigates firing alerts in the cluster, keeps the incident issue up to date, answers comments on incident issues and closes the incident when the alert resolves. + identityMode: autonomous + model: + family: anthropic + name: claude-haiku-5-5 + systemPrompt: | + You are on call for the Kubernetes cluster you reach with kubectl. You keep + the incident record as issues in the GitHub repository you are given. + + For a firing alert: + 1. Search the repository's open issues labelled incident for the alert + group in the prompt. If one exists, comment what changed and stop. + 2. Investigate with kubectl in the alert's namespace: the pods, their + logs and events, and the latest rollouts. Only read. + 3. Open an issue titled "[incident] in " with the + label incident. In the body, state the alert group, the likely cause, + the evidence and the impact. + 4. If a change to a file in the repository fixes the cause, state the + file and the change, and add the label remediate. + + For a resolved alert: + 1. Find the open incident issue for the alert group. + 2. Comment a summary with the timeline, the cause, the fix and any + follow-ups, then close the issue. + + For a comment on an incident issue, read the issue and answer it. + Investigate again when the comment asks you to. + environments: + - name: ops + default: true + connectors: + - connection: cluster + actions: [workloads:read, config:read, networking:read, cluster:read] + - connection: github + actions: [read, issues:write] + budget: + limitUSD: 30 + resetCadence: monthly diff --git a/factories/sre/factory/agents/remediator.yaml b/factories/sre/factory/agents/remediator.yaml new file mode 100644 index 0000000..71c166c --- /dev/null +++ b/factories/sre/factory/agents/remediator.yaml @@ -0,0 +1,36 @@ +apiVersion: runkora.io/v1alpha1 +kind: Agent +metadata: + name: remediator +spec: + displayName: Remediator + description: Makes the change an incident or cost issue recommends in a pull request to the deploy repository, and answers review and comments on its own pull requests. + identityMode: autonomous + model: + family: anthropic + name: claude-haiku-5-5 + systemPrompt: | + You make the change an issue recommends in the GitHub repository that + deploys the workload. The issue is an incident or a cost sweep. + + For an issue labelled remediate: + 1. Read the issue and the evidence it gives. + 2. Clone the repository, make the smallest change that does what the + issue recommends and push a branch named remediate/. + 3. Open a pull request that states what you changed and why. For an + incident, start the body with "Refs #". The incident stays + open until its alert resolves. For a cost issue, start the body with + "Fixes #". + 4. Comment on the issue with a link to the pull request. + + For a review or a comment on your pull request, read every review and + comment, address them and push to the same branch. Do not merge. + environments: + - name: ops + default: true + connectors: + - connection: github + actions: [read, contents:write, pulls:write, issues:write] + budget: + limitUSD: 20 + resetCadence: monthly diff --git a/factories/sre/factory/kustomization.yaml b/factories/sre/factory/kustomization.yaml new file mode 100644 index 0000000..203a6e9 --- /dev/null +++ b/factories/sre/factory/kustomization.yaml @@ -0,0 +1,32 @@ +# The SRE factory: a tenant team's repository that triages alerts, sweeps the +# cluster's costs, keeps the record in GitHub issues and proposes fixes to the +# deploy repository. Point your GitOps tool at this folder, or apply it with +# kubectl apply -k. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +# The factory's tenant namespace. The resource files leave namespace out. +namespace: kora-tenant-oncall +commonAnnotations: + # The console shows this value as "Managed by" on each declared item. + runkora.io/repository: github.com/runkora/examples +# Applies the resources in the order below. The OpenCost MCPServer must exist +# before its relay pod. +sortOptions: + order: fifo +resources: + - shared/cluster-connection.yaml + - shared/github-connection.yaml + - shared/github-source.yaml + - shared/alertmanager-source.yaml + - shared/opencost-mcp.yaml + - shared/ops-environment.yaml + - shared/sre-role.yaml + - agents/incident-triage.yaml + - agents/remediator.yaml + - agents/cost-sweeper.yaml + - triggers/triage-alert.yaml + - triggers/resolve-alert.yaml + - triggers/remediate.yaml + - triggers/incident-comment.yaml + - triggers/remediator-on-review.yaml + - triggers/sweep-weekly.yaml diff --git a/factories/sre/factory/shared/alertmanager-source.yaml b/factories/sre/factory/shared/alertmanager-source.yaml new file mode 100644 index 0000000..a094d65 --- /dev/null +++ b/factories/sre/factory/shared/alertmanager-source.yaml @@ -0,0 +1,35 @@ +# Alertmanager's webhook receiver. Point an Alertmanager webhook_configs entry +# at the API host plus status.webhookPath, with the secret stored under the +# remote key below as its bearer token. It posts alert.firing and +# alert.resolved events. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: alertmanager-webhook +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: example-secret-store + target: + name: alertmanager-webhook + template: + metadata: + labels: + # kora reads only Secrets that carry this label. + runkora.io/trigger-source: "true" + data: + - secretKey: secret + remoteRef: + key: oncall-alertmanager-webhook + property: secret +--- +apiVersion: runkora.io/v1alpha1 +kind: TriggerSource +metadata: + name: alertmanager +spec: + provider: alertmanager + signingSecretRef: + name: alertmanager-webhook + key: secret diff --git a/factories/sre/factory/shared/cluster-connection.yaml b/factories/sre/factory/shared/cluster-connection.yaml new file mode 100644 index 0000000..c7ce3d6 --- /dev/null +++ b/factories/sre/factory/shared/cluster-connection.yaml @@ -0,0 +1,36 @@ +# The factory's read-only access to the cluster its alerts come from. The +# egress gateway adds the ServiceAccount's token to each kubectl request from an +# agent's sandbox. The ClusterRole in ../../platform/cluster-access.yaml sets +# what the token can read. +# +# On your own cluster, set baseHost to the API server's hostname and caCert to +# the cluster's CA certificate, such as from aws eks describe-cluster. On the +# quickstart, make example fills in caCert from the cluster. See +# https://github.com/runkora/kora/blob/main/docs/guides/register-connection.md. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: cluster-reader +--- +# The cluster fills in the ServiceAccount's token. No token lives in git. +apiVersion: v1 +kind: Secret +metadata: + name: cluster-reader-token + annotations: + kubernetes.io/service-account.name: cluster-reader +type: kubernetes.io/service-account-token +--- +apiVersion: runkora.io/v1alpha1 +kind: Connection +metadata: + name: cluster +spec: + adapter: kubernetes + baseHost: k3d-kora-quickstart-server-0:6443 + secretRef: cluster-reader-token + secretKey: token + caCert: | + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- diff --git a/factories/sre/factory/shared/github-connection.yaml b/factories/sre/factory/shared/github-connection.yaml new file mode 100644 index 0000000..d58b63a --- /dev/null +++ b/factories/sre/factory/shared/github-connection.yaml @@ -0,0 +1,37 @@ +# The factory's GitHub App installation. Store the App's private key under the +# remote key below and set the config values from the App's settings. See +# https://github.com/runkora/kora/blob/main/docs/guides/register-connection.md. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: github-app +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: example-secret-store + target: + name: github-app + data: + - secretKey: key.pem + remoteRef: + key: oncall-github-app + property: key.pem +--- +apiVersion: runkora.io/v1alpha1 +kind: Connection +metadata: + name: github +spec: + adapter: github_app + baseHost: github.com + secretRef: github-app + secretKey: key.pem + config: + # The App ID from the App's settings page. + app_id: "123456" + # The number at the end of the installation's settings URL. + installation_id: 12345678 + # The organisation the App is installed on. kora denies requests for any + # other owner. + owner: example-org diff --git a/factories/sre/factory/shared/github-source.yaml b/factories/sre/factory/shared/github-source.yaml new file mode 100644 index 0000000..fa3455f --- /dev/null +++ b/factories/sre/factory/shared/github-source.yaml @@ -0,0 +1,34 @@ +# The GitHub webhook the factory's triggers read events from. Point a webhook +# on the deploy repository at the API host plus status.webhookPath, with the +# signing secret stored under the remote key below. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: github-webhook +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: example-secret-store + target: + name: github-webhook + template: + metadata: + labels: + # kora reads only Secrets that carry this label. + runkora.io/trigger-source: "true" + data: + - secretKey: secret + remoteRef: + key: oncall-github-webhook + property: secret +--- +apiVersion: runkora.io/v1alpha1 +kind: TriggerSource +metadata: + name: github +spec: + provider: github + signingSecretRef: + name: github-webhook + key: secret diff --git a/factories/sre/factory/shared/opencost-mcp.yaml b/factories/sre/factory/shared/opencost-mcp.yaml new file mode 100644 index 0000000..967ae8a --- /dev/null +++ b/factories/sre/factory/shared/opencost-mcp.yaml @@ -0,0 +1,88 @@ +# OpenCost's MCP server. OpenCost runs in the opencost namespace and serves MCP +# on port 8081. The tool gateway only reaches tenant namespaces, so a TCP relay +# here fronts the opencost Service. Each tool becomes the scope +# mcp:opencost::use. +# +# The MCPServer comes before the relay. kora reads it to admit the relay's pod +# as an MCP server. A pod admitted first is treated as an agent pod and never +# becomes ready. +apiVersion: runkora.io/v1alpha1 +kind: MCPServer +metadata: + name: opencost +spec: + displayName: OpenCost + description: Cluster cost allocation by namespace, workload and label, node and disk asset costs, and CPU and memory rightsizing. + endpoint: http://opencost-mcp.kora-tenant-oncall.svc:8000/mcp + transport: streamable-http + reintrospectIntervalSeconds: 300 + workload: + selector: + app: opencost-mcp +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: opencost-mcp + labels: + app: opencost-mcp +spec: + replicas: 1 + selector: + matchLabels: + app: opencost-mcp + template: + metadata: + labels: + app: opencost-mcp + spec: + # Tenant namespaces enforce the restricted Pod Security profile. + securityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + containers: + - name: relay + image: alpine/socat:1.8.1.3@sha256:22d77becaeeea8ef56b8c4bb2312323acafe85354be6a85714ff5050e3537842 + args: + - "TCP-LISTEN:8000,fork,reuseaddr" + - "TCP:opencost.opencost.svc.cluster.local:8081" + ports: + - name: mcp + containerPort: 8000 + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + readinessProbe: + tcpSocket: + port: 8000 + initialDelaySeconds: 5 + periodSeconds: 10 + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + cpu: 200m + memory: 64Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: opencost-mcp + labels: + app: opencost-mcp +spec: + selector: + app: opencost-mcp + ports: + # Istio reads the protocol from the port name. An unrecognised name is + # treated as plain TCP, and introspection fails. + - name: http + appProtocol: http + port: 8000 + targetPort: 8000 diff --git a/factories/sre/factory/shared/ops-environment.yaml b/factories/sre/factory/shared/ops-environment.yaml new file mode 100644 index 0000000..168441c --- /dev/null +++ b/factories/sre/factory/shared/ops-environment.yaml @@ -0,0 +1,15 @@ +# The sandbox the agents run kubectl and git in. kora-base ships kubectl, and +# the sandbox writes its kubeconfig for each cluster connection the agent holds. +apiVersion: runkora.io/v1alpha1 +kind: Environment +metadata: + name: ops +spec: + displayName: Ops + description: Inspects the cluster with kubectl and edits the deploy repository. + sandboxImage: kora-base + # Each run gets its own sandbox of this size. + size: + cpu: 500m + memory: 1Gi + disk: 5Gi diff --git a/factories/sre/factory/shared/sre-role.yaml b/factories/sre/factory/shared/sre-role.yaml new file mode 100644 index 0000000..89aeaa6 --- /dev/null +++ b/factories/sre/factory/shared/sre-role.yaml @@ -0,0 +1,14 @@ +# The sre role. People whose groups claim includes sre can see the factory's +# agents and triggers, and read every agent's runs. +apiVersion: runkora.io/v1alpha1 +kind: PolicyRole +metadata: + name: sre +spec: + role: sre + scopes: + - agents:types:read + - triggers:read + - agents:runs:incident-triage:read + - agents:runs:remediator:read + - agents:runs:cost-sweeper:read diff --git a/factories/sre/factory/triggers/incident-comment.yaml b/factories/sre/factory/triggers/incident-comment.yaml new file mode 100644 index 0000000..f755a44 --- /dev/null +++ b/factories/sre/factory/triggers/incident-comment.yaml @@ -0,0 +1,31 @@ +# GitHub sends comments on issues and pull requests as one event. The decision +# model sends a comment on an incident issue to the triage agent and a comment +# on a fix to the remediator, and joins the open run for it when there is one. +apiVersion: runkora.io/v1alpha1 +kind: Trigger +metadata: + name: incident-comment +spec: + source: github + eventType: issue_comment + match: + action: [created] + issue.user.login: ["example-sre[bot]"] + repository.full_name: [example-org/example-deploy] + senderRestriction: + roles: [OWNER, MEMBER, COLLABORATOR] + routing: + candidates: [incident-triage, remediator] + model: + family: typesafe + name: jev-1-14 + fallback: incident-triage + joinRuns: true + promptTemplate: | + {{ comment.user.login }} commented. Do what the comment asks. + + Repository: {{ repository.full_name }} + Issue or pull request: #{{ issue.number }} + Title: {{ issue.title }} + Link: {{ issue.html_url }} + Comment: {{ comment.body }} diff --git a/factories/sre/factory/triggers/remediate.yaml b/factories/sre/factory/triggers/remediate.yaml new file mode 100644 index 0000000..cdc4d2f --- /dev/null +++ b/factories/sre/factory/triggers/remediate.yaml @@ -0,0 +1,23 @@ +# The triage agent and the cost sweeper add the remediate label when a change +# to the deploy repository does what their issue recommends. A person can add +# it too. +apiVersion: runkora.io/v1alpha1 +kind: Trigger +metadata: + name: remediate +spec: + source: github + eventType: issues + match: + action: [labeled] + label.name: [remediate] + repository.full_name: [example-org/example-deploy] + agent: remediator + promptTemplate: | + Make the change this issue recommends. + + Repository: {{ repository.full_name }} + Issue: #{{ issue.number }} + Title: {{ issue.title }} + Link: {{ issue.html_url }} + Body: {{ issue.body }} diff --git a/factories/sre/factory/triggers/remediator-on-review.yaml b/factories/sre/factory/triggers/remediator-on-review.yaml new file mode 100644 index 0000000..32d8ea9 --- /dev/null +++ b/factories/sre/factory/triggers/remediator-on-review.yaml @@ -0,0 +1,28 @@ +apiVersion: runkora.io/v1alpha1 +kind: Trigger +metadata: + name: remediator-on-review +spec: + source: github + eventType: pull_request_review + match: + action: [submitted] + pull_request.user.login: ["example-sre[bot]"] + repository.full_name: [example-org/example-deploy] + senderRestriction: + roles: [OWNER, MEMBER, COLLABORATOR] + routing: + candidates: [remediator] + model: + family: typesafe + name: jev-1-14 + fallback: remediator + joinRuns: true + promptTemplate: | + {{ review.user.login }} reviewed your pull request. Address the review. + + Repository: {{ repository.full_name }} + Pull request: #{{ pull_request.number }} + Branch: {{ pull_request.head.ref }} + Link: {{ pull_request.html_url }} + Review: {{ review.body }} diff --git a/factories/sre/factory/triggers/resolve-alert.yaml b/factories/sre/factory/triggers/resolve-alert.yaml new file mode 100644 index 0000000..2901858 --- /dev/null +++ b/factories/sre/factory/triggers/resolve-alert.yaml @@ -0,0 +1,21 @@ +apiVersion: runkora.io/v1alpha1 +kind: Trigger +metadata: + name: resolve-alert +spec: + source: alertmanager + eventType: alert.resolved + routing: + candidates: [incident-triage] + model: + family: typesafe + name: jev-1-14 + fallback: incident-triage + joinRuns: true + promptTemplate: | + The alert resolved. Summarise the incident and close it. + + Incident repository: example-org/example-deploy + Alert group: {{ groupKey }} + Alert: {{ commonLabels.alertname }} + Namespace: {{ commonLabels.namespace }} diff --git a/factories/sre/factory/triggers/sweep-weekly.yaml b/factories/sre/factory/triggers/sweep-weekly.yaml new file mode 100644 index 0000000..8d7477b --- /dev/null +++ b/factories/sre/factory/triggers/sweep-weekly.yaml @@ -0,0 +1,11 @@ +apiVersion: runkora.io/v1alpha1 +kind: Trigger +metadata: + name: sweep-weekly +spec: + kind: scheduled + agent: cost-sweeper + schedule: + cronExpression: "0 9 * * 1" + timezone: UTC + promptTemplate: "Sweep the cluster's costs and report in example-org/example-deploy. Today is {{ var.date }}." diff --git a/factories/sre/factory/triggers/triage-alert.yaml b/factories/sre/factory/triggers/triage-alert.yaml new file mode 100644 index 0000000..1ee0d78 --- /dev/null +++ b/factories/sre/factory/triggers/triage-alert.yaml @@ -0,0 +1,30 @@ +# Alerts in the same group within a minute start one run. A later notification +# for the group joins the open run when there is one. +apiVersion: runkora.io/v1alpha1 +kind: Trigger +metadata: + name: triage-alert +spec: + source: alertmanager + eventType: alert.firing + coalesce: + key: "{{ groupKey }}" + windowSeconds: 60 + routing: + candidates: [incident-triage] + model: + family: typesafe + name: jev-1-14 + fallback: incident-triage + joinRuns: true + promptTemplate: | + An alert is firing. Triage it. + + Incident repository: example-org/example-deploy + Alert group: {{ groupKey }} + Alert: {{ commonLabels.alertname }} + Namespace: {{ commonLabels.namespace }} + Severity: {{ commonLabels.severity }} + Summary: {{ commonAnnotations.summary }} + Description: {{ commonAnnotations.description }} + Source: {{ alerts.0.generatorURL }} diff --git a/factories/sre/platform/cluster-access.yaml b/factories/sre/platform/cluster-access.yaml new file mode 100644 index 0000000..fa1ed9c --- /dev/null +++ b/factories/sre/platform/cluster-access.yaml @@ -0,0 +1,54 @@ +# The read-only ceiling for the factory's cluster connection. kora's grants on +# each agent can only narrow it. The resources mirror the kubernetes adapter's +# workloads, config, networking and cluster categories. Secrets are left out, +# and the adapter denies any resource it does not classify. +# +# The ServiceAccount lives in the tenant namespace, in +# ../factory/shared/cluster-connection.yaml. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kora-oncall-cluster-reader +rules: + - apiGroups: [""] + resources: + - pods + - pods/log + - replicationcontrollers + - configmaps + - services + - endpoints + - namespaces + - nodes + - events + - resourcequotas + - limitranges + verbs: ["get", "list", "watch"] + - apiGroups: ["apps"] + resources: ["deployments", "replicasets", "statefulsets", "daemonsets"] + verbs: ["get", "list", "watch"] + - apiGroups: ["batch"] + resources: ["jobs", "cronjobs"] + verbs: ["get", "list", "watch"] + - apiGroups: ["discovery.k8s.io"] + resources: ["endpointslices"] + verbs: ["get", "list", "watch"] + - apiGroups: ["networking.k8s.io"] + resources: ["ingresses", "ingressclasses", "networkpolicies"] + verbs: ["get", "list", "watch"] + - apiGroups: ["events.k8s.io"] + resources: ["events"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: kora-oncall-cluster-reader +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: kora-oncall-cluster-reader +subjects: + - kind: ServiceAccount + name: cluster-reader + namespace: kora-tenant-oncall diff --git a/factories/sre/platform/kustomization.yaml b/factories/sre/platform/kustomization.yaml new file mode 100644 index 0000000..f542807 --- /dev/null +++ b/factories/sre/platform/kustomization.yaml @@ -0,0 +1,20 @@ +# The platform team's repository for the SRE factory: the tenant, the models +# its agents run on and the read-only access its cluster connection has. Point +# your GitOps tool at this folder, or apply it with kubectl apply -k. +# +# These resources span kora-system and the cluster scope. This file sets no +# namespace. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +commonAnnotations: + # The console shows this value as "Managed by" on each declared item. + runkora.io/repository: github.com/runkora/examples +resources: + - tenants/oncall.yaml + - cluster-access.yaml + - models/claude-sonnet-5-5.yaml + - models/claude-haiku-5-5.yaml + - models/deepseek-v4-1-flash.yaml + - models/gpt-6-luna.yaml + - models/gpt-oss-20b.yaml + - models/jev-1-14.yaml diff --git a/factories/sre/platform/models/claude-haiku-5-5.yaml b/factories/sre/platform/models/claude-haiku-5-5.yaml new file mode 100644 index 0000000..a0a870c --- /dev/null +++ b/factories/sre/platform/models/claude-haiku-5-5.yaml @@ -0,0 +1,18 @@ +# anthropic/claude-haiku-5-5 is the model every agent in the factory runs on. It +# routes to the OpenRouter provider in quickstart/providers. On your own +# cluster, point the route at a provider you registered. +apiVersion: runkora.io/v1alpha1 +kind: VirtualModel +metadata: + name: anthropic-claude-haiku-5-5 + namespace: kora-system +spec: + family: anthropic + name: claude-haiku-5-5 + displayName: Claude Haiku 5.5 + description: Claude Haiku 5.5 through OpenRouter. + strategy: priority + routes: + - provider: compatible:openrouter + modelID: anthropic/claude-haiku-5.5 + priority: 1 diff --git a/factories/sre/platform/models/claude-sonnet-5-5.yaml b/factories/sre/platform/models/claude-sonnet-5-5.yaml new file mode 100644 index 0000000..2a7eb3a --- /dev/null +++ b/factories/sre/platform/models/claude-sonnet-5-5.yaml @@ -0,0 +1,18 @@ +# anthropic/claude-sonnet-5-5 is a larger model for harder investigations. It +# routes to the OpenRouter provider in quickstart/providers. On your own +# cluster, point the route at a provider you registered. +apiVersion: runkora.io/v1alpha1 +kind: VirtualModel +metadata: + name: anthropic-claude-sonnet-5-5 + namespace: kora-system +spec: + family: anthropic + name: claude-sonnet-5-5 + displayName: Claude Sonnet 5.5 + description: Claude Sonnet 5.5 through OpenRouter. + strategy: priority + routes: + - provider: compatible:openrouter + modelID: anthropic/claude-sonnet-5.5 + priority: 1 diff --git a/factories/sre/platform/models/deepseek-v4-1-flash.yaml b/factories/sre/platform/models/deepseek-v4-1-flash.yaml new file mode 100644 index 0000000..13ec540 --- /dev/null +++ b/factories/sre/platform/models/deepseek-v4-1-flash.yaml @@ -0,0 +1,17 @@ +# deepseek/deepseek-v4-1-flash is a low-cost chat model. It routes to the +# OpenRouter provider in quickstart/providers. +apiVersion: runkora.io/v1alpha1 +kind: VirtualModel +metadata: + name: deepseek-deepseek-v4-1-flash + namespace: kora-system +spec: + family: deepseek + name: deepseek-v4-1-flash + displayName: DeepSeek V4.1 Flash + description: DeepSeek V4.1 Flash through OpenRouter. + strategy: priority + routes: + - provider: compatible:openrouter + modelID: deepseek/deepseek-v4.1-flash + priority: 1 diff --git a/factories/sre/platform/models/gpt-6-luna.yaml b/factories/sre/platform/models/gpt-6-luna.yaml new file mode 100644 index 0000000..d5801dc --- /dev/null +++ b/factories/sre/platform/models/gpt-6-luna.yaml @@ -0,0 +1,17 @@ +# openai/gpt-6-luna is a low-cost chat model. It routes to the OpenRouter +# provider in quickstart/providers. +apiVersion: runkora.io/v1alpha1 +kind: VirtualModel +metadata: + name: openai-gpt-6-luna + namespace: kora-system +spec: + family: openai + name: gpt-6-luna + displayName: GPT-6 Luna + description: GPT-6 Luna through OpenRouter. + strategy: priority + routes: + - provider: compatible:openrouter + modelID: openai/gpt-6-luna + priority: 1 diff --git a/factories/sre/platform/models/gpt-oss-20b.yaml b/factories/sre/platform/models/gpt-oss-20b.yaml new file mode 100644 index 0000000..2e68970 --- /dev/null +++ b/factories/sre/platform/models/gpt-oss-20b.yaml @@ -0,0 +1,18 @@ +# openai/gpt-oss-20b runs gpt-oss 20B in Ollama on your machine. It routes to +# the quickstart's Ollama provider in quickstart/providers. To run an agent on +# it, set the agent's model to family openai and name gpt-oss-20b. +apiVersion: runkora.io/v1alpha1 +kind: VirtualModel +metadata: + name: openai-gpt-oss-20b + namespace: kora-system +spec: + family: openai + name: gpt-oss-20b + displayName: gpt-oss 20B (Ollama) + description: gpt-oss 20B running in Ollama on your machine. + strategy: priority + routes: + - provider: compatible:ollama + modelID: gpt-oss:20b + priority: 1 diff --git a/factories/sre/platform/models/jev-1-14.yaml b/factories/sre/platform/models/jev-1-14.yaml new file mode 100644 index 0000000..19f8253 --- /dev/null +++ b/factories/sre/platform/models/jev-1-14.yaml @@ -0,0 +1,17 @@ +# The decision model that routed triggers use to pick an agent or an open run. +# Routing needs a decision model. A chat model cannot route. +apiVersion: runkora.io/v1alpha1 +kind: VirtualModel +metadata: + name: typesafe-jev-1-14 + namespace: kora-system +spec: + family: typesafe + name: jev-1-14 + displayName: Jev 1.14 + description: TypeSafe Jev 1.14 decision model through OpenRouter. + strategy: priority + routes: + - provider: compatible:openrouter + modelID: typesafe/jev-1.14 + priority: 1 diff --git a/factories/sre/platform/tenants/oncall.yaml b/factories/sre/platform/tenants/oncall.yaml new file mode 100644 index 0000000..0272033 --- /dev/null +++ b/factories/sre/platform/tenants/oncall.yaml @@ -0,0 +1,25 @@ +# The oncall tenant. The operator creates its namespace, kora-tenant-oncall. +# ../factory applies the factory's agents, triggers and roles there. +# +# The oidc block points at the quickstart's Dex. On your own cluster, replace +# it with the client your identity provider gives you. See +# https://github.com/runkora/kora/blob/main/docs/guides/tenants.md. +apiVersion: runkora.io/v1alpha1 +kind: KoraTenant +metadata: + name: oncall +spec: + displayName: On-call + active: true + oidc: + issuerURL: https://dex.local.runkora.io:5557/dex + clientID: kora-quickstart + scopes: [openid, profile, email, groups] + # The console routes a person to this tenant by the domain of their email. + emailDomains: + - oncall.local.runkora.io + apiAuthoring: Enabled + # The tenant's spending limit across every agent and person. + budget: + limitUSD: 100 + resetCadence: monthly diff --git a/quickstart/.env.example b/quickstart/.env.example index e8c663e..9749c6c 100644 --- a/quickstart/.env.example +++ b/quickstart/.env.example @@ -13,6 +13,7 @@ OPENROUTER_API_KEY= EXA_API_KEY= # The absolute path to your GitHub App's private key file, for the github -# connections in ../multi-tenant/engineering and ../factories/autofix. GitHub -# gives you the .pem file when you generate a private key in the App's settings. +# connections in ../multi-tenant/engineering, ../factories/autofix and +# ../factories/sre. GitHub gives you the .pem file when you generate a private +# key in the App's settings. GITHUB_APP_PRIVATE_KEY_FILE= diff --git a/quickstart/Makefile b/quickstart/Makefile index 32f71f2..57c6466 100644 --- a/quickstart/Makefile +++ b/quickstart/Makefile @@ -21,7 +21,7 @@ TUNNEL_DIR := .tunnel # The examples make example applies. For each example: the tenants its # platform repository declares, its tenant repositories and the user to sign in # as. -EXAMPLES := single-tenant multi-tenant factories/autofix +EXAMPLES := single-tenant multi-tenant factories/autofix factories/sre TENANTS_single-tenant := example REPOS_single-tenant := tenant SIGN_IN_single-tenant := admin@local.runkora.io @@ -31,10 +31,13 @@ SIGN_IN_multi-tenant := admin@engineering.local.runkora.io or admin@sre.local.ru TENANTS_factories/autofix := autofix REPOS_factories/autofix := factory SIGN_IN_factories/autofix := admin@autofix.local.runkora.io +TENANTS_factories/sre := oncall +REPOS_factories/sre := factory +SIGN_IN_factories/sre := admin@oncall.local.runkora.io # The webhook signing secrets the examples' trigger sources read, one per # tenant and provider. make credentials generates each one. -WEBHOOK_SECRETS := example-github-webhook engineering-github-webhook engineering-slack-webhook sre-alertmanager-webhook autofix-github-webhook +WEBHOOK_SECRETS := example-github-webhook engineering-github-webhook engineering-slack-webhook sre-alertmanager-webhook autofix-github-webhook oncall-github-webhook oncall-alertmanager-webhook # The credentials you fill in from .env.example. The install runs without the # file, and make credentials skips each value that is not set. @@ -50,7 +53,7 @@ define apply $(KUBECTL) -n $(1) create $(2) --dry-run=client -o yaml | $(KUBECTL) apply -f - endef -.PHONY: up tools cluster namespaces secrets istio spire data dex eso kora credentials providers example check-example apply-example ask-tunnel tunnel tunnel-stop status down clean +.PHONY: up tools cluster namespaces secrets istio spire data dex eso kora credentials providers example check-example apply-example ask-tunnel tunnel tunnel-stop alert status down clean up: tools cluster namespaces secrets istio spire data dex eso kora credentials providers example ask-tunnel @@ -184,9 +187,10 @@ credentials: $(addprefix $(SECRETS)/,$(WEBHOOK_SECRETS)) $(call apply,kora-credentials,secret generic sre-exa --from-literal=apiKey="$$EXA_API_KEY"); \ fi @if [ -n "$$GITHUB_APP_PRIVATE_KEY_FILE" ]; then \ - echo "GITHUB_APP_PRIVATE_KEY_FILE -> Secrets engineering-github-app and autofix-github-app in kora-credentials"; \ + echo "GITHUB_APP_PRIVATE_KEY_FILE -> Secrets engineering-github-app, autofix-github-app and oncall-github-app in kora-credentials"; \ $(call apply,kora-credentials,secret generic engineering-github-app --from-file=key.pem="$$GITHUB_APP_PRIVATE_KEY_FILE"); \ $(call apply,kora-credentials,secret generic autofix-github-app --from-file=key.pem="$$GITHUB_APP_PRIVATE_KEY_FILE"); \ + $(call apply,kora-credentials,secret generic oncall-github-app --from-file=key.pem="$$GITHUB_APP_PRIVATE_KEY_FILE"); \ fi @for name in $(WEBHOOK_SECRETS); do \ echo "$(SECRETS)/$$name -> Secret $$name in kora-credentials"; \ @@ -213,9 +217,10 @@ example: echo " 1) single-tenant One tenant with one agent. Needs a model provider only."; \ echo " 2) multi-tenant Two teams with four use cases. Also needs an Exa key and a GitHub App."; \ echo " 3) autofix A factory that finds, verifies and fixes bugs. Needs a GitHub App and a repository."; \ - echo " 4) none Apply one later with make example."; \ - printf "Choose 1, 2, 3 or 4 [1]: "; read choice; \ - case "$$choice" in 2) example=multi-tenant;; 3) example=factories/autofix;; 4) example=none;; *) example=single-tenant;; esac; \ + echo " 4) sre A factory that triages alerts and proposes fixes. Needs a GitHub App and a repository."; \ + echo " 5) none Apply one later with make example."; \ + printf "Choose 1, 2, 3, 4 or 5 [1]: "; read choice; \ + case "$$choice" in 2) example=multi-tenant;; 3) example=factories/autofix;; 4) example=factories/sre;; 5) example=none;; *) example=single-tenant;; esac; \ fi; \ [ "$${example:-single-tenant}" = none ] || $(MAKE) --no-print-directory apply-example EXAMPLE=$${example:-single-tenant} @@ -253,6 +258,21 @@ ifeq ($(EXAMPLE),factories/autofix) echo "See factories/autofix/README.md. Then run make credentials example EXAMPLE=factories/autofix."; exit 1; \ fi endif +ifeq ($(EXAMPLE),factories/sre) + @missing=0; \ + if [ ! -f "$$GITHUB_APP_PRIVATE_KEY_FILE" ]; then \ + echo "sre: set GITHUB_APP_PRIVATE_KEY_FILE in quickstart/.env to your GitHub App's private key file."; missing=1; \ + fi; \ + if grep -q 'owner: example-org' ../factories/sre/factory/shared/github-connection.yaml; then \ + echo "sre: set app_id, installation_id and owner in factories/sre/factory/shared/github-connection.yaml."; missing=1; \ + fi; \ + if grep -rqF -e 'example-org/example-deploy' -e 'example-sre[bot]' ../factories/sre/factory/triggers; then \ + echo "sre: replace example-org/example-deploy and example-sre[bot] in factories/sre/factory/triggers with your deploy repository and your App's bot user."; missing=1; \ + fi; \ + if [ $$missing = 1 ]; then \ + echo "See factories/sre/README.md. Then run make credentials example EXAMPLE=factories/sre."; exit 1; \ + fi +endif # Applies an example: its platform repository first, then each tenant # repository once the operator has created the tenant's namespace. @@ -266,9 +286,20 @@ apply-example: check-example sleep 2; \ done; \ done +ifeq ($(EXAMPLE),factories/sre) + $(HELM_INSTALL) opencost opencost --repo https://opencost.github.io/opencost-helm-chart \ + --version $(OPENCOST_VERSION) -n opencost --create-namespace -f values/opencost.yaml +endif @for repo in $(REPOS_$(EXAMPLE)); do \ $(KUBECTL) apply -k ../$(EXAMPLE)/$$repo || exit 1; \ done +ifeq ($(EXAMPLE),factories/sre) + @echo "Setting the cluster connection's caCert to this cluster's CA" + @{ printf 'spec:\n caCert: |\n'; \ + $(KUBECTL) -n kora-tenant-oncall get configmap kube-root-ca.crt -o jsonpath='{.data.ca\.crt}' | sed 's/^/ /'; \ + } | $(KUBECTL) -n kora-tenant-oncall patch connection cluster --type merge --patch-file /dev/stdin + $(KUBECTL) apply -k ../factories/sre/demo +endif @echo @echo "$(EXAMPLE) is running." @echo " Console: https://local.runkora.io:9443" @@ -328,6 +359,19 @@ tunnel: tunnel-stop tunnel-stop: @if [ -f $(TUNNEL_DIR)/pid ]; then kill $$(cat $(TUNNEL_DIR)/pid) 2>/dev/null || true; rm -f $(TUNNEL_DIR)/pid $(TUNNEL_DIR)/url; fi +# Posts the alert Alertmanager would send for the sre example's crash-looping +# checkout pod. STATUS=resolved posts the resolution. The alert reaches kora on +# localhost and needs no tunnel. +STATUS ?= firing +alert: + @path=$$($(KUBECTL) -n kora-tenant-oncall get triggersource alertmanager -o jsonpath='{.status.webhookPath}'); \ + [ -n "$$path" ] || { echo "The sre example's alertmanager source is not ready. Run make example EXAMPLE=factories/sre."; exit 1; }; \ + now=$$(date -u +%Y-%m-%dT%H:%M:%SZ); \ + printf '%s' '{"version":"4","status":"$(STATUS)","receiver":"kora","groupKey":"{}:{alertname=\"KubePodCrashLooping\", namespace=\"shop\"}","truncatedAlerts":0,"groupLabels":{"alertname":"KubePodCrashLooping","namespace":"shop"},"commonLabels":{"alertname":"KubePodCrashLooping","namespace":"shop","deployment":"checkout","severity":"critical"},"commonAnnotations":{"summary":"checkout is crash-looping in shop","description":"A checkout pod in shop restarted more than three times in the last ten minutes."},"externalURL":"http://alertmanager.local","alerts":[{"status":"$(STATUS)","labels":{"alertname":"KubePodCrashLooping","namespace":"shop","deployment":"checkout","severity":"critical"},"annotations":{"summary":"checkout is crash-looping in shop"},"startsAt":"'"$$now"'","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://prometheus.local/graph","fingerprint":"5f1c0a6e2b7d9c31"}]}' | \ + curl -sS --fail-with-body --resolve api.local.runkora.io:9443:127.0.0.1 --cacert "$$(mkcert -CAROOT)/rootCA.pem" \ + -H "Authorization: Bearer $$(cat $(SECRETS)/oncall-alertmanager-webhook)" -H 'Content-Type: application/json' \ + --data-binary @- "https://api.local.runkora.io:9443$$path" && echo && echo "Posted a $(STATUS) KubePodCrashLooping alert for shop/checkout." + # Deletes the cluster. The next make up reuses the files in .secrets. down: tunnel-stop k3d cluster delete $(CLUSTER) diff --git a/quickstart/README.md b/quickstart/README.md index a08ecc8..20a8143 100644 --- a/quickstart/README.md +++ b/quickstart/README.md @@ -27,6 +27,7 @@ The quickstart installs runkora on a local k3d cluster. It installs Istio, SPIRE * **[`single-tenant`](../single-tenant):** One tenant with one agent that runs on the model provider alone. * **[`multi-tenant`](../multi-tenant):** Two tenants, engineering and SRE, with four use cases. It also needs an Exa API key and a GitHub App. * **[`autofix`](../factories/autofix):** A factory that finds, verifies and fixes bugs in a GitHub repository. It needs a GitHub App and a repository. + * **[`sre`](../factories/sre):** A factory that triages alerts, sweeps the cluster's costs with OpenCost and proposes fixes. It needs a GitHub App and a repository. * **`none`:** Installs kora with no tenants. You can apply an example later with `make example`. Set `EXAMPLE` to skip the question, such as `make up EXAMPLE=multi-tenant`. @@ -42,7 +43,7 @@ Each example declares four models in its `platform/models/`: * `deepseek/deepseek-v4-1-flash` and `openai/gpt-6-luna` route to OpenRouter. * `openai/gpt-oss-20b` routes to Ollama. -`multi-tenant` and `autofix` also declare `typesafe/jev-1-14`, a decision model on OpenRouter that their routed triggers use to pick an agent. +`multi-tenant`, `autofix` and `sre` also declare `typesafe/jev-1-14`, a decision model on OpenRouter that their routed triggers use to pick an agent. ### OpenRouter @@ -105,6 +106,9 @@ Every user signs in with the password `password`. kora routes each person to a t * **`autofix`:** * `admin@autofix.local.runkora.io`: Group `platform-admin`. * `maintainer@autofix.local.runkora.io`: Group `maintainer`. +* **`sre`:** + * `admin@oncall.local.runkora.io`: Tenant `oncall`, group `platform-admin`. + * `sre@oncall.local.runkora.io`: Tenant `oncall`, group `sre`. The users and their groups live in `manifests/dex/glauth.yaml`. kora reads a person's roles from the `groups` claim in their ID token. `platform-admin` holds every scope in kora, and `kora-viewer` reads everything in the tenant. Both are built-in roles that kora maps from the group name. Each example grants the other groups their scopes with `PolicyRole` resources. diff --git a/quickstart/manifests/dex/glauth.yaml b/quickstart/manifests/dex/glauth.yaml index 815db57..52e863e 100644 --- a/quickstart/manifests/dex/glauth.yaml +++ b/quickstart/manifests/dex/glauth.yaml @@ -15,6 +15,8 @@ # oncall@sre.local.runkora.io multi-tenant sre, kora-viewer # admin@autofix.local.runkora.io autofix platform-admin # maintainer@autofix.local.runkora.io autofix maintainer +# admin@oncall.local.runkora.io sre platform-admin +# sre@oncall.local.runkora.io sre sre apiVersion: v1 kind: ConfigMap metadata: @@ -122,6 +124,26 @@ data: othergroups = [5006] passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8" + [[users]] + name = "oncall-admin" + givenname = "Admin" + sn = "User" + mail = "admin@oncall.local.runkora.io" + uidnumber = 1009 + primarygroup = 1100 + othergroups = [5001] + passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8" + + [[users]] + name = "oncall-sre" + givenname = "SRE" + sn = "User" + mail = "sre@oncall.local.runkora.io" + uidnumber = 1010 + primarygroup = 1100 + othergroups = [5005] + passsha256 = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8" + [[groups]] name = "svcaccts" gidnumber = 5000 diff --git a/quickstart/values/kora.yaml b/quickstart/values/kora.yaml index 458712d..9c52340 100644 --- a/quickstart/values/kora.yaml +++ b/quickstart/values/kora.yaml @@ -36,6 +36,12 @@ networkPolicy: - cidr: 0.0.0.0/0 ports: [11434] +egressGateway: + # The sre example's cluster connection reaches the k3d API server, which + # listens on 6443. + upstreamPorts: + - 6443 + gateway: tls: enabled: true diff --git a/quickstart/values/opencost.yaml b/quickstart/values/opencost.yaml new file mode 100644 index 0000000..721696f --- /dev/null +++ b/quickstart/values/opencost.yaml @@ -0,0 +1,14 @@ +# OpenCost for the sre example's cost sweeper. The quickstart runs no +# Prometheus. The collector data source reads the cluster directly, and the MCP +# server on port 8081 serves the cost tools. +opencost: + exporter: + collectorDataSource: + enabled: true + prometheus: + internal: + enabled: false + mcp: + enabled: true + ui: + enabled: false diff --git a/versions.mk b/versions.mk index bbb7a8e..6a50d1f 100644 --- a/versions.mk +++ b/versions.mk @@ -2,7 +2,7 @@ # Override any of them on the command line, e.g. make up KORA_VERSION=0.11.0. # The kora release the examples install. Both charts share the version. -KORA_VERSION ?= 0.11.0-rc.5 +KORA_VERSION ?= 0.11.0-rc.6 KORA_CHARTS ?= oci://ghcr.io/runkora/charts # kora supports Istio 1.30 and 1.31. The operator's preflight rejects any other version. @@ -16,5 +16,9 @@ SPIRE_VERSION ?= 0.30.2 # the quickstart's credential store or your secret manager. ESO_VERSION ?= 2.14.0 +# The opencost chart 2.5.33 ships OpenCost 1.121.3. The sre example's cost +# sweeper reads it through OpenCost's MCP server. +OPENCOST_VERSION ?= 2.5.33 + # The kora charts require Kubernetes 1.34 or later. K3S_IMAGE ?= rancher/k3s:v1.36.4-k3s1