Repository navigation
feat: add the sre factory - #4
Merged
Merged
Conversation
Three agents triage Alertmanager alerts in the cluster through a read-only kubernetes connection, propose fixes to the deploy repository and sweep the cluster's costs weekly through OpenCost's MCP server.
make example installs OpenCost, fills in the cluster connection's CA and deploys the broken checkout service. make alert posts a sample Alertmanager alert. The egress gateway may reach the k3d API server on 6443.
rc.6 carries runkora/kora#812, which the sre factory's cluster connection needs to reach an API server on 6443.
justinwilkin
marked this pull request as ready for review
October 27, 2026 05:22
make validate reads quickstart/manifests as Kubernetes manifests, and a helm values file has no kind.
Visual WalkthroughThis pull request adds the SRE factory: triggers, agents and connections for alert triage, remediation and cost sweeps. Factory wiringTriggers route events to agents, and agents reach GitHub and the cluster through connections. flowchart TB
subgraph SRC["Event sources"]
AM[Alertmanager]:::context
GHW[GitHub webhook]:::context
end
subgraph FAC["Factory triggers and agents"]
TRA[Alert triggers]:::added
TRG[GitHub triggers]:::added
SCH[Weekly schedule]:::added
TRI[Incident triage]:::added
REM[Remediator]:::added
CST[Cost sweeper]:::added
end
subgraph CONN["Connections"]
CLC[Cluster connection]:::added
GHC[GitHub connection]:::added
end
K8S[Cluster API]:::context
GHAPI[GitHub API]:::context
AM --> TRA
GHW --> TRG
SCH --> CST
TRA --> TRI
TRG --> TRI
TRG --> REM
TRI --> CLC
TRI --> GHC
REM --> GHC
CST --> GHC
CLC --> K8S
GHC --> GHAPI
classDef added fill:#1f883d,stroke:#1a7f37,stroke-width:2px,color:#ffffff
classDef changed fill:#bf8700,stroke:#9a6700,stroke-width:2px,color:#ffffff
classDef removed fill:#cf222e,stroke:#a40e26,stroke-width:2px,stroke-dasharray:5,color:#ffffff
classDef context fill:transparent,stroke:#8c959f,stroke-width:1px,stroke-dasharray:3,color:#8c959f
linkStyle default stroke:#8c959f,stroke-width:2px
style SRC fill:transparent,stroke:#8c959f
style FAC fill:transparent,stroke:#8c959f
style CONN fill:transparent,stroke:#8c959f
Legend: green = added, amber = changed, red = removed, dashed grey = context
Alert to fix pathAn alert opens an incident issue, and a remediate label hands the fix to the remediator. The pull request refs the incident, which stays open until the alert resolves. sequenceDiagram
participant AM as Alertmanager
participant TT as Alert triggers
participant TI as Incident triage
participant K as Cluster API
participant GH as GitHub
participant RM as Remediator
AM->>TT: alert firing
TT->>TI: start or join run
TI->>GH: search open incident issues
alt issue already open
TI->>GH: comment new evidence
else no open issue
TI->>K: read pods, logs, events
K-->>TI: workload state
TI->>GH: open incident issue
opt fix in deploy repository
TI->>GH: add remediate label
GH->>RM: start remediate run
RM->>GH: open pull request refs incident
end
end
Cost sweep pathThe weekly sweep reads OpenCost through a relay in the tenant namespace and files a cost issue. flowchart TB
SCH[Weekly schedule]:::added
CST[Cost sweeper]:::added
subgraph NS["Tenant namespace"]
MCPS[OpenCost MCP server]:::added
RLY[OpenCost relay]:::added
end
OC[OpenCost]:::context
GHC[GitHub connection]:::added
SCH --> CST
CST --> MCPS
MCPS -->|served by| RLY
RLY --> OC
CST --> GHC
classDef added fill:#1f883d,stroke:#1a7f37,stroke-width:2px,color:#ffffff
classDef changed fill:#bf8700,stroke:#9a6700,stroke-width:2px,color:#ffffff
classDef removed fill:#cf222e,stroke:#a40e26,stroke-width:2px,stroke-dasharray:5,color:#ffffff
classDef context fill:transparent,stroke:#8c959f,stroke-width:1px,stroke-dasharray:3,color:#8c959f
linkStyle default stroke:#8c959f,stroke-width:2px
style NS fill:transparent,stroke:#8c959f
Legend: green = added, amber = changed, red = removed, dashed grey = context
|
Visual WalkthroughThis pull request adds the SRE factory, which triages alerts, files incident and cost issues, and proposes fixes to the deploy repository. Incident response loopThe factory triages a firing alert, records an incident issue, and hands a repository fix to the remediator. The remediator's pull request refs the incident, which stays open until the alert resolves. sequenceDiagram
participant AM as Alertmanager
participant T as triage-alert trigger
participant I as incident-triage agent
participant C as cluster connection
participant G as GitHub
participant R as remediator agent
AM->>T: alert firing
T->>I: start triage run
I->>C: read pods, logs and events
C-->>I: cluster state
alt agent reads a Secret
I->>C: read Secret
C-->>I: refused by cluster role
end
I->>G: open incident issue
alt fix is in the repository
I->>G: add remediate label
G->>R: label starts fix run
R->>G: open pull request refs incident
G->>R: review starts fix run
else no repository fix
I->>G: issue states the cause
end
AM->>I: alert resolved
I->>G: comment summary and close issue
Weekly cost sweepA weekly cron run reads OpenCost data and files a cost issue that the remediator can act on. Its pull request fixes the cost issue when merged. flowchart TB
subgraph FAC["SRE factory"]
SW["sweep-weekly trigger"]:::added
CS["cost-sweeper agent"]:::added
MCP["opencost MCP server"]:::added
RELAY["opencost-mcp relay"]:::added
RM["remediator agent"]:::changed
end
OC["OpenCost"]:::context
ISS["cost issue in GitHub"]:::context
PR["pull request in deploy repo"]:::context
SW --> CS
CS --> MCP
MCP --> RELAY
RELAY --> OC
CS -->|opens cost issue| ISS
ISS -->|remediate label| RM
RM -->|opens pull request| PR
style FAC fill:transparent,stroke:#8c959f
classDef added fill:#1f883d,stroke:#1a7f37,stroke-width:2px,color:#ffffff
classDef changed fill:#bf8700,stroke:#9a6700,stroke-width:2px,color:#ffffff
classDef removed fill:#cf222e,stroke:#a40e26,stroke-width:2px,stroke-dasharray:5,color:#ffffff
classDef context fill:transparent,stroke:#8c959f,stroke-width:1px,stroke-dasharray:3,color:#8c959f
linkStyle default stroke:#8c959f,stroke-width:2px
Legend: green = added, amber = changed, red = removed, dashed grey = context
|
![runkora[bot]](https://avatars.githubusercontent.com/in/5234669?s=60&v=4){kind=small}
There was a problem hiding this comment.
Walkthrough
The pull request adds an SRE factory with triage, remediation and cost-sweep agents, an oncall tenant and cluster access. It also wires the factory into the quickstart with OpenCost and new Dex users.
| Severity | Finding | Location |
|---|---|---|
 |
Restrict who can fire the remediator | factories/sre/factory/triggers/remediate.yaml:14 |
|
Narrow or document the cluster-wide ConfigMap and log read | factories/sre/platform/cluster-access.yaml:18 |
 |
Use the oncall tenant in the user table |
quickstart/manifests/dex/glauth.yaml:19 |
|
Check the other GitHub App placeholders too | quickstart/Makefile:266 |
Review effort: High
Risk: Medium, new agents with cluster read and repository write
Coverage: 38 of 38 files reviewed
Where to focus
| Area | Why it needs a person |
|---|---|
factories/sre/factory/triggers/ |
Decide who may trigger the remediator and how alert text reaches it. |
factories/sre/platform/cluster-access.yaml |
Confirm cluster-wide ConfigMap and log reads are acceptable. |
factories/sre/factory/agents/ |
Triage can label issues that start the remediator without a person. |
Lower risk
| Area | Why |
|---|---|
factories/sre/platform/models/ |
Model declarations; several are unused by the agents. |
factories/sre/demo/ |
Demo workload with an intentional fault. |
versions.mk |
Single version bump. |
Changes by area
| Area | Files | Change |
|---|---|---|
factories/sre/factory |
17 | Agents, triggers and shared resources |
factories/sre/platform |
9 | Cluster access, models, oncall tenant |
factories/sre/demo |
2 | Demo shop workload |
quickstart |
7 | Make targets, OpenCost values, Dex users |
| Root and docs | 3 | Makefile, README, versions |
Checked
Manifest cross-references, quickstart secret names, Dex user IDs.
The remediator referenced the incident with Fixes, and merging the fix closed it before resolve-alert could summarise it. Incidents now take Refs. Cost issues keep Fixes.
kora-agent
approved these changes
Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
factories/sre, a factory that triages Alertmanager alerts in the cluster, proposes fixes to the deploy repository and sweeps the cluster's costs with OpenCost.Needs kora 0.11.0-rc.6, which carries runkora/kora#812.
Changes
factories/sre: The platform repository (tenantoncall, models, a read-only ClusterRole for the cluster connection) and the factory repository (cluster and GitHub connections, Alertmanager and GitHub sources, OpenCost MCP server,opsenvironment,srerole, three agents, six triggers).incident-triagegroups an alert storm into one run, inspects the cluster withkubectl, opens anincidentissue and addsremediatewhen a manifest change fixes the cause. It closes the issue with a summary when the alert resolves.remediatoropens the fix as a pull request from theremediatelabel. Reviews and comments route back to it with a decision model, joining its open run.cost-sweeperruns weekly against OpenCost's MCP server and files acostissue with rightsizing recommendations.demo/is a deploy repository with a crash-loopingcheckoutservice.make exampleoffers sre and checks its placeholders. It installs OpenCost without Prometheus, fills in the cluster connection's CA from the cluster and deploysdemo/.make alertposts a sample Alertmanager alert, andSTATUS=resolvedresolves it.Testing
cost-sweeperfiled acostissue from its data.make alertrouted toincident-triage, which opened the incident issue.remediatoropened the fix from theremediatelabel in about a minute. A review routed back to its open run, and it pushed the requested change.incident-triageread the cluster withkubectlthrough the gateway on 6443, foundMODE: lvieinshop/checkout.yamlitself and addedremediate.remediatorthen opened the fix.