Skip to content

proposal: first-class alerting via PrometheusRule and Grafana #79

Description

@atkrad

Scraping is first-class (metrics: true makes a ServiceMonitor / PodMonitor). Alerting is not. Today you drop a PrometheusRule in .deployah/manifests/.

We want component-owned alerts, with Deployah labels, plan diffs, and a capability check. Two backends exist: Prometheus Operator (PrometheusRule) and Grafana Operator (GrafanaAlertRuleGroup). They are not the same CR, so we will not invent one DSL that pretends to emit both.

Platform picks the backends. The component owns the PromQL.

# deployah.platform.yaml
profiles:
  observability:
    metrics:
      monitorLabels:
        release: prometheus
    alerting:
      providers: [prometheusRule]
      prometheusRule:
        ruleLabels:
          release: prometheus
# deployah.yaml
components:
  api:
    metrics: true
    profiles: [observability]
    alerts:
      prometheusRule:
        groups:
          - name: api
            rules:
              - alert: APIHighErrorRate
                expr: |
                  sum(rate(http_requests_total{component="api",code=~"5.."}[5m]))
                    /
                  sum(rate(http_requests_total{component="api"}[5m]))
                    > 0.05
                for: 5m

v1 emits PrometheusRule only. Grafana (alerts.grafana) is phase 2. Do not run Helm tpl on expr or annotations. Do not install the operators. Extras stay valid for AlertmanagerConfig, contact points, and one-offs.

Related: #18 (scrape), extras for everything else.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/helmChart render, release, post-renderarea/specdeployah.yaml schema or loadingkind/proposalDesign discussion before or during implementation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions