From 1974d22fa8de993f2cf74d4c39fab7af6299b4f1 Mon Sep 17 00:00:00 2001 From: Derek Date: Sun, 11 Oct 2026 04:56:39 +1100 Subject: [PATCH] fix: hunts-pause switches the engine's hunt-runner secret group off too hunts-pause set huntRunner.enabled false and nothing else. dfe-extras then stops minting hunt-runner-clickhouse, but the engine Deployment still read it through a required secretKeyRef (DFE_CLICKHOUSE_HUNT_RUNNER_PASSWORD), so the engine pod sat in CreateContainerConfigError until a resume. hunts-pause now also sets secrets.hunt-runner.enabled false, hunts-resume sets both back so the engine adopts the runner's password again, and the governance README says a switch-off action carries its secret group. --- governance/README.md | 5 +++++ governance/actions/hunts-pause.yaml | 11 +++++++++-- governance/actions/hunts-resume.yaml | 9 +++++++-- 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/governance/README.md b/governance/README.md index 5b4c956..70c0254 100644 --- a/governance/README.md +++ b/governance/README.md @@ -35,6 +35,11 @@ instance file does not exist yet, an invoke creates it - and any `*-values.yaml` file IS an Argo application, so point actions only at instances this deployment actually runs. +An action that switches a component off also switches off the engine's secret +group for it (`secrets..enabled`, as `hunts-pause` does), and its +counterpart switches both back on. The component's Secret is only minted while +it is on, and the engine pod waits on any Secret it still reads. + Authoring a new action: validate first with `POST /api/v1/governance/admin/actions/validate` (returns every violation and the would-be diff without committing), then define via diff --git a/governance/actions/hunts-pause.yaml b/governance/actions/hunts-pause.yaml index 61fe0c2..a1c29cc 100644 --- a/governance/actions/hunts-pause.yaml +++ b/governance/actions/hunts-pause.yaml @@ -1,8 +1,11 @@ name: hunts-pause description: >- Stop the hunt runner (huntRunner.enabled=false on the engine instance) to - shed ClickHouse load during an incident. Checkpoint watermarks are advanced - only on committed runs, so paused hunts resume with no gap. Revert with + shed ClickHouse load during an incident. The engine's hunt-runner secret + group goes off with it (secrets.hunt-runner.enabled=false), because the + runner's password Secret is only minted while the runner is on and the + engine pod would wait on it. Checkpoint watermarks are advanced only on + committed runs, so paused hunts resume with no gap. Revert with hunts-resume. required_action: action:invoke:hunts-pause changes: @@ -10,3 +13,7 @@ changes: name: dfe-engine-default-values path: huntRunner.enabled value: false + - cls: helmvars + name: dfe-engine-default-values + path: secrets.hunt-runner.enabled + value: false diff --git a/governance/actions/hunts-resume.yaml b/governance/actions/hunts-resume.yaml index 378f7e6..5738f98 100644 --- a/governance/actions/hunts-resume.yaml +++ b/governance/actions/hunts-resume.yaml @@ -1,10 +1,15 @@ name: hunts-resume description: >- - Restart the hunt runner after a hunts-pause. Hunts resume from their last - committed checkpoint watermark - no window is lost. + Restart the hunt runner after a hunts-pause, and put the engine's hunt-runner + secret group back so the engine adopts the runner's password again. Hunts + resume from their last committed checkpoint watermark - no window is lost. required_action: action:invoke:hunts-resume changes: - cls: helmvars name: dfe-engine-default-values path: huntRunner.enabled value: true + - cls: helmvars + name: dfe-engine-default-values + path: secrets.hunt-runner.enabled + value: true