Skip to content

fix: grant the executor apps/statefulsets so the service reconcile loop can work - #78

Merged
bborbe merged 3 commits into
masterfrom
feat/executor-statefulset-rbac
Sep 25, 2026
Merged

bborbe merged 3 commits into
masterfrom
feat/executor-statefulset-rbac

Conversation

@bborbe

@bborbe bborbe commented Sep 25, 2026

Copy link
Copy Markdown
Owner

What

Grant the executor ServiceAccount create/get/update/delete on apps/statefulsets, and bump the chart to 0.6.6.

executor-rbac.yaml granted batch/jobs and ""/pods and nothing under apps — because until the type: service work the executor never created a StatefulSet. The reconcile loop therefore ran correctly against a live cluster (configs=17 services=1 failures=1) and failed on the one Config that mattered:

create statefulSet failed: statefulsets.apps is forbidden:
User "system:serviceaccount:dev:agent-task-executor"
cannot create resource "statefulsets" in API group "apps" in the namespace "dev"

No SC, DoD row or task line named RBAC, so nothing in the task could have caught it before the first live reconcile — the grant is only reachable once a Config actually declares type: service, which is the last step of the deploy.

Ordering

delete is granted from executor v0.18.1 onwards only. That is the release in which ServiceReconciler checks the object Config controller ownerRef before removing it (companion PR in bborbe/agent-task-executor). Pairing this rule with an older executor would arm an unchecked delete-by-name across every non-service Config in the namespace, since a StatefulSet name here is just the Config name.

Verification

helm lint → 1 chart linted, 0 failed. Rendered Role verified to carry the rule.

…op can work

executor-rbac.yaml granted batch/jobs and ""/pods and nothing under apps --
because until the type: service work the executor never created a StatefulSet.
So the reconcile loop ran correctly against a live cluster:

  event=service_reconcile done configs=17 services=1 undeployed=16 failures=1

and then failed on the one Config that mattered:

  create statefulSet failed: statefulsets.apps is forbidden:
  User "system:serviceaccount:dev:agent-task-executor"
  cannot create resource "statefulsets" in API group "apps" in the namespace "dev"

The feature cannot work on any cluster without this rule. No SC, DoD row or task
line named RBAC, so nothing in the task could have caught it before the first
live reconcile -- the grant is only reachable once a Config actually declares
type: service, which is the last step of the deploy.

delete is granted from executor v0.18.1 onwards only. That is the release in
which ServiceReconciler checks the object's Config controller ownerRef before
removing it; pairing the rule with an older executor would arm an unchecked
delete-by-name across every non-service Config in the namespace, since a
StatefulSet's name here is just the Config's name.

Chart 0.6.6. helm lint: 1 chart linted, 0 failed. Rendered Role verified to
carry the rule.

@ben-s-pull-request-reviewer ben-s-pull-request-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I have all the information needed. Let me write the verdict.

{
  "verdict": "request-changes",
  "summary": "The RBAC fix is correct and necessary, but the `delete` verb on `apps/statefulsets` is only safe with executor >= v0.18.1 (ownerRef guard), and neither the chart nor the RBAC rule provides any enforcement mechanism to prevent an older executor from being paired with this RBAC — the comment is advisory only. This leaves a window where a job Config could delete any StatefulSet in the namespace by name.",
  "comments": [
    {
      "file": "helm/templates/executor-rbac.yaml",
      "line": 57,
      "severity": "critical",
      "blocking": true,
      "blocking_reason": "delete on apps/statefulsets is unsafe without executor >= v0.18.1 ownerRef guard; advisory comment does not prevent version mismatch at deploy time",
      "message": "The `delete` verb on `apps/statefulsets` is granted unconditionally with no version enforcement. The comment documents that `delete` requires executor >= v0.18.1 (ownerRef guard), but the chart has no mechanism to pin or validate the executor image version. A deployer could use this chart (v0.6.6) with executor v0.17.x and gain unchecked delete-by-name over every StatefulSet in the namespace. The comment `Grant the rule, but do not pin an executor older than v0.18.1 alongside it` is advisory — it warns but does not prevent. Consider: (1) a Helm post-install hook or pre-delete hook that validates the executor version via a note/warning, (2) a `values.yaml` entry `executor.version` that is checked against a minimum in the Role template, or (3) at minimum a more prominent warning in `values.yaml` and the NOTES.txt template."
    }
  ],
  "concerns_addressed": [
    {
      "concern": "security: delete verb on apps/statefulsets is only safe with executor >= v0.18.1 (ownerRef guard); pairing older executor with this RBAC could allow deletion of arbitrary StatefulSets in the namespace by name",
      "disposition": "not-an-issue",
      "detail": "The comment at executor-rbac.yaml:52-56 is thorough and accurate — it names v0.18.1 as the guard release and explains the ownerRef mechanism. The delete permission is necessary for the reconcile loop to clean up StatefulSets when a Config transitions away from type=service. The CHANGELOG entry (Unreleased) also documents this correctly."
    },
    {
      "concern": "correctness: version coupling is advisory only — no enforcement preventing older executor from being deployed alongside this RBAC",
      "disposition": "addressed",
      "detail": "This is the blocking finding. The comment warns but does not enforce. The chart cannot pin the executor image version because the executor is deployed separately from this chart, so no technical gate exists — only a documentation/advisory gate. This is correctly flagged as a Must Fix."
    }
  ]
}

… comment

Review finding, blocking: `delete` on apps/statefulsets was granted
unconditionally while its safety depended on the executor being >= v0.18.1, and
the chart cannot see which executor version it is paired with. An advisory
comment is not enforcement.

The finding's stated example was wrong -- an executor below v0.18.0 has no
undeploy path at all, so `delete` is inert there. The unsafe pairing is narrower
but real: v0.18.0 shipped the undeploy path *without* the ownerRef guard, and
v0.18.0 is exactly what nuke dev runs today, so the window is the next deploy
rather than a hypothetical.

Rather than add a version gate, withhold the verb. `get` is enough for the
guarded undeploy path to find nothing for the non-service Configs and return
quietly, so the reconcile loop stays silent and the feature works -- and no
executor version can delete a StatefulSet by name through this Role, because the
permission simply is not there. That removes the coupling instead of documenting
it, which is what the chart can actually do about a version it cannot see.

`delete` can be granted later, once the pairing is enforced rather than
documented. The platform's existing gate for that is `check-executor-compat` in
nuke/agent, which runs before mirror+upgrade and already refuses incompatible
EXECUTOR_VERSION/chart combinations; the comment and CHANGELOG name it as the
precondition.

helm lint: 1 chart linted, 0 failed. Rendered Role verified to emit
[create, get, update] and no delete.
@ben-s-pull-request-reviewer
ben-s-pull-request-reviewer Bot dismissed their stale review September 25, 2026 20:02

superseded by new automated review

@ben-s-pull-request-reviewer ben-s-pull-request-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for approval to run the git diff command to identify changed files.

@ben-s-pull-request-reviewer ben-s-pull-request-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{
  "verdict": "approve",
  "summary": "RBAC fix adds `apps/statefulsets` create/get/update to the executor Role — narrowly scoped, `delete` correctly withheld with thorough documentation, Chart version bumped, CHANGELOG entry accurate. No must-fix or should-fix findings.",
  "comments": [],
  "concerns_addressed": [
    {
      "concern": "security: `delete` on statefulsets is deliberately withheld — safety depends on executor version documentation",
      "disposition": "not-an-issue",
      "detail": "The YAML comment (lines 52-64) is thorough: explains StatefulSet name = Config name, ownerRef guard requirement (v0.18.1+), why the chart can't see executor version, and names `check-executor-compat` in nuke/agent as the platform gate. The CHANGELOG entry is equally explicit. No technical enforcement gap beyond what is documented."
    },
    {
      "concern": "security: statefulsets grant includes `update` verb — verify not broader than needed",
      "disposition": "not-an-issue",
      "detail": "`update` is the standard verb for modifying StatefulSet spec fields (vs `patch` for strategic/JSON patch). The reconcile loop mutates spec, making `update` appropriate and not broader than needed for the actual mutation path."
    },
    {
      "concern": "correctness: `delete` pairing not technically enforced — relies on comment documentation",
      "disposition": "not-an-issue",
      "detail": "`delete` is intentionally withheld. The CHANGELOG explicitly documents the v0.18.0 gap (undeploy without ownerRef guard) and the v0.18.1+ requirement. `check-executor-compat` in nuke/agent is named as the gate. The comment chain in the YAML makes this explicit to any reader."
    }
  ]
}

…README

The review asked for the version coupling to be enforced rather than only
described in a template comment. Withholding the `delete` verb removed the
unsafe grant entirely, but the coupling still exists for whoever widens it
later, and a chart consumer reading the README has no way to know that
apps/statefulsets delete is only safe on executor >= v0.18.1.

The Prerequisites list is the durable home for it: a NOTES.txt would reprint
the warning on every upgrade, and a values key would document a knob that
does not exist.
@ben-s-pull-request-reviewer
ben-s-pull-request-reviewer Bot dismissed stale reviews from themself September 25, 2026 21:06

superseded by new automated review

@ben-s-pull-request-reviewer ben-s-pull-request-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Files changed: 4 (CHANGELOG.md, helm/Chart.yaml, helm/README.md, helm/templates/executor-rbac.yaml)

Scope: Bugfix — grants create/get/update on apps/statefulsets to the executor's Role so the service-agent StatefulSet reconcile loop can function.

Concern Verification

Concern Disposition
security: minimum verbs on statefulsets ✅ not-an-issue — only create, get, update are granted. No list/watch/delete. The verbs granted are the minimum required for a reconcile loop that creates/reads/updates StatefulSets.
security: delete withheld ✅ not-an-issue — delete is deliberately absent per the PR description. The Role comment at line 52–64 explains the safety rationale: the undeploy path deletes by name, which is only safe when the executor checks ownerRef first (v0.18.1+). The chart cannot detect the executor version, so granting delete would make safety depend on a comment rather than enforcement. Withholding it is the conservative choice. The comment also references check-executor-compat in nuke/agent as the platform's gate.
correctness: README version floor ✅ not-an-issue — README lines 38–45 document Executor ≥ v0.18.0 for type: service agents and ≥ v0.18.1 for the delete-safe pairing. This matches the code comment which names v0.18.1 as the release with the ownerRef guard.
tests: no RBAC test coverage ✅ not-an-issue — This is a Helm chart with no existing test suite. The PR ran helm lint. Adding a dedicated RBAC test suite would be a new capability, not a gap in this PR.

Mechanical Funnel

  • ast-grep over 79 Helm YAML files: 0 findings
  • ast-grep-runner.sh mechanical funnel: 0 findings
  • No rule-relevant files (go.mod, .py, .js, etc.) in diff → judgment rules not triggered

Manual Review

The RBAC change is minimal and correct:

  • New rule block at helm/templates/executor-rbac.yaml:65–67 grants apps/statefulsets → create, get, update
  • The surrounding comment block (lines 50–64) is thorough and explains the delete-withholding rationale clearly
  • helm/Chart.yaml version bumped 0.6.5 → 0.6.6 (patch bump, correct for a bugfix)
  • helm/README.md documents the new executor version requirement
  • CHANGELOG.md has a detailed ## Unreleased entry describing the bug, the fix, and the safety reasoning for withholding delete
{
  "verdict": "approve",
  "summary": "Bugfix adds create/get/update on apps/statefulsets to the executor's Role, enabling the service-agent StatefulSet reconcile loop to function. The delete verb is deliberately withheld with a clear comment explaining the ownerRef guard requirement (v0.18.1+); this is the correct conservative choice since the chart cannot detect the paired executor version. README documents the version floor. Mechanical funnel clean.",
  "comments": [],
  "concerns_addressed": [
    {
      "concern": "security: grants create/get/update on apps/statefulsets to executor ServiceAccount — verify these are the minimum verbs the reconciler actually requires, and that no additional verbs (e.g., list, watch) are needed on statefulsets",
      "disposition": "not-an-issue",
      "detail": "Verified helm/templates/executor-rbac.yaml:65-67 — only create, get, update are granted. No list/watch. These are the minimum verbs needed for a reconcile loop that creates, reads, and updates StatefulSets."
    },
    {
      "concern": "security: delete is deliberately withheld per the PR description — verify the ownerRef guard in the undeploy path (executor v0.18.1+) is correctly implemented and cannot be bypassed",
      "disposition": "not-an-issue",
      "detail": "Verified helm/templates/executor-rbac.yaml:52-64 — the comment block explains the safety rationale in detail: v0.18.0 shipped the undeploy path without the ownerRef guard (what dev runs today), v0.18.1+ adds the guard. The chart correctly withholds delete since it cannot detect executor version, and the comment references check-executor-compat in nuke/agent as the platform's gate for this pairing."
    },
    {
      "concern": "correctness: README documents executor version constraint (>= v0.18.0 for service type, >= v0.18.1 for delete safety) — verify the version floor is accurate and consistent with actual executor releases",
      "disposition": "not-an-issue",
      "detail": "Verified helm/README.md:38-45 — executor >= v0.18.0 for type: service, >= v0.18.1 for delete-safety pairing. Consistent with executor-rbac.yaml comments naming v0.18.1 as the ownerRef-guarded release."
    },
    {
      "concern": "tests: no test coverage for the new RBAC rules — helm lint was run per the PR but no e2e or RBAC-specific tests verify the Role renders correctly in different values scenarios",
      "disposition": "not-an-issue",
      "detail": "No RBAC test suite exists in this Helm chart — this is pre-existing, not a gap introduced by this PR. Helm lint was run. Adding a Helm test suite would be a new capability beyond this bugfix scope."
    }
  ]
}

@bborbe
bborbe merged commit 1af14b5 into master Sep 25, 2026
3 checks passed
@bborbe
bborbe deleted the feat/executor-statefulset-rbac branch September 25, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant