Skip to content

fix(cluster): let the low disk space alerts match WAL and tablespace PVCs - #1062

Open
abd-ulbasit wants to merge 1 commit into
cloudnative-pg:mainfrom
abd-ulbasit:dev/966
Open

abd-ulbasit wants to merge 1 commit into
cloudnative-pg:mainfrom
abd-ulbasit:dev/966

Conversation

@abd-ulbasit

Copy link
Copy Markdown

Closes #966

The PrometheusRule template builds podSelector as
<fullname>-([1-9][0-9]*)$. The low disk space rules append -wal and
-tbs.* to it, so they render as:

persistentvolumeclaim=~"<fullname>-([1-9][0-9]*)$-wal"
persistentvolumeclaim=~"<fullname>-([1-9][0-9]*)$-tbs.*"

$ only matches at the end of the value, so neither matcher can match
anything. A full WAL or tablespace volume raises neither
CNPGClusterLowDiskSpaceWarning nor CNPGClusterLowDiskSpaceCritical,
although the runbooks say both volumes are covered. This goes back to
the first version of the rules (#193).

The fix drops the $. Prometheus anchors =~ matchers at both ends
anyway, so the other matchers built from podSelector (the pod=~
matchers and the PGDATA persistentvolumeclaim=~ matchers) select the
same series as before. In the rendered monitoring test, these matchers
are the only lines that change.

The monitoring chainsaw test now checks that both low disk space rules
render the -wal and -tbs.* matchers without the $.

Testing:

  • I did not run the full chainsaw suite (no cluster on my side). I ran
    the new assertions with chainsaw assert -r (v0.2.15) against the
    helm template output. They pass with the fix. All four fail on main.
  • helm lint charts/cluster
  • ct lint --target-branch=main --check-version-increment=false (ct v3.14.0)
  • promtool test rules (v3.15.1) on the rendered rules. With a full WAL
    PVC or a full tablespace PVC, both alerts fire with the fix and stay
    silent on main. With no full PVC, nothing fires. Without the $,
    CNPGClusterInstancesOnSameNode still ignores pooler and job pods on
    the same node.

Disclosure: parts of this change and this description were written with
Claude Code (Claude Opus 5.5), as noted in the Assisted-by commit
trailer.

…PVCs

The podSelector passed to the PrometheusRule templates ends with "$".
The low disk space rules append "-wal" and "-tbs.*" to it, so those
matchers render as "<cluster>-([1-9][0-9]*)$-wal" and never match. A
full WAL or tablespace volume does not raise
CNPGClusterLowDiskSpaceWarning or CNPGClusterLowDiskSpaceCritical.

Drop the "$". Prometheus anchors =~ matchers at both ends, so the other
matchers built from podSelector (the pod matchers and the PGDATA PVC
matchers) select the same series as before.

Check the rendered WAL and tablespace matchers in the monitoring
chainsaw test.

Closes cloudnative-pg#966

Assisted-by: Claude Opus 5.5
Signed-off-by: abd-ulbasit <basit@basit.engineer>
@abd-ulbasit
abd-ulbasit requested review from a team and danishk as code owners October 22, 2026 08:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dollar sign in PrometheusRule template prevent low disk space alerts from firing

1 participant