OCPBUGS-112272: node ServiceMonitor serverName uses guest namespace on HyperShift

[bryan-cox]

What

On HyperShift, the node metrics ServiceMonitor that csi-operator creates in the guest cluster set its TLS serverName from the control-plane namespace (${NAMESPACE}) instead of the guest namespace. The name never matched the service serving certificate, so every node metrics scrape failed certificate verification, the node CSI driver metrics target was permanently down, and TargetDown fired continuously.

Impact is limited to observability — CSI provisioning and attach/detach are unaffected — but node CSI metrics are never collected and the permanently-failing target produces continuous alert noise.

Root cause

The shared patch assets/common/metrics/service_monitor_add_port.yaml.patch hardcoded ${NAMESPACE} in serverName. That patch is applied by generateDriverMetricsService() for both the controller monitor (where ${NAMESPACE} is correct, since the controller runs control-plane side) and the node monitor (where it is wrong, since the node DaemonSet and its metrics Service run guest side).

In pkg/driver/common/operator/replacer.go the two namespace variables resolve to different values under HyperShift:

• ${NAMESPACE} → control-plane namespace
• ${NODE_NAMESPACE} → guest namespace

On standalone both resolve to openshift-cluster-csi-drivers, which is why the defect was invisible there.

Fix

• Introduce a ${MONITOR_NAMESPACE} template variable in the shared ServiceMonitor patch.
• Resolve it per service prefix in the generator (monitorNamespaceVariable): ${NODE_NAMESPACE} for the node monitor, ${NAMESPACE} for the controller monitor. Wired into both generateDriverMetricsService and generateSidecarMetricsServices.
• Regenerated assets via make update.

Scope / safety

• HyperShift-only behavior change. On standalone, ${NAMESPACE} and ${NODE_NAMESPACE} both resolve to openshift-cluster-csi-drivers, so the rendered serverName is byte-for-byte identical to before — only the template variable changed.
• Controller ServiceMonitors unchanged across all drivers (still ${NAMESPACE}).
• Fix is in shared generator code, so any driver enabling node metrics inherits it. azure-disk is currently the only driver that generates a node metrics ServiceMonitor.

Testing

• Added regression tests at the generator seam (pkg/generator/asset_generator_test.go) asserting the node monitor resolves to ${NODE_NAMESPACE} and the controller monitor to ${NAMESPACE}, using the real embedded assets.
• make update is idempotent (no unintended diffs), go build ./... clean, go test ./pkg/... all pass.

Acceptance criteria

• Node ServiceMonitor serverName resolves to the guest namespace, matching the serving cert SANs.
• Generated HyperShift node asset renders serverName from ${NODE_NAMESPACE}.
• Controller metrics ServiceMonitor unchanged (${NAMESPACE}).
• Standalone generated assets remain functionally correct.
• Assets regenerated via the generator/make target; fix covers all affected driver overlays via shared code.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@bryan-cox: This pull request references Jira Issue OCPBUGS-112272, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.1.0) matches configured target version for branch (5.1.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What

On HyperShift, the node metrics ServiceMonitor that csi-operator creates in the guest cluster set its TLS serverName from the control-plane namespace (${NAMESPACE}) instead of the guest namespace. The name never matched the service serving certificate, so every node metrics scrape failed certificate verification, the node CSI driver metrics target was permanently down, and TargetDown fired continuously.

Impact is limited to observability — CSI provisioning and attach/detach are unaffected — but node CSI metrics are never collected and the permanently-failing target produces continuous alert noise.

Root cause

The shared patch assets/common/metrics/service_monitor_add_port.yaml.patch hardcoded ${NAMESPACE} in serverName. That patch is applied by generateDriverMetricsService() for both the controller monitor (where ${NAMESPACE} is correct, since the controller runs control-plane side) and the node monitor (where it is wrong, since the node DaemonSet and its metrics Service run guest side).

In pkg/driver/common/operator/replacer.go the two namespace variables resolve to different values under HyperShift:

• ${NAMESPACE} → control-plane namespace
• ${NODE_NAMESPACE} → guest namespace

On standalone both resolve to openshift-cluster-csi-drivers, which is why the defect was invisible there.

Fix

• Introduce a ${MONITOR_NAMESPACE} template variable in the shared ServiceMonitor patch.
• Resolve it per service prefix in the generator (monitorNamespaceVariable): ${NODE_NAMESPACE} for the node monitor, ${NAMESPACE} for the controller monitor. Wired into both generateDriverMetricsService and generateSidecarMetricsServices.
• Regenerated assets via make update.

Scope / safety

• HyperShift-only behavior change. On standalone, ${NAMESPACE} and ${NODE_NAMESPACE} both resolve to openshift-cluster-csi-drivers, so the rendered serverName is byte-for-byte identical to before — only the template variable changed.
• Controller ServiceMonitors unchanged across all drivers (still ${NAMESPACE}).
• Fix is in shared generator code, so any driver enabling node metrics inherits it. azure-disk is currently the only driver that generates a node metrics ServiceMonitor.

Testing

• Added regression tests at the generator seam (pkg/generator/asset_generator_test.go) asserting the node monitor resolves to ${NODE_NAMESPACE} and the controller monitor to ${NAMESPACE}, using the real embedded assets.
• make update is idempotent (no unintended diffs), go build ./... clean, go test ./pkg/... all pass.

Acceptance criteria

• Node ServiceMonitor serverName resolves to the guest namespace, matching the serving cert SANs.
• Generated HyperShift node asset renders serverName from ${NODE_NAMESPACE}.
• Controller metrics ServiceMonitor unchanged (${NAMESPACE}).
• Standalone generated assets remain functionally correct.
• Assets regenerated via the generator/make target; fix covers all affected driver overlays via shared code.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: fd4327c7-eaae-4f58-a942-79d4bf4ccf6a

📥 Commits

Reviewing files that changed from the base of the PR and between 9ce1841 and 82a49f1.

⛔ Files ignored due to path filters (2)

• assets/overlays/azure-disk/generated/hypershift/node_servicemonitor.yaml is excluded by !**/generated/**
• assets/overlays/azure-disk/generated/standalone/node_servicemonitor.yaml is excluded by !**/generated/**

📒 Files selected for processing (3)

• assets/common/metrics/service_monitor_add_port.yaml.patch
• pkg/generator/asset_generator.go
• pkg/generator/asset_generator_test.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

The ServiceMonitor TLS serverName now uses ${MONITOR_NAMESPACE}. The generator selects ${NODE_NAMESPACE} for node metrics and ${NAMESPACE} for controller and other metrics. Driver and sidecar monitoring patches receive the selected namespace variable. Regression tests verify rendered node and controller ServiceMonitor assets.

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The added tests use static Go test function names and contain no generated names, timestamps, UUIDs, node names, namespaces with random suffixes, IPs, or other run-dependent values.
Test Structure And Quality	✅ Passed	The added tests use Go's testing package, not Ginkgo. They perform no cluster operations or waits, create no resources, isolate one related behavior per test, and include diagnostic assertion messa...
Microshift Test Compatibility	✅ Passed	The PR adds standard Go tests using package testing, not new Ginkgo e2e tests; therefore the MicroShift API compatibility check does not apply.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The PR adds only standard Go testing.Test unit tests; no Ginkgo e2e tests or multi-node/SNO assumptions are present.
Topology-Aware Scheduling Compatibility	✅ Passed	The diff changes only ServiceMonitor TLS namespace substitution, generator replacements, and tests; it adds no affinity, topology spread, replica, selector, toleration, or PDB scheduling constraint.
Ote Binary Stdout Contract	✅ Passed	The diff changes YAML, generator logic, and unit tests only; no process-level OTE setup changed, and changed Go files contain no stdout, fmt, log, or klog writes.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The diff adds only standard-library unit tests in pkg/generator; it adds no Ginkgo e2e tests, IPv4 assumptions, external URLs, or connectivity requirements.
No-Weak-Crypto	✅ Passed	The PR changes only ServiceMonitor namespace templating and tests; scans of all introduced lines found no MD5, SHA1, DES, RC4, Blowfish, ECB, custom crypto, or secret comparisons.
Container-Privileges	✅ Passed	The PR diff changes ServiceMonitor serverName substitutions, generator replacements, generated assets, and tests; it adds no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or allowPrivilegeE...
No-Sensitive-Data-In-Logs	✅ Passed	The PR adds no application logging or logger imports. Changes contain only templated ServiceMonitor names and test diagnostics with synthetic values; the existing Logf call is unchanged.
Title check	✅ Passed	The title clearly identifies the HyperShift node ServiceMonitor serverName defect and matches the main change.
Description check	✅ Passed	The description directly explains the defect, root cause, fix, scope, and tests for the changeset.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[jsafrane]

/lgtm
/approve

[openshift-merge-bot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-upgrade
/test e2e-azure
/test e2e-azure-csi
/test e2e-azure-file-csi
/test e2e-azure-file-nfs-csi
/test e2e-azure-ovn-upgrade
/test e2e-gcp
/test e2e-gcp-csi
/test e2e-gcp-ovn-upgrade

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-azure-csi-extended
/test e2e-azure-csi-hypershift
/test e2e-azurestack-csi

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jsafrane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jsafrane]

@bryan-cox have you tested this change? Is its result visible in CI? Please add /verified by by yourself.

[bryan-cox]

@bryan-cox have you tested this change? Is its result visible in CI? Please add /verified by by yourself.

Hey @jsafrane - @clebs was planning on verifying it but I don't know where he's at on this atm. I'll check in with him today.

[clebs]

/verified by @clebs

Manual test plan execution to verify the fix:

Results for test plan 603

Test Scenarios

Scenario 1: Node ServiceMonitor serverName resolves to guest namespace

✅ Confirm the core fix — node metrics ServiceMonitor TLS serverName matches the guest namespace, not the control-plane namespace.

On the azure hosted cluster check the node metrics serverName:

~ oc get servicemonitor -n openshift-cluster-csi-drivers -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.endpoints[*].tlsConfig.serverName}{"\n"}{end}'

azure-disk-csi-driver-node-monitor: azure-disk-csi-driver-node-metrics.openshift-cluster-csi-drivers.svc

Scenario 2: Controller ServiceMonitor serverName resolves to control-plane namespace

✅ Verify the controller ServiceMonitor was not regressed — it should still use the control-plane namespace.

On the AWS management cluster check the control-plane-metrics servicemonitor serverName:

~ oc get servicemonitor -n clusters-bclement-hc-az -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.endpoints[*].tlsConfig.serverName}{"\n"}{end}' | grep control-plane-metrics

monitor-ovn-control-plane-metrics: ovn-kubernetes-control-plane.clusters-bclement-hc-az.svc

Scenario 3: Node metrics are actually being scraped

✅ Confirm the fix resolves the actual symptom — node CSI driver metrics were never collected because TLS verification failed.

On the azure hosted cluster check Prometheus targets for the node metrics endpoint:

~ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- \
curl -s http://localhost:9090/api/v1/targets | \
jq -r '.data.activeTargets[] | select(.labels.job | contains("node-metrics")) | "\(.labels.job) \(.health)"'

azure-disk-csi-driver-node-metrics up
azure-disk-csi-driver-node-metrics up

Scenario 4: No TargetDown alert firing for node CSI metrics

✅ Confirm the alert noise described in the bug is resolved.

On the azure hosted cluster check that no TargetDown alerts are firing:

oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- \
curl -s http://localhost:9090/api/v1/alerts | \
jq -r '.data.alerts[] | select(.labels.alertname | contains("TargetDown")) | select(.labels.job | ascii_downcase |
 contains("csi")) | "\(.labels.alertname) \(.state)"'

<empty output>

[openshift-ci-robot]

@clebs: This PR has been marked as verified by @clebs.

Details

In response to this:

/verified by @clebs

Manual test plan execution to verify the fix:

Results for test plan 603

Test Scenarios

Scenario 1: Node ServiceMonitor serverName resolves to guest namespace

✅ Confirm the core fix — node metrics ServiceMonitor TLS serverName matches the guest namespace, not the control-plane namespace.

On the azure hosted cluster check the node metrics serverName:

~ oc get servicemonitor -n openshift-cluster-csi-drivers -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.endpoints[*].tlsConfig.serverName}{"\n"}{end}'

azure-disk-csi-driver-node-monitor: azure-disk-csi-driver-node-metrics.openshift-cluster-csi-drivers.svc

Scenario 2: Controller ServiceMonitor serverName resolves to control-plane namespace

✅ Verify the controller ServiceMonitor was not regressed — it should still use the control-plane namespace.

On the AWS management cluster check the control-plane-metrics servicemonitor serverName:

~ oc get servicemonitor -n clusters-bclement-hc-az -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.endpoints[*].tlsConfig.serverName}{"\n"}{end}' | grep control-plane-metrics

monitor-ovn-control-plane-metrics: ovn-kubernetes-control-plane.clusters-bclement-hc-az.svc

Scenario 3: Node metrics are actually being scraped

✅ Confirm the fix resolves the actual symptom — node CSI driver metrics were never collected because TLS verification failed.

On the azure hosted cluster check Prometheus targets for the node metrics endpoint:

~ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- \
curl -s http://localhost:9090/api/v1/targets | \
jq -r '.data.activeTargets[] | select(.labels.job | contains("node-metrics")) | "\(.labels.job) \(.health)"'

azure-disk-csi-driver-node-metrics up
azure-disk-csi-driver-node-metrics up

Scenario 4: No TargetDown alert firing for node CSI metrics

✅ Confirm the alert noise described in the bug is resolved.

On the azure hosted cluster check that no TargetDown alerts are firing:

oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- \
curl -s http://localhost:9090/api/v1/alerts | \
jq -r '.data.alerts[] | select(.labels.alertname | contains("TargetDown")) | select(.labels.job | ascii_downcase |
contains("csi")) | "\(.labels.alertname) \(.state)"'

<empty output>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bryan-cox]

/retest-required

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD ea3179c and 2 for PR HEAD 82a49f1 in total

[openshift-ci]

@bryan-cox: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azurestack-csi	82a49f1	link	false	/test e2e-azurestack-csi
ci/prow/e2e-azure-csi-hypershift	82a49f1	link	false	/test e2e-azure-csi-hypershift

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[bryan-cox]

/retest-required

[openshift-ci-robot]

@bryan-cox: Jira Issue Verification Checks: Jira Issue OCPBUGS-112272
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-112272 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

What

On HyperShift, the node metrics ServiceMonitor that csi-operator creates in the guest cluster set its TLS serverName from the control-plane namespace (${NAMESPACE}) instead of the guest namespace. The name never matched the service serving certificate, so every node metrics scrape failed certificate verification, the node CSI driver metrics target was permanently down, and TargetDown fired continuously.

Impact is limited to observability — CSI provisioning and attach/detach are unaffected — but node CSI metrics are never collected and the permanently-failing target produces continuous alert noise.

Root cause

The shared patch assets/common/metrics/service_monitor_add_port.yaml.patch hardcoded ${NAMESPACE} in serverName. That patch is applied by generateDriverMetricsService() for both the controller monitor (where ${NAMESPACE} is correct, since the controller runs control-plane side) and the node monitor (where it is wrong, since the node DaemonSet and its metrics Service run guest side).

In pkg/driver/common/operator/replacer.go the two namespace variables resolve to different values under HyperShift:

• ${NAMESPACE} → control-plane namespace
• ${NODE_NAMESPACE} → guest namespace

On standalone both resolve to openshift-cluster-csi-drivers, which is why the defect was invisible there.

Fix

• Introduce a ${MONITOR_NAMESPACE} template variable in the shared ServiceMonitor patch.
• Resolve it per service prefix in the generator (monitorNamespaceVariable): ${NODE_NAMESPACE} for the node monitor, ${NAMESPACE} for the controller monitor. Wired into both generateDriverMetricsService and generateSidecarMetricsServices.
• Regenerated assets via make update.

Scope / safety

• HyperShift-only behavior change. On standalone, ${NAMESPACE} and ${NODE_NAMESPACE} both resolve to openshift-cluster-csi-drivers, so the rendered serverName is byte-for-byte identical to before — only the template variable changed.
• Controller ServiceMonitors unchanged across all drivers (still ${NAMESPACE}).
• Fix is in shared generator code, so any driver enabling node metrics inherits it. azure-disk is currently the only driver that generates a node metrics ServiceMonitor.

Testing

• Added regression tests at the generator seam (pkg/generator/asset_generator_test.go) asserting the node monitor resolves to ${NODE_NAMESPACE} and the controller monitor to ${NAMESPACE}, using the real embedded assets.
• make update is idempotent (no unintended diffs), go build ./... clean, go test ./pkg/... all pass.

Acceptance criteria

• Node ServiceMonitor serverName resolves to the guest namespace, matching the serving cert SANs.
• Generated HyperShift node asset renders serverName from ${NODE_NAMESPACE}.
• Controller metrics ServiceMonitor unchanged (${NAMESPACE}).
• Standalone generated assets remain functionally correct.
• Assets regenerated via the generator/make target; fix covers all affected driver overlays via shared code.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.