Skip to content

fix(gke-cluster): canonicalize monitoring_components order to prevent Day-2 drift - #6524

Closed
Neelabh94 wants to merge 1 commit into
GoogleCloudPlatform:developfrom
Neelabh94:fix-gke-monitoring-components-order
Closed

Neelabh94 wants to merge 1 commit into
GoogleCloudPlatform:developfrom
Neelabh94:fix-gke-monitoring-components-order

Conversation

@Neelabh94

Copy link
Copy Markdown
Contributor

Summary

Canonicalizes the ordering of monitoring_config.enable_components in modules/scheduler/gke-cluster to match the GKE control plane's canonical MonitoringComponentConfig.Component protobuf enum order:

  1. Reorders the default var.monitoring_components list in modules/scheduler/gke-cluster/variables.tf so "STORAGE" and "HPA" appear immediately after "SYSTEM_COMPONENTS".
  2. Normalizes local.monitoring_components in modules/scheduler/gke-cluster/main.tf against GKE's canonical enum order (SYSTEM_COMPONENTS, APISERVER, SCHEDULER, CONTROLLER_MANAGER, STORAGE, HPA, POD, DAEMONSET, DEPLOYMENT, STATEFULSET, CADVISOR, KUBELET, DCGM, JOBSET) while preserving any unrecognized future component strings at the end.
  3. Guards "DCGM" injection with length(var.monitoring_components) > 0 so setting monitoring_components = [] cleanly disables GKE monitoring even when enable_dcgm_monitoring is left at its default true.

Root Cause & Impact

In terraform-provider-google / google-beta 7.x, google_container_cluster.monitoring_config.enable_components has schema Type: schema.TypeList. On cluster reads, the GKE API always returns monitoringConfig.componentConfig.enableComponents sorted in GKE's canonical protobuf enum order. Because variables.tf previously placed "STORAGE" and "HPA" after "STATEFULSET" and main.tf appended "DCGM" after "JOBSET", every Day-2 gcluster deploy / terraform plan against an existing GKE cluster detected a spurious in-place update on google_container_cluster.gke_cluster:

  • Triggered an unnecessary UpdateClusterInternal control-plane operation that locked the cluster and blocked concurrent node-pool mutations.
  • Forced module.workload_identity data sources (depends_on = [google_container_cluster.gke_cluster]) to defer evaluation to apply time, recreating google_service_account_iam_member.main on every Day-2 run.

Compatibility

  • Existing Workflows & Deployed Clusters: Existing clusters already store GKE's canonical component order in .tfstate, so Day-2 terraform plan becomes a 0-diff no-op.
  • terraform-provider-google >= 8.0.0: Compatible with both 7.x (schema.TypeList) and >= 8.0.0 (schema.TypeSet).

… Day-2 drift

Sort monitoring_config.enable_components into GKE's canonical MonitoringComponentConfig.Component enum order (SYSTEM_COMPONENTS, APISERVER, SCHEDULER, CONTROLLER_MANAGER, STORAGE, HPA, POD, DAEMONSET, DEPLOYMENT, STATEFULSET, CADVISOR, KUBELET, DCGM, JOBSET) while preserving any unrecognized future component strings at the end, and guard DCGM injection on non-empty var.monitoring_components.
@Neelabh94 Neelabh94 added the release-bugfix Added to release notes under the "Bug fixes" heading. label Oct 19, 2026
@Neelabh94 Neelabh94 self-assigned this Oct 19, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a recurring Day-2 drift issue in GKE cluster configurations caused by mismatched ordering of monitoring components between Terraform and the GKE API. By enforcing the canonical enum order used by the GKE control plane, the changes ensure that Terraform plans remain stable and avoid unnecessary cluster updates that previously locked resources and caused secondary side effects during deployment.

Highlights

  • Canonical Ordering: Reordered monitoring components in variables and main configuration to match the GKE control plane's canonical protobuf enum order, preventing spurious Day-2 infrastructure updates.
  • DCGM Injection Guard: Added a safety check to ensure DCGM monitoring is only injected if monitoring components are enabled, allowing for a clean disablement of GKE monitoring.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request standardizes the ordering of GKE monitoring components in the GKE cluster module. It introduces a local ordered list gke_monitoring_component_order and logic to sort the enabled monitoring components accordingly, preventing potential configuration drift due to list ordering. The default values in variables.tf and the documentation in README.md have been updated to reflect this new order. I have no additional feedback to provide as the implementation is clean and follows best practices.

@Neelabh94 Neelabh94 closed this Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-bugfix Added to release notes under the "Bug fixes" heading.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants