Skip to content

fix(terraform)!: prevent unintended replacements during AIO upgrades - #884

Open
Katrien De Graeve (katriendg) wants to merge 6 commits into
mainfrom
fix/883-aio-upgrade
Open

Katrien De Graeve (katriendg) wants to merge 6 commits into
mainfrom
fix/883-aio-upgrade

Conversation

@katriendg

Copy link
Copy Markdown
Collaborator

Description

Prevent deferred Arc cluster lookups from causing unintended replacement cascades during Azure IoT Operations (AIO) upgrades, and keep existing Linux VM bootstrap payloads unchanged during routine reconciliation. Also correct storage-account provider compatibility and update the Bicep schema-registry API version.

Bootstrap content becomes create-only: later script and template-input changes require explicitly reviewed maintenance instead of automatically updating deployed secrets or rerunning VM setup.

Related Issue

Closes #883.

This implements the source safeguards and bootstrap maintenance documentation.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Blueprint modification or addition
  • Component modification or addition
  • Documentation update
  • CI/CD pipeline change
  • Other (please describe):

Implementation Details

  • Derive Arc cluster ID and name from configuration, retain live location and optional properties, and preserve lookup ordering. Explicit dependencies keep the four Arc RBAC assignments waiting for the cluster lookup.
  • Ignore changes only to K3s script-secret value and content_type, and Linux VM extension protected_settings. Other attributes remain managed; deletion or identity changes can still destroy or replace resources.
  • Document coordinated plaintext-to-compressed secret and loader migration before rerunning setup, rebuilding VMs, or adding nodes. The VM extension safeguard does not cover Arc-machine or SSH script delivery.
  • Use public_network_access_enabled to preserve the Boolean input behavior with AzureRM 5.3.0 and 5.9.0. Provider constraints remain unchanged.
  • Change the Bicep schema-registry API from 2024-09-01-preview to 2026-04-01 and regenerate its resource reference.

No cluster setup scripts, extension version pins, or deployment-specific dataflow graph inputs are changed by this commit.

Testing Performed

  • Terraform plan/apply
  • Blueprint deployment test
  • Unit tests
  • Integration tests
  • Bug fix includes regression test (see Test Policy)
  • Manual validation
  • Other: Terraform validation, lint, formatting, documentation generation, and committed patch hygiene

Evidence collected during implementation:

  • Terraform Build completed lint/format, validation, and documentation generation across 67 component and blueprint roots, including roots using AzureRM 5.3.0.
  • The full-multi-node blueprint validated with AzureRM 5.9.0.
  • Bicep documentation generation completed successfully.

The plan evidence is not a fresh-deployment or Kubernetes workload-health test. No regression tests were added. Hosted CI and first-deployment ordering verification remain pending.

Validation Steps

  1. Run formatting, lint, and validation checks for the changed Terraform components, and compile the changed Bicep module.
  2. Back up state for a representative existing VM-based deployment and review a full plan. Confirm no replacements arise solely from deferred cluster identity and no bootstrap secret or VM payload updates are scheduled.
  3. Review genuine configuration changes separately. Refresh-only reconciliation does not guarantee a no-op normal plan, and Container Storage version changes require their own reviewed upgrade procedure.
  4. Verify a fresh deployment preserves cluster lookup and RBAC ordering, and exercise the optional-cluster output path.
  5. Review secret-format and loader compatibility before intentionally rerunning setup or adding/rebuilding nodes; require hosted CI before merging.

Checklist

  • I have updated the documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed
  • I have run terraform fmt on all Terraform code
  • I have run terraform validate on all Terraform code
  • I have run az bicep format on all Bicep code
  • I have run az bicep build to validate all Bicep code
  • I have checked for any sensitive data/tokens that should not be committed
  • Lint checks pass (run applicable linters for changed file types)

Security Review

  • No credentials, secrets, or tokens are hardcoded or logged
  • RBAC and identity changes follow least-privilege principles
  • No new network exposure or public endpoints introduced without justification
  • Dependency additions or updates have been reviewed for known vulnerabilities
  • Container image changes use pinned digests or SHA references

Additional Notes

The storage Boolean argument is deprecated toward AzureRM v6; this change retains compatibility within the existing v5 constraint rather than upgrading the provider.

Bootstrap fixes, including changes to in-cluster administrator bindings, will no longer propagate automatically through the ignored payload fields. Maintainers must plan explicit maintenance for existing deployments.

Screenshots (if applicable)

Not applicable: infrastructure and documentation changes only.

…ipt handling

- change schema registry API version to 2026-04-01
- update public network access variable in storage account module
- enhance Key Vault script retrieval documentation
- add lifecycle rules to ignore changes in script values
- modify role assignment dependencies for Arc cluster

🔧 - Generated by Copilot
… steps

- clarify Terraform state refresh process
- enhance instructions for aligning configurations
- add warnings about bootstrap maintenance

🔧 - Generated by Copilot
@katriendg
Katrien De Graeve (katriendg) requested a review from a team October 14, 2026 11:34
@github-actions

Copy link
Copy Markdown

📚 Documentation Health Report

Generated on: 2026-10-14 11:39:06 UTC

📈 Documentation Statistics

Category File Count
Main Documentation 224
Infrastructure Components 238
Blueprints 40
GitHub Resources 26
AI Assistant Guides (Copilot) 17
Total 545

🏗️ Three-Tree Architecture Status

  • ✅ Bicep Documentation Tree: Auto-generated navigation
  • ✅ Terraform Documentation Tree: Auto-generated navigation
  • ✅ README Documentation Tree: Manual README organization

🔍 Quality Metrics

  • Frontmatter Validation:
    success
  • Link Validation: success

This report is automatically generated by the Documentation Automation workflow.

Co-authored-by: katriendg <838216+katriendg@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

📚 Documentation Health Report

Generated on: 2026-10-22 15:32:53 UTC

📈 Documentation Statistics

Category File Count
Main Documentation 224
Infrastructure Components 238
Blueprints 40
GitHub Resources 26
AI Assistant Guides (Copilot) 17
Total 545

🏗️ Three-Tree Architecture Status

  • ✅ Bicep Documentation Tree: Auto-generated navigation
  • ✅ Terraform Documentation Tree: Auto-generated navigation
  • ✅ README Documentation Tree: Manual README organization

🔍 Quality Metrics

  • Frontmatter Validation:
    success
  • Link Validation: success

This report is automatically generated by the Documentation Automation workflow.

@kgmwang1 kgmwang1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for helping tighten AIO updates to prevent unintended replacements. Only one non-blocking edit requested in-line but everything else looks great

Comment thread src/100-edge/100-cncf-cluster/terraform/main.tf Outdated
@github-actions

Copy link
Copy Markdown

📚 Documentation Health Report

Generated on: 2026-10-27 09:20:50 UTC

📈 Documentation Statistics

Category File Count
Main Documentation 224
Infrastructure Components 238
Blueprints 40
GitHub Resources 26
AI Assistant Guides (Copilot) 17
Total 545

🏗️ Three-Tree Architecture Status

  • ✅ Bicep Documentation Tree: Auto-generated navigation
  • ✅ Terraform Documentation Tree: Auto-generated navigation
  • ✅ README Documentation Tree: Manual README organization

🔍 Quality Metrics

  • Frontmatter Validation:
    success
  • Link Validation: success

This report is automatically generated by the Documentation Automation workflow.

Katrien De Graeve (katriendg) added a commit that referenced this pull request Oct 27, 2026
…tests (#907)

## Description

Updated `docusaurus-tests.yml` with `persist-credentials: false` to
address the checkout-credential requirement in #859. All other workflow
configuration is unchanged.

The branch also took in dependency vulnerability fixes after the failed
security checks on #884 exposed repository-wide blockers. These updates
address fixable findings needed for CI to pass; final validation remains
with CI, and the upstream Zod finding is still unresolved.

## Related Issue

Relates to #859. This PR addresses the Docusaurus checkout-credential
gap only; it does not complete the Merge Queue rollout or close the
issue.

Related CI failure: #884.

## Type of Change
<!-- What type of change does this PR introduce? Mark relevant options
with 'x' -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Blueprint modification or addition
- [x] Component modification or addition
- [ ] Documentation update
- [x] CI/CD pipeline change
- [x] Other (please describe): Dependency security updates and
Dependabot consolidation

## Implementation Details
<!-- Describe the implementation details and approach taken -->

- Disabled credential persistence on the existing Docusaurus test
checkout without changing its action pin, triggers, permissions, or
other workflow settings.
- Updated Tokio to 1.54.1 across ten affected Rust lockfiles to address
RUSTSEC-2026-0386.
- Updated affected npm dependency constraints and lockfiles for
`body-parser`, `undici`, `fast-uri`, `ip-address`, and
`serialize-javascript`.
- Updated MSAL Node to 7.1.1 and aligned the Azure Identity transitive
dependency with that patched version.
- Consolidated the Microsoft Graph Go SDK update from #906, the Vitest
updates from #904, and the certifi update from #905.
- Retained the existing security thresholds and advisory policy.

## Testing Performed
<!-- Describe the testing you have performed or plan to perform -->
<!-- For bug fixes: A regression test verifies the fix and prevents the
issue from recurring -->
- [ ] Terraform plan/apply
- [ ] Blueprint deployment test
- [x] Unit tests
- [ ] Integration tests
- [ ] Bug fix includes regression test (see [Test
Policy](https://github.com/microsoft/edge-ai/blob/main/docs/contributing/testing-validation.md))
- [ ] Manual validation
- [x] Other: Rust dependency audits, chat server compilation, Go module
verification, workflow permission validation, and patch hygiene

- All 22 Rust lockfiles passed `cargo audit --deny warnings` with the
repository's existing advisory and yanked-package policy.
- The chat server TypeScript build and all 38 tests passed after the
MSAL update, before the final Vitest patch update. CI must validate the
final locked test-tool versions.
- `go mod verify` passed for the updated Microsoft Graph dependency.
- Workflow permission validation passed for all 38 scanned workflows.
- `git diff --check origin/main...HEAD` passed.
- Full builds, Go vulnerability analysis, browser and deployment tests,
and final security scans remain for CI.

## Validation Steps
<!-- Provide steps that reviewers should follow to validate your changes
-->

1. Confirm the only workflow change is `persist-credentials: false` on
the Docusaurus test checkout.
2. Verify Docusaurus tests, dependency review, Cargo Audit, and Security
Scan results on the final branch commit.
3. Confirm the chat application builds and its tests pass with the final
npm lockfile.
4. Keep #859 open for the remaining trust gates, hosted activation, and
live Merge Queue proof.

## Checklist
<!-- Mark relevant options with 'x' -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] All new and existing tests passed
- [ ] I have run `terraform fmt` on all Terraform code
- [ ] I have run `terraform validate` on all Terraform code
- [ ] I have run `az bicep format` on all Bicep code
- [ ] I have run `az bicep build` to validate all Bicep code
- [ ] I have checked for any sensitive data/tokens that should not be
committed
- [ ] Lint checks pass (run applicable linters for changed file types)

## Security Review

- [ ] No credentials, secrets, or tokens are hardcoded or logged
- [ ] RBAC and identity changes follow least-privilege principles
- [ ] No new network exposure or public endpoints introduced without
justification
- [ ] Dependency additions or updates have been reviewed for known
vulnerabilities
- [ ] Container image changes use pinned digests or SHA references

## Additional Notes
<!-- Add any other context about the PR here -->

- Zod remains a known security-scan blocker because the upstream Agents
SDK requires v3. A forced v4 override was not applied because its schema
API is incompatible.
- The Checkov portion of #905 was not included because it introduces
`ecdsa`, which has an unpatched high-severity advisory.
- No security checks were disabled, and these changes do not assert that
hosted CI has passed.

## Screenshots (if applicable)
<!-- Add screenshots to show the changes, if applicable -->

Not applicable.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

📚 Documentation Health Report

Generated on: 2026-10-27 12:18:17 UTC

📈 Documentation Statistics

Category File Count
Main Documentation 224
Infrastructure Components 238
Blueprints 40
GitHub Resources 26
AI Assistant Guides (Copilot) 17
Total 545

🏗️ Three-Tree Architecture Status

  • ✅ Bicep Documentation Tree: Auto-generated navigation
  • ✅ Terraform Documentation Tree: Auto-generated navigation
  • ✅ README Documentation Tree: Manual README organization

🔍 Quality Metrics

  • Frontmatter Validation:
    success
  • Link Validation: success

This report is automatically generated by the Documentation Automation workflow.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(terraform): prevent destructive plans during Azure IoT Operations upgrades

3 participants