feat(metadata): centralize schemas, catalog sync and backfill

[Jared Holgate (jaredfholgate)]

Summary

Implements module-owned metadata across Bicep and Terraform, centrally managed schemas, temporary backfill tooling, repository creation, catalog publication, and the two-team metadata review policy in one tools change.

Current published and verified head: fb50ef1565088e84d7797fa756d199a7873243f2. The reported merge conflict is resolved: merge cd9b321322fe1d4ce01d8de927ae362c43e5a68f integrated main 25f1f4b2ca24505f9e1888542172322a6be6bc4a, preserving the provider-alias fix from #124. Only CHANGELOG needed manual resolution; both sets of notes were retained. No rebase or force-push.

The complete ownership branch from #120 remains included by history-preserving merge. That separate review is closed as superseded, not merged, and its branch is retained. This is the sole tools prerequisite.

Metadata-only generation and source CSV protection

The latest user decision replaces dual-source migration modes:

• Remove bicep_mode, terraform_mode, BicepMode, and TerraformMode. Every generated catalog entry and CSV row requires valid module metadata; no full record is reconstructed or retained from legacy CSV values.
• Fail by default if a source CSV row would disappear. Compare module implementation identities per source file, not row totals or module names alone. An equal-count replacement or another Terraform provider with the same module name cannot hide a removal.
• The baseline is the configured source CSV only, never an existing test-*.csv preview. The same guard works when a later reviewed manifest makes source and destination the same file.
• Add default-false manual workflow input force, plus -Force on Invoke-ModuleCatalog.ps1 and Publish-ModuleCatalog.ps1. Force permits the listed source-row removals only. Scheduled runs cannot select it.
• A forced generation artifact does not authorize publication by itself: publication requires its own explicit force input. Workflow expressions stay in environment values and are passed to generation, offline publication validation, and the publisher.
• The migration report includes sourceCsvRows, csvRowRemovals entries (sourceFile, moduleName, repoURL), and boolean csvRowRemovalsForced. Errors and warnings identify the affected source rows.
• Validate the hash-protected evidence against generated CSVs, then recheck it against the actual unchanged main-branch source CSVs before branch preparation or file writes. Force cannot bypass invalid metadata, incomplete registry/archive/profile evidence, altered output hashes, stale bases, schema/allow-list checks, WhatIf, or publication approvals.
• Preserve matching metadata-backed rows' compatibility fields, existing row order, child CSV aliases/comments including blank cells, and prior Deprecated status. Missing metadata remains reported; an unindexed missing module does not create a row. Existing source CSV proposals or retired records without metadata-backed replacements block by default unless their removal is explicitly permitted.

Simplified authored metadata

• Root owners is a flat array such as ["username", "@Azure/team-name"]. Usernames are bare and teams qualified. Empty arrays are valid, all owners are retained, and case-insensitive duplicates or malformed entries fail validation.
• Children omit owners and inherit them from the family root.
• No tier, authored schemaVersion, or lifecycle/status field. The required versioned $schema identifies the input format. No future tier automation is implemented.
• Generated catalogs retain their own format version, flat owner arrays, and repository-relative module paths: Bicep avm/..., Terraform root ., children modules/{name}. The output schema now requires metadataSource: "metadata".

Deprecation from existing signals

• Bicep DEPRECATED.md marks the containing module and all descendants Deprecated, not parents or siblings.
• Terraform's GitHub repository archived flag marks all modules in that repository Deprecated.
• Collection preserves these signals for offline generation. Missing/malformed archive evidence fails rather than defaulting to active. Prior CSV Deprecated status is preserved for matching metadata-backed entries, not by retaining whole legacy records.
• Catalog generation does not archive repositories, rewrite source, or perform physical retirement steps.

Permanent authoring, creation, and temporary backfill

avm metadata show only reads existing metadata. validate checks a file or explicit -InputObject; initialize creates missing files from supplied values without overwriting existing ones. Normal authoring chains validate root/child metadata, warning on missing files during rollout and failing invalid existing files. The catalog change does not alter that authoring policy. Permanent authoring commands do not depend on CSV conversion.

Repository creation initializes metadata from explicit request inputs before first publication. It retains primary/secondary compatibility parameters, additional usernames, optional qualified team, and resource namespace/type inputs required by the compatibility inventory. Inventory/app publication, authentication preflight, Portal controls and non-writing plans remain. No tier input; inventory CSVs never supply metadata.json values.

CSV conversion, backfill-only source inference, owner snapshots and migration adapters remain outside Avm.Authoring under repository-management/module-metadata. Existing files, branches and reviews are preserved. These temporary scripts can be retired after reconciliation; source CSV collection and row-retention checks are permanent catalog safeguards.

Backfill requires workflow_dispatch and explicit metadata_backfill=true; the runtime rejects non-manual workflow events. The catalog never creates metadata files. Metadata-only Terraform backfill skips legacy/BAMI tenant parsing, identity/state access, repository settings, unrelated formatting, CODEOWNERS and project updates.

Preview publication

• Publish six test-*.csv files beside canonical inputs under docs/static/module-indexes; do not rename, delete or overwrite the originals in this change.
• Keep v1/modules.json, the approved MAR string-array mirror, and the migration report in the public documentation repository.
• JSON retains every owner; compatibility CSVs project the first two usernames and first team without mixing team/user columns. Append CanonicalType, not Tier.
• The trusted manifest separates source and output paths and hashes both input/output bases. Manifest changes require fresh collection. Only public docs are publication destinations; no tools configuration or tier generation.
• Publication creates review changes and never auto-merges or force-pushes. Canonical CSV replacement remains a separate later reviewed change; it does not change the source-row guard or enable force.

Ownership and workflow safeguards

Both ownership templates and current-output validation require the exact final basename rule:

metadata.json @Azure/azure-verified-modules-engineering-owners @Azure/azure-verified-modules-module-owners

Either listed team's eligible approval may satisfy code-owner review; both are not required. General approval counts are unchanged. A metadata owners array does not grant permissions or reviewer eligibility.

Both teams must be visible and have repository write access. The user explicitly approved the default Terraform module-owners team configuration with repositoryPermission: "push" and environmentApproval: false. This is configuration for a future authorized ordinary sync, not a live permission change or part of metadata-only backfill. Existing teams, environment reviewers, engineering-only CODEOWNERS-file protection and the existing App bypass remain unchanged.

Keep the approved BAMI activation gate and separate Variables token, CI's Azure credential check, strict plans, trusted main/environment/app boundaries, and publication checks. Previously removed catalog/general pause variables remain removed; no new gate removal is included here.

Current rollout state and documentation

The Bicep prerequisite has already merged. Azure/bicep-registry-modules#7349 merged at 2026-09-16T16:57:27Z, merge commit d654b23520d7ed224201c9b74b0322df269dfefc, from reviewed head 2fe528df6fe20f849fb79055589677098357680c. Do not treat it as an outstanding merge or alter its historical branch/body. Its 572 flat metadata files preserve all 281 owner entries, nine empty roots, 14 telemetry omissions and unrelated raw values; this catalog change does not alter them. The shared template Azure/Azure-Verified-Modules#2929 is also merged.

The rollout plan records the originally planned merge sequence. The remaining tools-side compatibility and operator gates still apply: verify the actual Bicep Sync state, obtain approval for any pause/enable operation, and do not run incompatible tooling against the merged governance rules. This session has not verified or changed live workflow state or target permissions. No further Bicep merge is required for this prerequisite.

Azure/Azure-Verified-Modules#2936 remains a future-process draft at ef15455cf14a7f514e40d7615e5f78e9bdec2b7a. Its owner independently verified this published executable contract and updated the body/references, including the completed Bicep merge, without further file changes. The prose covers source-only generation, source CSV removal protection, ownership/orphaning/adoption, deprecation, and compatibility policies.

Human approval, compatible synchronization, actual access verification, canonical CSV changeover, pre-source proposal handling and internal Azure-Verified-Modules-Docs runbook alignment remain adoption gates. Proposal approval/issue tracking is separate from catalog visibility; no unsupported metadata home is invented.

Validation

• Latest source-only delta: full offline ./build.ps1 pre-commit passed 1,513 unit tests (eight existing skips) and 493 component tests, with zero errors, failures, invalid cases or unexecuted tests in persisted NUnit reports.
• All 25 publication/workflow tests passed after correcting one test-only empty-array assertion. Coverage includes generation and publication force boundaries, provider-aware/equal-count replacements, unresolved source identities, evidence tampering, actual source versus preview comparison, future canonical overwrite, and unchanged compatibility/status fields.
• Claude Opus 5 reviewed the consolidated implementation, two-team follow-up, and source-only/Force delta with no findings. Source-only review included offline helper probes, evidence round-trips, and publication ordering checks.
• Main merge passed the full local gate plus 33 native integration cases (31 MAPOTF/provider requirements and two metadata readers). Incoming MAPOTF 0.2.1 work is unchanged. Intermediate merge CI passed all 11 jobs after one isolated Windows source-check retry; that unchanged case also passed locally alone and with all 117 metadata component cases. No speculative runtime fix or weakened assertion was made.
• Public documentation reports all seven exact-head hosted checks successful at ef15455; Bicep reported 355 local regressions and all six checks successful at reviewed head 2fe528df before its merge.
• All fresh hosted checks passed for fb50ef1565088e84d7797fa756d199a7873243f2: full CI passed all 11 jobs, including three OS builds and six integration legs; configuration passed both jobs; CodeQL succeeded. All 17 reported checks on this exact head are successful, with no retry required for this head. Local and remote heads match and the worktree is clean.

GitHub reports OPEN/MERGEABLE rather than conflicting, with mergeStateStatus BLOCKED despite every reported check succeeding. The remaining policy reason is unverified; no merge was attempted. No production dispatch/fleet sync, live permission/variable/archival change, workflow enable/disable, new credentials/scopes, or force-push was performed by this implementation session.

[github-code-quality]

Code Coverage Overview

Languages: PowerShell

PowerShell / code-coverage-agent

The overall line coverage in commit fb50ef1 in the jaredfholgate-module... branch is 83%. The line coverage in commit 25f1f4b in the main branch is 89%.

Show a line coverage summary of the most impacted files.

File	main 25f1f4b	jaredfholgate-module... fb50ef1	+/-
src/Avm.Authori...erbRegistry.ps1	100%	100%	0%
src/Avm.Authori...tFreeSource.ps1	0%	0%	0%
src/Avm.Authori...tadataScope.ps1	0%	0%	0%
src/Avm.Authori...aSourcePlan.ps1	0%	0%	0%
src/Avm.Authori...etadataJson.ps1	0%	0%	0%
src/Avm.Authori...aModuleType.ps1	0%	69%	+69%
src/Avm.Authori...dataContent.ps1	0%	71%	+71%
src/Avm.Authori...dataLiteral.ps1	0%	94%	+94%
src/Avm.Authori...etadataJson.ps1	0%	98%	+98%
src/Avm.Authori...tadataIssue.ps1	0%	100%	+100%

---

_{Updated September 16, 2026 19:42 UTC}

[github-actions]

Test results

   12 files  +    6  1 224 suites  +342   43m 6s ⏱️ + 35m 36s
2 076 tests +  522  2 070 ✅ +  516   6 💤 + 6  0 ❌ ±0 
6 432 runs  +1 782  6 389 ✅ +1 764  43 💤 +18  0 ❌ ±0 

Results for commit fb50ef1. ± Comparison against base commit 25f1f4b.

This pull request removes 80 and adds 598 tests. Note that renamed tests count towards both.

# This file is generated automatically")
# This file is generated automatically","# This file is generated automatically")
Bicep CODEOWNERS workflow contract.offers reviewable plan-only dispatch by default with no target or credential overrides
Bicep CODEOWNERS workflow contract.restricts credentials to trusted tools main and requires explicit scheduled enablement
Bicep workflow isolation and trusted input boundary.preserves the original CODEOWNERS job, writer invocation, token, and scopes exactly
Both repository-sync entry points use one existing publication core.preserves the exact legacy return shape for no-change, plan, and apply outcomes(False,"NoChange",False)
Both repository-sync entry points use one existing publication core.preserves the exact legacy return shape for no-change, plan, and apply outcomes(True,"Merged",False)
Both repository-sync entry points use one existing publication core.preserves the exact legacy return shape for no-change, plan, and apply outcomes(True,"Planned",True)
Component: Invoke-AvmPreCommit + Invoke-AvmPrCheck (terraform engine end-to-end).pr-check composes eight steps (sync drift-check first) with the transform engine running a mapotf drift-check
Component: Invoke-AvmPreCommit + Invoke-AvmPrCheck (terraform engine end-to-end).pre-commit composes the five-step terraform chain end-to-end (sync first) via launcher-resolved stubs and the in-module check-convention rules
…

Bicep CODEOWNERS workflow contract.has no Bicep metadata backfill or intermediate approval-file path
Bicep CODEOWNERS workflow contract.offers a strict dry run by default with no target or credential overrides
Bicep CODEOWNERS workflow contract.restricts credentials to trusted tools main without the removed enable variable
Bicep literal metadata reader.decodes Bicep escapes and keeps escaped interpolation literal
Bicep literal metadata reader.reads a multiline literal without its opening newline
Bicep literal metadata reader.reads literals while ignoring comments and decoy strings
Bicep literal metadata reader.rejects a non-literal, missing, duplicate, or interpolated declaration: duplicate
Bicep literal metadata reader.rejects a non-literal, missing, duplicate, or interpolated declaration: expression
Bicep literal metadata reader.rejects a non-literal, missing, duplicate, or interpolated declaration: interpolation
Bicep literal metadata reader.rejects a non-literal, missing, duplicate, or interpolated declaration: invalid escape
…

♻️ This comment has been updated with latest results.