feat(config)!: remove legacy test-suites; fix pytest working-dir resolution

[bhagyapathak]

No description provided.

[Copilot]

🟡 Changes recommended

Schema snapshots and relative pytest path handling remain unresolved, with stale configuration documentation also requiring updates.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Removes legacy [test-suites] support in favor of project-level [tests] and [test-groups] configuration.

Changes:

• Removes legacy suite schema, loading, validation, and execution.
• Updates image test selection, runners, and related tests.
• Updates generated schema and CLI documentation.

File summaries

File	Reviewed changes
schemas/azldev.schema.json	Removes legacy schema definitions. Critical (3 votes): regenerate scenario snapshots.
internal/projectconfig/testsuite.go	Removes legacy suite metadata while retaining shared runner configuration.
internal/projectconfig/testsuite_test.go	Updates test references and removes legacy suite coverage.
internal/projectconfig/tests.go	Updates test path handling. Moderate (3 votes): preserve source-directory provenance for relative pytest paths.
internal/projectconfig/project.go	Removes top-level test-suite configuration and validation.
internal/projectconfig/loader.go	Removes legacy suite loading and merging.
internal/projectconfig/loader_test.go	Updates configuration loading tests.
internal/projectconfig/image.go	Removes legacy image test references.
internal/projectconfig/configfile.go	Removes legacy configuration validation.
internal/projectconfig/configfile_test.go	Removes legacy validation and deprecation-warning coverage.
internal/app/azldev/cmds/image/test.go	Uses new test selectors and runners. Nit (3 votes): update stale configuration documentation and agent skill references.
internal/app/azldev/cmds/image/test_internal_test.go	Updates selector and resolver tests.
internal/app/azldev/cmds/image/list_test.go	Updates image test fixtures.
internal/app/azldev/cmds/image/lisarunner.go	Removes the legacy LISA suite runner.
docs/user/reference/cli/azldev_image_test.md	Updates image-test CLI documentation.

Review details

• Files reviewed: 15/15 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Fix the schema freshness issue and add regression coverage for included-config pytest paths.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

internal/projectconfig/tests.go:99

• This provenance assignment is the behavior that makes relative working-dir values from included config files resolve against the file that defines them, but the added regression test only loads the root config file. Please add a test with a root file including a pytest definition from a different directory; otherwise a wrong loadedCfg.dir propagation could pass the current tests.

	result.dir = referenceDir

• Files reviewed: 22/22 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The cross-cutting migration warrants final human review before approval.

Review details

• Files reviewed: 22/22 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Nan Liu (liunan-ms)]

With --permissive-config, removing the test-suites fields causes both top-level [test-suites] definitions and images.<name>.tests.test-suites references to be ignored as unknown fields. This leaves resolvedTests empty, and returns success without executing anything. Would this be an expected behavior?

[bhagyapathak]

yes, this is expected, it's the designed behavior of --permissive-config

[Copilot]

🟡 Changes recommended

The documented pytest requirements and included-file provenance regression test still need updates.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

docs/user/reference/config/tests.md:98

• install = "none" does not make pytest available from an arbitrary PATH: the runner executes the venv interpreter with -m pytest (internal/app/azldev/cmds/image/pytestrunner.go:81-90), and the existing config comment requires pytest to be in that venv (internal/projectconfig/testsuite.go:90-92). Please say that the venv must already contain pytest and its dependencies.

| `none` (default) | Skips dependency installation entirely. Use when the venv is pre-populated or pytest is otherwise on `PATH`. |

internal/projectconfig/loader_test.go:1414

• The regression this change targets is per-config-file provenance, but this test loads only the root config at /project/azldev.toml; it never defines a test in an included file. As written, it would still pass if mergeTests stopped recording the included file's directory, so add an include under a different directory and assert that its relative working-dir resolves against that directory.

	// ...but it resolves to an absolute path relative to the defining config
	// file's directory at execution time (via recorded provenance), so relative
	// paths in included files remain correct.
	assert.Equal(t, "/project/tests", config.Tests["smoke"].PytestWorkingDir())

• Files reviewed: 22/22 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Add focused coverage for relative pytest working-directory resolution and clarify migration requirements.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

internal/projectconfig/tests.go:97

• WithAbsolutePaths no longer converts any paths; it only clones the maps and records the defining directory in dir. Keeping this name makes the new behavior misleading to callers and future maintainers—rename it to reflect provenance (for example, WithConfigDir/WithReferenceDir) and update its call site.

func (t TestDefinition) WithAbsolutePaths(referenceDir string) TestDefinition {

• Files reviewed: 22/22 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

WithConfigDir must deep-copy nested values or revise its API/documentation before approval.

Review details

Suppressed comments (1)

internal/projectconfig/tests.go:103

• WithConfigDir claims to return a deep copy, but cloneStringAnyMap only clones the outer maps; nested maps and slices (for example lisa.criteria or pytest.test-paths) remain aliased with the original definition. A caller that mutates a nested value through the returned definition can therefore mutate the source config unexpectedly. Either recursively clone the framework values or change the API/documentation to promise only a shallow copy.

// WithConfigDir returns a deep copy of the test definition, recording
// referenceDir (the defining config file's directory) so relative pytest paths
// (e.g. 'working-dir') can be resolved at test-execution time. Unlike the
// 'WithAbsolutePaths' helpers on other config types, this does not rewrite any
// path values: framework-specific subtables are preserved exactly as authored,
// so 'azldev config dump' shows the user-defined values; see
// [TestDefinition.PytestWorkingDir] for the execution-time resolution.
func (t TestDefinition) WithConfigDir(referenceDir string) TestDefinition {
	result := t
	result.dir = referenceDir
	result.Lisa = cloneStringAnyMap(t.Lisa)
	result.Tmt = cloneStringAnyMap(t.Tmt)
	result.Pytest = cloneStringAnyMap(t.Pytest)

• Files reviewed: 22/22 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

WithConfigDir must deep-copy RequiredCapabilities to avoid sharing the source slice.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 23/23 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The broad configuration and runtime changes warrant human review, with one documentation correction remaining.

Review details

Suppressed comments (1)

docs/user/reference/config/tests.md:28

• TestDefinition.Validate requires the matching pytest subtable when type = "pytest" (see internal/projectconfig/tests.go:270-305), so marking this field as not required makes the reference permit configurations that the loader rejects. Please mark it required for pytest tests.

| Pytest | `pytest` | table | No | pytest-specific configuration (see [Pytest fields](#pytest-fields)) |

• Files reviewed: 22/22 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Broad configuration, runtime, schema, and documentation changes warrant final human review.

Review details

• Files reviewed: 22/22 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

Legacy suite references are removed consistently, and pytest provenance resolution is covered by loader and execution-path tests.

Review details

• Files reviewed: 22/22 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Nan Liu (liunan-ms)]

LGTM