[Sanjay Singh (san360)]

Description

Add real packaged-VSIX testing alongside the existing unit and browser suites, with inspectable synthetic input data and screenshots of successful UI journeys.

• Install the actual VSIX into disposable VS Code installations using ExTester. Isolate home/AppData/Copilot directories, extension storage, profiles, and caches from developer data.
• Check in fully synthetic TypeScript, Python, and Go project definitions in tests/vscode/data/projects.json. Generate raw chat sessions and edit timelines: 3 projects, 21 sessions, 63 requests, 315 added lines and 294 net lines over 7 days. Reload adds one session, producing 22 sessions, 66 requests and 330 added lines. No personal logs were read or copied; input integrity is asserted.
• Cover activation, real parser/analysis workers, Dashboard metrics, overlapping Timeline sessions, nonzero Output by project, Patterns Work Hours/Projects, repeated reloads, reopening, and empty data. Discover suite files automatically and reject focused, skipped, empty, or environment-filtered desktop runs.
• Capture 9 milestone screenshots, wait for charts to finish rendering, and publish an offline HTML gallery with captions, suite outcomes, input counts, and the VSIX hash. Include source definitions, 45 initial raw input files, and 2 planned reload files in the diagnostic artifact. Preserve automatic failure screenshots separately.
• Run browser and desktop tests in PR/release workflows through a shared action, then publish the exact tested VSIX. Run the minimum/latest compatibility matrix on Linux, Windows, and macOS for desktop-harness PR changes, weekly, and on manual dispatch. Document local execution, isolation, dataset maintenance, and artifact viewing in CONTRIBUTING.md.

The review fixes handle ExTester's precise wrapped stale-outer-frame error within the original deadline, prevent inherited MOCHA_GREP/MOCHA_INVERT from silently selecting a subset, and avoid clicking while the replacement dashboard is still laying out.

Validation on the current revision:

• Main GitHub-hosted Linux CI: passed, including browser tests, all 7 desktop smoke tests, and VSIX upload.
• Hosted compatibility matrix: all 6 jobs passed: minimum and latest stable VS Code on Linux, Windows, and macOS.
• Download the screenshots and synthetic dataset. Extract the artifact and open vscode/<run>/gallery.html. Each suite also includes dataset.json, fixtures/initial, fixtures/reload, and individual PNGs. The empty suite intentionally has no session inputs.

Command	Result
npm run check	Passed locally on Node 22.23.3: 86 test files, 1,693 tests passed and 5 existing skips; also passed in hosted Linux CI. SQLite CLI was available for the existing database tests.
npm run test:e2e with CI=true	Passed in hosted Linux CI: 124 tests passed and 7 existing feature-gated skips.
npm run test:vscode -- --vsix ai-engineer-coach-0.1.0.vsix	Passed: all 7 desktop tests on the manifest minimum VS Code 1.125.0. Also verified locally with restrictive MOCHA_GREP and MOCHA_INVERT values inherited by the parent; the complete suite still ran.
npm run test:vscode -- --vsix ai-engineer-coach-0.1.0.vsix --vscode-version latest	Passed in the hosted compatibility matrix on all three platforms.
npm run check-size	Passed; extension bundle 1.63 MB and VSIX approximately 3.81 MB, within their budgets.
go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 .github\workflows\ci.yml .github\workflows\release.yml .github\workflows\vscode-compatibility.yml	Passed.

Try it locally: with Node 22, run npm ci then npm run test:vscode. It builds the VSIX and runs the isolated synthetic-data suite. To use an existing artifact without rebuilding, pass -- --vsix ai-engineer-coach-0.1.0.vsix. For headless Linux, first run npx playwright install-deps chromium, then prefix the desktop command with xvfb-run -a. Local galleries and raw fixture copies are under test-results/vscode/<run>.

Related Issues

Resolves #208.

Checklist

• npm run check passes (typecheck + lint + spellcheck + knip + tests)
• Changes are covered by tests (if applicable)
• Documentation updated (if applicable)

[github-actions]

Dependency Review

The following issues were found:

• ✅ 0 vulnerable package(s)
• ✅ 0 package(s) with incompatible licenses
• ✅ 0 package(s) with invalid SPDX license definitions
• ✅ 0 package(s) with unknown licenses.
• ⚠️ 20 packages with OpenSSF Scorecard issues.

View full job summary

[Copilot]

🔵 Needs a closer look

The new release-blocking desktop harness needs hosted Linux and macOS validation beyond the reported Windows results.

No open issues 🎉

What changed in this PR

Adds packaged-extension smoke tests inside real VS Code, complementing the existing unit and browser suites and addressing #208’s release-testing need.

Changes:

• Adds isolated synthetic fixtures and desktop coverage for activation, workers, navigation, reloads, reopening, and empty data.
• Integrates browser and desktop tests into CI and release gates, preserving the tested VSIX and failure diagnostics.
• Adds cross-platform compatibility runs, contributor guidance, and desktop-test integration with existing development checks.

File	Description
vitest.config.mts	Includes desktop fixture unit tests.
tsconfig.json	Type-checks desktop test sources.
tests/​vscode/​tsconfig.build.json	Compiles the desktop harness separately.
tests/​vscode/​suite/​populated.spec.ts	Tests real session analysis and dashboard interactions.
tests/​vscode/​suite/​empty.spec.ts	Tests isolated no-data behavior.
tests/​vscode/​run.ts	Orchestrates VSIX installation, scenarios, diagnostics, and hashing.
tests/​vscode/​fixtures.ts	Creates isolated fixtures and verifies input integrity.
tests/​vscode/​fixtures.test.ts	Tests fixture parsing, isolation, and runner helpers.
tests/​vscode/​dashboard.ts	Provides reusable desktop dashboard interactions.
playwright.config.ts	Adds failure diagnostics and tighter CI execution settings.
package.json	Adds desktop test commands and dependencies.
package-lock.json	Locks desktop testing dependencies and transitive updates.
knip.json	Registers desktop test sources and entry points.
eslint.config.mjs	Extends lint coverage to desktop tests.
CONTRIBUTING.md	Documents execution, isolation, and regression-test maintenance.
.vscodeignore	Excludes desktop runtime files and test artifacts from packaging.
.github/​workflows/​vscode-compatibility.yml	Tests minimum and latest VS Code across three operating systems.
.github/​workflows/​release.yml	Gates release publication on checks and E2E tests.
.github/​workflows/​ci.yml	Tests and uploads the exact packaged VSIX.
.github/​actions/​e2e/​action.yml	Shares E2E setup, execution, caching, and diagnostic uploads.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The release-blocking harness needs its frame-retry issue resolved and human confirmation of GitHub-hosted Linux and macOS execution.

1 open issue

• Retry stale outer iframe errors during frame switching · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The release-blocking desktop gate needs environment hardening and confirmed GitHub-hosted Linux and macOS execution.

2 open issues

• Prevent inherited Mocha grep filters from skipping tests · New
• Retry stale outer iframe errors during frame switching

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The new mandatory desktop release gate still needs GitHub-hosted Linux and macOS validation and final human review.

2 open issues

• Prevent inherited Mocha grep filters from skipping tests
• Retry stale outer iframe errors during frame switching

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The release-blocking desktop harness needs human confirmation of Linux and macOS execution results, which remain unverified in the supplied context.

No open issues 🎉

2 resolved since last review

• Prevent inherited Mocha grep filters from skipping tests
• Retry stale outer iframe errors during frame switching

_{Review effort: Balanced}