OPS-157454: Upload test coverage to Codacy from CI - #47
Conversation
Adds a coverage job to CI, on the pdl-public runner, that runs the offline tests under Microsoft.Testing.Extensions.CodeCoverage and uploads the Cobertura report to Codacy. Excluded: the existing Category=Integration classes under Integration/ (Account, Analysis, CodingStandards, Coverage, DebugRepositoryAccess, Issues, Organizations, People, Phase2 diagnostics/setup, QuickDiagnostic, Repositories, RepositoryManagement, Security, TestDataManagerExample, Version), which call the live Codacy API with a user-secret token. CI filters with Category!=Integration. Local line coverage (clean Linux container, no user secrets): 56.2%. The test project now sets UseMicrosoftTestingPlatformRunner, without which the xunit v3 executable runs its native runner and rejects --coverage. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Coverage variation | Report missing for ced585a1 |
| Diff coverage | ✅ ∅ diff coverage |
Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (ced585a) Report Missing Report Missing Report Missing Head commit (c446c68) 162 91 56.17% Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>
Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#47) 0 0 ∅ (not applicable) Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
1 Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.
AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Pull Request Overview
The PR successfully implements the required coverage job using the Microsoft.Testing.Platform on a hardened runner, aligning with the core security and functional requirements. However, the current configuration contains path mismatches that will likely cause the CI job to fail during the upload and artifact stages. Specifically, the test runner outputs reports to the root directory, while subsequent steps look for them in the project's bin directory.
Additionally, the coverage exclusion patterns are currently Windows-specific (using backslashes), which will fail to correctly exclude directories on the Linux-based runner. While the PR is 'up to standards' according to automated checks, these logic errors in the workflow and configuration should be resolved to ensure the coverage reporting functions correctly.
About this PR
- There is a systemic path mismatch in the workflow: the 'Run tests' step defines a specific output filename, but the Codacy upload and artifact steps look in a nested 'TestResults' directory that likely won't contain the file.
Test suggestions
- Execute the test runner with the --coverage and --filter 'Category!=Integration' flags.
- Apply coverage exclusions for CompilerGeneratedAttribute and Test.dll modules as defined in coverage.config.
- Upload the generated Cobertura report to Codacy using the project token.
- Store the Cobertura report as a workflow artifact with a 7-day retention period.
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| uses: codacy/codacy-coverage-reporter-action@v1 | ||
| with: | ||
| project-token: ${{ secrets.CODACY_PROJECT_TOKEN }} | ||
| coverage-reports: Codacy.Api.Test/bin/Debug/net10.0/TestResults/coverage.cobertura.xml |
There was a problem hiding this comment.
🔴 HIGH RISK
The coverage report path is likely incorrect. The 'Run tests' step uses '--coverage-output coverage.cobertura.xml' from the root, so the output will be at 'TestResults/coverage.cobertura.xml' rather than inside the bin directory. Verify the path to ensure the upload step finds the file.
| </Attributes> | ||
| <Sources> | ||
| <Exclude> | ||
| <Source>.*\\obj\\.*</Source> |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Suggestion: The path separator in the regex is Windows-specific. Since the job runs on a Linux runner ('pdl-public'), use a cross-platform pattern (e.g., forward slashes or '[/\\]') to ensure the 'obj' directory is correctly excluded.
Suggested fix:
| <Source>.*\\obj\\.*</Source> | |
| <Source>.*/obj/.*</Source> |
What
Adds a
coveragejob to CI, on the same push and pull_request triggers as the build and on thepdl-publicrunner (never a GitHub-hosted one). It builds the test project, runs the Microsoft.Testing.Platform test executable with--coverage(Microsoft.Testing.Extensions.CodeCoverage, settings incoverage.config, modelled on PanoramicData.NugetManagement), uploadscoverage.cobertura.xmlto Codacy withcodacy/codacy-coverage-reporter-actionusing theCODACY_PROJECT_TOKENsecret, and keeps the report as a 7-day artifact. The upload step iscontinue-on-errorso a Codacy outage cannot turn a passing build red.Which tests run
CI runs only tests that need no credentials;
xunit.runner.jsonsetsfailSkips: true, so an unconfigured test would fail rather than skip. No test was deleted and no assertion changed.Excluded: the existing Category=Integration classes under Integration/ (Account, Analysis, CodingStandards, Coverage, DebugRepositoryAccess, Issues, Organizations, People, Phase2 diagnostics/setup, QuickDiagnostic, Repositories, RepositoryManagement, Security, TestDataManagerExample, Version), which call the live Codacy API with a user-secret token. CI filters with Category!=Integration.
Measured locally in a clean
mcr.microsoft.com/dotnet/sdk:10.0Linux container with no user secrets, running exactly what CI runs: 56.2% line coverage.Other changes
The test project now sets UseMicrosoftTestingPlatformRunner, without which the xunit v3 executable runs its native runner and rejects --coverage.
Tracked on https://jira.panoramicdata.com/browse/OPS-157454
🤖 Generated with Claude Code