Skip to content

OPS-157454: Upload test coverage to Codacy from CI - #47

Merged
davidnmbond merged 1 commit into
mainfrom
ci/codacy-coverage
Sep 27, 2026
Merged

davidnmbond merged 1 commit into
mainfrom
ci/codacy-coverage

Conversation

@davidnmbond

Copy link
Copy Markdown
Contributor

What

Adds a coverage job to CI, on the same push and pull_request triggers as the build and on the pdl-public runner (never a GitHub-hosted one). It builds the test project, runs the Microsoft.Testing.Platform test executable with --coverage (Microsoft.Testing.Extensions.CodeCoverage, settings in coverage.config, modelled on PanoramicData.NugetManagement), uploads coverage.cobertura.xml to Codacy with codacy/codacy-coverage-reporter-action using the CODACY_PROJECT_TOKEN secret, and keeps the report as a 7-day artifact. The upload step is continue-on-error so a Codacy outage cannot turn a passing build red.

Which tests run

CI runs only tests that need no credentials; xunit.runner.json sets failSkips: true, so an unconfigured test would fail rather than skip. No test was deleted and no assertion changed.

Excluded: the existing Category=Integration classes under Integration/ (Account, Analysis, CodingStandards, Coverage, DebugRepositoryAccess, Issues, Organizations, People, Phase2 diagnostics/setup, QuickDiagnostic, Repositories, RepositoryManagement, Security, TestDataManagerExample, Version), which call the live Codacy API with a user-secret token. CI filters with Category!=Integration.

Measured locally in a clean mcr.microsoft.com/dotnet/sdk:10.0 Linux container with no user secrets, running exactly what CI runs: 56.2% line coverage.

Other changes

The test project now sets UseMicrosoftTestingPlatformRunner, without which the xunit v3 executable runs its native runner and rejects --coverage.

Tracked on https://jira.panoramicdata.com/browse/OPS-157454

🤖 Generated with Claude Code

Adds a coverage job to CI, on the pdl-public runner, that runs the offline
tests under Microsoft.Testing.Extensions.CodeCoverage and uploads the
Cobertura report to Codacy.

Excluded: the existing Category=Integration classes under Integration/ (Account, Analysis, CodingStandards, Coverage, DebugRepositoryAccess, Issues, Organizations, People, Phase2 diagnostics/setup, QuickDiagnostic, Repositories, RepositoryManagement, Security, TestDataManagerExample, Version), which call the live Codacy API with a user-secret token. CI filters with Category!=Integration.

Local line coverage (clean Linux container, no user secrets): 56.2%.

The test project now sets UseMicrosoftTestingPlatformRunner, without which the xunit v3 executable runs its native runner and rejects --coverage.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@codacy-production

codacy-production Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Coverage ∅ diff coverage

Metric Results
Coverage variation Report missing for ced585a1
Diff coverage ✅ ∅ diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (ced585a) Report Missing Report Missing Report Missing
Head commit (c446c68) 162 91 56.17%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#47) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

1 Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

@codacy-production codacy-production Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

The PR successfully implements the required coverage job using the Microsoft.Testing.Platform on a hardened runner, aligning with the core security and functional requirements. However, the current configuration contains path mismatches that will likely cause the CI job to fail during the upload and artifact stages. Specifically, the test runner outputs reports to the root directory, while subsequent steps look for them in the project's bin directory.

Additionally, the coverage exclusion patterns are currently Windows-specific (using backslashes), which will fail to correctly exclude directories on the Linux-based runner. While the PR is 'up to standards' according to automated checks, these logic errors in the workflow and configuration should be resolved to ensure the coverage reporting functions correctly.

About this PR

  • There is a systemic path mismatch in the workflow: the 'Run tests' step defines a specific output filename, but the Codacy upload and artifact steps look in a nested 'TestResults' directory that likely won't contain the file.

Test suggestions

  • Execute the test runner with the --coverage and --filter 'Category!=Integration' flags.
  • Apply coverage exclusions for CompilerGeneratedAttribute and Test.dll modules as defined in coverage.config.
  • Upload the generated Cobertura report to Codacy using the project token.
  • Store the Cobertura report as a workflow artifact with a 7-day retention period.

TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback

Comment thread .github/workflows/ci.yml
uses: codacy/codacy-coverage-reporter-action@v1
with:
project-token: ${{ secrets.CODACY_PROJECT_TOKEN }}
coverage-reports: Codacy.Api.Test/bin/Debug/net10.0/TestResults/coverage.cobertura.xml

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 HIGH RISK

The coverage report path is likely incorrect. The 'Run tests' step uses '--coverage-output coverage.cobertura.xml' from the root, so the output will be at 'TestResults/coverage.cobertura.xml' rather than inside the bin directory. Verify the path to ensure the upload step finds the file.

Comment thread coverage.config
</Attributes>
<Sources>
<Exclude>
<Source>.*\\obj\\.*</Source>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM RISK

Suggestion: The path separator in the regex is Windows-specific. Since the job runs on a Linux runner ('pdl-public'), use a cross-platform pattern (e.g., forward slashes or '[/\\]') to ensure the 'obj' directory is correctly excluded.

Suggested fix:

Suggested change
<Source>.*\\obj\\.*</Source>
<Source>.*/obj/.*</Source>

@davidnmbond
davidnmbond merged commit 643c2ee into main Sep 27, 2026
7 checks passed
@davidnmbond
davidnmbond deleted the ci/codacy-coverage branch September 27, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant