Skip to content

Publish test and code-quality results as CI check annotations #3141

Description

@AlexSkrypnyk

Summary

Test runs already emit machine-readable result files, and CircleCI consumes them to show a test summary. On GitHub Actions they are not surfaced at all: a failed run means opening the job log and scrolling back to find which assertion failed. Code-quality tools emit no machine-readable output at all, so a coding-standards failure is always a log-reading exercise on both providers.

Details

phpunit.xml already logs JUnit output to .logs/test_results/phpunit/phpunit.xml, and .circleci/config.yml has two store_test_results steps that consume it. .github/workflows/build-test-deploy.yml copies .logs out of the container and extracts the coverage percentage, but never publishes the test results themselves, so the richest failure information the pipeline produces is discarded.

The quality tools are invoked bare in .ahoy.yml - vendor/bin/phpcs, vendor/bin/phpstan, vendor/bin/rector --dry-run, vendor/bin/twig-cs-fixer lint - with no report format, so there is nothing to publish even if a publisher existed. Each of these tools can emit a standard report format that maps a finding to a file and a line.

Publishing the results as check annotations puts each failure against the offending file and line in the pull request diff. Annotations are preferable to a comment here, since the coverage summary already occupies the pull request comment slot and test failures are better read next to the code.

Acceptance criteria

  • Test results are published on GitHub Actions as check annotations, appearing against the failing file and line in the pull request.
  • Results are published whether the test step passed or failed, so a failed run still annotates.
  • The code-quality tools write machine-readable reports into the existing logs directory alongside the test results, and those findings annotate too.
  • Report writing does not change the exit codes or the human-readable output of the existing ahoy lint commands.
  • An existing off-the-shelf action is used for publishing rather than a bespoke script.
  • The steps are fenced per tool, so a project that deselected a tool does not receive its publishing step.
  • CircleCI keeps its current behaviour, extended to the new quality reports where the provider supports it.
  • Test result files remain available as build artifacts.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A1Board worker 1ConfirmedThe issue was triaged and confirmed for development

    Type

    No type

    Projects

    • Status
      To do for current release

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions