Repository navigation
[SYCL][E2E][CI] Migrate E2E tests to shared UR test reporting #23327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: sycl
Are you sure you want to change the base?
Changes from all commits
b371ad9
529861c
8830384
7564b80
eebb84b
32ca47e
2458761
c63e963
823dcbd
7e62dd0
0709e64
fd56839
21033a8
5e9209e
0633b69
57a2a34
8a5f4ea
3129c21
fd878c7
2ecdd0d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -81,7 +81,8 @@ jobs: | |
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| sparse-checkout: | | ||
| devops/ | ||
| devops/actions | ||
| devops/scripts | ||
|
|
||
| # download build artefact | ||
| - name: Download toolchain | ||
|
|
@@ -100,6 +101,7 @@ jobs: | |
| ref: ${{ github.ref || github.sha }} | ||
| testing_mode: full | ||
| target_devices: native_cpu:cpu | ||
| job_name: ${{ github.job }} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can we not do this inside the final workflow/step that actually needs it?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point. I added
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I looked into this further, but |
||
| sycl_compiler: $GITHUB_WORKSPACE/toolchain/bin/dpclang++ | ||
| extra_lit_opts: --param sycl_build_targets="native_cpu" | ||
| extra_cmake_args: -DSYCL_TEST_E2E_TARGETS="native_cpu:cpu" -DSYCL_TEST_E2E_STANDALONE=ON | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,6 +18,10 @@ inputs: | |
| required: false | ||
| sycl_compiler: | ||
| required: false | ||
| job_name: | ||
| description: 'Name of the calling job, used to build a unique artifact name' | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can you explain why we didnt need this until now?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Previously, E2E used its own log reporting and didn't upload log/XML artifacts through the shared
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Small update: I kept |
||
| required: false | ||
| default: '' | ||
|
|
||
|
|
||
| runs: | ||
|
|
@@ -43,6 +47,7 @@ runs: | |
| run: | | ||
| mkdir build-e2e | ||
| tar -I 'zstd' -xf e2e_binaries.tar.zst -C build-e2e | ||
| devops/scripts/validate-e2e-binaries build-e2e | ||
|
|
||
| - name: Extract E2E tests from container image | ||
| if: ${{ inputs.testing_mode == 'run-only' && inputs.binaries_artifact == 'in-container' }} | ||
|
|
@@ -51,6 +56,7 @@ runs: | |
| mkdir build-e2e llvm | ||
| tar -I 'zstd' -xf /sycl-prebuilt/e2e_binaries.tar.zst -C build-e2e | ||
| tar -I 'zstd' -xf /sycl-prebuilt/e2e_sources.tar.zst -C llvm | ||
| devops/scripts/validate-e2e-binaries build-e2e | ||
|
|
||
| - name: Deduce E2E CMake options | ||
| if: inputs.testing_mode != 'run-only' | ||
|
|
@@ -67,32 +73,32 @@ runs: | |
| shell: bash | ||
| run: | | ||
| cmake -GNinja -B./build-e2e -S./llvm/sycl/test-e2e -DCMAKE_CXX_COMPILER="${{ inputs.sycl_compiler || '$(which dpclang++)'}}" -DLLVM_LIT="$PWD/llvm/llvm/utils/lit/lit.py" ${{ steps.cmake_opts.outputs.opts }} | ||
| - name: SYCL End-to-end tests | ||
| id: run_e2e | ||
| continue-on-error: true | ||
|
|
||
| # Composed here, not in ur-test, since these depend on caller-only workflow inputs. | ||
| - name: Compose E2E LIT options | ||
| id: lit_opts | ||
| shell: bash | ||
| env: | ||
| LIT_OPTS: -v --no-progress-bar --show-unsupported --show-pass --show-xfail --max-time 3600 --time-tests --param print_features=True --param test-mode=${{ inputs.testing_mode }} --param sycl_devices=${{ inputs.target_devices }} ${{ inputs.extra_lit_opts }} | ||
| TESTING_MODE: ${{ inputs.testing_mode }} | ||
| TARGET_DEVICES: ${{ inputs.target_devices }} | ||
| USER_LIT_OPTS: ${{ inputs.extra_lit_opts }} | ||
| run: | | ||
| ninja -C build-e2e check-sycl-e2e > e2e.log 2>&1 | ||
| # Two steps below are duplicated between Lin/Win actions, updates must change both | ||
| - name: E2E logs | ||
| if: ${{ always() }} | ||
| shell: bash | ||
| run: | | ||
| echo "::group::Show Full E2E Log" | ||
| cat e2e.log | ||
| echo "::endgroup::" | ||
| - name: Report E2E Failures | ||
| if: steps.run_e2e.outcome != 'success' | ||
| # --timeout kills an individual hung test (e.g. a GPU driver hang) instead of | ||
| # letting it block the whole suite until the job's own timeout-minutes fires. | ||
| echo "opts=--max-time 3600 --timeout 600 --param print_features=True --param test-mode=${TESTING_MODE} --param sycl_devices=${TARGET_DEVICES} ${USER_LIT_OPTS}" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Run E2E tests | ||
| uses: ./devops/actions/run-tests/lit | ||
| with: | ||
| test_type: 'e2e' | ||
| build_dir: 'build-e2e' | ||
| extra_lit_opts: ${{ steps.lit_opts.outputs.opts }} | ||
| job_name: ${{ inputs.job_name }} | ||
|
|
||
| - name: Verify E2E binaries were built | ||
| if: ${{ !cancelled() && inputs.binaries_artifact != '' && inputs.testing_mode != 'run-only' }} | ||
| shell: bash | ||
| # For some reason Github uses the first line from the `run: |` section for | ||
| # the folded entry when displaying instead of this step's name. | ||
| run: | | ||
| # Report E2E Failures | ||
| awk '/^Failed Tests|Unexpectedly Passed Tests|Unresolved tests|Timed Out Tests|Testing Time/{flag=1}/FAILED: CMakeFiles/{flag=0}flag' e2e.log >> $GITHUB_STEP_SUMMARY | ||
| awk '/^Failed Tests|Unexpectedly Passed Tests|Unresolved tests|Timed Out Tests|Testing Time/{flag=1}/FAILED: CMakeFiles/{flag=0}flag' e2e.log | ||
| exit 1 | ||
| run: devops/scripts/validate-e2e-binaries build-e2e | ||
|
|
||
| - name: Pack E2E binaries | ||
| if: ${{ !cancelled() && inputs.binaries_artifact != '' && inputs.testing_mode != 'run-only' }} | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i just checked the ci results for this pr and say the new github step summaries which are really nice!
however i noticed one issue, for example:
IMO we need to print all results here (and similarly for unsuppoted/xfail/etc), I use this often to make sure my test ran
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Glad you like the new summaries! The intention was to have complete, collapsible lists of test names for every status, without truncation. I'll fix the step summary to work that way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done