Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
b371ad9
Apply UR test summary/stats/artifacts tooling to E2E tests
kekaczma Sep 30, 2026
529861c
Install defusedxml on demand in run-tests/ur action
kekaczma Sep 30, 2026
8830384
Include devops/scripts in Windows E2E sparse checkout
kekaczma Sep 30, 2026
7564b80
Include devops/scripts in build-job sparse checkouts
kekaczma Oct 6, 2026
eebb84b
Install ur-test Python deps into workspace dir, not $HOME
kekaczma Oct 6, 2026
32ca47e
Avoid writing to $GITHUB_ENV, use step output instead
kekaczma Oct 6, 2026
2458761
Add per-test --timeout to E2E lit invocations
kekaczma Oct 6, 2026
c63e963
Fail fast when E2E binaries artifact is empty/broken
kekaczma Oct 6, 2026
823dcbd
Fix Slowest Tests/Test Times header matching for lit's count suffix
kekaczma Oct 6, 2026
7e62dd0
Fix zizmor code-injection-via-template-expansion finding
kekaczma Oct 6, 2026
0709e64
Propagate PYTHONPATH to all ur-test invoking steps
kekaczma Oct 6, 2026
fd56839
Fix darker formatting in test_result_reconciliation.py
kekaczma Oct 6, 2026
21033a8
Move ur-test Python dependency management to a dedicated setup action
kekaczma Oct 6, 2026
5e9209e
idempotent provisioning, clearer description
kekaczma Oct 6, 2026
0633b69
Fix PYTHONPATH propagation gap and missing E2E artifact names
kekaczma Oct 7, 2026
57a2a34
Fix Windows XML output path being mangled by shlex in LIT_OPTS
kekaczma Oct 7, 2026
8a5f4ea
Skip redundant venv when ur-test deps are already importable
kekaczma Oct 8, 2026
3129c21
Never truncate test name lists in summaries/step summaries
kekaczma Oct 23, 2026
fd878c7
Merge remote-tracking branch 'origin/sycl' into e2e-test-stats
kekaczma Oct 23, 2026
2ecdd0d
Rename run-tests/ur to run-tests/lit, extract binary validation script
kekaczma Oct 26, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/sycl-linux-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,7 @@ jobs:
with:
sparse-checkout: |
devops/actions
devops/scripts

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i just checked the ci results for this pr and say the new github step summaries which are really nice!

however i noticed one issue, for example:

Passed (1469)
<...> // removed by me
      ... and 1369 more

IMO we need to print all results here (and similarly for unsuppoted/xfail/etc), I use this often to make sure my test ran

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad you like the new summaries! The intention was to have complete, collapsible lists of test names for every status, without truncation. I'll fix the step summary to work that way.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

# Cleanup will be run after all actions are completed.
- name: Register cleanup after job is finished
uses: ./devops/actions/cleanup
Expand Down Expand Up @@ -332,6 +333,7 @@ jobs:
testing_mode: build-only
target_devices: all
binaries_artifact: ${{ inputs.e2e_binaries_artifact }}
job_name: ${{ inputs.e2e_binaries_artifact }}
sycl_compiler: $GITHUB_WORKSPACE/toolchain/bin/dpclang++
extra_lit_opts: --param sycl_build_targets="spir;nvidia" --param gpu_aot_target=${{ env.GPU_AOT_TARGET }}

Expand All @@ -347,6 +349,7 @@ jobs:
testing_mode: build-only
target_devices: all
binaries_artifact: ${{ inputs.e2e_binaries_new_offload_model_artifact }}
job_name: ${{ inputs.e2e_binaries_new_offload_model_artifact }}
sycl_compiler: $GITHUB_WORKSPACE/toolchain/bin/dpclang++
extra_lit_opts: --param sycl_build_targets="spir;nvidia" --param enable_new_offload_model=True --param gpu_aot_target=${{ env.GPU_AOT_TARGET }}

Expand All @@ -358,6 +361,7 @@ jobs:
testing_mode: build-only
target_devices: all
binaries_artifact: ${{ inputs.e2e_binaries_spirv_backend_artifact }}
job_name: ${{ inputs.e2e_binaries_spirv_backend_artifact }}
sycl_compiler: $GITHUB_WORKSPACE/toolchain/bin/dpclang++
extra_lit_opts: --param spirv-backend=True --param gpu_aot_target=${{ env.GPU_AOT_TARGET }}

Expand All @@ -369,5 +373,6 @@ jobs:
testing_mode: build-only
target_devices: all
binaries_artifact: ${{ inputs.e2e_binaries_preview_artifact }}
job_name: ${{ inputs.e2e_binaries_preview_artifact }}
sycl_compiler: $GITHUB_WORKSPACE/toolchain/bin/dpclang++
extra_lit_opts: --param test-preview-mode=True --param gpu_aot_target=${{ env.GPU_AOT_TARGET }}
4 changes: 3 additions & 1 deletion .github/workflows/sycl-linux-precommit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,8 @@ jobs:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
sparse-checkout: |
devops/
devops/actions
devops/scripts

# download build artefact
- name: Download toolchain
Expand All @@ -100,6 +101,7 @@ jobs:
ref: ${{ github.ref || github.sha }}
testing_mode: full
target_devices: native_cpu:cpu
job_name: ${{ github.job }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we not do this inside the final workflow/step that actually needs it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I added job_name to generate unique names for the uploaded test log/XML artifacts, but we can derive the job ID from github.job inside the E2E action instead. I'll simplify this and keep an explicit name only where needed to distinguish matrix configurations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into this further, but github.job alone isn't sufficient because it has the same value for all matrix configurations. We still need to pass an explicit name to avoid artifact collisions. I've moved the artifact naming logic into the shared run-tests/lit action, but kept job_name as an explicit input.

sycl_compiler: $GITHUB_WORKSPACE/toolchain/bin/dpclang++
extra_lit_opts: --param sycl_build_targets="native_cpu"
extra_cmake_args: -DSYCL_TEST_E2E_TARGETS="native_cpu:cpu" -DSYCL_TEST_E2E_STANDALONE=ON
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/sycl-linux-run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,8 @@ jobs:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
sparse-checkout: |
devops
devops/actions
devops/scripts
sycl/cts_exclude_filter
- name: Register cleanup after job is finished
uses: ./devops/actions/cleanup
Expand Down Expand Up @@ -366,6 +367,7 @@ jobs:
target_devices: ${{ inputs.target_devices }}
extra_lit_opts: ${{ inputs.extra_lit_opts }}
retention-days: ${{ inputs.retention-days }}
job_name: ${{ inputs.name }}

- name: Run SYCL CTS Tests
if: inputs.tests_selector == 'cts'
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/sycl-prebuilt-e2e-container.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ jobs:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
sparse-checkout: |
devops
devops/actions
devops/scripts
- run: |
mkdir toolchain
tar -I 'zstd' -xf /sycl-prebuilt/toolchain.tar.zst -C toolchain
Expand All @@ -116,3 +117,4 @@ jobs:
with:
testing_mode: run-only
target_devices: level_zero:gpu
job_name: ${{ github.job }}
3 changes: 3 additions & 0 deletions .github/workflows/sycl-windows-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ jobs:
with:
sparse-checkout: |
devops/actions
devops/scripts
- uses: TheMrMilchmann/setup-msvc-dev@368ef7d1ee4d1171b31d4a7f67f4d954f903f5a9 # v4.1.0
with:
arch: x64
Expand Down Expand Up @@ -262,6 +263,7 @@ jobs:
testing_mode: build-only
target_devices: all
binaries_artifact: ${{ inputs.e2e_binaries_artifact }}
job_name: ${{ inputs.e2e_binaries_artifact }}
extra_lit_opts: --param sycl_build_targets="spir" --param gpu_aot_target=${{ env.GPU_AOT_TARGET }}
cxx: ${{ inputs.cxx }}

Expand All @@ -277,6 +279,7 @@ jobs:
testing_mode: build-only
target_devices: all
binaries_artifact: ${{ inputs.e2e_binaries_new_offload_model_artifact }}
job_name: ${{ inputs.e2e_binaries_new_offload_model_artifact }}
extra_lit_opts: --param sycl_build_targets="spir" --param enable_new_offload_model=True --param gpu_aot_target=${{ env.GPU_AOT_TARGET }}
cxx: ${{ inputs.cxx }}

Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/sycl-windows-run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ jobs:
with:
sparse-checkout: |
devops/actions
devops/scripts
sycl/cts_exclude_filter
- uses: TheMrMilchmann/setup-msvc-dev@368ef7d1ee4d1171b31d4a7f67f4d954f903f5a9 # v4.1.0
with:
Expand Down Expand Up @@ -152,6 +153,7 @@ jobs:
extra_lit_opts: ${{ inputs.extra_lit_opts }}
retention-days: ${{ inputs.artifact_retention_days }}
cxx: ${{ inputs.cxx }}
job_name: ${{ inputs.name }}

- name: Run SYCL CTS Tests
if: inputs.tests_selector == 'cts'
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ur-build-hw.yml
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ jobs:
env:
# Enable Level Zero loader debug trace for L0 adapters only
ZE_ENABLE_LOADER_DEBUG_TRACE: ${{ startsWith(matrix.adapter.name, 'L0') && '1' || '' }}
uses: ./devops/actions/run-tests/ur
uses: ./devops/actions/run-tests/lit
with:
test_type: 'adapter-specific'
build_dir: build
Expand All @@ -194,7 +194,7 @@ jobs:
env:
# Enable Level Zero loader debug trace for L0 adapters only
ZE_ENABLE_LOADER_DEBUG_TRACE: ${{ startsWith(matrix.adapter.name, 'L0') && '1' || '' }}
uses: ./devops/actions/run-tests/ur
uses: ./devops/actions/run-tests/lit
with:
test_type: 'conformance'
build_dir: build
Expand Down
50 changes: 28 additions & 22 deletions devops/actions/run-tests/linux/e2e/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ inputs:
required: false
sycl_compiler:
required: false
job_name:
description: 'Name of the calling job, used to build a unique artifact name'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain why we didnt need this until now?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, E2E used its own log reporting and didn't upload log/XML artifacts through the shared run-tests/ur action. With the migration, we need unique artifact names to avoid collisions between E2E jobs. However, as you pointed out in the other comment, I can derive the job ID from the GitHub context instead of passing it explicitly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small update: I kept job_name after looking into matrix job naming, as explained in the other thread.

required: false
default: ''


runs:
Expand All @@ -43,6 +47,7 @@ runs:
run: |
mkdir build-e2e
tar -I 'zstd' -xf e2e_binaries.tar.zst -C build-e2e
devops/scripts/validate-e2e-binaries build-e2e

- name: Extract E2E tests from container image
if: ${{ inputs.testing_mode == 'run-only' && inputs.binaries_artifact == 'in-container' }}
Expand All @@ -51,6 +56,7 @@ runs:
mkdir build-e2e llvm
tar -I 'zstd' -xf /sycl-prebuilt/e2e_binaries.tar.zst -C build-e2e
tar -I 'zstd' -xf /sycl-prebuilt/e2e_sources.tar.zst -C llvm
devops/scripts/validate-e2e-binaries build-e2e

- name: Deduce E2E CMake options
if: inputs.testing_mode != 'run-only'
Expand All @@ -67,32 +73,32 @@ runs:
shell: bash
run: |
cmake -GNinja -B./build-e2e -S./llvm/sycl/test-e2e -DCMAKE_CXX_COMPILER="${{ inputs.sycl_compiler || '$(which dpclang++)'}}" -DLLVM_LIT="$PWD/llvm/llvm/utils/lit/lit.py" ${{ steps.cmake_opts.outputs.opts }}
- name: SYCL End-to-end tests
id: run_e2e
continue-on-error: true

# Composed here, not in ur-test, since these depend on caller-only workflow inputs.
- name: Compose E2E LIT options
id: lit_opts
shell: bash
env:
LIT_OPTS: -v --no-progress-bar --show-unsupported --show-pass --show-xfail --max-time 3600 --time-tests --param print_features=True --param test-mode=${{ inputs.testing_mode }} --param sycl_devices=${{ inputs.target_devices }} ${{ inputs.extra_lit_opts }}
TESTING_MODE: ${{ inputs.testing_mode }}
TARGET_DEVICES: ${{ inputs.target_devices }}
USER_LIT_OPTS: ${{ inputs.extra_lit_opts }}
run: |
ninja -C build-e2e check-sycl-e2e > e2e.log 2>&1
# Two steps below are duplicated between Lin/Win actions, updates must change both
- name: E2E logs
if: ${{ always() }}
shell: bash
run: |
echo "::group::Show Full E2E Log"
cat e2e.log
echo "::endgroup::"
- name: Report E2E Failures
if: steps.run_e2e.outcome != 'success'
# --timeout kills an individual hung test (e.g. a GPU driver hang) instead of
# letting it block the whole suite until the job's own timeout-minutes fires.
echo "opts=--max-time 3600 --timeout 600 --param print_features=True --param test-mode=${TESTING_MODE} --param sycl_devices=${TARGET_DEVICES} ${USER_LIT_OPTS}" >> "$GITHUB_OUTPUT"

- name: Run E2E tests
uses: ./devops/actions/run-tests/lit
with:
test_type: 'e2e'
build_dir: 'build-e2e'
extra_lit_opts: ${{ steps.lit_opts.outputs.opts }}
job_name: ${{ inputs.job_name }}

- name: Verify E2E binaries were built
if: ${{ !cancelled() && inputs.binaries_artifact != '' && inputs.testing_mode != 'run-only' }}
shell: bash
# For some reason Github uses the first line from the `run: |` section for
# the folded entry when displaying instead of this step's name.
run: |
# Report E2E Failures
awk '/^Failed Tests|Unexpectedly Passed Tests|Unresolved tests|Timed Out Tests|Testing Time/{flag=1}/FAILED: CMakeFiles/{flag=0}flag' e2e.log >> $GITHUB_STEP_SUMMARY
awk '/^Failed Tests|Unexpectedly Passed Tests|Unresolved tests|Timed Out Tests|Testing Time/{flag=1}/FAILED: CMakeFiles/{flag=0}flag' e2e.log
exit 1
run: devops/scripts/validate-e2e-binaries build-e2e

- name: Pack E2E binaries
if: ${{ !cancelled() && inputs.binaries_artifact != '' && inputs.testing_mode != 'run-only' }}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,35 +1,69 @@
name: 'Run UR Tests'
name: 'Run LIT Tests'

inputs:
test_type:
description: 'Type of tests to run (adapter-specific or conformance)'
description: 'Type of tests to run (adapter-specific, conformance or e2e)'
required: true
build_dir:
required: false
default: 'build'
artifact_name:
description: 'Name for uploaded test artifacts (logs, XML)'
description: 'Name for uploaded test artifacts (logs, XML). If empty and job_name is set, a name is derived from test_type and job_name.'
required: false
default: ''
job_name:
description: 'Human-readable job identifier used to derive artifact_name when it is not given explicitly.'
required: false
default: ''
extra_lit_opts:
description: 'Extra options appended to LIT_OPTS (e.g. E2E --param/--max-time flags)'
required: false
default: ''

runs:
using: "composite"
steps:
- name: Set up ur-test
id: ur_test
uses: ./devops/actions/setup-ur-test

- name: Resolve artifact name
id: artifact
shell: bash
env:
ARTIFACT_NAME: ${{ inputs.artifact_name }}
JOB_NAME: ${{ inputs.job_name }}
TEST_TYPE: ${{ inputs.test_type }}
run: |
name="$ARTIFACT_NAME"
if [ -z "$name" ] && [ -n "$JOB_NAME" ]; then
safe_name=$(echo "$JOB_NAME" | tr -c '[:alnum:]._-' '-' | sed 's/-\+/-/g; s/^-//; s/-$//')
name="${TEST_TYPE}-${safe_name}"
fi
echo "name=$name" >> "$GITHUB_OUTPUT"

- name: Run ${{ inputs.test_type }} tests
id: run_tests
continue-on-error: true
shell: bash
env:
BUILD_DIR: ${{ inputs.build_dir }}
TEST_TYPE: ${{ inputs.test_type }}
EXTRA_LIT_OPTS: ${{ inputs.extra_lit_opts }}
PY: ${{ steps.ur_test.outputs.python }}
# Needed by lit (spawned via cmake/ninja's system Python) to see psutil,
# and by ur-test itself under the --target fallback; set on every step below too.
PYTHONPATH: ${{ steps.ur_test.outputs['site-packages'] }}
run: |
devops/scripts/ur-test run "$TEST_TYPE" "$BUILD_DIR" "$GITHUB_WORKSPACE"
"$PY" devops/scripts/ur-test run "$TEST_TYPE" "$BUILD_DIR" "$GITHUB_WORKSPACE"

- name: Filter test log
if: ${{ always() && steps.run_tests.outputs['skip-artifacts'] != '1' }}
shell: bash
env:
LOG_FILE: ${{ steps.run_tests.outputs['log-file'] }}
PY: ${{ steps.ur_test.outputs.python }}
PYTHONPATH: ${{ steps.ur_test.outputs['site-packages'] }}
run: |
if [ -z "$LOG_FILE" ]; then
echo "::warning::Test log path not provided by runner; skipping log filtering."
Expand All @@ -42,21 +76,23 @@ runs:
fi

echo "::group::Test Log"
devops/scripts/ur-test filter-log "$LOG_FILE" || { echo "::error::Filter failed"; head -1000 "$LOG_FILE"; }
"$PY" devops/scripts/ur-test filter-log "$LOG_FILE" || { echo "::error::Filter failed"; head -1000 "$LOG_FILE"; }
echo "::endgroup::"

- name: Report test failures
if: steps.run_tests.outcome != 'success'
shell: bash
env:
LOG_FILE: ${{ steps.run_tests.outputs['log-file'] }}
PY: ${{ steps.ur_test.outputs.python }}
PYTHONPATH: ${{ steps.ur_test.outputs['site-packages'] }}
run: |
if [ -z "$LOG_FILE" ] || [ ! -f "$LOG_FILE" ]; then
echo "::warning::Test execution failed, but no test log is available."
exit 1
fi

devops/scripts/ur-test extract-errors "$LOG_FILE"
"$PY" devops/scripts/ur-test extract-errors "$LOG_FILE"
exit 1

- name: Test summary
Expand All @@ -65,9 +101,11 @@ runs:
env:
LOG_FILE: ${{ steps.run_tests.outputs['log-file'] }}
XML_FILE: ${{ steps.run_tests.outputs['xml-file'] }}
ARTIFACT_NAME: ${{ inputs.artifact_name }}
ARTIFACT_NAME: ${{ steps.artifact.outputs.name }}
TEST_TYPE: ${{ inputs.test_type }}
WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
PY: ${{ steps.ur_test.outputs.python }}
PYTHONPATH: ${{ steps.ur_test.outputs['site-packages'] }}
run: |
if [ -z "$LOG_FILE" ]; then
echo "::warning::Test log path not provided by runner; skipping summary."
Expand All @@ -84,7 +122,7 @@ runs:
exit 0
fi

devops/scripts/ur-test summary "$LOG_FILE" "${XML_FILE:-}"
"$PY" devops/scripts/ur-test summary "$LOG_FILE" "${XML_FILE:-}"

if [ -n "$ARTIFACT_NAME" ]; then
echo ""
Expand All @@ -96,11 +134,11 @@ runs:

if [ -n "$GITHUB_STEP_SUMMARY" ]; then
echo "### ${TEST_TYPE} tests" >> "$GITHUB_STEP_SUMMARY"
devops/scripts/ur-test step-summary "$LOG_FILE" "${XML_FILE:-}" >> "$GITHUB_STEP_SUMMARY"
"$PY" devops/scripts/ur-test step-summary "$LOG_FILE" "${XML_FILE:-}" >> "$GITHUB_STEP_SUMMARY"
fi

- name: Copy artifacts
if: ${{ always() && inputs.artifact_name != '' && (inputs.test_type != 'adapter-specific' || steps.run_tests.outputs['skip-artifacts'] != '1') }}
if: ${{ always() && steps.artifact.outputs.name != '' && (inputs.test_type != 'adapter-specific' || steps.run_tests.outputs['skip-artifacts'] != '1') }}
shell: bash
env:
LOG_FILE: ${{ steps.run_tests.outputs['log-file'] }}
Expand All @@ -120,10 +158,10 @@ runs:
fi

- name: Upload test artifacts
if: ${{ always() && inputs.artifact_name != '' && (inputs.test_type != 'adapter-specific' || steps.run_tests.outputs['skip-artifacts'] != '1') }}
if: ${{ always() && steps.artifact.outputs.name != '' && (inputs.test_type != 'adapter-specific' || steps.run_tests.outputs['skip-artifacts'] != '1') }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ inputs.artifact_name }}
name: ${{ steps.artifact.outputs.name }}
path: test_artifacts/
retention-days: 7
if-no-files-found: warn
Loading
Loading