From 304874694e5dc36bd61a6a7a92fb13fb4e41813b Mon Sep 17 00:00:00 2001
From: Shengliang Xu <shengliangx@nvidia.com>
Date: Wed, 21 Oct 2026 22:06:23 +0000
Subject: [PATCH 1/7] Run the small examples of a lane in one job

Every example had its own job, and each job spent a container start (2.6-4.2 min),
checkout and base install before its first test, while llm_distill, llm_sparsity,
specdec_bench and gpt-oss run only 1-3 min of tests each.

The runner now accepts several examples ("llm_eval gpt-oss") and runs them one
after the other, each in its own step (new composite action run-example: its
requirements, its tests, its coverage upload under the same examples-<name> flag
as before). A failing example does not stop the next one, as when each had its
own job; the job fails at the end. Requirements are installed right before each
example's tests, so an example that upgrades packages does not change the
environment of the ones before it. The allow-failure list (repo variable) is now
read per example inside the runner instead of once per job by the caller.

Grouped: llm_distill, llm_sparsity and specdec_bench in the torch lane; llm_eval
and gpt-oss in the trtllm lane (gpt-oss upgrades datasets and trl, so it runs
last). Kept apart: llm_qat, speculative_decoding and hf_ptq (long jobs), and
torch_trt, whose requirements downgrade torch 2.13 to 2.12.1 and torchvision to
0.27, which torch_onnx must not run on. Lane gating, images, extras, timeouts and
the required-check aggregation are unchanged.

Testing: actionlint clean; PyYAML comparison of the old and new jobs (13 -> 10
example jobs, every example still run once); stub test of the new shell step;
pre-commit on the changed files. Not run on GitHub runners.

Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
---
 .github/actions/run-example/action.yml      | 67 ++++++++++++++
 .github/workflows/_example_tests_runner.yml | 97 +++++++++++----------
 .github/workflows/example_tests.yml         | 27 ++++--
 3 files changed, 134 insertions(+), 57 deletions(-)
 create mode 100644 .github/actions/run-example/action.yml

diff --git a/.github/actions/run-example/action.yml b/.github/actions/run-example/action.yml
new file mode 100644
index 00000000000..736dbf91912
--- /dev/null
+++ b/.github/actions/run-example/action.yml
@@ -0,0 +1,67 @@
+name: Run example tests
+description: Install the requirements of one example, run its tests with coverage and upload the coverage under the flag `examples-<example>`. Used by _example_tests_runner.yml,
+  once per example, so that examples sharing a job still get their own step, requirements and Codecov flag.
+
+inputs:
+  example:
+    description: Example directory name, e.g. `llm_distill` (tests/examples/<example>, examples/<example>).
+    required: true
+  allow-failure-examples:
+    description: Comma-separated example names whose test failures are reported as a warning instead of failing the step.
+    required: false
+    default: ""
+  codecov-token:
+    description: Codecov upload token (composite actions cannot read secrets).
+    required: false
+    default: ""
+
+runs:
+  using: composite
+  steps:
+    # Installed right before this example's tests, not up front for all examples of the job: an example
+    # that pins or upgrades packages (e.g. gpt-oss upgrades datasets, torch_trt downgrades torch) must not
+    # change the environment of the examples that run before it.
+    - name: Install requirements of ${{ inputs.example }}
+      shell: bash
+      run: find examples/${{ inputs.example }} -name "requirements.txt" | while read req_file; do python3 -m pip install -r "$req_file" || exit 1; done
+    - name: Run tests
+      id: run_tests
+      shell: bash
+      continue-on-error: ${{ contains(format(',{0},', inputs.allow-failure-examples), format(',{0},', inputs.example)) }}
+      env:
+        # Absolute paths so subprocesses running from different working directories
+        # all find the config and write .coverage.* files to the same location.
+        COVERAGE_PROCESS_START: ${{ github.workspace }}/pyproject.toml
+        COVERAGE_FILE: ${{ github.workspace }}/.coverage
+        # sys.monitoring records the same lines as the default C tracer at a fraction of the
+        # overhead; the tracer slowed each hf_ptq example run by ~60%. Needs Python 3.12+.
+        COVERAGE_CORE: sysmon
+      run: |
+        echo "Running tests for: ${{ inputs.example }}"
+        pytest_args=()
+        if [[ "${{ inputs.example }}" == "diffusers" ]]; then
+          # Keep legacy ONNX/TRT tests available manually during deprecation.
+          pytest_args+=(--ignore=tests/examples/diffusers/test_diffusers_onnx_trt.py)
+        fi
+        if [[ "${{ inputs.example }}" == "hf_ptq" && "${{ github.ref }}" == refs/heads/pull-request/* ]]; then
+          # One GPU on PR runners: small-model tests share it and the rest run alone on it
+          # (tests/examples/hf_ptq/conftest.py). The 2-GPU nightly stays serial.
+          pytest_args+=(-n 4)
+        fi
+        python3 -m pytest tests/examples/${{ inputs.example }} --cov "${pytest_args[@]}"
+    - name: Flag allowed failure
+      if: ${{ steps.run_tests.outcome == 'failure' }}
+      shell: bash
+      run: |
+        echo "::warning title=Allowed example failure::'${{ inputs.example }}' failed but is in the allow-failure list (vars.ALLOW_FAILURE_EXAMPLE_TESTS); not blocking. Remove it from the variable once fixed."
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
+      with:
+        token: ${{ inputs.codecov-token }}
+        files: coverage.xml
+        # One flag per example, not a shared `examples`: carryforward only applies to a flag
+        # with no upload, so a shared flag would replace every lane's coverage with the subset
+        # that ran once lanes are gated independently.
+        flags: examples-${{ inputs.example }}
+        fail_ci_if_error: false # test may be skipped if relevant file changes are not detected
+        verbose: true
diff --git a/.github/workflows/_example_tests_runner.yml b/.github/workflows/_example_tests_runner.yml
index 18979ce0f48..737dbddca8d 100644
--- a/.github/workflows/_example_tests_runner.yml
+++ b/.github/workflows/_example_tests_runner.yml
@@ -9,7 +9,7 @@ on:
         required: true
         type: string
       example:
-        description: "Example name to test (e.g. 'hf_ptq')"
+        description: "Example name to test (e.g. 'hf_ptq'), or up to 3 names separated by spaces: they share the job (one container start and one base install) and run one after the other, each in its own step, in the order given. List an example that pins or upgrades packages last."
         required: true
         type: string
       timeout_minutes:
@@ -27,11 +27,11 @@ on:
         required: false
         type: string
         default: "linux-amd64-gpu-rtxpro6000-latest-1"
-      allow_failure:
-        description: "If true, test failures are reported as a warning and do not fail the job (used to keep a known-broken example non-blocking)"
+      allow_failure_examples:
+        description: "Comma-separated example names whose test failures are reported as a warning and do not fail the job (used to keep a known-broken example non-blocking)"
         required: false
-        type: boolean
-        default: false
+        type: string
+        default: ""
 
 jobs:
   run-test:
@@ -50,6 +50,15 @@ jobs:
         # Build CUDA kernels only for the runner's RTX PRO 6000 (sm_120), not the image's ~6 archs.
         TORCH_CUDA_ARCH_LIST: "12.0"
     steps:
+      - name: Split the examples of this job
+        id: examples
+        shell: bash
+        run: |
+          read -ra examples <<< "${{ inputs.example }}"
+          (( ${#examples[@]} <= 3 )) || { echo "::error::At most 3 examples can share a job: ${{ inputs.example }}"; exit 1; }
+          for i in 0 1 2; do echo "e$i=${examples[$i]:-}" >> "$GITHUB_OUTPUT"; done
+          # Expressions have no replace(), and a cache key is better without spaces
+          echo "key=$(IFS=+; echo "${examples[*]}")" >> "$GITHUB_OUTPUT"
       - name: Install git for vLLM image
         if: startsWith(inputs.docker_image, 'vllm/') || startsWith(inputs.docker_image, 'docker.io/vllm/')
         run: apt-get update && apt-get install -y git
@@ -58,8 +67,8 @@ jobs:
       - uses: ./.github/actions/cache-extensions
         id: ext-cache
         with:
-          # Per example: jobs sharing an image build different extensions, and the first to save wins the key.
-          cache-key: rtxpro6000-example-${{ inputs.example }}-${{ inputs.docker_image }}
+          # Per job (its examples): jobs sharing an image build different extensions, and the first to save wins the key.
+          cache-key: rtxpro6000-example-${{ steps.examples.outputs.key }}-${{ inputs.docker_image }}
       - name: Setup environment variables
         run: |
           echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/include:/usr/lib/x86_64-linux-gnu:/usr/local/tensorrt/targets/x86_64-linux-gnu/lib" >> $GITHUB_ENV
@@ -75,6 +84,8 @@ jobs:
           if [[ "${{ inputs.docker_image }}" == *"/tensorrt:"* ]]; then
             echo "torch<2.14" > /tmp/pip-constraints.txt
             export PIP_CONSTRAINT=/tmp/pip-constraints.txt
+            # The requirements of each example are installed later, in their own step
+            echo "PIP_CONSTRAINT=/tmp/pip-constraints.txt" >> "$GITHUB_ENV"
           fi
 
           # Use `python3 -m pip` instead of `pip` to avoid conflicts with system pip for nemo containers
@@ -86,50 +97,40 @@ jobs:
             python3 -m pip uninstall -y apex || true
           fi
 
-          find examples/${{ inputs.example }} -name "requirements.txt" | while read req_file; do python3 -m pip install -r "$req_file" || exit 1; done
-      - name: Run tests
-        id: run_tests
-        shell: bash
-        continue-on-error: ${{ inputs.allow_failure }}
-        env:
-          # Absolute paths so subprocesses running from different working directories
-          # all find the config and write .coverage.* files to the same location.
-          COVERAGE_PROCESS_START: ${{ github.workspace }}/pyproject.toml
-          COVERAGE_FILE: ${{ github.workspace }}/.coverage
-          # sys.monitoring records the same lines as the default C tracer at a fraction of the
-          # overhead; the tracer slowed each hf_ptq example run by ~60%. Needs Python 3.12+.
-          COVERAGE_CORE: sysmon
-        run: |
-          echo "Running tests for: ${{ inputs.example }}"
-          pytest_args=()
-          if [[ "${{ inputs.example }}" == "diffusers" ]]; then
-            # Keep legacy ONNX/TRT tests available manually during deprecation.
-            pytest_args+=(--ignore=tests/examples/diffusers/test_diffusers_onnx_trt.py)
-          fi
-          if [[ "${{ inputs.example }}" == "hf_ptq" && "${{ github.ref }}" == refs/heads/pull-request/* ]]; then
-            # One GPU on PR runners: small-model tests share it and the rest run alone on it
-            # (tests/examples/hf_ptq/conftest.py). The 2-GPU nightly stays serial.
-            pytest_args+=(-n 4)
-          fi
-          python3 -m pytest tests/examples/${{ inputs.example }} --cov "${pytest_args[@]}"
+      # One step per example. A failing example does not stop the ones after it (they are independent, as
+      # when each had its own job); the last step of the job fails it. A job with one example fails at the
+      # example's own step.
+      - name: "Example 1: ${{ steps.examples.outputs.e0 }}"
+        id: example0
+        continue-on-error: ${{ steps.examples.outputs.e1 != '' }}
+        uses: ./.github/actions/run-example
+        with:
+          example: ${{ steps.examples.outputs.e0 }}
+          allow-failure-examples: ${{ inputs.allow_failure_examples }}
+          codecov-token: ${{ secrets.CODECOV_TOKEN }}
+      - name: "Example 2: ${{ steps.examples.outputs.e1 || '-' }}"
+        id: example1
+        if: steps.examples.outputs.e1 != ''
+        continue-on-error: ${{ steps.examples.outputs.e2 != '' }}
+        uses: ./.github/actions/run-example
+        with:
+          example: ${{ steps.examples.outputs.e1 }}
+          allow-failure-examples: ${{ inputs.allow_failure_examples }}
+          codecov-token: ${{ secrets.CODECOV_TOKEN }}
+      - name: "Example 3: ${{ steps.examples.outputs.e2 || '-' }}"
+        id: example2
+        if: steps.examples.outputs.e2 != ''
+        uses: ./.github/actions/run-example
+        with:
+          example: ${{ steps.examples.outputs.e2 }}
+          allow-failure-examples: ${{ inputs.allow_failure_examples }}
+          codecov-token: ${{ secrets.CODECOV_TOKEN }}
       - name: Save extensions
         # Not tied to the tests passing, see the action description
         if: always() && steps.ext-cache.outputs.cache-hit != 'true'
         uses: ./.github/actions/save-extensions
         with:
           cache-key: ${{ steps.ext-cache.outputs.cache-key }}
-      - name: Flag allowed failure
-        if: ${{ inputs.allow_failure && steps.run_tests.outcome == 'failure' }}
-        run: |
-          echo "::warning title=Allowed example failure::'${{ inputs.example }}' failed but is in the allow-failure list (vars.ALLOW_FAILURE_EXAMPLE_TESTS); not blocking. Remove it from the variable once fixed."
-      - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
-        with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          files: coverage.xml
-          # One flag per example, not a shared `examples`: carryforward only applies to a flag
-          # with no upload, so a shared flag would replace every lane's coverage with the subset
-          # that ran once lanes are gated independently.
-          flags: examples-${{ inputs.example }}
-          fail_ci_if_error: false # test may be skipped if relevant file changes are not detected
-          verbose: true
+      - name: Fail the job for an example that failed before the last one
+        if: steps.example0.outcome == 'failure' || steps.example1.outcome == 'failure'
+        run: exit 1
diff --git a/.github/workflows/example_tests.yml b/.github/workflows/example_tests.yml
index 703642ad9af..75c1ac93dd5 100644
--- a/.github/workflows/example_tests.yml
+++ b/.github/workflows/example_tests.yml
@@ -14,8 +14,14 @@ concurrency:
   group: ${{ github.workflow }}-${{ startsWith(github.ref, 'refs/heads/pull-request/') && github.ref || github.sha }}
   cancel-in-progress: true
 
-# Each job's `allow_failure` reads the repo variable ALLOW_FAILURE_EXAMPLE_TESTS:
+# Every job passes the repo variable ALLOW_FAILURE_EXAMPLE_TESTS to the runner as `allow_failure_examples`:
 # a comma-separated list of example names whose test failures should be non-blocking, e.g. "torch_trt,llm_qat"
+#
+# An `example` of several names separated by spaces runs them one after the other in one job, each in its own
+# step, which saves the container start and the base install (3-5 min each) of every job after the first. Only
+# small examples with the same image and extras are grouped; each example installs its own requirements just
+# before its tests, so a group lists first the examples that leave the environment alone and last the ones that
+# pin or upgrade packages.
 
 jobs:
   # One changed-files pass decides which lanes run. Lane granularity, not per-example: a job's
@@ -126,7 +132,9 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        example: [llm_distill, llm_qat, llm_sparsity, specdec_bench, speculative_decoding]
+        # llm_qat and speculative_decoding take 9-16 min and stay on their own jobs; the three small ones share one.
+        # llm_distill pins trl and torchao, so it goes last.
+        example: ["specdec_bench llm_sparsity llm_distill", llm_qat, speculative_decoding]
     uses: ./.github/workflows/_example_tests_runner.yml
     permissions:
       contents: read
@@ -137,7 +145,7 @@ jobs:
       timeout_minutes: 30
       pip_install_extras: "[hf,dev-test]"
       runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
-      allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }}
+      allow_failure_examples: ${{ vars.ALLOW_FAILURE_EXAMPLE_TESTS }}
 
   ##### vLLM Example Tests #####
   vllm:
@@ -157,7 +165,7 @@ jobs:
       timeout_minutes: 30
       pip_install_extras: "[hf,dev-test]"
       runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
-      allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }}
+      allow_failure_examples: ${{ vars.ALLOW_FAILURE_EXAMPLE_TESTS }}
 
   ##### TensorRT-LLM Example Tests #####
   trtllm:
@@ -166,7 +174,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        example: [gpt-oss, hf_ptq, llm_eval]
+        # hf_ptq is the longest and stays alone. gpt-oss upgrades datasets and trl, so it goes after llm_eval.
+        example: ["llm_eval gpt-oss", hf_ptq]
     uses: ./.github/workflows/_example_tests_runner.yml
     permissions:
       contents: read
@@ -178,7 +187,7 @@ jobs:
       example: ${{ matrix.example }}
       pip_install_extras: "[hf,dev-test]"
       runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
-      allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }}
+      allow_failure_examples: ${{ vars.ALLOW_FAILURE_EXAMPLE_TESTS }}
 
   ##### Megatron Example Tests #####
   megatron:
@@ -195,7 +204,7 @@ jobs:
       timeout_minutes: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 30 || 75 }}
       pip_install_extras: "[hf,puzzletron,dev-test]"
       runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
-      allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), ',megatron_bridge,') }}
+      allow_failure_examples: ${{ vars.ALLOW_FAILURE_EXAMPLE_TESTS }}
 
   ##### Diffusers Example Tests #####
   diffusers:
@@ -212,7 +221,7 @@ jobs:
       timeout_minutes: 60
       pip_install_extras: "[onnx,hf,dev-test]"
       runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
-      allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), ',diffusers,') }}
+      allow_failure_examples: ${{ vars.ALLOW_FAILURE_EXAMPLE_TESTS }}
 
   ##### ONNX/TensorRT Example Tests #####
   onnx:
@@ -235,7 +244,7 @@ jobs:
       timeout_minutes: 45
       pip_install_extras: "[onnx,hf,dev-test]"
       runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
-      allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }}
+      allow_failure_examples: ${{ vars.ALLOW_FAILURE_EXAMPLE_TESTS }}
 
   ##### Required Check for PR #####
   example-pr-required-check:

From ab0a36f7405e497c922bec18a4fb643bdef14733 Mon Sep 17 00:00:00 2001
From: Shengliang Xu <shengliangx@nvidia.com>
Date: Thu, 22 Oct 2026 07:30:05 +0000
Subject: [PATCH 2/7] Guard the example coverage upload and the example count
 of a job

Skip the Codecov upload after an allowed failure, which continue-on-error
leaves green, so its partial coverage is not reported under the example's
flag. Require 1 to 3 names in a job's example list (an empty list made the
first action run all of tests/examples) and pass the list through the
environment instead of expanding it into the script.

Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
---
 .github/actions/run-example/action.yml      | 2 ++
 .github/workflows/_example_tests_runner.yml | 6 ++++--
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/.github/actions/run-example/action.yml b/.github/actions/run-example/action.yml
index 736dbf91912..1bdea3d69ec 100644
--- a/.github/actions/run-example/action.yml
+++ b/.github/actions/run-example/action.yml
@@ -55,6 +55,8 @@ runs:
       run: |
         echo "::warning title=Allowed example failure::'${{ inputs.example }}' failed but is in the allow-failure list (vars.ALLOW_FAILURE_EXAMPLE_TESTS); not blocking. Remove it from the variable once fixed."
     - name: Upload coverage to Codecov
+      # continue-on-error leaves the job green after an allowed failure; do not upload its partial coverage
+      if: ${{ steps.run_tests.outcome == 'success' }}
       uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
       with:
         token: ${{ inputs.codecov-token }}
diff --git a/.github/workflows/_example_tests_runner.yml b/.github/workflows/_example_tests_runner.yml
index 737dbddca8d..7fee9b8e6c2 100644
--- a/.github/workflows/_example_tests_runner.yml
+++ b/.github/workflows/_example_tests_runner.yml
@@ -53,9 +53,11 @@ jobs:
       - name: Split the examples of this job
         id: examples
         shell: bash
+        env:
+          EXAMPLES: ${{ inputs.example }}
         run: |
-          read -ra examples <<< "${{ inputs.example }}"
-          (( ${#examples[@]} <= 3 )) || { echo "::error::At most 3 examples can share a job: ${{ inputs.example }}"; exit 1; }
+          read -ra examples <<< "$EXAMPLES"
+          (( ${#examples[@]} >= 1 && ${#examples[@]} <= 3 )) || { echo "::error::A job runs 1 to 3 examples, got: '$EXAMPLES'"; exit 1; }
           for i in 0 1 2; do echo "e$i=${examples[$i]:-}" >> "$GITHUB_OUTPUT"; done
           # Expressions have no replace(), and a cache key is better without spaces
           echo "key=$(IFS=+; echo "${examples[*]}")" >> "$GITHUB_OUTPUT"

From 05e423d5d45ff8eefc456d3319e7f979e3359e8e Mon Sep 17 00:00:00 2001
From: Shengliang Xu <shengliangx@nvidia.com>
Date: Thu, 22 Oct 2026 17:10:28 +0000
Subject: [PATCH 3/7] Make the allow-failure warning and the example cap
 explicit

Restore the allow-list check in the condition of the allowed-failure warning,
and derive the example loop bound from the validated cap, so the validation
and the outputs cannot drift apart.

Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
---
 .github/actions/run-example/action.yml      | 2 +-
 .github/workflows/_example_tests_runner.yml | 8 +++++---
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/.github/actions/run-example/action.yml b/.github/actions/run-example/action.yml
index 1bdea3d69ec..1c3c2a77123 100644
--- a/.github/actions/run-example/action.yml
+++ b/.github/actions/run-example/action.yml
@@ -50,7 +50,7 @@ runs:
         fi
         python3 -m pytest tests/examples/${{ inputs.example }} --cov "${pytest_args[@]}"
     - name: Flag allowed failure
-      if: ${{ steps.run_tests.outcome == 'failure' }}
+      if: ${{ steps.run_tests.outcome == 'failure' && contains(format(',{0},', inputs.allow-failure-examples), format(',{0},', inputs.example)) }}
       shell: bash
       run: |
         echo "::warning title=Allowed example failure::'${{ inputs.example }}' failed but is in the allow-failure list (vars.ALLOW_FAILURE_EXAMPLE_TESTS); not blocking. Remove it from the variable once fixed."
diff --git a/.github/workflows/_example_tests_runner.yml b/.github/workflows/_example_tests_runner.yml
index 7fee9b8e6c2..9c828e47ca0 100644
--- a/.github/workflows/_example_tests_runner.yml
+++ b/.github/workflows/_example_tests_runner.yml
@@ -56,9 +56,10 @@ jobs:
         env:
           EXAMPLES: ${{ inputs.example }}
         run: |
+          max=3  # the cap; the "Example N" steps and the final failure check below must match it
           read -ra examples <<< "$EXAMPLES"
-          (( ${#examples[@]} >= 1 && ${#examples[@]} <= 3 )) || { echo "::error::A job runs 1 to 3 examples, got: '$EXAMPLES'"; exit 1; }
-          for i in 0 1 2; do echo "e$i=${examples[$i]:-}" >> "$GITHUB_OUTPUT"; done
+          (( ${#examples[@]} >= 1 && ${#examples[@]} <= max )) || { echo "::error::A job runs 1 to $max examples, got: '$EXAMPLES'"; exit 1; }
+          for (( i = 0; i < max; i++ )); do echo "e$i=${examples[$i]:-}" >> "$GITHUB_OUTPUT"; done
           # Expressions have no replace(), and a cache key is better without spaces
           echo "key=$(IFS=+; echo "${examples[*]}")" >> "$GITHUB_OUTPUT"
       - name: Install git for vLLM image
@@ -101,7 +102,8 @@ jobs:
 
       # One step per example. A failing example does not stop the ones after it (they are independent, as
       # when each had its own job); the last step of the job fails it. A job with one example fails at the
-      # example's own step.
+      # example's own step. These three steps and that final check hold the job's cap of 3 (`max` in the
+      # first step).
       - name: "Example 1: ${{ steps.examples.outputs.e0 }}"
         id: example0
         continue-on-error: ${{ steps.examples.outputs.e1 != '' }}

From e332c2e1522469da33ae7a9a346927eaf5e67593 Mon Sep 17 00:00:00 2001
From: Shengliang Xu <shengliangx@nvidia.com>
Date: Thu, 22 Oct 2026 17:25:38 +0000
Subject: [PATCH 4/7] Report a failed example instead of failing its step

A failed step ends the job at once, whatever continue-on-error the caller puts
on the action's step (checked on a hosted runner), so a failing example skipped
the examples after it. The action now reports `failed` and the last step of the
job fails it after the remaining examples have run. The allow-failure verdict
moves into the action.

Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
---
 .github/actions/run-example/action.yml      | 35 ++++++++++++++++-----
 .github/workflows/_example_tests_runner.yml | 14 ++++-----
 2 files changed, 34 insertions(+), 15 deletions(-)

diff --git a/.github/actions/run-example/action.yml b/.github/actions/run-example/action.yml
index 1c3c2a77123..929a82d2576 100644
--- a/.github/actions/run-example/action.yml
+++ b/.github/actions/run-example/action.yml
@@ -1,13 +1,14 @@
 name: Run example tests
 description: Install the requirements of one example, run its tests with coverage and upload the coverage under the flag `examples-<example>`. Used by _example_tests_runner.yml,
-  once per example, so that examples sharing a job still get their own step, requirements and Codecov flag.
+  once per example, so that examples sharing a job still get their own step, requirements and Codecov flag. Failing tests do not fail this action; it reports
+  them in `failed`, so that the caller can run the remaining examples first.
 
 inputs:
   example:
     description: Example directory name, e.g. `llm_distill` (tests/examples/<example>, examples/<example>).
     required: true
   allow-failure-examples:
-    description: Comma-separated example names whose test failures are reported as a warning instead of failing the step.
+    description: Comma-separated example names whose test failures are reported as a warning instead of as a failed example.
     required: false
     default: ""
   codecov-token:
@@ -15,6 +16,11 @@ inputs:
     required: false
     default: ""
 
+outputs:
+  failed:
+    description: "'true' if the example's tests failed and the example is not in the allow-failure list."
+    value: ${{ steps.verdict.outputs.failed }}
+
 runs:
   using: composite
   steps:
@@ -27,7 +33,9 @@ runs:
     - name: Run tests
       id: run_tests
       shell: bash
-      continue-on-error: ${{ contains(format(',{0},', inputs.allow-failure-examples), format(',{0},', inputs.example)) }}
+      # The verdict below decides what a failure means. A step that fails here would fail the whole job at
+      # once, whatever continue-on-error the caller puts on this action's step, and skip the later examples.
+      continue-on-error: true
       env:
         # Absolute paths so subprocesses running from different working directories
         # all find the config and write .coverage.* files to the same location.
@@ -49,13 +57,26 @@ runs:
           pytest_args+=(-n 4)
         fi
         python3 -m pytest tests/examples/${{ inputs.example }} --cov "${pytest_args[@]}"
-    - name: Flag allowed failure
-      if: ${{ steps.run_tests.outcome == 'failure' && contains(format(',{0},', inputs.allow-failure-examples), format(',{0},', inputs.example)) }}
+    - name: Verdict
+      id: verdict
       shell: bash
+      env:
+        OUTCOME: ${{ steps.run_tests.outcome }}
+        EXAMPLE: ${{ inputs.example }}
+        ALLOW_FAILURE_EXAMPLES: ${{ inputs.allow-failure-examples }}
       run: |
-        echo "::warning title=Allowed example failure::'${{ inputs.example }}' failed but is in the allow-failure list (vars.ALLOW_FAILURE_EXAMPLE_TESTS); not blocking. Remove it from the variable once fixed."
+        failed=false
+        if [[ "$OUTCOME" == "failure" ]]; then
+          if [[ ",$ALLOW_FAILURE_EXAMPLES," == *",$EXAMPLE,"* ]]; then
+            echo "::warning title=Allowed example failure::'$EXAMPLE' failed but is in the allow-failure list (vars.ALLOW_FAILURE_EXAMPLE_TESTS); not blocking. Remove it from the variable once fixed."
+          else
+            echo "::error title=Example failed::The tests of '$EXAMPLE' failed (see its Run tests log). The job fails after the remaining examples have run."
+            failed=true
+          fi
+        fi
+        echo "failed=$failed" >> "$GITHUB_OUTPUT"
     - name: Upload coverage to Codecov
-      # continue-on-error leaves the job green after an allowed failure; do not upload its partial coverage
+      # A failed run, allowed or not, only has partial coverage
       if: ${{ steps.run_tests.outcome == 'success' }}
       uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
       with:
diff --git a/.github/workflows/_example_tests_runner.yml b/.github/workflows/_example_tests_runner.yml
index 9c828e47ca0..48759ef7ea9 100644
--- a/.github/workflows/_example_tests_runner.yml
+++ b/.github/workflows/_example_tests_runner.yml
@@ -100,13 +100,12 @@ jobs:
             python3 -m pip uninstall -y apex || true
           fi
 
-      # One step per example. A failing example does not stop the ones after it (they are independent, as
-      # when each had its own job); the last step of the job fails it. A job with one example fails at the
-      # example's own step. These three steps and that final check hold the job's cap of 3 (`max` in the
-      # first step).
+      # One step per example. A failing example does not fail its step (the action reports `failed`
+      # instead, because a failed step would end the job at once), so the examples after it still run, as
+      # when each had its own job, and the last step of the job fails it. These three steps and that final
+      # check hold the job's cap of 3 (`max` in the first step).
       - name: "Example 1: ${{ steps.examples.outputs.e0 }}"
         id: example0
-        continue-on-error: ${{ steps.examples.outputs.e1 != '' }}
         uses: ./.github/actions/run-example
         with:
           example: ${{ steps.examples.outputs.e0 }}
@@ -115,7 +114,6 @@ jobs:
       - name: "Example 2: ${{ steps.examples.outputs.e1 || '-' }}"
         id: example1
         if: steps.examples.outputs.e1 != ''
-        continue-on-error: ${{ steps.examples.outputs.e2 != '' }}
         uses: ./.github/actions/run-example
         with:
           example: ${{ steps.examples.outputs.e1 }}
@@ -135,6 +133,6 @@ jobs:
         uses: ./.github/actions/save-extensions
         with:
           cache-key: ${{ steps.ext-cache.outputs.cache-key }}
-      - name: Fail the job for an example that failed before the last one
-        if: steps.example0.outcome == 'failure' || steps.example1.outcome == 'failure'
+      - name: Fail the job for an example whose tests failed
+        if: steps.example0.outputs.failed == 'true' || steps.example1.outputs.failed == 'true' || steps.example2.outputs.failed == 'true'
         run: exit 1

From 5d6dfe5128110ecf310be98b07ac5b875df328f4 Mon Sep 17 00:00:00 2001
From: Shengliang Xu <shengliangx@nvidia.com>
Date: Thu, 22 Oct 2026 18:05:11 +0000
Subject: [PATCH 5/7] Tolerate a failing install and whitespace in the
 allow-failure list

The requirements install ran in a step of its own, so a broken requirements.txt
failed the job at once, skipped the later examples of its group and bypassed
the allow-failure list. The install and the tests now run in one step that
records the exit status instead of failing, and the verdict reads it, so
nothing depends on continue-on-error inside the action. The allow-failure
match ignores whitespace, so "diffusers, llm_qat" works.

Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
---
 .github/actions/run-example/action.yml      | 64 +++++++++++----------
 .github/workflows/_example_tests_runner.yml |  2 +-
 2 files changed, 36 insertions(+), 30 deletions(-)

diff --git a/.github/actions/run-example/action.yml b/.github/actions/run-example/action.yml
index 929a82d2576..62ca148adaf 100644
--- a/.github/actions/run-example/action.yml
+++ b/.github/actions/run-example/action.yml
@@ -1,14 +1,14 @@
 name: Run example tests
 description: Install the requirements of one example, run its tests with coverage and upload the coverage under the flag `examples-<example>`. Used by _example_tests_runner.yml,
-  once per example, so that examples sharing a job still get their own step, requirements and Codecov flag. Failing tests do not fail this action; it reports
-  them in `failed`, so that the caller can run the remaining examples first.
+  once per example, so that examples sharing a job still get their own step, requirements and Codecov flag. A failing install or failing tests do not fail
+  this action; it reports them in `failed`, so that the caller can run the remaining examples first.
 
 inputs:
   example:
     description: Example directory name, e.g. `llm_distill` (tests/examples/<example>, examples/<example>).
     required: true
   allow-failure-examples:
-    description: Comma-separated example names whose test failures are reported as a warning instead of as a failed example.
+    description: Comma-separated example names (whitespace is ignored) whose failures are reported as a warning instead of as a failed example.
     required: false
     default: ""
   codecov-token:
@@ -18,24 +18,15 @@ inputs:
 
 outputs:
   failed:
-    description: "'true' if the example's tests failed and the example is not in the allow-failure list."
+    description: "'true' if the example's install or tests failed and the example is not in the allow-failure list."
     value: ${{ steps.verdict.outputs.failed }}
 
 runs:
   using: composite
   steps:
-    # Installed right before this example's tests, not up front for all examples of the job: an example
-    # that pins or upgrades packages (e.g. gpt-oss upgrades datasets, torch_trt downgrades torch) must not
-    # change the environment of the examples that run before it.
-    - name: Install requirements of ${{ inputs.example }}
-      shell: bash
-      run: find examples/${{ inputs.example }} -name "requirements.txt" | while read req_file; do python3 -m pip install -r "$req_file" || exit 1; done
     - name: Run tests
       id: run_tests
       shell: bash
-      # The verdict below decides what a failure means. A step that fails here would fail the whole job at
-      # once, whatever continue-on-error the caller puts on this action's step, and skip the later examples.
-      continue-on-error: true
       env:
         # Absolute paths so subprocesses running from different working directories
         # all find the config and write .coverage.* files to the same location.
@@ -45,39 +36,54 @@ runs:
         # overhead; the tracer slowed each hf_ptq example run by ~60%. Needs Python 3.12+.
         COVERAGE_CORE: sysmon
       run: |
-        echo "Running tests for: ${{ inputs.example }}"
-        pytest_args=()
-        if [[ "${{ inputs.example }}" == "diffusers" ]]; then
-          # Keep legacy ONNX/TRT tests available manually during deprecation.
-          pytest_args+=(--ignore=tests/examples/diffusers/test_diffusers_onnx_trt.py)
-        fi
-        if [[ "${{ inputs.example }}" == "hf_ptq" && "${{ github.ref }}" == refs/heads/pull-request/* ]]; then
-          # One GPU on PR runners: small-model tests share it and the rest run alone on it
-          # (tests/examples/hf_ptq/conftest.py). The 2-GPU nightly stays serial.
-          pytest_args+=(-n 4)
+        # Nothing here may fail the step: a failed step ends the whole job at once, whatever continue-on-error
+        # the caller puts on this action's step, and the later examples would be skipped. The exit status goes
+        # to the verdict below instead.
+        status=0
+
+        # Installed right before this example's tests, not up front for all examples of the job: an example
+        # that pins or upgrades packages (e.g. gpt-oss upgrades datasets, torch_trt downgrades torch) must not
+        # change the environment of the examples that run before it.
+        echo "::group::Install requirements of ${{ inputs.example }}"
+        find examples/${{ inputs.example }} -name "requirements.txt" | while read req_file; do python3 -m pip install -r "$req_file" || exit 1; done || status=$?
+        echo "::endgroup::"
+
+        if (( status == 0 )); then
+          echo "Running tests for: ${{ inputs.example }}"
+          pytest_args=()
+          if [[ "${{ inputs.example }}" == "diffusers" ]]; then
+            # Keep legacy ONNX/TRT tests available manually during deprecation.
+            pytest_args+=(--ignore=tests/examples/diffusers/test_diffusers_onnx_trt.py)
+          fi
+          if [[ "${{ inputs.example }}" == "hf_ptq" && "${{ github.ref }}" == refs/heads/pull-request/* ]]; then
+            # One GPU on PR runners: small-model tests share it and the rest run alone on it
+            # (tests/examples/hf_ptq/conftest.py). The 2-GPU nightly stays serial.
+            pytest_args+=(-n 4)
+          fi
+          python3 -m pytest tests/examples/${{ inputs.example }} --cov "${pytest_args[@]}" || status=$?
         fi
-        python3 -m pytest tests/examples/${{ inputs.example }} --cov "${pytest_args[@]}"
+        echo "status=$status" >> "$GITHUB_OUTPUT"
     - name: Verdict
       id: verdict
       shell: bash
       env:
-        OUTCOME: ${{ steps.run_tests.outcome }}
+        STATUS: ${{ steps.run_tests.outputs.status }}
         EXAMPLE: ${{ inputs.example }}
         ALLOW_FAILURE_EXAMPLES: ${{ inputs.allow-failure-examples }}
       run: |
         failed=false
-        if [[ "$OUTCOME" == "failure" ]]; then
-          if [[ ",$ALLOW_FAILURE_EXAMPLES," == *",$EXAMPLE,"* ]]; then
+        if [[ "$STATUS" != "0" ]]; then
+          if [[ ",${ALLOW_FAILURE_EXAMPLES//[[:space:]]/}," == *",$EXAMPLE,"* ]]; then
             echo "::warning title=Allowed example failure::'$EXAMPLE' failed but is in the allow-failure list (vars.ALLOW_FAILURE_EXAMPLE_TESTS); not blocking. Remove it from the variable once fixed."
           else
-            echo "::error title=Example failed::The tests of '$EXAMPLE' failed (see its Run tests log). The job fails after the remaining examples have run."
+            echo "::error title=Example failed::The requirements install or the tests of '$EXAMPLE' failed (see the log of its Run tests step). The job fails after the remaining examples have run."
             failed=true
           fi
         fi
         echo "failed=$failed" >> "$GITHUB_OUTPUT"
     - name: Upload coverage to Codecov
       # A failed run, allowed or not, only has partial coverage
-      if: ${{ steps.run_tests.outcome == 'success' }}
+      if: ${{ steps.run_tests.outputs.status == '0' }}
       uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
       with:
         token: ${{ inputs.codecov-token }}
diff --git a/.github/workflows/_example_tests_runner.yml b/.github/workflows/_example_tests_runner.yml
index 48759ef7ea9..e0f45d8fb16 100644
--- a/.github/workflows/_example_tests_runner.yml
+++ b/.github/workflows/_example_tests_runner.yml
@@ -133,6 +133,6 @@ jobs:
         uses: ./.github/actions/save-extensions
         with:
           cache-key: ${{ steps.ext-cache.outputs.cache-key }}
-      - name: Fail the job for an example whose tests failed
+      - name: Fail the job for an example that failed
         if: steps.example0.outputs.failed == 'true' || steps.example1.outputs.failed == 'true' || steps.example2.outputs.failed == 'true'
         run: exit 1

From ba8aebb5e3480f43e3435dcd7316b132d0bc79d2 Mon Sep 17 00:00:00 2001
From: Shengliang Xu <shengliangx@nvidia.com>
Date: Thu, 22 Oct 2026 20:44:50 +0000
Subject: [PATCH 6/7] Log the GPU and package changes between the examples of a
 job

Examples of one job share the GPU and the environment. Log the GPU state before
each example and the packages its requirements changed, so a failure caused by
an earlier example can be traced. The apex uninstall for diffusers moves from
the job-wide install step, whose substring check would also remove apex for any
example grouped with diffusers, into the action, where it applies to that
example only. The grouping comment now says an example that changes torch must
not be grouped.

Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
---
 .github/actions/run-example/action.yml      | 17 +++++++++++++++++
 .github/workflows/_example_tests_runner.yml |  5 -----
 .github/workflows/example_tests.yml         |  4 +++-
 3 files changed, 20 insertions(+), 6 deletions(-)

diff --git a/.github/actions/run-example/action.yml b/.github/actions/run-example/action.yml
index 62ca148adaf..57c1ee98adc 100644
--- a/.github/actions/run-example/action.yml
+++ b/.github/actions/run-example/action.yml
@@ -41,12 +41,29 @@ runs:
         # to the verdict below instead.
         status=0
 
+        # Examples of one job share the GPU and the environment one after the other: log both, so a failure
+        # caused by an earlier example is easy to trace.
+        echo "::group::GPU state before ${{ inputs.example }}"
+        nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv || true
+        nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv || true
+        echo "::endgroup::"
+
+        if [[ "${{ inputs.example }}" == "diffusers" ]]; then
+          echo "Uninstalling apex for diffusers: T5 Int8 (PixArt) + Apex is not supported as per https://github.com/huggingface/transformers/issues/21391"
+          python3 -m pip uninstall -y apex || true
+        fi
+
         # Installed right before this example's tests, not up front for all examples of the job: an example
         # that pins or upgrades packages (e.g. gpt-oss upgrades datasets, torch_trt downgrades torch) must not
         # change the environment of the examples that run before it.
+        packages_before="${RUNNER_TEMP:-/tmp}/packages-before-${{ inputs.example }}.txt"
+        python3 -m pip freeze > "$packages_before" || true
         echo "::group::Install requirements of ${{ inputs.example }}"
         find examples/${{ inputs.example }} -name "requirements.txt" | while read req_file; do python3 -m pip install -r "$req_file" || exit 1; done || status=$?
         echo "::endgroup::"
+        echo "::group::Packages changed by the requirements of ${{ inputs.example }} (< before, > after)"
+        { python3 -m pip freeze | diff "$packages_before" - || true; } | grep "^[<>]" || echo "none"
+        echo "::endgroup::"
 
         if (( status == 0 )); then
           echo "Running tests for: ${{ inputs.example }}"
diff --git a/.github/workflows/_example_tests_runner.yml b/.github/workflows/_example_tests_runner.yml
index e0f45d8fb16..12e35c789ae 100644
--- a/.github/workflows/_example_tests_runner.yml
+++ b/.github/workflows/_example_tests_runner.yml
@@ -95,11 +95,6 @@ jobs:
           # Editable install so example scripts launched as subprocesses resolve modelopt to the same source path as the test process
           python3 -m pip install -e ".${{ inputs.pip_install_extras }}"
 
-          if [[ "${{ inputs.example }}" == *"diffusers"* ]]; then
-            echo "Uninstalling apex for diffusers: T5 Int8 (PixArt) + Apex is not supported as per https://github.com/huggingface/transformers/issues/21391"
-            python3 -m pip uninstall -y apex || true
-          fi
-
       # One step per example. A failing example does not fail its step (the action reports `failed`
       # instead, because a failed step would end the job at once), so the examples after it still run, as
       # when each had its own job, and the last step of the job fails it. These three steps and that final
diff --git a/.github/workflows/example_tests.yml b/.github/workflows/example_tests.yml
index 75c1ac93dd5..d6ae425d556 100644
--- a/.github/workflows/example_tests.yml
+++ b/.github/workflows/example_tests.yml
@@ -21,7 +21,9 @@ concurrency:
 # step, which saves the container start and the base install (3-5 min each) of every job after the first. Only
 # small examples with the same image and extras are grouped; each example installs its own requirements just
 # before its tests, so a group lists first the examples that leave the environment alone and last the ones that
-# pin or upgrade packages.
+# pin or upgrade packages. An example that changes torch (torch_trt downgrades it) must not be grouped: the
+# extension cache key is the group name plus the image, with no torch version, and a cache built against another
+# torch fails with undefined symbols.
 
 jobs:
   # One changed-files pass decides which lanes run. Lane granularity, not per-example: a job's

From f1d797af7b73c25fa6126f4713cd98a6c32b0db7 Mon Sep 17 00:00:00 2001
From: Shengliang Xu <shengliangx@nvidia.com>
Date: Fri, 23 Oct 2026 17:17:25 +0000
Subject: [PATCH 7/7] Hide the stderr of pip freeze in the example package diff

On the pytorch image every `pip freeze` prints about ten lines of `git config ... exit code 128` and
`fatal: not a git repository: /opt/pytorch/ao` to stderr, twice per example, one of them outside any
log group. The freeze is only logged (`|| true`) and the diff is unaffected, so stderr goes to
/dev/null (review suggestion).

Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
---
 .github/actions/run-example/action.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/actions/run-example/action.yml b/.github/actions/run-example/action.yml
index 57c1ee98adc..339860e6860 100644
--- a/.github/actions/run-example/action.yml
+++ b/.github/actions/run-example/action.yml
@@ -57,12 +57,12 @@ runs:
         # that pins or upgrades packages (e.g. gpt-oss upgrades datasets, torch_trt downgrades torch) must not
         # change the environment of the examples that run before it.
         packages_before="${RUNNER_TEMP:-/tmp}/packages-before-${{ inputs.example }}.txt"
-        python3 -m pip freeze > "$packages_before" || true
+        python3 -m pip freeze 2>/dev/null > "$packages_before" || true
         echo "::group::Install requirements of ${{ inputs.example }}"
         find examples/${{ inputs.example }} -name "requirements.txt" | while read req_file; do python3 -m pip install -r "$req_file" || exit 1; done || status=$?
         echo "::endgroup::"
         echo "::group::Packages changed by the requirements of ${{ inputs.example }} (< before, > after)"
-        { python3 -m pip freeze | diff "$packages_before" - || true; } | grep "^[<>]" || echo "none"
+        { python3 -m pip freeze 2>/dev/null | diff "$packages_before" - || true; } | grep "^[<>]" || echo "none"
         echo "::endgroup::"
 
         if (( status == 0 )); then