Repository navigation
ci: Vulkan validation with cost-balanced test-backend-ops sharding - #52
Merged
lalalune merged 12 commits intoOct 14, 2026
Merged
Conversation
github-actions
Bot
added
examples
testing
conversion
devops
ggml
model
Apple Metal
server
WebGPU
Vulkan
labels
lalalune
force-pushed
the
fix/bge-nonspacing-marks-31518
branch
from
October 14, 2026 03:30
cc0cfdf to
fd4e328
Compare
lalalune
changed the title
lalalune
changed the base branch from
codex/aosp69-vulkan-core-storage
to
eliza/develop-pin-ea8b3f2
October 14, 2026 03:31
--test-shard assigned occurrences by index modulo the shard count, so the
expensive hsk=512/576 FLASH_ATTN_EXT cases could stack on one worker. On
llvmpipe shard 86 of 128 exceeded the 600 s limit (exit 124) while most
shards finished in a few minutes.
test-backend-ops now estimates each matching case (op FLOPs plus a
per-byte and per-case term, from the same metadata-only graph eval()
builds) and assigns longest-first to the lightest shard. The assignment
is deterministic, printed in every inventory line ("Shard inventory N ->
S: ..."), and the verifier requires every worker to report the same
assignment, run exactly its own occurrences, and cover each occurrence
once across the union.
The workflow gives sharded invocations 1200 s (worker 0 also runs every
unsharded op, so the job limit rises to 75 minutes) and reports each
invocation's elapsed time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixed-artifact and partition-children diagnostics replayed one
runtime artifact from run 35506984389 to time cases from the shard that
timed out. They qualify nothing ("Diagnostic only, not matrix
qualification"), the fixed-artifact run exits 1 whenever one of its
pinned slow cases does not complete in its bound (as on the last push),
and the pinned artifact has a 14-day retention that has since expired,
so both would now fail on any pull request touching them. The shard
timing they measured is addressed by cost-balanced sharding.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lalalune
force-pushed
the
fix/bge-nonspacing-marks-31518
branch
from
October 14, 2026 03:43
fd4e328 to
b8fa7fe
Compare
lalalune
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Retargeted to
eliza/develop-pin-ea8b3f2and narrowed to Vulkan CI (2026-10-13).codex/aosp69-vulkan-core-storageis fully contained in it. The branch was rebuilt on develop-pin (c6e523b6).MTL0gate, MUL_MAT capability narrowing, TBQ3/TBQ4 attention-score kernels, matmul support test) is superseded by metal: enforce kernel capabilities and restore numerical parity gates #61 (codex/metal-capability-validation), which re-implements the same changes against develop-pin. Those commits were dropped here so the two PRs don't conflict; this branch merges cleanly with metal: enforce kernel capabilities and restore numerical parity gates #61's head (git merge-tree).What remains
test-backend-ops --test-shard index/count: deterministic partition of the matching occurrences (duplicates and whole-graph cases included), with each occurrence's owner printed in the inventory. Occurrences are assigned by estimated cost (op FLOPs + bytes + per-case term, longest first onto the lightest shard). Index-modulo assignment let the large hsk=512/576 FLASH_ATTN_EXT cases stack on one worker: shard 86 hit the 600 s limit (exit 124) in run 35630168295.--threadsalso applies to tensor initialization and the CPU reference backend.GGML_NATIVE=OFFruntime build shared by 128 workers (MUL_MAT and FLASH_ATTN_EXT sharded; unsharded ops on worker 0), 1200 s per sharded invocation with elapsed time logged, and an aggregate job that re-verifies every worker's raw log (scripts/ci_verify_vulkan.py): same inventory and assignment on every worker, each worker ran exactly its own occurrences, union covers every occurrence once, no FAIL, unsupported counted separately.Local: verifier tests pass (7);
--test-shardassignment identical across workers, 1,420 MUL_MAT and 5,096 FLASH_ATTN_EXT occurrences fully covered.Hosted CI on this head
Pending.
🤖 Generated with Claude Code