[lalalune]

Summary

Rebased onto eliza/develop-pin-ea8b3f2 (single commit on top of c6e523b).

• Metal kernels: quantize/dequantize helpers plus SET_ROWS (i32/i64) and CPY (f32→q, q→f32/f16) kernels for TBQ3_0, TBQ4_0, Q4_POLAR; advertised in ggml_metal_device_supports_op (CPY/CONT and SET_ROWS only). Conflict with develop-pin's quantize_iq4_nl_full / kernel_cpy_f32_iq4_nl_full resolved by keeping both.
• FA dequant gating (changed from the original PR): the original made TBQ3_0/TBQ4_0/Q4_POLAR K and V always cast F32→F16 before flash attention on every backend. CPU already scores these natively (vec_dot/to_float), so that would have added a full KV cast per layer per step on Android CPU. Now llama_context probes each offloaded KV layer's (non-CPU) device with a FLASH_ATTN_EXT on the requested cache types and sets cparams.fa_dequant_kv only when that device lacks native FA support (e.g. Metal). The existing always-dequant set (QJL1_256/TBQ3_TCQ K, Q4_POLAR V) is unchanged. CPU-only, --no-kv-offload and CPU-assigned layers never take the extra cast.
• Tests: test-backend-ops SET_ROWS cases for the three types; Q4_POLAR→F32 CPY now covered.
• Base branch's extra commit (diarizer fix) is not needed and not included.

Test evidence (Apple M4 Max, Metal)

• test-backend-ops -b MTL0 -o SET_ROWS: 147/147 passed (12 new tbq3_0/tbq4_0/q4_polar cases OK vs CPU)
• test-backend-ops -b MTL0 -o CPY: 99/99 passed (f32↔tbq3_0/tbq4_0/q4_polar OK)
• test-backend-ops -b MTL0 -o FLASH_ATTN_EXT: 4752/4752 passed
• test-backend-ops -b MTL0 -o MUL_MAT -p "type_a=(f32|f16|bf16|q4_0|q4_1|q5_0|q5_1|q8_0|q2_K|q3_K|q4_K|q5_K|q6_K|iq4_nl|iq4_xs),": 868/868 passed. (Unfiltered MUL_MAT aborts on q1_0_g32 — Metal mul_mv has no case for it; pre-existing, not touched here.)
• llama-completion with eliza-1-2b-32k, -fa on -ctk/-ctv tbq3_0 and tbq4_0, -ngl 99: Metal logs the probe ("device MTL0 has no native flash attention … dequantizing") and outputs "Paris."; graph nodes 1491.
• CPU-only build (GGML_METAL=OFF): builds; same run with tbq3_0 shows no probe/dequant (graph nodes 1467, i.e. no cast nodes) and outputs "Paris."

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 95ce213e-fb68-46dc-b5cd-c8224c549f2d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/metal-v-cache-set-rows-9258

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[lalalune]

Triage (#11386 M1): NOT superseded — keeping open. This adds Metal set_rows support for the custom KV quant types TBQ3_0 / TBQ4_0 / Q4_POLAR. Merged main (ad56033f) still carries only the QJL-only path (// // ELIZA-QJL-SET-ROWS-V1 at ggml/src/ggml-metal/ggml-metal-device.m:1539); this PR's markers (ELIZA-CUSTOM-KV-SET-ROWS-V1, k_tbq3_codebook_set_rows) are absent from main. So it is a genuine, unlanded enhancement — not part of the develop-pinned lineage.

Two follow-ups before it can land, both out of scope for the M1 pointer reconciliation:

1. Retarget base from feat/gemma4-assistant-arch (now merged via Reconcile main with the develop-pinned lineage (perf/kokoro-accelerate-blas + #11612 Metal guards) #42) to main and resolve the ggml-metal-device.m conflict.
2. Real Metal test-backend-ops verification of the new set_rows path for TBQ3_0/TBQ4_0/Q4_POLAR (GPU-gated).

Leaving open and tracking separately.