Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,26 @@

## [Unreleased]

Headline: **the browser lane stops falling to the reference kernel (SKEEP-007, Tier 0).**

### Added

- **SKEEP-007 "The browser lane"** (`docs/modules/skeep/pages/007-browser-lane.adoc`): the design record for making inference in a browser usable and quick — common-code fixes now, a SIMD128 sidecar kernel pack with pthreads and quantize-at-load for SafeTensors as the next pull requests, WebGPU as a later proposal.
- **`ByteArrayRandomAccessSource`** (`skainet-io-core`, `sk.ainet.io`): the engine's in-memory `RandomAccessSource`, single array or equal-sized chunks for files above the 2 GB array limit. Every web sample had written its own.
- **`NarrowFloatDenseTensorData` is a `RowDequantSource`**: FP16 and BF16 tables serve rows on demand, so a 16-bit token-embedding table loaded `KEEP_NATIVE` is read row by row by `ops.gather` instead of being widened to FP32 at load.
- **`skainet-backend-wasm-simd` (SKEEP-007 Tier 1): the browser kernel pack.** The engine's own C kernels (`skainet-backend-native-cpu/native/src`) compiled by Emscripten into a SIMD128 sidecar module — single-threaded (`skainet_kernels_st`) and pthreads (`skainet_kernels_mt`, for cross-origin-isolated pages and Node). `WasmSimd.install()` loads it, registers `WasmKernelProvider` (dense FP32 GEMM and the feed-order packed kernels, priority 50) and `WasmSimdKernelPack` (row-major Q8_0/Q4_0/Q4_K/Q5_K/Q6_K/Q5_0/Q5_1 kernels keyed like the JVM FFM pack), and makes every `DirectCpuExecutionContext` built afterwards use `WasmCpuOps`, which routes dense `x · Wᵀ` through the new `skainet_fp32_matmul_wt` kernel. Weights are uploaded into the sidecar once per weight and stay resident; activations cross per call through Kotlin's linear memory, no interop call per element. Without `install()` the engine runs the scalar lane as before. Explicit `__wasm_simd128__` bodies for the Q8_0 and Q4_0 block dot products; `skainet_fp32_matmul_wt` joins the native library for every tier.
- **`WasmCpuOpsOverride`** (`skainet-backend-cpu`, wasmJs): the one hook a kernel pack has into context construction on a target without `ServiceLoader`.
- **Quantize-at-load (SKEEP-007 Tier 2).** `Q8_0Quantizer` (`skainet-lang-core`, ggml-faithful, beside the existing `Q4_0Quantizer`) and the `Requantizers` registry (Q8_0, Q4_0, BITNET_PLANES). `SafeTensorsParametersLoader` and `ShardedSafeTensorsParametersLoader` take `weightForm` / `weightFormFor`: `EncodingRequest.RequantizeTo(Q8_0)` turns every rank-2 float matrix whose rows block into packed Q8_0 at load (vectors, norms and non-blocking rows stay dense), with `KERNEL_FEED` honoured; a request no quantizer can serve is refused at construction. `PlannerProfile.quantizeDenseTo` lets a profile ask `WeightFormResolver` for it on dense sources when a kernel can feed the encoding. Every Hugging Face checkpoint can now be served packed on every target; before, only GGUF files could.
- **`skainet-quantize`** (`skainet-apps/skainet-tensor-tools`, `./gradlew :skainet-apps:skainet-tensor-tools:quantize --args=...`): SafeTensors → GGUF with the weight matrices quantized offline (Q8_0 or Q4_0), tensor names kept, `--keep` prefixes written dense as F16 — for downloads that should not carry F16 weights.
- **`skainet-backends/benchmarks/wasm-bench`**: a Kotlin/Wasm executable that times the matmul shapes of the shipped models and prints `[SKAINET_PERF_WASM]` lines — the baseline for the following tiers.

### Changed

- **`matmulWeightTransposed` on a canonical packed weight** (every GGUF weight) now prefers, in order: an exact kernel for the row-major view, the one-time cached relayout into kernel feed order that the scalar packed kernels on every target read, and only then the decoding reference kernel. On JS, wasm and Kotlin/Native without the cinterop pack this replaces a ~1000× slower per-element decode on every call; the JVM overrides the path and is unaffected.
- **`matmulWeightTransposed` on a dense heap FP32 `[out, in]` weight** computes the product straight from the weight's rows (4 × 4 tiles, one sequential accumulator per output, bit-identical to the transpose-then-`matmul` result) on tiers without a vectorised `[k, n]` kernel, instead of making and caching a transposed copy of every weight. `DefaultCpuOpsBase.preferDirectWeightTransposed()` is the hook; the JVM and Apple ops keep their cached transpose.
- **GGUF `RequantizeTo`** accepts every encoding in `Requantizers` (was `BITNET_PLANES` only, #1150); freshly quantized Q8_0/Q4_0 honour `KERNEL_FEED`.
- **`JsBlobRandomAccessSource`** no longer logs every read to the console. (The byte-by-byte `ArrayBuffer` conversion stays: Kotlin/Wasm 2.4 has no typed-array adapter; SKEEP-007 Tier 1 replaces it with the sidecar's linear-memory bridge.)

## [0.60.0] - 2026-10-15

Headline: **key-padding lengths and window metadata for attention (#1343).**
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/ROOT/pages/explanation/eager-execution.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ mindmap
Android — scalar, Native-JNI
Native linux — scalar, Native-cinterop
Native apple — scalar, Accelerate, Native-cinterop
JS and WASM — scalar only
JS and WASM — scalar; packed weights relaid once into kernel feed order (SKEEP-007)
----

== Kernel × provider (matmul, FP32 activations)
Expand Down Expand Up @@ -88,7 +88,7 @@ not per call.
| *Android* | Scalar + Native-JNI | Panama (`jdk.incubator.vector`) and FFM (`java.lang.foreign`) are JDK-only — ART has neither, so Android's native tier is JNI, not a degraded JVM. Includes the NEON `bitnet_gemv` pack.
| *Kotlin/Native — linux x64/arm64* | Scalar + Native-cinterop | Static archive embedded in the klib; manual `installNativeKernels()`, no ServiceLoader on K/N.
| *Kotlin/Native — macOS/iOS* | Scalar + Apple Accelerate + Native-cinterop | Accelerate covers dense FP32 and reductions; cinterop covers packed quant with runtime `FEAT_DotProd` dispatch, so one archive serves A12 through M-series.
| *JS / WASM (Js, Wasi)* | Scalar | No SIMD.
| *JS / WASM (Js, Wasi)* | Scalar; wasmJs + `skainet-backend-wasm-simd` | No SIMD in Kotlin code: packed weights are relaid once per weight into kernel feed order and served by the scalar packed kernels; dense `[out, in]` weights are multiplied in place without a transpose copy (SKEEP-007 Tier 0). On wasmJs, `WasmSimd.install()` adds the SIMD128 sidecar pack — the engine's C kernels compiled by Emscripten, with pthreads on cross-origin-isolated pages (SKEEP-007 Tier 1).
|===

Packed-quant matmul works on every target: those formats have a `commonMain` scalar kernel and
Expand Down
9 changes: 7 additions & 2 deletions docs/modules/ROOT/pages/explanation/kernel-selection.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,15 @@ platform support is absent (a missing native library, no vector unit).
| `ServiceLoader`, same caveat

| Kotlin/Native, wasm, JS
| manual — e.g. `installNativeKernels()`
| manual — call the pack's `install()` yourself
| manual — e.g. `installNativeKernels()`; on wasmJs `WasmSimd.install()` (SKEEP-007)
| manual — call the pack's `install()` yourself; `WasmSimd.install()` registers the browser pack and its provider
|===

Without a row-major pack, a canonical packed weight is relaid **once per weight** into kernel feed
order and served by the scalar packed kernels (`DefaultCpuOpsBase.matmulWeightTransposed`, SKEEP-007
Tier 0); the decoding reference kernel is the last resort, reached only for an encoding with no
scalar kernel at all.

A backend module declares its service the usual way, e.g. `skainet-backend-native-cpu` ships
`META-INF/services/sk.ainet.backend.api.kernel.ViewKernelPack` naming
`FfmRowMajorKernelPackFactory`. `ServiceLoader` needs a public no-arg constructor, which a Kotlin
Expand Down
16 changes: 8 additions & 8 deletions docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ Each cell is the best (highest-priority) provider that serves `Float32 × format
|===
| Weight format | JVM | Android | Native·linux | Native·apple | JS/WASM

| `Float32` | native-ffm | scalar | scalar | scalar | scalar
| `Float32` | native-ffm | scalar | scalar | scalar | wasm-simd128
| `BFloat16` | native-ffm | scalar | scalar | scalar | scalar
| `Q8_0` | native-ffm | native-jni | native-cinterop | native-cinterop | scalar
| `Q4_0` | native-ffm | native-jni | native-cinterop | native-cinterop | scalar
| `Q4_K` | native-ffm | native-jni | native-cinterop | native-cinterop | scalar
| `Q6_K` | panama-vector | native-jni | native-cinterop | native-cinterop | scalar
| `Q5_K` | native-ffm | native-jni | native-cinterop | native-cinterop | scalar
| `Q5_1` | native-ffm | native-jni | native-cinterop | native-cinterop | scalar
| `Q5_0` | native-ffm | native-jni | native-cinterop | native-cinterop | scalar
| `Q8_0` | native-ffm | native-jni | native-cinterop | native-cinterop | wasm-simd128
| `Q4_0` | native-ffm | native-jni | native-cinterop | native-cinterop | wasm-simd128
| `Q4_K` | native-ffm | native-jni | native-cinterop | native-cinterop | wasm-simd128
| `Q6_K` | panama-vector | native-jni | native-cinterop | native-cinterop | wasm-simd128
| `Q5_K` | native-ffm | native-jni | native-cinterop | native-cinterop | wasm-simd128
| `Q5_1` | native-ffm | native-jni | native-cinterop | native-cinterop | wasm-simd128
| `Q5_0` | native-ffm | native-jni | native-cinterop | native-cinterop | wasm-simd128
|===

== Mapped serving (row-major, off-heap)
Expand Down
1 change: 1 addition & 0 deletions docs/modules/skeep/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@
** xref:skeep:004-virtual-tensor-layout.adoc[SKEEP-004: Virtual tensor layout]
** xref:skeep:005-schedules-structured-concurrency.adoc[SKEEP-005: Schedules — structured concurrency]
** xref:skeep:006-input-frontends-and-model-identity.adoc[SKEEP-006: Input front ends and model identity]
** xref:skeep:007-browser-lane.adoc[SKEEP-007: The browser lane]
Loading
Loading