Repository navigation
Docs: adding Living Weights to a family - #905
Conversation
…unchanged The learner, its whole-lesson and routed paths, the pocket and its fits and helpers move from families/nemotron to a `living` module that both builds import, as 19 renames with no change to their contents. The family-neutral half of learn_shape.zig (block sizes, Adam settings, direction lengths) moves with them; Nemotron's learn_shape.zig keeps where its blocks sit and names the shared constants from the module. Nemotron's files import the moved ones through the module. The learner's tests now also run on Linux, as `zig build test-living` and under test-cuda-host. No behaviour change.
… layout, neuron kind and shard format A family now describes itself once, as data (living/families.zig): where its shared experts sit and their tensor names, the neuron's activation (relu^2, or SwiGLU with an optional per-row gate), the quantization, the config keys and the object they live in, and its MTP head's expert. Its host gives the learner a layout and the pocket hooks (living/family.zig). Nemotron is the one family listed, and its lessons, routed blocks, edits and probes are unchanged. - The pocket reads its layers and width from the family's layout, not from Nemotron's config, and the learner offers routed blocks, edits and probes only to a trainer with `sites` (Nemotron's). - living/shards.zig widens any family's experts at its group and width; Nemotron's writer is now a wrapper over it. shard_edit sets and reads nested config keys (text_config). - A SwiGLU neuron fitted as u is written as gate row 16u and up row u/16: relu(u x)^2 on its fact's rows, at most 8e-5 of it at a 0.5 margin; a gated family's outputs fit gate * act(x). No listed family uses it yet; the tests use a stand-in family shaped as Qwen3.8 Flash Next's shared expert. - The server's view reads layers, pockets and shards through the family its config's model_type names, and native/nemotron_slide.zig is native/slide_adapter.zig. - pocket.zig's quiet refit and its loading move to pocket_quiet.zig and pocket_load.zig, for the 600-line rule. - Tests: the generic writer equals affine4_host at 4 bits in groups of 64, Nemotron's save test over the wrapper, a SwiGLU neuron fitted and stored at 4 bits in groups of 32, and a whole pocket lesson over a stand-in host for each activation (test-living, also under test-cuda-host).
…ig family What a family gives the learner in zig/src/living, in order: its description as data (the pocket beside an MoE layer's shared expert, behind a gated one, or in a dense MLP), the pocket in every pass, the host's hooks, the trainer's API with its keeps and backward kernels, the shard writer, --slide on the native host, the receipts a PR should carry, and the traps found bringing up Flash Next and the Qwen3.8-27B. The recipe book links it.
|
Thank you, @ZackSample, for writing the family recipe around shared metadata, the serving forward and saving into the model's own shards. We checked f47c203 separately from its #878 parents. The useful new piece is the recipe and its index entry. One claim needs narrowing: We'll fold that wording and the inherited family-validation guards in at landing if you prefer, preserving your docs commit's credit. We'll check the recipe against the corrected shared learner and run the documented untaught, retained-fact, quantized save/reload and lane-equality cases as each family is integrated. Posted by Ash's AI Assistant Synthena |
|
Thanks, @ashhart. Yes, please fold the narrower wording in at landing (the tested input range and the 1e-6 relative tolerance, with the leakage and quantized-save checks kept separate), along with the family-validation guards. |
|
Thank you, @ZackSample, for the Living Weights family recipe and for aligning its promises with the tests. The recipe moves with Living Weights to 1.1.1, the next patch release. It describes the shared family data, pocket wiring, trainer, persistence and required receipts, keeping the stated range and relative tolerance separate from leakage and quantized-save checks. We need the recipe to match the final learner and the qualified family ports before it ships. Your credit stays with that work. Posted by Ash's AI Assistant Synthena |
… and its tests hold them A SwiGLU neuron written as gate sharp u and up u / sharp fires z^2 sigmoid(sharp z), so it equals relu(z)^2 only well above zero: the recipe now gives neuron.zig's tested points and relative tolerance (1e-6 at z = 1, 1.5, 3) in place of a universal 1e-7, the 0.0017 gap at z = 0.1, the silent rows' leak bound (0.25 sigmoid(-8)), and when the power-of-two sharp gives u back exactly. Family.check is described as family.zig refuses today: zero or non-dividing code widths, empty groups or pockets, and a sharp that is not a whole power of two. The Flash Next and 27B numbers are marked as from their unlanded branches, and the receipts add a plain zig build. CHANGELOG line for #905. Co-authored-by: ZackSample <134018356+ZackSample@users.noreply.github.com>
…moves to 1.1.1 with the series it describes This reverts the merge 3295baaca (its first parent kept) and our recipe corrections in 949667a33.
…Next on CUDA's lines say what was measured The six Living Weights rows (#876, #877, #878, #891, #905 and #992) leave Unreleased with the series. The EXL3 operators' line names GLM-5.3-Flash's two-rank engine as the family that serves them; #995's and #979's launches have their records on nvcc 13.0 and 13.2; and Flash Next on CUDA states what a GB10 served from the MLX 4-bit checkpoint, text and PNG images.
|
Thank you, @ZackSample, for the Living Weights family recipe and for aligning its promises with the tests. The recipe moves with Living Weights to the next patch release that passes its gates, 1.1.1 if they pass in time. It describes the shared family data, pocket wiring, trainer, persistence and required receipts, keeping the stated range and relative tolerance separate from leakage and quantized-save checks. We need the recipe to match the final learner and the qualified family ports before it ships. Your credit stays with that work. Posted by Ash's AI Assistant Synthena |
2 participants
(#878). Title: Docs: adding Living Weights to a family
Stacks on #878: the page describes the
livingmodule,Familyand the host and trainer hooks that #878 adds, so itopens against #878's branch (or as a draft until #878 lands; the diff to review is the one commit).
What this changes
As you suggested on #878, the list of what a host and a trainer give the learner is now a recipe page beside
adding-a-zig-family.md:docs/recipes/adding-living-weights-to-a-family.md, linked from the recipe book's README.No code changes.
It goes in order, as the Zig family recipe does:
living/families.zig): a table of everyFamilyfield for Nemotron, Flash Next and adense model (the Qwen3.8-27B), where the pocket goes (beside an MoE layer's shared expert, behind a gated shared
expert, or in a dense MLP), and why a SwiGLU neuron is written as
sharp uandu / sharpwith a 0.5 margin."draft": false, zeroing the pocket onuntaughtstreams, a config check that accepts a taught folder, the MTPhead widened with zeros.
family.zig), and theLayout.on a GB10, and the checks: the trainer's forward byte for byte against serving, causality, packs equal alone, and
finite differences as a smoke check only.
living/shards.zig) and its test.--slideon the native host:slide,learnModel,linked, and the refusals.higher ones on prompts no lesson read, checks that only see single-turn questions (Living Weights: facts learned and checked in conversation, not only asked cold #891), captured kernel sets.
The Flash Next details refer to #895 and #896; the 27B is our port on #726, not sent yet, and is named as such.
Checks
python3 tools/zig/lean_check.py: 527 problems against 527 onlw-b-core(0 new). Prose lines are at most 120columns outside tables, as in the other recipes.