The lookahead head forecasts future expert-routing distributions. It does not predict the future token identity and is not a draft decoder. Its output guides expert-weight residency rather than entering the language-model readout directly.
That architectural separation does not make prediction errors harmless. A wrong prediction can waste I/O, displace useful weights or cause a later miss. Under a shared-fallback policy, changed residency can also change the model output. During training, lookahead losses can affect the backbone unless their input is detached.
Current computation and future prediction use separate paths. MoE, mixture of experts; FP32, 32-bit floating point. Dashed future-request edges carry control, not a contribution added to the current token's activation output.
Use
Here lookahead_steps configuration value.
The head additionally emits slot zero. Its output is
when layer_aware_lookahead=False, or
when that flag is true. Softmax is over the final expert axis. Group
| Axis | Meaning | What it is not |
|---|---|---|
| Current input-token position | Recurrent depth | |
| Future-token offset | Next physical layer | |
| Recurrent physical group | A new parameter block at every iteration | |
| Repeated call of a recurrent layer | Future token position | |
| Expert ID within a physical MoE layer | A global expert network shared by all layers |
Future supervision and prefetch use slots
flowchart TB
State["Current token state at position t"] --> Head["One lookahead projection"]
Head --> Q0["Slot 0: interface output"]
Head --> Q1["Slot 1: experts at token t+1"]
Head --> QK["Slot K: experts at token t+K"]
Q1 --> Groups["Optional recurrent-group axis"]
QK --> Groups
Groups --> Expert["Distribution over expert IDs"]
R["Recurrent iteration r"] -. "Changes teacher scope, not<br/>token horizon" .-> Groups
Prediction axes. Token horizon, physical group and recurrent iteration have different meanings even when their numeric sizes happen to coincide.
Let
with bias-free matrices
It uses SiLU, not a SwiGLU body, and one projection that is reshaped into offset/group slices. There is no current lookahead.hidden=256 setting or collection of separately instantiated per-layer output modules. Torch head, MLX head.
The exact matrix parameter count is
For the explicitly named bundled translation configuration,
These are configuration/algebraic counts, not measured latency or a claim that every 50M recipe has this shape. The matrix FMA count per input token has the same leading expression; SiLU, softmax, tensor handling and scheduling add work. Named configuration.
flowchart TB
H["Hidden input: B × T × d"] --> L1["Bias-free linear: d to floor(d/4)"]
L1 --> Act["SiLU activation"]
Act --> L2["Bias-free linear: m to E(K+1)G_out"]
L2 --> Shape["Reshape token, offset, group,<br/>expert axes"]
Shape --> Prob["Softmax over expert axis"]
Prob --> Future["Use future slots 1 through K"]
The implemented lightweight prediction head. Its hidden width follows the backbone rather than a separate fixed-width knob.
For active recurrent execution, the logical input is the state after the prelude, before recurrent refinement. For the depth-one ordinary path, it is the state after physical block zero. Torch computes the head from the saved post-prelude state after the recurrent helper returns; MLX evaluates it at that observation point before entering the recurrent portion. Do not confuse the source state with an identical backend scheduling order. Torch calls, MLX calls.
The MoE dispatch router explicitly uses FP32 softmax on both backends. The lookahead head currently differs: MLX casts its projected logits to FP32, while Torch applies softmax without an explicit cast in this head. Torch therefore inherits the projected logits' dtype. These operations should not be documented as unconditional FP32 parity. Torch softmax, MLX cast.
The teacher is a soft future routing distribution, not just hard expert IDs. It is detached before lookahead supervision. For each physical layer, the current modes are:
lookahead_teacher_scope="first": use the first recorded dispatch distribution.lookahead_teacher_scope="iterations": average the recorded distributions across that physical layer's iterations.
The accepted string is "iterations", not "all". Averaging probabilities across recurrent calls is not taking a union of their hard selected IDs. The separate future-union objective below concerns token offsets. Scope validation, teacher sources.
When predictions lack a group axis, teacher probabilities are averaged across physical layers. With a group axis, the trainer selects physical recurrent layers
Input detachment is a different decision:
lookahead_input_detach defaults to false. MLX applies the switch at the head call. In the reviewed Torch call sites, the flag is not consulted: changing it alone does not create a stopped-gradient input there. This is an implementation-parity boundary, not a feature that documentation can assume works identically. Config, MLX switch, Torch call.
flowchart TD
Source["Post-prelude or block-zero hidden state"] --> Switch{"Input detached?"}
Switch -->|no| Live["Prediction loss can update upstream backbone"]
Switch -->|yes, MLX switch| Frozen["Stop gradient at head input"]
Live --> Head["Lookahead parameters"]
Frozen --> Head
Future["Actual future router probabilities"] --> Teacher["Detached teacher distribution"]
Head --> Loss["Soft CE and coverage proxies"]
Teacher --> Loss
Loss -. "Update prediction branch" .-> Head
Teacher -. "No teacher-target gradient" .-> Stop["Teacher is not pulled toward predictor"]
Gradient ownership. Teacher detachment prevents target feedback. Input detachment prevents a different path through the prediction input. They are not substitutes.
For valid future offset
The loss averages valid token/group examples within each offset, then averages available offsets:
Offsets with no valid pair are skipped. Expanding cross-entropy gives
Because
The displayed formula assumes unmasked pairs. Document boundaries and valid-token masks determine which pairs may enter the mean on the relevant trainer path. Their switches must be recorded with a packed-document experiment; a temporal offset crossing unrelated documents is not a valid continuation target.
Let
The division by
For a horizon of
The implemented proxy is
The maximum across offsets is not a normalized categorical distribution or a probabilistic union formula. It summarizes each expert's largest forecast probability over the prefetch window. This term rewards coverage of distinct future needs, while soft cross-entropy preserves offset-specific structure. Union implementation.
The trainer combines the available terms as
It adds them only when prediction/teacher tensors and future offsets are available. Assembly.
The runtime does not use every prediction for every prefix position. The Torch scheduler selects the final token row, removes slot zero and takes top num_experts_per_tok predictions per future offset. With group-aware caching it maps group IDs to physical-layer page keys. It submits earlier offsets before later ones and deduplicates IDs. Scheduler.
For one sample and one group, before page-layout expansion, the union of
flowchart TB
Q["Final-token prediction tensor"] --> Future["Keep slots 1 through K"]
Future --> IDs["Top-k IDs for each offset and<br/>group"]
IDs --> Keys["Map group IDs to physical page<br/>keys"]
Keys --> Queue["Bounded asynchronous request queue"]
Queue --> Store["Backend-specific page or<br/>cluster load"]
Store --> Resident["Admit weights under residency<br/>budget"]
Resident --> Demand["Future actual dispatch checks<br/>residency"]
Demand --> Exact["Execute requested expert if<br/>available"]
Demand --> Miss["Otherwise apply selected miss<br/>policy"]
Prediction-to-residency path. KV, key–value. None of these requests writes an attention KV entry or directly supplies a language-model activation.
A queue slot can contain a list of expert IDs; queue depth is not automatically a per-token byte or expert cap. The scheduler currently derives its selected count from num_experts_per_tok; there is no lookahead.top_k or prefetch.budget_per_token=8 field in this code path. Queue construction.
Inference mode removes autograd bookkeeping, not the need to compute or hold prediction tensors. Prefill output has a
| Prediction/runtime event | Possible consequence | What cannot be concluded automatically |
|---|---|---|
| A needed expert was not predicted | Miss, demand load or fallback | Miss rate is not identical to head prediction error |
| An unused expert was predicted | Extra loading, cache occupancy or eviction | One prediction does not always equal one physical read |
| Correct cluster, wrong expert | Potential neighbor reuse | Not guaranteed if the requested neighbor was not admitted or was evicted |
| Exact miss policy waits for expert | More latency on a cold miss | Not a zero-cost miss |
| Approximate fallback handles a miss | Shared output substitutes for the routed contribution | Not guaranteed output equivalence to full residency |
Torch's clustered path selects requested/budgeted pages before materialization. MLX's cluster path loads/evaluates the cluster before bounded admission. Thus “one correct cluster prediction covers every member” is not a backend-independent guarantee. Torch, MLX.
Likewise, the residency benchmark is not a clean head-on/head-off ablation. Its LRU and planned modes change eviction policy; LRU can still receive predictive prefetches. The observed miss-rate difference cannot be attributed solely to the lookahead network. Selection misses also depend on cache budget, evictions, cluster layout, scheduling and warm/cold state.
The named CPU/float32, 16-prompt benchmark reports selection miss rates 0.2175 for plain LRU and 0.1394 for planned residency. These are measured downstream runtime counters on the stated checkpoint/workload, not standalone forecast accuracy. Benchmark JSON.
The 15M, one-seed iteration-cluster agreement summarized in the changelog is a separate experiment. The synthetic topic-cluster purity of 0.999 at locality coefficient 0.3 is another metric on another tiny model. They should not be combined into a single end-to-end accuracy claim. Loop report summary, synthetic record.
| Actual configuration field | Class default | Role |
|---|---|---|
lookahead_steps |
2 |
Number |
layer_aware_lookahead |
False |
Add recurrent physical-group output axis |
layer_aware_cache |
False |
Interpret group-aware page predictions |
lambda_lookahead |
0.1 |
Soft-target CE coefficient |
lambda_lookahead_topk |
0.15 |
Top-k target-mass coefficient |
lambda_lookahead_union |
0.05 |
Future-union coefficient |
lookahead_teacher_scope |
"first" |
"first" or "iterations"
|
lookahead_input_detach |
False |
Optional input stop-gradient; observed backend boundary above |
num_experts_per_tok |
1 |
Scheduler's per-offset selected count, before recipe override |
prefetch_depth |
2 |
Queue-related setting, not future horizon |
Sources: head-related defaults, loss defaults, teacher/input policies, base top-k. Stored checkpoints and planner recipes may override these values.
The standalone prefetch_depth.json benchmark is not present in the repository. Useful next measurements are offset/group-specific top-k recall, predicted-page precision, bytes loaded per useful expert, queue delay, and task quality under each miss policy. A matched head-off/head-on test with the same eviction policy and cache budget is needed to isolate the predictor's contribution. These are proposed measurements, not results already obtained.
Notation: MoE, mixture of experts; MLP, multilayer perceptron; CE, cross-entropy; KL, Kullback–Leibler divergence; KV, key–value; FMA, fused multiply-add; FP32, 32-bit floating point; LRU, least recently used;
