A small vision-language model, running locally, that hears a commercial VR game, looks at the frame the game just rendered, decides what to do, and does it through a simulated humanoid body. The brain is one 2B model, LFM3-Omni-2B. The body is a simulated Unitree G1 that the game sees as a player. The whole stack, game included, runs on a single GPU in ~6.5 GB of VRAM.
clank-demo-trimmed.mp4
What works: one task step, end to end. JobBot asks for his card to be taken. The model hears him, decides to take the card and finds it in the frame. The body reaches, closes the hand, and the game registers the grab.
One number: on a measured run with the game live, the generated reach ended 12.2 mm from its target. The hand-written reach it replaced missed by 49 to 79 mm.
What doesn't yet: anything past that first step. I have not scored the runs, so there is no success rate on this page. Every run since the generated reach replaced the hand-written one has grabbed the card; that is my observation, not a measurement.
The hard part turned out to be geometry rather than the model: a head that will not hold a commanded pose, and a pixel that goes stale the moment it is marked.
Stack and role: solo project. The application in clank/ is Python 3 with the standard library only. The body runs in its own environments with MuJoCo 3.17 and ONNX Runtime, the depth layer is C and GLSL, and NixOS and Proton sit underneath. NVIDIA's GEAR-SONIC and ARDY-G1, and Liquid AI's LFM3-Omni-2B, run as published. Nothing here is trained or fine-tuned.
Clank is a small physical-AI experiment. I wanted to know how much of a task a 2B model can carry when nobody hands it state. No object positions, no task list, no goal written by me: only what it hears and what it sees. The game is the examiner. Either the fingers close on the card or the card stays on the table.
A VR game is a fair test, not just a convenient one. It renders, it has physics, it fails a grab the way a real one fails, and it has no idea anything unusual is driving it. The relay writes the robot's head and hands into the game's OpenXR session, where a headset's runtime would put them. I changed no game files.
The model's part ends at a goal in words and a located target. It never writes a joint or a pose; everything after that is code.
The game's audio goes to a PipeWire sink that only the game feeds, so nothing else on the desktop can be mistaken for JobBot. A speech gate cuts out each line, and the model transcribes it in its own call (238 ms prefill, 310 ms decode). Asking for the words and the decision in one call got the decision wrong, so they are two calls now.
The game never exposes its eye target. RenderDoc can capture a frame, but replaying one takes about 16 seconds. So I wrote a Vulkan implicit layer that copies the depth target after the final eye pass, converts it to metres on the GPU, and leaves a 480×270 depth map and a 960×540 colour image in shared memory. The copy costs 13.9 ms at the median.
The trap took a while to find: the game writes that target twice per frame, once to clear it and once to add the other eye and its glow. Inject into both and the preview alternates between dim and bright frames.
Each turn the model gets the newest still and one prompt with what was heard, the state of its hands and the last outcome. It replies with a goal in its own words:
{"goal": "take the card from JobBot", "object": "card", "destination": "none", "side": "right"}I gave it a list of six actions first, and it did not work. On four live stills it answered hold, release, release, look, and never grabbed. Asked what it was trying to do, it named taking the card on all four. So the model states the goal, and skill_from_goal reads the verb out of the model's own sentence in code. This is the part of the design I would defend hardest.
It also remembers the run. Every call carries every line heard and every earlier situation, reply and result, in order. History is capped at about 8k tokens, and a heard line is never trimmed. Only the newest still is an image, because a pixel from last turn is a guess.
Each run writes llm.jsonl with the exact prompt and raw reply of every call, so a bad decision can be replayed instead of guessed at.
The prompt, the memory rules and the checks around the model live in clank/brain.py, and the measurements behind them are in research/brain-tools.md.
Finding the card takes seven more calls to the same model. The first asks which quadrant holds it. Five more ask for its box in five crops of that quadrant, sent at once to the server's four slots. The last describes what is in the box, and code checks that against the noun.
A locate takes about 1.6 s, and on a measured frame it put the card 21 px from where I had marked it. The pixel is then unprojected through the measured head into a point in metres.
The model is bad at searching a whole frame and good at answering about a crop, so code does the cropping (spike/wbc_game/locate.py). Details and numbers: research/locate.md.
The goal becomes an intent: a skill, a side and a target in metres. A hand action becomes one constraint, right hand at the located point with both feet planted. ARDY-G1 turns that into 52 frames at 25 Hz, and SONIC tracks them on the simulated G1 in MuJoCo (50 Hz policy, 500 Hz PD loop).
The achieved poses leave as 376-byte packets for the relay, and the same poses draw the body tile on the dashboard, so the game and I are looking at the same robot.
Most rows have an earlier attempt beside them, because that comparison is the only reason to believe the number. Each number comes from the source in the last column.
| Stage | Earlier attempt | This | Conditions | Source |
|---|---|---|---|---|
| See | RenderDoc frame, about 16 s | 13.9 ms median, p95 24.8 ms, transport 1.4 ms, 29.8 fps | 480×270 depth, 960×540 colour | research/evidence/depth-stream/latency.txt |
| Hear | one call for words and decision, wrong decision | 238 ms prefill + 310 ms decode, split into two calls | one utterance | research/hearing.md |
| Decide | six-action list: 0 grabs in 4 | goal in words, verb recovered in code: 4 of 4 | four live stills, one sentence | research/brain-tools.md |
| Decide, cost | n/a | 210 ms prompt + 203 ms decode | one still plus the sentence | research/brain-tools.md |
| Locate | full frame 360 px; crop with a VL model 22 px in 19.3 s | 21.0 px in about 1.6 s, seven calls (2.4 s before the reads ran in parallel) | one live frame | research/locate.md, commit 1254d59 |
| Generate | hand-written reach tracked by the policy, 49 to 79 mm | 16.1 mm commanded, 12.2 mm achieved at the hand, 0.313 s, 0.94 GB | game and layer live, 52 frames | research/evidence/ardy-in-the-loop/README.md |
| Track | n/a | 44.2 Hz with the game running, 44.5 Hz without; the body adds 177 MiB of GPU memory | two render views | research/r6-co-residency.md |
| Drive | n/a | look_left −28.6°, look_right +54.3°, from the body's −86.8° baseline |
mode body, 50 Hz relay |
docs/archive/specs/2026-10-30-targeting-adapter.md, baseline in docs/archive/plans/2026-10-30-hands-follow-the-head.md |
| Walk | n/a | 39.79 m path in 60 s, 0.663 m/s, no fall | SONIC default, headless, no game | research/sonic-gate-results.md |
The control tick costs 6.8 ms with ONNX Runtime's default threading (47 threads, about 19 of 20 CPU threads busy) or 17.66 ms single-threaded at 32% of one core. Both fit the 20 ms budget. Single-threaded is the default, because the other setting starved the game and the desktop (research/wbc-policy-proposal.md).
Test machine for every row: one 8 GB NVIDIA GPU, 20 CPU threads, NixOS.
The failures are the useful part, so here they are with their numbers.
Picking from a list. The six-action list, as above. This one cost me the most time, because it looked like a model quality problem for a while and was a question-shape problem.
Asking the locator to search. This model collapses any image-global grid toward the centre. It picked the right third of the frame 3 times in 12, the right cell 0 in 12 on a nine-cell grid, and 0 in 12 on a 4×4. The same question about a crop I supply scored 44 of 44 on the tray. So the crop is my job, not the model's.
Telling it when it was wrong. Told reach right card -> not_located, it re-issued reach on three stills out of three, and writing the rule into the prompt changed nothing. When an action needs a pixel the loop does not have, the loop now runs the search itself.
Matching by description alone. A box described as a "green robot computer" matched "job cartridge" on the shared colour word and pointed at a monitor. Identity is a vocabulary comparison in code now, and an unknown noun is refused even when the box was right.
Trusting it with a greeting. One turn heard an introduction, "welcome to the job simulator, my name is JobBot", and planned place(job cartridge into console), a chore nobody had given it. The schema caught it (place needs a side) and held. The loop refused the plan; the model never withdrew it. An utterance that names no task no longer reaches the model at all.
Trusting the pipeline to stay up. Two live runs died on the body's control socket, because the new viewer answered the old one's request and then unlinked the socket on its way out. The run now waits for the socket, restarts a child that died, and the relay retries a refused socket for 20 seconds instead of giving up.
The ear. In run 190131-679de2, the diary ends on no word from JobBot after the grab, and that is wrong: the recording has him asking for a job cartridge at 42 s. The speech gate's threshold came from a window that included his loudest line, so it heard nothing in the quiet one. The gate now takes its floor from the whole period before it opens.
- It only does one thing. No level has been finished. Walking to a target, the cartridge rack, the lever and the console are all untried.
- Memory is text only. Earlier stills are not kept, and history is capped near 8k tokens to hold prefill near 1.5 s.
- The torso does not turn. It started 63.9° off the card and the reach did not turn it, so the body reaches sideways at something it is facing away from.
- The head will not hold a commanded pose. Held at one target for ten seconds it wobbled 11.7° of yaw, and the card moved 1009 → 1122 → 865 → 1007 px inside one four-command reach. The target is kept in metres and projected back through the measured head every tick, not remembered as a pixel.
- The body does not walk in the game. It is clip-anchored: one generated window reaches; it does not step.
- The body loop runs at 44.2 Hz against a 50 Hz design point (44.5 Hz even without the game), so the game sees 0.89× real time.
- The clasp threshold is a guess. 80 mm has fired twice, at 60 and 78 mm, and I have not swept it.
- Controller orientation is unverified. Positions are exact, but the wrist frame is not an OpenXR grip, so the in-game aim may be rotated by a fixed offset.
Score the card grab over a fixed number of runs from a reset save and publish the success rate. Then hand JobBot's second request, the job cartridge, to the same loop.
You need NixOS with Steam, your own copy of Job Simulator (not included), Monado built with the remote driver, an NVIDIA GPU, and llama.cpp serving LFM3-Omni-2B-Q8_0 with its mmproj on port 11434. The server flags are pinned in AGENTS.md; a server started with defaults answers differently.
git clone https://github.com/HasNate618/Clank && cd Clank
cp config.example.toml config.toml # steam_bin, game_exe, openxr_active_runtime, launch ids
nix develop --command spike/e5/build.sh # the Vulkan layer
./run.sh # dashboard on http://127.0.0.1:8765Then press Start run. It boots the stack if it is down, refuses before spending anything if the model server is not up, wipes the game's save so the job starts at its first frame, stands the body up, and starts the loop. It takes about 25 s to reach running. One run leaves a directory with the diary, the frames, llm.jsonl and a video.
Setup, ports, VNC and the control table are in docs/setup.md.
XDG_RUNTIME_DIR=/tmp/clank-xdg python3 -m unittest discover -s tests -t .1532 tests in about 135 seconds, standard library only: no network, no GPU, no game. They cover the command bus and its validator, the wire format, the run state machine, the recording manifest, the depth reader, the dashboard routes and the transcription boundary. They do not cover what needs hardware. The layer's latency, the policy's tracking and anything touching the game are measured by hand and recorded in research/.
| Path | What |
|---|---|
clank/ |
The application. brain.py holds the prompt assembly, the session memory and every model call; the rest is config, the session orchestrator, the run state machine, the dashboard, the pose relay, recording, the ears and the depth reader. Standard library only. |
spike/e5/ |
The Vulkan depth layer: C, GLSL, the launcher, the calibration. |
spike/wbc_game/ |
The loop (auto.py), the locator, the reach adapters, the pixel picker. |
spike/ardy/, spike/e3/ |
The motion generator and the MuJoCo body. Each has its own virtualenv. |
spike/c0/ |
Scripted pose clients. Still the fastest way to tell a body fault from a pipe fault. |
research/ |
The measurements: one document per spike, with conditions, negative results and evidence. |
docs/ |
The brain and perception write-ups, designs and their archive. docs/README.md says which is current. |
Mine: the system design, the perception chain (the Vulkan depth layer, the locator, unprojection through the head), the turn loop and the checks around the model, and every experiment and measurement in research/, failures included. Much of the code was written with the pi coding agent (DeepSeek V4.1 Flash), and I checked it against the tests and those measurements.
Not mine: GEAR-SONIC (the whole-body controller) and ARDY-G1 (the motion generator) are NVIDIA's, under the NVIDIA Open Model License. LFM3-Omni-2B is Liquid AI's. I run all three as published, with no training, fine-tuning or changes. Monado's remote driver carries the poses into the game; I patched Monado for a full visibility mask and a 60 Hz null compositor (spike/e5/monado-no-visibility-mask.patch). Job Simulator is proprietary and not distributed here.
MIT for this repository; see LICENSE.

