Historical implementation contract for the Live Wallpaper sampler/extension. See
README.mdfor current installation, usage and configuration.
Companion to: gnome-live-wallpaper-design-doc.md (read that first for rationale — this document is for implementation, not motivation)
This document uses MUST / MUST NOT / SHOULD / MAY in the RFC 2119 sense. Every numeric constant below is a decision, not an example — implement the exact value given. If a value is genuinely unspecified for a case you hit, stop and ask rather than inventing one; do not silently pick a "reasonable" default for anything covered by this contract. Section 18 lists things you MUST NOT build regardless of how naturally they'd extend the design.
In scope: the telemetry sampler, the audio analyzer, the IPC layer between them and the Shell extension, the GNOME Shell extension itself (particle rendering, placement, fullscreen/multi-monitor handling), the color system, and the wallpaper layering + palette-matching feature.
Explicitly out of scope for the first implementation pass (see §18 for the full list): X11 support, any non-GNOME compositor, a DMA-BUF zero-copy render path, an external Rust/wgpu renderer bridged via GStreamer ("Path B" in the design doc).
- Target: GNOME Shell 50.x, Wayland session only. GNOME Shell 50 removed X11 support — do not add an X11 code path, do not guard for
XDG_SESSION_TYPE=x11. - Telemetry sampler + audio analyzer: Rust, stable toolchain.
- GNOME Shell extension: TypeScript compiled to GJS-compatible JS (matching the toolchain used by the
weather-effectreference extension), targeting the GNOME Shell 50 extension API. - Hardware assumption for GPU telemetry: AMD
amdgpudriver (iGPU). Do not add an NVML/NVIDIA code path unless explicitly asked.
live-wallpaper/
sampler/ # Rust binary — telemetry + audio + IPC server + image ops
src/main.rs
src/telemetry.rs
src/audio.rs
src/scene_state.rs
src/ipc.rs
src/layering.rs # cutout generation + palette extraction
extension/ # GNOME Shell extension
src/extension.ts
src/lib/sceneClient.ts # IPC socket client
src/lib/sceneState.ts # type defs + defaults, mirrors scene_state.rs
src/lib/monitorManager.ts
src/lib/obscurationManager.ts
src/lib/particleLayer.ts
src/lib/wallpaperConfig.ts # per-wallpaper JSON store
src/prefs.ts # GTK4/Adwaita prefs window, incl. cutout picker
schemas/ # GSettings schema
metadata.json
Do not deviate from this layout without flagging it — file names are referenced by exact path elsewhere in this document.
Both the sampler's telemetry path and its audio path MUST publish this exact shape. All floats are normalized to [0.0, 1.0] unless stated otherwise; values MUST be clamped to range before publishing.
interface Pulse {
kind: "disk_io" | "beat";
strength: number; // 0.0–1.0
timestamp_ms: number; // monotonic clock, ms
}
interface Stream {
direction: "rx" | "tx" | "pan_left" | "pan_right";
strength: number; // 0.0–1.0
}
interface SceneState {
source: "telemetry" | "audio";
intensity: number; // system: blended CPU+RAM load | audio: RMS
population: number; // system: RAM used fraction | audio: mid-band energy
heat: number; // system: normalized temp | audio: spectral centroid position
pulses: Pulse[]; // events since last publish, MUST be cleared each publish
streams: Stream[]; // current directional/pan state, replaces previous value
spawn_rate: number; // system: process churn | audio: treble density
gravity_bias: number; // system: 0 at idle | audio: bass-driven
timestamp_ms: number; // monotonic clock, ms, when this state was computed
}intensity for the system source MUST be computed as 0.6 * cpu_load + 0.4 * ram_load, both already normalized 0–1. This weighting is a decision, not a placeholder — do not rebalance it without flagging.
- Transport: Unix domain socket at
$XDG_RUNTIME_DIR/live-wallpaper/scene.sock. The sampler binary is the server (creates and binds the socket on startup, removes it on clean shutdown). The extension is the client. - Framing: newline-delimited JSON. Each message is one JSON object followed by
\n. No length prefix. - Message types,
typefield required on every message:scene_update— body is aSceneStateobject as above. Sampler pushes this continuously; no request needed.generate_layers(extension → sampler) —{ "type": "generate_layers", "wallpaper_uri": string, "output_dir": string }.layers_result(sampler → extension) —{ "type": "layers_result", "candidates": [{ "id": "low"|"balanced"|"high", "coverage": number, "path": string }] }. Candidates whose coverage fails the sanity check in §11 MUST be omitted from this array, not included with a flag.error—{ "type": "error", "code": string, "message": string }.
- Reconnect policy (extension side): on disconnect or connect failure, retry with backoff
1s, 2s, 4s, 8s, 10s, then hold at 10s intervals indefinitely. MUST NOT crash the extension or throw an uncaught error on any connection failure — log viaconsole.warnand continue rendering the last knownSceneState, degrading toward the idle end of the color/intensity range if no update has arrived in 5 seconds.
Poll interval: 300ms, fixed. Do not make this configurable in the first pass.
| Field | Exact source | Notes |
|---|---|---|
| CPU load | sysinfo crate, global CPU usage |
normalize 0–100% → 0.0–1.0 |
| RAM load | sysinfo crate, used/total |
0.0–1.0 |
Disk I/O → pulses |
delta of /proc/diskstats, fields 6 and 10 (sectors read/written) between polls |
emit a disk_io pulse when delta exceeds a fixed threshold of 512 sectors between polls; strength = min(1.0, delta_sectors / 8192) |
Network → streams |
delta of /proc/net/dev, RX and TX bytes |
emit one rx and one tx stream entry every publish (not just on change), strength = min(1.0, delta_bytes / 5_000_000) per 300ms window |
CPU temp → heat |
first temp*_input under /sys/class/hwmon/hwmon*/ whose temp*_label matches Tctl or Tdie (AMD) |
normalize: heat = clamp((temp_c - 30) / 70, 0, 1) — 30°C floor, 100°C ceiling |
| GPU load | /sys/class/drm/card0/device/gpu_busy_percent |
0–100 → 0.0–1.0 |
Process spawn/death → spawn_rate |
PID set diff between polls (list /proc/[0-9]*) |
spawn_rate = min(1.0, (spawned + died) / 20) per 300ms window |
| Battery | /sys/class/power_supply/BAT0/capacity if present, else omit from computation entirely (do not synthesize a value) |
not currently wired into any SceneState field — reserved for future use, MUST be sampled and logged but MUST NOT affect rendering yet |
population (system) = RAM load directly (no separate formula). gravity_bias (system) = 0.0 always in this version — it is a Music Mode concept; do not invent a system-mode meaning for it.
- Capture: default sink monitor stream via PipeWire's Pulse-compatible API (
cpal, Pulse backend). - Sample rate: use whatever the default sink reports (commonly 48000 Hz) — do not resample.
- FFT size: 2048 samples, Hann window, hop size 1024 (50% overlap).
- Band boundaries (Hz): bass = 20–250, mid = 250–4000, treble = 4000–16000. Bin-to-Hz mapping uses the actual sample rate reported by the sink, not a hardcoded 44100/48000 assumption.
intensity= normalized RMS of the full-band signal:floor = -60 dBFS → 0.0,ceiling = -6 dBFS → 1.0, clamp outside this range.population= normalized mid-band energy, same dBFS floor/ceiling as above applied to the mid band only.heat= spectral centroid position mapped into 0–1 across the analyzed range (20 Hz → 0.0, 16000 Hz → 1.0).spawn_rate= normalized treble-band energy, same dBFS floor/ceiling.gravity_bias= normalized bass-band energy, same dBFS floor/ceiling.- Beat detection (→
pulses): spectral flux of the bass+mid bands against a rolling average of the last 43 frames (~1s at this hop size). Fire abeatpulse when current flux exceeds1.5 × rolling_averageAND at least 120ms have passed since the last fired beat (debounce).strength = min(1.0, flux / (2 × rolling_average)). - Stereo →
streams: compare L/R channel RMS; emit onepan_leftorpan_rightentry (whichever channel is louder) withstrength = abs(L_rms - R_rms) / (L_rms + R_rms), omit entirely if the two are within 5% of each other.
States: System, Music. Start state: System.
System → Music: audio-sourceintensity(§7) stays> 0.15continuously for>= 400ms.Music → System: audio-sourceintensitystays< 0.05continuously for>= 3000ms.- While in
System, the extension renders from the most recentscene_updatewheresource == "telemetry". While inMusic, from the most recent wheresource == "audio". The sampler MUST publish both continuously regardless of which mode is currently displayed — mode switching is a rendering-side decision, not something the sampler needs to know about. - Transition MUST NOT reset particle positions/state — only the data source driving them changes.
Two independent axes computed per-particle from the current SceneState, both fed by different fields depending on mode as described in the design doc.
Hue arc, input t (a 0–1 value — heat for system-sourced state, or the bass/treble balance gravity_bias - spawn_rate remapped to 0–1 for audio-sourced state):
- Two-segment piecewise interpolation. Segment A,
tin[0.0, 0.5]: hue goes 235° → 300°. Segment B,tin[0.5, 1.0]: hue goes 300° → 375°, then take the resultmod 360. This wrap handling is required — interpolating segment B directly toward 15° instead of 375° will route the gradient through green, which is wrong. - Saturation:
S = clamp(20 + 70 * intensity, 20, 90)(percent). - Lightness:
L = clamp(15 + 45 * intensity, 15, 60)(percent).
Punctate event flash (on any pulses entry, regardless of kind): set that particle's L = 90%, S = 10% immediately, then ease back to its ambient S/L (computed above) over 200ms using ease-out-cubic. This is the same treatment for disk_io and beat pulses — do not give them different curves.
Directional accents (streams entries): rx and pan_left/pan_right-favoring-left render at hue 190° (cyan-leaning); tx and right-favoring pan at hue 20° (coral-leaning); both at S = 50% flat (not intensity-driven), regardless of the ambient hue arc — these are annotations, not part of the main population.
- On
enable(): spawn the sampler binary viaGio.Subprocess(path resolved relative to the extension's own install directory), then connect to the IPC socket per §5. Ondisable(): disconnect, then terminate the subprocess. - Actor placement: one actor per entry in
Main.layoutManager.monitors, inserted intoMain.layoutManager._backgroundGroup, geometry matching that monitor exactly. - Fullscreen: watch
Main.layoutManager.monitors[i].inFullscreenper monitor. On transition totrue, stop and disconnect that monitor'sClutter.Timeline— do not merely setvisible = false. Reconnect and restart on transition back tofalse. - Multi-monitor churn: on the layout manager's monitors-changed signal, fully rebuild the actor set (destroy all, recreate from the current
Main.layoutManager.monitors) rather than trying to diff and patch it. - Per-monitor desync: each monitor's particle system MUST seed its own local animation phase independently (e.g.
Math.random()at actor creation) even though every monitor consumes the identicalSceneState. - Render tick: driven by
Clutter.Timeline/actor.easeAsync()(GNOME Shell 50's async ease helper), not a JSsetInterval. - Particle budget: base cap 600 particles at
intensity == 0. Scale ascount = max(100, round(600 * (1 - 0.5 * system_cpu_load))), recomputed every telemetry publish. This applies in both modes — Music Mode does not get a separate budget.
- Trigger: watch
org.gnome.desktop.background'spicture-uri(andpicture-uri-dark) for changes. - On change, look up
$XDG_CONFIG_HOME/live-wallpaper/wallpapers.json, keyed bysha256(picture_uri)(hex string). If no entry exists, request cutout generation (below) before deciding a default. - Cutout generation (sampler-side, via
generate_layersIPC message): compute per-pixel luma viaY = 0.2126R + 0.7152G + 0.0722Bon the full-resolution image. Build the luminance histogram; take three threshold candidates at the 25th, 50th, and 75th percentile of that histogram — these correspond tolow/balanced/highsensitivity respectively. For each candidate: threshold below the cutoff = foreground, apply one morphological open (5×5 kernel) then one morphological close (5×5 kernel), then feather the resulting mask edge with a 3px-radius Gaussian blur. Export as RGBA PNG (original pixels where foreground, transparent elsewhere) tooutput_dir. - Sanity check, per candidate independently: compute foreground coverage as a fraction of total pixels. Discard the candidate if coverage is below
0.0005(a genuinely small object — an icon, a boat — is still a valid foreground) or above0.95(hiding particles behind virtually the whole screen is pointless). The mask is faded toward transparent as coverage approaches either bound (0.002fade on the low side so small objects get the full effect,0.02on the high side so the effect disappears gracefully). Only surviving candidates are returned inlayers_result. If zero candidates survive,layers_result.candidatesMUST be an empty array — the extension then appliesflatoverlay mode automatically and MUST NOT show the picker UI for that wallpaper. - Picker UI (
prefs.ts): render surviving candidates as selectable cards. On hover (Gtk.EventControllerMotionenter), scale the preview and draw the mask boundary viaGtk.Overlay. Selecting a candidate, or explicitly choosing "none of these," writes the result towallpapers.jsonfor that wallpaper's hash and MUST NOT prompt again for that same hash.
- Downsample the source image (or, in layered mode, only its background pixels — i.e. everywhere the accepted cutout mask is not foreground) to 100×100 using nearest-neighbor.
- Convert to HSL. Discard any pixel with
S < 0.08(near-neutral) before clustering. - If fewer than 40 pixels remain after filtering, abort match-wallpaper extraction and fall back to the
defaultpalette for this wallpaper — do not cluster on an under-populated set. - Otherwise run k-means with
k = 4on the remaining pixels in HSL space. - Sort the 4 resulting centroids by
Lascending. Centroid 0 (lowest L) becomes the idle-end hue for §9's arc; centroid 3 (highest L) becomes the peak-end hue; the median of centroids 1 and 2 becomes the midpoint used in place of the fixed 300° default. The two-segment interpolation logic in §9 is otherwise unchanged, including the wrap-around handling — apply it with these derived hues instead of the default 235°/300°/375° stops.
- Global settings: GSettings schema
org.gnome.shell.extensions.live-wallpaper, keys:enabled(boolean, defaulttrue),default-palette-mode(enumdefault|custom|match-wallpaper, defaultmatch-wallpaper). - Per-wallpaper settings: not in GSettings — stored as JSON at
$XDG_CONFIG_HOME/live-wallpaper/wallpapers.json, structure:
{
"<sha256-hex-of-picture-uri>": {
"mode": "flat | layered | replace | disabled",
"palette": "default | custom | match-wallpaper",
"cutout_id": "low | balanced | high | null",
"custom_colors": null
}
}- Cutout PNGs cached at
$XDG_CACHE_HOME/live-wallpaper/cutouts/<sha256-hex>/{low,balanced,high}.png. These MAY be regenerated (overwritten) if missing; the extension MUST NOT treat a missing cached file as an error requiring user intervention — it should silently regenerate viagenerate_layers.
- Sampler process: average CPU
< 3%on the reference AMD Ryzen 7 / Radeon iGPU system, RSS< 40MB. - Extension (Shell process contribution): average CPU
< 3%atSystemmode idle,< 8%at peak particle budget; do not exceed these without flagging the specific mechanism causing it. - No component may poll faster than the intervals specified in §6/§7 "to be safe" — that inflates both these budgets and is explicitly against the resource-discipline goal in the design doc.
| Failure | Required behavior |
|---|---|
| Sampler binary missing/fails to start | Extension logs via console.error once, does not retry-spawn more than 3 times per session, renders a static minimal-idle particle field with no data-driven behavior |
| IPC socket unreachable | Reconnect per §5 backoff; render last known SceneState, decaying toward idle after 5s of silence (§5) |
| PipeWire unavailable / no default sink | Sampler continues publishing telemetry-sourced SceneState only; Mode FSM MUST remain in System permanently (never transitions to Music) rather than erroring |
| hwmon temp node not found | heat for system source defaults to 0.0; do not crash the sampler |
| No battery present | Skip battery sampling silently (§6); this is not an error condition |
| Wallpaper image unreadable (permissions, missing file) | generate_layers returns an error message; extension treats this identically to "zero surviving candidates" (§11) — falls back to flat, no picker shown |
| GNOME Shell version mismatch at runtime | Extension MUST declare shell-version: ["50"] in metadata.json and rely on GNOME's own compatibility gating — do not add internal version-branching logic for other Shell versions |
Follow the phase order in the design doc §8. Each phase's acceptance criterion:
- Sampler logs valid
SceneStateJSON matching §4's schema at 300ms cadence; values sanity-checked by hand againsthtop/sensorsoutput. - A windowed (non-wallpaper) renderer displays particles driven by a hand-written mock
SceneState, with the color system from §9 visibly correct (verify the hue wrap in segment B specifically — this is the easiest part of the spec to get subtly wrong). - Same windowed renderer, now driven live by the sampler's telemetry-sourced
scene_updatemessages over the real IPC socket. - Same window, audio-sourced
scene_updatemessages correctly producing band-driven behavior; verify beat detection debounce (§7) actually prevents double-fires on sustained bass. - Mode FSM transitions verified against §8's exact thresholds using a stopwatch, not by eye.
- Extension running as an actual GNOME 50 wallpaper: verify per-monitor placement, verify fullscreen stop (not hide) via a system CPU monitor during a fullscreened window, verify multi-monitor desync is visibly not synchronized.
- Wallpaper layering end-to-end on at least one bright-sky/dark-silhouette image and at least one image expected to fail the sanity check (e.g. a flat-color or portrait image) — confirm the latter skips the picker automatically per §11.
- Resource budgets in §14 measured with
top/psunder both idle and heavy-load (e.g. amake -j$(nproc)build) conditions.
- SceneState: the single shared data contract in §4 — the only thing the renderer reads.
- Source: whether a given
SceneStatewas produced by the telemetry path or the audio path; not the same as the current Mode FSM state. - Layered mode: the three-actor depth compositing described in §11, as opposed to flat overlay (particles over the whole image, no depth split) or replace (no underlying image at all).
- Candidate: one of the (up to three) auto-generated cutout mask options presented in the picker UI.
- X11 support of any kind.
- Support for compositors other than GNOME Shell (no
wlr-layer-shellpath). - The external Rust/wgpu + GStreamer bridge ("Path B") — the in-Shell GJS/Clutter renderer (§10) is the only renderer in scope for this contract.
- DMA-BUF zero-copy texture sharing.
- NVIDIA/NVML GPU telemetry.
- Any telemetry field, color rule, or threshold not listed in this document — if the design doc's prose suggests something this contract doesn't pin down numerically, treat the absence as intentional and ask rather than filling the gap.
- Making any of the fixed constants in this document (poll intervals, thresholds, particle budget formula, hue stops) user-configurable in the first implementation pass, unless explicitly asked.