PRSim2

PengRui humanoid robot simulation platform — desktop Electron studio for GLB/GLTF characters: Shape Keys, facial control bones, studio lighting, MediaPipe face capture, and a local WebSocket control API.

About

PRSim2 is a desktop application from the PengRui team — a humanoid robot simulation platform for previewing and driving 3D facial expressions on rigged GLB/GLTF characters. Adjust Shape Keys and control bones, capture live expressions via webcam (MediaPipe / ARKit blendshapes), or automate rigs through a local WebSocket API in a cross-platform Electron studio viewer.

Official website: https://pengrui.ai

Overview

PRSim2 is a cross-platform Electron desktop application from the PengRui team for previewing and controlling humanoid 3D characters exported as GLB/GLTF. It targets facial animation workflows that combine Shape Keys (morph targets) with facial control bones, and adds real-time MediaPipe face tracking (52 ARKit blendshapes) plus an external automation API over WebSocket.

Features

Area	Capability
Model loading	Drag-and-drop `.glb` / `.gltf` onto the 3D view, or File → Open (`Ctrl/Cmd+O`)
Shape Keys	Deduplicated morph target sliders (same name across multiple meshes → one control)
Bones	Facial control bones (`c_` prefix + facial keywords), search filter, Show all bones* toggle
Lighting	Ambient, hemisphere, key/fill/rim lights, exposure; Reset lighting to defaults
Stage reveal	5-second cinematic lighting ramp when a model loads
Face capture	Separate window: webcam + MediaPipe Face Landmarker → 52 ARKit blendshapes → main viewer
Video drive	Toolbar opens a video window: play `.mp4` / `.webm` / `.mov`, track faces per frame, drive the model
WebSocket API	`ws://localhost:8765` (configurable via `PRSIM2_WS_PORT`) — blendshapes, bones, reset, state
Packaging	`electron-builder` → `dist/` (macOS dmg/zip, Windows NSIS/portable, Linux AppImage/deb)
Window chrome	Frameless on Windows/Linux; macOS hidden inset title bar with traffic lights

Video drive

Open from the main toolbar Video drive button (separate window).
Load local video: MP4, WebM, or MOV.
MediaPipe Face Landmarker extracts 52 ARKit blendshapes and head pose each frame and sends them to the main viewer (same mapping as live capture).
Face mesh in a small preview panel while tracking; replay after the video ends without re-selecting the file.
Load a GLB in the main window first so morph targets and head bones can follow the video.

Default lighting

Parameter	Default
Ambient intensity	`0.50`
Hemisphere intensity	`1.15`
Key light intensity	`1.35`
Key azimuth / elevation	`35°` / `45°`
Fill intensity	`1.50`
Rim intensity	`0.38`
Exposure	`1.30`

Tech stack

Layer	Technology
Desktop shell	Electron 36
Build tooling	electron-vite 3, Vite 6, TypeScript 5
3D rendering	Three.js 0.176 (`GLTFLoader`, `OrbitControls`)
Face tracking	@mediapipe/tasks-vision 0.10.35
External API	ws 8

Architecture

flowchart TB
  subgraph MainProcess["Electron Main Process"]
    Menu["Application Menu"]
    Dialog["File Dialog / IPC"]
    WS["WsControlServer :8765"]
    CapWin["Capture BrowserWindow"]
  end

  subgraph Preload["Preload (contextBridge)"]
    API["electronAPI"]
  end

  subgraph Renderer["Renderer — index.html"]
    MV["ModelViewer (Three.js)"]
    Expr["ExpressionPanel"]
    Bone["BonePanel"]
    Light["LightingPanel"]
  end

  subgraph Capture["Renderer — capture.html"]
    MP["MediaPipe FaceLandmarker"]
    Cam["getUserMedia"]
  end

  subgraph External["External Clients"]
    Py["scripts/test_ws_control.py"]
    Any["Any WebSocket client"]
  end

  Menu --> Dialog
  Dialog --> API
  API --> MV
  WS <-->|"ws:control / ws:response"| API
  CapWin --> MP
  MP -->|"blendshapes:update"| MV
  Cam --> MP
  Any --> WS
  Py --> WS
  MV --> Expr
  MV --> Bone
  MV --> Light

Control flow

GLB load — Renderer parses morph targets (deduplicated by name) and skeleton; optional 5s stage-reveal animation runs.
UI / WS — Sliders and WebSocket messages update the same ModelViewer state; bone rotations use radians (Three.js convention).
Face capture — Capture window streams blendshape coefficients to the main window via IPC; ARKit names are fuzzy-mapped to GLB morph target names.
WebSocket — Main process forwards JSON commands to the renderer and returns state / ack / error; model_loaded events are broadcast to all clients.

Getting Started

Prerequisites

Node.js 18+ (20 LTS recommended)
npm 9+
For face capture: system camera permission
For Python WebSocket tests: pip install websocket-client

Install & run (development)

git clone <repository-url>
cd prsim2
npm install
npm run dev

On first npm run build / packaging, the MediaPipe face model is downloaded to resources/mediapipe/face_landmarker.task (~4 MB) for offline use.

Usage

Launch the app (npm run dev or packaged binary from dist/).
Load a model:
- Drag & drop .glb / .gltf onto the viewer, or
- File → Open / toolbar Open GLB.
Use sidebar sections:
- Shape Keys — morph sliders (deduplicated names; multi-mesh targets move together).
- Bones — search, optional show-all, per-axis rotation sliders (radians).
- Lighting — adjust studio lights; Reset restores defaults.
Face capture — open the capture window, select camera, start tracking; blendshapes drive the loaded model when ARKit mapping succeeds.
Video drive — toolbar Video drive → pick a video file → play; per-frame blendshapes and head pose drive the model; face mesh shows in the small preview panel.
WebSocket — status bar shows API :8765 when the server is active.

WebSocket API

Item	Value
URL	`ws://localhost:8765`
Port override	Environment variable `PRSIM2_WS_PORT`
Format	JSON text frames
Concurrency	Multiple clients; control commands are last-wins per channel
Connect	Server may push initial `state` if a model is already loaded

Client → server messages

`type`	Fields	Description
`blendshapes`	`values: Record<string, number>`	Set ARKit/MediaPipe blendshape weights `0…1` (partial updates OK)
`bones`	`values: Record<string, {x,y,z}>`	Set bone rotation in radians
`reset`	`target: "blendshapes" \| "bones" \| "all"`	Reset morphs and/or bones
`get_state`	—	Query current state (works without a loaded model)

Examples

{ "type": "blendshapes", "values": { "jawOpen": 0.5, "mouthSmileLeft": 0.3 } }
{ "type": "bones", "values": { "c_jaw.x": { "x": 0, "y": 0.2, "z": 0 } } }
{ "type": "reset", "target": "all" }
{ "type": "get_state" }

Server → client messages

`type`	Fields	Description
`state`	`blendshapes`, `bones`, `modelLoaded`, `availableBones`, `availableMorphTargets`	Full snapshot
`ack`	`ok: true`, optional `applied`, optional `invalid[]`	Success (bone `invalid` lists unknown names)
`error`	`message`	e.g. `No model loaded`, `Invalid JSON`, `Renderer timeout`
`event`	`event: "model_loaded"`, `modelLoaded`, `availableBones`, `availableMorphTargets`	Broadcast after GLB load

Example state response

{
  "type": "state",
  "blendshapes": { "jawOpen": 0.5 },
  "bones": { "c_jaw.x": { "x": 0, "y": 0.2, "z": 0 } },
  "modelLoaded": true,
  "availableBones": ["c_jaw.x", "c_head.x"],
  "availableMorphTargets": ["smile", "jawOpen"]
}

ARKit blendshape names (52)

Supported names for blendshapes messages (MediaPipe / ARKit convention):

browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, cheekSquintRight, eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookDownRight, eyeLookInLeft, eyeLookInRight, eyeLookOutLeft, eyeLookOutRight, eyeLookUpLeft, eyeLookUpRight, eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight, jawForward, jawLeft, jawOpen, jawRight, mouthClose, mouthDimpleLeft, mouthDimpleRight, mouthFrownLeft, mouthFrownRight, mouthFunnel, mouthLeft, mouthLowerDownLeft, mouthLowerDownRight, mouthPressLeft, mouthPressRight, mouthPucker, mouthRight, mouthRollLower, mouthRollUpper, mouthShrugLower, mouthShrugUpper, mouthSmileLeft, mouthSmileRight, mouthStretchLeft, mouthStretchRight, mouthUpperUpLeft, mouthUpperUpRight, noseSneerLeft, noseSneerRight, tongueOut

Morph targets in the GLB are matched by exact name, case-insensitive name, or keyword fuzzy mapping (src/renderer/src/viewer/blendshapeMapping.ts).

Python test client

pip install websocket-client
python scripts/test_ws_control.py
python scripts/test_ws_control.py --host localhost --port 8765

The script demonstrates: connect + initial state, get_state, jawOpen animation loop, c_jaw.x bone rotation, and reset / all.

Blender GLB Export Guide

1. Prepare the character

Complete ARP rigging and skinning.
Build facial expressions with Shape Keys (typical ARP face workflow).
On the Armature modifier, disable Preserve Volume (most game engines do not use dual-quaternion skinning).
Set scene Unit Scale = 1; character height ≈ 1.8 m (one Blender unit ≈ 1 meter).

2. Shape Keys + drivers (recommended)

ARP recommends driving shape keys from custom properties on control bones (e.g. c_head.x) via drivers so shape and bone animation share one Action.

Example:

Add property smile (0–1) on c_head.x.
Copy as New Driver from the property.
Paste driver onto Shape Key smile Value.

3. Export with ARP

Select the Armature.
File → Export → Auto-Rig Pro FBX/GLTF.
Format: GLTF.
Important options:
- Shape Keys — enabled (morph targets)
- Apply Modifiers — if Subdivision etc. affect topology
- Full Facial (Humanoid) — full facial bone set when using bone-driven faces
- Engine preset: Generic / Godot / etc. (PRSim2 focuses on morph targets + c_* bones)

4. Verify in PRSim2

If Shape Keys count is 0 after load:

Shape keys must live on exported meshes, not only on the armature.
Confirm Shape Keys were exported.
Avoid export modes that omit skinned meshes.

Blender	Three.js
Shape Keys	`morphTargetDictionary` / `morphTargetInfluences`
Basis key	Skipped in UI
Bone-driven expressions	Bone panel + WebSocket `bones`

Project Structure

prsim2/
├── build/                          # electron-builder resources (macOS entitlements)
├── resources/mediapipe/            # face_landmarker.task (fetched by prebuild script)
├── scripts/
│   ├── ensure-mediapipe-model.mjs  # Download MediaPipe model before build
│   └── test_ws_control.py          # WebSocket API demo client
├── src/
│   ├── main/
│   │   ├── index.ts                # Windows, menu, IPC, capture window
│   │   └── wsServer.ts             # WebSocket server
│   ├── preload/
│   │   └── index.ts                # contextBridge API
│   ├── shared/
│   │   ├── wsProtocol.ts           # Message types & default port
│   │   └── appInfo.ts
│   └── renderer/
│       ├── index.html              # Main 3D viewer UI
│       ├── capture.html            # Face capture window
│       └── src/
│           ├── main.ts             # App wiring, drag-drop, WS handlers
│           ├── capture/main.ts     # MediaPipe pipeline
│           ├── viewer/
│           │   ├── ModelViewer.ts  # Scene, GLB, lights, reveal, morph/bone API
│           │   └── blendshapeMapping.ts
│           └── ui/
│               ├── ExpressionPanel.ts
│               ├── BonePanel.ts
│               └── LightingPanel.ts
├── out/                            # electron-vite build output (gitignored in dev)
├── dist/                           # Packaged installers (electron-builder)
├── package.json
└── README.md / README.zh-CN.md

Development

Command	Description
`npm run dev`	Dev server + Electron with HMR
`npm run build`	Production build → `out/`
`npm run preview`	Run production build locally
`npm run pack`	Build + unpackaged app dir (debug packaging)

Type-checking is via TypeScript; main/preload/renderer each have dedicated tsconfig paths under electron-vite defaults.

Build & Deploy

Uses electron-builder. Output directory: dist/.

npm install
npm run build:mac      # macOS dmg + zip (x64 + arm64 universal)
npm run build:win      # Windows NSIS installer + portable exe
npm run build:linux    # AppImage + deb
npm run build:all      # macOS + Windows + Linux in one run

Platform	Command	Typical artifacts
macOS	`npm run build:mac`	`dist/PRSim2-0.1.0.dmg`, `dist/PRSim2-0.1.0-mac.zip`
Windows	`npm run build:win`	`dist/PRSim2 Setup 0.1.0.exe`, `dist/PRSim2 0.1.0.exe`
Linux	`npm run build:linux`	`dist/PRSim2-0.1.0.AppImage`, `dist/prsim2_0.1.0_amd64.deb`

Cross-platform notes

Scenario	Notes
macOS → macOS	Native `npm run build:mac`; Universal binary (x64 + arm64).
macOS → Windows	May require Wine for NSIS; prefer CI (`windows-latest`).
macOS → Linux	Some targets (e.g. deb) may fail on macOS; use Linux VM or CI.
Code signing	macOS notarization requires Apple Developer certs; current config allows unsigned local builds.

Packaged app behavior

WebSocket — same port 8765 (or PRSIM2_WS_PORT).
MediaPipe — WASM and .task model unpacked via asarUnpack (out/renderer/mediapipe/**).
Camera — macOS NSCameraUsageDescription + build/entitlements.mac.plist.

Config: package.json → "build"; entitlements: build/entitlements.mac.plist.

Changelog

v0.1.0

Initial release: GLB viewer for rigged humanoid characters
Shape Keys panel with deduplicated morph targets
Facial control bone panel (search, show all)
Lighting panel + 5s stage reveal on load
MediaPipe face capture window (52 ARKit blendshapes)
WebSocket control API on port 8765
Cross-platform packaging (macOS / Windows / Linux)

Contributors

Xavier Chen (@xavier-chen) — PengRui Team

License

This project is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later).

Under the AGPL, if you modify this software and provide interactive access to users over a network (for example as a network service), you must offer those users the corresponding source code. See the LICENSE file for the full terms.

Name		Name	Last commit message	Last commit date
build		build
resources/mediapipe		resources/mediapipe
scripts		scripts
src		src
.gitignore		.gitignore
CONTRIBUTORS.md		CONTRIBUTORS.md
LICENSE		LICENSE
README.md		README.md
README.zh-CN.md		README.zh-CN.md
electron.vite.config.ts		electron.vite.config.ts
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
tsconfig.node.json		tsconfig.node.json
tsconfig.web.json		tsconfig.web.json

Navigation Menu

Repository files navigation

PRSim2

About

Table of Contents

Overview

Features

Video drive

Default lighting

Tech stack

Architecture

Getting Started

Prerequisites

Install & run (development)

Usage

WebSocket API

Client → server messages

Server → client messages

ARKit blendshape names (52)

Python test client

Blender GLB Export Guide

1. Prepare the character

2. Shape Keys + drivers (recommended)

3. Export with ARP

4. Verify in PRSim2

Project Structure

Development

Build & Deploy

Cross-platform notes

Packaged app behavior

Changelog

v0.1.0

Contributors

License

References

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

PRSim2

About

Table of Contents

Overview

Features

Video drive

Default lighting

Tech stack

Architecture

Getting Started

Prerequisites

Install & run (development)

Usage

WebSocket API

Client → server messages

Server → client messages

ARKit blendshape names (52)

Python test client

Blender GLB Export Guide

1. Prepare the character

2. Shape Keys + drivers (recommended)

3. Export with ARP

4. Verify in PRSim2

Project Structure

Development

Build & Deploy

Cross-platform notes

Packaged app behavior

Changelog

v0.1.0

Contributors

License

References

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages