WebAR Gesture Prototypes

A collection of browser-based AR interaction experiments built with webcam input, MediaPipe tracking, HTML5 Canvas, and Three.js.

The repository explores three different interaction patterns: gesture-triggered aiming and shooting, finger-trail slicing, and face-anchored deformation driven by pinch gestures.

Demos

1. Gesture Shooter

A camera-based shooting prototype that uses a hand pose as the input device. The index finger controls aiming, a thumb motion acts as the trigger, and nearby targets receive a small magnetic aim-assist effect.

Core ideas

• MediaPipe Hands for real-time hand landmarks
• Pistol-pose recognition
• Gesture-based trigger detection
• Screen-space aiming and target hit testing
• Three.js overlays, particles, score, hit, and miss feedback

Open: demos/gesture-shooter.html

2. Fruit Slice

A hand-tracked slicing prototype inspired by motion-based arcade interaction. The index fingertip becomes a blade, and the system detects sufficiently fast trajectory intersections with moving fruit.

Core ideas

• MediaPipe Hands fingertip tracking
• Finger trajectory and blade-trail rendering
• Velocity thresholding for slice detection
• Segment-to-object collision testing
• Combo scoring, particles, split-fruit feedback, and dynamic difficulty

Open: demos/fruit-slice.html

3. Skin Peel / Code Underlayer

An experimental face-and-hand interaction where a pinch gesture near the face can grab and pull a face-anchored membrane. Pulling beyond the rupture threshold creates persistent tears that reveal a moving code layer underneath.

Core ideas

• MediaPipe Hands + Face Mesh
• Pinch detection and hand-to-face proximity testing
• Face-landmark anchored interaction
• Stretch, tension, and rupture states
• Canvas-based deformation and code-underlayer rendering

Open: demos/skin-peel.html

Technology

• HTML5 / CSS3 / JavaScript
• MediaPipe Hands
• MediaPipe Face Mesh
• Three.js
• Canvas 2D
• WebRTC getUserMedia() camera access
• Web Audio API

Repository Structure

webar-gesture-prototypes/
├── index.html
├── README.md
└── demos/
    ├── gesture-shooter.html
    ├── fruit-slice.html
    └── skin-peel.html

Running Locally

Because the demos use camera access and external JavaScript/WASM assets, run them from a local web server rather than opening the files directly with file://.

With Python:

python3 -m http.server 8000

Then open:

http://localhost:8000

Allow camera access when prompted.

Browser Requirements

A modern browser with support for:

• navigator.mediaDevices.getUserMedia
• WebGL
• Canvas
• Web Audio
• ES6 JavaScript

Chrome or Edge is recommended for the most consistent MediaPipe behavior.

Design Goal

The project is less about building one finished game and more about exploring how computer-vision landmarks can become direct interaction primitives in browser-based spatial interfaces.

The three prototypes intentionally cover different input mappings:

Prototype	Tracking	Input Mapping	Main Interaction
Gesture Shooter	Hand	Pose + thumb trigger	Aim and shoot
Fruit Slice	Hand	Fingertip trajectory	Slice moving objects
Skin Peel	Hand + Face	Pinch + anchored drag	Pull and rupture a face-bound surface

Together they form a compact testbed for gesture recognition, continuous motion input, collision logic, and face-relative interaction.

Limitations

• Tracking quality depends on lighting, camera position, and hand visibility.
• The prototypes use heuristic gesture thresholds rather than trained gesture classifiers.
• Camera coordinates are screen-space approximations rather than calibrated world-space AR tracking.
• External MediaPipe/Three.js assets require an internet connection unless vendored locally.

Future Work

• Shared tracking and rendering modules across all demos
• Performance profiling and adaptive inference rates
• Mobile layout and interaction tuning
• WebXR integration
• Gesture calibration per user
• Recording and replay for usability testing
• A unified launcher with live preview thumbnails

License

Add a license before redistributing the project publicly if required by your intended use and third-party dependency terms.