A Python program that maps body/hand movement + simple voice commands to Minecraft controls (mouse and keyboard).
It uses MediaPipe for pose/hand tracking and Vosk for offline speech commands, plus pyautogui/pynput to control the game.
Warning: This program controls your mouse and keyboard. Run it when Minecraft is focused (or otherwise in a safe environment) and be ready to kill the script if it acts unexpectedly (press
qin the OpenCV window or use your OS-level input lock).
- Also note that this program was written and run entirely on Windows 11, so if your operating system differs then some features (such as the
screeninfomodule, may not work). If that is the case, remove them from the program.
- If you want to use this program for a video or want to fork it, make sure to credit both my GitHub and YouTube!
- YouTube: https://www.youtube.com/@BlacBrdDev
- GitHub: https://github.com/Blacbrd
- Check out the YouTube video related to this project for a showcase of how I made it/use it!:
- YouTube link: https://www.youtube.com/watch?v=QgRdTzry0po
- Punch / mining detection (single click / hold for consecutive punches)
- Walking with alternating knee raise pattern
- Jumping and crouch with body posture detection
- Block placement when wrists cross
- Inventory opening with wrist-to-face gesture; in-menu finger-as-mouse hand mode
- Voice commands (Vosk) for recalibration and switching modes
- Head-turn → camera movement (mouse) support
- Basic calibration via voice commands:
calibrate arms,calibrate legs,calibrate head,calibrate box - Mode switching via voice:
change hand,change body
- Python 3.12.x (the project was tested with Python 3.12).
- Camera + Microphone.
- Laptop that can run Minecraft along with other apps.
- Minecraft client (and Vivecraft if using VR).
- Preferably 2 monitors; works with 1.
audio.py- Handles audio input logicconfigurables.py- User can change thresholds heremain.py- The main file the user will runstate.py- Controls the state of the program (global variables)tracking.py- Main Mediapipe tracking scriptutils- Helpful functions for programbodyTrackerVR.py- Body controller with VRrequirements.txt- Python dependencies (see below).env.example- Example.envthat a user should replicateREADME.md- This file
General idea: create and activate a venv, upgrade pip/tools, install requirements, download a Vosk model, set
VOSK_MODEL_PATHin.env, then runmain.py.
Windows (PowerShell):
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip setuptools wheel
macOS / Linux (bash / zsh):
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel
pip install -r requirements.txt
If
PyAudiofails on your platform see Troubleshooting below.
Download a Vosk model and set VOSK_MODEL_PATH (absolute path) in a .env file. The official model list and downloads are here:
https://alphacephei.com/vosk/models
For example, a small English model:
vosk-model-small-en-us-0.15orvosk-model-en-us-0.22(larger / more accurate).
I used vosk-model-en-us-0.22-lgraph
Unzip the model and point VOSK_MODEL_PATH to the folder:
VOSK_MODEL_PATH = r"/absolute/path/to/vosk-model-small-en-us-0.15"
If you don't do this the camera turning won't work!
- Pause Game → Options → Controls → Key Binds... → Change Sneak to "C"
- Pause Game → Options → Controls → Set Sneak to Toggle

-
If using VR, install the Vivecraft mod (or any other compatible VR mod that allows movement with keyboard and mouse) (Can be installed using Cursed Forge)
Tutorial on how to install Vivecraft: (Cursed Forge) https://www.youtube.com/watch?v=N_77a6dXMJ0
-
Pause Game → Options → VR Settings... → Play Mode: Seated
-
Pause Game → Options → VR Settings... → Seated Settings → Put settings as shown in image below:

-
Pause Game → Options → Controls → Key Binds... → Rotate Left (N) Rotate Right (M)

python main.py
- The OpenCV window shows the camera feed and landmark overlays. Press
qinside the OpenCV window to quit.
Supported words are limited by the small recognizer grammar. Useful commands you can say while the program runs:
calibrate arms→ Dummy function, doesn't do anythingcalibrate legs→ sets knee thresholdcalibrate head→ recalibrates base head rotationcalibrate box→ repositions the inventory box for hand modechange hand→ switch to hand mode (inventory / menus)change body→ switch back to body mode
SMALL_ANGLE_THRESHOLD,LARGE_ANGLE_THRESHOLD- punch detection anglesCONSECUTIVE_PUNCH_THRESHOLD- time window for “mining” detectionKNEE_OFFSET- how far above knees the purple threshold line is drawn (pixels)WALK_THRESHOLD_ENABLED,CONSECUTIVE_WALK_THRESHOLD- walking behaviourDOMINANT_HAND-"Right"or"Left"RECTANGLE_WIDTH/HEIGHT- hand-mode pointing box sizeMONITORselection is automatic fromscreeninfo
- This program will click and press keys. Keep a spare keyboard/mouse or an OS-level input lock if you need to regain control quickly.
- Test with Minecraft already opened and in focus to prevent the mouse from drifting onto other programs.
- Do not run on an account or system where unexpected keypresses could be dangerous.
- Make sure stretch/warm up before using the program, and take regular breaks.
- The head turning feature is very finecky and hard to get used to, make sure to avoid fast head movements as these can cause neck strain (trust me I've been there).
MediaPipe currently publishes Python wheels and supports desktop Python versions including Python 3.12; if you see an error when pip install mediapipe, please ensure pip is up to date using pip install --upgrade pip and try again.
- Windows: use
pipwinor a prebuilt wheel ifpip install PyAudiofails. - macOS:
brew install portaudiothenpip install pyaudio. - Ubuntu/Debian:
sudo apt-get install portaudio19-dev python3-devthenpip install pyaudio.
Vosk itself installs via pip install vosk. Models are downloaded separately (alpha-cephei models page and mirrors). Point vosk.Model() at the unzipped model directory.
