MineMatrix

MineMatrix is a rule-guarded Reinforcement Learning (RL) environment and agent framework for Minecraft. It uses Mineflayer as a Node.js client bridge to run discrete actions on a local Minecraft server, guided by hardcoded survival safety rules (Rule Guards), coupled with a Gymnasium environment and Stable-Baselines3 (PPO) in Python for learning optimal survival policies.

---

1. Prerequisites

Before setting up, ensure your local Mac environment has the following installed:

• Java 21: Required by Minecraft Paper server.

  brew install openjdk@21

• Node.js 18+: Required by the Mineflayer client and HTTP bridge.
• Python 3.10+: Required for the RL training scripts.

---

2. Server Setup (Minecraft Paper)

To run the training environment, you need a local Minecraft server:

1. Download a Minecraft server JAR from PaperMC (e.g., version 1.20.x).
2. Place the JAR in a dedicated server directory, and run it for the first time:

   java -jar paper-*.jar --nogui

3. Open the generated eula.txt file and accept the license:

   eula=true
   

4. Open the generated server.properties and disable online mode (allowing the bot to connect offline):

   online-mode=false
   

5. Run the server again:

   java -jar paper-*.jar --nogui

---

3. Node.js Bot Setup & Execution

The Node.js project serves as an HTTP bridge exposing the state of the bot and accepting actions to execute.

1. Navigate to the bot directory:

   cd bot

2. Install the Node.js dependencies:

   npm install

3. Start the bridge server (connects to local Minecraft server on port 25565 under the username manoj):

   npm start

---

4. Python RL Setup & Training

The Reinforcement Learning agent is written in Python using Stable-Baselines3.

Important

You must run pip install directly on the target Mac (not inside a Docker container) so that PyTorch can automatically detect and install the correct MPS build optimized for Apple Silicon GPU acceleration.

1. Navigate to the rl directory:

   cd rl

2. Create and activate a Python virtual environment:

   python -m venv venv
   source venv/bin/activate

3. Install the dependencies:

   pip install -r requirements.txt

4. Start training the pure RL model:

   python train.py --timesteps 50000

5. Or start training the model guided by the high-level LLM planner:

   # Ensure Ollama is running locally: ollama run qwen2.5:3b
   python train_with_planner.py --timesteps 50000

---

5. How It Works: Rule-Guards + RL Split

MineMatrix splits decision-making into two distinct layers to accelerate learning and protect the agent from fatal mistakes:

                  +--------------------------------+
                  |         Python PPO RL          |
                  |  Decides action ID (0-9) based  |
                  |     on 12-dim state vector     |
                  +---------------+----------------+
                                  |
                                  | POST /action
                                  v
                  +--------------------------------+
                  |         Rule Guards            |
                  | Checks for high-risk threats;  |
                  | overrides RL choice if needed  |
                  +---------------+----------------+
                                  |
                                  | Executes chosen / overridden action
                                  v
                  +--------------------------------+
                  |         Mineflayer Bot         |
                  | Executes pathfinding / actions |
                  | in the Minecraft world         |
                  +--------------------------------+

1. Gymnasium Environment (rl/env.py): Receives the state vector, selects a discrete action index (0-9), and issues it via POST /action.
2. Rule-Guards Layer (bot/ruleGuards.js): Intercepts the action before execution. If the bot is in immediate danger, a hardcoded rule overrides it:
   • Low Health + Hostile Nearby: Overrides action to FLEE (2).
   • Zero Hunger + Food Available: Overrides action to EAT (3).
   • Very Low Oxygen: Overrides action to IDLE (6) with a swim-up jump command.
3. Action Execution (bot/actions.js): Executes the final (possibly overridden) action asynchronously using the mineflayer-pathfinder engine.
4. Reward Shaping: The reward is calculated using state differences on the Python environment, referencing constants from config.py.

---

6. Tuning Reward and Behavior

To adjust the bot's behavior, priorities, and action tendencies, edit rl/config.py. The primary levers are:

• LIVING_REWARD: Small penalty per step to encourage active progress rather than idling.
• DAMAGE_PENALTY_SCALE: Negative penalty multiplier for taking damage (e.g. from mobs or falling).
• DEATH_PENALTY: Severe penalty for dying.
• EAT_REWARD: Bonus for successfully consuming food and restoring hunger.
• MINE_REWARD: Bonus for mining resources (stone, wood, ores).
• FLEE_SUCCESS_REWARD: Bonus for moving away from hostile threats.
• WASTED_ACTION_PENALTY: Small penalty for attempting actions that couldn't be fulfilled (e.g., trying to eat with no food).
• HAS_BED_NIGHT_REWARD: Positive reward multiplier for having a bed at night.
• CRAFTING_TABLE_REWARD: Positive one-time reward when placing a crafting table.

---

7. V2/V3: Goals, Building, and the LLM Planner

V2: Scripted Goal-Actions (Deterministic Sequencing)

The bot's action space is extended to 10 discrete actions (0-9) where actions 7-9 invoke long-running, multi-step routines implemented in bot/goalActions.js:

• GATHER_WOOD (7): Locates nearby trees, paths to them, and mines logs (5 logs by default).
• CRAFT_AND_BUILD (8): Sequentially executes CRAFT_TABLE $\rightarrow$ GATHER_WOOL (shear/kill sheep) $\rightarrow$ CRAFT_BED (crafts and places a bed near the bot), skipping tasks whose requirements are already met.
• BUILD_SHELTER (9): Places block items from inventory around the bot's location in a 3x3 box wall structure to form a safe house.

V3: High-Level LLM Planner Layer

The Python training loop can run with LLM guidance by running train_with_planner.py. This setup coordinates a separate planner loop:

1. Ollama Integration: Uses a local Ollama instance (defaulting to the qwen2.5:3b model at http://localhost:11434/api/generate) with temperature $0.0$ for consistency.
2. Periodic Assessment: Every 45 seconds, the planner queries Ollama with a system prompt detailing the bot's situation (inventory counts, daytime state, vitals) and parses a JSON goal response:

   {"goal": "GATHER_WOOD"}

   Allowed goal outputs: SURVIVE, GATHER_WOOD, BUILD_SHELTER, CRAFT_AND_BUILD.
3. Soft Reward Nudging: While a non-SURVIVE goal is active, the environment temporarily boosts the reward weights in config.py for actions matching the goal. This nudges the PPO agent to pursue the planner's selected path without hard-locking it. Rule guards can still override actions at any point to save the bot's life.
4. Planner Logs: Every decision is appended to rl/logs/planner_decisions.jsonl in JSONLines format to act as training data for future fine-tuning.

Future Roadmap

• Terminal Dashboard UI: We plan to port over a blessed-contrib dashboard (from the previous mine-companion project) that visually displays live bot health, hunger, active LLM thoughts, and live inventory directly in the console.
• Modular Handlers: Refactoring the monolithic action macros into clean, object-oriented handler modules.