RMSW - ROS2 Autonomous Explorer Evaluation Project

IL2202 Research Methodology and Scientific Writing - ROS2 Project

Project Overview

This project evaluates LLM (Large Language Model) generated code for autonomous robot exploration using ROS2. The project includes:

• A ROS2-based TurtleBot3 autonomous exploration system
• LLM-generated explorer implementations at 3 difficulty levels
• Automated testing framework for running and evaluating explorer implementations
• Code analysis tools for measuring complexity and lines of code

Table of Contents

• Project Structure
• ROS2 Project
  • Prerequisites
  • Installation
  • Running the Explorer
• Testing Scripts
  • Automated Explorer Testing (run_explorer_tests.py)
• Analysis Scripts
  • Line Counting Script (count_lines.py)
  • Code Complexity Analysis (check_code_complexity.py)

---

Project Structure

RMSW/
├── src/
│   └── irob_assignment_1/           # ROS2 package
│       ├── irob_assignment_1/       # Python module
│       │   ├── explorer_empty.py    # Template explorer (TODO skeleton)
│       │   ├── explorer_generated.py # Active explorer implementation
│       │   ├── performance_monitor.py # Metrics collection node
│       │   ├── LLM_codes_lvl1/      # Level 1 LLM-generated explorers
│       │   ├── LLM_codes_lvl2/      # Level 2 LLM-generated explorers
│       │   └── LLM_codes_lvl3/      # Level 3 LLM-generated explorers
│       ├── launch/                  # ROS2 launch files
│       ├── config/                  # Configuration files
│       └── urdf/                    # Robot description files
├── run_explorer_tests.py            # Automated testing script (Python)
├── run_explorer_tests.sh            # Automated testing script (Bash)
├── count_lines.py                   # Line counting analysis script
├── check_code_complexity.py         # Code complexity analysis script
├── exploration_results.csv          # Test results output
├── code_complexity_results.csv      # Complexity analysis output
└── line_count_results.csv           # Line count analysis output

---

ROS2 Project

The ROS2 project implements a frontier-based autonomous explorer for a TurtleBot3 robot in a simulated Gazebo environment. The explorer node:

1. Subscribes to the occupancy grid map (/map)
2. Detects frontier cells (boundary between known and unknown space)
3. Selects and navigates to frontier points using Nav2
4. Repeats until exploration is complete

Prerequisites

• Ubuntu 20.04
• ROS2 Foxy
• Python 3.8
• TurtleBot3 packages
• Nav2 navigation stack
• SLAM Toolbox
• Gazebo simulator

Installation

1. Source ROS2 environment:

   source /opt/ros/foxy/setup.bash

2. Create and setup workspace:

   mkdir -p ~/ros_ws/src
   cd ~/ros_ws/src
   # Clone or copy the irob_assignment_1 package here

3. Build the workspace:

   cd ~/ros_ws
   colcon build
   source install/setup.bash

4. Set TurtleBot3 model:

   export TURTLEBOT3_MODEL=burger

Running the Explorer

1. Start the simulator with navigation stack:

   export TURTLEBOT3_MODEL=burger
   source /opt/ros/foxy/setup.bash
   source ~/ros_ws/install/setup.bash
   
   # Launch simulator (Gazebo + RViz)
   ros2 launch irob_assignment_1 simulator.launch.py

2. In a new terminal, start navigation:

   ros2 launch irob_assignment_1 start_navigation.launch.py

3. In a new terminal, start the explorer node:

   ros2 launch irob_assignment_1 start.launch.py explorer:=explorer_generated

4. Optional: Start performance monitor:

   ros2 run irob_assignment_1 performance_monitor

Launch Files

Launch File	Description
simulator.launch.py	Starts Gazebo simulator with TurtleBot3 world and RViz visualization
start_navigation.launch.py	Launches Nav2 navigation stack
slam.launch.py	Starts SLAM Toolbox for mapping (not needed)
start.launch.py	Launches the explorer node (specify which explorer with explorer:= argument)

---

Testing Scripts

Automated Explorer Testing

Script: run_explorer_tests.py

This Python script automates testing of multiple LLM-generated explorer implementations. It:

• Copies each explorer implementation to explorer_generated.py
• Rebuilds the ROS2 package
• Launches the full simulation stack
• Monitors exploration progress and completion
• Collects performance metrics
• Records results to CSV

Usage

# Test all explorers from a specific level
python3 run_explorer_tests.py --level 1

# Test multiple levels
python3 run_explorer_tests.py --level 1 2 3

# Test only explorers from a specific LLM
python3 run_explorer_tests.py --llm chatgpt
python3 run_explorer_tests.py --llm claude
python3 run_explorer_tests.py --llm geminipro

# Test specific explorers by name
python3 run_explorer_tests.py --specific ChatGpt_level1_01 Claude_level1_02

# Limit number of tests
python3 run_explorer_tests.py --count 15

Configuration

The script has configurable timeouts at the top of the file:

• TIMEOUT_SECONDS: Maximum time per test (default: 600 seconds / 10 minutes)
• GAZEBO_STARTUP_WAIT: Wait time for Gazebo initialization
• NAVIGATION_STARTUP_WAIT: Wait time for Nav2 stack
• POST_DONE_WAIT: Wait time after completion for final metrics

Output

Results are saved to exploration_results.csv with metrics including:

• Exploration completion status
• Coverage percentage
• Navigation success rate
• Time elapsed
• Number of frontiers explored

Note: There is also a Bash version (run_explorer_tests.sh) with similar functionality.

---

Analysis Scripts

Line Counting Script

Script: count_lines.py

Counts actual code lines in Python files, excluding comments, docstrings, and blank lines.

Usage

# Count all files in all levels
python count_lines.py

# Count files in a specific level
python count_lines.py --level 1
python count_lines.py --level 2
python count_lines.py --level 3

# Count files from a specific LLM
python count_lines.py --model ChatGpt
python count_lines.py --model Claude
python count_lines.py --model GeminiPro

# Combine filters
python count_lines.py --level 1 --model Claude

# Analyze a specific file
python count_lines.py --file ChatGpt_level3_01.py

# Output in CSV format
python count_lines.py --csv

# Show only summary statistics
python count_lines.py --summary

Output

The script outputs:

• Per-file breakdown: Total lines, code lines, blank/comment lines
• Summary by model: Average, min, max code lines per LLM
• Summary by level: Average, min, max code lines per difficulty level

---

Code Complexity Analysis

Script: check_code_complexity.py

Analyzes code complexity metrics for all explorer implementations using the radon library.

Prerequisites

pip install radon

Usage

python check_code_complexity.py

Metrics Collected

Metric Category	Metrics
Cyclomatic Complexity (CC)	Average, Maximum, Total, Worst Rank, Number of Functions
Maintainability Index (MI)	Score (0-100), Rank (A-F)
Raw Metrics	LOC, LLOC, SLOC, Comments, Multi-line strings, Blank lines
Halstead Metrics	Vocabulary, Length, Difficulty, Effort, Volume, Bugs estimate, Time estimate

Output

Results are saved to code_complexity_results.csv with columns:

• File identification (filename, model, level, variant)
• All complexity metrics listed above

Summary statistics are also printed to console showing averages per level.

---

LLM Code Variants

The project includes explorer implementations generated by three LLMs at three difficulty levels:

Level	Files per LLM
Level 1	5 variants
Level 2	5 variants
Level 3	5 variants

LLMs tested:

• ChatGPT-5 Thinking Mini
• Claude Sonnet 4.5
• Gemini 3 Pro

---

License

Apache-2.0

---

Authors

• Mátyás Gergo
• Daniele Priola
• Project maintained for IL2202 course at KTH

References

• Maciej Wozniak (maciejw@kth.se) - Original ROS2 project (https://github.com/maxiuw/irob1)