An intelligent, computer-vision-based sorting and storage system. The project classifies uploaded images or live video feeds of household/workshop items and dynamically illuminates the corresponding drawer in a physical cabinet to show where they belong.
This project is designed to integrate with a modular, 3D-printable storage tower.
- 3D Model Reference: Storage Tower Organizer - Modular Storage Box on MakerWorld.
- Hardware Integration: Five indicator LEDs (one per drawer/category) can be embedded into this organizer's drawer slots and connected to the ESP32. When an item is scanned and classified, the corresponding LED blinks to guide your sorting.
- AI Classification: Leverages a MobileNetV2 deep learning classifier trained to categorize 5 classes of items:
apples,coins,guns,pencils, andscrews. - Centralized Flask API: Orchestrates image receiving, model inference, prediction evaluation, error logging, and communication with physical hardware.
- Rich Web UI: A modern dashboard where users can drag-and-drop images or capture items using their webcams.
- Desktop Webcam Scanner: A Python OpenCV utility (
client_webcam.py) for live scanning via an external camera. - ESP32 Firmware: IoT endpoints that handle HTTP commands over WiFi to blink LED indicators on specified GPIO pins.
- Correction Feedback Loop: Stores misclassifications or user adjustments in a local JSONL log (
feedback_log.jsonl) for future model training.
graph LR
A["Web Browser<br/>(index.html)"] -->|"POST /predict<br/>(image)"| B["Flask API Server<br/>(app.py, port 5000)"]
C["Webcam Client<br/>(client_webcam.py)"] -->|"POST /predict<br/>(image)"| B
B -->|"Load model"| D["TensorFlow Model<br/>(storage_classifier.keras)"]
B -->|"GET /drawer/N<br/>(WiFi HTTP)"| E["ESP32<br/>(esp32.ino)"]
E -->|"Blink LED"| F["LED on GPIO Pin<br/>(Drawer 1–5)"]
├── app.py # Flask API backend server
├── inference.py # AI inference helper logic
├── index.html # Main web browser dashboard
├── client_webcam.py # Webcam client application (OpenCV)
├── requirements.txt # Python dependency file
├── setup_guide.md # Complete step-by-step installation guide
├── esp32/
│ ├── esp32.ino # Arduino sketch for the ESP32 microcontroller
│ ├── platformio.ini # PlatformIO project configuration
│ └── config.h.example # Template for WiFi SSID/password configuration
├── model/
│ ├── train.py # Model training script
│ ├── prepare_dataset.py # Dataset creation script
│ ├── predict.py # Basic inference testing script
│ └── training_history.png # Model training loss/accuracy curve
└── frontend/ # Alternate modular frontend
├── index.html
├── app.js
└── style.css
Detailed, step-by-step setup guides for all components are provided in the setup_guide.md. Below is a quick summary:
- Python 3.10 or 3.11 (TensorFlow compatibility)
- Arduino IDE (for ESP32 programming)
- Create and activate a virtual environment:
python -m venv venv source venv/bin/activate # On Windows: .\venv\Scripts\activate
- Install dependencies:
pip install -r requirements.txt
- Copy environment configuration:
(Edit
cp .env.example .env
.envto match your local IP setup and mock settings.) - Run the API server:
python app.py
- Open
http://localhost:5000in your browser to access the dashboard.
- Copy the WiFi credentials template:
cp esp32/config.h.example esp32/config.h
- Open
esp32/config.hand enter your 2.4 GHz WiFi credentials. - Wire the LEDs to the GPIO pins:
- Drawer 1 (apples) -> GPIO 2
- Drawer 2 (coins) -> GPIO 4
- Drawer 3 (guns) -> GPIO 5
- Drawer 4 (pencils) -> GPIO 18
- Drawer 5 (screws) -> GPIO 19
- Compile and flash
esp32/esp32.inoto your ESP32 board using the Arduino IDE or PlatformIO.