SmartClass Analytics is a deployment-oriented classroom behavior analysis system built for long surveillance-style academic video, combining GPU-efficient perception, track-level temporal reasoning, and interactive web-based review.
- Project Vision
- Why This System Exists
- System Architecture
- Core Capabilities
- Web Platform
- Interface Modules
- Project Structure
- Environment Setup
- Model Selection
- Quick Start
- Single-Video Inference
- Batch Processing
- Output Schema
- Performance Philosophy
- Practical Use Cases
- Extensibility
- Roadmap
- Summary
Classroom video analysis is a deceptively difficult problem.
A real classroom is not a curated benchmark scene. It is a dense, long-duration, partially occluded, behavior-rich environment where multiple people act simultaneously, identities must remain consistent over time, and the same posture may correspond to different semantic meanings depending on context.
This project is designed to make that problem operational.
SmartClass Analytics transforms long classroom recordings into structured behavioral timelines through a hybrid pipeline that emphasizes:
- speed on local GPU hardware
- stable multi-person identity tracking
- interpretable first-pass behavior inference
- selective semantic refinement for ambiguous cases
- human-in-the-loop inspection and correction
- analytics-oriented outputs for later reporting and research
Rather than treating classroom understanding as a pure VLM problem or a pure rule-based pipeline, the system deliberately combines both philosophies in a way that is practical for real deployment.
Running a large multimodal model over every frame of a long classroom video is usually impractical.
A one-hour recording can easily contain tens of thousands of frames. If every frame is pushed into a heavy image-text model, the system becomes expensive, slow, and difficult to scale. At the same time, relying only on simple heuristics often produces labels that are too coarse for educational analytics.
This system addresses that trade-off with a two-layer design:
A GPU-friendly perception stack handles the full video stream:
- person detection
- tracking
- pose estimation
- movement and posture cues
- first-pass behavior classification
Only selected person crops or representative track snapshots are sent to an optional VL image-to-text model for more expressive descriptions when the first-pass stage is insufficient.
This means the system can remain:
- fast enough for long-form video
- expressive enough for meaningful behavior summaries
- structured enough for analytics dashboards
- flexible enough for future classroom intelligence tasks
Long classroom video
│
▼
[Frame sampling / stride control]
│
▼
[YOLO person detection]
│
▼
[Multi-object tracking]
│
▼
[Per-person pose estimation]
│
▼
[Lightweight behavior rules]
│
├──► annotated video
├──► per-frame CSV
├──► per-track CSV
├──► per-track JSON
└──► web live stream + review UI
│
▼
[optional selective visual-language refinement]
│
▼
refined labels + activity reports + teacher analytics
The first stage must be able to process all sampled frames of long classroom videos without collapsing under computational cost.
The system does not only care about frame-wise detections. It is organized around consistent person identities over time, which allows activity summaries, per-ID timelines, and teacher-focused analysis.
Not all frames deserve expensive semantic interpretation. Refinement happens only where it adds value.
The web interface is not an afterthought. It is part of the pipeline’s core philosophy: analysts should be able to inspect, verify, and correct outputs interactively.
- Ultralytics YOLO-based person detection
- GPU-accelerated inference
- stable track IDs via lightweight multi-object tracking
- per-person keypoint extraction using YOLO pose
The rule-based stage produces coarse but useful actions such as:
standingsittingwalkingraising_handhead_downlying_on_deskunknown
These labels serve as the operational backbone for all downstream analytics.
The pipeline produces identity-aware temporal summaries, not just isolated frame predictions. This makes it possible to answer questions such as:
- what a given student was doing over time
- when specific activities occurred
- which students appeared most focused or most distracted
- how teacher actions correlated with classroom response
The platform supports live and post-run analysis through a browser-based interface for:
- current frame inspection
- re-tagging
- ID filtering
- per-person behavior review
- track summaries
- teacher-centered analytics
A small number of representative crops per track can be passed to an optional VL image-to-text stage to reduce vague labels and improve semantic specificity.
The web application is designed as the operational control center for the pipeline.
It supports both analysis-time visualization and post-analysis review, allowing users to move from raw video to structured classroom insight within one interface.
- upload a classroom video or provide an absolute server path
- configure stride, device, and person limits
- launch analysis
- monitor live detections and identity assignments
- inspect per-ID behavior over time
- optionally run selective refinement
- review teacher behavior and classroom response statistics
- FastAPI backend
- SSE streaming for progressive frame delivery
- browser-based control and visualization
- structured output persistence for repeated review
Each interface page is designed for a specific stage of the classroom analysis workflow.
This is the operational landing page of the system. It provides the entry point for the full analysis pipeline, including video ingestion, run configuration, live execution controls, and refinement triggers.
Users can:
- upload a local classroom video
- enter a server-side absolute path
- configure
stride,max persons, anddevice - start or stop analysis
- launch selective refinement
- enable Focus ID mode for isolated per-person inspection
This page turns the system from a research prototype into a usable operator-facing tool. Instead of relying on command-line only workflows, it centralizes the critical controls needed for practical classroom video processing.
The Live View page streams the currently processed frame together with the detections recognized at that moment.
It includes:
- current annotated frame
- bounding boxes and ID labels
- first-pass or refined behavior labels
- frame index and current timestamp
- detection cards on the right panel
- dropdown-based re-tagging for each visible identity
- timeline scrubbing support
This page is especially useful for:
- debugging tracking quality
- validating current labels
- checking whether identity switches are happening
- performing human-in-the-loop correction during review
A particularly valuable feature is Focus ID mode, which isolates a selected identity and displays only that person’s crop and label panel. This is useful when a classroom is crowded and the analyst needs to inspect a single subject without distraction.
The Activity Report page reorganizes the raw frame-level analysis into a cleaner identity-time representation.
It allows users to:
- switch between By ID and By Activity
- filter by activity type
- search refine tags
- inspect a compact time-segment timeline for each person
- click segment chips to inspect corresponding moments
Each ID receives a readable sequence such as:
0:00–0:04 sitting, looking_at_phone0:05–0:10 sitting, working_on_computer
This makes the output much more interpretable than raw CSV rows.
The Activity Report is the bridge between algorithmic output and behavioral understanding. It helps transform low-level predictions into a timeline narrative for each student or classroom participant.
The Refine Summary page provides a macro-level view of the semantic results accumulated so far.
It highlights:
- refined label distribution
- track overview
- behavior frequency composition
- which labels dominate the classroom recording
- which identities have richer semantic descriptions after refinement
This page is useful for quickly understanding the behavioral composition of the entire session without manually reviewing every frame.
A good analytics system should not force the user to inspect raw detections only. The refine summary offers a higher-level snapshot of what the classroom looked like behaviorally, which is important for reporting, presentation, and trend discovery.
The Teacher Behaviour module is one of the most valuable pages in the system because it elevates the project beyond generic surveillance analytics.
This page focuses on a selected teacher track and correlates teacher-side actions with classroom-side response. It can display:
- teacher activity timeline
- teacher activity breakdown
- student activity breakdown
- student focus trend over time
- engagement score
- focused student list
- distracted student list
- correlations between teacher actions and student responses
Typical questions supported by this page include:
- when was the teacher reading, writing, or lecturing
- during which teacher actions did student focus rise or fall
- which students remained most attentive across the session
- which students were most frequently distracted
This module turns the system into a classroom insight platform rather than only a detection engine. It supports educational analytics, teaching evaluation, behavioral study, and structured classroom observation.
Teacher Behavior Analysis Page – Statistics on the Correlation between Teacher Actions and Student Status
classroom_behavior_hybrid/
app.py # Core single-video processing pipeline
batch_run.py # CLI for single and batch processing
web_server.py # FastAPI backend + SSE streaming
templates/
live_viewer.html # Browser UI
detector.py # YOLO detector wrapper
tracker.py # Lightweight IOU tracker
pose.py # YOLO pose wrapper
behavior_rules.py # Pose/motion-based first-pass action logic
qwen_refiner.py # Optional visual-language refinement
draw_utils.py # Rendering utilities
io_utils.py # Video / CSV / JSON utilities
config.py # Global configuration dataclass
requirements.txt
README.md
outputs/ # Output videos and structured artifacts
Recommended target environment:
- Ubuntu 24.04
- NVIDIA RTX 4090
- Python 3.10
- CUDA-capable PyTorch environment
cd /home/mmr/smart_classroom/open_model_zoo
conda create -n classroom-hybrid python=3.10 -y
conda activate classroom-hybrid
pip install -r classroom_behavior_hybrid/requirements.txtIf your machine requires a specific CUDA-enabled PyTorch build, install torch and torchvision according to the official PyTorch instructions before or after installing the remaining requirements.
Recommended Ultralytics models for a strong speed/quality balance on RTX 4090:
yolo11n.pt— fastest option for long videoyolo11s.pt— stronger than nano with moderate overheadyolo11m.pt— higher accuracy at higher cost
yolo11n-pose.pt— efficient per-person keypoint extraction
Ultralytics will automatically download these model weights on first use if they are not already available locally.
cd /home/mmr/smart_classroom/open_model_zoo
conda activate classroom-hybrid
uvicorn classroom_behavior_hybrid.web_server:app --host 0.0.0.0 --port 8000Open in browser:
http://localhost:8000
- choose a video file or provide a server path
- click Start analysis
- inspect detections in Live View
- review semantic distribution in Refine Summary
- inspect per-ID behavior in Activity Report
- analyze classroom teaching dynamics in Teacher Behaviour
To run the pipeline from the command line for one video:
cd /home/mmr/smart_classroom/open_model_zoo
conda activate classroom-hybrid
python -m classroom_behavior_hybrid.batch_run \
--single_video "/home/mmr/Desktop/Видео уроки/01.10.2025 12-40о13-40/B02-R116_01_10_2025 13.42.44.mp4" \
--device cuda:0 \
--frame_stride 5 \
--detector_model yolo11n.pt \
--pose_model yolo11n-pose.pt \
--max_persons_per_frame 20The generated files are typically saved under classroom_behavior_hybrid/outputs/, including:
- annotated video
- frame-level CSV
- track-level CSV
- track-level JSON
For recursive processing of a directory of classroom videos:
cd /home/mmr/smart_classroom/open_model_zoo
conda activate classroom-hybrid
python -m classroom_behavior_hybrid.batch_run \
--input_root "/home/mmr/Desktop/Видео уроки" \
--output_root "classroom_behavior_hybrid/outputs" \
--device cuda:0 \
--frame_stride 5 \
--detector_model yolo11n.pt \
--pose_model yolo11n-pose.pt \
--max_persons_per_frame 20The file discovery logic supports paths containing spaces and Cyrillic characters.
The structured outputs are designed for both direct inspection and downstream analytics.
Typical fields include:
source_videoframe_indextimestamp_sectrack_idbbox_xminbbox_yminbbox_xmaxbbox_ymaxfirst_pass_actionfirst_pass_confidenceqwen_refined_actionqwen_confidence
Provides a visual audit trail of the full inference process.
Suitable for low-level debugging, visualization, and temporal statistics.
Useful for track-centric reporting and aggregation.
Useful for integration into web dashboards, APIs, and later analytics pipelines.
This project is not designed around benchmark vanity. It is designed around operational value.
Its core philosophy is:
- do not overuse heavy multimodal inference
- preserve identity over time
- keep the first-stage pipeline fast and stable
- expose the outputs in a way that humans can inspect
- make the system extensible for real educational environments
For long classroom videos, this matters far more than maximizing complexity on paper.
SmartClass Analytics can support:
Structured review of student behaviors and teaching dynamics.
Quantifying how teacher actions correlate with student focus and classroom response.
Exporting time-resolved structured signals for later analysis.
Using the interface to accelerate semi-manual dataset building or correction.
Applying the system to environments where persistent identity and behavior timelines matter.
The current design intentionally keeps several modules replaceable.
You can later extend or replace:
- the tracker with ByteTrack or BoT-SORT
- the first-pass rules with a learned action head
- the refinement stage with another image-text reasoning model
- the analytics layer with richer scoring logic
- the web frontend with institution-specific reporting needs
Because the pipeline is modular, it can evolve without rewriting the full system.
Potential next directions include:
- stronger classroom-specific behavior taxonomy
- better social interaction modeling
- teacher–student interaction graphs
- engagement scoring over longer horizons
- automatic session reports
- multi-camera classroom fusion
- deployment integration into larger smart campus systems
SmartClass Analytics is a hybrid classroom behavior analysis platform built for real long-form educational video.
It combines:
- fast GPU-based perception
- track-aware temporal analysis
- interpretable behavior outputs
- interactive web review
- optional selective semantic refinement
- dedicated teacher behavior analytics
The result is a system that is not only technically capable, but also operationally useful for classroom understanding, educational analytics, and future smart-campus deployment.