MathBoard

Draw an equation. Get the solution. MathBoard is a handwriting-to-solution math whiteboard: sketch an expression on a canvas, and it is recognized, converted to LaTeX, solved symbolically, and independently cross-checked.

Live demo · Setup guide

The demo frontend is hosted on GitHub Pages and talks to a free-tier Render backend, so the first request after a period of inactivity can take a while to wake up.

Highlights

• Hybrid recognition. A locally trained CNN handles clean single-symbol drawings on the server; anything ambiguous or multi-part falls through to Google Gemini for OCR. The app keeps working with no trained model at all.
• Symbolic solving with a second opinion. A SymPy solver produces the answer, and an optional R Plumber sidecar re-solves it with Ryacas. The UI shows a check mark when the two engines agree and a warning when they do not.
• Observability built in. A "Model & Activity" tab (R Shiny) shows live request statistics, the classifier's confusion matrix, solver agreement, and the local-vs-Gemini split.
• Tested. About 70 Python tests across the solver, API, classifier, datasets, IDX parser, activity log, Ryacas client and a training smoke test, plus R tests for the LaTeX-to-Yacas conversion.

How it works

flowchart LR
    C[React canvas] -->|PNG| API[FastAPI /convert]
    API --> G{Simple single symbol?}
    G -->|yes: confidence >= 0.85| CNN[Local CNN]
    G -->|no| GEM[Gemini OCR]
    CNN --> L[LaTeX]
    GEM --> L
    L --> S[SymPy solver]
    S --> R[Ryacas cross-check via R Plumber]
    S --> UI[KaTeX result in the UI]
    R --> UI
    API --> LOG[(Activity log)] --> SH[R Shiny dashboard]

Loading

For each drawing the backend decodes the image and counts connected components. With more than four components it goes straight to Gemini. Otherwise it crops, resizes to 32x32, runs the CNN, and accepts the prediction locally only if the softmax confidence is at least 0.85.

Local classifier results

Trained on HASYv2 (top 100 symbol classes) plus MNIST digits, 110 classes in total. These are validation-split numbers from backend/ml/artifacts/metrics.json, measured at the end of epoch 8:

Metric	Value
Top-1 accuracy (all validation samples)	94.1%
Coverage at the 0.85 confidence threshold	86.0% of samples handled locally
Accuracy on the accepted (confident) predictions	99.1%

The accepted-only accuracy is a confident subset, so it is higher than the overall figure by construction. Treat 94.1% as the headline generalization number; the confidence gate is what lets the app skip the Gemini call for most clean symbols.

Tech stack

Layer	Tools
Frontend	React, Vite, KaTeX
Backend	Python, FastAPI, SymPy, PyTorch (CNN), Pillow, NumPy
OCR	Google Gemini (gemini-2.5-flash)
Cross-check and dashboard	R, Plumber, Ryacas, Shiny
Data	HASYv2, MNIST
Deployment	GitHub Pages (frontend), Render (backend, see render.yaml)

Quick start

You need Python 3.10+, Node 18+, and a free Gemini API key. R is optional.

# Backend
cd backend
cp .env.example .env            # add GEMINI_API_KEY
python -m venv .venv && .venv\Scripts\activate   # macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload       # http://localhost:8000

# Frontend (second terminal)
cd frontend/mathboard
npm install
npm run dev                     # http://localhost:5173

Optional extras (the core solver works without them):

python -m ml.train              # train the local CNN, about 30 min on CPU
Rscript run_plumber.r           # Ryacas cross-check on :8003
Rscript run_shiny.r             # Model & Activity dashboard on :3838

See SETUP.md for the full walkthrough, the HASYv2 dataset download, and troubleshooting.

Tests

cd backend && pytest tests/ -v
Rscript -e "testthat::test_file('tests/test_plumber.R')"

Project structure

backend/
  main.py              FastAPI app (/health, /convert)
  solver.py            SymPy solving and step output
  ryacas_client.py     client for the R cross-check service
  activity_log.py      request logging for the dashboard
  ml/                  CNN model, training, preprocessing, datasets
  plumber.R, shiny_app.R
frontend/mathboard/    React + Vite UI
render.yaml            Render deployment config

Limitations

• Recognition quality depends on how cleanly the expression is drawn; complex multi-line expressions rely on Gemini.
• The classifier artifacts are gitignored, so you need to train it yourself (or accept the Gemini-only path).
• The classifier numbers above are from the validation split; there is no separately reported test-set figure yet.