An AI Calculator that learns to do arithmetic using only a neural network — no Python eval, no calculators, no symbolic math.
The model is trained from scratch (or fine-tuned) to predict the correct answer to simple arithmetic problems by learning patterns from synthetic data. This is a pure "neural arithmetic" demonstration.
- Generate high-quality synthetic training data for arithmetic
- Train a small Transformer model to solve arithmetic as a sequence modeling problem
- Rigorously evaluate the model on unseen problems
- Provide a clean GUI to explore every stage of the pipeline
- Never use actual calculation during inference — the answer comes only from the neural network
- Creates synthetic problems of the form
a {op} b - Supported operations:
+,-(multiplication optional) - Configurable difficulty: number of digits (1–6)
- Produces both the question and the ground truth answer
- Exports to JSONL + optional CSV
- Creates properly split train / validation / test sets with controlled distribution shifts
- Tokenizer: Tiny custom character-level tokenizer (
ArithmeticTokenizer)- Vocabulary: digits
0-9, operators+ - *,=, space, special tokens[PAD],[EOS],[UNK] - Extremely small vocab (~18 tokens) — forces the model to truly learn the algorithm
- Vocabulary: digits
- Model: Small decoder-only Transformer (GPT-2 style architecture via
transformers)- Configurable size presets:
tiny(~6-8M),small(~20-30M) - Trained with causal language modeling
- Loss is masked so the model only learns to predict the answer portion after
=
- Configurable size presets:
- Standard next-token prediction on formatted strings:
47 + 128 = 175 - Only the answer tokens contribute to the loss (standard SFT-style masking)
- Supports:
- Resume from checkpoint
- Mixed precision (bf16 / fp16)
- MPS (Apple Silicon), CUDA, or CPU
- Training metrics logged live (loss, accuracy on answer tokens)
- Exact match accuracy (model output string == correct answer string)
- Optional relaxed metrics (numeric closeness)
- Breakdown by difficulty (number of digits, operation type)
- Failure case analysis (shows what the model actually output)
- Pure neural prediction only
- User provides an expression → model completes after
= - Generation parameters exposed (temperature, max length, beams)
- Zero access to Python arithmetic in the inference path
The primary way to use Calc-AI is through the interactive web UI:
python app.pyThe GUI has these main sections:
| Tab | Purpose |
|---|---|
| Dashboard | Quick inference + project status + "Load Demo Model" |
| Data Lab | Generate datasets, preview samples, view statistics, export |
| Training Lab | Configure model size, data, hyperparameters. Launch training with live loss curves |
| Evaluation Lab | Load a checkpoint, run full evaluation on test set or custom data, analyze errors |
| AI Calculator | The star of the show — type expressions and watch the neural net answer |
The GUI is designed so you can navigate the entire lifecycle without touching the command line.
From the project directory, simply run:
./start.shThis will:
- Create the Python virtual environment (if needed)
- Install all dependencies
- Launch the GUI automatically
Even better for first-time users:
./start.sh --quickThis does everything above plus automatically generates a small dataset and trains a tiny demo model (~2-3 minutes). After it finishes, the AI Calculator will already give reasonable answers instead of random output.
If you prefer doing it step by step:
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.pyThen open: http://127.0.0.1:7860 (or http://localhost:7860)
- Go to Data Lab → Generate a small dataset (or use the one from
--quick) - Go to Training Lab → Train a "tiny" model for 3–5 epochs
- Go to AI Calculator → Type expressions like
47 + 128or300 - 147
Running ./start.sh --quick creates a usable demo model automatically and places it in examples/demo_model. You can also load previous checkpoints from the Model Manager tab.
-
"The link goes nowhere" / blank page:
Always usehttp://127.0.0.1:7860orhttp://localhost:7860.
The app deliberately binds to127.0.0.1(not0.0.0.0) because0.0.0.0is unreliable when accessed from the same Mac. -
Make sure you are inside the virtual environment (
source .venv/bin/activate). -
If port 7860 is busy, you can change it temporarily by editing the
server_portline near the bottom ofapp.py. -
Try a different browser (Chrome / Firefox work best).
Raw training example (after tokenization):
Input (prompt): 47 + 128 =
Target (model learns to generate): 175
Full training string:
47 + 128 = 175
The loss is only applied to the characters after =.
This project demonstrates several important ideas:
- Neural networks can learn algorithmic behavior (addition with carry, subtraction with borrow)
- You can supervise only the final answer — the model invents its own internal algorithm
- Small models (~10M params) are surprisingly capable when the task is narrow and data is clean
- "AI calculators" are a fun and rigorous way to study generalization, distribution shift, and length extrapolation
Ideas for further work (great for learning):
- Add multiplication (significantly harder — needs more capacity/data)
- Support multi-operand expressions or parentheses
- Add chain-of-thought supervision (
47 + 128 = 175 <think> carry 1...) - Try encoder-decoder architecture (T5 style) instead of decoder-only
- Length extrapolation experiments (train on 3 digits, test on 5-6 digits)
- LoRA fine-tuning on top of a larger base model
Calc-AI/
├── app.py # Main Gradio application (start here)
├── calc_ai/
│ ├── __init__.py
│ ├── data_generator.py # Synthetic arithmetic data creation
│ ├── tokenizer.py # ArithmeticTokenizer (character level)
│ ├── model.py # Small transformer model definition
│ ├── trainer.py # Training loop / HF Trainer wrapper
│ ├── evaluator.py # Evaluation metrics and analysis
│ └── inference.py # Pure neural inference (no cheating!)
├── configs/
│ └── default.yaml
├── data/ # Generated datasets (gitignored)
├── runs/ # Training run outputs
├── checkpoints/ # Saved models
├── examples/ # Demo checkpoints + sample data
├── requirements.txt
└── README.md
Educational project. Built as a demonstration of neural algorithmic learning.
Important: This calculator does not use any traditional arithmetic libraries at inference time. Every answer you see is generated token-by-token by the neural network.# Calc-AI