Skip to content
MenuuTUXPublic

About

A small transformer trained from scratch to do arithmetic with no calculation at inference, with synthetic data generation, evaluation on held-out problems, and a Gradio GUI.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Calc-AI

An AI Calculator that learns to do arithmetic using only a neural network — no Python eval, no calculators, no symbolic math.

The model is trained from scratch (or fine-tuned) to predict the correct answer to simple arithmetic problems by learning patterns from synthetic data. This is a pure "neural arithmetic" demonstration.

Project Goals

  • Generate high-quality synthetic training data for arithmetic
  • Train a small Transformer model to solve arithmetic as a sequence modeling problem
  • Rigorously evaluate the model on unseen problems
  • Provide a clean GUI to explore every stage of the pipeline
  • Never use actual calculation during inference — the answer comes only from the neural network

Architecture Overview

1. Data Generator (data_generator.py)

  • Creates synthetic problems of the form a {op} b
  • Supported operations: +, - (multiplication optional)
  • Configurable difficulty: number of digits (1–6)
  • Produces both the question and the ground truth answer
  • Exports to JSONL + optional CSV
  • Creates properly split train / validation / test sets with controlled distribution shifts

2. Model & Tokenizer

  • Tokenizer: Tiny custom character-level tokenizer (ArithmeticTokenizer)
    • Vocabulary: digits 0-9, operators + - *, =, space, special tokens [PAD], [EOS], [UNK]
    • Extremely small vocab (~18 tokens) — forces the model to truly learn the algorithm
  • Model: Small decoder-only Transformer (GPT-2 style architecture via transformers)
    • Configurable size presets: tiny (~6-8M), small (~20-30M)
    • Trained with causal language modeling
    • Loss is masked so the model only learns to predict the answer portion after =

3. Training Pipeline

  • Standard next-token prediction on formatted strings: 47 + 128 = 175
  • Only the answer tokens contribute to the loss (standard SFT-style masking)
  • Supports:
    • Resume from checkpoint
    • Mixed precision (bf16 / fp16)
    • MPS (Apple Silicon), CUDA, or CPU
  • Training metrics logged live (loss, accuracy on answer tokens)

4. Evaluation Pipeline

  • Exact match accuracy (model output string == correct answer string)
  • Optional relaxed metrics (numeric closeness)
  • Breakdown by difficulty (number of digits, operation type)
  • Failure case analysis (shows what the model actually output)

5. Inference

  • Pure neural prediction only
  • User provides an expression → model completes after =
  • Generation parameters exposed (temperature, max length, beams)
  • Zero access to Python arithmetic in the inference path

GUI (Gradio)

The primary way to use Calc-AI is through the interactive web UI:

python app.py

The GUI has these main sections:

Tab Purpose
Dashboard Quick inference + project status + "Load Demo Model"
Data Lab Generate datasets, preview samples, view statistics, export
Training Lab Configure model size, data, hyperparameters. Launch training with live loss curves
Evaluation Lab Load a checkpoint, run full evaluation on test set or custom data, analyze errors
AI Calculator The star of the show — type expressions and watch the neural net answer

The GUI is designed so you can navigate the entire lifecycle without touching the command line.

Quick Start (Recommended)

One-command setup + launch

From the project directory, simply run:

./start.sh

This will:

  • Create the Python virtual environment (if needed)
  • Install all dependencies
  • Launch the GUI automatically

Even better for first-time users:

./start.sh --quick

This does everything above plus automatically generates a small dataset and trains a tiny demo model (~2-3 minutes). After it finishes, the AI Calculator will already give reasonable answers instead of random output.

Alternative: Manual setup

If you prefer doing it step by step:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py

Then open: http://127.0.0.1:7860 (or http://localhost:7860)

Fastest way to see it working

  1. Go to Data Lab → Generate a small dataset (or use the one from --quick)
  2. Go to Training Lab → Train a "tiny" model for 3–5 epochs
  3. Go to AI Calculator → Type expressions like 47 + 128 or 300 - 147

Pre-trained Demo Model

Running ./start.sh --quick creates a usable demo model automatically and places it in examples/demo_model. You can also load previous checkpoints from the Model Manager tab.

Troubleshooting the GUI not loading

  • "The link goes nowhere" / blank page:
    Always use http://127.0.0.1:7860 or http://localhost:7860.
    The app deliberately binds to 127.0.0.1 (not 0.0.0.0) because 0.0.0.0 is unreliable when accessed from the same Mac.

  • Make sure you are inside the virtual environment (source .venv/bin/activate).

  • If port 7860 is busy, you can change it temporarily by editing the server_port line near the bottom of app.py.

  • Try a different browser (Chrome / Firefox work best).

Data Format Example

Raw training example (after tokenization):

Input (prompt):  47 + 128 =
Target (model learns to generate): 175

Full training string:

47 + 128 = 175

The loss is only applied to the characters after =.

Why This Matters

This project demonstrates several important ideas:

  • Neural networks can learn algorithmic behavior (addition with carry, subtraction with borrow)
  • You can supervise only the final answer — the model invents its own internal algorithm
  • Small models (~10M params) are surprisingly capable when the task is narrow and data is clean
  • "AI calculators" are a fun and rigorous way to study generalization, distribution shift, and length extrapolation

Extending the Project

Ideas for further work (great for learning):

  • Add multiplication (significantly harder — needs more capacity/data)
  • Support multi-operand expressions or parentheses
  • Add chain-of-thought supervision (47 + 128 = 175 <think> carry 1...)
  • Try encoder-decoder architecture (T5 style) instead of decoder-only
  • Length extrapolation experiments (train on 3 digits, test on 5-6 digits)
  • LoRA fine-tuning on top of a larger base model

Project Structure

Calc-AI/
├── app.py                 # Main Gradio application (start here)
├── calc_ai/
│   ├── __init__.py
│   ├── data_generator.py  # Synthetic arithmetic data creation
│   ├── tokenizer.py       # ArithmeticTokenizer (character level)
│   ├── model.py           # Small transformer model definition
│   ├── trainer.py         # Training loop / HF Trainer wrapper
│   ├── evaluator.py       # Evaluation metrics and analysis
│   └── inference.py       # Pure neural inference (no cheating!)
├── configs/
│   └── default.yaml
├── data/                  # Generated datasets (gitignored)
├── runs/                  # Training run outputs
├── checkpoints/           # Saved models
├── examples/              # Demo checkpoints + sample data
├── requirements.txt
└── README.md

License & Credits

Educational project. Built as a demonstration of neural algorithmic learning.


Important: This calculator does not use any traditional arithmetic libraries at inference time. Every answer you see is generated token-by-token by the neural network.# Calc-AI

About

A small transformer trained from scratch to do arithmetic with no calculation at inference, with synthetic data generation, evaluation on held-out problems, and a Gradio GUI.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages