A production-oriented assessment implementation that classifies support messages as Billing, Technical, Account, Delivery, or General. The production model uses pretrained MiniLM sentence embeddings with Logistic Regression; FastAPI exposes cached inference and an optional low-confidence LLM fallback.
flowchart TD
A[Customer message] --> B[Validation and whitespace normalization]
B --> C[all-MiniLM-L6-v2 encoder]
C --> D[384-dimensional semantic embedding]
D --> E[Logistic Regression]
E --> F[Class probabilities]
F --> G{Confidence at least threshold?}
G -->|Yes| H[ML result]
G -->|No; fallback disabled| H
G -->|No; fallback enabled| I[Groq or Gemini]
I --> J[Validated label or safe ML result]
MiniLM is a frozen pretrained feature extractor; it is not fine-tuned. models/classifier.joblib contains only Logistic Regression. models/model_metadata.json identifies the encoder, labels, parameters, split, timestamp, and measured metrics. The encoder, classifier, and metadata load once and are reused.
data/support_tickets.csv remains unchanged at 100 unique examples, exactly 20 per class. Validation checks schema, missing/empty values, duplicates, labels, and class counts. The fixed random_state=42 stratified split contains 80 training and 20 held-out examples. Hyperparameters were selected with stratified five-fold cross-validation on the training partition; the holdout was not used for tuning.
Semantic preprocessing only validates input, trims it, and normalizes repeated whitespace. It deliberately preserves case and punctuation because pretrained transformers can use those signals. The original lightweight cleanup remains available for reproducible TF-IDF baselines.
Traditional sparse-text models established a fast, interpretable baseline but depend heavily on lexical overlap. MiniLM brings external pretrained semantic knowledge, so “I want my money back” and “I need a refund” can be close even without identical wording.
| Model | CV accuracy | CV std. | CV macro F1 | Held-out accuracy | Held-out macro F1 | Billing F1 |
|---|---|---|---|---|---|---|
| Multinomial Naive Bayes | 76.2% | 7.3% | 76.2% | 70% | 69.1% | 66.7% |
| TF-IDF Logistic Regression | 73.8% | 7.3% | 74.5% | 70% | 69.9% | 66.7% |
| TF-IDF Linear SVM | 75.0% | 8.8% | 75.8% | 70% | 69.1% | 66.7% |
| Character TF-IDF + Logistic Regression | 77.5% | 10.2% | 78.1% | 70% | 69.6% | 85.7% |
| Combined TF-IDF + Logistic Regression | 77.5% | 10.2% | 78.4% | 75% | 74.8% | 85.7% |
| MiniLM + Logistic Regression (production) | 97.5% | 3.1% | 97.3% | 95% | 94.9% | 100% |
| MiniLM + Linear SVM | 97.5% | 3.1% | 97.3% | 95% | 94.9% | 100% |
Production uses Logistic Regression (C=0.1, balanced weights) because it matches semantic SVM performance and provides predict_proba. Held-out macro precision is 96%, macro recall 95%, and Billing precision/recall/F1 are 100% on four Billing examples.
The held-out test set contains only 20 examples, so the 95% result represents 19 correct predictions out of 20 and should not be interpreted as a precise estimate of production performance.
Probabilities are useful confidence signals, not guaranteed calibration. A larger dataset should support calibration curves and possibly CalibratedClassifierCV.
Requires Python 3.11+ and internet access for the first MiniLM download.
python -m venv venvWindows:
venv\Scripts\activatemacOS/Linux:
source venv/bin/activatepip install -r requirements.txt
python scripts/validate_data.py
python scripts/compare_models.py
python scripts/train.py
python scripts/evaluate.py
pytest
python run.pyAlways regenerate models/classifier.joblib after changing Python or scikit-learn environments. Joblib model artifacts are not guaranteed to be compatible across scikit-learn versions.
If MiniLM is not cached and the machine is offline, training/startup fails with a message explaining that the model must be downloaded once. For a truly offline deployment, package the Hugging Face cache or a local model directory.
GET /— service statusGET /health— component health and semantic model identityPOST /api/v1/predict— predictionGET /docs— Swagger UI
Open http://127.0.0.1:8000/docs, or call:
curl -X POST http://127.0.0.1:8000/api/v1/predict -H "Content-Type: application/json" -d '{"message":"I forgot my password"}'Blank messages return HTTP 422. Missing or unloadable model components return HTTP 503.
Fallback is disabled by default and the semantic classifier requires no LLM key. Copy .env.example to .env, configure Groq or Gemini, and set LLM_FALLBACK_ENABLED=true. Groq defaults to the configurable GROQ_MODEL=openai/gpt-oss-20b. Below CONFIDENCE_THRESHOLD=0.22, the service may request one strict allowed label. The old 0.55 threshold flagged every held-out prediction because strong regularization compresses probabilities near 0.20–0.27. The 0.22 default was selected from training-only out-of-fold confidence behavior (12.5% flagged), not by optimizing the test set. Confidence did not cleanly separate all cross-validation errors, so this remains a routing heuristic that needs calibration and production monitoring. Missing keys, timeouts, network failures, malformed results, or unsupported labels safely retain the ML result. Secrets and message bodies are not logged.
docker build -t ticket-classifier .
docker run -p 8000:8000 ticket-classifierThe Docker build installs CPU-only PyTorch, downloads MiniLM before training, and makes first container startup independent of network access. This substantially increases build time and image size: Sentence Transformers, PyTorch, and model weights are much heavier than the former ~90 KB TF-IDF artifact.
Overall accuracy can conceal a business-critical class failure. Inspect Billing precision, recall, F1, false positives, false negatives, and its confusion-matrix row and column. Audit imbalance, coverage, annotation quality, vocabulary drift, products/channels, and ambiguous multi-intent tickets. “My card was charged but my order never arrived” requires a documented policy deciding Billing, Delivery, multi-label handling, or escalation.
Improve representative Billing data and annotation rules, correct labels, compare features/classifiers, consider class weighting or a business-justified threshold, and route uncertain/high-impact cases to humans. Monitor Billing metrics separately in production. Current test Billing recall is 100%, but it covers only four examples and is not evidence of perfect real-world performance.
This is production-style, not fully production-ready. The dataset is small, synthetic, English-only, and lacks real operational diversity. Monitor request volume, latency, errors, class/confidence distributions, fallback rate, drift, and reviewed per-class performance. Human corrections should feed a validated retraining and approval workflow.
A real deployment also needs authentication, rate limiting, HTTPS, PII controls, managed secrets, model versioning, CI/CD, scalable serving, drift monitoring, alerting, rollback, and a larger historical dataset.