Skip to content

docs: repo polish pass (v2) #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
OneByJorah merged 1 commit into from
Oct 10, 2026
Merged

docs: repo polish pass (v2) #5

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 23 additions & 109 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,20 @@
<div align="center">

![BenchDash banner](docs/assets/banner.svg)

# BenchDash

**A benchmarking platform for local LLMs on Ollama** — auto-discover models, run structured tests, and compare accuracy, latency, and throughput.

<a href="https://github.com/OneByJorah/BenchDash/stargazers"><img src="https://img.shields.io/github/stars/OneByJorah/BenchDash?style=flat-square" alt="Stars"></a>
<a href="https://github.com/OneByJorah/BenchDash/commits"><img src="https://img.shields.io/github/last-commit/OneByJorah/BenchDash?style=flat-square" alt="Last commit"></a>
<a href="LICENSE"><img src="https://img.shields.io/github/license/OneByJorah/BenchDash?style=flat-square" alt="License"></a>
<img src="https://img.shields.io/badge/python-3.11%2B-blue?style=flat-square&logo=python&logoColor=white" alt="Python">
<img src="https://img.shields.io/badge/ollama-000000?style=flat-square&logo=ollama&logoColor=white" alt="Ollama">
<img src="https://img.shields.io/badge/status-pre--alpha-orange?style=flat-square" alt="Pre-alpha">
> A benchmarking platform for local LLMs on Ollama — auto-discover models, run structured test suites, and compare accuracy, latency, and token throughput.

</div>
[![License](https://img.shields.io/github/license/OneByJorah/BenchDash?style=for-the-badge&color=FFB300&labelColor=0a0a09)](https://github.com/OneByJorah/BenchDash)
[![Top Language](https://img.shields.io/github/languages/top/OneByJorah/BenchDash?style=for-the-badge&color=FFB300&labelColor=0a0a09)](https://github.com/OneByJorah/BenchDash)
[![Stars](https://img.shields.io/github/stars/OneByJorah/BenchDash?style=for-the-badge&color=FFB300&labelColor=0a0a09)](https://github.com/OneByJorah/BenchDash/stargazers)
[![Last Commit](https://img.shields.io/github/last-commit/OneByJorah/BenchDash?style=for-the-badge&color=FFB300&labelColor=0a0a09)](https://github.com/OneByJorah/BenchDash/commits)

![BenchDash screenshot](docs/assets/screenshot.png)
![BenchDash dashboard](docs/screenshots/dashboard-desktop.png)

## What This Is

Comparing local models usually means ad-hoc prompts and a stopwatch. BenchDash is a design-first platform that aims to make that repeatable: discover the models Ollama has loaded, run a fixed task suite, and rank results on accuracy, latency, and token throughput. Today the collector, dashboard, and packaging exist; the benchmark engine and scoring pipeline are planned.
Comparing local LLMs usually means ad-hoc prompts and a stopwatch. BenchDash aims to make that repeatable: discover the models Ollama has loaded, run a fixed task suite, and rank results on accuracy, latency, and throughput. Today the system-telemetry collector, dashboard UI, Docker packaging, and install scripts exist; the benchmark engine and scoring pipeline are on the roadmap.

> [!WARNING]
> **Pre-alpha.** The dashboard currently displays sample/demo data. The benchmark engine, scoring pipeline, JSON persistence, and backend server are not yet implemented — this scaffolding is not functional end-to-end.
> **Pre-alpha.** The dashboard currently displays sample data. The benchmark engine, scoring pipeline, JSON persistence, and backend server are not yet implemented — the scaffolding does not run end-to-end.

## Quick Start

Expand All @@ -32,113 +24,35 @@ cd BenchDash
docker compose up -d
```

Open **http://localhost:8081**. Run the dashboard locally with `python -m http.server 8081` instead if you prefer.
Open **http://localhost:8081**. Or serve the dashboard directly: `python -m http.server 8081`.

## Features

- **System telemetry collector** — gathers CPU, GPU, VRAM, CUDA, drivers, memory, and OS info in Python.
- **Static dashboard** — standalone `index.html` with no build step; shows model comparisons, scores, and system metrics.
- **Ollama integration (planned)** — queries the local Ollama API to enumerate models and run inference benchmarks.
- **Configurable benchmarks** — YAML task definitions for categories, weights, and scoring.
- **JSON persistence (planned)** — timestamped result files for historical comparison.
- **Docker support** — Alpine image and Compose file for containerized deployment.
- **Install scripts** — `install.sh` (Linux/macOS) and `install.ps1` (Windows).
- **System telemetry collector** — CPU, GPU, VRAM, CUDA, drivers, memory, and OS info gathered in Python (`collector/system_info.py`).
- **Static dashboard** — standalone `index.html` (no build step) showing model comparisons and system metrics.
- **Ollama integration (planned)** — enumerate local models via the Ollama API and run inference benchmarks.
- **Configurable benchmark tasks** — YAML definitions for categories, weights, and scoring (`j1.yaml`).
- **JSON persistence (planned)** — timestamped results for historical regression tracking.
- **Container packaging** — Alpine image with healthcheck; `install.sh` and `install.ps1` installers.

## Architecture

```mermaid
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#0a0a09','primaryTextColor':'#FFB300','lineColor':'#FFB300'}}}%%
graph LR
A[Dashboard<br/>HTML/JS] --> B[Collector<br/>Python]
B --> C[Ollama<br/>Local LLMs]
B --> D[JSON Store<br/>Results]
```
┌─────────────┐ ┌──────────────┐ ┌────────────────┐
│ Dashboard │────▶│ Collector │────▶│ Ollama │
│ (HTML/JS) │ │ (Python) │ │ (Local LLMs) │
└─────────────┘ └──────────────┘ └────────────────┘
│ │
│ ▼
│ ┌──────────────┐
└───────────▶│ JSON Store │
│ (Results) │
└──────────────┘
```

## Implementation Status

| Component | Status |
|---|---|
| System info collector | Implemented |
| Dashboard UI (sample data) | Implemented |
| Docker packaging | Implemented |
| Install scripts | Implemented |
| Benchmark engine | Planned |
| Scoring pipeline | Planned |
| JSON persistence | Planned |
| Scheduler (cron) | Planned |
| Notifications | Planned |
| Flask/FastAPI backend | Planned |

## Configuration

| Variable | Default | Description |
|---|---|---|
| `OLLAMA_HOST` | `http://host.docker.internal:11434` | Ollama API endpoint |
| `DASHBOARD_PORT` | `8081` | Dashboard port |
| `DASHBOARD_HOST` | `<ip-address>` | Dashboard bind host |
| `BENCH_SKIP_MEDIA` | `0` | Set to `1` to skip image/audio/video tests |
| `BENCH_DATA_DIR` | `/app/results` | Directory for benchmark results |
| `TELEGRAM_BOT_TOKEN` / `TELEGRAM_CHAT_ID` | — | Optional notifications |

## Development

```bash
git clone https://github.com/OneByJorah/BenchDash.git
cd BenchDash

python -m venv .venv
source .venv/bin/activate # Linux/macOS
# .venv\Scripts\activate # Windows

python -m http.server 8081 # serve the dashboard locally
```

See [INTENT.md](INTENT.md) for the design specification.

## Project Structure

```
BenchDash/
├── index.html # Standalone dashboard UI (no build step)
├── collector/
│ └── system_info.py # System telemetry collector (CPU, GPU, RAM)
├── docs/assets/ # Banner, screenshots
├── Dockerfile # Alpine + thttpd, serves on port 8081
├── docker-compose.yml # Container orchestration with healthcheck
├── install.sh # Linux/macOS installer
├── install.ps1 # Windows installer
├── requirements.txt # Python dependencies
├── j1.yaml # Benchmark task definitions
├── INTENT.md # Design specification
└── .env.example # Environment variable template
```

## Tech Stack
## Stack

Python 3.11+ · Ollama · HTML/CSS/JS · Docker (Alpine + thttpd) · Docker Compose

## Screenshots

| View | |
|---|---|
| ![desktop](docs/screenshots/dashboard-desktop.png) | ![full dashboard](docs/screenshots/dashboard-full.png) |
| ![mobile](docs/screenshots/dashboard-mobile.png) | ![main viewport](docs/screenshots/main.viewport.full.png) |

## Contributing

Contributions are welcome. Please read [INTENT.md](INTENT.md) for the design specification before submitting a PR. [Open an issue](https://github.com/OneByJorah/BenchDash/issues).
Read [INTENT.md](INTENT.md) for the design spec before opening a PR. [Open an issue](https://github.com/OneByJorah/BenchDash/issues) for bugs or ideas.

## License

MIT — see [LICENSE](LICENSE).

## Connect

- [jorahone.com](https://jorahone.com)
- [GitHub Org](https://github.com/OneByJorah)
- [info@jorahone.com](mailto:info@jorahone.com)
Loading