Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,404 changes: 2,404 additions & 0 deletions reports/20261028T114129/chart.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
99 changes: 99 additions & 0 deletions reports/20261028T114129/report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# jero-benchmarks — run `20261028T114129`

_Generated 2026-10-28T11:54:36+00:00_

## Run configuration

| | |
| --- | --- |
| **Run ID** | `20261028T114129` |
| **Created** | 2026-10-28 11:42:46 UTC |
| **Host** | ip-172-31-29-179.eu-central-1.compute.internal |
| **AWS region** | eu-central-1 |
| **EC2 instance type** | c9g.2xlarge |
| **EC2 AMI** | `ami-0f7d895aa37851cb2` |
| **Commit** | [`e8c5c8cd1556`](https://github.com/RogerThomas/api-benchmarks/commit/e8c5c8cd1556baade2979a5fc7d7c5ca65bfe434) |
| **Images** | `jero-2026.10.28.01`, `postgres-2026.10.28.01`, `runner-2026.10.28.01`, `upstream-2026.10.28.01` |
| **Load generator** | wrk |
| **Connections** | 128 |
| **Duration per attempt** | 60s |
| **Best of** | 3 |
| **Server workers** | 1 |
| **Python server** | peregrine |
| **Event loop** | uvloop |
| **Frameworks** | jero |
| **Results backend** | TaglyDB |

## Results

![jero-benchmarks results grid](chart.svg)

### 1 - GET /info

| Framework | req/s | vs | mean | vs | p99 | vs | succ% | memPk | memAv | cpuAv | cpuPk | cpu ~ | mem ~ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| jero | ~100.7k | x1.00 | 1.26ms | x1.00 | 1.31ms | x1.00 | 100.00 | 44M | 43M | 100% | 103% | ▇▇▇▇▇▇▇▇█▇▇▇▇▇▇▇ | ▇▇▇▇▇▇▇█▇▇▇▇▇▇▇▇ |

## How these results were made

**Isolation.** Exactly one framework container is ever alive at a time, alongside
the shared Postgres/upstream/load-generator infra -- brought up via `compose.base.yml`
plus that framework's own `compose.<fw>.yml`, torn down before the next framework
starts. No two frameworks ever share a core or contend for host resources.

**One worker, one core.** Every service runs exactly 1 worker (peregrine
`--workers 1`, Gin `GOMAXPROCS=1`, Bun's single JS thread, Bolt `--processes 1`),
pinned to one dedicated CPU core via `cpuset` affinity -- not a CFS quota, which
throttles a saturated container every 100ms scheduler period and adds
tail-latency stalls; affinity confines all of a framework's threads (including
GIL-releasing extensions like `msgspec`/`fastpg`) to one core with no such stalls.

**Equal resource budgets.** DB pool and outbound-HTTP pool are capped at **64**
connections for every framework/language.

**Load generator.** wrk (C) or oha (Rust), on two dedicated cores -- fast
enough that the framework, not the load generator, is what saturates.

**Best of N.** Each (framework, test) pair starts with an untimed 3s warm-up,
then runs multiple attempts; the attempt with the highest req/s is kept.

**Success %** is the share of requests that got the expected status. Before
the load, a correctness pass also checks the *body* of 100 responses (key
fields per test); if any is wrong, the whole test is reported as 0% success,
so a fast-but-wrong response can never pass as a success.

**Resource-use columns.** `memPk`/`memAv` = peak/average resident memory (MB);
`cpuAv`/`cpuPk` = average/peak CPU as % of one core (pinned via `cpuset`, so
these sit ≤100%); `cpu ~`/`mem ~` = a 0-to-max-scaled sparkline of the CPU/memory
time series during the attempt. Only the framework's own container is measured
-- Postgres and the upstream are excluded.

**Equal work, per test** -- every framework does the same task; only the
*native* tool differs:

| Framework | Language | Server | HTTP client | DB driver | Validates body (test 2) | Parses upstream (test 3) | JSON serializer |
| --- | --- | --- | --- | --- | --- | --- | --- |
| jero | Python 3.14 | peregrine ASGI | lothc | fastpg | msgspec Struct | typed (msgspec) | msgspec |

**One deliberate exception to per-framework idiom:** Django Bolt
reaches Postgres through raw fastpg -- the same driver the rest of the async
Python fleet uses -- rather than Django's ORM, whose sync-driver-behind-an-async-facade
design would make test 4 a measurement of Django's data layer instead of the
framework. For the same reason, every JWT endpoint hand-decodes the bearer
token with ryjwt rather than using each framework's own auth integration.

**Known, intended differences** (each framework's real idiomatic behaviour, kept
rather than normalised away): the JSON serializer column (stdlib `json` is
slower than msgspec/orjson/native); FastAPI and Django Bolt additionally
re-validate the *response* against its declared model (both frameworks'
default); Elysia returns the upstream payload parsed but unvalidated;
Spring Boot's Jackson binding coerces a numeric field given
as a JSON string rather than rejecting it like pydantic/msgspec do. These are
visible here so a number is never mistaken for pure framework speed.

## Disclosure

The author of this benchmark suite is also the author of [jero](https://pypi.org/project/jero/), one of the frameworks benchmarked here. Every measure above -- equal resource budgets, idiomatic code per framework, isolated single-framework runs, documented intended differences -- applies identically to jero and everyone else; nothing here is tuned in its favor.

---
_Generated by `export_results.py` from `results.is_best` rows in TaglyDB._
84 changes: 84 additions & 0 deletions reports/20261028T114129/results.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
{
"run": {
"run_id": "20261028T114129",
"created_at": "2026-10-28T11:42:46.490784+00:00",
"hostname": "ip-172-31-29-179.eu-central-1.compute.internal",
"vus": 128,
"duration": "60s",
"best_of": 3,
"workers": 1,
"python_server": null,
"loadgen": "wrk",
"ec2_ami_id": "ami-0f7d895aa37851cb2",
"ec2_instance_type": "c9g.2xlarge",
"aws_region": "eu-central-1"
},
"results": [
{
"framework": "jero",
"test": "test1",
"attempt": 1,
"reqs_per_sec": 100535.40902728825,
"latency_avg_ms": 1.265420357,
"latency_p50_ms": 1.266,
"latency_p75_ms": 1.278,
"latency_p90_ms": 1.288,
"latency_p99_ms": 1.314,
"failed_rate": 0.0,
"total_requests": 6038839,
"is_best": false,
"peak_rss_mb": 43.55,
"avg_rss_mb": 43.053225806451614,
"avg_cpu_pct": 96.75612903225807,
"peak_cpu_pct": 101.88,
"cpu_spark": "\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2588\u2587\u2587\u2587\u2584",
"mem_spark": "\u2588\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587",
"images": "postgres-2026.10.28.01 upstream-2026.10.28.01 runner-2026.10.28.01 jero-2026.10.28.01",
"commit": "e8c5c8cd1556baade2979a5fc7d7c5ca65bfe434"
},
{
"framework": "jero",
"test": "test1",
"attempt": 2,
"reqs_per_sec": 100516.88188612513,
"latency_avg_ms": 1.263878125,
"latency_p50_ms": 1.266,
"latency_p75_ms": 1.278,
"latency_p90_ms": 1.29,
"latency_p99_ms": 1.315,
"failed_rate": 0.0,
"total_requests": 6039178,
"is_best": false,
"peak_rss_mb": 43.48,
"avg_rss_mb": 43.146,
"avg_cpu_pct": 100.236,
"peak_cpu_pct": 101.24,
"cpu_spark": "\u2587\u2588\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587",
"mem_spark": "\u2587\u2587\u2587\u2587\u2588\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587",
"images": "postgres-2026.10.28.01 upstream-2026.10.28.01 runner-2026.10.28.01 jero-2026.10.28.01",
"commit": "e8c5c8cd1556baade2979a5fc7d7c5ca65bfe434"
},
{
"framework": "jero",
"test": "test1",
"attempt": 3,
"reqs_per_sec": 100675.52946785002,
"latency_avg_ms": 1.263971352,
"latency_p50_ms": 1.266,
"latency_p75_ms": 1.278,
"latency_p90_ms": 1.29,
"latency_p99_ms": 1.314,
"failed_rate": 0.0,
"total_requests": 6047392,
"is_best": true,
"peak_rss_mb": 43.51,
"avg_rss_mb": 43.211999999999996,
"avg_cpu_pct": 100.17066666666666,
"peak_cpu_pct": 102.53,
"cpu_spark": "\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2588\u2587\u2587\u2587\u2587\u2587\u2587\u2587",
"mem_spark": "\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2588\u2587\u2587\u2587\u2587\u2587\u2587\u2587\u2587",
"images": "postgres-2026.10.28.01 upstream-2026.10.28.01 runner-2026.10.28.01 jero-2026.10.28.01",
"commit": "e8c5c8cd1556baade2979a5fc7d7c5ca65bfe434"
}
]
}
20 changes: 10 additions & 10 deletions services/elysia/bun.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion services/elysia/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@
},
"dependencies": {
"@elysiajs/jwt": "^1.4.2",
"elysia": "^1.4.30"
"elysia": "^1.4.32"
}
}
30 changes: 15 additions & 15 deletions services/gin/go.mod
Original file line number Diff line number Diff line change
@@ -1,46 +1,46 @@
module gin-bench

go 1.26.0
go 1.26.10

require (
github.com/gin-gonic/gin v1.12.0
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.11.0
github.com/jackc/pgx/v5 v5.13.0
)

require (
github.com/bytedance/gopkg v0.1.4 // indirect
github.com/bytedance/sonic v1.15.4 // indirect
github.com/bytedance/sonic v1.15.5 // indirect
github.com/bytedance/sonic/loader v0.5.3 // indirect
github.com/cloudwego/base64x v0.1.7 // indirect
github.com/gabriel-vasile/mimetype v1.4.16 // indirect
github.com/gin-contrib/sse v1.1.2 // indirect
github.com/gin-contrib/sse v1.2.1 // indirect
github.com/go-playground/locales v0.14.2 // indirect
github.com/go-playground/universal-translator v0.18.2 // indirect
github.com/go-playground/validator/v10 v10.30.5 // indirect
github.com/goccy/go-json v0.11.2 // indirect
github.com/go-playground/validator/v10 v10.30.6 // indirect
github.com/goccy/go-json v0.12.0 // indirect
github.com/goccy/go-yaml v1.19.2 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.3 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/cpuid/v2 v2.4.1 // indirect
github.com/klauspost/cpuid/v2 v2.4.2 // indirect
github.com/leodido/go-urn v1.5.0 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pelletier/go-toml/v2 v2.4.3 // indirect
github.com/quic-go/qpack v0.6.0 // indirect
github.com/quic-go/quic-go v0.64.1 // indirect
github.com/quic-go/quic-go v0.65.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.3.2 // indirect
go.mongodb.org/mongo-driver/v2 v2.9.2 // indirect
golang.org/x/arch v0.32.0 // indirect
golang.org/x/crypto v0.58.0 // indirect
golang.org/x/net v0.61.0 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/sys v0.49.0 // indirect
golang.org/x/text v0.43.0 // indirect
go.mongodb.org/mongo-driver/v2 v2.10.0 // indirect
golang.org/x/arch v0.33.0 // indirect
golang.org/x/crypto v0.59.0 // indirect
golang.org/x/net v0.62.0 // indirect
golang.org/x/sync v0.25.0 // indirect
golang.org/x/sys v0.50.0 // indirect
golang.org/x/text v0.44.0 // indirect
google.golang.org/protobuf v1.36.12 // indirect
)
Loading