[leeclemnet]

What does this PR do?

Runtimes and processors replace per-runtime subclasses

What it does: Each benchmark row is now a runtime class (how to load and time an artifact) plus a processor class (how one model family does preprocessing and postprocessing). All 11 scripts use this design. Runs now resume, can be filtered, and write each row as soon as it is complete. uv replaces requirements.txt and setup.py.
Why: Prepares SAB for OpenVINO, LiteRT, ExecuTorch, Core ML and Hailo runtimes. This PR adds no new runtime.

Change map

• Runtimes (sab/runtimes/) — Runtime base class. ONNXRuntime(device) replaces the ONNX CPU and CUDA classes. TRTRuntime replaces TRTInference and now builds its engine itself. The timed region is unchanged.
• Processors (sab/processors.py, sab/models/benchmark_*.py) — Processor holds preprocess, postprocess and extra_inputs. Pipeline joins a runtime and a processor. Each script has one <Family>Processor and a build_requests() that gives the same rows as before.
• Request, run control, results (sab/request.py, runner.py, results.py, artifacts.py, host.py, benchmark_all.py) — artifact_path, runtime, processor, device and precision replace onnx_path, inference_class and needs_fp16. The runner filters, skips, resumes and saves atomically after each row. .zip artifacts are unpacked.
• Monitors (sab/monitors/) — the NVML monitor moved from clock_watch.py with no change. CpufreqMonitor replaces CPUFrequencyMonitor, which read /proc/cpuinfo and always said "stable" on ARM and macOS.
• Packaging, tests, CI (pyproject.toml, uv.lock, tests/, .github/workflows/tests.yml) — uv extras for each runtime and host. TensorRT is pinned to 10.x (cu13) from the NVIDIA index. There is a pytest suite, and CI runs it on Ubuntu and macOS.

Behavior changes

• Install: pip install -r requirements.txt → uv sync --extra nvidia.
• Results JSON: the is_trt, is_cpu and needs_fp16 keys → runtime, device, precision, processor, plus a host block. pretty_print_results reads only the new schema.
• Output file: written once at the end → rewritten after each row. A second run keeps the rows that are already complete, unless you pass --rerun.
• New flags on benchmark_all and each script: --runtimes, --devices, --max_images, --rerun.
• throttled: always false on hosts with no CPU frequency signal → null, and the table shows ?. The monitor now watches only the timed pass, not the COCO scoring after it.
• A results file from the old SAB: SAB drops its rows at load, and prints the file name and the number of rows.
• TensorRT: tensorrt==10.4 → tensorrt-cu13>=10.13,<11. Engines cached from 10.4 do not load; SAB builds them again.
• yololite ONNX rows: preallocated outputs → ONNX Runtime allocates the outputs inside the timed call.

Watch out for

• sab/runtimes/tensorrt.py:334 — orig_target_sizes (dfine, rtdetr) now comes from Processor.extra_inputs, not from fill_(1) on the persistent tensor. It must stay all ones, or the boxes change.
• sab/runtimes/onnxruntime.py:59 — dynamic_output_shapes=True moves output allocation into the timed call. yololite ONNX latency can rise a little.
• sab/monitors/cpufreq.py:71 — the monitor counts only the polls inside busy(), which run_timed_pass puts around each infer. The baseline is the first busy poll, not the boosted read after the warm-up. So a scaling governor that slows idle cores between images does not give a throttle report.
• pyproject.toml:27 — TensorRT 11 removes EXPLICIT_BATCH and the FP16/INT8 builder flags, so the <11 cap must stay until the build code changes.
• sab/runner.py:136 — the resume key is (artifact_path, runtime, device, precision, max_images, buffer_time, max_dets). A changed artifact under the same name keeps its old row unless you pass --rerun.

Related Issue(s): n/a

Type of Change

• Other: a refactor with no change to the timed region. It breaks the results JSON schema and the install method.

Testing

• Unit tests
• Before/after parity test for rfdetr and yolo26 on T4 VM

Old/new parity on a Tesla T4 (lee-t4-dev-cu128): rfdetr and yolo26, first 500 COCO val2017 images, buffer time 0.

Both versions ran in the same venv (TensorRT 10.16 cu13, torch 2.14 cu130, onnxruntime-gpu 1.30) and in the same directory. So the new code used the TensorRT engines that the old code built, and only the code differs.

• mAP: all 12 COCOeval values are identical on 24/24 rows.
• TensorRT, no throttling in either run (4 rows): the largest median latency difference is 0.5%.
• TensorRT, throttled in both runs (12 rows): differences from −7.3% to +3.4%. SAB marks these latencies as unreliable. With buffer time 0, the larger models throttle the T4.
• ONNX Runtime CPU (8 rows): the largest median latency difference is 4.9%. The new code shows ? for throttling, because this VM has no cpufreq in /sys. The old code read a fixed cpu MHz value from /proc/cpuinfo, and so it always reported no.

Artifact	Runtime	Device	Precision	mAP50-95 old	mAP50-95 new	COCOeval stats	Median ms old	Median ms new	Δ latency	Throttled old / new
rf-detr-nano.onnx	tensorrt	gpu	fp32	53.8	53.8	identical	8.81	9.11	+3.4%	yes / yes
rf-detr-nano.onnx	tensorrt	gpu	fp16	53.9	53.9	identical	2.33	2.34	+0.5%	no / no
rf-detr-nano.onnx	onnxruntime	cpu	fp32	53.8	53.8	identical	105.82	108.26	+2.3%	no / ?
rf-detr-small.onnx	tensorrt	gpu	fp32	58.3	58.3	identical	17.78	18.00	+1.2%	yes / yes
rf-detr-small.onnx	tensorrt	gpu	fp16	58.3	58.3	identical	3.46	3.49	+1.0%	yes / yes
rf-detr-small.onnx	onnxruntime	cpu	fp32	58.3	58.3	identical	170.89	173.44	+1.5%	no / ?
rf-detr-medium.onnx	tensorrt	gpu	fp32	59.3	59.3	identical	23.95	23.98	+0.1%	yes / yes
rf-detr-medium.onnx	tensorrt	gpu	fp16	59.3	59.3	identical	4.38	4.40	+0.4%	yes / yes
rf-detr-medium.onnx	onnxruntime	cpu	fp32	59.3	59.3	identical	226.04	237.09	+4.9%	no / ?
yolo26n.onnx	tensorrt	gpu	fp32	44.5	44.5	identical	3.10	3.11	+0.3%	no / no
yolo26n.onnx	tensorrt	gpu	fp16	44.4	44.4	identical	1.61	1.61	+0.1%	no / no
yolo26n.onnx	onnxruntime	cpu	fp32	44.5	44.5	identical	40.23	38.92	-3.3%	no / ?
yolo26s.onnx	tensorrt	gpu	fp32	52.1	52.1	identical	6.71	6.60	-1.7%	yes / yes
yolo26s.onnx	tensorrt	gpu	fp16	52.2	52.2	identical	2.49	2.49	+0.0%	no / no
yolo26s.onnx	onnxruntime	cpu	fp32	52.1	52.1	identical	67.88	65.54	-3.5%	no / ?
yolo26m.onnx	tensorrt	gpu	fp32	57.7	57.7	identical	18.07	17.33	-4.1%	yes / yes
yolo26m.onnx	tensorrt	gpu	fp16	57.7	57.7	identical	4.45	4.40	-0.9%	yes / yes
yolo26m.onnx	onnxruntime	cpu	fp32	57.7	57.7	identical	157.69	152.24	-3.5%	no / ?
yolo26l.onnx	tensorrt	gpu	fp32	59.5	59.5	identical	22.57	21.85	-3.2%	yes / yes
yolo26l.onnx	tensorrt	gpu	fp16	59.6	59.6	identical	5.83	5.75	-1.4%	yes / yes
yolo26l.onnx	onnxruntime	cpu	fp32	59.5	59.5	identical	193.24	188.71	-2.3%	no / ?
yolo26x.onnx	tensorrt	gpu	fp32	61.7	61.7	identical	50.71	46.99	-7.3%	yes / yes
yolo26x.onnx	tensorrt	gpu	fp16	61.7	61.7	identical	11.39	10.74	-5.8%	yes / yes
yolo26x.onnx	onnxruntime	cpu	fp32	61.7	61.7	identical	378.24	384.46	+1.6%	no / ?

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have updated the documentation accordingly (if applicable)

Additional Context

• Not in this PR: benchmark_yololite.main still takes onnx_path as its first argument, so benchmark_all cannot run it (as before).

🤖 Generated with Claude Code

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​numpy@​2.5.3
	pypi/​tqdm@​4.70.1
	pypi/​torch@​2.14.0
	pypi/​onnxruntime@​1.30.0
	pypi/​onnxruntime-gpu@​1.30.0
	pypi/​onnx@​1.23.0
	pypi/​torchvision@​0.29.0
	pypi/​tensorrt-cu13@​10.16.1.11
	pypi/​tensorrt-cu13-bindings@​10.16.1.11
	pypi/​tensorrt-cu13-libs@​10.16.1.11
	pypi/​pillow@​12.3.0
	pypi/​pytest@​9.1.1
	pypi/​supervision@​0.30.5
	pypi/​fire@​0.7.1
	pypi/​requests@​2.34.2
	pypi/​faster-coco-eval@​1.8.0
	pypi/​pycocotools@​2.0.11

View full report