Skip to content

Add an overhead benchmark workflow for AD vs double builds - #55

Merged
auto-differentiation-dev merged 3 commits into
mainfrom
feature/overhead-benchmarking-v2
Nov 12, 2026
Merged

auto-differentiation-dev merged 3 commits into
mainfrom
feature/overhead-benchmarking-v2

Conversation

@auto-differentiation-dev

Copy link
Copy Markdown
Contributor

Adds a workflow that measures what XAD's active type costs QuantLib when no derivatives are computed. It builds QuantLib twice on the same runner with QuantLibAAD's build defaults, once with Real = xad::AReal<double> and once with Real = double, times every test-suite case and the QuantLib benchmark in interleaved, repeated runs, and reports AD/double time ratios per test. Ratios rather than absolute times, because hosted runners are noisy. Builds on #54, whose defaults it measures.

Changes

  • .github/workflows/overhead.yaml: manual run with inputs for the XAD ref (default v2.2.0), the QuantLib ref (default a recent master commit that builds with AD; master itself sometimes needs an upstream fix first) and the number of repeats (default 3), and a repository_dispatch hook of type xad-release (client_payload.xad_ref, optionally ql_ref) that other repositories can use. Jobs for Linux GCC, Linux Clang and Windows MSVC.
  • tools/benchmark/run_benchmarks.py: runs the test suite (one Boost.Test JUNIT log per run) and the benchmark of both builds, alternating which build goes first. It checks that the two build dirs are a double and an AD build made with the same compiler, and that they differ in nothing else, and records the build options with the results. A failing test or a benchmark run without its timing table fails the job, but the summary is still written.
  • tools/benchmark/aggregate.py: per-test medians and ratios to results.csv, and a job summary with the total run time ratio, the median and geometric mean of the per-test ratios, the 20 tests with the highest overhead and the number of runs used. Failed tests, benchmark tests that threw and runs that crashed or can't be read are left out; a source without any usable run fails the job.
  • tools/benchmark/README.md: how to run it on GitHub and locally, and what the numbers mean.
  • .github/workflows/ci.yaml: a small job that runs the scripts' unit tests on every push and pull request.

Testing

  • Unit tests for the scripts: python3 -m pytest tools/benchmark/tests (log parsing, broken or empty logs, interleaving, medians, ratios, geometric mean, worst tests, CSV and summary output, build and argument checks).
  • Workflow checked with actionlint.
  • Scripts run end to end on Linux with GCC, on top of Faster AD and builds by default #54, against XAD v2.2.0 and QuantLib: a filtered suite run, then the full suite and the benchmark three times on an AMD and an Intel machine.

Closes #33

@auto-differentiation-dev

Copy link
Copy Markdown
Contributor Author

CI against QuantLib master plus lballabio/QuantLib#3068 (which fixes master's test suite with AD types): all jobs pass. https://github.com/auto-differentiation/QuantLibAAD/actions/runs/43262133585 The red checks on this PR come from QuantLib master's test suite not compiling with AD until lballabio/QuantLib#3068 is merged.

@auto-differentiation-dev
auto-differentiation-dev merged commit def38e3 into main Nov 12, 2026
29 of 43 checks passed
@auto-differentiation-dev
auto-differentiation-dev deleted the feature/overhead-benchmarking-v2 branch November 12, 2026 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Overhead Benchmarking

1 participant