Skip to content

Refresh the large-bibliography benchmark results (Phase A4) - #153

Merged
dknauss merged 1 commit into
mainfrom
claude/keen-tesla-49orwb-google-key
Oct 14, 2026
Merged

dknauss merged 1 commit into
mainfrom
claude/keen-tesla-49orwb-google-key

Conversation

@dknauss

@dknauss dknauss commented Oct 14, 2026

Copy link
Copy Markdown
Owner

Summary

Phase A, ticket 4 of docs/planning/refactor-and-standards-memo.md.

This reruns the May 2026 benchmark against 1.11.0 plus main, which now has the rewritten CSL styles and the particle-aware sort. It uses the existing harnesses: scripts/benchmark-formatter.php for PHP and npm run benchmark:perf for JS, 5 runs per cell. The results doc gets new tables, an honest machine note, and a comparison section.

Worst p95 at 200 entries:

Layer Worst p95 Budget
PHP formatter (citeproc-php) 45.3 ms (author-date, cold) 250 ms
JS style switch 14.8 ms 250 ms
JS mutation 26.8 ms 150 ms

Every cell is at least 5× under budget.

Against May. Absolute times are about 2–3× higher in PHP and about 1.3–1.8× higher in JS. The runs came from different machines (a Linux cloud container on a 2.1 GHz Xeon, not an M-series Mac), so the doc treats the change as direction only, not as a regression.

The one real change: author-date is now the slowest style at 200 entries (about 45 ms against notes' 36 ms), where in May they matched. That could be the rewritten style or noise. Only a same-machine run of old styles against new ones would settle it.

A follow-up this doesn't cover. This PR's sibling, #152, ran the runtime-matrix timing of a 200-entry /format request over real HTTP (nginx, PHP 8.3). It measured a p50 of about 205 ms for Chicago notes, against about 71–73 ms for author-date and IEEE. In-process, the same style formats in about 34 ms. So most of the HTTP cost for notes sits outside citeproc's render: request boot, sanitization, cache miss, or CiteProc setup for that style. It's still under the 250 ms budget, but it's the closest number to it anywhere. Worth profiling separately; it's not something this doc PR should chase.

Validation

  • Docs only. Every number comes from runs made for this PR. The raw reports went to the gitignored output/benchmarks/.
  • The PHP harness already existed on main; no new code.

Checklist

  • No code changed
  • Numbers reproducible with the commands named in the doc

🤖 Generated with Claude Code

https://claude.ai/code/session_01KbVuMPN7YTcV27vwsG3LZ2


Generated by Claude Code

Re-run the PHP formatter (scripts/benchmark-formatter.php) and JS
orchestration (npm run benchmark:perf) benchmarks on current main, after
the CSL style rewrite and particle-aware sorter. Notes the Linux container
vs. May macOS difference and adds a comparison section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbVuMPN7YTcV27vwsG3LZ2
@dknauss
dknauss marked this pull request as ready for review October 14, 2026 23:18
@codecov

codecov Bot commented Oct 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.28%. Comparing base (4d638a2) to head (2757f98).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main     #153   +/-   ##
=========================================
  Coverage     88.28%   88.28%           
  Complexity       22       22           
=========================================
  Files            61       61           
  Lines          7290     7290           
  Branches        690      690           
=========================================
  Hits           6436     6436           
  Misses          291      291           
  Partials        563      563           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dknauss
dknauss merged commit 53623ec into main Oct 14, 2026
19 checks passed
@dknauss
dknauss deleted the claude/keen-tesla-49orwb-google-key branch October 14, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants