Skip to content

Add Proofline detector submission - #202

Open
trigeochiral wants to merge 4 commits into
liamdugan:mainfrom
trigeochiral:trigeochiral-english-final
Open

trigeochiral wants to merge 4 commits into
liamdugan:mainfrom
trigeochiral:trigeochiral-english-final

Conversation

@trigeochiral

Copy link
Copy Markdown

Adds a leaderboard submission for Proofline, a CPU-only statistical detector from TriGeoChiral Engineering.

Submission

leaderboard/submissions/proofline/

  • predictions.json — 672,000 rows, full coverage of test.csv (all domains, generator models, decoding strategies and attacks)
  • metadata.json — per leaderboard/template-metadata.json

Detector

  • No neural inference and no LLM in the analysis path; runs on CPU, ~20 docs/sec single-threaded.
  • Scores are P(machine-generated) in [0, 1] — higher means more likely AI.

Training disclosure

Please classify this under the trained-on-RAID section of the leaderboard. The reference corpus was fit on the RAID train split (attack=none baseline, 8 English domains), sampling 3,000 human and 3,000 machine generations with seed 42. No text from test.csv was used at any fit step — the encoder, scaler and classifier are all fit before the test set is read.

Reproducibility

A separately signed benchmark artifact covering the RAID extra split (code, German, Czech × 10 attacks, mean AUC 0.9302) is published at https://github.com/trigeochiral/proofline-proof. It is Ed25519-signed and carries per-document scores, so every published AUC can be independently recomputed:

pip install proofline-verify
proofline-verify --result RAID.signed.json --pubkey pubkey.pem

Happy to adjust the format or rerun anything if something here doesn't match what the eval bot expects.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BMXHCeesVWM1B621qufPJd

@github-actions

Copy link
Copy Markdown

Eval run succeeded! Link to run: link

Here are the results of the submission(s):

proofline

Release date: 2026-09-11

I've committed detailed results of this detector's performance on the test set to this PR.

On the RAID dataset as a whole (aggregated across all generation models, domains, decoding strategies, repetition penalties, and adversarial attacks), it achieved an AUROC of 63.86 and a TPR of 19.29% at FPR=5% and 8.81% at FPR=1%.
Without adversarial attacks, it achieved AUROC of 76.04 and a TPR of 24.23% at FPR=5% and 10.90% at FPR=1%.

If all looks well, a maintainer will come by soon to merge this PR and your entry/entries will appear on the leaderboard. If you need to make any changes, feel free to push new commits to this PR. Thanks for submitting to RAID!

@8bpdkhpryv-pixel

8bpdkhpryv-pixel commented Sep 17, 2026

Copy link
Copy Markdown

If you wouldn't mind I'd appreciate the ability to resubmit tonight. I discovered an additional last step was added to the measurement code on 9/9 that flipped everything like a dipole. If you would be willing to hold off submission and inclusion for tonight I can have it run on the correct code this evening.

Thank you for your consideration. You've put together one of the most important pieces of work in our generation and I'm grateful to be able to utilize it.

David
Proofline

This branch was successfully deployed

1 active (outdated) deployment
raid-main 5ea66c50 Deployed Sep 11, 2026 by trigeochiral via evaluate #343
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants