Repository navigation
OIE: reduce cost of full OWASP ingestion (~$200 per 700 repos) #1160
Description
Context
A full OWASP ingestion run (700 repos via $0.29/repo). Embeddings are cheap, so the spend is almost certainly LLM calls (extraction, mapping judge, shortlist pick). The cost split has not been profiled yet.scripts/run_oie_exhaustive_incremental.py) costs roughly $200 (
Gate
Do not build until the accuracy work gives good combinations (golden-set scores acceptable). Each lever below must be measured against the golden set, and must not regress accuracy.
Levers (roughly by expected savings per effort)
- 0. Profile first: log tokens and cost per pipeline stage and per repo, so the levers below can be ranked on real numbers.
- 1. Embedding prefilter: compare chunks against existing CRE/standard embeddings and only send those above a similarity threshold to the LLM (likely 5–10x).
- 2. File selection heuristics: README, docs/, SECURITY.md and top-level markdown only. Skip lockfiles, vendored, tests and generated files. Cap bytes per repo.
- 3. Tiered models: a cheap model (Haiku / Flash-Lite) for the bulk pass, escalating to a larger one only on low confidence or a thin margin (extends
candidate_picker.py). - 4. Finer incremental skipping: per-file content hashes on top of the existing per-repo SHA skip.
- 5. Batch API + prompt caching for backfills (typically ~50% off; cached taxonomy/system prefix).
- 6. Tighter output: structured IDs only, no rationale text.
- 7. Tiered depth by project: full pass for flagship/production repos, README-only for the long tail.
- 8. Retry audit: make sure the "retry failures ≤5" loop does not resend full contexts on parse/timeout failures.
Target
Plausibly ~$200 → $15–30 per full run with levers 1–3, and roughly half again with 5. These are estimates, to be validated by lever 0.
Acceptance
- Cost per full run is measured before and after.
- Golden-set accuracy is no worse than baseline (state the tolerance when picking this up).
Activity
northdpole commentedon Oct 20, 2026
Progress on the comprehension-stage cost levers (PR #1158), mapped to this ticket's levers. Nothing here changes the existing OIE extraction/mapping stages yet, and the golden-set (B2) baseline for the 1-point tolerance is still to be measured.
scripts/comprehend_standard.py --estimateprints per-repo and total estimated cost (digest / overview / embeddings / raw-only) from document and chunk sizes with no LLM calls. Not yet run across all ~700 repos.--skip-doneskips repos whose comprehension overview already exists. Per-file content hashes: not yet.Measured finding that matters for the cost of a full import (252-question sample, same questions): raw source chunks only scored 50% correct, per-document digests only 51%, both together 53%. So a full import can be raw chunks plus a cheap context/question enrichment pass (Flash-Lite, about 11,500 chunks cost ~$2 for 16 repos) instead of full digests. My estimate for all ~700 repos is about $25 versus $100-170 with digests; please treat those as estimates until the pilot runs.
Per-question cost note: the best accuracy lever so far (reading a whole named project instead of snippets, +3 points on the full 589-question set) raises the input per question. Worst case is about 175k tokens (~$0.05 at Flash list prices); typical projects are far smaller and measured latency stayed at ~20 s. Context caching for hot projects would cut this and belongs under lever 5.
northdpole commentedon Oct 20, 2026
Lever 0 numbers: I ran
scripts/comprehend_standard.py --estimateover all 700 harvested repos (14,865 documents, 84,300 chunks). It uses public list prices and document/chunk sizes only, so treat it as an estimate until a pilot reports real token counts.Most expensive repos by the estimate: cornucopia ($7.5), WSTG ($4.5), MASTG ($4.4), the LLM Top 10 ($2.6).
Given the earlier finding (raw only 50% vs digests only 51% vs both 53% on the same 252 questions) the plan for the full import is raw chunks + enrichment, which is roughly a fifth of the full-digest cost. Still to do for this ticket: a ~20-repo pilot with real token counts, batch API / context caching, per-file content hashes, and the B2 golden-set baseline.
northdpole commentedon Oct 21, 2026
Cost pilot result (staging, 19 repos, raw chunks + skipped files + enrichment with Flash-Lite, 8 workers):
$0.00027 per chunk all-in. Extrapolated to the full corpus (84,300 chunks): **$23** (LLM ~$17.8 + embeddings ~$5), ~7.7 h at this speed.--skip-unchanged(resume / incremental): re-running DevGuide and FIASSE found 0 of 471 chunks changed and made 0 LLM calls (cost $0). Unchanged files are skipped by comparing the stored chunk text.--supplement(index repo markdown the harvester skipped) found +0 files in 17 of 19 repos; ASVS (+265 chunks) and WSTG (+160) were the exceptions.Next: worker-count/speed measurement, then the cost PR on top of the stack.