Repository navigation
feat(comprehension): ingestion-time LLM digests so ask/chat answer structure + combination questions - #1158
Open
northdpole wants to merge 52 commits into
Open
feat(comprehension): ingestion-time LLM digests so ask/chat answer structure + combination questions#1158northdpole wants to merge 52 commits into
northdpole wants to merge 52 commits into
Conversation
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review: 1
Please check the settings in the CodeRabbit UI or the
 |
northdpole
force-pushed
the
fix/ask-embedding-retrieval
branch
from
October 20, 2026 08:20
21890d0 to
1129c1f
Compare
northdpole
force-pushed
the
feat/ingest-comprehension
branch
from
October 20, 2026 08:20
016ec56 to
2972c67
Compare
northdpole
changed the base branch from
fix/ask-embedding-retrieval
to
claude/fix-agent-index-80b1a2
October 20, 2026 08:21
northdpole
force-pushed
the
claude/fix-agent-index-80b1a2
branch
from
October 20, 2026 08:29
7dd9cd2 to
f5ba231
Compare
northdpole
force-pushed
the
feat/ingest-comprehension
branch
from
October 20, 2026 08:29
2972c67 to
8306880
Compare
northdpole
force-pushed
the
claude/fix-agent-index-80b1a2
branch
from
October 20, 2026 08:48
f5ba231 to
d14089a
Compare
northdpole
force-pushed
the
feat/ingest-comprehension
branch
from
October 20, 2026 08:48
8306880 to
5860a9d
Compare
northdpole
force-pushed
the
claude/fix-agent-index-80b1a2
branch
from
October 20, 2026 09:21
d14089a to
cb51c1f
Compare
northdpole
force-pushed
the
feat/ingest-comprehension
branch
from
October 20, 2026 09:21
c306180 to
6b2c573
Compare
9 tasks
northdpole
force-pushed
the
feat/ingest-comprehension
branch
from
October 21, 2026 21:09
b6ded96 to
0a34ae0
Compare
northdpole
force-pushed
the
claude/fix-agent-index-80b1a2
branch
from
October 21, 2026 23:09
cb51c1f to
fe62660
Compare
northdpole
force-pushed
the
feat/ingest-comprehension
branch
2 times, most recently
from
October 22, 2026 07:59
c4b23d4 to
594e1d0
Compare
northdpole
force-pushed
the
claude/fix-agent-index-80b1a2
branch
from
October 22, 2026 07:59
fe62660 to
044e610
Compare
northdpole
marked this pull request as ready for review
October 22, 2026 11:17
northdpole
requested review from
Pa04rth,
paoga87 and
robvanderveer
as code owners
October 22, 2026 11:17
…es at ingestion Replace hand-curated structural facts with LLM comprehension of harvested standards. Per-document claims must carry a verbatim quote found in the source; a per-standard overview must cite existing documents. Results are embedded as evidence docs in the existing agent channel; at query time the router synthesizes an answer from the top comprehension hits and strips uncited output (UNKNOWN falls through). - application/utils/comprehension: sources, digest, store, synthesize, answer - scripts/comprehend_standard.py: run for one harvested repo - router hook behind OWASP_COMPREHENSION_ENABLED (default on) - design note: docs/oie/comprehension-design.md Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, prefer latest version, overview-first ordering Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…hesis is UNKNOWN; sharper synthesis rules Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…row abstain to non-CRE questions Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…artially instead of abstaining Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…dings (and merge the two heads) Vector search was a full scan; the index is used when OWASP_PGVECTOR_HALFVEC=1. No-op on SQLite and without the vector extension. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…roject questions (L4) Flagged off by default. Falls back to normal retrieval when the question names no single project, the project is too big, or the answer is unknown. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… winning flag set Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…_CX_LONGCTX_PROJECTS) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nd OWASP_CX_DECOMPOSE (multi-part questions) Both off by default; measured before being turned on. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ed lower) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…inking budget Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The async tool handler called the blocking ask directly on the event loop with a single worker, so concurrent users were answered one after another (3 parallel users saw ~3x the single-user latency). Each call now runs in a thread with its own Flask app context. OPENCRE_MCP_ASK_CONCURRENCY caps threads (default 8). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…concurrency cap Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first question after a restart took ~30 s (imports, embedding client, database connection). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…stion names Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…imilarity when none is named Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… skipped Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…w fit) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…comprehend_standard Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…incremental Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s 90 s on a 250-chunk repo) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…art and cross-standard questions Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… get, overview of indexed sources) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…slices, so a provider outage no longer discards a repo's enrichment Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…estions (off by default) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…wers per standard (off by default) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s and issue titles as retrievable documents Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… per OWASP YouTube talk) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…skips the record Nest cannot serialise Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
northdpole
force-pushed
the
claude/fix-agent-index-80b1a2
branch
from
October 25, 2026 10:35
044e610 to
f6c15e0
Compare
northdpole
force-pushed
the
feat/ingest-comprehension
branch
from
October 25, 2026 10:35
5406406 to
4c92743
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1155.
Why
#1155 answered structural questions (e.g. SAMM levels) from hand-written facts, which cannot scale and was partly wrong. This derives that knowledge at ingestion instead.
How
application/utils/comprehension/: reads whole harvested documents, extracts claims that must include a verbatim quote found in the source (unverified claims dropped), builds a per-standard overview that must cite existing documents, and stores everything as embedded evidence docs in the existing agent channel.UNKNOWNfalls through to existing paths. Flag:OWASP_COMPREHENSION_ENABLED(default on).scripts/comprehend_standard.py --repo OWASP/SAMMruns it per harvested repo.docs/oie/comprehension-design.md.Verified on staging (homeserver)
SAMM/ASVS/MASVS/AI guide digested; e.g. "how many levels in owasp samm" returns three maturity levels per practice (also surfaces a v1.5 vs v2 conflict, to fix by preferring the latest version), with GitHub citations.
Not done yet
Generated source-grounded eval (Gemini questions, Claude judge), held-out standard test, version preference, removal of hand-written
node_facts, people/talks layer, scheduling into the pipeline.🤖 Generated with Claude Code