Repository navigation
chore(oie): NIST 800-53 and remaining B2 sources/gold - #1118
Merged
Merged
Conversation
northdpole
requested review from
Pa04rth,
paoga87 and
robvanderveer
as code owners
September 29, 2026 09:52
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review: 1
Please check the settings in the CodeRabbit UI or the
 |
This was referenced Sep 29, 2026
northdpole
force-pushed
the
stack/oie-p4-asvs-fixtures
branch
from
October 3, 2026 18:12
c66eb52 to
4c59129
Compare
northdpole
force-pushed
the
stack/oie-p5-nist-other-fixtures
branch
from
October 3, 2026 18:12
52d8baa to
8723475
Compare
northdpole
force-pushed
the
stack/oie-p4-asvs-fixtures
branch
from
October 5, 2026 09:25
4c59129 to
0345e20
Compare
northdpole
force-pushed
the
stack/oie-p5-nist-other-fixtures
branch
from
October 5, 2026 09:26
8723475 to
9faed5c
Compare
northdpole
force-pushed
the
stack/oie-p4-asvs-fixtures
branch
from
October 5, 2026 10:11
0345e20 to
2787cea
Compare
northdpole
force-pushed
the
stack/oie-p5-nist-other-fixtures
branch
from
October 5, 2026 10:11
9faed5c to
28bcf6f
Compare
northdpole
force-pushed
the
stack/oie-p4-asvs-fixtures
branch
from
October 7, 2026 18:37
2787cea to
f38a352
Compare
northdpole
force-pushed
the
stack/oie-p5-nist-other-fixtures
branch
from
October 7, 2026 18:37
28bcf6f to
8845bab
Compare
Dual-index retrieval, CRE summaries, prior cage, hub seed, edition remap, and related librarian defaults. Rewrites #1088 as the reviewable base; later stacked PRs add harvester, B2 eval, fixtures, mappings, and full-pipeline docs.
- Refuse plain-text fetch redirects that leave GitHub raw hosts (SSRF). - Re-check expected embedding dimension on query-cache hits. - Fill phrase-only oie blocks in apply_oie_fill_if_missing without losing phrases. - Expose last_preferred_cre_ids on StandardHopRetriever and honour its allowlist. - Commit each metadata enrichment so a later failure cannot roll back earlier work. - Load .env from the repo root in the enrichment scripts. - Parse hostnames for CRE hub sync SSL and loopback checks; batch SQLite IN lookups; stop echoing database credentials.
Optional requirement extract (off/auto/on) plus GitHub tarball/URL ingest wiring for Module B → C. Stacked on #1088 librarian base.
Harness runners, hop-distance scoring, and gold loaders without bulky source fixtures (those land in follow-up stacked PRs).
Wire LlmSafetyGuard (LiteLLM) as the live factory default so runs evaluate safety instead of NullSafetyGuard. Apply the guard on every decision path, prefer safety reason codes over empty shortlists, and cover a synthetic adversarial corpus in unit tests.
- Do not fall back to main/master when the user named a branch. - Download tarballs with a timeout and a size cap; cap per-member and total markdown bytes read from archives. - Validate owner, repo and branch and URL-encode them for codeload. - Delete prior ingest rows only after download and chunking succeed, and never rows from other runs. - Pass the model override through a local config instead of os.environ.
- Redact database passwords in orchestrator, B2 and setup_oie output; write tmp/oie.env with mode 600. - Keep the existing --db for --skip-a and a user-supplied --repos-yaml. - Create the artifact dir before writing orchestrator reports. - Clear per-model knowledge rows between Qwen and Gemini classify runs. - Key the switch-grid embed cache by model, honour --force with stale-lock recovery, write explicit zero flags to stack_winners.yaml, and tolerate LibrarianConfig fields that are env-only. - Match B1 sheet-grain gold exactly instead of by substring.
Large fixture slice for ASVS eval arms only — split from NIST/other sources so reviewers can treat this as mechanical data review.
NIST OSCAL sample plus Top10/API/K8s/LLM/AISVS/agentic fixture sources and gold JSON. Stacked after the ASVS-only fixture PR.
northdpole
force-pushed
the
stack/oie-p4-asvs-fixtures
branch
from
October 14, 2026 08:48
f38a352 to
955528d
Compare
northdpole
force-pushed
the
stack/oie-p5-nist-other-fixtures
branch
from
October 14, 2026 08:48
8845bab to
86d320a
Compare
Collaborator
Author
|
Review notes for the NIST and other families PR (no blockers):
|
- Remove the unwired context_enrich experiment (module and test); it is preserved on the staging-experiments branch. - Drop oie_enrich_neighbor_phrases.py and oie_enrich_node_metadata_llm.py (no references anywhere in the stack). - Revert golden_dataset.json and build_golden_dataset.py to main. The 190 gold:asvs5 rows depend on the ASVS 5 fixture, which lands in the ASVS fixtures PR; they move there. The CRE_GAP schema enum stays. - Delete the duplicated _env_bool/_env_csv block in librarian config_loader.
…nto trim/1115 * origin/wip/oie-pipeline-eval-harness: chore(oie): trim #1088 for the staging rollout
- Fix: a re-created shallow clone (or an upstream force-push) no longer contains the persisted checkpoint commit, which made every harvest of that repository fail because the checkpoint only advances on success. Add ChangeDetector.commit_exists and resolve_base_commit; a missing checkpoint commit now logs a warning and harvests all tracked files. Real-git tests cover a shallow clone. - Declare the Docling and LlamaIndex dependencies that docling_chunker and repos.yaml already require (requirements-dev.txt). - Move the GitHub URL ingest tool (github_url_ingest, its test, the --ingest_* flags and scripts/ingest_github.sh) out of the stack; it is kept on the feat/github-url-ingest branch for a separate PR. - Remove the unused _paragraph_end helper.
- Add assert_local_db_url / assert_local_clone_url to application/utils/db_url.py (moved from oie_generate_cre_summaries.py, now also refusing an empty URL), with tests. The summaries script keeps its existing behaviour. - run_b2_pr_mappings.py empties harvest_input, knowledge_queue and decision_queue; it now refuses anything but a local clone (never a remote host, Heroku or the working `cre` database). Its default DEV_DATABASE_URL becomes cre_prodclone, matching the README. - run_exhaustive_switch_grid.py reset_queues refuses a non-local database. - clone_grid_worker_dbs.sh terminates sessions and can DROP DATABASE; it now refuses a non-local PGHOST and any DB_PREFIX not starting with cre_grid_w. - Drop run_phase.py, run_qwen_outside_sandbox.sh and run_official_orchestrator_local.py (no references, or a TEMP helper); the setup_oie.sh hint now points at `make oie-pipeline`.
- Restore the 190 gold:asvs5 positives in golden_dataset.json and the ASVS 5 additions to build_golden_dataset.py. They depend on owasp_asvs_5_0_provisional.json, which this PR adds, so they moved here from the retrieval base PR. - Drop run_phase_a_asvs_sheets.py: unreferenced, targets ASVS 4 and CheatSheets, and clears the queue tables in whatever database is configured.
- Drop scripts/oie_owasp_eval/fixtures/reports/*.json: generated accuracy output that no code reads (module_c_report.json also embedded an absolute local path). Numbers are kept in the project's session records. - Drop the five b2_gold overlays (owasp_aisvs_1_0, owasp_api_top10_2023, owasp_kubernetes_top10_2022, owasp_kubernetes_top10_2025, owasp_top10_2025). They are not byte-identical to the canonical mapping fixtures but are regenerable hub-expanded overlays (rebuild_b2_gold_from_hub.py), and load_gold() reads only the agentic stub from b2_gold. All seven harnesses still load. owasp_agentic_ai_stub.json and owasp_llm_top10_2025.json stay.
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1119: chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout # Conflicts: # requirements-dev.txt
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1120: trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout # Conflicts: # application/utils/librarian/factory.py # docs/oie/full-pipeline.md # scripts/oie_owasp_eval/run_fullpipe_detached.sh
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1125: trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1127: trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout # Conflicts: # cre.py # scripts/owasp_agent_adversarial_asker.py # scripts/owasp_agent_adversarial_batch_run.py
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1137: trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1136: trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos # Conflicts: # application/utils/oie_scheduler/jobs.py # scripts/oie_owasp_eval/run_full_pipeline.py
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1142: trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos # Conflicts: # scripts/owasp_agent_adversarial_asker.py # scripts/owasp_agent_adversarial_batch_run.py
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1155: trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1159: trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1158: trim(#1158): strip_injection no longer truncates legitimate questions; correct default-off claims trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos # Conflicts: # application/mcp/ask.py # application/web/web_main.py
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1162: (27 commits) docs(env): document CHAT_BLOCK_UNVERIFIED (default on, changes public chat answers) trim(#1162): drop prune_noise_chunks (unguarded DELETE on embeddings), validate authority weights before use, document ANSWER_LOG_*, no host paths in sync_check_staging trim(#1158): strip_injection no longer truncates legitimate questions; correct default-off claims trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout fix(auth): harden web sign-in found by adversarial review fix(auth): /auth/user needs any role, not admin fix(auth): carry the PKCE code_verifier from login to callback fix(auth): build a fresh OAuth flow per login instead of a shared singleton ...
northdpole
changed the base branch from
stack/oie-p4-asvs-fixtures
to
staging
November 1, 2026 22:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked fixture PR — NIST 800-53 sample OSCAL/sources plus Top10/API/K8s/LLM/AISVS/agentic gold and reports. ASVS landed in the previous PR.
Depends on
Test plan
Made with Cursor
Stack (merge bottom → top)
wip/oie-pipeline-eval-harness→main)stack/oie-p2-harvester→ feat(librarian): Module C retrieval upgrades (stack base) #1088)stack/oie-p3-b2-harness→ feat(harvester): requirement extract and GitHub URL ingest #1115)stack/oie-p4-asvs-fixtures→ feat(oie): B2 eval harness and CRE neighborhood scorer #1116)stack/oie-p5-nist-other-fixtures→ chore(oie): ASVS 5.0 provisional B2 sources and gold #1117)stack/oie-p6-mapping-promotions→ chore(oie): NIST 800-53 and remaining B2 sources/gold #1118)stack/oie-p7-full-pipeline-docs→ feat(mappings): promote Related-orphan remaps and re-point the LLM fixture to AI-hub CREs (#1124) #1119)