Admin showcase panel for signed-in users

.cursor/skills/brevity-bot/SKILL.md

@@ -0,0 +1,56 @@
+---
+name: brevity-bot
+description: >-
+  Review a diff like a long-tenured greybeard coworker: shorten AI blurbs,
+  simplify code, keep correctness/security/Bugbot fixes. Use after Bugbot and
+  Security reviews, or when asked for /review-brevity.
+---
+
+# Brevity bot
+
+You are the ponytailed, greybearded, Linux-using god of coworker who has been
+with the company since the beginning. You can shorten or simplify any codebase.
+Your job is to dramatically increase readability and quality and shorten AI
+blurbs while preserving correctness, quality, and security.
+
+You simplify code **without dropping the fixes** from Bugbot or Security
+review. If a check, error path, redaction, auth gate, or test exists because
+those reviews required it, keep it. Prefer a shorter form of the same fix.
+
+## Diff contract (same as Bugbot)
+
+The parent gives:
+
+```text
+Full Repository Path: <absolute path>
+Diff: <branch changes | uncommitted changes>
+```
+
+Review that diff only. Do not invent drive-by refactors outside it.
+
+## What to cut
+
+- Restated comments, essay docstrings, "as an AI" padding, changelog comments
+- Needless wrappers, duplicate branches, unused helpers, speculative abstraction
+- Tests that only restate the same assertion five ways
+
+## What to keep
+
+- Auth, allowlists, redaction, abort codes, CSRF/login gates
+- Bugbot/Security fixes (narrower code is fine; weaker checks are not)
+- Names and structure that match this repo (`application/tests/`, Makefile)
+
+## Output
+
+Either:
+
+1. **Patches** — file path, short why, and the smaller replacement, or
+2. **`NO_COMMENTS`** — nothing left to shorten without harming clarity, tests,
+   or security.
+
+End with exactly one of:
+
+- `VERDICT: NO_COMMENTS`
+- `VERDICT: COMMENTS` (and the patch list)
+
+Do not praise the diff. Do not recap the plan.

.cursor/skills/brevity-bot/examples.md

@@ -0,0 +1,40 @@
+# Brevity bot fixtures
+
+## AI blurb → short
+
+Before:
+
+```python
+def is_enabled() -> bool:
+    """Return True if the feature flag is enabled.
+
+    This function checks the environment variable and compares it against
+    a set of truthy values so that operators can enable the feature.
+    """
+    value = os.getenv("FLAG", "")
+    normalized = value.strip().lower()
+    if normalized in {"1", "true", "yes"}:
+        return True
+    return False
+```
+
+After:
+
+```python
+def is_enabled() -> bool:
+    return os.getenv("FLAG", "").strip().lower() in {"1", "true", "yes"}
+```
+
+## Do not drop a security fix
+
+Security required redacting secrets. Illegal brevity (drops the fix):
+
+```python
+return {"GEMINI_API_KEY": os.environ.get("GEMINI_API_KEY")}
+```
+
+Legal brevity (keeps redaction):
+
+```python
+return {"GEMINI_API_KEY": "***"} if os.environ.get("GEMINI_API_KEY") else {}
+```

.cursor/skills/review-brevity/SKILL.md

@@ -0,0 +1,24 @@
+---
+name: review-brevity
+description: Launch Brevity bot on the current diff after Bugbot and Security.
+---
+
+# Review Brevity
+
+When asked to run `/review-brevity` or Brevity bot:
+
+Launch **three** independent `generalPurpose` subagents (`run_in_background: false`)
+with the contents of `.cursor/skills/brevity-bot/SKILL.md` as instructions.
+
+Prompt each with:
+
+```text
+Full Repository Path: <absolute repository path>
+Diff: branch changes
+```
+
+Default to `branch changes`. Use `uncommitted changes` only if asked.
+
+Collect the three `VERDICT:` lines. Majority `NO_COMMENTS` (2 of 3) is clean.
+If majority is `COMMENTS`, apply agreed simplifications that do **not** revert
+Bugbot or Security fixes, then re-run.

.env.example

@@ -5,6 +5,10 @@
 # Never commit your real `.env` file to version control.
 # ------------------------------------------------------------------------------
 # Database
+# Local Docker (make docker-postgres / make admin-local):
+#   postgresql://cre:password@127.0.0.1:5432/cre
+# `make admin-local` starts that container when the URL is unreachable and
+# never falls back to SQLite for the admin showcase.
 
 DEV_DATABASE_URL=postgresql://user:password@localhost:5432/opencre
 
@@ -42,14 +46,17 @@ CRE_ENABLE_MYOPENCRE=false
 CRE_ENABLE_LOGIN=false
 
 # Experimental OIE OWASP metadata agent (chat + MCP only — never CRE graph UI).
-# Off by default. When enabled, community/meta questions are answered from
-# the local owasp_agent index instead of CRE RAG. Concepts/tags stay in
-# OWASP_AGENT_DB and are reachable only from /completion and MCP owasp_meta_* tools.
+# Off by default. Uses the main app Postgres (DEV_DATABASE_URL / DATABASE_URL)
+# for the agent index + pgvector. Reachable from /completion and MCP owasp_meta_* tools.
 OWASP_AGENT_ENABLED=false
+
+# Module C.1 graph filing (Automatically linked to). Off by default — leave Module C
+# decisions in decision_queue for review; set true to auto-add high-confidence links.
+OIE_GRAPH_FILING_ENABLED=false
+
 # Nest REST API key (https://nest.owasp.org — create via GraphQL while logged in)
 NEST_API_KEY=
 NEST_API_BASE=https://nest.owasp.org/api/v0
-OWASP_AGENT_DB=tmp/owasp_agent.sqlite
 # Set to 1 to crawl all www-chapter / www-project repos during sync (slow).
 OWASP_AGENT_CRAWL_REPOS=0
 

.gitignore

@@ -49,6 +49,8 @@ v/
 .cursor/*
 !.cursor/rules/
 !.cursor/rules/**
+!.cursor/skills/
+!.cursor/skills/**
 .claude/
 
 ### Frontend 
@@ -67,6 +69,7 @@ standards_cache.sqlite
 ### Docs
 *.md
 !AGENTS.md
+!.cursor/skills/**/*.md
 !docs/faq.md
 !docs/api/mcp.md
 !docs/Mid_eval_blog_gsoc2026/module_B_mideval_blog.md
@@ -94,6 +97,8 @@ neo4j/
 .mypy_cache
 tmp/
 
+import_telemetry.json
+
 ### CREs dir
 cres/*
 ### Local project management tooling

Makefile

@@ -1,6 +1,6 @@
 .ONESHELL:
 
-.PHONY: run test covers install-deps dev docker lint frontend clean all e2e e2e-db owasp-agent-sync owasp-agent-merge-concepts owasp-agent-synth-eval
+.PHONY: run test covers install-deps dev dev-flask docker lint frontend clean all e2e e2e-db admin-local owasp-agent-sync owasp-agent-merge-concepts owasp-agent-synth-eval
 
 prod-run:
 	gunicorn cre:app --log-file=-
@@ -74,11 +74,19 @@ start-worker:
 	. ./venv/bin/activate && FLASK_APP=`pwd`/cre.py python cre.py --start_worker
 
 upstream-sync:
-	. ./venv/bin/activate && python cre.py --upstream_sync
+	. ./venv/bin/activate &&\
+	python cre.py --upstream_sync --cache_file "$(or $(DEV_DATABASE_URL),$(CRE_CACHE_FILE),$(CURDIR)/standards_cache.sqlite)"
 
 PORT?=5000
 
-dev-flask:
+# Local app against Docker Postgres: migrate, upstream-sync if cre/node empty, Flask.
+# Same entrypoint as admin showcase / OIE local (no SQLite). macOS AirPlay often
+# holds :5000 — use PORT=5001 make dev if bind fails.
+dev dev-flask admin-local:
+	bash ./scripts/run_admin_local.sh --port $(PORT) $(ADMIN_LOCAL_ARGS)
+
+# Bare Flask without Postgres bootstrap (legacy / CI helpers). Prefer `make dev`.
+dev-flask-sqlite:
 	. ./venv/bin/activate && INSECURE_REQUESTS=1 FLASK_APP=`pwd`/cre.py  FLASK_CONFIG=development flask run --port $(PORT)
 
 dev-flask-docker:
@@ -136,7 +144,14 @@ install-deps-typescript:
 install-deps: install-deps-python install-deps-typescript
 
 install-python:
-	virtualenv -p python3 venv
+	@if [ ! -d "./venv" ]; then \
+		if command -v virtualenv >/dev/null 2>&1; then \
+			virtualenv -p python3 venv; \
+		else \
+			echo "virtualenv not on PATH — using python3 -m venv"; \
+			python3 -m venv venv; \
+		fi; \
+	fi
 	. ./venv/bin/activate &&\
 	make install-deps-python &&\
 	playwright install  # Python embeddings/scraping (prompt_client); NOT frontend e2e — keep when migrating to Cypress
@@ -148,7 +163,9 @@ install-python:
 # `make install`. Rebuild with `make frontend` when you change the UI.
 install-typescript: install-deps-typescript
 
-install: install-typescript install-python migrate-upgrade
+# Deps + Docker Postgres schema + CRE upstream graph when empty (see run_admin_local.sh).
+install: install-typescript install-python
+	bash ./scripts/run_admin_local.sh --migrate-only $(ADMIN_LOCAL_ARGS)
 
 docker-dev:
 	docker build -f Dockerfile-dev -t opencre-dev:$(shell git rev-parse HEAD) .
@@ -273,6 +290,21 @@ monitor-chatbot-health-prod:
 		--base-url "https://opencre.org" \
 		--output-json "$(CURDIR)/tmp/prod-chatbot-health.json"
 
+# Scheduled OIE jobs (Postgres only). Cron `make oie-tick` every 10 minutes: each
+# job runs once per slot (owasp every 10 min, cre_expansion daily) and is a no-op
+# otherwise. Overrides: OIE_JOB=owasp|cre_expansion|all, OIE_FORCE=1, OIE_DRY_RUN=1.
+oie-tick:
+	@[ -d "./.venv" ] && . ./.venv/bin/activate || ([ -d "./venv" ] && . ./venv/bin/activate); \
+	export CRE_CACHE_FILE="$${CRE_CACHE_FILE:-postgresql://cre:password@127.0.0.1:5432/cre}"; \
+	python cre.py --cache_file "$$CRE_CACHE_FILE" --run_scheduled "$${OIE_JOB:-all}" \
+		$${OIE_FORCE:+--scheduled_force} $${OIE_DRY_RUN:+--scheduled_dry_run}
+
+monitor-oie-health:
+	@[ -d "./.venv" ] && . ./.venv/bin/activate || ([ -d "./venv" ] && . ./venv/bin/activate); \
+	export CRE_CACHE_FILE="$${CRE_CACHE_FILE:-postgresql://cre:password@127.0.0.1:5432/cre}"; \
+	export PYTHONPATH="$(CURDIR)"; \
+	python scripts/monitor_oie_health.py --output-json "$(CURDIR)/tmp/oie-health.json"
+
 verify-ga-parity-local:
 	@[ -d "./.venv" ] && . ./.venv/bin/activate || ([ -d "./venv" ] && . ./venv/bin/activate); \
 	export CRE_CACHE_FILE="$${CRE_CACHE_FILE:-postgresql://cre:password@127.0.0.1:5432/cre}"; \

README.md

@@ -49,44 +49,33 @@ If you want to develop on OpenCRE or docker is not available in your environment
 ### Command Line
 
 To run outside of Docker you need to install OpenCRE.
-To install this application you need Python 3.11.9, Node.js, Yarn Classic (the repository uses a Yarn v1 lockfile), and `virtualenv`. The Python version is pinned in `.python-version`; the repository does not currently pin a Node.js version.
+To install this application you need Python 3.11.9, Node.js, Yarn Classic (the repository uses a Yarn v1 lockfile), and Docker (local Postgres). `virtualenv` is optional — `make install-python` falls back to `python3 -m venv`. The Python version is pinned in `.python-version`; the repository does not currently pin a Node.js version.
 
 Clone the repository:
 
 ```bash
 git clone https://github.com/OWASP/OpenCRE
 ```
 
-`make install` creates and uses the project virtual environment. Do not create a second environment with `python3 -m venv`; install the prerequisites above, then let the Makefile manage `venv`.
-
-Install dependencies:
+`make install` creates the project `venv` (if missing), installs deps, starts Docker Postgres when needed, migrates, and pulls the CRE graph from opencre.org when the local DB is empty.
 
 ```bash
 make install
+make dev                 # same as make dev-flask / make admin-local
 ```
 
-`make install` creates the local schema as part of installation. Download the latest CRE graph from upstream with:
-
-```bash
-make upstream-sync
-```
+Then open http://127.0.0.1:5000/admin and http://127.0.0.1:5000/explorer. If port 5000 is busy (common on macOS with AirPlay Receiver), use `PORT=5001 make dev`. To reuse an existing DB without re-pulling the CRE graph: `SKIP_UPSTREAM_SYNC=1 PORT=5001 make dev`.
 
 For contributors, this is the supported local data bootstrap path. You do
 **not** need access to the internal OpenCRE Google Sheet to work on the
-project.
+project. From **Admin → Config**, use **Run golden-set harvest** and **Sync OWASP agent** (progress under Pipeline), or `make owasp-agent-sync` from the CLI.
 
 You can precompute local gap-analysis cache after imports with:
 
 ```bash
 make backfill-gap-analysis
 ```
 
-To run CRE locally then you can do:
-
-```bash
-make dev-flask
-```
-
 To run the CLI application, you can run:
 
 ```bash

application/cmd/cre_main.py

@@ -880,6 +880,39 @@ def backfill_gap_analysis_only(
     logger.info("GA backfill complete: final_missing_pairs=%s", final_missing)
 
 
+def run_scheduled_cli(args: argparse.Namespace, collection: db.Node_collection) -> int:
+    """``--run_scheduled``: run the selected OIE job(s); return a process exit code.
+
+    Postgres only -- the advisory-lock lease and the run table assume it, and the
+    default ``standards_cache.sqlite`` must not silently become the scheduler's
+    database. Exit 1 when a job errored, 2 for a non-Postgres database.
+    """
+    import json
+
+    from application.utils.oie_scheduler.runner import run_due_jobs
+
+    dialect = collection.session.get_bind().dialect.name
+    if dialect != "postgresql":
+        logger.error(
+            "--run_scheduled is Postgres-only (got %s); point --cache_file at "
+            "a postgresql:// URL (make docker-postgres for a local one)",
+            dialect,
+        )
+        return 2
+    selected = args.run_scheduled
+    force = bool(getattr(args, "scheduled_force", False))
+    outcomes = run_due_jobs(
+        collection,
+        cache_file=args.cache_file,
+        jobs=None if selected == "all" else [selected],
+        force=force,
+        dry_run=bool(getattr(args, "scheduled_dry_run", False)),
+        trigger="manual" if force else "scheduled",
+    )
+    print(json.dumps([o.to_dict() for o in outcomes], indent=2))
+    return 1 if any(o.failed for o in outcomes) else 0
+
+
 def run(args: argparse.Namespace) -> None:  # pragma: no cover
     script_path = os.path.dirname(os.path.realpath(__file__))
     os.path.join(script_path, "../cres")
@@ -892,6 +925,14 @@ def run(args: argparse.Namespace) -> None:  # pragma: no cover
         logger.info("Exported %s rows to %s", rows, csv_out)
         return
 
+    if getattr(args, "run_scheduled", ""):
+        import sys
+
+        code = run_scheduled_cli(args, db_connect(args.cache_file))
+        if code:
+            sys.exit(code)
+        return
+
     if getattr(args, "run_harvester", False):
         import sys
 
@@ -905,6 +946,12 @@ def run(args: argparse.Namespace) -> None:  # pragma: no cover
             args.run_id.strip(),
             repos_yaml=repos_yaml,
             dry_run=getattr(args, "harvester_dry_run", False),
+            kinds=[
+                k.strip()
+                for k in (getattr(args, "harvester_kinds", "") or "").split(",")
+                if k.strip()
+            ]
+            or None,
         )
         print(summary.to_json())
         if summary.status == "degraded":