Skip to content

2.4.0: verified reviews, specialist /audit, supply-chain scanning, closed trigger holes - #34

Merged
iamrraj merged 8 commits into
mainfrom
feat/forge-review-hardening
Nov 1, 2026
Merged

iamrraj merged 8 commits into
mainfrom
feat/forge-review-hardening

Conversation

@iamrraj

@iamrraj iamrraj commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Why

Forge posted whatever its first pass suspected, let anyone who could comment trigger it, read review rules from the pull request itself, and had no dependency or supply-chain checks. This brings it to the shape of the managed reviewers (find, verify, post) while closing the security holes found along the way.

Security

  • Only collaborators with write access can start a run from a comment, @mention or issue.
  • Fork PRs are read by number (refs/pull/N/head); Forge never pushes a follow-up or CI fix to a fork.
  • REVIEW.md, FORGE.md, AGENTS.md, security guidance and skills come from the base branch.
  • /secrets on a PR scans the PR's code (it scanned the default branch and passed the PR's check).
  • Dependabot alerts on a review are limited to touched manifests and never block.
  • Dependencies: 0 critical / 0 high (simple-git 4, openai 7, zod 4, patched transitive pins). 27 moderate remain inside probot 13.

Review

  • Every finding is re-checked by a second agent before posting; review_depth (light / standard / deep) now works, within one spend cap.
  • A push re-reviews only the files it changed; if none of the PR's files moved, the previous review stands and nothing is spent.

Scanning and audit

  • Dependency scanner: changed lockfiles vs OSV.dev (CVEs, malicious releases, new install scripts, off-registry sources, missing integrity).
  • /audit: up to eight specialist slices adapted from pentest-skills, a verifier, a coverage map; semgrep, gitleaks, trivy, osv-scanner, bandit and checkov bundled in the Action image (pinned, checksum-verified, no token in their environment). audit_skills: owner/repo@<sha> overrides the slices.

Site and dashboard

  • Animated landing page, comparison with Claude Code Review and Cursor Bugbot (public docs only), updated docs, builder split and given review/security options, one ember theme.
  • Dashboard: finding details (explanation and fix), pagination, top rules, distinct chart colours.

Behaviour changes to note

  • Community-opened issues are no longer auto-analysed; triage-only users cannot start agent-fix.
  • Standard reviews cost more (verification is on by default); review_depth: light restores the old behaviour.
  • The dependency scan sends package names and versions to api.osv.dev.

Tests

  • 867 β†’ 1192 tests; typecheck clean (agent and site); site builds; scanner eval 100% with no drop.
  • External scanners run for real on a sample repository.

Not covered

  • Not yet exercised against live GitHub: the write-access check, refs/pull checkout, the compare call for incremental review, loading private audit_skills.
  • The Docker image was not built locally (no Docker); two pinned binaries were downloaded and their SHA-256 verified.
  • The OpenAI adapter on SDK v7 was not run against the live API.
  • The model-review eval (--model-review) was built but not run (no credentials).

…nd closed trigger holes

Security
- Only collaborators with write access can start a run from a comment,
  @mention or issue; everyone else is ignored before any model call.
- Fork pull requests are checked out by number (refs/pull/N/head) and Forge
  never pushes a follow-up or a CI fix to a fork.
- REVIEW.md, FORGE.md, AGENTS.md, security guidance and skills are read from
  the base branch, so a pull request cannot rewrite its own review rules.
- /secrets on a pull request scans the pull request's code instead of the
  default branch.
- Dependabot alerts on a review are limited to manifests the PR touches and
  never block it.

Review
- Every finding is re-checked by a second agent before it is posted.
- review_depth now works: light (one pass), standard (find, then verify),
  deep (three focused reviewers, then verify). The spend cap is shared.

Scanning
- New dependency scanner: changed lockfiles checked against OSV.dev for CVEs
  and malicious releases, new install scripts, off-registry sources and
  missing integrity hashes (dep_scan).

Audit
- /audit runs up to eight specialist slices adapted from pentest-skills, then
  one verifier with known false-positive shapes, and reports a coverage map.
- semgrep, gitleaks, trivy, osv-scanner, bandit and checkov run when
  installed, with no token or model key in their environment; the Action
  image bundles them pinned and checksum-verified (external_scanners).
- audit_skills: owner/repo@<sha> loads slices from another repository.

Evals
- forge eval --model-review scores the model review against planted bugs
  and safe look-alikes.
- Hero: a live, looping Forge review (scan, find, verify, post, fixed) over an
  animated grid and glow; a smaller three-line headline.
- New sections: an agent-flow diagram with step cards, a hub of what Forge
  reads, checks and posts (with provider logos), a comparison with Claude Code
  Review and Cursor Bugbot drawn only from their public docs, a replayed
  supply-chain check, and what makes Forge safe on public repositories.
- "What it does" and "Skills" become interactive players with real run logs
  and each skill's enforced tool allowlist.
- Landing split into one component per section under sections/landing.
- Docs: the audit section rewritten for the specialist team and scanners; a
  review depth and verification tab; new configuration keys.
- Tests hold the page's claims to the code: scanner list, review depths,
  skill names and tool access.
…lines

- New "Review & security" step: review depth, what fails the scan check,
  dependency scan, bundled scanners on /audit, and audit skills pinned to a
  commit (validated as owner/repo@<40-character SHA).
- The generated workflow emits each input only when it differs from the
  Action's default; a test checks every input exists in action.yml.
- /how-to added to the skill list.
- WorkflowBuilder.tsx (711 lines) split into one component per form group
  under workflow/builder/; generateWorkflow.ts split into validate, agentYml
  and yaml helpers, keeping its exports.
- simple-git 3 -> 4 (critical; it clones the repositories Forge reviews).
- openai 4 -> 7 and zod 3 -> 4 (high). The tool schemas sent to the model
  are hand-written JSON, unchanged; zod only validates arguments.
- js-yaml 4.3.2, and overrides pinning patched proxy-addr, body-parser,
  nanoid, undici, source-map-js and vite within their current majors.
- web: npm audit fix, 0 vulnerabilities.

Remaining: 27 moderate advisories inside probot 13, fixed only by probot 14.
- Findings keep their explanation, suggested fix and start line (secrets
  redacted, capped), through a new additive migration.
- /api/findings/list pages with an offset and leaves the text out, so a page
  stays small; /api/findings/detail returns one finding in full.
- Findings page: 50 per page, a "Top rules" panel with readable CWE names,
  and a drawer with the explanation, the fix (copyable), links to the line on
  GitHub and to the run that found it.
- Charts: eight distinct series hues (no two orange), donut legends that wrap
  instead of truncating model names, wider bar labels.
- Site-wide: accent tokens move to ember in both themes; toggles, the active
  nav link, form labels and the scroll bar stop borrowing syntax colours;
  selected chips use dark text on the accent for contrast.
- Each review records the head commit it covered in a hidden marker on
  Forge's review comment. On the next push, Forge asks GitHub which files
  changed since and reviews the pull request's changes in those files only.
- When none of the PR's files moved (or the head has not moved), the previous
  review stands: no clone, no model call.
- A force-push, a missing marker, a compare error or a capped file list falls
  back to a full review. Opening, readiness, an invitation and /review are
  always full.
- Thread resolution is scoped to the files a partial review read, so it never
  closes a thread in a file it did not look at. Threads now carry their path.
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown

πŸ›‘οΈ ShipIT Forge β€” security scan

βœ… Nothing found. No committed credentials, no infrastructure or workflow misconfiguration, and nothing matching the source-code security rules, and no vulnerable or malicious dependency.

What was scanned
  • Committed credentials β€” Named shapes from thirty-one providers β€” GitHub, AWS, Anthropic, OpenAI, Google, Slack, Stripe, GitLab, SendGrid, Twilio, Shopify, HuggingFace, Linear, npm, JWT, PEM private keys, database URLs and more β€” and, because a provider list is never complete, a generic pass that reads the variable name and the randomness of the value rather than the vendor. Placeholders, names and empty PEM headers are not findings; a real key in documentation is one, at lower severity.
  • Infrastructure and workflows β€” Dockerfile and Compose (running as root, :latest bases, remote ADD, privileged services), Kubernetes (privileged pods, host-path mounts), Terraform (0.0.0.0/0, public buckets, encryption switched off), and GitHub Actions (mutable action pins, pull_request_target, untrusted event text reaching a shell).
  • Source code β€” Command injection (CWE-78), path traversal (CWE-22), clear-text logging of a credential (CWE-532), information exposure through an exception (CWE-209), open redirect (CWE-601), unsafe deserialization (CWE-502), binding every interface (CWE-1327), and workflows that never declare their permissions (CWE-732). Each needs something attacker-controlled and something dangerous done with it on the same line.
  • Dependencies β€” Lockfiles this change touches (npm, pnpm, Yarn, pip, Poetry, uv, Pipenv, Cargo): every added or upgraded package is checked against OSV.dev for known vulnerabilities (CWE-1395) and for releases listed as malicious (CWE-506); install scripts that are new or appear in an update (CWE-829); packages fetched over plain HTTP, from git or from arbitrary URLs (CWE-494); and entries with no integrity hash (CWE-353).

All of it is deterministic: the same commit gives the same answer, every time, with no model call and no token cost. The model review judges what is reachable and worth worrying about; this pass judges nothing and simply never gets bored.

The dependency check sends package names and versions, and nothing else, to api.osv.dev.

Scanned 154 files in the files this pull request changes. No model call β€” this scan is deterministic and costs nothing.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

πŸ‘€ ShipIT Forge β€” what this run looked at:

Pass Looking for
πŸ”‘ Secrets Committed credentials β€” provider shapes, entropy, file context
πŸ—οΈ Infrastructure Dockerfile, Compose, Kubernetes, Terraform, workflows
🧩 Code Injection, traversal, credentials in logs, unsafe deserialization
πŸ“¦ Dependencies Known vulnerabilities and malicious releases (OSV.dev), install scripts, package sources
πŸ€– Code review The model reads the diff β€” correctness, security, and what it would break

The scans are deterministic β€” no model call, no token cost β€” and run before the review, so their findings survived even if the model run stopped early. Findings arrive as inline comments, plus one security-scan comment grouped by rule.


πŸ” ShipIT Forge reviewed this PR β€” πŸ’¬ commented (no blocking issues)

No findings.

πŸ”Ž Reviewed by one reviewer; nothing to verify.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ShipIT Forge review

βœ… No blocking issues found. I reviewed the changed files and ran:

  • πŸ›‘οΈ Security checks β€” SSRF, injection (SQL/command/template), broken auth/authz, hardcoded secrets, unsafe deserialization, path traversal, weak crypto.
  • πŸ”§ Code review β€” correctness, error handling, missing tests, clarity.

Nothing to flag. This is a comment, not an approval β€” ShipIT Forge never approves PRs; a human reviewer should approve and merge.

@iamrraj
iamrraj merged commit a1ea154 into main Nov 1, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant