Skip to content

ci: unify Dependabot, Trivy and Docker Scout behind one gate, report and alert - #344

Closed
zeevmoney wants to merge 3 commits into
ci/continuous-image-vuln-scanningfrom
ci/scanner-unification
Closed

zeevmoney wants to merge 3 commits into
ci/continuous-image-vuln-scanningfrom
ci/scanner-unification

Conversation

@zeevmoney

Copy link
Copy Markdown
Member

Linear: PER-15358

Stacked on #338 (base: ci/continuous-image-vuln-scanning). Merge #338 first.

#338 made the CVE gate run on releases and added a scheduled re-scan of the published tags. This PR gives all three scanners — Dependabot, Trivy, Docker Scout — the same treatment: a PR gate with a readable report, alerts that reach Slack, and a release gate with teeth. It adopts the pattern from permitio/agent-security's dependency-review.yml.

What each scanner does now

PR gate Scheduled alert Release gate
Trivy blocks, in the sticky comment published tags, in the shared digest scans the shipped tag by digest, both platforms
Docker Scout blocks, same comment latest, same digest same job
Dependabot new dependency-review job alerts API poll, same digest —

PR gate and report

The Trivy step moves from exit-code: 1 to scan-to-JSON → format → a separate fail step, so findings exist as a file and can be reported rather than living only in a step log. The gate does not weaken: severity, trivyignores, ignore-unfixed and vuln-type carry over byte-identical, so critical + high > 0 is the old predicate evaluated two steps later — plus three report-unreadable failure modes the old exit code could not express.

One sticky comment per PR, marker <!-- pdp-image-scan -->, covering Trivy and Scout together. Scout's own write-comment goes to false on both steps, so a PR gets at most two bot comments instead of three. Guards for fork PRs and Dependabot PRs (read-only token there), with $GITHUB_STEP_SUMMARY and one ::error:: per finding as fallbacks.

One schedule, one message

scheduled-security-scan.yml runs all three scanners as parallel jobs on 17 6 */3 * * and a digest job merges them:

*Trivy* - 2 published tag(s): 1 clean, 1 not clean.
  • permitio/pdp-v2:latest -> SOURCE (3 findings: 1 CRITICAL, 2 HIGH)
  • permitio/pdp-v2:v0.9.15 -> CLEAN
*Docker Scout* - found 3 unwaived CRITICAL/HIGH finding(s) in permitio/pdp-v2:latest...
*Dependabot* - 1 new unwaived CRITICAL alert: CVE-2026-99999 in requests.

They ran on separate crons at first, each sending its own message, so one CVE produced three notifications on the same morning — three scanners agreeing, which reads as three problems. The digest keys each scanner on its job result, not its finding count: Scout exports a body on the clean path too, so the body alone cannot tell "the gate passed" from "the gate never ran", and a scanner that did not run must not read as clean. status only escalates ok → warn → fail.

The Dependabot leg filters through .trivyignore.yaml, and this is load-bearing. All three currently-open HIGH alerts (CVE-2026-50271, CVE-2026-54283, CVE-2026-48818) are CVEs already waived there. An unfiltered feed would repeat the same three every run until the channel was muted. Verified against the live feed: 6 open → 3 at critical/high → 0 reported, 3 recognised as waived. Consequence worth knowing: adding a waiver also silences its Dependabot alert.

There is no dependabot_alert workflow trigger, so that leg is a poll. It reports a delta over a 73-hour window (72 + 1h overlap) and the standing backlog, so a run GitHub drops does not lose that day permanently.

Release gate

New verify-published-image job scans the tag that actually shipped, by digest, per platform — the only place linux/arm64 is ever scanned, since tests.yml builds amd64 only. update-pdp-api-ecs-service now lists it in needs:, so an unwaived CRITICAL/HIGH in a freshly published image stops the rollout. It runs after the push, so it cannot un-publish the tag; what it prevents is rolling that image to the fleet.

Supporting changes

  • format_scan_report.py, check_waiver_parity.py, check_dependabot_alerts.py — 81 tests in horizon/tests/, which is the path the required pytests job actually runs.
  • classify_image_cves.py now fails closed. A zero-byte report was reported as CLEAN with exit 0; it now exits 2 with verdict=ERROR.
  • Waiver parity gap closed, 5 → 7 (CVE-2026-48710, CVE-2026-48817 were VEX-only), and enforced by a waiver-parity pre-commit hook rather than a comment.
  • notify-slack.yml — reusable notifier, a green no-op with a ::notice:: until SLACK_WEBHOOK_URL exists, and it reports its own delivery failures.
  • The rolling [image-cve] GitHub issue is removed, not deprecated.
  • PyYAML declared; it was reaching a required check only via uvicorn[standard].

Verification

  • actionlint: 8 findings, all pre-existing — zero introduced, one cleared vs the baseline.
  • zizmor: clean on every new workflow; 17 pre-existing findings cleared across the branch.
  • 81 tests pass; every mutation applied during review was caught.
  • pre-commit run --all-files: all 17 hooks pass.
  • All 10 action pins re-resolved live against the API — zero version comments that disagree with their SHA.
  • Gate strength re-derived independently by extracting the real run: bodies and driving them over 13 fixtures; the merged digest over 12 more.

Before this does anything

  1. SLACK_WEBHOOK_URL does not exist. Every Slack path is a green no-op until it does. A green notify job proves nothing on its own — look for the ::notice::.
  2. Copy CLONE_REPO_TOKEN, DOCKERHUB_USERNAME, DOCKERHUB_TOKEN into the Dependabot secret store. That store is empty, so build-pdp-image fails on every Dependabot PR and takes pdp-tester and docker-scout with it — the image gate does not run on the PRs it exists for. Pre-existing, not introduced here.
  3. Create the dependencies label. ci: CVE-gate PRs, releases and published images; harden workflows and credentials (PER-15358) #338's dependabot.yml sets it; the repo has only the nine defaults, so those PRs arrive unlabeled.
  4. Unproven assumption: that Scout resolves registry://permitio/pdp-v2:latest to the PURL pkg:docker/permitio/pdp-v2@latest. Docker's docs say it does. If not, all 7 waivers stop suppressing. One workflow_dispatch after merge settles it.

Known gap, documented rather than fixed

dependency-review is a third waiver surface that does not read .trivyignore.yaml, so a PR touching the starlette or ddtrace pins can be failed on an already-waived advisory. check_waiver_parity.py covers 2 of 3 surfaces.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VvR24Cibs2Z8cVsEqkp3Es

zeevmoney and others added 3 commits September 16, 2026 23:51
…and alert

Adopts the agent-security pattern for all three scanners: a PR gate with a
sticky report, immediate Slack on CRITICAL/HIGH, and a release gate with teeth.

PR gate and report
- Trivy moves from `exit-code: 1` to scan-to-JSON, format, then a separate fail
  step, so findings exist as a file and can be reported. The gate blocks on the
  same predicate as before plus four report-unreadable paths the old exit code
  could not express.
- One sticky comment per PR, marker `<!-- pdp-image-scan -->`, covering Trivy
  and Scout together. Scout's own write-comment goes off on both steps, so a PR
  gets at most two bot comments instead of three.
- New dependency-review job gates the dependency diff and posts on failure only.

Immediate Slack, not a weekly batch
- notify-slack.yml, a reusable notifier that is a green no-op with a ::notice::
  until SLACK_WEBHOOK_URL exists, and that reports its own delivery failures.
- Trivy: daily published-tag scan alerts when any tag is not CLEAN.
- Docker Scout: new daily leg on `latest`. Every OpenVEX statement gains
  pkg:docker/permitio/pdp-v2@latest so the seven waivers keep suppressing.
- Dependabot: daily poll of the alerts API. There is no dependabot_alert
  workflow trigger, so this is a schedule. It filters through .trivyignore.yaml
  because all three currently-open HIGH alerts are already waived, and an
  unfiltered feed would ping daily about triaged work.
- Gate failures alert on push and release, never on pull_request, where the red
  check and the comment already reach the author.
- The weekly digest stays, demoted to a roll-up. It uniquely carries the
  waiver-expiry warning: all seven waivers expire 2026-12-31 together.

Release gate
- verify-published-image scans the tag that actually shipped, by digest, on both
  platforms. This is the only place linux/arm64 is ever scanned.
- update-pdp-api-ecs-service now needs it, so a CVE-bearing image does not reach
  the fleet. It runs after the push, so it cannot un-publish the tag.

Supporting
- format_scan_report.py, check_waiver_parity.py, check_dependabot_alerts.py,
  with 81 tests. classify_image_cves.py now fails closed: a zero-byte report was
  reported as CLEAN with exit 0.
- Waiver parity gap closed, 5 -> 7, and enforced by a pre-commit hook rather than
  a comment. The rolling GitHub issue is removed, not deprecated.
- PyYAML declared; it was reaching a required check only via uvicorn[standard].

actionlint 8 findings, all pre-existing, one cleared. zizmor clean on the new
workflows. Slack stays inert until the secret is added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sage

Trivy, Docker Scout and Dependabot ran on separate crons (06:17, same run but
its own notify, and 07:00) and each sent its own Slack message. One CVE in a
published image therefore produced three notifications on the same morning -
three scanners agreeing with each other, which reads as three problems and is
how a channel gets muted.

They now run as parallel jobs under one trigger in daily-security-scan.yml,
and the `digest` job merges their verdicts into one body with a line per
scanner, so agreement is visible at a glance. The cost is that the message
waits for the slowest scanner; on a daily cadence that is minutes.

- image-scan-published.yml -> daily-security-scan.yml, and the `dependabot`
  job moves in from the deleted dependabot-alert-watch.yml.
- `digest` now needs [scan, scout-latest, dependabot] and keys each scanner on
  its JOB RESULT, not on a finding count: Scout's summary step exports a body
  on the clean path too, so the body alone cannot tell "the gate passed" from
  "the gate never ran", and a scanner that did not run must not read as clean.
- `status` only escalates ok -> warn -> fail, so a clean Dependabot result
  cannot talk down a Trivy SOURCE verdict.
- notify-scout is gone; there is now exactly one Slack sender in the workflow,
  plus verify-delivery so a dead webhook reddens the run rather than passing
  silently on a workflow that is quiet by design.
- The dispatch input is `all_alerts` now that it shares a workflow with `tags`.
- Cron stays 06:17 - off the hour, which GitHub documents as the slot most
  likely to be dropped, and the Dependabot leg carries a --new-since window.

Digest logic exercised over 12 cases against the extracted run: silent only
when all three are clean; notifies on REBUILD, SOURCE, ERROR, a failed or
skipped Scout job, a new Dependabot alert, a backlog-only alert, a failed
Dependabot job, and on no verdict files at all. shellcheck clean.

actionlint 8 findings (unchanged, all pre-existing), zizmor clean, 81 tests
pass, all 17 pre-commit hooks pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VvR24Cibs2Z8cVsEqkp3Es
Cron moves from `17 6 * * *` to `17 6 */3 * *`, and the workflow is renamed
daily-security-scan.yml -> scheduled-security-scan.yml because a file called
"daily" that runs every third day is the same silently-wrong config this
branch exists to remove.

The Dependabot leg moves with it. It reports a DELTA over a time window, so a
window sized to the old cadence would have gone blind for two days in three:
--new-since goes 25 -> 73 hours (72 plus one hour of overlap, so an alert
opened between two runs cannot fall through the gap). The workflow now passes
it explicitly, next to the cron it derives from, and the script's default
moves in step for anyone running it by hand.

Be precise about what `*/3` in the day-of-month field is: days 1, 4, 7 ... 31,
resetting each month, so the 31st-to-1st gap is one day and February's is one
or two. Cron has no true "every 72 hours". Both consequences are bounded - a
SHORT gap only means the window overlaps and may repeat an alert once, and a
LONG gap cannot occur - and the digest already branches on the standing
backlog (total_unwaived) rather than only on what is new in the window, which
is what makes a missed run recoverable.

actionlint 8 findings (unchanged, all pre-existing), zizmor clean, 81 tests
pass, all 17 pre-commit hooks pass. Verified against the live alert feed at
both the default and the explicit window: 0 new, 0 unwaived, 3 waived.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VvR24Cibs2Z8cVsEqkp3Es
Copilot AI lite review requested due to automatic review settings September 23, 2026 12:59
@zeevmoney

Copy link
Copy Markdown
Member Author

Closing — not stacking. These commits go onto #338's branch directly.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown

PDP image vulnerability report

Image: permitio/pdp-v2:next

✅ No CRITICAL or HIGH findings after waivers.

Trivy

No CRITICAL or HIGH findings.

Docker Scout

No CRITICAL or HIGH findings.


Scanned by Trivy, Docker Scout. Waivers: .trivyignore.yaml + .docker/scout/pdp-v2.vex.json. View run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants