Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/eval.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,15 @@ jobs:
--corpus specs/065-evaluation-foundation/datasets/detect_corpus_v1.json \
--gate --min-recall 0.90 --max-fp 0.05

# Spec 122: the review-tier checks (anti-refusal, padding, composite
# exfiltration, prohibition handling, reconstruction limits) measured by
# check id; a review sample must never become a quarantine verdict.
- name: Run Spec-122 review-tier gate (offline, blocking)
run: |
go run ./cmd/scan-eval \
--corpus specs/065-evaluation-foundation/datasets/detect_review_corpus_v1.json \
--review-gate --min-review-recall 0.90 --max-review-fp 0.10

- name: Checkout mcp-eval (public, pinned)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ tail -f ~/Library/Logs/mcpproxy/main.log # main log (macOS; Linux: ~/.mcpproxy/
- **Windows installer**: [docs/github-actions-windows-wix-research.md](docs/github-actions-windows-wix-research.md). **Prerelease** (`next` branch + `v*-rc.*` tags, opt-in, off stable channels): [docs/prerelease-builds.md](docs/prerelease-builds.md).

## Recent Changes
- 122-scanner-skillspector-checks: Go 1.26, backend-only — SkillSpector-informed detect checks (Apache-2.0 attribution in `THIRD_PARTY_NOTICES.md`): `refusal.directive` (anti-refusal / safety nullification / risk concealment, review), `padding.hidden` (horizontal ≥64, vertical ≥12 lines, block >1 KiB, repeated char ≥256, repeated line ≥16, content must follow; also "analysis incomplete" on inspection-budget exhaustion), `exfil.composite` (credential access + transmission + external collector within 400 bytes of one field, authentication excluded, review), `payload.reconstructed` (bounded declared-marker removal: quoted/tag/bare/passive/empty-replacement forms, action-governed payload ≤700 B, ≤64 removals, ≤8 active directives, forwarded shell match is HARD, limits → "analysis incomplete"), and a direct-prohibition cap in `phrase.injection` (`detect.DirectProhibition`: adverb-only binding, same raw line via `detect.LineMap`, no exception/reversal cue). Shared `detect.FieldStrings` walks description + every schema string with JSON-pointer paths and exact offsets (`FieldSpan`). Eval: `--review-gate` over `detect_review_corpus_v1.json`; hard gate gains the `reconstruction` category (self-check corpus copied). Docs: `docs/features/tool-scanner.md` "SkillSpector-informed checks". **No new dependencies.**
- 120-scan-structural-gate: Go 1.26, Vue 3 Web UI — a deterministic structural change gate (`internal/runtime/tool_contract_delta.go`, `contractDelta(approved, live) []Signal`) runs in front of the spec-119 scan-mode approval at both `checkToolApprovals` gate sites: any effective safety-hint change (`GATE-HINT-CHANGE`, safer flips also `GATE-HINT-SAFER-FLIP`), a recursive input-schema widening (`GATE-SCHEMA-SHAPE:<reason>:<path>`: new required, type, enum added/dropped, pattern/format, loosened bound, additionalProperties relaxed, default/const, new steering property, combinators, unknown approved schema), an output-schema change (`GATE-OUTPUT-SCHEMA`) and a hash-equal dispatch-tier change (`DeriveCallWith` moved, FR-009: closes the hash-match/formula/content shortcuts, `previousHintsProveRevert` requires the variant too) hold the change whatever the verdict; signals go first under the `MaxToolHeldSignals` cap. `tool_auto_approved` for a changed contract carries the before-state (`emitToolAutoApprovedChange`). One-time startup WARN (`warnScanDefaultOnce`, marker `scan_default_notice_done`) and `config.ScanDefaultTrustFinding` name servers with no `trust_mode`. Terminology family `hold_signal` (`contracts.AllHoldSignals`, `HoldSignal*` TS consts, `HOLD_SIGNAL_LABELS`), shared fixture `internal/runtime/testdata/contract_delta_cases.json` (Go exact signals + vitest review-line parity), qapilot `Contract.Shape`/`BenignWide`/QP-20/S28/S29. Docs: `docs/features/security-quarantine.md` "Structural changes are held". **No new dependencies.**
- 101-tpa-db US3 + 087-tpa-daily-refresh (tpa-bundle-refresh): Go 1.26, Vue 3 Web UI — a refresh cycle (`internal/security/scanner/tpa_refresh.go`: file leg then optional fetch leg, jittered `tpa_refresh_interval` default 24h, single-flight joins, `Nudge` on `config.reloaded`, joined in `Runtime.Close`) feeds the same source-neutral `evaluateCandidate` pipeline; the opt-in fetch (`tpa_fetch_enabled` default off, `tpa_fetch_url` no compiled-in default, env `MCPPROXY_TPA_FETCH_ENABLED`; Releases-API `latest` or direct `<url>` + `<url>.sig`, signed-only via `evalEnv.MustSign`, https/loopback-only transport with dial-time egress checks, host-only logs, outcomes kept apart from `last_rejection`) works today only with operator-signed bundles (publisher key set still empty); the activation self-check (`tpa_selfcheck.go`, parity-tested port of the `cmd/scan-eval` gate, embedded `bundled/selfcheck-corpus.json`) refuses a candidate as `gate_failure`; fetched bundles survive reload/restart (`externalConfigured`) and disabling fetch keeps them serving; activation observers drive the cached-only re-protection sweep (`internal/server/scan_bundle_reprotect.go`, held-evidence leg + `Service.StartScanFromCachedContracts` gated by `auto_baseline_scan`, marker `storage.BundleReprotectState`, RT-02: never approves, unquarantines or revokes); `BundleInfo` gains `last_checked_at`, `last_activated_at`, `stale`, `refresh`, `reprotect`; surfaces `POST /api/v1/security/bundle/refresh`, `mcpproxy security bundle refresh`, `BundleDoctorWarnings`, telemetry schema 16 refresh counters. Docs: `docs/features/tpa-bundle-signing.md` "Daily refresh and optional fetch", `docs/features/security-quarantine.md` "Re-protection after a bundle activates". **No new dependencies.**
- 101-tpa-db US1 (tpa-db-format): Go 1.26, Vue 3 Web UI — external TPA bundles are signed artifacts: `<bundle>.sig` four-key Ed25519 sidecar (`internal/security/bundlesig`, pure, contract `specs/101-tpa-db/contracts/sidecar-format.md`), verify-before-parse source-neutral `Candidate` pipeline (`scanner/tpa_candidate.go`, FR-008 order, ceilings 8 MiB / 2000 rules / 4 KiB / 5 s via `candidate_open_{unix,windows}.go` no-follow descriptor opens), per-authority (publisher/operator) epoch-scoped watermark + ratchet + pin/deny-list in BBolt `tpa_bundle_state` (one tx with the history row) and content-addressed LKG files `<datadir>/tpa-lkg/`, full-digest rollback, `coverageOK` + async scan-mode admission degrade on the four FR-007/009a/009b conditions. Config: `require_signed_bundle` (default off), `tpa_active_root`, `tpa_operator_signers`, `tpa_operator_baseline`, `tpa_operator_epoch`. Surfaces: `GET/POST /api/v1/security/bundle{,/history,/rollback,/unpin,/reload}`, `mcpproxy security bundle …`, `cmd/tpa-db keygen|sign|verify`, telemetry schema 15 bundle keys. Publisher key set ships EMPTY. Docs: `docs/features/tpa-bundle-signing.md`. **No new dependencies.**
Expand Down
1 change: 1 addition & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -1178,3 +1178,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—`
| [118-pin-visibility-boundary](./specs/118-pin-visibility-boundary/) | — | — |
| [119-scan-default-trust](./specs/119-scan-default-trust/) | — | — |
| [120-scan-structural-gate](./specs/120-scan-structural-gate/) | — | — |
| [122-scanner-skillspector-checks](./specs/122-scanner-skillspector-checks/) | `shipped` | 31/31 (100%) |
33 changes: 33 additions & 0 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Third-Party Notices

MCPProxy includes material adapted from the following third-party open-source
projects. Each entry lists the license, the copyright notice, the upstream
files that were adapted and the files in this repository that carry the
adaptation. The full license texts live under `third_party/`.

## NVIDIA SkillSpector

- **License:** Apache-2.0 (`third_party/skillspector/LICENSE`)
- **Copyright:** Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
- **URL:** https://github.com/NVIDIA/SkillSpector
- **Reviewed at commit:** `5cb9385ab0dffa974088ee21c4376c7e560b2a52` (2026-11-02)
- **Nature of use:** pattern families, thresholds and a bounded grammar were
studied and re-implemented in Go for MCP tool metadata; no Python code,
YARA bundle or test fixture file is copied verbatim. Individual test
sentences reused as regression cases are marked "(upstream)" in the Go test
files. Each adapting Go file carries the upstream copyright line, the
SPDX identifier, the upstream path, the reviewed commit and a summary of the
modifications, as Apache-2.0 §4 requires.

| Upstream file | Adapted in |
|---|---|
| `src/skillspector/nodes/analyzers/static_patterns_anti_refusal.py` (AR1–AR3) | `internal/security/detect/checks/anti_refusal.go` |
| `src/skillspector/nodes/analyzers/whitespace_padding.py` (P9) | `internal/security/detect/checks/padding_hidden.go` |
| `src/skillspector/security_reconstruction.py` | `internal/security/detect/checks/marker_reconstruction.go` |
| `src/skillspector/nodes/analyzers/prohibition_context.py` | `internal/security/detect/prohibition.go` |
| `src/skillspector/yara_rules/agent_skills.yar` (`agent_skill_credential_exfiltration_webhook`), `src/skillspector/nodes/analyzers/static_patterns_data_exfiltration.py` (prose patterns) | `internal/security/detect/checks/composite_exfil.go` |
| `tests/nodes/analyzers/test_static_patterns_anti_refusal.py`, `tests/nodes/analyzers/test_whitespace_padding.py`, `tests/nodes/analyzers/test_security_reconstruction.py`, `tests/nodes/analyzers/test_prohibition_review_regressions.py` (selected sentences) | `internal/security/detect/checks/*_test.go`, `internal/security/detect/prohibition_test.go`, `specs/065-evaluation-foundation/datasets/detect_review_corpus_v1.json` |

The upstream repository's other YARA bundles (`cryptominers.yar`,
`hacktools.yar`, `webshells.yar`, `malware.yar.b64`) and its third-party
notices are NOT included.
8 changes: 5 additions & 3 deletions cmd/scan-eval/gate.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,9 @@ var categoryCheck = map[string]string{
"unicode_smuggling": "unicode.hidden",
"decoded_payload": "payload.decoded",
"shadowing": "shadowing.cross_server",
"phrase_injection": "phrase.injection", // Spec 077 US1 — curated hard check
"capability_mismatch": "capability.mismatch", // US2 (T016) — not yet registered
"phrase_injection": "phrase.injection", // Spec 077 US1 — curated hard check
"capability_mismatch": "capability.mismatch", // US2 (T016) — not yet registered
"reconstruction": "payload.reconstructed", // Spec 122 US3 — declared-marker reconstruction (forwarded shell match is hard)
}

// gateChecks is the canonical set of detect checks the gate runs. It MUST mirror
Expand All @@ -80,7 +81,8 @@ func gateChecks() []detect.Check {
&checks.UnicodeHidden{},
&checks.Shadowing{},
&checks.PayloadDecoded{},
&checks.PhraseInjection{}, // Spec 077 US1 — curated hard injection/exfil check
&checks.PhraseInjection{}, // Spec 077 US1 — curated hard injection/exfil check
&checks.MarkerReconstruction{}, // Spec 122 US3 — reconstructed payloads reach the hard tier through the shell matcher
}
}

Expand Down
12 changes: 12 additions & 0 deletions cmd/scan-eval/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ func run(args []string, stdout, stderr io.Writer) int {
gate := fs.Bool("gate", false, "run the Spec-076 detect.Engine over the corpus, print per-category recall/FP metrics, and exit non-zero on a recall/FP breach (CI regression gate)")
minRecall := fs.Float64("min-recall", 0.90, "gate: minimum malicious recall over gated categories (FR-013)")
maxFP := fs.Float64("max-fp", 0.05, "gate: maximum false-positive rate over benign + hard-negative samples (FR-013)")
reviewGate := fs.Bool("review-gate", false, "run the full baseline check set over a REVIEW-tier corpus (Spec 122): per-category recall by check id (any tier), hard-negative FP by check id, and no soft sample may become dangerous")
minReviewRecall := fs.Float64("min-review-recall", 0.90, "review-gate: minimum recall over the review categories")
maxReviewFP := fs.Float64("max-review-fp", 0.10, "review-gate: maximum false-positive rate over the review hard negatives")

if err := fs.Parse(args); err != nil {
return exitConfigError
Expand All @@ -60,6 +63,15 @@ func run(args []string, stdout, stderr io.Writer) int {
// Gate mode is a distinct pipeline: it runs the offline detect.Engine over
// the labeled detect corpus and enforces recall/FP thresholds. It does not
// emit the per-detector verdict report (that is the default --detectors path).
if *reviewGate {
rc, err := loadReviewCorpus(*corpusPath)
if err != nil {
fmt.Fprintf(stderr, "error: %v\n", err)
return exitConfigError
}
return runReviewGate(rc, *minReviewRecall, *maxReviewFP, stdout, stderr)
}

if *gate {
gc, err := loadGateCorpus(*corpusPath)
if err != nil {
Expand Down
Loading
Loading