Skip to content

feat(kubescape): add vulnerability drill-down tools and bound CVE output - #77

Open
slashben wants to merge 2 commits into
kagent-dev:mainfrom
slashben:feat/kubescape-vulnerability-drilldown
Open

feat(kubescape): add vulnerability drill-down tools and bound CVE output#77
slashben wants to merge 2 commits into
kagent-dev:mainfrom
slashben:feat/kubescape-vulnerability-drilldown

Conversation

@slashben

@slashben slashben commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What's wrong

The obvious question — "what are the worst vulnerabilities in my cluster?" — can't be answered with the current tools without blowing the context window.

  • kubescape_list_vulnerabilities returns 200,793 B (~50k tokens) for a single image.
  • kubescape_get_vulnerability_details needs a cve_id, and the only tool that emits CVE IDs is that 200 KB call.
  • Nothing aggregates across images, so a cluster-wide question means one 200 KB call per image.

This PR adds two cheap tools that answer cluster- and workload-level questions, and bounds the expensive one.

Every number below was measured by running the built binary over MCP stdio against a live cluster.

Results

Call Before After
vulnerability_overview (whole cluster) no such tool 1,114 B
vulnerability_overview (one namespace) no such tool 1,116 B
list_vulnerable_workloads (all namespaces) no such tool 3,807 B
list_vulnerable_workloads (namespace, limit 2) no such tool 2,057 B
list_vulnerabilities (nginx:1.14.0, default) 200,793 B 3,013 B — 67× smaller
list_vulnerabilities (limit=0, all 293 CVEs) 200,793 B 38,707 B
get_vulnerability_details 5,104 B 5,104 B — unchanged

A full drill-down — cluster → worst workloads → that image's CVEs → one CVE in detail — now costs ~11 KB.

Background: where the cheap data comes from

Kubescape's scanner writes three levels of object, which this provider only ever read the most expensive of:

Level What it holds Size
per-namespace summary severity totals for a namespace ~1 KB for the cluster
per-workload summary severity totals + the name of the object holding that image's CVEs ~1.9 KB each
per-image CVE list every CVE, full detail 2.7 MB each

The top two levels are just integers plus a reference. The provider went straight to the bottom level for everything, which is why every question cost 200 KB.

The two new tools

kubescape_vulnerability_overview — severity totals per namespace, worst first, in one call. Answers "where is the risk concentrated?", which has no tool today.

kubescape_list_vulnerable_workloads — workloads ranked by severity, in one call, where each row carries the manifest_name that list_vulnerabilities takes as input. The agent is handed its next call rather than guessing an object name.

Real output (verified against kubectl):

{"workload": "deployment/vuln-nginx", "container": "nginx",
 "image": "docker.io/library/nginx:1.14.0",
 "severities": {"Critical": {"all": 76, "relevant": 30}, "High": {"all": 133, "relevant": 34}},
 "manifest_name": "docker.io-library-nginx-1.14.0-e34030",
 "manifest_namespace": "kubescape"}

relevant is a Kubescape feature nothing surfaced before: it counts CVEs whose code was actually observed loaded at runtime. "76 critical, 30 reachable" is a far better triage signal than a raw count. (See the caveat at the bottom — it's reported carefully.)

⚠️ The one thing I'd ask you to look at

Reading those summary objects requires a non-standard query parameter, and getting it wrong fails silently in the worst possible direction.

Kubescape's API server strips object contents from every LIST response by default and only returns them if you pass resourceVersion=fullSpec. Without it, severity counts come back as zeros — not an error, and indistinguishable from a clean cluster.

Measured on the same objects, same moment:

Objects default LIST with resourceVersion=fullSpec
per-namespace summaries 903 B — all zeros 996 B — correct (99 critical)
per-workload summaries 12,908 B — all zeros 7,648 B — correct (76 critical for nginx)
per-image CVE lists 14,054 B — contents omitted 3,635,500 B ⚠️ never do this

So nginx:1.14.0 has 76 critical CVEs, and a default LIST of its summary reports 0. This is the same failure shape as the bug in #76, one level up — writing these tools the obvious way would report a vulnerable cluster as clean.

Two things follow, and both are deliberate:

1. The parameter is verified, not assumed. I wrote a throwaway Go program that logs the outgoing HTTP request, to confirm the Kubernetes client actually transmits this non-standard value rather than validating it away:

List(ctx, ListOptions{})                            → GET .../vulnerabilitymanifestsummaries?
List(ctx, ListOptions{ResourceVersion:"fullSpec"})  → GET .../vulnerabilitymanifestsummaries?resourceVersion=fullSpec

It's passed through verbatim, no error or warning, and the returned counts match what a single-object GET reports. A RESTClient() fallback also works if that ever changes. (Verified for List() only — Watch() treats resourceVersion differently and isn't used here.)

2. overview will not report an all-zero cluster as clean. If every namespace reports zero it says the result could not be confirmed and points at a per-manifest check, because "no vulnerabilities" is the most expensive wrong answer this provider can give and it should never be produced by a silent mechanism failure. Happy to drop this if you find it too defensive — it's the one piece of opinion in the PR.

list_vulnerabilities reshaped

  • The severity breakdown always covers the whole image, even when the CVE array is filtered or truncated — so a bounded response still carries the real totals.
  • The array is capped by limit (default 20), worst severity first, with total_count / returned_count / truncated so the agent knows it has partial data.
  • Each entry carries id, severity, fix_state, affected_artifacts. Removed: description (56% of the old payload, and truncated at 200 chars mid-word"...to prevent command inje..." — so it was lossy prose inviting reasoning from a fragment), data_source (19.5%), fix_versions (7.9%). All three are still available in full from get_vulnerability_details.
  • New severity and fixable_only filters.

Also fixes a miscount: the severity breakdown had no Negligible bucket, so nginx's 102 Negligible CVEs were reported as "Unknown": 102. Visible in the live output before this change.

Two bugs the live run caught

Duplicate CVE rows (mine). The scanner emits one row per affected package, so the same CVE recurs. Once I trimmed the package fields, those rows became byte-identical and duplicates ate the limit budget — the first 20 records held two CVEs twice each. Now collapsed by CVE id with an affected_artifacts count, and a CVE unfixed in any package is never reported as fixed. 466 rows → 293 distinct CVEs.

A reference in Kubescape's own data is wrong. The per-workload summary points at its CVE object with a namespace that doesn't contain it:

$ kubectl get vulnerabilitymanifestsummary -n verify-targets deployment-vuln-nginx-nginx \
    -o jsonpath='{.spec.vulnerabilitiesRef.all}'
{"name":"docker.io-library-nginx-1.14.0-e34030","namespace":"verify-targets"}

$ kubectl get vulnerabilitymanifest -n verify-targets docker.io-library-nginx-1.14.0-e34030
Error from server (NotFound)
$ kubectl get vulnerabilitymanifest -n kubescape docker.io-library-nginx-1.14.0-e34030
docker.io-library-nginx-1.14.0-e34030   2026-09-02T07:51:06Z

So only the name is taken from that reference; the namespace comes from the operator's namespace (overridable via a kubescape_namespace argument). Passing it through verbatim would hand the agent a pointer that 404s. Looks like an upstream Kubescape bug — I'll report it there separately.

Risk / compatibility

Testing

go test ./pkg/kubescape/... — 68 pass, no cluster needed. go build, go vet, golangci-lint, go test -tags=test ./pkg/... ./internal/... (747 pass) all clean. The pre-existing test/e2e suite needs a live kind cluster and was not run.

All tests written first and observed failing. One note: the fake clientset silently drops resourceVersion from the actions it records, so a test reactor can't check that the parameter was sent. The tests instead wrap the client to record the options each handler actually passes — otherwise the single most important invariant here would be untested, and losing it would reintroduce the #76 bug at cluster scale.

Open questions

  1. Does kubescape_list_vulnerability_manifests still earn its place? Once list_vulnerable_workloads exists it's strictly weaker — same index, no counts, no pointers. Left untouched here.
  2. Is the all-zero warning in overview right, or noise?
  3. 10 → 12 tools — acceptable, or merge the two new ones?

Relationship to the other PRs

Independent of #75 (health checks) and #76 (vulnerability_count: 0); all three branch from main. #76 and this one touch nearby code, so whichever lands second needs a trivial rebase.

Not included, and worth separate PRs: the provider's client is built once and never retried, so if it starts before the API is reachable all its tools stay dead until the pod restarts; and internal/errors marks unrecoverable errors Retryable: Yes — visible in this PR's own validation error, where invalid severity "Nope" reports Retryable: Yes and suggests checking your kubeconfig.

Ticket

None.

An agent could not answer "what are the worst vulnerabilities in my cluster?"
without either being lied to or blowing its context window. The only route to
any severity information was list_vulnerabilities, which returned 200,793 B
(~50k tokens) for a single image, and nothing aggregated across manifests at
all, so cluster-scope questions had no answer at any price.

The Kubescape storage API already exposes a full aggregation ladder that this
provider never used: cluster -> namespace -> workload -> manifest -> CVE, where
every level except the last is a handful of integers plus a reference.

Two new tools walk it:

  kubescape_vulnerability_overview
      Severity totals per namespace from the server-side aggregates, worst
      first. One call, ~1 KB, from vulnerabilitysummaries (cluster-scoped, one
      object per namespace).

  kubescape_list_vulnerable_workloads
      Workloads ranked by severity, each row carrying the manifest_name that
      kubescape_list_vulnerabilities takes, read from the summary's
      spec.vulnerabilitiesRef. One call, ~1.9 KB per workload.

Both take ListOptions.ResourceVersion = "fullSpec".

That sentinel is load-bearing and not obvious. The storage server strips spec
from EVERY list response by default, so severity counters come back zero and
vulnerabilitiesRef comes back empty -- not an error, and indistinguishable from
a clean cluster. Measured on storage v0.0.298, nginx:1.14.0 has 76 critical CVEs
and a default LIST of its summary reports 0. This is the same failure shape as
the vulnerability_count bug, one level up, so listing these resources without
the sentinel would report a vulnerable cluster as clean.

Verified against a live cluster that the typed client transmits it: a wire
capture shows resourceVersion=fullSpec on the request, and the returned counts
match the values an individual GET reports.

Because that correctness rests on a vendor sentinel, the overview also refuses
to present an all-zero cluster as good news: it says the result could not be
confirmed and points at a per-manifest check. A wrong "no vulnerabilities" is
the most expensive answer this provider can give.

kubescape_list_vulnerabilities is reshaped to fit the ladder:

  - severity_summary always describes the WHOLE manifest, even when the array
    is filtered or truncated, so the aggregate survives a bounded response.
  - The array is capped by limit (default 20), ordered worst severity first with
    id as a tiebreak for stable output, and reports total_count, returned_count
    and truncated so the agent knows it holds partial data.
  - Records carry only id, severity and fix_state. Measured field shares of the
    old payload: description 56.1%, data_source 19.5%, fix_versions 7.9%. The
    description was truncated at 200 chars mid-word, so it was lossy prose that
    invited reasoning from a fragment; the full text, data source and fix
    versions are all in get_vulnerability_details.
  - New severity and fixable_only filters for drill-down.

  - Grype emits one match per affected package, so the same CVE recurs. With the
    package fields trimmed away those rows are byte-identical, and duplicates
    would consume the limit budget: the first 20 records for nginx:1.14.0 held
    CVE-2017-12424 and CVE-2017-15670 twice each. Matches are now collapsed by
    CVE id with an affected_artifacts count, and a CVE left unfixed in any
    package is never reported as fixed. 466 matches become 293 distinct CVEs.

  Measured against the same image on a live cluster: 3,013 B, against 200,793 B
  before -- 67x smaller, with a complete and correct severity summary.

Also fixes a miscount in that summary: severityCounts had no Negligible bucket,
so the 102 Negligible CVEs in nginx:1.14.0 were reported as "Unknown": 102.
Severity handling is now driven by one ordered list used for both bucketing and
ranking, so a severity cannot be dropped into the wrong bucket again.

Relevancy is surfaced where the API provides it -- "relevant" counts CVEs whose
code node-agent observed loaded at runtime, e.g. 76 critical of which 30 are
reachable. It is emitted only when non-zero: the field is `json:"relevant,omitempty"`
upstream, so a zero is indistinguishable from "relevancy was never computed",
and node-agent needs a learning period before it reports anything. Rendering
absent as 0 would claim nothing is reachable when nobody has looked yet.

One value from the summaries cannot be trusted. vulnerabilitiesRef reports the
WORKLOAD's namespace, but the manifests live in the Kubescape namespace, so
following it verbatim yields NotFound -- confirmed on a live cluster, where a GET
of docker.io-library-nginx-1.14.0-e34030 in the referenced namespace fails while
the same GET in `kubescape` succeeds. Only the name is taken from the ref; the
namespace comes from the operator namespace, overridable with kubescape_namespace.
Handing the agent a pointer that 404s would defeat the point of the chain.

Tests are fake-client based and need no cluster. Note that client-go's fake
drops ResourceVersion from recorded actions, so a reactor cannot observe the
sentinel; the tests wrap the typed client to record the ListOptions each handler
actually passes, which pins the invariant at the real call site.

Signed-off-by: Ben Hirschberg <ben@armosec.io>

Docs-exempt: no existing doc describes the vulnerability tool surface
Signed-off-by: Ben <ben@armosec.io>
…cription

The description asserted that a missing 'relevant' count means relevancy was
not computed. That overstates what can be known: the upstream field is
`json:"relevant,omitempty"`, so an absent key is equally a genuine zero, and
the two are indistinguishable in the source data.

The design proposed deriving a relevancy availability verdict from the summary's
kubescape.io/status annotation. Measurement rules that out -- workloads with
status=ready were observed with 'relevant' absent -- so no such field is
emitted. The description now states the ambiguity in both directions and tells
the caller not to read an absent value as a measured zero.

Signed-off-by: Ben Hirschberg <ben@armosec.io>

Docs-exempt: tool description wording, no behavioral change
Signed-off-by: Ben <ben@armosec.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant