feat(kubescape): add vulnerability drill-down tools and bound CVE output - #77
Open
slashben wants to merge 2 commits into
Open
feat(kubescape): add vulnerability drill-down tools and bound CVE output#77slashben wants to merge 2 commits into
slashben wants to merge 2 commits into
Conversation
An agent could not answer "what are the worst vulnerabilities in my cluster?"
without either being lied to or blowing its context window. The only route to
any severity information was list_vulnerabilities, which returned 200,793 B
(~50k tokens) for a single image, and nothing aggregated across manifests at
all, so cluster-scope questions had no answer at any price.
The Kubescape storage API already exposes a full aggregation ladder that this
provider never used: cluster -> namespace -> workload -> manifest -> CVE, where
every level except the last is a handful of integers plus a reference.
Two new tools walk it:
kubescape_vulnerability_overview
Severity totals per namespace from the server-side aggregates, worst
first. One call, ~1 KB, from vulnerabilitysummaries (cluster-scoped, one
object per namespace).
kubescape_list_vulnerable_workloads
Workloads ranked by severity, each row carrying the manifest_name that
kubescape_list_vulnerabilities takes, read from the summary's
spec.vulnerabilitiesRef. One call, ~1.9 KB per workload.
Both take ListOptions.ResourceVersion = "fullSpec".
That sentinel is load-bearing and not obvious. The storage server strips spec
from EVERY list response by default, so severity counters come back zero and
vulnerabilitiesRef comes back empty -- not an error, and indistinguishable from
a clean cluster. Measured on storage v0.0.298, nginx:1.14.0 has 76 critical CVEs
and a default LIST of its summary reports 0. This is the same failure shape as
the vulnerability_count bug, one level up, so listing these resources without
the sentinel would report a vulnerable cluster as clean.
Verified against a live cluster that the typed client transmits it: a wire
capture shows resourceVersion=fullSpec on the request, and the returned counts
match the values an individual GET reports.
Because that correctness rests on a vendor sentinel, the overview also refuses
to present an all-zero cluster as good news: it says the result could not be
confirmed and points at a per-manifest check. A wrong "no vulnerabilities" is
the most expensive answer this provider can give.
kubescape_list_vulnerabilities is reshaped to fit the ladder:
- severity_summary always describes the WHOLE manifest, even when the array
is filtered or truncated, so the aggregate survives a bounded response.
- The array is capped by limit (default 20), ordered worst severity first with
id as a tiebreak for stable output, and reports total_count, returned_count
and truncated so the agent knows it holds partial data.
- Records carry only id, severity and fix_state. Measured field shares of the
old payload: description 56.1%, data_source 19.5%, fix_versions 7.9%. The
description was truncated at 200 chars mid-word, so it was lossy prose that
invited reasoning from a fragment; the full text, data source and fix
versions are all in get_vulnerability_details.
- New severity and fixable_only filters for drill-down.
- Grype emits one match per affected package, so the same CVE recurs. With the
package fields trimmed away those rows are byte-identical, and duplicates
would consume the limit budget: the first 20 records for nginx:1.14.0 held
CVE-2017-12424 and CVE-2017-15670 twice each. Matches are now collapsed by
CVE id with an affected_artifacts count, and a CVE left unfixed in any
package is never reported as fixed. 466 matches become 293 distinct CVEs.
Measured against the same image on a live cluster: 3,013 B, against 200,793 B
before -- 67x smaller, with a complete and correct severity summary.
Also fixes a miscount in that summary: severityCounts had no Negligible bucket,
so the 102 Negligible CVEs in nginx:1.14.0 were reported as "Unknown": 102.
Severity handling is now driven by one ordered list used for both bucketing and
ranking, so a severity cannot be dropped into the wrong bucket again.
Relevancy is surfaced where the API provides it -- "relevant" counts CVEs whose
code node-agent observed loaded at runtime, e.g. 76 critical of which 30 are
reachable. It is emitted only when non-zero: the field is `json:"relevant,omitempty"`
upstream, so a zero is indistinguishable from "relevancy was never computed",
and node-agent needs a learning period before it reports anything. Rendering
absent as 0 would claim nothing is reachable when nobody has looked yet.
One value from the summaries cannot be trusted. vulnerabilitiesRef reports the
WORKLOAD's namespace, but the manifests live in the Kubescape namespace, so
following it verbatim yields NotFound -- confirmed on a live cluster, where a GET
of docker.io-library-nginx-1.14.0-e34030 in the referenced namespace fails while
the same GET in `kubescape` succeeds. Only the name is taken from the ref; the
namespace comes from the operator namespace, overridable with kubescape_namespace.
Handing the agent a pointer that 404s would defeat the point of the chain.
Tests are fake-client based and need no cluster. Note that client-go's fake
drops ResourceVersion from recorded actions, so a reactor cannot observe the
sentinel; the tests wrap the typed client to record the ListOptions each handler
actually passes, which pins the invariant at the real call site.
Signed-off-by: Ben Hirschberg <ben@armosec.io>
Docs-exempt: no existing doc describes the vulnerability tool surface
Signed-off-by: Ben <ben@armosec.io>
…cription The description asserted that a missing 'relevant' count means relevancy was not computed. That overstates what can be known: the upstream field is `json:"relevant,omitempty"`, so an absent key is equally a genuine zero, and the two are indistinguishable in the source data. The design proposed deriving a relevancy availability verdict from the summary's kubescape.io/status annotation. Measurement rules that out -- workloads with status=ready were observed with 'relevant' absent -- so no such field is emitted. The description now states the ambiguity in both directions and tells the caller not to read an absent value as a measured zero. Signed-off-by: Ben Hirschberg <ben@armosec.io> Docs-exempt: tool description wording, no behavioral change Signed-off-by: Ben <ben@armosec.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's wrong
The obvious question — "what are the worst vulnerabilities in my cluster?" — can't be answered with the current tools without blowing the context window.
kubescape_list_vulnerabilitiesreturns 200,793 B (~50k tokens) for a single image.kubescape_get_vulnerability_detailsneeds acve_id, and the only tool that emits CVE IDs is that 200 KB call.This PR adds two cheap tools that answer cluster- and workload-level questions, and bounds the expensive one.
Every number below was measured by running the built binary over MCP stdio against a live cluster.
Results
vulnerability_overview(whole cluster)vulnerability_overview(one namespace)list_vulnerable_workloads(all namespaces)list_vulnerable_workloads(namespace, limit 2)list_vulnerabilities(nginx:1.14.0, default)list_vulnerabilities(limit=0, all 293 CVEs)get_vulnerability_detailsA full drill-down — cluster → worst workloads → that image's CVEs → one CVE in detail — now costs ~11 KB.
Background: where the cheap data comes from
Kubescape's scanner writes three levels of object, which this provider only ever read the most expensive of:
The top two levels are just integers plus a reference. The provider went straight to the bottom level for everything, which is why every question cost 200 KB.
The two new tools
kubescape_vulnerability_overview— severity totals per namespace, worst first, in one call. Answers "where is the risk concentrated?", which has no tool today.kubescape_list_vulnerable_workloads— workloads ranked by severity, in one call, where each row carries themanifest_namethatlist_vulnerabilitiestakes as input. The agent is handed its next call rather than guessing an object name.Real output (verified against
kubectl):{"workload": "deployment/vuln-nginx", "container": "nginx", "image": "docker.io/library/nginx:1.14.0", "severities": {"Critical": {"all": 76, "relevant": 30}, "High": {"all": 133, "relevant": 34}}, "manifest_name": "docker.io-library-nginx-1.14.0-e34030", "manifest_namespace": "kubescape"}relevantis a Kubescape feature nothing surfaced before: it counts CVEs whose code was actually observed loaded at runtime. "76 critical, 30 reachable" is a far better triage signal than a raw count. (See the caveat at the bottom — it's reported carefully.)Reading those summary objects requires a non-standard query parameter, and getting it wrong fails silently in the worst possible direction.
Kubescape's API server strips object contents from every LIST response by default and only returns them if you pass
resourceVersion=fullSpec. Without it, severity counts come back as zeros — not an error, and indistinguishable from a clean cluster.Measured on the same objects, same moment:
resourceVersion=fullSpecSo
nginx:1.14.0has 76 critical CVEs, and a default LIST of its summary reports 0. This is the same failure shape as the bug in #76, one level up — writing these tools the obvious way would report a vulnerable cluster as clean.Two things follow, and both are deliberate:
1. The parameter is verified, not assumed. I wrote a throwaway Go program that logs the outgoing HTTP request, to confirm the Kubernetes client actually transmits this non-standard value rather than validating it away:
It's passed through verbatim, no error or warning, and the returned counts match what a single-object GET reports. A
RESTClient()fallback also works if that ever changes. (Verified forList()only —Watch()treatsresourceVersiondifferently and isn't used here.)2.
overviewwill not report an all-zero cluster as clean. If every namespace reports zero it says the result could not be confirmed and points at a per-manifest check, because "no vulnerabilities" is the most expensive wrong answer this provider can give and it should never be produced by a silent mechanism failure. Happy to drop this if you find it too defensive — it's the one piece of opinion in the PR.list_vulnerabilitiesreshapedlimit(default 20), worst severity first, withtotal_count/returned_count/truncatedso the agent knows it has partial data.id,severity,fix_state,affected_artifacts. Removed:description(56% of the old payload, and truncated at 200 chars mid-word —"...to prevent command inje..."— so it was lossy prose inviting reasoning from a fragment),data_source(19.5%),fix_versions(7.9%). All three are still available in full fromget_vulnerability_details.severityandfixable_onlyfilters.Also fixes a miscount: the severity breakdown had no
Negligiblebucket, so nginx's 102 Negligible CVEs were reported as"Unknown": 102. Visible in the live output before this change.Two bugs the live run caught
Duplicate CVE rows (mine). The scanner emits one row per affected package, so the same CVE recurs. Once I trimmed the package fields, those rows became byte-identical and duplicates ate the
limitbudget — the first 20 records held two CVEs twice each. Now collapsed by CVE id with anaffected_artifactscount, and a CVE unfixed in any package is never reported as fixed. 466 rows → 293 distinct CVEs.A reference in Kubescape's own data is wrong. The per-workload summary points at its CVE object with a namespace that doesn't contain it:
So only the name is taken from that reference; the namespace comes from the operator's namespace (overridable via a
kubescape_namespaceargument). Passing it through verbatim would hand the agent a pointer that 404s. Looks like an upstream Kubescape bug — I'll report it there separately.Risk / compatibility
list_vulnerabilitiesoutput shape changes — fields removed from each entry, array now bounded by default. This is the breaking part and the reason this is a separate PR from fix(kubescape): report health correctly and make the level filter work #75/fix(kubescape): stop reporting vulnerability_count 0 for every image #76. If you'd rather the array default to unbounded, that's a one-line change.scopeargument, at some cost in clarity for the calling model.pkg/kubescape.Testing
go test ./pkg/kubescape/...— 68 pass, no cluster needed.go build,go vet,golangci-lint,go test -tags=test ./pkg/... ./internal/...(747 pass) all clean. The pre-existingtest/e2esuite needs a live kind cluster and was not run.All tests written first and observed failing. One note: the fake clientset silently drops
resourceVersionfrom the actions it records, so a test reactor can't check that the parameter was sent. The tests instead wrap the client to record the options each handler actually passes — otherwise the single most important invariant here would be untested, and losing it would reintroduce the #76 bug at cluster scale.Open questions
kubescape_list_vulnerability_manifestsstill earn its place? Oncelist_vulnerable_workloadsexists it's strictly weaker — same index, no counts, no pointers. Left untouched here.overviewright, or noise?Relationship to the other PRs
Independent of #75 (health checks) and #76 (
vulnerability_count: 0); all three branch frommain. #76 and this one touch nearby code, so whichever lands second needs a trivial rebase.Not included, and worth separate PRs: the provider's client is built once and never retried, so if it starts before the API is reachable all its tools stay dead until the pod restarts; and
internal/errorsmarks unrecoverable errorsRetryable: Yes— visible in this PR's own validation error, whereinvalid severity "Nope"reportsRetryable: Yesand suggests checking your kubeconfig.Ticket
None.