fix(kubescape): report health correctly and make the level filter work

[slashben]

What's wrong

kubescape_check_health reports "healthy": false on a fully working Kubescape install. It's the tool an agent calls first, and answering "is this working?" is its whole job — so an agent tells the user Kubescape is broken and recommends reinstalling something that is already installed correctly.

Verified on a live cluster with all 7 Kubescape pods Running and every other tool returning correct data:

"healthy": false,
"summary": "Kubescape has issues that need attention",
"vulnerability_crd": {"status":"error","message":"VulnerabilityManifests CRD not installed..."},
"configuration_crd": {"status":"error","message":"WorkloadConfigurationScans CRD not installed..."},
"storage_pods":      {"status":"warning","message":"No storage pods found..."}

…while kubectl get vulnerabilitymanifests -A returned data and the storage pod was 1/1 Running.

Three separate bugs cause this. Also fixed: an advertised filter argument that silently does nothing.

1. It checks for CRDs that never exist

Background: most operators define their data types as CRDs. Kubescape doesn't — it runs its own extension API server (a pod named storage, registered via an APIService) which serves its data types. They look like normal Kubernetes resources to a client, but no CRD objects exist for them, ever, in any install.

The health check did four CRD lookups:

k.apiExtClient.ApiextensionsV1().CustomResourceDefinitions().Get(ctx, "vulnerabilitymanifests.spdx...", ...)

These return NotFound on every install, forever. Confirmed on two clusters:

$ kubectl get crd | grep spdx.softwarecomposition          # -> nothing, ever
$ kubectl api-resources --api-group=spdx.softwarecomposition.kubescape.io | wc -l
15                                                          # -> 15 resources, all served, zero CRDs

Fix: check availability by listing each resource through the client the data tools already use. If the list succeeds, the API is there. This proves the exact path the tools depend on rather than a proxy for it — a CRD (or discovery) check can pass while the storage pod is dead and every tool still fails.

It also uses fewer API calls than before: one list per resource answers both "is the API reachable?" and "is there data?", replacing 4 CRD gets + 8 lists with 4 lists.

The apiextensions client is no longer needed anywhere and is removed. Check keys are unchanged (vulnerability_crd, etc.) so existing consumers are unaffected; only the lookup and the message wording change.

This also fixes a latent bug: two checks (application_profiles_data, network_neighborhoods_data) were nested inside the always-failing CRD lookup, so they never ran at all.

2. The pod checks select the wrong pods

app.kubernetes.io/name=kubescape-operator is the chart-wide label on every pod the Helm chart creates, not the operator's own label. Per-component identity is in the plain app label:

$ kubectl get pods -n kubescape -o custom-columns='NAME:.metadata.name,CHART_LABEL:.metadata.labels.app\.kubernetes\.io/name,APP:.metadata.labels.app'
kubescape-64c56c75c6-hjnfc   kubescape-operator   kubescape
kubevuln-c6bb59f9c-v5xlq     kubescape-operator   kubevuln
node-agent-7vfq2             kubescape-operator   node-agent
operator-5d854fdc8f-j75sb    kubescape-operator   operator
storage-5ff6f76c7f-ngb2d     kubescape-operator   storage

$ kubectl get pods -n kubescape -l app.kubernetes.io/name=kubescape-operator | wc -l   # 7
$ kubectl get pods -n kubescape -l app.kubernetes.io/name=storage            | wc -l   # 0

So "operator_pods": "7/7 pods running" was really an all-pods count — if node-agent died you'd get 6/7 with no way to tell which component failed — and "storage_pods": "No storage pods found" was emitted while the storage pod was healthy.

Fix: select on app=operator and app=storage.

A missing storage pod now fails the health check instead of warning. Every read this provider makes goes through that pod, so the one check that would catch the failure mode breaking all nine data tools previously matched nothing and wouldn't have failed the check if it had.

Reviewer note: app=operator is verified on chart 1.40.x. If an older chart labelled differently, this would swap a false pass for a false fail. I think that's the right trade — a false fail is visible and fixable, while today's false fail is unconditional — but I'm happy to add a fallback to the chart-wide selector if you'd prefer.

3. The level filter silently does nothing

kubescape_list_vulnerability_manifests advertises level: image, workload, or both. All three returned byte-identical output:

call	bytes	sha256[:12]
no level	1898	6f1e27dfb8fb
level=image	1898	6f1e27dfb8fb
level=workload	1898	6f1e27dfb8fb

The provider passed the filter as a Kubernetes label selector, but the Kubescape API server version shipped by the chart ignores label selectors on list and returns everything regardless (a nonsense selector also returns every row).

This is fixed upstream — kubescape/storage#362, released in storage v0.0.305 — but the chart still pins v0.0.298, seven releases earlier, and a client can't know which server version it's talking to.

Fix: filter client-side, using the same image-level/workload-level test the response already computes for its own output fields. So the filter and the reported fields can't disagree, and it returns the same answer against either server version. An unrecognised level value is now a validation error instead of silently meaning "both".

Risk / compatibility

• No output keys added, removed or renamed. Message text changes; statuses change where they were wrong.
• Behaviour change: a missing storage pod now fails the health check (previously a warning).
• One client dependency removed (apiextensions), no new ones.
• Confined to pkg/kubescape.

Testing

go test ./pkg/kubescape/... — 60 pass, no cluster needed. go build, go vet, golangci-lint, go test -tags=test ./pkg/... ./internal/... (739 pass) all clean. The pre-existing test/e2e suite needs a live kind cluster and was not run.

All tests written first and observed failing. Two notes:

• Health tests seed pods carrying both labels — the chart-wide one and app=<component> — i.e. the real shape. That's what makes the old selector fail and the new one pass.
• The level test installs a reactor that reproduces the real server ignoring label selectors. Without it the fake clientset honours selectors, so a server-side filter would pass the test while returning unfiltered data on a real cluster — which is exactly how this survived.

Two existing tests whose premise was "no CRDs installed" are rewritten to exercise genuine API unavailability, including a partial-failure case (one resource down, another still reporting ok).

Ticket

None.