[Suselz]

What type of PR is this?

/kind bug

Any specific area of the project related to this PR?

/area plugins

What this PR does / why we need it:

The webhook handler no longer waits without a limit for room in the event queue,
and it no longer queues a batch whose request the API server has already cancelled.
Without this, Falco stops seeing most audit events after the queue fills up once, until it is restarted.

Today the handler writes 200 and then blocks on serverEvtChan <- b (50 batches) without watching req.Context().
The status reaches the API server only when the handler returns.
With a full queue, the API server cancels the request after its 30-second timeout and sends the batch again,
while the cancelled handler still queues its copy.
Every batch is processed at least twice, so the queue never drains, even after the load goes down.

With this change, the handler:

• queues the batch at once when the queue has room, and answers 200 only after that;
• waits for room for at most webhookEnqueueTimeout seconds, then answers 429 without Retry-After;
• answers 503 and does not queue the batch when the request is cancelled while waiting, or when the server shuts down;
• logs refused batches separately for a full queue and for a cancelled request,
  at most once in two enqueue timeouts and at least one second apart.

Why 429 without Retry-After:

• the REST client of the API server repeats a request on its own only when a 429 or 5xx response
  carries Retry-After (checkWait).
  With the header, it would resend the same batch within the same 30-second request timeout and keep the queue under pressure;
• without the header, the error goes to the audit webhook back-off
  (WithExponentialBackoff,
  DefaultRetryBackoffWithInitialDelay):
  five attempts with pauses of 10, 15, 22.5 and 33.75 seconds by default, each with up to 20% of jitter.
  After the last attempt, the API server drops the batch and counts it in apiserver_audit_error_total.

New init config parameter webhookEnqueueTimeout: the maximum time in seconds a webhook request waits for room
in the event queue before it is refused with 429. The default is 5, well below the 30-second request timeout
of the audit webhook. A value of 0 refuses a request at once when the queue is full.
The parameter is described in the init schema and in the plugin README.

How it was tested:

• go build ./..., go vet ./... and go test ./... in plugins/k8saudit pass, also with -race -count=3 for the new tests;
• the new tests in pkg/k8saudit/webserver_test.go fail with the source.go of main:
  the request over a full queue hangs until the client gives up, and the cancelled request is still being handled after 5 seconds;
• make in plugins/k8saudit builds libk8saudit.so, and golangci-lint run --new-from-rev=main reports no issues in the change;
• Falco 0.41.3 loads the plugin with webhookEnqueueTimeout set, rejects a non-integer value through the init schema,
  answers 200 to a webhook batch and triggers a rule from k8s_audit_rules.yaml;
• on a Kubernetes 1.35 cluster with a single control-plane node and Falco limited to 200m CPU, the audit webhook
  was overloaded for about 4 minutes. Without the change, events were not delivered after the load went down until Falco was restarted.
  With the change, all test events sent after the overload were delivered in three runs,
  and apiserver_audit_error_total stopped growing a few minutes after the overload.

Which issue(s) this PR fixes:

Fixes #1599

Special notes for your reviewer:

The event parser goroutine of OpenWebServer does not watch the instance context while it pushes events
(source.go#L388).
In the tests that fill the queue, nothing reads the events, so the parser stays blocked on its output channel after Close,
and its goroutine leaks until the test binary exits. It holds no port and does not affect other tests.
This PR does not change the parser, since the leak does not affect the webhook behaviour fixed here.

A slot that frees up at the moment the request is cancelled can still take the batch.
This is no worse than a cancellation right after the batch is queued, which the handler cannot detect either.

I did not bump the plugin version or edit CHANGELOG.md, since recent changes leave both to separate release PRs.

Does this PR introduce a user-facing change?:

fix(plugins/k8saudit): refuse webhook batches with 429 after `webhookEnqueueTimeout` seconds when the event queue is full, and never queue a batch whose request has been cancelled

[poiana]

Welcome @Suselz! It looks like this is your first PR to falcosecurity/plugins 🎉

[github-actions]

Rules files suggestions

rules

Comparing 3f5c9f0b478cddc736dc19e071174fd789d7c479 with latest tag plugins/k8saudit/v0.18.0

No changes detected

[leogr]

LGTM, thanks! 🙏

[poiana]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: leogr, Suselz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[poiana]

LGTM label has been added.

Details

Git tree hash: 2d6a828658e1493a9b20d061b5a59055b47941fe