Repository navigation
fix(shared/go/aws/cloudwatchlogs): do not skip late-ingested events - #1589
roeeelnekave-spec wants to merge 1 commit into
Conversation
Each poll restarted at the newest event timestamp seen. CloudWatch Logs ingests each stream of a log group with its own delay, so an event that arrives after a newer event from another stream (e.g. a second EKS API server) was never read. The boundary event was also sent twice. Re-read a 60s reorder window before the newest event on every poll and drop events already sent, by event ID. Signed-off-by: Roee Elnekave <roee.elnekave@infinitylabs.co.il>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: roeeelnekave-spec The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @roeeelnekave-spec! It looks like this is your first PR to falcosecurity/plugins 🎉 |
|
Tried this on an EKS cluster with 2 apiservers, 60 I have tests for the window and the dedup against a fake |
What type of PR is this?
/kind bug
Any specific area of the project related to this PR?
/area plugins
What this PR does / why we need it:
The shared CloudWatch Logs reader (
shared/go/aws/cloudwatchlogs, used byk8saudit-eks) loses events.After each poll it sets
StartTimeto the newest event timestamp it has seen. A log group is not ingested in timestamp order: each stream has its own delivery delay. EKS writes onekube-apiserver-audit-*stream per API server, and we measured 0 to 5 s ingestion lag per stream. An event ingested after a newer event from the other stream has a timestamp beforeStartTimeat the next poll, so it is never read. The dedup check (*input.StartTime == *e.Timestamp && lastIngestionTime >= *e.IngestionTime) runs after the event has already been sent, so events at the boundary timestamp are also delivered twice.In practice a
kubectl get secretby a user was in CloudWatch but never reached Falco, so "K8s Secret Get Successfully" did not fire.This PR re-reads a 60 s reorder window before the newest event on every poll and skips events already sent, using the CloudWatch event ID. The ID set is pruned to the window, so memory stays bounded.
Measured on a two-API-server EKS cluster (~20 audit events/s): stock
k8saudit-eks0.11.0 and a build with this patch ran side by side for 4 minutes, compared with theauditID/stageof every eventFilterLogEventsreturned for the same window:Which issue(s) this PR fixes:
None filed.
Special notes for your reviewer:
ReorderWindow, 60 s), not an option, to keepCreateOptionsunchanged for other consumers. Happy to make it configurable.FilterLogEventshas no per-event charge; read volume per poll grows by roughly60 / polling_interval.