Repository navigation
Conversation
An engine that answered its ping but did not finish its initial listing within the engine timeout was skipped for good: bootstrapEngines logged it and moved on, so Falco stayed blind to every container of that runtime for the rest of the run, even after the runtime recovered. Defer the whole engine to the fetcher instead, like a listing that returned ListIncompleteError already does. DeferredContainers gains a RetryList marker: the worker hands the engine over without running it in the synchronous startup, and the fetcher keeps retrying its listing in the background, one bounded attempt per serve iteration at the final backoff delay. Until the listing answers, the engine takes part in no on-demand lookup, since its containers would miss every probe; once it answers, its containers are published and its getter joins the lookups. Follow-up of falcosecurity#1535. Signed-off-by: Konstantin Khlopkov <konstantin.khlopkov93@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kokhlo The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
There is not a label identifying the kind of this issue or PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Rules files suggestions |
What this PR does
An engine that answered its ping but did not finish its initial listing within the engine timeout was skipped for good in
bootstrapEngines: the plugin stayed blind to every container of that runtime for the rest of the run, even after the runtime recovered from its slow start.This is the second follow-up listed in #1535. Instead of dropping the engine, the whole engine is now deferred to the fetcher, the same way a listing interrupted by the engine timeout already is (
ListIncompleteErrorpath):DeferredContainersgains aRetryListmarker for a group holding an engine whose listing timed out.WithEngineTimeout) per serve iteration, re-queued at the final backoff delay while it keeps timing out.IsCreateand its getter joins the on-demand lookups. A partial answer (ListIncompleteError) also joins the existing deferred-container recovery.How to verify it
Unit tests, no runtime required:
TestFetcherRecoversAnEngineDroppedAtStartup: the listing is held by the runtime, then answers; the containers are published and the engine serves lookups.TestFetcherDroppedEngineListingRetryIsBounded: a listing that never answers keeps being retried without blocking requests.TestFetcherDroppedEnginePartialListing: an answer withListIncompleteErrorfeeds the existing deferred recovery.TestBootstrapEnginesSkipsUnresponsiveEngine: the unresponsive engine comes back as aRetryListgroup instead of being dropped.On
main(dbc25c5), the dropped engine never comes back:bootstrapEnginesreturned an emptydeferredfor it.TestContainerdneeds a live containerd socket and fails the same way onmainin this environment.Special notes for your reviewer
Closing the engine was the alternative discussed in #1535, but engines expose no
Close; retrying the listing reuses the deferred machinery already in place for interrupted listings.This follows #1535. cc @fedeGP