Repository navigation
fix(hcloud): back off on rate limits and report failures on the service - #37
Merged
Merged
Conversation
lexfrei
force-pushed
the
fix/hcloud-rate-limit
branch
from
September 29, 2026 09:47
a1547b9 to
785e5bf
Compare
This was referenced Sep 29, 2026
pando85
added a commit
to forkline/robotlb
that referenced
this pull request
Oct 13, 2026
Guard Hetzner API calls across all managed Services after a 429, using a 60-second to 16-minute exponential pause. Avoid extending the cooldown for already-in-flight requests. Preserve typed errors from balancer creation so 429s are distinguishable. Ported-from: Treetscom#37
s3rius
added this pull request to stack #80
October 13, 2026 08:43
Once the Hetzner Cloud API budget ran out, every service was retried every 30 seconds, and each retry spent the budget being waited for. The budget is per project, so after a 429 all services now pause together, starting at one minute and doubling up to 16 minutes while the limit keeps being hit. Adding targets stops at the first 429 instead of calling the API for every remaining node. The generated client drops response headers, so RateLimit-Reset cannot be used. A failed reconciliation now leaves a SyncLoadBalancerFailed warning event on the service, so kubectl describe shows the last error. Errors are logged on one line with the status and the message Hetzner reported, and the API token, which Hetzner quotes in some messages, is redacted from logs and events. The chart role gains permission to create events. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Services paused by the rate limit gate were all requeued for the moment it reopened, so they hit the API in one burst while the budget was still nearly empty. Each wait now grows by up to a quarter, derived from the service name, so services wake up spread over that window. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
With wakeups spread by service name, the same service wakes first after every pause, hits the limit and closes the gate again. Every other service only ever saw the closed gate, which published no event and logged at debug level, so after the event TTL they showed nothing at all. A service that wakes up to a closed gate now gets an event and an info line. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The comment claimed the default grants all permissions, while it lists only what robotlb needs. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
A reconcile that failed for any reason other than a rate limit was retried after a fixed 30 seconds, however long the error persisted. When the failing step is a Hetzner API call, a service stuck on a permanent error repeats that call twice a minute and spends the per-project request budget that every other service shares. Retry each service after 5 seconds, doubling up to 5 minutes, the per-item backoff the upstream cloud-provider service controller uses. A successful reconcile, or a skip once the service is no longer a robotlb load balancer, starts the delay short again. A wait at the rate limit gate neither lengthens nor resets it. Entries that have not failed for an hour are dropped, so services deleted before they got the finalizer do not accumulate. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
s3rius
force-pushed
the
fix/hcloud-rate-limit
branch
from
October 13, 2026 10:13
d9927e5 to
f5dd223
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Once the Hetzner Cloud API budget runs out, robotlb retries every service every 30 seconds, and each retry spends the budget it is waiting for. The failure is also hard to see from the cluster.
The budget is per project, so after a 429 all services now pause together. The pause starts at one minute and doubles up to 16 minutes while the limit keeps being hit. A 429 from a call that was already in flight does not lengthen it, and adding targets stops at the first 429 instead of calling the API for every remaining node. Services wake up spread over up to a quarter of the pause, so they don't all hit the API at the moment it ends.
RateLimit-Resetis not used: the generated hcloud client drops response headers, so the header never reaches robotlb.A failed reconciliation now leaves a
SyncLoadBalancerFailedwarning event on the service, the same reason the upstream service controller uses, sokubectl describeshows the last error. A service that wakes up to the paused gate gets one too. Errors are logged on one line with the HTTP status and the code and message Hetzner reported. The API token, which Hetzner quotes in some error messages, is redacted from logs and events. The chart role gets permission to create events.Charts that override
serviceAccount.permissionsreplace the default list, so they needcreateonevents.k8s.ioevents added by hand. Without it, a failure only logs that the event could not be published.Other errors are still retried every 30 seconds, see #46, and kube 0.96 does not aggregate events, see #45.
Closes #36