Repository navigation
fix(lb): retry targets rejected because the balancer is busy - #59
Merged
Merged
Conversation
This was referenced Sep 29, 2026
s3rius
requested changes
Oct 6, 2026
s3rius
left a comment
s3rius
left a comment
Member
There was a problem hiding this comment.
I'm not sure if we even need retries, because kubernetes handles reconcilidation itself.
So in any case the retry will happen on kubernetes side. Maybe we don't need this PR, it adds functionality that already exists but on a platform level.
pando85
added a commit
to forkline/robotlb
that referenced
this pull request
Oct 13, 2026
Retry 409/423/5xx and robot_unavailable responses on target updates, balancer type and algorithm changes with 1s/2s delay. Treat a target already added by a previous attempt as success. 429 stays governed by the shared rate-limit gate. Ported-from: Treetscom#59 Related-to: Treetscom#66
s3rius
added this pull request to stack #80
October 13, 2026 08:43
s3rius
approved these changes
Oct 13, 2026
Hetzner answers 423 locked while an action is still running on the balancer, and the operator skipped the target on that answer as if it were permanently invalid. A node added right after another change could stay out of the balancer until the next reconcile. Tell temporary rejections (locked, conflict, robot_unavailable, 5xx) from permanent ones and retry the former up to twice, after 1s and 2s. A target that still fails is skipped as before, and a 429 is still returned at once for the rate limit gate. Once one target has used up its retries, the rest of the run does not retry: the lock is on the whole balancer, and waiting per target would stall the service for minutes on a large balancer. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
s3rius
force-pushed
the
fix/retry-busy-balancer
branch
from
October 13, 2026 10:13
128fd31 to
2e700c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A target added while another action is still running on the balancer gets skipped as if the IP were invalid (#54). Hetzner answers 423
lockedin that window.The add-target loop now retries a temporary rejection (
locked,conflict,robot_unavailableor a 5xx) twice, after 1s and 2s. A permanent one, like an IP outside the vSwitch subnet, is skipped as before. A 429 is still returned at once for the rate limit gate. After one target has used up its retries, the other targets in that run are not retried, because the lock is on the whole balancer.I did not add waiting for the running action to finish. That costs extra API calls per target, and the retry plus the existing 30 second requeue covers the short window. Other balancer calls (removing targets, changing services) still fail on
lockedas before.tokio::timerelies on the tokiotimefeature; #52 declares it explicitly.Stacked on #49.
Closes #54