Repository navigation
Conversation
Once the Hetzner Cloud API budget ran out, every service was retried every 30 seconds, and each retry spent the budget being waited for. The budget is per project, so after a 429 all services now pause together, starting at one minute and doubling up to 16 minutes while the limit keeps being hit. Adding targets stops at the first 429 instead of calling the API for every remaining node. The generated client drops response headers, so RateLimit-Reset cannot be used. A failed reconciliation now leaves a SyncLoadBalancerFailed warning event on the service, so kubectl describe shows the last error. Errors are logged on one line with the status and the message Hetzner reported, and the API token, which Hetzner quotes in some messages, is redacted from logs and events. The chart role gains permission to create events. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Services paused by the rate limit gate were all requeued for the moment it reopened, so they hit the API in one burst while the budget was still nearly empty. Each wait now grows by up to a quarter, derived from the service name, so services wake up spread over that window. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
With wakeups spread by service name, the same service wakes first after every pause, hits the limit and closes the gate again. Every other service only ever saw the closed gate, which published no event and logged at debug level, so after the event TTL they showed nothing at all. A service that wakes up to a closed gate now gets an event and an info line. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The comment claimed the default grants all permissions, while it lists only what robotlb needs. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every LoadBalancer service was reconciled every 30 seconds, and each run cost one or two Hetzner API requests even when nothing changed. The short interval was there because only Services were watched, so a new node or a moved pod reached the balancer only through that requeue. Node changes that affect targets now reconcile every service. Heartbeat updates are ignored, since only labels, cordoning, readiness and addresses are compared. An endpoint slice change reconciles its service when it is a robotlb balancer with the Local traffic policy and the nodes its targets follow changed: all endpoint nodes when targets come from pods, ready endpoint nodes otherwise. Readiness flaps and address changes that leave those nodes alone cost nothing. The controller's watch mapper cannot tell a deletion from an update, so a second watch tracks which slices exist; a deleted slice with endpoints reconciles every service. The periodic resync is now ROBOTLB_RESYNC_INTERVAL, five minutes by default and at most a year. A reconcile that could not add some targets, for a reason other than the rate limit, is retried after 30 seconds like a failed one, so a temporarily rejected node does not wait for the resync. Pods that have finished or have no IP yet no longer count as targets. Neither is in the endpoint slice, so deleting them would leave their node a target until the resync. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Collaborator
Author
|
Checked on a live cluster with a build of this branch and debug logging, on a LoadBalancer Service with a nodePort:
The API budget can't show the difference over a window this short, since it refills by one request per second. The skip line and the shorter log per reconcile are the evidence. |
…56) A Local service with a selector took its targets from the pods the selector matched. That kept nodes where kube-proxy had no endpoint to send traffic to, such as the node of a pod that is not ready yet, and so nodes that fail the balancer's health check. A cordoned or not-ready node running a matching pod stayed a target too, unlike under the Cluster policy, with nothing saying whether that was intended. Every Local service now takes its targets from its EndpointSlices, the path already used for services without a selector. A node is a target when kube-proxy serves Local traffic from it: it has a ready endpoint, or a terminating endpoint that still serves, which kube-proxy falls back to while the node has no ready one. Missing conditions are read the way kube-proxy reads them. Targets and the slice trigger share this filter, so the trigger no longer needs a separate pod-based mode. robotlb no longer reads pods at all, and the chart's default role drops its access to them. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Base automatically changed from
fix/local-targets-from-slices
to
perf/watch-cluster-changes
October 6, 2026 08:12
Every write to a Service starts a reconcile, including status writes by another LoadBalancer controller that also takes Services without a class. When that controller clears status.loadBalancer, robotlb writes it back, the other controller clears it again, and each round costs Hetzner requests until the project hits the rate limit. Services carry no metadata.generation, so a generation predicate cannot filter these events, and filtering the main stream needs an unstable kube-runtime feature. robotlb now remembers, per Service UID, a hash of the spec, the robotlb/ annotations and the computed targets and ports after each successful reconcile. A reconcile that finds the same hash before the next check is due makes no Hetzner requests and writes no status. Node and EndpointSlice changes still get through because they change the targets. An entry is valid until the requeue of the reconcile that recorded it, so while Hetzner refuses some targets events are skipped only until the 30-second retry. The entry is dropped after a failed reconcile, when the Service is released or deleted, and when it has no port to expose. The README now says which classes robotlb handles, that setting loadBalancerClass: robotlb keeps other controllers away, and that recreating a Service to add the class costs it its balancer and public IP. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
lexfrei
force-pushed
the
fix/skip-unchanged-reconcile
branch
from
October 6, 2026 09:34
3821dc6 to
5f018d3
Compare
pando85
added a commit
to forkline/robotlb
that referenced
this pull request
Oct 13, 2026
Prevent repeated Service watch events caused by writing the same ingress status after every reconcile. Adapt the API-call suppression principle from upstream while retaining Forkline's controller structure. Related-to: Treetscom#72
s3rius
force-pushed
the
perf/watch-cluster-changes
branch
from
October 13, 2026 10:13
f245740 to
82d632b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
robotlb now skips the Hetzner calls when nothing it acts on has changed since the last successful reconcile. The README now explains how to keep another LoadBalancer controller away from robotlb's Services.
Two controllers that both take Services without a class keep undoing each other's
status.loadBalancer. MetalLB started without--lb-classis one example. Every status write is a watch event and starts a reconcile, and every reconcile talks to Hetzner. The loop runs until the project hits the rate limit. Services have nometadata.generation, so a generation predicate does not filter these events. A filter on the main stream needsController::for_stream, which is still unstable.After each successful reconcile, robotlb keeps a hash per Service UID. It covers the spec, the
robotlb/annotations, and the targets and ports robotlb computed. The next reconcile computes targets and ports as before, which costs only Kubernetes reads. If the hash is the same and the next check is not due yet, it stops before the first Hetzner call and does not write the status. It requeues for the time left until that check. The next check is the resync, or the 30-second retry while Hetzner refuses some targets. Node and EndpointSlice changes still get through, because they change the targets.The hash is dropped after a failed reconcile, and when the Service is released or deleted. It is also dropped when the Service has no port to expose, because that path clears the status and the status has to come back with the ports.
The README now says that robotlb handles Services with no
loadBalancerClassor withrobotlb. SettingloadBalancerClass: robotlbkeeps the other controller away. The field can only be set when the Service is created or its type is changed toLoadBalancer. Recreating a Service deletes its balancer, and the new one gets a new public IP, so the README says to set the class at creation.There are two trade-offs:
robotlbclass ends the fight over the status.Some smaller limits I left alone. If a Service is deleted without robotlb seeing its deletion (for example after someone removes the finalizer by hand), its hash stays in memory until the process restarts. If the balancer has no public IP yet on the first reconcile, the status is written at the next check, not at the next Service event. A reconcile that waits at the rate limit gate also counts as failed and drops the hash, so every Service does a full reconcile once the gate opens. Keeping the hash there would save requests right when the budget is lowest, but I kept "any failure drops the hash" as the simpler rule.
Unit tests cover the skip decision before and after the check is due, and with the same or a different hash. They check that the hash changes with a spec field, a robotlb annotation, the targets and a port. They also check that it stays the same when only the status,
resourceVersionor a foreign annotation changes, and that the hash is dropped on failure, release and deletion. I broke the code under each test and saw it go red: without the targets, without the annotations, with the expiry ignored, with a fixed expiry instead of the one recorded, and with the hash kept after a failure. Two mutations survive. One records the hash before the Hetzner calls succeed, the other bypasses the check. Both sit inreconcile_load_balancer, and catching them needs a mocked Kubernetes API and Hetzner API. A failed reconcile drops the hash anyway, so the first one only changes how long the hash stays valid when Hetzner refuses some targets.Stacked on #56, only the last commit is new.
Closes #69