Skip to content

fix(controller): skip Hetzner calls when nothing robotlb acts on changed - #72

Open
lexfrei wants to merge 7 commits into
perf/watch-cluster-changesfrom
fix/skip-unchanged-reconcile
Open

lexfrei wants to merge 7 commits into
perf/watch-cluster-changesfrom
fix/skip-unchanged-reconcile

Conversation

@lexfrei

@lexfrei lexfrei commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator

robotlb now skips the Hetzner calls when nothing it acts on has changed since the last successful reconcile. The README now explains how to keep another LoadBalancer controller away from robotlb's Services.

Two controllers that both take Services without a class keep undoing each other's status.loadBalancer. MetalLB started without --lb-class is one example. Every status write is a watch event and starts a reconcile, and every reconcile talks to Hetzner. The loop runs until the project hits the rate limit. Services have no metadata.generation, so a generation predicate does not filter these events. A filter on the main stream needs Controller::for_stream, which is still unstable.

After each successful reconcile, robotlb keeps a hash per Service UID. It covers the spec, the robotlb/ annotations, and the targets and ports robotlb computed. The next reconcile computes targets and ports as before, which costs only Kubernetes reads. If the hash is the same and the next check is not due yet, it stops before the first Hetzner call and does not write the status. It requeues for the time left until that check. The next check is the resync, or the 30-second retry while Hetzner refuses some targets. Node and EndpointSlice changes still get through, because they change the targets.

The hash is dropped after a failed reconcile, and when the Service is released or deleted. It is also dropped when the Service has no port to expose, because that path clears the status and the status has to come back with the ports.

The README now says that robotlb handles Services with no loadBalancerClass or with robotlb. Setting loadBalancerClass: robotlb keeps the other controller away. The field can only be set when the Service is created or its type is changed to LoadBalancer. Recreating a Service deletes its balancer, and the new one gets a new public IP, so the README says to set the class at creation.

There are two trade-offs:

  • While the hash matches, a status cleared by another controller stays empty until the next check. This change stops the quota burn, and only the robotlb class ends the fight over the status.
  • A change made to the balancer directly in Hetzner is not noticed until the next check either. Before, any watch event on the Service caught it.

Some smaller limits I left alone. If a Service is deleted without robotlb seeing its deletion (for example after someone removes the finalizer by hand), its hash stays in memory until the process restarts. If the balancer has no public IP yet on the first reconcile, the status is written at the next check, not at the next Service event. A reconcile that waits at the rate limit gate also counts as failed and drops the hash, so every Service does a full reconcile once the gate opens. Keeping the hash there would save requests right when the budget is lowest, but I kept "any failure drops the hash" as the simpler rule.

Unit tests cover the skip decision before and after the check is due, and with the same or a different hash. They check that the hash changes with a spec field, a robotlb annotation, the targets and a port. They also check that it stays the same when only the status, resourceVersion or a foreign annotation changes, and that the hash is dropped on failure, release and deletion. I broke the code under each test and saw it go red: without the targets, without the annotations, with the expiry ignored, with a fixed expiry instead of the one recorded, and with the hash kept after a failure. Two mutations survive. One records the hash before the Hetzner calls succeed, the other bypasses the check. Both sit in reconcile_load_balancer, and catching them needs a mocked Kubernetes API and Hetzner API. A failed reconcile drops the hash anyway, so the first one only changes how long the hash stays valid when Hetzner refuses some targets.

Stacked on #56, only the last commit is new.

Closes #69

Once the Hetzner Cloud API budget ran out, every service was retried
every 30 seconds, and each retry spent the budget being waited for. The
budget is per project, so after a 429 all services now pause together,
starting at one minute and doubling up to 16 minutes while the limit
keeps being hit. Adding targets stops at the first 429 instead of
calling the API for every remaining node. The generated client drops
response headers, so RateLimit-Reset cannot be used.

A failed reconciliation now leaves a SyncLoadBalancerFailed warning
event on the service, so kubectl describe shows the last error. Errors
are logged on one line with the status and the message Hetzner reported,
and the API token, which Hetzner quotes in some messages, is redacted
from logs and events. The chart role gains permission to create events.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Services paused by the rate limit gate were all requeued for the moment
it reopened, so they hit the API in one burst while the budget was
still nearly empty. Each wait now grows by up to a quarter, derived
from the service name, so services wake up spread over that window.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
With wakeups spread by service name, the same service wakes first after
every pause, hits the limit and closes the gate again. Every other
service only ever saw the closed gate, which published no event and
logged at debug level, so after the event TTL they showed nothing at
all. A service that wakes up to a closed gate now gets an event and an
info line.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The comment claimed the default grants all permissions, while it lists
only what robotlb needs.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every LoadBalancer service was reconciled every 30 seconds, and each run
cost one or two Hetzner API requests even when nothing changed. The
short interval was there because only Services were watched, so a new
node or a moved pod reached the balancer only through that requeue.

Node changes that affect targets now reconcile every service. Heartbeat
updates are ignored, since only labels, cordoning, readiness and
addresses are compared. An endpoint slice change reconciles its service
when it is a robotlb balancer with the Local traffic policy and the
nodes its targets follow changed: all endpoint nodes when targets come
from pods, ready endpoint nodes otherwise. Readiness flaps and address
changes that leave those nodes alone cost nothing. The controller's
watch mapper cannot tell a deletion from an update, so a second watch
tracks which slices exist; a deleted slice with endpoints reconciles
every service. The periodic resync is now ROBOTLB_RESYNC_INTERVAL, five
minutes by default and at most a year.

A reconcile that could not add some targets, for a reason other than
the rate limit, is retried after 30 seconds like a failed one, so a
temporarily rejected node does not wait for the resync.

Pods that have finished or have no IP yet no longer count as targets.
Neither is in the endpoint slice, so deleting them would leave their
node a target until the resync.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@lexfrei

lexfrei commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Checked on a live cluster with a build of this branch and debug logging, on a LoadBalancer Service with a nodePort:

  1. Adding a foreign annotation (example.com/skip-test=1) gave one reconcile that logged "Nothing robotlb acts on changed since the last reconcile. Skipping...". The health check interval in Hetzner stayed at 15.
  2. Setting robotlb/lb-check-interval=16 gave a full reconcile, and the interval in Hetzner became 16.
  3. Removing the foreign annotation was skipped again.
  4. Removing robotlb/lb-check-interval gave a full reconcile, and the interval went back to 15.

The API budget can't show the difference over a window this short, since it refills by one request per second. The skip line and the shorter log per reconcile are the evidence.

…56)

A Local service with a selector took its targets from the pods the
selector matched. That kept nodes where kube-proxy had no endpoint to
send traffic to, such as the node of a pod that is not ready yet, and
so nodes that fail the balancer's health check. A cordoned or
not-ready node running a matching pod stayed a target too, unlike
under the Cluster policy, with nothing saying whether that was
intended.

Every Local service now takes its targets from its EndpointSlices, the
path already used for services without a selector. A node is a target
when kube-proxy serves Local traffic from it: it has a ready endpoint,
or a terminating endpoint that still serves, which kube-proxy falls
back to while the node has no ready one. Missing conditions are read
the way kube-proxy reads them.

Targets and the slice trigger share this filter, so the trigger no
longer needs a separate pod-based mode. robotlb no longer reads pods
at all, and the chart's default role drops its access to them.

Assisted-by: LLM

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Base automatically changed from fix/local-targets-from-slices to perf/watch-cluster-changes October 6, 2026 08:12
Every write to a Service starts a reconcile, including status writes
by another LoadBalancer controller that also takes Services without a
class. When that controller clears status.loadBalancer, robotlb writes
it back, the other controller clears it again, and each round costs
Hetzner requests until the project hits the rate limit. Services carry
no metadata.generation, so a generation predicate cannot filter these
events, and filtering the main stream needs an unstable kube-runtime
feature.

robotlb now remembers, per Service UID, a hash of the spec, the
robotlb/ annotations and the computed targets and ports after each
successful reconcile. A reconcile that finds the same hash before the
next check is due makes no Hetzner requests and writes no
status. Node and EndpointSlice changes still get through because they
change the targets. An entry is valid until the requeue of the
reconcile that recorded it, so while Hetzner refuses some targets
events are skipped only until the 30-second retry. The entry is
dropped after a failed reconcile, when the Service is released or
deleted, and when it has no port to expose.

The README now says which classes robotlb handles, that setting
loadBalancerClass: robotlb keeps other controllers away, and that
recreating a Service to add the class costs it its balancer and
public IP.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@lexfrei
lexfrei force-pushed the fix/skip-unchanged-reconcile branch from 3821dc6 to 5f018d3 Compare October 6, 2026 09:34
pando85 added a commit to forkline/robotlb that referenced this pull request Oct 13, 2026
Prevent repeated Service watch events caused by writing the same ingress
status after every reconcile. Adapt the API-call suppression principle
from upstream while retaining Forkline's controller structure.

Related-to: Treetscom#72
@s3rius
s3rius force-pushed the perf/watch-cluster-changes branch from f245740 to 82d632b Compare October 13, 2026 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant