fix(service): release the balancer when a service stops needing it by lexfrei · Pull Request #47 · Treetscom/robotlb

lexfrei · 2026-09-29T09:36:59Z

When a Service stops being a robotlb load balancer, its Hetzner balancer is now deleted and the finalizer removed. Until now such a Service was skipped, so the balancer stayed billed and the finalizer stayed.

The API server clears loadBalancerClass together with the type, so robotlb decides ownership by its finalizer. A Service that carries it but is no longer a robotlb LoadBalancer, after a type change or a change back with another class, goes through the same release path as a deleted Service. The status is left alone, since the API server clears it on the type change.

Release finds the balancer by name, as before. If the robotlb/balancer annotation is changed or removed in the same edit as the type, release deletes the balancer under the name the Service has now. That can be the balancer of another Service with the same name, see #41. #62 replaces this lookup with an ownership label. No unit test covers the lookup in the release path, since it sits in async code that talks to Hetzner.

A Service with no TCP port that has a node port now keeps its balancer and external IP and gets a warning event. Before, it lost the IP while the balancer stayed.

On upgrade, Services changed from LoadBalancer before this release still carry the finalizer, and their balancer is deleted on the first start. The README has a command to list them. Its balancer is found by name, so a same-named Service in another namespace may share it, see #41.

This is stacked on #37 and targets its branch.

Closes #38

Once the Hetzner Cloud API budget ran out, every service was retried every 30 seconds, and each retry spent the budget being waited for. The budget is per project, so after a 429 all services now pause together, starting at one minute and doubling up to 16 minutes while the limit keeps being hit. Adding targets stops at the first 429 instead of calling the API for every remaining node. The generated client drops response headers, so RateLimit-Reset cannot be used. A failed reconciliation now leaves a SyncLoadBalancerFailed warning event on the service, so kubectl describe shows the last error. Errors are logged on one line with the status and the message Hetzner reported, and the API token, which Hetzner quotes in some messages, is redacted from logs and events. The chart role gains permission to create events. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>

Services paused by the rate limit gate were all requeued for the moment it reopened, so they hit the API in one burst while the budget was still nearly empty. Each wait now grows by up to a quarter, derived from the service name, so services wake up spread over that window. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>

With wakeups spread by service name, the same service wakes first after every pause, hits the limit and closes the gate again. Every other service only ever saw the closed gate, which published no event and logged at debug level, so after the event TTL they showed nothing at all. A service that wakes up to a closed gate now gets an event and an info line. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>

The comment claimed the default grants all permissions, while it lists only what robotlb needs. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>

A service changed from LoadBalancer to another type was skipped, so its Hetzner balancer kept running and its finalizer stayed. The API server clears loadBalancerClass with the type, so ownership is now decided by the finalizer: a service robotlb no longer serves has its balancer deleted and the finalizer removed, the same path a deleted service takes. The balancer to delete is found by the Hetzner ID robotlb now records on the service, since the name annotation may be gone by then. The ID is bound to the service UID, so a manifest exported and applied under another name cannot release the original's balancer. Without a recorded ID, or when no balancer has it, the balancer is looked up by name as before. A service with no TCP port that has a nodePort used to lose its external IP while the balancer behind it stayed. The balancer and the address are now kept, and the service gets a warning event instead, since a missing nodePort is more likely a mistake than a request to delete the balancer. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>

s3rius · 2026-09-29T12:17:53Z

+    }
+    // Hetzner IDs are positive; anything else was edited by hand and must not
+    // make every cleanup attempt fail.
+    id.parse().ok().filter(|id| *id > 0)


I don't think that this check is required, since we assume that those annotations are not edited by hand anyway.

@s3rius Both go away with #62, which is stacked on this PR. It removes the robotlb/balancer-id annotation completely. robotlb then finds its balancer by the robotlb/service-uid label it puts on the balancer when it creates it. If you'd rather not have the annotation land here at all, I can drop it from this PR. Until #62 the release would then find the balancer only by name.

s3rius · 2026-09-29T12:19:35Z

+    Some(json!({
+        "metadata": {
+            "annotations": {
+                consts::LB_ID_ANN_NAME: format!("{uid}/{id}")


Why do we need service uid? I guess we can just set ID of an actual hetzner's loadbalancer.

@s3rius Same here, #62 removes this annotation, UID included.

Then we can close this PR and rebase the #62 on top of main.

@s3rius This PR is more than the annotation. It also fixes #38: robotlb releases the balancer when a Service changes its type or class, and #57 is already merged into this branch. #62 builds on that release logic, so closing #47 would drop the #38 fix.

The annotation is a leftover. I built the stack step by step and didn't get to the final shape right away, so I missed that #62 makes it unnecessary. I'll remove it from this PR, so it never lands.

cleanup removed every service and every target one request at a time and only then deleted the balancer, so a balancer with 3 ports and 10 targets cost 14 API calls instead of 1. That matters when the project is close to the hourly rate limit. The API reference only says "Deletes a Load Balancer" and does not describe what happens to its services and targets. The change follows how Hetzner's own cloud controller manager and CLI delete a balancer: one delete call, with no prior removal of services or targets and no network detach. Network handling is left as it was. A failure part-way no longer leaves a balancer with some services already removed. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>

Recording the balancer ID on the service as an annotation adds state that users can copy between objects or edit by hand, and the operator has to guard against both. Drop the annotation and look the balancer up by name when releasing it, as robotlb did before. A balancer whose name annotation was changed or removed before the release is not found, and the README now says so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>

Associate newly provisioned Hetzner balancers with the immutable Service UID. Resolve or delete resources by ownership instead of name. Conservatively adopt recognizable pre-upgrade balancers in place, keeping public IPs and existing labels. Ignore foreign balancers. Release on deletion or type/class change without parsing annotations; reject unrecognized name collisions rather than taking over LBs. Ported-from: Treetscom#62 Ported-from: Treetscom#47 Ported-from: Treetscom#64

lexfrei added 4 commits September 29, 2026 01:05

docs(helm): describe the default service account permissions

785e5bf

The comment claimed the default grants all permissions, while it lists only what robotlb needs. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>

This was referenced Sep 29, 2026

Update kube, k8s-openapi and hcloud to their current releases #50

Open

fix(deps): update dependencies with security advisories #52

Open

lexfrei force-pushed the fix/release-balancer branch from 76182d6 to d6c6ee0 Compare September 29, 2026 15:15

This was referenced Sep 29, 2026

fix(lb): delete the balancer with a single request #57

Merged

fix(lb): act only on balancers labelled for the service #62

Open

s3rius reviewed Oct 6, 2026

View reviewed changes

lexfrei mentioned this pull request Oct 6, 2026

fix(lb): reject a balancer name Hetzner does not accept #67

Closed

lexfrei force-pushed the fix/release-balancer branch from 96a8ddf to bea258d Compare October 6, 2026 09:54

s3rius added this pull request to stack #81 October 13, 2026 10:16

Navigation Menu

Repository navigation

fix(service): release the balancer when a service stops needing it - #47

fix(service): release the balancer when a service stops needing it#47
lexfrei wants to merge 7 commits into
fix/hcloud-rate-limitfrom
fix/release-balancer

lexfrei commented Sep 29, 2026 •

edited

Loading

Uh oh!

s3rius Sep 29, 2026

Uh oh!

lexfrei Oct 6, 2026

Uh oh!

s3rius Sep 29, 2026

Uh oh!

lexfrei Oct 6, 2026

Uh oh!

s3rius Oct 6, 2026

Uh oh!

lexfrei Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lexfrei commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s3rius Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

lexfrei Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

s3rius Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

lexfrei Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

s3rius Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

lexfrei Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lexfrei commented Sep 29, 2026 •

edited

Loading