Skip to content

feat: port upstream reconciliation efficiency, failure backoff, multicluster naming and image cleanup - #65

Merged
pando85 merged 47 commits into
masterfrom
feat/upstream-operability-and-multicluster-2026-10
Oct 13, 2026
Merged

pando85 merged 47 commits into
masterfrom
feat/upstream-operability-and-multicluster-2026-10

Conversation

@pando85

@pando85 pando85 commented Oct 13, 2026 •

Copy link
Copy Markdown

Scope

Single integration PR adapting six upstream changes on top of Forkline's existing UID-only ownership, safe capacity handling, and Warning Event throttling:

Upstream Adaptation
Treetscom#72 Successful desired-state fingerprint of Service spec, robotlb/ annotations, targets and ports; bypass Hetzner calls on unchanged events; retain periodic drift resync
Treetscom#55 Per-Service bounded exponential error retries (5–300 s), with separate existing project-wide 429 gate
Treetscom#49 Watch Node target-relevant state without reacting to heartbeats; retain Forkline's existing EndpointSlice watch and chosen pod-based Local behavior
Treetscom#77 Show underlying DNS/TLS/connectivity causes for Hetzner API failures, including structured API response messages
Treetscom#58 Remove redundant jemalloc build/runtime packages and install only required runtime TLS libraries
Treetscom#68 Optional validated cluster prefix for default LB names, stable FNV-1a suffix if longer than 128 chars; respect explicit LB names

Upstream provenance

Ported or adapted from original upstream commits:

Original SHAs were not cherry-picked verbatim, because upstream PRs were stacked on older, incompatible controller structures and are in a separate GitHub repository. Each port has a commit referencing the upstream SHA; the existing Forkline commits and ancestry remain intact.

Review follow-up: controller correctness

Addresses the three issues found in the deep review:

  1. Provisioning without an ingress address: an LB that has no assignable public/private ingress returns a deferred reconcile and retries after 15 seconds. It is not fingerprint-memoized until Kubernetes can advertise an ingress address.
  2. Rate limit versus failure backoff: successful, deferred and skipped reconciliations are distinguished. A shared project-wide 429 pause no longer clears an existing per-Service failure-backoff progression.
  3. EndpointSlice churn: only a change in relevant target nodes triggers the Service watcher. Selector-backed Local Services retain Pod-based placement and ignore readiness/address churn on the same nodes; selectorless Local Services continue to follow ready endpoints. A metadata-only watch detects slice deletions and missed deletions on relist.

New tests cover the deferred outcome, backoff resets, mapping modes, readiness/address churn and slice deletions.

The Dockerfile now builds with --locked; CI builds the final release image and checks shared-library linkage and robotlb --help.

Safety and compatibility

  • Never adopt unowned Hetzner resources or silently truncate LB targets.
  • Local target selection is unchanged; Node watches only trigger reconciliation.
  • Successful memoization does not eliminate periodic Hetzner drift checks.
  • Existing uid-labelled LBs remain owned even if the configured cluster prefix changes.
  • No automatic merge; test with disposable Hetzner resources before production rollout.

Validation

Run cargo fmt, cargo clippy --all-targets -- -D warnings, cargo test --all-targets --locked, Helm unit tests and container build.

Final review follow-up (October 13)

  • Cache the successful ingress snapshot alongside the Service desired-state fingerprint. Unchanged desired state now repairs externally modified status.loadBalancer.ingress without any Hetzner API call; a status patch failure invalidates the memo.
  • Detect changes to kubernetes.io/service-name in the existing metadata-only EndpointSlice watcher, including relabels discovered during relists. Wake reconciliation for both old and new owners through one existing global signal, and invalidate stale slice snapshots.
  • Keep ingress provisioning polling fixed at 15 seconds. After 180 seconds, return LoadBalancerProvisioningTimeout, emit the existing throttled Kubernetes Warning Event and failure metric, but continue polling every 15 seconds without exponential backoff. Clear the deadline after successful provisioning or Service removal.
  • Regression tests cover cached ingress, deadline/reset behavior, label changes, relist reconciliation and the fixed retry policy.

@forkline

forkline Bot commented Oct 13, 2026

Copy link
Copy Markdown

⏸️ CI Auto-Fix paused

Maximum consecutive fix attempts (3/3) reached for this PR without success.
CI auto-fix is now paused to prevent infinite retry loops.

You can:

  • Review the CI logs and fix the issue manually
  • Use /reset-ci-auto-fix to give CI auto-fix another chance after making changes

To configure the maximum attempts, go to Repository Settings > Automation > CI Auto-Fix.

@pando85
pando85 marked this pull request as ready for review October 13, 2026 10:15
@forkline

forkline Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

✅ Fix pushed:

All CI issues from commit 988f645 have already been fixed in subsequent commits on this branch. The fixes include:

  • chrono dependency: Replaced k8s_openapi::chrono::Utc::now() with JSON deserialization (commit a73bae7)
  • missing resync_interval: Added the field to test fixture (commit 63810bd)
  • Clippy doc_markdown: Added backticks around LoadBalancer (commit d105d08)
  • Clippy missing_errors_doc: Added # Errors section to parse_cluster_name (commit d105d08)
  • Clippy collapsible_if: Collapsed nested if using let chains (commit d105d08)
  • rustfmt: Applied formatting changes across multiple files (commits c7bf240, b13fb20, f1d4978, aeba254, 81afb8a)

Current state:

  • cargo fmt --check ✓
  • cargo clippy --all-targets --all-features -- -D warnings ✓
  • cargo test - 78 tests pass ✓

No additional changes needed. The branch is up to date with origin.

…le handling slice deletion via metadata watch
@pando85
pando85 merged commit fc17cbb into master Oct 13, 2026
3 checks passed
@pando85
pando85 deleted the feat/upstream-operability-and-multicluster-2026-10 branch October 13, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant