Repository navigation
bug(kiosk): console self-rate-limits on first boot (HTTP 429) causing card refresh failures #199
Description
Bug Report
Symptoms
On first boot and initial dashboard hydration, several cards in the KubeStellar console report "Refresh failed" / "3 consecutive refresh failures", and API requests return 429 Too Many Requests.
Root Cause
The console's internal rate-limiting middleware (github.com/kubestellar/console/pkg/api/middleware/ratelimit.go) enforces a token bucket with strict burst caps. When the SPA boots, it fires simultaneous requests to:
/api/mcp/clusters/api/mcp/services/api/mcp/gpu-nodes/stream/api/kagent/status/api/stellar/state
The volume of simultaneous calls on first page load exhausts the bucket immediately, returning HTTP 429 and breaking initial card rendering.
Recommendation
Tune the rate-limiting burst threshold in the console deployment or configure the frontend router to stagger/batch initial API hydration requests.
Attribution
Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Activity
feat(kubernetes): replace the k0s sysext with upstream kubeadm/kubelet
feat(kubernetes): replace the k0s sysext with upstream kubeadm/kubelet
feat(kubernetes): replace the k0s sysext with upstream kubeadm/kubelet
jeefy commentedon Oct 3, 2026
Still relevant after #358/#363: the Console is now the homelab
kubestellaradd-on (ghcr.io/kubestellar/consolev0.3.42), and its first-load burst against the rate limiter is upstream Console behaviour. The fix belongs either upstream (stagger the SPA's initial hydration) or in the add-on's values (the rate-limit burst, if exposed). Leaving this open against the add-on.jeefy commentedon Oct 6, 2026
Status: root cause found. It's upstream. The Console has no setting for this limit, so I can't fix it in the add-on. Leaving this issue open.
Root cause (Console v0.3.42 is the version we pin, and current
mainstill has the same code)pkg/api/routes_auth.gocreatespublicAPI := app.Group("/api", publicLimiterWithSkip). In Fiber v2,Group(prefix, handlers...)registers the handlers asUsemiddleware on the whole/apiprefix. The "public" limiter therefore runs on every/api/*request, including authenticated ones. That limiter is a fixed window of 120 requests per minute perc.IP()./api/mcp/clusters) never get a say. The only skips that apply are/api/me,/api/version,/api/feedback/requests,/api/github/*and/api/auth/*./api/active-users,/api/youtube/playlist,/api/medium/blogand/api/public/nightly-e2e/runs, also namepublicLimiteron the route itself. Those requests count twice./apirequests, about 35 of them in the first 2 seconds, and costs about 45–50 of the 120 budget. Two or three loads (reloads, tabs, a kiosk) inside one minute get 429s, which the cards report as refresh failures.Not a Gateway IP collapse. Envoy Gateway 1.9.2 runs with
use_remote_addressand appends the client address toX-Forwarded-For. Our Gateway Service usesexternalTrafficPolicy: Local. The Console trusts XFF from RFC 1918 sources, which includes the Envoy pods, so each client gets its own bucket. (Side finding:EnableIPValidationis off, soc.IP()returns the raw XFF header. A client-supplied XFF therefore produces a fresh key. I'm reporting that upstream separately.)Evidence
ghcr.io/kubestellar/console:v0.3.42@sha256:8961…7717), run directly in podman. I fired the SPA's ~30 initial GETs in parallel, 5 rounds back to back:load1-3: 28×200 1×502 1×503(the 502/503 come from having no cluster),load4: 15×200 15×429,load5: 30×429. After that,/api/mcp/clustersreturns 429 (it is on the 2000/min skip list, so this is the 120/min limiter) and/api/mereturns 200 (it is skipped)./apirequests in the first 40 s, then about 10/min steady.dogfood-homelab-templateswith an added probe, through the Gateway over HTTPS with thekc_authsession:PROBE cp-console-ratelimit secs=25 paths=30 … clusters-after=429 xff-spoofed=200,200,200, hdr=HTTP/2 429 retry-after: 60. Console log:429 | GET /api/mcp/clusters,429 | GET /api/health, … The Gateway Service showsLoadBalancer Local. All the usual template checks still PASS.Next: an upstream issue at kubestellar/console asking to scope
publicLimiterto the public routes only (or to make the limits configurable). When a fixed Console release is out, bump the pin inscripts/render-homelab-manifests.pyand close this.Karibhou commentedon Oct 20, 2026
Summary: The dashboard can still reject its own requests when it first opens, leaving cards unable to refresh. This server project already uses the latest stable Console release, so there is no fixed version it can ship yet.
The dependency is a KubeStellar Console fix that limits the public request cap to public routes, followed by a stable Console image and chart release. The server's
scripts/render-homelab-manifests.pypins chart and image v0.3.42; v0.3.42 remains the latest stable Console release. The Console has no deployment setting for this limit, as documented in the root-cause investigation. Once a fixed stable release exists, update both pins and verify the first-load request pattern before closing this issue.— hive: backend=codex
33 remaining items
hivecommons-hive commentedon Oct 26, 2026
🛠️ Hive agent
scanner(hive hosted-projectbluefin-knuckle-gjvq) started work on this issue at 2026-10-26T21:48:23Z. A pull request referencing this issue supersedes this mark; it lapses on its own at 2026-10-26T23:48:23Z if none appears.hivecommons-hive commentedon Oct 26, 2026
Summary: I looked at fixing this from the server side and found nothing we can safely change. The request limit is hard-coded inside the Console itself, so the fix has to come from the Console project first.
Re-checked against the root-cause comment above: the 120 req/min limiter is a Go constant attached to the whole
/apiprefix, with no env var, flag or chart value. Nothing in this repo (render script, manifests, Gateway config) can change it, and the latest stable Console is still v0.3.42. No PR opened; this stays open until an upstream release fixes it, then bump the pins inscripts/render-homelab-manifests.py. Releasing my claim is left to the hive.🐝 Hive Agent:
scanner| Instance:hosted-projectbluefin-knuckle-gjvq| SHA:fcf0e78— hive: agent=scanner backend=copilot model=claude-opus-5.5 copilot=1.0.88
hivecommons-hive commentedon Oct 27, 2026
stage 020-claimed → 010-actionable: actionable
kylerankin commentedon Oct 27, 2026
Summary: The dashboard on this project's homelab build can still reject its own requests when it first opens, leaving the on-screen cards unable to refresh until the limit resets a minute later. The cause lives inside the dashboard software this project ships (KubeStellar Console), not in this project's own code. This project only decides which version of that dashboard to include, and the corrected version has not been published yet, so there is nothing in this project to change right now. It stays blocked until the dashboard project fixes the request limit and ships a stable release containing that fix.
scripts/render-homelab-manifests.py(lines 366 and 370) and at v0.3.34 infiles/k0s/manifests/kubestellar/40-kubestellar-console.yaml.v0.3.43-nightly.*pre-releases, which are not suitable as a production pin.mainbranch (commit 88bf46e) still hard-codes 120 requests per minute and attaches that limit to every/apirequest, including logged-in dashboard requests, with no setting, flag or environment variable to change it. This confirms the maintainer's root-cause investigation (bug(kiosk): console self-rate-limits on first boot (HTTP 429) causing card refresh failures #199, by jeefy).This is the external dependency identified in that investigation. Once a fixed stable dashboard release exists, update the pins in
scripts/render-homelab-manifests.py(and the k0s manifest) and verify repeated dashboard loads before closing #199. No server files changed and no PR was opened.— hive: backend=pi model=lemonade/ornith-1.5-35b-a3b-gguf-q6_k
🐝 Hive Agent:
contributor| SHA:3772407