Skip to content

fix: registration no longer drops accounts on transient network errors - #1

Merged
B3hnamR merged 1 commit into
B3hnamR:masterfrom
amirshaker000:fix/registration-network-resilience
Oct 27, 2026
Merged

B3hnamR merged 1 commit into
B3hnamR:masterfrom
amirshaker000:fix/registration-network-resilience

Conversation

@amirshaker000

Copy link
Copy Markdown
Contributor

Problem

Generated accounts were silently never added to the pool. Worker logs always ended with \1 fully created, but the gateway log showed the real story:

\
ERROR zcode_gateway.dash: generator: pool registration failed for
requests.exceptions.SSLError: ... UNEXPECTED_EOF_WHILE_READING
\\

Two stacked issues:

  1. Silent system-proxy inheritance. On Windows,
    equests\ with \proxies=None\ still inherits the IE/registry proxy (e.g. a VPN client on \127.0.0.1:10808) via \ rust_env\ — even when the generator config says \ip_source: direct. Every device-flow call rode that tunnel, which drops TLS connections at random.
  2. One flake killed the whole registration. \get_zcode_jwt/\get_plan\ had no per-request retry or exception handling. A single escaping \RequestException\ jumped straight past
    egister_account's outer 3-attempt loop, dropped the account, and left only a log line. (The catch-up scan only runs at startup, so nothing rescued it.)

Fix

*\generator/plan_check.py*

  • Shared
    equests.Session\ with \ rust_env = False: direct actually means direct; keep-alive across the init/authorize/callback/poll chain. \ZAI_PROXY\ still wins when explicitly set.
  • New _request()\ helper: per-request retries (3x, capped backoff) on transient \RequestException, used by every device-flow / billing / claim / telemetry call.
  • \get_plan\ / \get_current\ / \preview_plans\ return \None/[]\ on exhausted retries instead of raising.
  • Test-shim compatibility: when tests monkeypatch \plan_check.requests.get/post, _request\ dispatches through the patched functions (canned envelopes keep working, no live calls).

*\zcode_gateway/api_dash.py*


  • egister_account\ now catches per-attempt exceptions inside both retry loops, so a single network flake costs one attempt instead of the whole registration.

Evidence

  • Targeted suites green: \ est_claim_start, \ est_generator, \ est_accounts_dir, \ est_login_resilience\ — 55 passed.
  • Full suite: identical results vs pristine \master\ on the same machine (62 failed / 289 passed both trees; the 62 are env-dependent, e.g. missing optional deps / live-socket guards).
  • Live smoke test (real account token): a mid-flow \ConnectionError\ on /oauth/cli/poll\ was retried and recovered — JWT minted, plan read ['zcode-v3-start-plan-nt-1026', 'zcode-v3-start-plan-0817']. On master that same flake dropped the account.

Notes

  • No behavior change for proxied setups (\ZAI_PROXY\ set): retries and session pooling apply the same way.
  • The _MEI\ PyInstaller copy is unaffected — this only changes source; a rebuild picks it up.

Account registration died wholesale on flaky edges:

- requests silently inherited the Windows registry proxy (VPN client on
  127.0.0.1) even with ip_source=direct, so every 'direct' device-flow
  call rode a tunnel that drops TLS at random (SSLEOFError / ProxyError)
- get_zcode_jwt/get_plan had no per-request retry, so ONE escaping
  RequestException jumped past register_account's outer 3-attempt loop
  and dropped the account with only a log line

generator/plan_check.py:
- shared Session with trust_env=False: direct actually means direct;
  keep-alive across the init/authorize/callback/poll chain
- _request() helper: per-request retries (3x, capped backoff) on
  transient RequestException for every device-flow/billing call
- get_plan/get_current/preview_plans now return None/[] instead of
  raising on exhausted retries

zcode_gateway/api_dash.py:
- register_account catches per-attempt exceptions inside both retry
  loops so a single flake costs one attempt, not the whole registration

Tests: test_claim_start / test_generator / test_accounts_dir /
test_login_resilience all green (55 passed); full suite unchanged vs
master. Live-verified: a mid-flow ConnectionError now retries and the
account still mints + reads its plan.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 issues found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="zcode_gateway/api_dash.py">

<violation number="1" location="zcode_gateway/api_dash.py:882">
P3: The new catch-all handler logs only the exception type, so non-network bugs inside `get_zcode_jwt` lose the message and traceback the outer `log.exception` used to record. Include the message here, and do the same in the plan read handler below. The attempt also logs a second 'no zcodeJWT' warning right after this one.</violation>
</file>

<file name="generator/plan_check.py">

<violation number="1" location="generator/plan_check.py:118">
P2: Retrying POSTs on any RequestException can resend state-changing requests after the server already handled them. Limit the retry loop to connect-level failures for non-idempotent calls, or pass a `retry_post=False` flag for claim and event POSTs.</violation>

<violation number="2" location="generator/plan_check.py:118">
P2: `_request` does not retry the documented `502-with-empty-body` response because it returns every HTTP response without checking its status. Retry transient 5xx statuses such as 502/503/504 before returning the response.</violation>

<violation number="3" location="generator/plan_check.py:119">
P2: `get_plan` now swallows balance-request failures and returns None. `main.py` relies on an escaping exception to retry the plan check direct, so a proxy failure during the balance read now records the account as not starter with no retry. Let the exception propagate, or keep the direct-retry path.</violation>

<violation number="4" location="generator/plan_check.py:262">
P2: One burst of three failed polls raises out of `get_zcode_jwt` and discards the flow. Catch `requests.RequestException` in the poll loop, log it, and `continue` so the remaining polls can still succeed.</violation>
</file>

Reply to a comment to ask cubic a question or push back. It learns from your replies.

View guided diff | Re-trigger cubic

Comment thread generator/plan_check.py
last = None
for i in range(1, tries + 1):
try:
return _session.request(method, url, proxies=_proxies(), **kw)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Retrying POSTs on any RequestException can resend state-changing requests after the server already handled them. Limit the retry loop to connect-level failures for non-idempotent calls, or pass a retry_post=False flag for claim and event POSTs.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At generator/plan_check.py, line 118:

<comment>Retrying POSTs on any RequestException can resend state-changing requests after the server already handled them. Limit the retry loop to connect-level failures for non-idempotent calls, or pass a `retry_post=False` flag for claim and event POSTs.</comment>

<file context>
@@ -84,6 +84,47 @@ def _proxies():
+    last = None
+    for i in range(1, tries + 1):
+        try:
+            return _session.request(method, url, proxies=_proxies(), **kw)
+        except requests.RequestException as e:
+            last = e
</file context>

Comment thread generator/plan_check.py
for _ in range(15):
r = requests.get(POLL + flow_id, headers=build_headers(poll_token, include_api_key=False),
timeout=30, verify=_VERIFY, proxies=_proxies())
r = _request("GET", POLL + flow_id, headers=build_headers(poll_token, include_api_key=False),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: One burst of three failed polls raises out of get_zcode_jwt and discards the flow. Catch requests.RequestException in the poll loop, log it, and continue so the remaining polls can still succeed.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At generator/plan_check.py, line 262:

<comment>One burst of three failed polls raises out of `get_zcode_jwt` and discards the flow. Catch `requests.RequestException` in the poll loop, log it, and `continue` so the remaining polls can still succeed.</comment>

<file context>
@@ -213,11 +255,12 @@ def get_zcode_jwt(chat_token, log=print):
     for _ in range(15):
-        r = requests.get(POLL + flow_id, headers=build_headers(poll_token, include_api_key=False),
-                         timeout=30, verify=_VERIFY, proxies=_proxies())
+        r = _request("GET", POLL + flow_id, headers=build_headers(poll_token, include_api_key=False),
+                     timeout=30, verify=_VERIFY, log=log)
         try:
</file context>

Comment thread generator/plan_check.py
for i in range(1, tries + 1):
try:
return _session.request(method, url, proxies=_proxies(), **kw)
except requests.RequestException as e:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: get_plan now swallows balance-request failures and returns None. main.py relies on an escaping exception to retry the plan check direct, so a proxy failure during the balance read now records the account as not starter with no retry. Let the exception propagate, or keep the direct-retry path.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At generator/plan_check.py, line 119:

<comment>`get_plan` now swallows balance-request failures and returns None. `main.py` relies on an escaping exception to retry the plan check direct, so a proxy failure during the balance read now records the account as not starter with no retry. Let the exception propagate, or keep the direct-retry path.</comment>

<file context>
@@ -84,6 +84,47 @@ def _proxies():
+    for i in range(1, tries + 1):
+        try:
+            return _session.request(method, url, proxies=_proxies(), **kw)
+        except requests.RequestException as e:
+            last = e
+            if i < tries:
</file context>

Comment thread generator/plan_check.py
last = None
for i in range(1, tries + 1):
try:
return _session.request(method, url, proxies=_proxies(), **kw)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: _request does not retry the documented 502-with-empty-body response because it returns every HTTP response without checking its status. Retry transient 5xx statuses such as 502/503/504 before returning the response.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At generator/plan_check.py, line 118:

<comment>`_request` does not retry the documented `502-with-empty-body` response because it returns every HTTP response without checking its status. Retry transient 5xx statuses such as 502/503/504 before returning the response.</comment>

<file context>
@@ -84,6 +84,47 @@ def _proxies():
+    last = None
+    for i in range(1, tries + 1):
+        try:
+            return _session.request(method, url, proxies=_proxies(), **kw)
+        except requests.RequestException as e:
+            last = e
</file context>
Suggested change
return _session.request(method, url, proxies=_proxies(), **kw)
response = _session.request(method, url, proxies=_proxies(), **kw)
if response.status_code in (502, 503, 504) and i < tries:
log(f" plan: {method.lower()} {short} failed "
f"(HTTP {response.status_code}), retry {i}/{tries - 1}")
time.sleep(min(backoff * i, 6))
continue
return response

Comment thread zcode_gateway/api_dash.py
Comment on lines +882 to +884
log.warning("generator: zcodeJWT mint error for %s "
"(attempt %d/3): %s", email, attempt,
type(exc).__name__)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new catch-all handler logs only the exception type, so non-network bugs inside get_zcode_jwt lose the message and traceback the outer log.exception used to record. Include the message here, and do the same in the plan read handler below. The attempt also logs a second 'no zcodeJWT' warning right after this one.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At zcode_gateway/api_dash.py, line 882:

<comment>The new catch-all handler logs only the exception type, so non-network bugs inside `get_zcode_jwt` lose the message and traceback the outer `log.exception` used to record. Include the message here, and do the same in the plan read handler below. The attempt also logs a second 'no zcodeJWT' warning right after this one.</comment>

<file context>
@@ -873,7 +873,16 @@ def register_account(info: dict):
+                        # transient network failure: cost ONE attempt, not all
+                        # three (an escaping exception used to jump straight
+                        # to the outer handler and drop the account).
+                        log.warning("generator: zcodeJWT mint error for %s "
+                                    "(attempt %d/3): %s", email, attempt,
+                                    type(exc).__name__)
</file context>
Suggested change
log.warning("generator: zcodeJWT mint error for %s "
"(attempt %d/3): %s", email, attempt,
type(exc).__name__)
log.warning("generator: zcodeJWT mint error for %s "
"(attempt %d/3): %s: %s", email, attempt,
type(exc).__name__, exc)

@B3hnamR
B3hnamR merged commit 147ea99 into B3hnamR:master Oct 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants