Skip to content

fix(tron): let balance Retry reach the network while the circuit breaker is open - #526

Draft
rvelaz wants to merge 1 commit into
mainfrom
fix/stale-balances-08-tron-retry-open-breaker
Draft

rvelaz wants to merge 1 commit into
mainfrom
fix/stale-balances-08-tron-retry-open-breaker

Conversation

@rvelaz

@rvelaz rvelaz commented Oct 26, 2026 •

Copy link
Copy Markdown

Draft. Manifest shasum not yet regenerated, so snap checks may fail.
Built and tested on 80b21f6; main is one commit ahead (#331, stellar only) and git merge-tree shows no conflicts.

  • Package: @metamask/tron-wallet-snap

Explanation

Bug

When TronGrid or the Tron provider has failed enough times to open the shared circuit breaker, the TRX row's Retry does nothing: the snap rejects the request locally, sends no request, and reports another error to Sentry on every click.

How to reproduce

  1. Build extension release/13.54.0 (acf000d) with yarn build:dev. For the "after" case, also build this branch and copy its dist/bundle.js, snap.manifest.json and preinstalled-snap.json into node_modules/@metamask/tron-wallet-snap. To see the wallet-side "Balance unavailable" + Retry, add core PR 10 and extension PR 12.
  2. Load it unpacked in Chrome or Chromium with default settings (HTTP/2 on). This is profile A.
  3. Import a test wallet that holds BTC, TRX, XLM and SOL (we used a team test wallet). Onboarding creates the Bitcoin, Tron, Stellar and Solana accounts.
  4. Let the snaps' requests coalesce. In Chrome, a request to tron-mainnet.infura.io can reuse the HTTP/2 connection already open to another *.infura.io host (in our runs the Solana snap usually opened it first). The reply is an empty 404 without CORS headers, so the snap sees Failed to fetch. We reproduced this on a test machine where every *.infura.io name resolved to the same IP. It depends on the network path and may not reproduce on every network. In our runs we moved the shared connection between hosts on purpose: we dropped the test browser's sockets while briefly blocking the other Infura hosts, which "heals" one chain and breaks the others.
  5. While tron requests fail, wait until the snap logs 'circuit breaker is open', then click Retry on the Tron row 10–20 s after tron's last failure, a second time 2 s later, and a third time about 13 s later.
  6. Control (profile B): the same steps with Chrome started with --disable-http2 give no coalescing, and all four balances are correct.

Expected: the first click sends tron requests at once; a click within 10 s reuses the result; a later click sends a new attempt. Sentry gets one 'Failed to fetch account info' per failure streak and none for the open circuit.
Actual (before this change): no tron request for clicks inside the breaker's open window, and one trackError per click.

Without a browser: keyring.openCircuit.test.ts drives getAccountBalances / getAccountAssets with the breaker open.

Summary

When TronGrid or the Tron provider has failed enough times to open the shared fetch-policy circuit breaker, the wallet's Retry button did nothing: the snap rejected the request locally and sent no request. This PR lets the wallet's own Retry requests (keyring getAccountAssets / getAccountBalances) bypass an open breaker. Each upstream request gets at most one direct attempt every 10 s, and concurrent callers share it. Background and cron reads still respect the breaker. It also stops repeat failures from flooding Sentry (snap_trackError).

Why

In Chrome, HTTP/2 connection coalescing across *.infura.io can send a snap's request over a connection opened for another Infura host. The reply is an empty 404 without CORS headers, so the snap sees Failed to fetch. Tron reads then fail in a burst and the circuit breaker opens. The wallet shows "Balance unavailable" + Retry, but:

  • core's SnapDataSource calls getAccountAssets before getAccountBalances, and both short-circuited with BrokenCircuitError, so Retry sent no request until the breaker cooled down (the EM A-v5 test: 0 requests for 5 clicks made 9–23 s after the last failure);
  • each failed fetch or Retry sent its own snap_trackError ('Failed to fetch account info'), and the open-circuit error itself was also tracked (A-v5: 34 + 1 for one outage).

What changed

  • utils/openCircuitBypass.ts (new): OpenCircuitBypass. When a guarded read throws BrokenCircuitError and the caller asked for bypassOpenCircuit, it makes one direct request with no retries, not counted by the policy. It is keyed by network + method + URL + body and limited to once every 10 s. Concurrent callers share the in-flight attempt, a result can be reused for up to 5 s, and a direct success resets the state.
  • TrongridApiClient, TronHttpClient: accept the bypass option on account info, TRC20, account resources and reward reads.
  • AccountsService, AssetsService, SnapAssetsAdapter, handlers/keyring/keyring.ts: keyring getAccountAssets/getAccountBalances pass { bypassOpenCircuit: true }. Cron and background paths do not.
  • utils/errors.ts: shouldTrack drops the open-circuit error, matched by message, because the keyring SDK rethrows it as a plain Error.
  • utils/failureStreakGate.ts (new): repeated sync failures such as 'Failed to fetch account info' are reported once per failure streak, keyed by message, with a reminder every 15 min. A success ends the streak.
  • Tests: keyring.openCircuit.test.ts (new; tests 1 and 3 fail without the fix), openCircuitBypass.test.ts, failureStreakGate.test.ts, plus updates to the client, service and error tests.

Validation

Checked locally: the test files changed in this PR were run with the parent's source (tests from this branch, every other file from the parent), then on this branch.

Unit tests that fail before and pass after (parent: 80b21f6)

  • TronHttpClient › open circuit bypass (v5) › lets an explicit request reach the network once while the circuit is open, and the same test in TrongridApiClient
  • KeyringHandler › getAccountBalances › fetches live balances for the account and returns the balance record; KeyringHandler › getAccountAssets › fetches live assets for all account scopes and returns their asset types
  • AssetsService › fetchAccountAssets › fetches live assets from the chain for all the account scopes
  • AccountsService › synchronize (assets) › does not report accounts short-circuited by an open circuit (v5)
  • keyring.openCircuit.test.ts, openCircuitBypass.test.ts and failureStreakGate.test.ts cannot run on the parent because their modules are new.
  • In total, 6 tests fail on the parent, and 134 pass, plus the 3 suites that cannot run.

Full run on this branch

  • jest --runInBand in packages/tron-wallet-snap: 64 suites, 958 tests pass.
  • tsc --noEmit (package): clean.

Manual A/B

A = Chrome with default HTTP/2 (reproduces the coalescing); B = Chrome with --disable-http2 (control, no coalescing). Every run used the same test wallet and a local yarn build:dev of extension 13.54.0 (acf000d), with the snaps and assets-controller wired into node_modules by hand. "Unfixed 13.54.0" is that build with no fixes. The fixes-vN builds are cumulative: each one has every fix up to that pass, so an "after" number shows this PR together with the others in that build. Run names (A-v6, B-v7, ...) are our local test passes on those builds; times are CEST.

Check Before After
A: Retry with the breaker open fixes-v3 build, A-v3: the Retry click sent no tron request (trackError without a request). fixes-v5 build (an earlier, incomplete version of this fix), A-v5: 5 clicks 9–23 s after the failure → 0 tron requests fixes-v6 build, A-v6: click 18.7 s after the failure → /v1/accounts, getaccountresource, getReward within 0.1 s; +2 s reused; +13 s new requests. fixes-v7, A-v7: the same (request in the same second at 18 s)
A: tron trackErrors A-v5: 34 'Failed to fetch account info' + 1 'circuit breaker is open' A-v6: 2 (one per failure streak), 0 open-circuit; A-v7: 3 for 3 streaks, 0 open-circuit
A: TRX after a heal + Retry — A-v6: a 200 in the same second as the click; TRX 24.109793 22 s later
B: tron trackErrors (control) B-v5: 0 B-v6, B-v7: 0; TRX correct at the first dump

Risks

  • The bypass sends up to one extra request per endpoint every 10 s while the breaker is open, but only on user-triggered keyring reads. The 5-second request cache from feat(tron): deduplicate identical requests within 5 seconds #500 still applies.
  • Matching the open-circuit error by message is brittle if the message text in @metamask/snap-networks-utils changes. It is covered by a test.
  • Failure streaks are tracked in memory, so a snap restart starts a new streak and can send one more report.

Changelog (packages/tron-wallet-snap/CHANGELOG.md, Unreleased → Fixed)

  • Let explicit balance and asset requests (the wallet's Retry) reach TronGrid and the Tron provider while the circuit breaker is open, at most once every 10 s per request, and report repeated sync failures to Sentry once per failure streak instead of on every attempt (#526)

References

Dependencies

Stale non-EVM balances series (Chrome HTTP/2 coalescing across *.infura.io):

Checklist

  • I've updated the test suite for new or updated code as appropriate
  • I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
  • I've communicated my changes to consumers by updating changelogs for packages I've changed
  • I've introduced breaking changes in this PR and have prepared draft pull requests for clients and consumer packages to resolve them

…ker is open and dedupe failure reports

Explicit keyring balance and asset requests (the wallet's Retry) bypass an open circuit breaker with one direct attempt per upstream request at most every 10 s, shared by concurrent callers. Background reads never bypass. Open-circuit short-circuits are no longer sent to trackError, and 'Failed to fetch account info' is reported once per failure streak.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant