Skip to content

perf(bitcoin): scan new accounts with a gap of 5 and deepen only if used - #522

Draft
rvelaz wants to merge 1 commit into
mainfrom
fix/stale-balances-04-btc-startup-cost
Draft

rvelaz wants to merge 1 commit into
mainfrom
fix/stale-balances-04-btc-startup-cost

Conversation

@rvelaz

@rvelaz rvelaz commented Oct 26, 2026 •

Copy link
Copy Markdown

Update (28 Oct 2026): Rebased onto main on its own. It no longer includes #519, #520 or #521. #520 (Bitcoin request retries and timeouts) was closed because Julien's Bitcoin HTTP retry-policy work covers it, and that work is already on main as #530. The address-history gap check now goes through #530's per-network fetchWithPolicy (retries, circuit breaker, default per-attempt timeout) instead of #520's REST policy, and the 10 s GAP_CHECK_REQUEST_TIMEOUT_MS is removed.
On main, the gap-5 first scan applies in AccountUseCases.fullScan, which the FullScanAccount event runs for new accounts. The PR adds the small hasSynchronized() helper itself (the same code as in #519) to tell a first scan from a rescan. The follow-up scan for a reused discovery scan moved to #521, which adds that reuse. The B-v3 onboarding test needs #521, so it was dropped here.
It merges independently of #521. Whichever of #521 and #522 merges second needs a small rebase in AccountUseCases.fullScan (on top of #521 the discovery-gap first scan moves into its #runFullScan). No conflict with #519.
Draft. Manifest shasum not yet regenerated, so snap checks may fail.

  • Package: @metamask/bitcoin-wallet-snap

Explanation

Bug

Each new BTC account gets a full gap-20 first scan (on main, and also with PR 03). With the 5 accounts that onboarding creates, that is 257 bitcoin requests in the first 2 minutes, a burst that adds load on Infura and makes coalescing failures more likely. The unfixed build made about 40.

How to reproduce

  1. Build extension release/13.54.0 (acf000d) with yarn build:dev. For the "after" case, also build this branch and copy its dist/bundle.js, snap.manifest.json and preinstalled-snap.json into node_modules/@metamask/bitcoin-wallet-snap. To see the wallet-side "Balance unavailable" + Retry, add core PR 10 and extension PR 12.
  2. Load it unpacked in Chrome or Chromium with default settings (HTTP/2 on). This is profile A.
  3. Import a test wallet that holds BTC, TRX, XLM and SOL (we used a team test wallet). Onboarding creates the Bitcoin, Tron, Stellar and Solana accounts.
  4. Let the snaps' requests coalesce. In Chrome, a request to bitcoin-mainnet.infura.io can reuse the HTTP/2 connection already open to another *.infura.io host (in our runs the Solana snap usually opened it first). The reply is an empty 404 without CORS headers, so the snap sees Failed to fetch. We reproduced this on a test machine where every *.infura.io name resolved to the same IP. It depends on the network path and may not reproduce on every network. In our runs we moved the shared connection between hosts on purpose: we dropped the test browser's sockets while briefly blocking the other Infura hosts, which "heals" one chain and breaks the others.
  5. For this PR use profile B (--disable-http2), so that every request succeeds, and count the requests to bitcoin-mainnet.infura.io (and the scripthash lookups) in the first 2 minutes after onboarding.
  6. Control (profile B): the same steps with Chrome started with --disable-http2 give no coalescing, and all four balances are correct.

Expected: about 137 requests (116 scripthash), with all balances correct.
Actual (before this change): 257 requests (236 scripthash) on the fixes-v3 build (PRs 01–03, gap 20).

Without a browser: AccountUseCases.test.ts › fullScan › completes the initial scan to the standard gap when the discovery gap finds activity and the EsploraClientAdapter › extendDiscoveryScan tests (e.g. checks only the addresses the standard gap adds and skips the full scan when they are unused (run B-v4 account 0)).

Summary

The first full scan of a new account uses the discovery stop gap (5) instead of the standard gap (20). When that scan finds activity, the snap checks only the addresses the standard gap would add, and runs the gap-20 scan only if one of them has history or the check fails. Bitcoin requests in the first 2 minutes after onboarding went from 257 to 137.

Why

In Chrome, HTTP/2 connection coalescing across *.infura.io can send the snap's Esplora request over a connection opened for another Infura host. The provider router answers with an empty 404 and no CORS headers, which shows up as Failed to fetch. Every extra request at startup is another chance to hit that, and it also loads the shared Infura Esplora endpoint. On a restored SRP, the account-tree sync creates account groups 1..4 before discovery runs. No discovery-scan reuse applies to them, so each got a gap-20 initial scan: about 42 requests for an empty account, where gap 5 needs about 10.

What changed

  • use-cases/AccountUseCases.ts: the first full scan of an account (one that never synchronized) runs in 'discovery' mode. If the result shows activity, it calls chain.extendDiscoveryScan(account) instead of a full gap-20 rescan. Rescans keep the standard gap.
  • entities/chain.ts, infra/EsploraClientAdapter.ts: new extendDiscoveryScan.
    • It derives the addresses between the discovery gap and the standard gap on both keychains, past the last used index.
    • It checks each with /scripthash/:hash/txs through the network's shared fetch policy (fetchWithPolicy from feat(bitcoin): use shared fetch policy helpers #530: retries, circuit breaker and the default per-attempt timeout).
    • It runs the gap-20 full scan only if one has history or any check fails.
    • toScripthash() computes the Esplora script hash.
  • entities/account.ts, infra/BdkAccountAdapter.ts, infra/StoredAccountAdapter.ts: Keychain type, peekKeychainAddress, lastUsedIndex and lastRevealedIndex. The metadata-only stored view throws "unsupported". Also hasSynchronized() (same code as PR 01).
  • Tests: AccountUseCases (gap-5 first scan, extension only on activity), EsploraClientAdapter (toScripthash, the gap check, the fallback to the full scan on history or failure), BdkAccountAdapter, StoredAccountAdapter.

Validation

Run on a Cursor cloud VM against main at c2e8d19, with this PR's commit (c872652) applied with git am:

  • ESLint: pass.
  • tsc --noEmit: pass.
  • Build: pass. The only difference is the expected snap.manifest.json shasum mismatch (not regenerated yet).
  • jest in packages/bitcoin-wallet-snap: 27 suites, 697 tests, 0 failures.
  • yarn workspace @metamask/bitcoin-wallet-snap run test (with coverage): pass.

The earlier fail-before list (against PR 03) is no longer current: this PR is now based on main.

Manual A/B

A = Chrome with default HTTP/2 (reproduces the coalescing); B = Chrome with --disable-http2 (control, no coalescing). Every run used the same test wallet and a local yarn build:dev of extension 13.54.0 (acf000d), with the snaps and assets-controller wired into node_modules by hand. "Unfixed 13.54.0" is that build with no fixes. The fixes-vN builds are cumulative: each one has every fix up to that pass, so an "after" number shows this PR together with the others in that build. Run names (A-v6, B-v7, ...) are our local test passes on those builds; times are CEST.

All numbers are B runs: bitcoin requests / scripthash lookups in the first 2 minutes after onboarding.

Build Requests / scripthash Source
Unfixed 13.54.0 ~40 / ~27 as quoted in the v2 notes (old build)
fixes-v3 (PRs 01–03, gap 20) 257 / 236 B-v3
fixes-v4 (gap 5, then a full gap-20 rescan on activity) 158 / 136 B-v4
fixes-v5 to v7 (this PR: gap 5 + gap check) 137 / 116 B-v5, B-v6, B-v7

The first BTC balance was correct at the first dump (≤ 9 s) in B-v4 to B-v7. The load is still above the unfixed build's ~40, because the old build did not run a full first scan per account. That is the trade-off for a reliable first balance.

Risks

  • Coverage: an account with no history in the first 5 addresses of either keychain is treated as empty, the same assumption account discovery already makes with gap 5. Once activity is found, the check covers each keychain from (last used + 6) to (max(last used, last revealed) + 20), so the account ends with the same gap-20 coverage. Any history or check failure runs the full gap-20 scan.
  • The gap check adds one REST request per checked address on used accounts. That is still fewer than a full rescan, which re-fetches every index already scanned.

Changelog (packages/bitcoin-wallet-snap/CHANGELOG.md, Unreleased → Changed)

  • Scan new accounts with the discovery gap (5) and check the standard-gap addresses only when the account has activity, instead of a gap-20 initial scan, roughly halving the requests at startup (#522)

References

Dependencies

Stale non-EVM balances series (Chrome HTTP/2 coalescing across *.infura.io):

Checklist

  • I've updated the test suite for new or updated code as appropriate
  • I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
  • I've communicated my changes to consumers by updating changelogs for packages I've changed
  • I've introduced breaking changes in this PR and have prepared draft pull requests for clients and consumer packages to resolve them

The first full scan of a new account uses the discovery stop gap (5)
instead of the standard gap (20). When that scan finds activity,
extendDiscoveryScan checks only the addresses the standard gap adds (one
Esplora history request each, through the network's shared fetch policy)
and runs the gap-20 full scan only if one of them has history or the
check fails, so coverage is unchanged. BitcoinAccount.hasSynchronized()
tells a first scan (genesis checkpoint) from a rescan. In the A/B runs
this cut the first-2-minute bitcoin requests from 257 to 137.
@rvelaz
rvelaz force-pushed the fix/stale-balances-04-btc-startup-cost branch from 1a8ef47 to c872652 Compare October 28, 2026 10:30
@rvelaz
rvelaz changed the base branch from fix/stale-balances-03-btc-scan-reliability to main October 28, 2026 10:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant