Skip to content

Add streaming SQLite snapshot installation - #32

Open
findolor wants to merge 1 commit into
codex/extend-sqlite-wasm-rs-opfsfrom
codex/streaming-snapshot-poc
Open

findolor wants to merge 1 commit into
codex/extend-sqlite-wasm-rs-opfsfrom
codex/streaming-snapshot-poc

Conversation

@findolor

@findolor findolor commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Chained PRs

  1. Vendor sqlite-wasm-rs 0.3.0 #33 — vendor sqlite-wasm-rs 0.3.0.
  2. Harden and extend sqlite-wasm-rs OPFS #34 — harden and extend the WASM/OPFS layer.
  3. This PR: implement generic streaming snapshot installation and coordinated shutdown.

Motivation

Importing a large SQL dump in application code creates avoidable startup work, large transient allocations, and repeated application/worker crossings. Consumers need an application-agnostic way to install an authenticated SQLite snapshot safely across multiple tabs.

Implementation

  • Adds installSnapshot(url, compression, sha256, uncompressedSize) with raw and gzip streaming.
  • Streams response bytes through the database worker directly into an OPFS staging slot.
  • Verifies exact size, SHA-256, SQLite structure, and PRAGMA quick_check before activation.
  • Activates through staging/backup paths and restores the previous database on failure.
  • Coalesces identical installs while preserving independent waiters.
  • Coordinates leader/follower installs across tabs with explicit cancellation messages.
  • Adds asynchronous await db.close(), followed by db.free().

Multi-tab shutdown contract

Every close performs an explicit coordinator shutdown handshake:

  1. stop admitting new forwarded query, batch, and snapshot work;
  2. authoritatively reject new forwarded requests;
  3. drain or settle all already-accepted work;
  4. acknowledge shutdown;
  5. terminate workers and release OPFS ownership.

Snapshot cancellation is separately acknowledged and fenced immediately before activation. The caller never receives a cancellation/timeout failure followed by a hidden successful activation. Cancel-before-request ordering is tombstoned with bounded expiry, last-waiter cancellation does not absorb successor jobs, and leader close cannot strand follower-owned work.

Safety

  • Existing databases survive download, digest, validation, cancellation, timeout, and activation failures.
  • Corruption beyond the first SQLite page is caught before activation.
  • Size/path inputs are checked without imposing snapshot-only path limits on ordinary database opens.
  • Close/wipe races cannot resurrect a worker.
  • Shutdown owns no application schema, Raindex manifest, contract, or RPC logic.

Review-loop fixes

Codex review drove fixes for recovery ordering, stream cancellation, leader loss, install coalescing, safe integer/path bounds, terminal close semantics, follower detach, cancellation acknowledgement, finalization fences, bounded cancel tombstones, successor-job ordering, leader/follower close interleavings, and explicit coordinator draining. The teammate review-loop skill was removed after verification at the user's request.

Verification

  • sqlite-web-core Chrome WASM: 120/120
  • public sqlite-web Chrome WASM: 42/42
  • browser snapshot/shutdown integration: 13/13
  • workspace wasm32 check passed
  • Svelte check: 0 errors, 0 warnings
  • formatting and diff checks passed

The browser regression suite covers raw/gzip install, digest/corruption preservation, oversize cancellation, two-client coalescing, close/wipe races, zero-yield install+close, follower cancellation, finalization cancellation, leader close with coalesced/follower-only work, and OPFS reopen handoff.

Downstream

Raindex integration: rainlanguage/raindex#2872, #2873, and #2871. The SQLite Web package must be released and Raindex's dependency version bumped before standalone runtime use.

@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: a0782254-fb3e-46a0-a2fd-1b3f199e74be

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

findolor commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@findolor
findolor changed the base branch from main to graphite-base/32 September 13, 2026 13:09
@findolor
findolor force-pushed the codex/streaming-snapshot-poc branch from 519560c to 82e975b Compare September 13, 2026 13:09
@findolor
findolor changed the base branch from graphite-base/32 to codex/extend-sqlite-wasm-rs-opfs September 13, 2026 13:09
@findolor
findolor marked this pull request as ready for review September 13, 2026 13:09
@findolor
findolor force-pushed the codex/streaming-snapshot-poc branch from 82e975b to 2cb1419 Compare September 13, 2026 13:36
@findolor
findolor force-pushed the codex/extend-sqlite-wasm-rs-opfs branch from c92adf4 to 197c158 Compare September 13, 2026 13:36
@findolor

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@findolor
findolor force-pushed the codex/extend-sqlite-wasm-rs-opfs branch from 197c158 to 6c41f37 Compare September 16, 2026 12:13
@findolor
findolor force-pushed the codex/streaming-snapshot-poc branch from 2cb1419 to fa4d9d2 Compare September 16, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant