Skip to content

Replace top-level metadata keys on SQLite merges - #1533

Merged
bgentry merged 2 commits into
masterfrom
sqlite-shallow-metadata-merge
Oct 24, 2026
Merged

bgentry merged 2 commits into
masterfrom
sqlite-shallow-metadata-merge

Conversation

@bgentry

@bgentry bgentry commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

SQLite metadata updates currently use JSON Merge Patch, which recursively merges nested objects and deletes keys set to JSON null. River's intended behavior is to preserve unrelated top-level keys and replace each supplied value completely, as Postgres's metadata || updates does. This difference can leave a stale resumable cursor on SQLite and cause a completed step to run again on a later retry.

All resumable cursors live inside the same top-level key, river:resumable_cursor. Each cell below shows the full metadata object: Postgres replaces that outer key's entire value, while the existing SQLite merge retains nested entries or interprets null as deletion.

Stored metadata Metadata updates Postgres result SQLite result before this fix
{"river:resumable_cursor":{"c":1}} {"river:resumable_cursor":{"d":7}} {"river:resumable_cursor":{"d":7}} {"river:resumable_cursor":{"c":1,"d":7}}
{"river:resumable_cursor":{"c":1}} {"river:resumable_cursor":{"d":null}} {"river:resumable_cursor":{"d":null}} {"river:resumable_cursor":{"c":1}}
{"river:resumable_cursor":{"c":1}} {"river:resumable_cursor":null} {"river:resumable_cursor":null} {}

For example, c saves a cursor and fails on attempt 1. On attempt 2, c completes, so the worker removes its cursor from the map; then d saves a cursor and fails. The update replaces the cursor map with {"d":7}. A recursive merge retains the old c entry, making c look partly done on attempt 3 and causing it to run again.

Go uses one shared metadata_merge.sql expression to apply each supplied top-level value with jsonb_set. The update list is materialized once, avoiding expression-depth limits and repeated work on Turso. The three generated queries insert this fixed expression through the existing template mechanism because sqlc cannot resolve its CTEs. Fixed SQL and parameter names keep the template replacer's cache enabled.

Java, JavaScript, Rust, and Ruby use fixed SQLite queries that rebuild metadata from retained keys and replacement values. Values are read with -> to preserve JSON types and numeric precision.

The change covers Go, Java, JavaScript, Rust, and Java's conformance helper. Ruby already replaces top-level values; its SQLite merge now also handles large updates without nesting one setter per key. Java's transactional checkpoint fallback keeps its precedence over stored metadata, with attempt updates taking precedence over both. Go keeps malformed stored metadata intact and uses top-level JSON paths without reconstructing existing key names.

Regression coverage includes completed cursor steps, running and stale completions, large updates, literal key names, empty objects, nulls, booleans, and numeric precision. Changelog entries cover every affected language. A separate commit records the cross-language parity expectations in AGENTS.md.

@bgentry
bgentry force-pushed the sqlite-shallow-metadata-merge branch 2 times, most recently from 796bf38 to 73d32f0 Compare October 24, 2026 21:33

@brandur brandur left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah wow, okay that seems like a bad one. Nice find!

Metadata updates replace whole top-level values on Postgres. SQLite's
JSON Merge Patch instead merges nested objects and deletes null values,
leaving stale resumable cursors that can rerun completed steps.

Restore top-level replacement in every affected port and preserve Java's
transactional checkpoint fallback. Keep JSON nulls and numeric value
tokens intact. Use fixed SQL so large updates avoid expression-depth
limits.

Share one SQL template across Go's updates and both completion paths.
Apply supplied values with `jsonb_set` without reconstructing existing
metadata or decoding its keys. Keep the template cache enabled and
materialize the update list once for Turso. Preserve malformed stored
metadata and Postgres behavior for non-object stored values.

Cover cursor retries, large updates, literal keys, empty objects, nulls,
and numeric precision in shared driver and port regression coverage.
Keep shared features and fixes in one pull request across the Go
reference implementation and every affected port. Require equivalent
coverage and documentation of intentional behavior differences.
@bgentry
bgentry force-pushed the sqlite-shallow-metadata-merge branch from 73d32f0 to 398a149 Compare October 24, 2026 21:42
@bgentry
bgentry marked this pull request as ready for review October 24, 2026 21:46
@bgentry
bgentry merged commit f1de3b0 into master Oct 24, 2026
68 checks passed
@bgentry
bgentry deleted the sqlite-shallow-metadata-merge branch October 24, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants