Skip to content

contextdb: add v2 library and shell with typed columns, transactions, and concurrent access - #20

Merged
lucidprogrammer merged 2 commits into
devfrom
v2-library-and-shell
Nov 10, 2026
Merged

lucidprogrammer merged 2 commits into
devfrom
v2-library-and-shell

Conversation

@lucidprogrammer

@lucidprogrammer lucidprogrammer commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Adds a new workspace under v2/, beside the current crates, which this change does not touch. Nothing that builds from crates/ changes; the root Cargo.toml and Cargo.lock are untouched.

What is in it

  • v2/contextdb: the library. Tables with typed columns, a primary key, NOT NULL, UNIQUE, DEFAULT and indexes; INSERT, UPDATE, DELETE and SELECT over one table; transactions; one process writing a store while any number of other processes read it; a memory limit that holds. A store is one folder. Statements this version does not run yet (joins, vector search, relationships, sync clauses and others) are refused and change nothing.
  • v2/contextdb-cli: the contextdb shell for a store, read-only by default, --write to write.
  • v2/redb: the storage library. The first commit is upstream redb 4.1.0, source, tests and examples, unchanged. The second commit holds the changes, listed in v2/redb/README.md: a file its writer holds can be opened for reading at the newest whole commit, pages a reader still needs are not reused, and a page's checksum is checked when it is read.
  • .github/workflows/check.yml: format, lint with warnings denied, tests on Linux and macOS, the storage library's own tests, a build on Rust 1.89, and two ceilings (shipped lines at most 150,000, tests at most 1,000). The existing workflow gains two paths-ignore lines so that a change touching only v2/ does not start it.

Commits

  • 2c9a26956 storage: add redb 4.1.0 upstream source with tests and examples. 71 files, upstream unchanged; it is built from the next commit on, which adds it to the v2 workspace.
  • bde61bdb7 contextdb: add v2 library and shell with typed columns, transactions, and concurrent access. 94 files.

Measured at this commit

By v2/numbers.sh: a release build on a 16-core AMD Ryzen 9 9955HX with 64 GB of memory and an NVMe drive, with nothing else running.

20,000 rows of 1.6 KB, one commit per row 17.2 s to insert, 60.3 MiB on disk
COUNT(*) of those rows 0.026 ms, no row read
Open that store after a clean close 0.6 ms
One million rows under a 64 MiB limit loaded in 11.9 s, scanned in 246 ms; peak 13.4 MiB allocated, 18.0 MiB resident
First build of a program that embeds the library 5 s, 46 packages, a 2.7 MiB program
Fresh clone to a green cargo test 2.1 min
Shipped lines; tests 7,369; 87 tests and 11 SQL script files

Verification

Known and stated in the README

  • After a writer is killed, readers open at once; the next writer first runs the storage library's repair, which reads every page in use (about half a second per GiB from a cold disk). An open after a clean close is under a millisecond.
  • The smallest memory limit that opens a store is 9 MiB.
  • A memory limit set inside a transaction takes effect at once and is not undone by a rollback.

@lucidprogrammer
lucidprogrammer merged commit 6b44713 into dev Nov 10, 2026
115 checks passed
@lucidprogrammer
lucidprogrammer deleted the v2-library-and-shell branch November 10, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant