Repository navigation
feat: Account for model memory and retain worker launch ownership - #835
Draft
buildingjoshbetter wants to merge 176 commits into
Draft
buildingjoshbetter wants to merge 176 commits into
buildingjoshbetter wants to merge 176 commits into
Conversation
buildingjoshbetter
changed the title
The Mac native acceptance harness and Windows ownership diagnostics are private validation tooling with workstation paths. They do not test package code, and the kebab-case collection added here ran them on Windows CI. Derive the TEXT affinity expectation from stored values so SQLite 3.48+ round-trip float text is classified correctly.
Header reads no longer charge the RSS budget, and a refusal while admitting a connection rejects that connection instead of escaping the accept loop and exiting the server. Error and busy replies are not charged, so clients receive server_busy rather than a dropped connection that triggers another server launch. Model construction is admitted with the known model size, models refused after loading are collected immediately, and an MPS OOM while building a model degrades that model kind to CPU like an inference OOM. MPS cache flushes now collect dropped models first. Small hosts keep room for the Base and Pro models (up to half of memory, at most 3 GiB), and single-text requests wait briefly for state bookkeeping before building a CPU clone.
The MCP backlog drainer still spawned the non-automatic ingest CLI on the live transcript and left the .processing claim for the CLI to clear. Completion removed that clearing, so cleanup_stale_processing restored every claim after 30 minutes and the drainer re-ingested the same sessions forever. Each cycle also rewrote the claim, so the 30-day backlog prune never removed it, and the loop slowly consumed the shared extraction budget that completion dispatch depends on. adopt_legacy_pointer admits a pointer's current transcript bytes as an immutable capture and then retires the pointer under one legacy lock, so SessionStart conversion and every MCP drainer process each pointer exactly once. Completion owns attempts, the budget and the spawn cap from there, which keeps the #422 guarantee: a worker that exits non-zero is retried from the retained capture rather than from a claim. The MCP drainer uses the pointer's own validated scope and host, keeps the M-90 allowlist through backlog_transcript_source, quarantines invalid markers and stops on completion backpressure. A pointer whose transcript was removed retires without admission. The #422 MCP drainer test is replaced by tests for the new contract, and the M-50 spawn-site check follows the spawn into completion.dispatch.
Every SessionEnd kept a full immutable copy of the transcript and nothing ever deleted one. After 64 MiB or 1024 captures every later Claude session was refused, and the stop hook returned without a spawn, a marker or a visible reason. Four 15.7 MB sessions were enough. Admission now collects under state.lock before the quota check. Bytes of receipted, exhausted and superseded captures are released while their receipts and outcomes stay, a lost final-attempt worker is settled, the oldest released entries are forgotten to keep room for one more capture and to bound state.json, and orphan bytes from an interrupted admission are removed. State is saved before any unlink, so a crash leaves orphans for a later pass, never a live entry without its bytes. A queued or retrying capture of the same native source is superseded only when the incoming bytes begin with its exact bytes, and only in the same save that admits the replacement. Bytes of a forgotten receipted capture are not copied again. The stop hook checks MIN_MESSAGES before admitting, so short sessions retain nothing, and dispatch settles a short capture from any other path instead of keeping it queued forever. A backpressure refusal now logs its reason and queues a legacy backlog pointer, which is adopted once completion has room.
A capture whose coverage could never be proven was re-extracted in full five times: heuristic extraction without an LLM provider, parser outcomes such as image, document or redacted_thinking blocks, and chunk-limited transcripts. With claude_cli that is up to 100 provider calls for one long session, with no possible change in the outcome. Parser outcomes are a function of the immutable capture, and heuristic, unobserved and chunk-limited extraction repeat for the same bytes and configuration, so these outcomes are recorded once as exhausted and the capture is released at the next admission. Provider, storage and unreadable outcomes still retry with backoff.
The status guard compared normalized SQL text and expected `pressure REAL`, while create_db (and 0.7.6.1) declare `pressure REAL DEFAULT 0`. Every edge to Base/Pro switch on a real database therefore refused with "Unsupported rebuild status schema" and status reads returned unknown. The guard now compares PRAGMA table_xinfo names, types, NOT NULL, primary-key and hidden flags against the reference declaration, and still refuses virtual, WITHOUT ROWID or separately indexed primary keys and any status trigger. A new subprocess regression test runs the real switch preparation and activation on a create_db database.
Base and Pro keep 1.67 GiB of weights on MPS and one long batch needs about 1.5 GiB more (measured on Apple silicon), so the 1.5-2.5 GiB caps sent the first long memory to CPU for the server lifetime. The automatic MPS cap is now at least 3.25 GiB. PyTorch has no fast bfloat16 CPU path on Apple silicon: Qwen3-Embedding took 709 ms per query in bfloat16 and 83 ms in float32. CPU-resident half-precision models on macOS, including OOM recovery and the fast-lane encoder, now run in float32.
Readiness waiting censused the workload's slice while the workload was still gating itself with short-lived systemctl helpers in its own cgroup, so a launch could fail after the barrier was released and leave a running server with no owner. The wait now watches the pidfd and polls only for the ready receipt (tolerating a receipt that is still being written), then proves the domain once after the helpers are reaped. A launch that fails after a child exists is no longer dropped. An owned workload is terminated only through its retained pidfd, retried briefly because the same helpers can make one census transiently non-exact; a workload that cannot be terminated stays owned and watched. A start that fails before returning a record closes its barrier so the waiting child exits, then retires that generation's units. Units used to be stopped only when the launcher outlived the server, and failed transient services stayed loaded and marked the user manager degraded. The service now runs with --collect, an owned stop clears any failed state, an exited workload whose terminal evidence was collected is retired from the empty-domain proof alone, a joined client retires units of a daemon whose launcher is gone, and every launch sweeps orphaned generations and prunes retired evidence beyond the newest eight. Units are stopped only when the manager reports no main process, the cgroup is the generation's own path and the recursive census is empty. The console launcher retires its claim on failure and keeps owning a workload it could not terminate until that workload exits.
After an unclean capped-server exit, including an OOM kill by its own cap, model.sock stayed on disk and every capped request raised because the claimed daemon was dead, so the relaunch path was never reached. A claim that is absent, pending or names an incarnation that no longer exists now reports no live daemon: the stale endpoint is never contacted and the launch path replaces it once it holds the bind lock. A live claimed daemon that fails inspection is still refused. A failed capped launch kept its claim, so every client treated the live launcher as a launch in progress and waited out its timeout until that process exited. The claim is now removed under the start lock as on the uncapped path, and a launched workload is terminated or kept watched. MCP startup with TRUEMEMORY_REQUIRE_PROCESS_MEMORY_CAP set crashed on an invalid flag value or unavailable enforcement because only connection errors were caught. Those errors are now logged and the server keeps serving; capped requests still fail closed and the flag value is still rejected explicitly. Uncapped startup behavior is unchanged.
A rebuild held maintenance ownership for its whole run (up to 9000 s), so every fresh open (the MCP server after configure, hooks, the CLI) failed with MaintenanceBusyError. Any commit from another connection changed PRAGMA data_version and aborted the job, and exceptions left the job marker selected and its status row running, so every retry was refused until a manual cancel-rebuild with exact IDs. Unusable inactive pairs (0.7.6.1 switch-back caches, previously served pairs, partial pairs after a forget or a schema change) could never be reset, so switching back to a group was permanently impossible. Workers now hold a per-database tier-job lock instead of maintenance and take maintenance only around the activation commit, which fresh opens wait for (bounded) through an activation marker. The source fence no longer treats foreign commits as invalidation: appended rows are embedded as a delta of the same generation, while corrections and schema changes discard the inactive pair and restart it (bounded). Unusable inactive pairs are discarded and recreated before a rebuild; serving pairs are refused, and the group's registry row is removed so an older binary rebuilds instead of trusting a stale cache. Exceptions end the job and terminate its status row on a fresh connection; a selected marker found under a free tier-job lock is ended as abandoned, cancel accepts unbound markers by exact job id, and status reports interrupted when no process holds the lock. The target reranker is proven before the durable commit, and a projection fault after it restores the exact previous journal, registry, marker and config so the previous selection keeps serving. Cold legacy initialization attaches to an already-stable pair instead of failing while another maintenance owner runs. Tests cover live fresh opens and foreign commits during a rebuild, corrections mid-run, SIGKILL recovery, unbound marker cancellation, switch-back caches, round trips and partial pairs, on real create_db databases in isolated interpreters.
The hooks counted human turns by parsing the whole transcript, and the completion worker read, decoded and kept every record. A 600 MiB Claude transcript peaked at 3.2 GiB in the worker parse, and every SessionEnd, UserPromptSubmit and PreCompact hook paid a full parse. Record handling moves into one accumulator shared by whole-text and streamed reads. With compact parsing, JSONL files are read in 1 MiB chunks, split on newline bytes and hashed as read; tool and system payloads are emptied and assistant text keeps its first 501 stripped characters, which format_for_extraction renders identically, so extraction input and provenance indices are unchanged. The worker peak for the same 600 MiB transcript is 79 MiB. has_human_messages stops reading once the threshold is met, so the hook cost no longer grows with the transcript. parse_transcript_outcome also accepts a byte limit. A prefix read of a growing append-only file is stable when the same file still holds at least those bytes; the caller proves their content by hash.
Completion refused any transcript over 16 MiB with capture_limit, and the stop hook then dropped the session. Real Claude transcripts reach hundreds of MiB, so long sessions that 0.7.6.1 ingested were lost. Admission now reads the source once in 1 MiB chunks. Sources up to 16 MiB are still copied, streamed through a private partial file in incoming/ that is renamed into place. Sources up to 2 GiB are referenced instead of copied: the capture records the file identity, byte count and SHA-256 of the observed prefix, appends during the read are allowed, and the worker parses exactly that prefix and proves it by the same hash, so a receipt still certifies exact bytes. Retained disk no longer grows with transcript size and the 64 MiB account quota only covers copies. A growing session supersedes its queued prefix using digests taken during the same pass. Dispatch checks a referenced source by identity and counts turns with an early-exit reader, so the hook never rehashes it; a rewritten, truncated or replaced source settles the capture with a source_prefix_changed outcome instead of retrying. Settled copies are released when their receipt or terminal outcome is recorded rather than at the next admission. Retained evidence is opened with O_BINARY on Windows. Tradeoff: a referenced prefix depends on the transcript staying in place and append-only until its worker runs; if it is removed or rewritten the capture settles without ingestion, and a newer SessionEnd capture of the same file covers what remains.
Root cause: the MCP server opens its Memory at startup only in local
mode, inside _preload_models (mcp_server.py, `if not
use_model_server()`). In the default shared mode _get_memory() runs only
from tool handlers, so a session with no TrueMemory tool call never
opened the store: this long-lived process requested no maintenance pass,
seeded no legacy vector gap and ran no NaN re-embed. Review finding V2-1.
Fix: after ensure_server_running() in shared mode, main() calls
_open_store_for_maintenance(), which runs _get_memory()._engine.
_ensure_connection() on a daemon thread ("truememory-store-open"). That
admission uses models="none", so the open is SQL only. Its first
maintenance wake requests the pass and starts the gap seed. A failure is
logged at debug level, and the first tool call opens the store as
before. Local mode is unchanged.
Evidence:
- tests/test-mcp-startup-store-open.py:
- The startup open, with get_model, get_reranker and model-server
requests made to fail, opens the store, requests maintenance and
seeds the cursor of a synthetic store with 10 legacy gaps.
- main()'s shared-mode branch calls it (AST).
- Both fail on 76dd56a and pass here.
- Real main() in shared mode, driven by scratchpad
maint-close-2/v3/mcp_startup_probe.py. Stubs: no model-server spawn,
model-server requests recorded and refused, mcp.run returns once the
background open finished. 5 runs per tree on fresh copies:
- 10k memories with a 13% legacy gap: before, store not opened,
0 wakes, no cursor; after, opened, 1 wake ("eligible"), cursor 500.
- 5000-memory eligible seed: before not opened, 0 wakes; after
opened, 1 wake.
- Every run, both trees: 0 get_model calls, 0 get_reranker calls,
0 model-server requests, no torch, transformers,
sentence_transformers or model2vec imported during startup.
- main() start to mcp.run, median (min to max): 10k store 52.8 ms
(31.7 to 109.6) before, 41.1 ms (40.0 to 86.1) after; 5k store
78.4 ms (58.5 to 80.4) before, 46.2 ms (35.6 to 91.5) after. No
measurable change: the open runs in the background.
- The background open, wake and seed finished 0.36 to 0.74 s after
mcp.run was reached.
Root cause (bfe9a38): reembed_served_vectors replaced the served rows in one writer transaction (DELETE every covered row, INSERT every shadow row, both tables). A SQL replica of that transaction on synthetic stores held the writer 4.8 s at 10k memories and 49.9 s at 100k. Other writers wait at most busy_timeout (10 s), so on a large affected store foreground adds would fail with "database is locked". sqlite-vec cannot rename a vec0 table (the renamed table looks for the old shadow-table names: "no such table: main.vec_messages_chunks"), so a drop-and-rename swap is not an option. Fix: _swap_shadow replaces the rows a complete shadow covers 1024 ids per writer transaction, with point deletes and inserts. Each memory keeps a vector throughout: the old one until its batch, the new one after. Each batch rechecks the shadow (complete, same model, source append-only) under the model fence and the writer, and stores its cursor (metadata vec_reembed_swapped:<shadow>). An interrupted swap continues from it; a new shadow generation clears it. A last transaction drops the shadows, their manifests and cursors, and writes the mark. Evidence (scratchpad maint-close-2/v3): - Batched SQL replica (swap_sql_batched*.py), 100k memories: 2048 ids per batch 0.66 s median, 2.16 s at most (98 batches, 74 s in total); 1024 ids 0.27 s median, 1.0 s at most (196 batches, 57 s). 10k, 2048: 0.54 s median, 1.1 s at most. - The product function end to end (swap_cost.py, 10k memories, mock encoder): 22 swap transactions, 0.84 s in total, 51 ms at most. The builds dominate the 65 s run and are unchanged. - test_interrupted_swap_keeps_a_vector_for_every_memory_and_resumes: an interruption in the third 16-id batch leaves all 60 memories with a vector (new below id 32, old above). The rerun encodes nothing (both shadows complete) and finishes the swap: the new vectors in both tables, no shadows, cursors or manifests left, mark set. Fails on 50a6e56 (DID NOT RAISE: one transaction per table), passes here. - tests/test-nan-reembed-crash-safe.py and tests/test-vector-gap-seed.py: 8 passed.
The seed (e7ca212) and backfill threads ran statements on the Engine's connection. Callers may use and close engine.conn directly (as test_issue_650 does), and closing a sqlite3 connection under another thread's running statement segfaults the process: the whole suite on dd112cf crashed in truememory-vector-backfill while test_issue_650 closed engine.conn. Both threads now open the store themselves (mode=rw, never creating it) inside a background_writes run, so the last Engine.close() stops them and stop_background_backfill fences a run that outlives its 1 s join. In-memory stores start no backfill thread. The Engine connection is never touched off its owner thread, which makes the reopen guard (_backfill_local, EngineClosedError) unnecessary. Tests (fail on dd112cf, pass here; run on a non-framework Python so the expected child crash on dd112cf shows no dialog): - background work runs no statement on the Engine connection - a direct engine.conn.close() during the seed or a pass does not crash (in a child process) - close() fences a seed without the last-close wait - the backfill never reopens a closed Engine Targeted files on the fix: 27 passed, no crash reports. Taken over from audit-code-maint-close-3, stopped mid-task.
Review v7 M-v7-1. On a store with no tier selection or policy, every add and update ran INSERT OR REPLACE on the embed_model and embed_dim metadata rows with a new updated_at, even when both values already matched. The clusters dependency identity hashed key, value and updated_at for those rows, so any add between a clusters snapshot and its publication changed the fingerprint and the build deferred with SourceChanged. 4f4f6b5's append tolerance therefore never applied to clusters on legacy stores, and each deferral discarded a full HDBSCAN fit (5 of 7 fits at 10k in pK10). Two changes, each pinned by a test: - _write_foreground_embedder_metadata_no_commit returns early when both rows already hold the serving identity. No write, no new timestamp. - _cluster_dependency_identity hashes key and value only. A same-value rewrite is no new embedding space: 0.7.6.2 still rewrites both rows on every add, and a rebuild's last write does too. The source fingerprint already covers every vector byte and message row up to the watermark. Model, dimension, vec table, schema, registry rows and metadata values are still in the identity, so a real model change still defers. Tests: - test-tier-metadata-publication-795: a matching publication writes no row and keeps updated_at. - test-tier-switch-liveness SnapshotPublicationOverForegroundAdds: a real engine.add before every layer's publication leaves no layer deferred and every layer append_pending; on a legacy store clusters publish over an add (the add keeps its vector and stays unclustered) and over an updated_at-only rewrite, and still defer on an embed_model value change. - The adapter test's comment points at the real-engine coverage. Evidence (clean archives, uv CPython 3.12.8, SQLite 3.47.1): - 9e87ddb code with these tests: all 3 new tests fail (11 != 9 total changes; clusters DEFERRED (SourceChanged); added_report DEFERRED). - This change: 15 passed in the touched classes; 16 passed under Homebrew CPython 3.14.6 including SmallStoreClustering. - Mutant with only the early return: the liveness clusters test fails at the rewrite case. Mutant with only the identity change: the 795 test and the metadata_kept check fail. - Reviewer probe pclusterid.py (300 memories, real hdbscan, 3.14): the add case goes from DEFERRED (SourceChanged) on 9e87ddb to "2 clusters", success at revision 301, identity rows unchanged.
engine.update commits the new source text, then publishes the new vector in a second writer transaction. That transaction did not use the foreground busy retry, so a writer held past busy_timeout made it fail; update caught the error, logged a warning and returned the new content with the old vector in place (review-tier-fixes-v7 L-v7-2). Backfill only fills missing vectors, so nothing repaired it. _foreground_vector_publication takes retry_busy, and update passes it. Only BEGIN is retried; the DELETE plus INSERT of the same rowid is idempotent. Test test_an_update_publishes_its_vector_past_a_writer_busy_after_the_source_commit: on d3a0a39 the stored vector stays the old one (similarity to new 0.03, to old 1.00); here it is the new one. Finished from audit-code-tier-4's work-in-progress after that agent was stopped.
test_pipeline_sets_busy_timeout_on_memory never closed the pipeline's Memory. Since e7ca212 the ingest pipeline (a default-mode process) seeds legacy vector gaps on a background thread at open, so that thread could still be writing the store directory when TemporaryDirectory removed it: "Directory not empty" in the python3.14 whole suite on 4426b5e (1 of 3 failures; the other 2 are the known NaN tests). close() stops and fences that thread, as designed. Test only; the file passes 5/5.
…usy writer Review v7 L-v7-1. eeb2cf0's busy retry ran inside begin_immediate, which add, update, delete and delete_all enter under the engine connection lock (admitted_call's hold plus their own with block); add and update's vector publication also enter it under the model fence (vector_search._lock). The retry loop, up to 30 s from the first attempt, held both, so every other thread of the process waited: get, search, a second add's model capture, a rebuild publication (it refuses a busy fence) and clustering. The reviewer's pretry measured a same-process read waiting 18.88 s (writer held 25 s) and 29.96 s (held 40 s), against 10.61 s before the retry existed. Fix: - begin_immediate takes a backoff context, entered around each wait between attempts. The shortened busy timeout is set for one attempt and restored right after it, so the connection keeps its own timeout while the locks are free. After a wait, a transaction found open on the connection is never joined (the last busy error is raised), and a connection closed meanwhile raises ProgrammingError. - writer_transaction, for an owned retrying BEGIN, backs off through runtime.released_connection_locks (every depth of this thread's engine locks, only when no transaction is open: the rule model and LLM work already follow) and a caller-supplied release for the model fence. The fence is let go first and taken back last, keeping the engine lock -> fence order. - _foreground_model_fence yields that release. It frees _lock for the wait, takes it back, and checks the model identity again (VectorPublicationChanged if it changed). add and _foreground_vector_publication pass it on. - Unchanged: only BEGIN is retried, so nothing was written; the budget is 30 s from the first attempt; borrowed writers (a SAVEPOINT, or an admission judged borrowed) never retry and never let go of a lock; the selection and model are checked under the writer on the attempt that wins; the WAL-reset guard code (its idle hook now also runs when a wait frees the engine lock, as it does for encode). - update's vector publication now checks, under its writer, that the memory still holds the text it embedded (content, plus sender, recipient and timestamp for the SEP vector), and publishes nothing otherwise. Another connection could always write between update's source commit and its publication, and the older update then replaced the newer vector for good (backfill fills only missing vectors). Now that the wait lets go of the engine lock, this engine's own threads can write there too. - The streaming harness execs extracted vector_search definitions; it now compiles them with postponed annotations, like the module, and copies _released_model_fence. Tests (test-foreground-busy-retry, test-tier-switch-liveness ForegroundWriteRetry): each backoff runs outside a transaction with the busy timeout restored; a transaction or close during a backoff is never joined; a selection committed during the wait refuses the winning attempt; through the engine, with a writer held 2 s past a 200 ms busy timeout, a same-process get and a model-fence acquire stay under 1 s on add, update (source and vector), delete and delete_all; an update publishes no vector over a newer write (another connection, or this engine during the wait); closing the engine during the wait fails the add with nothing stored and the engine reopens; a model change during the wait refuses the add with nothing stored; a nested admitted caller gets every lock depth back, and a caller transaction is not waited in; 4 same-process writers against a churning holder fail no add. Evidence (clean archives; uv CPython 3.12.8 / SQLite 3.47.1 via run312.sh unless noted): - 045270e code with these tests: 8 fail, the 2 pins pass. Writer held 2 s, worst same-process get / model-fence wait, 045270e -> here: add 1.99 / 1.99 -> 0.23 / 0.23 s; update source 2.00 / 0 -> 0.23 / 0 s; update vector 2.16 / 2.14 -> 0.22 / 0.21 s; delete 2.15 -> 0.24 s; delete_all 2.00 -> 0.23 s. The newer-write update kept the older vector (similarity to the newer text 0.009); the in-wait hooks never ran. - Mutants: without the source check both newer-write tests fail (similarity 0.009); with the fence never let go, the add's fence wait is 2.0 s while reads pass. - pretry (one add against a writer held by another connection, a second thread calling get/search every 100 ms), worst read wait, 045270e -> here: 3.47.1, 25 s hold 20.69 -> 10.26 s, 40 s hold 29.91 -> 10.26 s; 3.53.3 (Homebrew CPython 3.14.6), 25 s 18.57 -> 10.24 s, 40 s 29.94 -> 10.58 s. Outcomes unchanged: stored once at 25 s; "database is locked", nothing stored, at 40 s. The 10 s left is the first attempt, which keeps the configured busy timeout as before eeb2cf0. - Saturated 4-writer psat (10,484-memory store, 4 writer processes plus a consolidate() loop, 60 s), here: engine adders 0 failed of 6,466 (3.47.1, max 1.71 s) and of 12,726 (3.53.3, max 9.35 s; 045270e 0 of 13,281, max 13.18 s); raw loop 0 failed of 480,847 (3.47.1, max 12.39 s) and of 409,555 and 361,881 (3.53.3, max 28.88 and 8.29 s; 045270e 0 of 389,421, max 18.41 s). No lost rows, integrity ok. WAL-guard saturate test: 5 of 5 pass on each SQLite.
045270e passes retry_busy=True to _foreground_vector_publication from engine.update. The stub that test-source-update-fence-748 installs in its place did not accept the keyword, so the call raised TypeError, update caught it as a failed embedding and logged a warning, and test_content_publication_failure_keeps_historical_committed_source failed with "PublicationChanged not raised". The stub takes retry_busy and ignores it, as it does no writer wait. Evidence (uv CPython 3.12.8): the test passes on d3a0a39, fails on 045270e and 79f7722 with that TypeError, and the file passes 13 of 13 with this change.
auto_detect prefers the claude CLI whenever `claude` is on PATH, and hydrate_config left its model empty, so every extraction ran on the CLI's default model (usually Opus). Extraction makes up to 20 calls per ingested session, and on the user's machine it exhausted their Claude subscription; once the limit was hit every chunk failed with provider_error and no memories were extracted. The claude_cli backend now defaults to `haiku` (an explicit model still wins). Verified on the user's account: `claude -p --model haiku` answers with claude-haiku-5-5. Tests in test-claude-cli-default-model.py: the default and the command line fail before this change and pass after; an explicit model is kept.
_swap_shadow committed a batch and took the writer again within microseconds. SQLite's busy handler sleeps up to 100 ms between retries, so a writer in another process almost never found the lock free and waited for the whole swap: on 100k memories a hook add failed "database is locked" after 31 s (review-maint-close-v3 F1). Each batch now ends with a 100 ms pause, after the writer and model fence are released. The reviewer's measurement with this pause: 2,099 cross-process adds overlapped a 33 s swap, max 153 ms, 0 errors. Test test_swap_pauses_between_batches_with_the_writer_free: another connection takes BEGIN IMMEDIATE with no busy wait during every pause. Fails before (no pause), passes after; the file passes 4/4.
d3a0a39 (review v7 M-v7-1) made a same-value rewrite of the embedder metadata stop changing the clusters dependency identity, because every legacy-store add rewrote both rows and defeated every publication. The adapter test still required an updated_at-only change to reject an in-flight publication (whole suite on 3231b0f: 'success_empty' != 'failed'). The test now pins both sides of the new contract: a timestamp-only rewrite publishes, and an embed_model value change during the build still rejects it with no output and unverified coverage. File: 50 passed.
Second test pinning the pre-d3a0a39 contract: the source-fence matrix in test-cluster-publication-749 required an updated_at-only rewrite of embed_model to reject publication (whole suite on 3231b0f, subtest stored_identity_stamp). After review v7 M-v7-1 only metadata values are in the clusters identity, so that case now publishes; it moves to its own test. Value changes (stored_model, stored_dimension) and registry changes, cache_identity_stamp included, still reject. File: 31 passed.
…ners Hosted CI on 1d7ad97 failed on every OS: - Windows: PidfdFallback.linux() patched sys.platform to "linux" and then failed patching os.uname (absent on Windows). The exception left its ExitStack unclosed, so sys.platform stayed "linux" for the rest of the session and later tests took POSIX paths (fcntl imports, POSIX ownership checks). The helper now closes its stack on failure, and PidfdFallback, OwnedAbandonAndRelease and UnitRetirement (pidfd and systemd units) are skipped on Windows like the file's other Linux-only classes. Source reads in that file pass encoding="utf-8" (cp1252 failed to decode mcp_server.py). - macOS: the runner's sqlite3 cannot load extensions, so the vec0 fixtures in test-vector-gap-seed, test-nan-reembed-crash-safe and test-mcp-startup-store-open now carry requires_sqlite_ext. Two runtime-recovery tests read the runner's real free MPS memory, which refused their synthetic loads; mps_free_bytes is pinned unmeasured. - Ubuntu: the gate job was cancelled at its 20 minute limit at 61%; the test job now allows 45 minutes. Local: the five touched test files pass (166 passed).
The re-promotion backoff, the fast-lane idle period and both reranker idle periods were checked as `now - since >= period`. With monotonic clock values such as 588.062 (Windows) or 453.569018625 (macOS), `(since + period) - since` evaluates to 599.9999999999999, so the step that is due at exactly `since + period` was still refused. Hosted CI failed 12 model-server tests on Windows and 2 on macOS this way. Each check now compares against the deadline, `now >= since + period`, which is exact at the boundary. Behavior is otherwise unchanged. Model-server test files: 164 passed.
…pers - conftest's unconfigured-connection guard now ignores a connection that is already closed. A background run fenced by a previous test's close between its connect and configure_connection closes that connection at once; hosted Ubuntu flagged it in a later test's teardown (test_l5_surprise_rerank, maintenance.py:643). Plain sqlite3 connections, which take no weak reference, are still always reported. - test-source-capture-admission-native-v1 counts only os.fdopen calls on the source file when injecting the first or second wrapper failure. A state save also uses os.fdopen and, depending on timing, ran first (hosted Ubuntu: failure raised in _save_state).
…e DACL adopt_legacy_pointer opened every pointer with _open_private, which on Windows requires a private DACL. Hooks, and every older release, write these pointers with _atomic_write_text, which sets no DACL, so on Windows every pointer was refused (nonprivate_file) and no legacy backlog was ever adopted. Hosted Windows CI showed it in test-ingest-maintenance-busy and test-mcp-backlog-adoption-422. On Windows a pointer is now opened without the private check, still refusing links, non-regular files and extra hard links, and the existing _source_owned check on the open handle decides, exactly as for transcript sources. POSIX pointers keep the private-file check.
…file delete On Windows, Path.home() reads USERPROFILE, not HOME. Fixtures that set only HOME left completion and tier scenarios looking at the runner's real profile: transcripts were refused (source_not_admitted) and tier scenarios read another config (hosted Windows CI). The completion, backlog adoption, ingest-busy and large-transcript fixtures and the tier-switch scenario environment now set USERPROFILE too. test_close_then_delete_leaves_nothing_for_a_fenced_worker_to_recreate deletes the store while a blocked worker holds it open, which Windows never allows; it is skipped there. Local: the touched files and related completion tests pass (126 passed).
…t sqlite-vec Hosted macOS CI on 6b20f32: - test_no_retry_without_a_budget_or_for_other_errors allowed 0.5 s for a non-retried BEGIN against a 100 ms busy timeout and took 0.61 s. The bound is now 0.9 s, still below the 1 s hold a retry would wait out. - test_each_backoff_runs_outside_a_transaction_with_the_busy_timeout_restored needed more than one backoff within a 1 s hold, but a jittered backoff can approach 1 s; the hold is now 3 s. - test_openers_disable_autocheckpoint_only_on_affected_builds opened the tier _open_db, which refuses interpreters that cannot load sqlite-vec (the runner's Python); that opener is checked only where it can load. Both files: 21 passed.
…guard a grace period - test-source-capture-admission-native-v1: admission no longer wraps the source in os.fdopen; it streams the raw descriptor with os.read (_stream). The two wrapper tests injected failures into a call the capture no longer makes (hosted Ubuntu: CompletionError not raised; the ValueError variant passed only because a state save also uses os.fdopen). They now fail os.read on the source descriptor: an OSError surfaces as source_capture_failed, a ValueError propagates, and the descriptor is closed either way. All 8 tests in the file pass with the Linux gate bypassed on macOS; the old pair fails there too. - conftest: a background connection still open and unconfigured at teardown gets up to 2 s to reach configure_connection before the test fails. Hosted Ubuntu caught a background run between its connect and configure in unrelated tests' teardown. Plain connections are still reported at once.
Remove an unused io import left by the source-read rewrite, and set the requires_sqlite_ext pytestmark after the module imports (E402). ruff check truememory/ tests/: all checks passed.
On Python 3.10, sqlite3's Cursor.executescript calls the connection's Python-level commit(). A fenced connection runs executescript inside its run's statement gate, and its commit() override takes the same gate, so the maintenance thread deadlocked on itself in create_db's schema migrations. Hosted CI on Python 3.10 timed out about ten maintenance tests at 120 s each and hit the job limit; 3.11+ commit from C and were unaffected. The gate is now an RLock. Other threads, fence() included, wait for it exactly as before; only the thread already holding it re-enters. Test test_a_statement_that_commits_inside_its_gate_does_not_deadlock nests a COMMIT statement inside a statement on a worker thread: it deadlocks on the previous code (detected by a bounded join) and passes here, and a fenced run still refuses statements. Maintenance and related files: 274 passed.
…iter per case - test-tier-active-initialization-795 runs maintenance definitions with a stand-in threading module that had no RLock; since ca498b4 the run gate needs it (hosted macOS: AttributeError, 30+ tests). - test_a_transaction_or_close_during_the_backoff_is_never_joined reused one 1 s writer hold for two cases; on a slow runner the hold had ended before the second case, so its BEGIN never reached the backoff that closes the connection (ProgrammingError not raised). The second case now waits for the first hold and takes a fresh one. Both files pass on the 837 tree (43 passed).
test_no_module_copies_moves_or_opens_database_files AST-scans every module. It takes 6 s locally but 84 s, then over the suite's 120 s per-test limit, on hosted Python 3.10 runners. It now carries a 600 s pytest timeout; the scan itself is unchanged.
A fresh engine open refuses at once (MaintenanceBusyError) while another
process owns maintenance, which an ingest worker does for its pass.
In the MCP server that surfaced as a failed tool call ("Error executing
tool truememory_search: Maintenance is already running for this
database"), and the startup open in shared mode gave up after one try.
The MCP server now waits a bounded time instead; hooks keep failing
fast:
- Before an operation opens the store, _get_memory probes the
maintenance lock the way maintenance does (a separate descriptor,
released at once, never this process's own owner) and waits up to
10 s while another process holds it. The store still opens lazily
inside the operation.
- The startup open retries for up to 5 min in its background thread.
Tests in test-mcp-open-busy-wait.py (4 fail before, all pass after): a
held lock is waited out without opening early, the wait is bounded, an
open store or free lock never waits, and the startup open retries only
the busy error. MCP-related test files: 879 passed.
Each extraction chunk ran `claude -p`, which started every MCP server in the user's configuration before answering. On the user's Mac one call measured 306 s, past the 120 s timeout, so every chunk of a session failed with provider_error and nothing was extracted. Extraction uses no tools; with --strict-mcp-config (and no --mcp-config) the same call took 4 s. The claude_cli backend now passes --strict-mcp-config. The command-line test checks the flag (fails before, passes after).
SessionStart and UserPromptSubmit recall for Claude Code was emitted as a
top-level {"additionalContext": ...}. Claude Code reads hook context only
from hookSpecificOutput.additionalContext with a hookEventName, reports a
top-level field as unrecognized, and drops it. Hooks exited 0, so nothing
looked wrong, but no recall or directives reached Claude sessions, at
startup or after compaction.
Claude now gets the same hookSpecificOutput shape as Codex and Gemini.
The protocol fixture decodes Claude responses only in that shape and
rejects the old top-level payload; it and the onboarding, debounce and
intensity tests that read the old field fail before the change and pass
after.
codex-cli 0.164 labels each typed user block "user.text" with user provenance, gives the UserMessage event its own id instead of the canonical message id, and adds thread_settings_applied events and Extension items. The pinned grammar rejected all four, so every user turn was dropped: a seven-turn session parsed as assistant replies only, the Stop hook's five-turn minimum never passed, and no Codex session since the upgrade was captured. "user.text" now counts only with user provenance (any other provenance is still rejected), a UserMessage event matches its canonical turn on turn and exact text, and the two new record types are context. Scaffold blocks (AGENTS.md, environment context, harness content) stay excluded, and an event whose text differs from the canonical turn stays partial. The synthetic 0.164 rollout test fails before the change and passes after. On the user's eight most recent Codex rollouts, the seven-turn session now parses complete with seven human turns; sessions with images or question prompts remain partial, as before for unsupported content.
Codex builds before 0.164 (rollouts from Oct 20 to 25 on the user's Mac) label typed input "user.text" but write no content_item_metadata, so the previous commit still read those sessions as having no human turns. A "user.text" block now counts when provenance is user or absent; any other provenance still contradicts the label and is rejected. The same builds write environment context and the Codex app's open-page state without provenance; both are harness scaffold and are now excluded as such. Across the user's 30 Codex rollouts in this format, 431 human turns are now read (none before); 13 parse complete and 17 remain partial for images and question prompts. The synthetic test gains an earlier-build case that fails before and passes after.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Local embedding and reranking could start allocations without a shared reservation, worker startup could lose ownership evidence across failures, and Qwen embedding calls retained unused decoder caches. This composition adds shared model memory accounting and launch ownership records, and disables decoder caches in the three Qwen embedding constructors. The original models, precision, padding, reranker and output contract remain unchanged. It also carries the previously tested ingestion, provenance and adapter reliability fixes.
Changes since 001e8f1 (a full audit of this stack, each area independently reviewed):
Model server memory (macOS)
SQLite safety
Maintenance and foreground writes
updated_at). Stores too small for HDBSCAN publish no clusters instead of failing.mode=rwconnections and never create a deleted store.Memory(auto_maintenance=False)); long-lived processes run it, and the MCP server opens the store at startup in shared mode without loading models.Ingestion
Tier switching
Validation at 1d7ad97 (code identical to 3231b0f, which the whole suite ran on; beb58e9 and 1d7ad97 change tests only):
test_issue_465_flag_set_immediately,test_issue_485_nan_flag_set_after_completion) that also fail on the previous head.Release note: releases up to 0.7.6.1 back up a database with a plain file copy before a full rebuild. On SQLite older than 3.53 that copy can corrupt a live database or lose writes. Upgrade before switching tiers.
Known limitations: about 1.4 GB of allocator memory stays resident after a CPU fallback or a deep CPU rerank; the 1.2 GB room gate can delay the return to MPS; hook requests can wait while a CPU batch runs; a TERM can take up to 16 s while Metal drains; the first attempt of a busy foreground write still waits the full busy timeout; HDBSCAN (optional clustering extra) holds the GIL for up to about 17 s per fit; extraction processes at most 20 transcript chunks; Intel Macs with AMD GPUs need
TRUEMEMORY_DEVICE=cpu. The batch-32 versus batch-8 embedding comparison still fails its original numerical tolerance.This remains a draft. The complete code-only diff excludes README, public documentation, setup and compatibility guides, benchmark methodology and public benchmark results. No merges or releases.