Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions docs/plans/database-working-store.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Database working store for external projects

Status: stages 1–2 implemented; stage 3 revised (2026-11-02). Owner decision recorded with the user.

## Decision

The local PostgreSQL database is the persistent **working copy** of an opened
project and is kept across sessions. The external project folder
(`.tuldok/project.json` plus content-addressed `records/`, tracked in Git) is
only:

- an **import source**, read when its files changed since Tuldok last imported
or exported them, or when the project is not yet in this workspace; and
- an **export target**, written only by an explicit Save.

Edits during a session are committed to the database and stay there. Closing a
project does not write the external folder. Complete domain validation runs
only when the record set changes (import of a changed folder, or a commit).

There is exactly one durable copy of the record set (the database) and no
complete in-memory copy: owners read the record families or records they need.

## Why

Opening the 54k-record, 1.77 GB Rheon portfolio rebuilt a fresh SQL projection
from every JSON file on each Open (9+ minutes, 17–21 GB RAM: the Store, the
Session and owner copies each held the whole record set). JSON exists only so
the project can be tracked in Git.

## Identity and integrity

- Checkpoint identity is unchanged: the working copy keeps the same manifest
(inventory of `family/id/format/bytes/sha256`, roots, receipts) and the same
checkpoint digest as the file format, so Save writes byte-identical files and
existing receipts, validation markers and custody records stay meaningful.
- Each record is stored once as its exact canonical bytes (not `jsonb`, which
would change number spelling such as `-0.0`) with its SHA256.
- Import verifies `project.json`, then only reads, SHA256-checks and
canonical-checks records whose digest the working copy does not already hold.
- Save re-checks that the external `project.json` is still the one last
imported/exported (refusing to overwrite external changes), writes only
missing record files, then publishes `project.json` atomically.

## Open

1. Read external `project.json` (one file) and its SHA256.
2. Working copy exists and its last imported/exported manifest hash matches:
attach. No record file is read; the SQL projection is reused.
3. External folder changed, working copy has no unsaved changes: incremental
import, validate (cached per checkpoint), rebuild projection tables.
4. External folder changed **and** working copy has unsaved changes: refuse with
an explicit conflict (keep working copy and overwrite on Save, or discard and
re-import). Never merge silently.

## Stages

1. **Working store — done.** `WorkingProjectStore` (project_working_store.py)
with the ProjectStore interface, incremental import, commits joined to the
projection's open SQL transaction, Save as the only folder write. Workspace
keeps a private folder→working-copy registry.
2. **No whole-project memory copy — done.** Sessions over a working copy bind
checkpoint metadata only; family/record/identity reads come from
PostgreSQL, commit hooks never materialize the record set, Study authoring
restores from Study/Trial documents and Run history loads per Run.
3. **Projection reuse — revised.** Reusing the SQL projection across sessions
would carry live execution state (runs, admissions, execution grants) into
a later session, which a cold open deliberately starts without. Instead the
projection is still reconstructed fresh, from cheap family-scoped reads.
4. **Edit-scoped validation — done.** `project_validation_delta` declares each
validation component's input families (from a per-component audit); an
edit re-runs only components whose inputs it changed and only the old
receipts bound to changed records or sharing a key space. Identity, claim,
fork and receipt-identity checks always run; unclassified operations are
always validated. `TULDOK_VALIDATION_CROSSCHECK=1` runs complete validation
alongside and fails on disagreement (the project suite passes in that mode).
A text import on the portfolio: 161 s → ~31 s; validation itself 1–3 s. The
remainder is the receipt digest over every record's bytes (format-defined)
and the full record read for owner captures, both dominated by stage 5.
5. **Smaller project files — open.** Store each captured runtime file listing
once and reference it from Runs (format migration with exact round trip).

## Measured on the Rheon portfolio (54k records, 1.77 GB metadata)

| | before | reopen now |
|---|---|---|
| Open | ~9 min (CPU) | ~11 s |
| Peak memory | 21.7 GB | 0.76 GB |
| First history/progression read | ~100 s | 0.5 s |

First import into a new workspace takes ~50 s plus one complete validation
(~2.5 min) per checkpoint and validator version.

Payload blobs (`artifacts/sha256/*`, 16 GB for the portfolio) remain
content-addressed files under the project folder in stages 1–4.
6 changes: 5 additions & 1 deletion model_definitions.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,13 @@ def natural(value, name, minimum=0, maximum=MAX_INTEGER):
return value


# Same code points as any(ord(c)<32 or ord(c)==127 or 0xD800<=ord(c)<=0xDFFF), scanned in C.
_TEXT_FORBIDDEN=re.compile('[\x00-\x1f\x7f\ud800-\udfff]')


def text(value, name, maximum=120):
if (type(value) is not str or not value.strip() or len(value)>maximum or
any(ord(c)<32 or ord(c)==127 or 0xD800<=ord(c)<=0xDFFF for c in value)):
_TEXT_FORBIDDEN.search(value)):
raise WorkbenchError(name+' must be bounded nonempty text without controls.')
return value

Expand Down
25 changes: 25 additions & 0 deletions project_composition.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
The lifecycle owner must validate each source and the complete final candidate,
clone primary edits and publish all required bytes through the native owners.
"""
import contextvars
from contextlib import contextmanager
import hashlib

from project_fork import ARCHIVE_FAMILIES, _sha, _uuid, decode_fork
Expand All @@ -23,7 +25,30 @@ def _fail(message, code='invalid', status=400):
raise ProjectCompositionError(message, code, status)


_KNOWN_ENVELOPES = contextvars.ContextVar('known_envelope_digests', default=None)


@contextmanager
def known_envelope_digests(known):
"""Supply verified envelope digests for exact record objects during validation.

known maps id(record) -> (record, digest) for envelopes whose document is the
stored canonical revision. Only that same object uses its digest; any other
envelope (for example a constructed observation variant) is encoded.
"""
token = _KNOWN_ENVELOPES.set(known)
try:
yield
finally:
_KNOWN_ENVELOPES.reset(token)


def _record_sha(record):
known = _KNOWN_ENVELOPES.get()
if known is not None:
entry = known.get(id(record))
if entry is not None and entry[0] is record:
return entry[1]
return hashlib.sha256(_encode(record)).hexdigest()


Expand Down
35 changes: 18 additions & 17 deletions project_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
from saved_searches import SavedSearches, criteria
from saved_selections import SavedSelections
from dataset_releases import Releases
from project_session import session_active, session_records, session_records_for

FAMILIES = {
'workbench_record': 'workbench_record_v1',
Expand Down Expand Up @@ -971,11 +972,10 @@ def capture_runtime_changes(self, before_records, *, operation, key):
return captured

def _sync_sources(self):
records = self.session.snapshot['records']
from project_fork import active_records
active = active_records(records, project_id=self.session.project_id)
self.context.project_sources = {r['id']: copy.deepcopy(r['document']['source']) for r in active if r['family'] == 'workbench_record'}
self._declarations = {r['id']: r['document'] for r in records if r['family'] == 'project_artifact'}
# Only the selected families are copied; fork history is still checked.
active = session_active(self.session, 'workbench_record')
self.context.project_sources = {r['id']: r['document']['source'] for r in active if r['family'] == 'workbench_record'}
self._declarations = {r['id']: r['document'] for r in session_records(self.session, 'project_artifact')}

def _available(self, record_id):
source = self.context.project_sources.get(record_id)
Expand Down Expand Up @@ -1005,9 +1005,8 @@ def _publish(self, raw, kind, role, media_type):
media_type=media_type, kind=kind, role=role, preservation='unique', required_for=['inspect', 'release'])

def _require_record(self, record_id, *, extended=False):
from project_fork import active_records
snapshot = self.session.snapshot
for item in active_records(snapshot['records'], project_id=snapshot['project_id']):
# Only these families are inspected; fork history is still checked.
for item in session_active(self.session, 'workbench_record', 'workbench_text'):
if item['id'] == record_id and item['family'] in ('workbench_record', 'workbench_text'):
if extended and item['family'] == 'workbench_text':
fail('This operation requires an explicitly converted data record.', 'unsupported', 409)
Expand Down Expand Up @@ -1047,9 +1046,8 @@ def _mutate(self, operation, target, body, expected_checkpoint, key, invoke):

def _legacy_replay(self, operation, target, body, key):
"""Retain M1 request meaning after the shared M2 session cutover."""
records = self.session.snapshot['records']
from project_fork import active_records
records = active_records(records, project_id=self.session.project_id)
# Only receipts and legacy text are inspected below.
records = session_active(self.session, 'project_operation', 'workbench_text')
if any(r['family']=='project_operation' and r['document']['request_key']==key for r in records):
return None
if operation not in ('data.import_text','data.annotate'):
Expand Down Expand Up @@ -1291,7 +1289,8 @@ def _capture(self, result, before_records, operation, key):
for family, table, owner in (('saved_selection', 'saved_selections', self.selections), ('saved_search', 'saved_searches', self.searches)):
live = {r[0] for r in self.db.execute('SELECT id FROM ' + table)}
from project_fork import active_records
all_ids = live | {r['id'] for r in active_records(before_records, project_id=self.session.project_id) if r['family'] == family}
all_ids = live | {r['id'] for r in active_records(before_records, project_id=self.session.project_id,
families=frozenset({family})) if r['family'] == family}
for record_id in all_ids:
old = before.get((family, record_id))
current = owner._get(record_id) if record_id in live else copy.deepcopy(old['document']['current'])
Expand Down Expand Up @@ -1333,16 +1332,18 @@ def _capture(self, result, before_records, operation, key):
return {'records': list(unique.values()), 'result': copy.deepcopy(result)}

def restore(self, snapshot=None):
snapshot = snapshot or self.session.snapshot
if snapshot != self.session.snapshot:
if snapshot is not None and snapshot != self.session.snapshot:
fail('Data reconstruction snapshot is stale.', 'stale_project', 409)
owned = decode_data_records(snapshot['records'])
# The decoder reads its own families and project_fork overlays, and its
# fork validation needs every record identity; unrelated documents are
# shared read-only instead of copying unrelated Run history.
families = (*FAMILIES, 'workbench_text', 'project_artifact', 'project_fork')
owned = decode_data_records(session_records_for(self.session, *families))
self._sync_sources()
# Validate present source bytes through the existing semantic owners
# before touching any selected SQL row. Missing bytes retain metadata.
self._verify_sources(owned)
from project_fork import active_records
active = active_records(snapshot['records'], project_id=self.session.project_id)
active = session_active(self.session, *families)
identities = {(r['family'], r['id']) for r in active}
owned = [r for r in owned if (r['family'], r['id']) in identities]
with self.db.lock, self.db.transaction():
Expand Down
4 changes: 2 additions & 2 deletions project_dispatch.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import copy

from workbench import WorkbenchError
from project_session import session_records


_BOUND_FIELDS = frozenset(('binding', 'checkpoint', 'expected_checkpoint', 'request_key'))
Expand Down Expand Up @@ -61,8 +62,7 @@ def _origin_items(items, arguments):


def _declarations(projection):
return {item['id']: item['document'] for item in projection.session.snapshot['records']
if item['family'] == 'project_artifact'}
return {item['id']: item['document'] for item in session_records(projection.session, 'project_artifact')}


def _artifact(projection, sha):
Expand Down
22 changes: 19 additions & 3 deletions project_execution.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
from project_history import capture_run, declarations
from project_session import operation_records
from workbench import WorkbenchError
from project_session import session_operations


OPERATIONS = frozenset(('runs.submit_external', 'runs.submit', 'runs.retry', 'runs.cancel', 'runs.approve_external'))
Expand Down Expand Up @@ -527,11 +528,26 @@ def capture_dirty(self, before_records):
merged[identity] = record
return list(merged.values())

def _capture_pending(self):
"""Whether capture_dirty could return any record (see its owners)."""
if self._dirty_runs or self._pending_studies or any(self._pending_producers.values()) or self.native.has_dirty:
return True
study = self.study_execution
if study is not None and (study._dirty or study._package_records):
return True
producers = self.producer_execution
if producers is not None and (producers._runtime_data is not None or any(producers._dirty.values())):
return True
return False

def before_commit(self):
if self._reconstructing or self.session.foreground_publication:
return
if not self._dirty_runs and not self._pending_studies and not any(self._pending_producers.values()) and not self.native.has_dirty and self.study_execution is None and self.producer_execution is None:
return
if not self._capture_pending():
# Nothing can be captured; never materialize the record set.
return
snapshot = self.session.snapshot
records = self.capture_dirty(snapshot['records'])
if records:
Expand Down Expand Up @@ -600,7 +616,7 @@ def native_preflight(self,graph_id,revision,expected_checkpoint,key):
graph_id,revision,expected_checkpoint,key)

def _native_replay(self,operation,target,body,expected_checkpoint,key):
prior=operation_records(self.session.snapshot['records']).get(key)
prior=session_operations(self.session).get(key)
if prior is None:return None
if (prior['operation']!=operation or self.db.execute('SELECT id FROM workspace_runs WHERE id=%s',
(prior['result'].get('run_id'),)).fetchone() is None):
Expand Down Expand Up @@ -656,7 +672,7 @@ def submit_external(self, body, expected_checkpoint, key):
authority = self.runtime_access.require_external_submission()
with self.access.lock, self.session.lock, self.db.lock:
self.session.check_current()
prior = operation_records(self.session.snapshot['records']).get(key)
prior = session_operations(self.session).get(key)
if prior is None and expected_checkpoint != self.session.summary['checkpoint']:
raise WorkbenchError('Expected project checkpoint changed.', 'stale_project', 409)
if prior is not None:
Expand All @@ -679,7 +695,7 @@ def approve_external(self, body, expected_checkpoint, key):
authority = self.runtime_access.require('approve-execution', 'runs.approve_external', owner_only=True)
with self.access.lock, self.session.lock, self.db.lock:
self.session.check_current()
prior = operation_records(self.session.snapshot['records']).get(key)
prior = session_operations(self.session).get(key)
if prior is None and expected_checkpoint != self.session.summary['checkpoint']:
raise WorkbenchError('Expected project checkpoint changed.', 'stale_project', 409)
if prior is not None:
Expand Down
Loading
Loading