Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ jobs:
--health-retries 12
env:
TULDOK_TRAINING_PYTHON: ${{ github.workspace }}/build/workspace-training/bin/python
TULDOK_NATIVE_READER_PYTHON: ${{ github.workspace }}/build/workspace-training/bin/python
TULDOK_DATABASE_URL: postgresql://tuldok_test:isolated-ci-fixture@127.0.0.1:5432/tuldok_test
steps:
- uses: actions/checkout@v4
Expand All @@ -49,6 +50,11 @@ jobs:
- run: build/workspace-training/bin/python -m unittest discover -s tests -p test_pointclouds.py
- run: build/workspace-training/bin/python -m unittest discover -s tests -p test_volume_assets.py
- run: python -m unittest discover -s tests
- run: node --check static/scientific-fields.js
- run: build/workspace-training/bin/python tests/qualify_scientific_fields.py --output build/qa/scientific-neutral.zip --report build/qa/scientific-neutral.json
- run: node tests/browser_scientific_fields.cjs
env:
BROWSER: /usr/bin/google-chrome
- run: node --check static/workspace-projects.js
- run: node tests/browser_workspace_project.cjs
env:
Expand Down
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ current record; saving/reviewing and freezing a new release leaves old results
unchanged. Viewing test results records durable exposure; reused records or
pixels cannot be presented as fresh holdout evidence.

Fresh catalogs use schema 5. Existing PostgreSQL schema-1 through schema-4 catalogs require an
Fresh catalogs use schema 6. Existing PostgreSQL schema-1 through schema-5 catalogs require an
explicit stopped-writer backup and upgrade, never an automatic startup change:

```sh
Expand Down Expand Up @@ -852,8 +852,12 @@ fixtures and do not download model weights or call paid inference providers.

### Point-cloud and volume workflows

In Data, import `points.ply` with `points.json`, or `volume.npy` with `volume.json`. The bounded profiles are [ordered ASCII PLY points](docs/contracts/point-cloud-data.md) and [uint8 NPY1 volumes](docs/contracts/volume-data.md). Inspect native points or index slices/physical coordinates, save human review, select whole assets and freeze a canonical release. In Pipeline, create and edit the corresponding validation/export recipe; Runs retains its integrity report and the unchanged release. Classification releases also offer validation/export without installing a trainer. Existing catalogs require the explicit schema5 upgrade described in the storage runbook; the application never silently migrates them. [Accepted workspace scope, evidence and limitations](docs/plans/dataset-centered-workspaces/reports/final-20261013.md).
In Data, import `points.ply` with `points.json`, or `volume.npy` with `volume.json`. The bounded profiles are [ordered ASCII PLY points](docs/contracts/point-cloud-data.md) and [uint8 NPY1 volumes](docs/contracts/volume-data.md). Inspect native points or index slices/physical coordinates, save human review, select whole assets and freeze a canonical release. In Pipeline, create and edit the corresponding validation/export recipe; Runs retains its integrity report and the unchanged release. Classification releases also offer validation/export without installing a trainer. Existing catalogs require the explicit schema6 upgrade described in the storage runbook; the application never silently migrates them. [Accepted workspace scope, evidence and limitations](docs/plans/dataset-centered-workspaces/reports/final-20261013.md).

## Open a local project

Start with `python app.py --data ./catalog --project-root research=/absolute/research`, then select **Open project** in `/workspaces`. A [project manifest](docs/contracts/projects.md) references files beneath that explicitly selected root. Data shows sources and missing requirements, Pipeline shows recipe references, and Runs shows external evidence. This stage is **inspection only**: no project code, model loading, runtime probing or training executes. Existing catalog workflows remain separate. Opening registers a snapshot in Tuldok; it does not modify the opened project.
Start with `python app.py --data ./catalog --project-root research=/absolute/research`, then select **Open project** in `/workspaces`. A [project manifest](docs/contracts/projects.md) references files beneath that explicitly selected root. Data shows sources and missing requirements, Pipeline shows recipe references, and Runs shows external evidence. Opening performs inspection: no project code, model loading, runtime probing or training executes. Existing catalog workflows remain separate. Opening registers a snapshot in Tuldok; it does not modify the opened project.

## Scientific field sequences

For a conforming [scientific field bundle](docs/contracts/scientific-fields.md), use **Import fields** on its project Data reference. Inspect frames, fields and index slices with physical coordinates, units and producer stage labels; save whole-trajectory review and freeze a canonical release. Original bundle and floating-point bytes remain unchanged. First-observed-state and declared-family grouping does not certify independent initial conditions. The cold reader `python scientific_release.py RELEASE.zip` verifies frozen custody; generation and training require later execution milestones.
16 changes: 15 additions & 1 deletion app.py
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,13 @@ def do_GET(self):
if path == '/api/workbench/recipes':
return self.reply(dataset_recipes.RECIPES)
if path.startswith('/api/workbench/records/'):
if path.endswith('/scientific'):
options = parse_qs(urlsplit(self.path).query, keep_blank_values=True)
if set(options) - {'frame', 'field', 'axis', 'index'} or any(len(values) != 1 for values in options.values()):
raise workbench.WorkbenchError('Choose one frame, field, axis and slice index.')
return self.reply(dataset.workbench.scientific.inspect(path.split('/')[-2],
frame=int(options.get('frame', ['0'])[0]), field=options.get('field', [None])[0],
axis=options.get('axis', ['z'])[0], index=int(options.get('index', ['0'])[0])))
if path.endswith('/preferences'):
return self.reply(dataset.workbench.preferences.list(path.split('/')[-2]))
if path.endswith('/responses'):
Expand Down Expand Up @@ -557,6 +564,7 @@ def do_GET(self):
'/workspace-projects.js': ('workspace-projects.js', 'text/javascript'),
'/pointclouds.js': ('pointclouds.js', 'text/javascript'),
'/volume-editor.js': ('volume-editor.js', 'text/javascript'),
'/scientific-fields.js': ('scientific-fields.js', 'text/javascript'),
'/audio-editor.js': ('audio-editor.js', 'text/javascript'),
'/audio-editor.css': ('audio-editor.css', 'text/css'),
'/meshes.css': ('meshes.css', 'text/css'),
Expand Down Expand Up @@ -605,14 +613,20 @@ def do_POST(self):
if not 0 < length <= limit:
raise ValueError('Request is too large or empty.')
raw_body = self.rfile.read(length)
body = projects.decode(raw_body) if path == '/api/workspace/projects' else spatial_assets.parse_request(raw_body) if path in ('/api/workbench/pointcloud-import', '/api/workbench/volume-import') else native_detection_import.parse_request(raw_body) if path in ('/api/workbench/native-detection-import/prepare', '/api/workbench/native-detection-import/row') else saved_searches.parse_request(raw_body) if path == '/api/workbench/searches' or path.startswith('/api/workbench/searches/') else sequence_batch_import.parse_request(raw_body) if path in ('/api/workbench/sequence-import', '/api/workbench/sequence-import-item') else meshes.parse_json(raw_body) if path == '/api/workbench/mesh-import' else json.loads(raw_body)
body = projects.decode(raw_body) if path == '/api/workspace/projects' or path.startswith('/api/workspace/projects/') else spatial_assets.parse_request(raw_body) if path in ('/api/workbench/pointcloud-import', '/api/workbench/volume-import') else native_detection_import.parse_request(raw_body) if path in ('/api/workbench/native-detection-import/prepare', '/api/workbench/native-detection-import/row') else saved_searches.parse_request(raw_body) if path == '/api/workbench/searches' or path.startswith('/api/workbench/searches/') else sequence_batch_import.parse_request(raw_body) if path in ('/api/workbench/sequence-import', '/api/workbench/sequence-import-item') else meshes.parse_json(raw_body) if path == '/api/workbench/mesh-import' else json.loads(raw_body)
if not isinstance(body, dict):
raise ValueError('Expected an object.')
path = urlsplit(self.path).path
if path.startswith('/api/workspace/'):
parts = path.removeprefix('/api/workspace/').split('/')
if parts == ['projects']:
return self.reply(dataset.projects.open(body), 201)
if len(parts) == 4 and parts[0] == 'projects' and parts[2] == 'import-data':
projects.closed(body, ('expected_sha256',), ('name', 'rights', 'groups', 'parents'))
raw, origin = dataset.projects.read_data_reference(parts[1], parts[3],
body['expected_sha256'], projects.DATA_LIMIT)
admission = {key: value for key, value in body.items() if key != 'expected_sha256'}
return self.reply(dataset.workbench.scientific.admit(raw, admission, origin), 201)
if parts == ['graphs']:
return self.reply(dataset.pipelines.create(body), 201)
if len(parts) == 2 and parts[0] == 'graphs':
Expand Down
41 changes: 7 additions & 34 deletions audio_assets.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
import base64
import hashlib
import io
import os
import struct
import uuid
import wave
Expand Down Expand Up @@ -92,21 +91,13 @@ def validate_annotation(value, record):
return {'segments': result}


def _sync_directory(path):
descriptor = os.open(path, os.O_RDONLY)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)


class AudioAssets:
def __init__(self, workbench):
self.workbench = workbench
self.path = workbench.dataset.path / 'audio'

def admit(self, body):
from metadata import CommitOutcomeUnknown
from immutable_assets import publish_file
if type(body) is not dict or set(body) - {'wav', 'name', 'rights', 'groups', 'parents'}:
invalid('import expects wav, name, rights, groups and optional parents.')
encoded = body.get('wav')
Expand All @@ -132,35 +123,17 @@ def admit(self, body):
record_id, created = uuid.uuid4().hex, timestamp()
folder = self.path / record_id
w = self.workbench
def discard():
if folder.exists():
(folder / 'source.wav').unlink(missing_ok=True)
folder.rmdir()
try:
with w.lock, w.db.transaction():
for parent in parents:
w._get(parent)
if w.db.execute("SELECT 1 FROM workbench_records WHERE kind='audio' AND content_hash=%s", (digest,)).fetchone():
invalid('this original WAV already exists; open its existing record.', 'conflict', 409)
self.path.mkdir(exist_ok=True)
folder.mkdir()
# Successful savepoints pass custody to their outer transaction.
w.db.on_rollback(discard)
with (folder / 'source.wav').open('xb') as target:
target.write(raw); target.flush(); os.fsync(target.fileno())
_sync_directory(folder); _sync_directory(self.path); _sync_directory(self.path.parent)
with w.lock, w.db.transaction():
for parent in parents:
w._get(parent)
if w.db.execute("SELECT 1 FROM workbench_records WHERE kind='audio' AND content_hash=%s", (digest,)).fetchone():
invalid('this original WAV already exists; open its existing record.', 'conflict', 409)
with publish_file(w.db, folder, 'source.wav', raw):
w.db.execute('INSERT INTO workbench_records VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)',
(record_id, 'audio', name, None, None, digest, None, encode(groups), encode(parents),
encode(origin), TASK, None, 'draft', 1, created, created))
record = w._get(record_id)
w._history(record)
except CommitOutcomeUnknown:
# A lost PostgreSQL commit acknowledgement may already have published this reference.
# Retain its bytes; the existing catalog reconciliation owns the next action.
raise
except Exception:
discard()
raise
return record

def metadata(self, record_id, provenance):
Expand Down
5 changes: 3 additions & 2 deletions canonical_release.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ def archive(raw, maximum_entries, maximum_bytes):

def validate_bytes(raw, expected_sha256, *, kind, validate_row):
"""Return custody and subtype summaries for one supported whole-asset kind."""
if kind not in ('mesh', 'sequence', 'pointcloud', 'volume'):
if kind not in ('mesh', 'sequence', 'pointcloud', 'volume', 'field_sequence'):
raise ValueError('Unsupported canonical release reader.')
require(type(raw) is bytes and 0 < len(raw) <= MAX_RELEASE_BYTES, 'select a release of at most 64 MiB.')
digest = sha(raw)
Expand Down Expand Up @@ -106,11 +106,12 @@ def validate_bytes(raw, expected_sha256, *, kind, validate_row):
and set(report['actual_counts']) <= set(SPLITS)
and all(type(n) is int and n >= 0 for n in report['actual_counts'].values())
and {split: report['actual_counts'].get(split, 0) for split in SPLITS} == counts, 'split report counts disagree with records.')
row_limit = MAX_METADATA_BYTES if kind == 'field_sequence' else MAX_ROW_BYTES
for split in SPLITS:
raw_rows = release.read(split + '/records.jsonl')
require(not raw_rows or raw_rows.endswith(b'\n'), 'split JSONL must end with a newline.')
lines = raw_rows.splitlines()
require(all(0 < len(line) <= MAX_ROW_BYTES for line in lines), 'split JSONL row exceeds 256 KiB or is empty.')
require(all(0 < len(line) <= row_limit for line in lines), 'split JSONL row exceeds its subtype metadata bound or is empty.')
require([encode(decode(line)) for line in lines] == [encode(row) for row in rows if row['split'] == split], 'split JSONL differs from frozen manifest records.')
coco = decode(release.read(split + '/coco.json'))
require(type(coco) is dict and set(coco) == {'info', 'licenses', 'images', 'annotations', 'categories'}
Expand Down
4 changes: 2 additions & 2 deletions caption_import.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ def snapshot(row):
invalid('Invalid protected source snapshot fields.')
if not isinstance(row['id'], str) or not IDENTIFIER.fullmatch(row['id']):
invalid('Invalid origin source ID.')
if row['kind'] not in ('image', 'text', 'sequence', 'mesh', 'pointcloud', 'volume') or type(row['revision']) is not int or row['revision'] < 1:
if row['kind'] not in ('image', 'text', 'sequence', 'mesh', 'pointcloud', 'volume', 'field_sequence') or type(row['revision']) is not int or row['revision'] < 1:
invalid('Invalid origin kind or revision.')
if type(row['source_available']) is not bool or row['source_lineage_known'] is not True:
invalid('Protected source lineage must be known, including deleted members.')
Expand All @@ -89,7 +89,7 @@ def snapshot(row):
invalid('Invalid retained book/session links.')
if row['kind'] == 'text' and (row['book_id'] is not None or row['session_id'] is not None or not is_hash(row['content_hash']) or not is_hash(row['source_sha256']) or row['pixel_hash'] is not None):
invalid('Invalid protected text source snapshot.')
if row['kind'] in ('sequence', 'mesh', 'pointcloud', 'volume') and (row['book_id'] is not None or row['session_id'] is not None
if row['kind'] in ('sequence', 'mesh', 'pointcloud', 'volume', 'field_sequence') and (row['book_id'] is not None or row['session_id'] is not None
or not is_hash(row['content_hash']) or row['source_sha256'] != row['content_hash']
or row['pixel_hash'] is not None or type(row['source_revision']) is not int
or row['source_revision'] != 1 or row['source_split'] != 'unassigned'):
Expand Down
Loading
Loading