Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,10 @@ jobs:
- run: build/workspace-training/bin/python -m unittest discover -s tests -p test_pointclouds.py
- run: build/workspace-training/bin/python -m unittest discover -s tests -p test_volume_assets.py
- run: python -m unittest discover -s tests
- run: node --check static/workspace-projects.js
- run: node tests/browser_workspace_project.cjs
env:
BROWSER: /usr/bin/google-chrome
- run: node --check static/workbench.js
- run: node --check static/audio-editor.js
- run: node --check static/saved-selections.js
Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ current record; saving/reviewing and freezing a new release leaves old results
unchanged. Viewing test results records durable exposure; reused records or
pixels cannot be presented as fresh holdout evidence.

Fresh catalogs use schema 4. Existing PostgreSQL schema-1, schema-2 or schema-3 catalogs require an
Fresh catalogs use schema 5. Existing PostgreSQL schema-1 through schema-4 catalogs require an
explicit stopped-writer backup and upgrade, never an automatic startup change:

```sh
Expand Down Expand Up @@ -852,4 +852,8 @@ fixtures and do not download model weights or call paid inference providers.

### Point-cloud and volume workflows

In Data, import `points.ply` with `points.json`, or `volume.npy` with `volume.json`. The bounded profiles are [ordered ASCII PLY points](docs/contracts/point-cloud-data.md) and [uint8 NPY1 volumes](docs/contracts/volume-data.md). Inspect native points or index slices/physical coordinates, save human review, select whole assets and freeze a canonical release. In Pipeline, create and edit the corresponding validation/export recipe; Runs retains its integrity report and the unchanged release. Classification releases also offer validation/export without installing a trainer. Existing catalogs require the explicit schema4 upgrade described in the storage runbook; the application never silently migrates them. [Accepted workspace scope, evidence and limitations](docs/plans/dataset-centered-workspaces/reports/final-20261013.md).
In Data, import `points.ply` with `points.json`, or `volume.npy` with `volume.json`. The bounded profiles are [ordered ASCII PLY points](docs/contracts/point-cloud-data.md) and [uint8 NPY1 volumes](docs/contracts/volume-data.md). Inspect native points or index slices/physical coordinates, save human review, select whole assets and freeze a canonical release. In Pipeline, create and edit the corresponding validation/export recipe; Runs retains its integrity report and the unchanged release. Classification releases also offer validation/export without installing a trainer. Existing catalogs require the explicit schema5 upgrade described in the storage runbook; the application never silently migrates them. [Accepted workspace scope, evidence and limitations](docs/plans/dataset-centered-workspaces/reports/final-20261013.md).

## Open a local project

Start with `python app.py --data ./catalog --project-root research=/absolute/research`, then select **Open project** in `/workspaces`. A [project manifest](docs/contracts/projects.md) references files beneath that explicitly selected root. Data shows sources and missing requirements, Pipeline shows recipe references, and Runs shows external evidence. This stage is **inspection only**: no project code, model loading, runtime probing or training executes. Existing catalog workflows remain separate. Opening registers a snapshot in Tuldok; it does not modify the opened project.
42 changes: 38 additions & 4 deletions app.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,12 @@
import meshes
import spatial_assets
import pipeline
import projects
import runs
import base64
import hashlib
import io
import ipaddress
import json
import math
import shutil
Expand Down Expand Up @@ -133,7 +135,7 @@ def validate_annotation(body):


class Dataset:
def __init__(self, path, *, dsn=None):
def __init__(self, path, *, dsn=None, project_roots=None):
self.path = Path(path).resolve()
self.path.mkdir(parents=True, exist_ok=True)
(self.path / 'images').mkdir(exist_ok=True)
Expand All @@ -142,6 +144,7 @@ def __init__(self, path, *, dsn=None):
self.db = Metadata(self.path, dsn=dsn)
self.lock = self.db.lock
try:
self.projects = projects.Projects(self, project_roots)
self.generation_jobs = synthetic.Jobs(self)
with self.lock:
self._recover_deletions()
Expand All @@ -158,6 +161,8 @@ def __init__(self, path, *, dsn=None):
self.pipelines = pipeline.Pipelines(self)
self.runs = runs.Runs(self, self.pipelines)
except BaseException:
if hasattr(self, 'projects'):
self.projects.close()
self.db.close()
raise

Expand All @@ -167,6 +172,7 @@ def close(self):
self.caption_proposals.close()
self.grounded.close()
self.generation_jobs.close()
self.projects.close()
self.db.close()

def rows(self):
Expand Down Expand Up @@ -402,11 +408,26 @@ def reply(self, data, status=200, content_type='application/json'):
except (BrokenPipeError, ConnectionResetError):
pass

def project_access(self):
# This opt-in file inspection surface is local-only until actor authentication.
host = urlsplit('http://' + self.headers.get('Host', '')).hostname
if (host not in ('localhost', '127.0.0.1', '::1') or
not ipaddress.ip_address(self.client_address[0]).is_loopback):
raise workbench.WorkbenchError('Project inspection requires a local loopback client and host.', 'forbidden', 403)

def do_GET(self):
path = urlsplit(self.path).path
try:
if path == '/api/workspace/projects' or path.startswith('/api/workspace/projects/'):
self.project_access()
if path.startswith('/api/workspace/'):
parts = path.removeprefix('/api/workspace/').split('/')
if parts == ['projects']:
return self.reply(dataset.projects.list())
if len(parts) == 2 and parts[0] == 'projects':
return self.reply(dataset.projects.get(parts[1]))
if len(parts) == 4 and parts[0] == 'projects' and parts[2] == 'evidence':
return self.reply(dataset.projects.evidence(parts[1], parts[3]))
if parts == ['context']:
return self.reply({'version': 1, 'mode': 'real', 'dataset_id': dataset.db.dataset_id,
'recipe': 'image_classification_v1', 'execution': 'optional_local_cpu'})
Expand Down Expand Up @@ -533,6 +554,7 @@ def do_GET(self):
assets = {'/native-detection-import.js': ('native_detection_import.js', 'text/javascript'), '/saved-searches.js': ('saved-searches.js', 'text/javascript'), '/sequence-batch.js': ('sequence-batch.js', 'text/javascript'), '/pumas-typed.js': ('pumas-typed.js', 'text/javascript'), '/pumas-gateways.js': ('pumas-gateways.js', 'text/javascript'), '/meshes.js': ('meshes.js', 'text/javascript'), '/sequences.js': ('sequences.js', 'text/javascript'), '/text-classification-proposals.js': ('text-classification-proposals.js', 'text/javascript'), '/caption-proposals.js': ('caption-proposals.js', 'text/javascript'), '/curation.js': ('curation.js', 'text/javascript'), '/caption-import.js': ('caption_import.js', 'text/javascript'), '/bulk-import.js': ('bulk_import.js', 'text/javascript'), '/saved-selections.js': ('saved-selections.js', 'text/javascript'), '/workbench': ('workbench.html', 'text/html'), '/workbench.js': ('workbench.js', 'text/javascript'), '/workbench.css': ('workbench.css', 'text/css'), '/': ('index.html', 'text/html'), '/app.js': ('app.js', 'text/javascript'), '/style.css': ('style.css', 'text/css'), '/rights-note.js': ('rights-note.js', 'text/javascript'), '/preferences.js': ('preferences.js', 'text/javascript'), '/instruction-responses.js': ('instruction-responses.js', 'text/javascript'), '/native-text-import.js': ('native_text_import.js', 'text/javascript')}
# Opt-in workspace shell keeps fixture and real service boundaries distinct.
assets.update({
'/workspace-projects.js': ('workspace-projects.js', 'text/javascript'),
'/pointclouds.js': ('pointclouds.js', 'text/javascript'),
'/volume-editor.js': ('volume-editor.js', 'text/javascript'),
'/audio-editor.js': ('audio-editor.js', 'text/javascript'),
Expand Down Expand Up @@ -573,6 +595,8 @@ def do_POST(self):
raise ValueError('Send JSON.')
length = int(self.headers.get('Content-Length', '0'))
path = urlsplit(self.path).path
if path == '/api/workspace/projects' or path.startswith('/api/workspace/projects/'):
self.project_access()
limit = native_detection_import.MAX_PREPARE_REQUEST if path == '/api/workbench/native-detection-import/prepare' else native_detection_import.MAX_REQUEST if path == '/api/workbench/native-detection-import/row' else saved_searches.MAX_REQUEST if path == '/api/workbench/searches' or path.startswith('/api/workbench/searches/') else meshes.MAX_REQUEST if path == '/api/workbench/mesh-import' else rheon_sequences.MAX_REQUEST if path in ('/api/workbench/sequence-import', '/api/workbench/sequence-import-item') else MAX_BODY
if path in ('/api/workbench/pointcloud-import', '/api/workbench/volume-import'):
limit = spatial_assets.MAX_REQUEST
Expand All @@ -581,12 +605,14 @@ def do_POST(self):
if not 0 < length <= limit:
raise ValueError('Request is too large or empty.')
raw_body = self.rfile.read(length)
body = spatial_assets.parse_request(raw_body) if path in ('/api/workbench/pointcloud-import', '/api/workbench/volume-import') else native_detection_import.parse_request(raw_body) if path in ('/api/workbench/native-detection-import/prepare', '/api/workbench/native-detection-import/row') else saved_searches.parse_request(raw_body) if path == '/api/workbench/searches' or path.startswith('/api/workbench/searches/') else sequence_batch_import.parse_request(raw_body) if path in ('/api/workbench/sequence-import', '/api/workbench/sequence-import-item') else meshes.parse_json(raw_body) if path == '/api/workbench/mesh-import' else json.loads(raw_body)
body = projects.decode(raw_body) if path == '/api/workspace/projects' else spatial_assets.parse_request(raw_body) if path in ('/api/workbench/pointcloud-import', '/api/workbench/volume-import') else native_detection_import.parse_request(raw_body) if path in ('/api/workbench/native-detection-import/prepare', '/api/workbench/native-detection-import/row') else saved_searches.parse_request(raw_body) if path == '/api/workbench/searches' or path.startswith('/api/workbench/searches/') else sequence_batch_import.parse_request(raw_body) if path in ('/api/workbench/sequence-import', '/api/workbench/sequence-import-item') else meshes.parse_json(raw_body) if path == '/api/workbench/mesh-import' else json.loads(raw_body)
if not isinstance(body, dict):
raise ValueError('Expected an object.')
path = urlsplit(self.path).path
if path.startswith('/api/workspace/'):
parts = path.removeprefix('/api/workspace/').split('/')
if parts == ['projects']:
return self.reply(dataset.projects.open(body), 201)
if parts == ['graphs']:
return self.reply(dataset.pipelines.create(body), 201)
if len(parts) == 2 and parts[0] == 'graphs':
Expand Down Expand Up @@ -742,10 +768,18 @@ def main():
parser.add_argument('--data', type=Path, default=ROOT / 'data')
parser.add_argument('--host', default='127.0.0.1')
parser.add_argument('--port', type=int, default=8091)
parser.add_argument('--project-root', action='append', default=[], metavar='ALIAS=PATH',
help='Allow read-only project inspection beneath a local directory; repeat for multiple roots.')
args = parser.parse_args()
roots = {}
for binding in args.project_root:
alias, separator, location = binding.partition('=')
if not separator or not location or alias in roots:
parser.error('Each project root needs a unique ALIAS=PATH.')
roots[alias] = location
try:
dataset = Dataset(args.data)
except StorageError as error:
dataset = Dataset(args.data, project_roots=roots)
except (StorageError, workbench.WorkbenchError) as error:
parser.exit(2, str(error) + "\n")
server = ThreadingHTTPServer((args.host, args.port), make_handler(dataset))
print('Tuldok: http://' + args.host + ':' + str(server.server_port), flush=True)
Expand Down
119 changes: 119 additions & 0 deletions docs/contracts/projects.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Project inspection v1

Owner: `projects.py:Projects`; PostgreSQL schema5 owns immutable registration
snapshots. HTTP and the workspace UI project the same owner. This first stage
inspects references only: project files are not imported as Data records,
recipes are not executable graphs, and external evidence is not a Tuldok Run.

## Opening and reference schema

Configure selected directories locally when starting the application:

```sh
python app.py --data ./catalog --project-root research=/absolute/research
```

Then choose **Open project** in `/workspaces`, select `research` and enter the
manifest's relative path. Root aliases use 1–64 letters/digits/dot/dash/underscore
and start with a letter or digit. The server does not accept browser-supplied
absolute root paths. Project endpoints require loopback client addresses and
loopback Host names; existing broader application authentication is unchanged.
Only expose directories whose referenced files local application clients may
read. Project opening grants no actor/execution authority.

An external manifest can live beside an existing source checkout, so opening
does not require adding files to that checkout. All file references are relative
to the configured root, **not** the manifest's directory:

```json
{
"schema_version": 1,
"id": "8bd3793c538d4a0bba1d1e0c675a86d3",
"name": "Pressure experiment",
"data": [{"id": "capture", "label": "Trajectory capture", "path": "lab/runs/capture.npz"}],
"recipes": [{"id": "driver", "label": "Experiment driver", "path": "lab/scripts/trajectory_experiment.py"}],
"evidence": [{"id": "freeze", "label": "Frozen protocol", "path": "lab/experiments/trajectory-v2/freeze.json"}],
"requirements": [{"id": "torch", "label": "Compatible PyTorch environment"}]
}
```

Every top-level field shown is required; unknown fields/versions are refused.
Use a new 32-character lowercase hex ID per independent project snapshot.
Names are at most120 characters. Each section has at most64 entries with unique
IDs (same alias syntax) within that section and labels of at most160 characters.
Data/recipe/evidence entries require `id,label,path` and optionally a lowercase
64-character `sha256`. Requirements require `id,label` and optionally `path`.
Requirements are declarations: file presence does not establish interpreter,
package, device, version or scientific compatibility. No subprocess probe,
module import, checkpoint deserialization or dependency installation occurs.

Manifest input is at most64KiB of strict UTF-8 JSON. Duplicate keys, nonfinite
numbers, lone surrogates, unsupported fields and unsafe paths fail before
registration. Paths are at most512 characters and16 components; absolute paths,
backslashes, empty/dot/parent components and control characters are refused.
The registered snapshot includes the parsed manifest and SHA256 of its original
bytes, selected root alias and root device/inode identity. Reopening identical
input returns that registration. A changed manifest or root for the same ID
returns409; there is no replacement/delete/edit operation in this stage.

## Filesystem and evidence boundary

Startup pins open directory handles for configured roots. Inspection walks
directory descriptors with no-follow flags, refusing symlinks at every component,
including final files. Only regular files are read; FIFOs/devices/directories
cannot block the file reader. Root selection itself is a trusted local startup
action and resolves the configured directory. All project access is read-only;
opening writes only Tuldok's catalog registration.

Inspection hashes at most8MiB per referenced file and64MiB total per detail
request. Larger regular files are reported present with unverified contents.
Bounded reads check size and modification identity before/after and refuse
observed changes. Presence, measured byte hash, equality to a declared hash and
scientific verification are distinct. `verified` means only equality to the
manifest's declared SHA256, not source authenticity or scientific correctness.

Text evidence previews permit only `.json`, `.txt` and `.md`, at most64KiB of
UTF-8 without binary control characters. The response is JSON text, never active
HTML/Markdown. An expected hash mismatch refuses preview. Other files may be
listed/hashed but cannot be parsed or downloaded through this first-stage API.

## HTTP operations and observation

- `GET /api/workspace/projects` returns configured alias names and registered
snapshot summaries. No absolute root path or credential is returned.
- `POST /api/workspace/projects` accepts exactly `{root,manifest_path}` and
registers a valid immutable snapshot, returning detail. A lost reply can be
reconciled by listing; repeating identical opening is idempotent.
- `GET /api/workspace/projects/ID` inspects the saved references. It reports
manifest `unchanged`, `changed`, `missing` or `unavailable`; a changed source
never replaces the saved references implicitly.
- `GET /api/workspace/projects/ID/evidence/REFERENCE_ID` returns bounded external
text plus its hash and `scientific_verification: not_performed`.

Detail is always `mode: inspection`, `execution: unsupported`. Referenced file
states are `available`, `missing`, `unavailable`, with separate integrity
`verified`, `mismatch`, `unverified`. Requirements may be `present_unqualified`,
`missing`, `unavailable`, `unverified`. Paths with missing files leave the rest
of the project usable. Removing/rebinding a configured root leaves the snapshot
visible but unavailable; matching alias alone cannot silently redirect reads.

Browser source selection remembers `project:ID` and Data/Pipeline/Runs navigation
without selecting the current catalog as that project's data. Switching projects
does not move catalog records or alter existing dirty editors. External project
mode cannot submit training. Its three panels show file references, recipe
references and external evidence respectively.

## Persistence and compatibility

Schema5 adds `workspace_projects(id TEXT PRIMARY KEY, document TEXT NOT NULL)`.
Schema1–4 definitions and stored bytes are retained exactly; use the explicit
quiesced [upgrade/backup procedure](workspace-storage.md). Complete backup
preserves registration rows, but does not copy external project roots. Back up
external research files separately. Restoring on a different filesystem leaves
root identity unavailable; explicit rebinding/migration is future work, not a
silent reassignment. Existing catalog-only startup requires no project roots.

Schema version, project manifest version and later scientific/recipe versions
have independent owners. Runtime requirements are declared file references in
v1; later qualified execution must use its own admitted runtime/permission
contract rather than treating presence as capability.
Loading
Loading