Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: Dataset lifecycle tests
on:
push:
branches: [develop/dataset-workflows, feature/grounded-text-candidates, feature/image-caption-exports, feature/selected-release-preview-20261006, feature/saved-dataset-selections-20261006, fix/saved-selection-intent-20261006, feature/explicit-metadata-filters-20261006, feature/bulk-real-import-20261006, integration/dataset-candidate-20261006, feature/annotated-caption-import-20261006, integration/dataset-caption-stage-20261006, feature/curation-diagnostics-20261006, feature/rights-note-correction-20261006, feature/instruction-responses-20261007, feature/labeled-text-import-20261007, feature/text-classification-proposals, feature/local-postgres-metadata-20261011, integration/workspaces-postgres-20261011, feature/workspace-image-runs-20261011]
branches: [main, develop/dataset-workflows, feature/grounded-text-candidates, feature/image-caption-exports, feature/selected-release-preview-20261006, feature/saved-dataset-selections-20261006, fix/saved-selection-intent-20261006, feature/explicit-metadata-filters-20261006, feature/bulk-real-import-20261006, integration/dataset-candidate-20261006, feature/annotated-caption-import-20261006, integration/dataset-caption-stage-20261006, feature/curation-diagnostics-20261006, feature/rights-note-correction-20261006, feature/instruction-responses-20261007, feature/labeled-text-import-20261007, feature/text-classification-proposals, feature/local-postgres-metadata-20261011, integration/workspaces-postgres-20261011, feature/workspace-image-runs-20261011]
pull_request:
permissions:
contents: read
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/workspaces-m1.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: M1 workspace interface
on:
push:
branches: [ui/dataset-workspaces-m1-20261011, integration/workspaces-postgres-20261011, feature/workspace-image-runs-20261011]
branches: [main, ui/dataset-workspaces-m1-20261011, integration/workspaces-postgres-20261011, feature/workspace-image-runs-20261011]
pull_request:
paths:
- 'static/workspace*'
Expand Down
200 changes: 182 additions & 18 deletions backup_metadata.py

Large diffs are not rendered by default.

9 changes: 9 additions & 0 deletions docs/contracts/captured-cpu-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,3 +122,12 @@ This extension does not enlarge captured input/source or diagnostic streams,
alter cancellation/drain deadlines, enable network isolation, or qualify GPU/JIT.
Actual recipes retain explicit smaller output budgets and require measured
capacity admission. See the external recipe contract for custody and downloads.


## Backup permission preservation

Captured executable (`0500`) and data (`0400`) permissions survive new backups
and restores. Historical byte receipts do not authenticate modes: qualify the
actual snapshot at its restored root before claiming runtime acceptance.
The canonical transfer, retry and partial-cleanup policy is in
[local storage](local-storage.md#backup-permission-preservation).
27 changes: 27 additions & 0 deletions docs/contracts/local-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,3 +227,30 @@ complete backup, never automatically deleted. Deletion staging is reconciled
against committed metadata on reopen. Nested deletion defers physical removal
until the outermost commit. Release publication and worker dispatch refuse an
enclosing transaction so they cannot publish or execute uncommitted intent.


## Backup permission preservation

The existing PostgreSQL backup and restore owner preserves ordinary file
permission bits, including captured runtime executables (`0500`) and data
(`0400`). It applies directory permissions after copying children, with the
catalog root last. Source identity and permissions must remain stable during
copying. Resumed files must match both bytes and permissions; a missing file
inside read-only staging is refused without widening directory permissions.
Writable staging directories are intermediate construction state; existing
read-only directories must already match source permissions. A completed restore
retry compares all permissions against the retained backup. On backup failure,
only the invocation-owned private partial backup directories are made removable
for cleanup through retained directory descriptors and no-follow child opens.
Source and resumable restore directories are never widened. If private-partial
identity or traversal is uncertain, cleanup stops, retains the partial and adds
a diagnostic note to the original error rather than masking that failure.
The operator must preserve refused staging for diagnosis; the transfer never
repairs source files or rewrites runtime manifests.

The version-1 byte manifest and receipt format are unchanged. Permissions are
copied from the ordinary backup files, not authenticated by the historical byte
receipt. Older backups made without permission preservation cannot establish
original permissions or usable captured runtimes. Cold runtime verification is
required before claiming runtime restore acceptance; synthetic executable
fixtures do not qualify a real model or runtime installation.
9 changes: 9 additions & 0 deletions docs/contracts/workspace-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,3 +229,12 @@ does not interpret executable recipes or synthesize approval. Integer/text
columns use the existing backup format; retained schema-7 backups restore
exactly and must be explicitly upgraded. Complete schema-8 backups preserve
approval state and identities; recovery does not implicitly refund approvals.


## Backup permission preservation

Complete PostgreSQL/artifact transfers preserve file and directory permissions
without changing the retained byte receipt format or historical Run identities.
Resumed and completed restores refuse contradictory permissions.
The canonical transfer, retry and partial-cleanup policy is in
[local storage](local-storage.md#backup-permission-preservation).
34 changes: 34 additions & 0 deletions docs/plans/generic-scientific-projects/execution-ledger.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,3 +228,37 @@ verified durable publication; all89 published counterparts rehashed afterwards.
Preserved old cleanup evidence under its original names and recorded this new
cleanup separately. No failed staging, snapshot, source/input or custody removed.
P6 source checkpoint b02004a is draftPR53; no merge/deployment.

## P7 isolated source checkpoint and merge authority

P7 permission-preservation fix and explicit staged restore/browser procedures
are implemented in an isolated feature branch. Focused tests and independent
review passed; actual transfer and acceptance remain pending. See
[verification](reports/p7-verification.md) and [admission](reports/p7-admission.md).
The user's subsequent explicit instruction admits merging qualified Tuldok PRs,
assessing owner32/33 and verifying integrated main. PRs38–42 were merged in
dependency order with exact qualified trees retained. No incomplete/failing
work or deployment is admitted by this new authority.

## P6 hosted acceptance and P7 actual qualification

PR52 `fb3a5a0` and PR53 `5c0e7f9` passed both final hosted checks. P6's
patched full discovery passed 900 tests with 36 explicit skips; dedicated
local optional qualification remains separately recorded. G5/P6 is accepted.

P7's actual separate-database restore passed raw table/schema/sequence/tree
parity, cold runtime and completed-artifact/release reads, and public
Data/Pipeline/Run inspection at desktop and narrow widths. All 89 evaluation
artifacts matched full authenticated HTTP hashes. The original database and
verified backup remain unchanged; only the admitted task-created source tree
was removed after recovery proofs. The first logout failure, exact session
cleanup and fresh successful browser attempt remain recorded.

Final review additionally admitted exclusive fresh-catalog permission
initialization for private manifests/cross-umask restores. This refinement passed independent review and all 38 affected tests with
zero skips and frozen source. P7 final hosted regression remains pending. PR32 owner screenshots
and independent main CI trigger PR54 were qualified and merged, preserving
owner image bytes. Integrated main is `f2751c3`; its UI check passed and its
lifecycle check is queued. PR33 requires separate deliberate reconciliation.
PR43 and its dependent stack remain unmerged while the actual original v2
packet gate is blocked. A8 lacks a live supported model/runtime. No deployment.
18 changes: 9 additions & 9 deletions docs/plans/generic-scientific-projects/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

**Plan status:** `Active`

**Current phase:** P5 qualified; P6 actual scientific MCP read passed and final test/CI qualification is active. See [P5c](reports/p5c-verification.md) and [P6](reports/p6-verification.md).
**Current phase:** P5 and P6 are qualified. P7 actual scientific backup, separate-database restore, cold reads and public UI/artifact checks passed. The narrowly scoped catalog creation-permission refinement passed final affected tests and independent review. Final hosted regression and original legacy-v2 acceptance remain under verification. See [P5c](reports/p5c-verification.md), [P6](reports/p6-verification.md) and [P7](reports/p7-verification.md).

**Next slice:** Finish P6 regression/CI and P7 complete PostgreSQL/artifact restore; acceptance remains partial.
**Next slice:** Finish P7 hosted regression; verify integrated main. Original v2 packet access and A8 real image/text inference remain external blockers to the overall task.

**Acceptance status:** `partial`

Expand Down Expand Up @@ -36,10 +36,10 @@ The user approved publishing this plan and staged implementation on 2026-10-13.
| G2 | Scientific fields preserve exact float bytes, dimensions, staggered placement, units, time and trajectory identity through import, freeze, download and cold read | contract | representative | automated | satisfied | [Actual native and neutral custody workflows](reports/p2-verification.md) |
| G3 | An optional Rheon adapter exposes bounded generation controls and produces real generic scientific data; removing it leaves generic project/data use intact | integration | required-real | automated | satisfied | [P4 actual generation and no-adapter startup](reports/p4-verification.md) |
| G4 | Saved graph executes existing lab fitting and autonomous evaluation scripts; verified checkpoint, full-field comparisons and physics diagnostics survive restart with complete provenance | system | required-real | automated | satisfied | [P5b training](reports/p5b-verification.md) and [P5c real evaluation/restart](reports/p5c-verification.md) |
| G5 | Agent and UI clients reach the same domain owners for the admitted bounded matrix and equivalent authority; owner-only review/grants remain separate. Stale requests, forged human review, unauthorized execution, path escape and approval reuse fail before side effects | contract | representative | automated | pending | P3/P6 public-interface denial and parity tests |
| G5 | Agent and UI clients reach the same domain owners for the admitted bounded matrix and equivalent authority; owner-only review/grants remain separate. Stale requests, forged human review, unauthorized execution, path escape and approval reuse fail before side effects | contract | representative | automated | satisfied | [P6 real SDK parity, denial and final hosted regression](reports/p6-verification.md) |
| G6 | Admission is idempotent; cancel, crash and restart never duplicate work, publish partial success or silently leave descendants running | system | representative | automated | satisfied | [P3 custody and fault tests](reports/p3b-verification.md), retained by final P5 regression |
| G7 | Existing imports/releases/runs retain identities and bytes, including Rheon v1/v2; explicit migration and complete PostgreSQL/artifact restore preserve old and new records | integration | representative | automated | pending | P2/P7 retained-history and restore workflow |
| G8 | Desktop/narrow/keyboard UI supports open → generate → review/freeze → edit graph → train/evaluate → inspect results without explanatory clutter or false qualification labels | user-workflow | required-real | either | pending | P7 reviewed screenshots and actual-run browser receipt |
| G7 | Existing imports/releases/runs retain identities and bytes, including Rheon v1/v2; explicit migration and complete PostgreSQL/artifact restore preserve old and new records | integration | representative | automated | pending | [P7 actual scientific restore and retained fixture matrix](reports/p7-verification.md) passed; original legacy-v2 packet remains blocked |
| G8 | Desktop/narrow/keyboard UI supports open → generate → review/freeze → edit graph → train/evaluate → inspect results without explanatory clutter or false qualification labels | user-workflow | required-real | either | satisfied | [P1 desktop/narrow/keyboard](reports/p1-verification.md), [P4 real generation/review](reports/p4-verification.md), [P5 real train/evaluate](reports/p5c-verification.md), [P7 restored UI](reports/p7-verification.md) |

## Scope

Expand Down Expand Up @@ -236,21 +236,21 @@ Statuses below track the approved staged implementation. Before starting each mi

**Acceptance gate:** Complete G5 against the admitted matrix through a real workflow with separate owner review/grants. Compare domain evidence; exercise revisions, lost acknowledgements, revoked credentials, human-review spoofing, grant reuse, fresh external attempts, scoped test exposure and exact artifact reassembly. Excluded capabilities remain absent from MCP; existing HTTP authority is unchanged. No new actor-specific privilege bypass.

**Status:** `Planned`
**Status:** `Accepted`

**Preparation admission:** [P6 exact contract and sequencing](reports/p6-admission.md). Source preparation is active; objective acceptance remains pending.
**Preparation admission:** [P6 exact contract and sequencing](reports/p6-admission.md). The admitted matrix passed actual SDK scientific parity and final local/hosted gates.

### P7: Complete workflow and compatibility qualification

**Goal:** Deliver an installable local workflow with preserved old data, explicit limitations and meaningful real scientific acceptance.

**Allowed write set:** existing CI configuration, deployment/dependency documentation, existing backup/upgrade and browser acceptance tests, scoped packaging needed by P1–P6, README and this plan's reports. Runtime fixes remain in the owning milestone's boundary and receive affected re-review.
**Allowed write set:** existing CI configuration, deployment/dependency documentation, existing backup/upgrade, metadata initialization and browser acceptance tests, scoped packaging needed by P1–P6, README and this plan's reports. Runtime fixes remain in the owning milestone's boundary and receive affected re-review.

**Tasks:** Fresh setup/reopen, full PostgreSQL/artifact restore, old+new results, offline operation after installation, full actual browser workflow, all relevant lint/static/type/unit/integration/UI checks supported by chosen technologies; independent architecture/security/scientific review and applicable hosted CI once publication is authorized.

**Acceptance gate:** G1–G8 satisfied with linked evidence; no unqualified hardware/performance/model claims; all material review findings resolved. Separate outstanding dataset-workflows A8 and actual legacy-v2 packet qualification remain visible rather than hidden by this plan's completion.

**Status:** `Planned`
**Status:** `Verifying`

## Blockers

Expand Down
19 changes: 19 additions & 0 deletions docs/plans/generic-scientific-projects/reports/p6-verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,22 @@ API-test patch the only change from the original full discovery inventory.
`7b1de6a33f76b3cf08ad45a91af346bfbb5ff13840a459937d73b5f4120e979a`.
The failed original900-test discovery remains recorded; fresh hosted full
discovery on the patched source is the final regression acceptance gate.

## Final hosted acceptance

Both final checks passed at PR53 head
`5c0e7f92914e5aeac6ec939e3a667fc85b22aa90`:
[lifecycle](https://github.com/MrScripty/Tuldok/actions/runs/38478255351) and
[workspace UI](https://github.com/MrScripty/Tuldok/actions/runs/38478255410).
Hosted full discovery passed 900 tests in 672.693 seconds with 36 explicit
skips. The separately recorded local optional environments cover the admitted
reader, training, evaluation and SDK paths with zero skips. Hosted agent
22-test and MCP 13-test modules also passed. The broader suite is not claimed
to have zero skips. Independent review and final JavaScript/browser checks
passed. This supersedes the pending full-regression status above and satisfies
G5 for the explicitly admitted operation matrix.

The later caption-browser change only waits for the loaded page before DOM
reads; it changes no scientific or MCP production behavior. Actual scientific
SDK evidence remains tied to source `b02004a`. Original v2 packet and A8
real-model acceptance remain separate blockers to overall completion.
Loading
Loading