Implement in-place SPSS transformations

Author: TonisOrmisson

CHANGELOG.md

@@ -4,8 +4,21 @@ All notable changes to this reference implementation are documented here.
 
 ## Unreleased
 
+### Added
+
+- Added a bounded SPSS-like `RECODE`, `VARIABLE LABELS`, and `VALUE LABELS`
+  parser with stable diagnostics and conformance fixtures.
+- Added canonical Transformation Plan serialization, RFC 8785 hashes, schema
+  binding, deterministic typed operations, and an agent-facing CLI/API.
+- Added product-neutral in-place execution with same dataset/table identity,
+  direct data and metadata mutation, and compact operation audit. Dolt adds
+  expected branch/HEAD and clean-working-set checks.
+
 ### Changed
 
+- Version history, diff, rollback, restoration, and commit remain database
+  responsibilities; the transformer creates no derived/copy/snapshot/recovery
+  layer and performs no `DOLT_COMMIT`.
 - SQL server support now distinguishes conservative family claims from exact
   CI evidence: PostgreSQL 17.x/18.x at 17.10/18.4, MySQL 8.4.x/9.7.x at
   8.4.11/9.7.2, and MariaDB 11.4.x/11.8.x/12.3.x at

README.md

@@ -61,6 +61,46 @@ See [the SQL transformation workflow](docs/sql-transformation-workflow.md) for
 Python and CLI examples, migration behavior, hashing, atomicity, and the exact
 implemented capability boundary.
 
+## SPSS-like transformation frontend
+
+The transformation frontend accepts `RECODE`, `VARIABLE LABELS`, and
+`VALUE LABELS`, lowers them to a canonical OpenStatSpec Transformation Plan,
+and mutates the same logical dataset and physical wide table. SQLite,
+PostgreSQL, MySQL, MariaDB, and Dolt connections are allowed. It creates no
+derived dataset, copied table, snapshot, or OpenStatSpec rollback/version layer.
+
+Install the compact operation-audit relation separately, then apply syntax:
+
+```python
+from openstatspec import (
+    apply_spss_in_place,
+    install_in_place_transformation_schema,
+)
+
+database_url = "postgresql+psycopg://user:password@host/database"
+install_in_place_transformation_schema(database_url=database_url)
+result = apply_spss_in_place(
+    database_url=database_url,
+    dataset_id="...",
+    actor="agent@example.org",
+    source_text="""
+      RECODE age (18 THRU 34 = 1) (35 THRU 64 = 2) INTO age_group.
+      VARIABLE LABELS age_group 'Age group'.
+      VALUE LABELS age_group 1 '18-34' 2 '35-64'.
+    """,
+)
+```
+
+Existing-target recodes and metadata mutations use direct DML. SQLite and
+PostgreSQL may also add a target column in the same native transaction. MySQL,
+MariaDB, and Dolt reject such schema-changing plans before mutation because
+their implicit-commit DDL could otherwise leave a partial apply. On Dolt, the
+caller additionally supplies expected branch and HEAD identities, and the
+working set must be clean. The transformer never calls `DOLT_COMMIT`.
+
+See [in-place transformations](docs/in-place-transformation.md) for the full
+execution boundary and CLI form.
+
 ## Current support status
 
 The adapter requires `openstatspec-pyspssio==0.5.1.post2` as its sole SPSS

docs/in-place-transformation.md

@@ -0,0 +1,41 @@
+# In-place SPSS-like transformation
+
+`openstatspec.apply_spss_in_place` is the public execution path for the
+SPSS-like frontend. It accepts `RECODE`, `VARIABLE LABELS`, and `VALUE LABELS`,
+binds them to the existing core dataset, and applies the canonical plan to that
+same SQL wide table and metadata catalog.
+
+The caller supplies a supported SQL URL, the existing normative `dataset_id`,
+and a non-empty actor identity. For Dolt, the caller also supplies the expected
+active branch and current `HEAD` hash.
+
+Install the compact audit relation once with
+`openstatspec.install_in_place_transformation_schema(database_url=...)` before
+the first apply. Apply never creates schema-management objects itself.
+
+The adapter uses the engine's ordinary transaction behavior. On Dolt it first
+checks branch and HEAD and requires `dolt_status` to be empty. Existing-target
+recodes are one direct `UPDATE`. SQLite and PostgreSQL can add a new numeric
+`INTO` target to the same table. MySQL, MariaDB, and Dolt require that target
+column and variable metadata to exist before apply because their DDL can commit
+independently of the following data and metadata changes. Label commands
+update/replace the same dataset's normative and compatibility metadata rows.
+
+One compact `transformation_apply` row records operation identity, canonical
+plan/source hashes, actor, status, timestamps, and the observed Dolt branch and
+HEAD. It contains no row values and points to no copied table.
+
+The adapter does not create `derived_dataset` rows, persistent output tables,
+full-table copies, staging datasets, snapshots, rollback tables, retirement
+records, or a recovery/version catalog. It does not call `DOLT_COMMIT`, change
+branches, merge, reset, or tag. After success, the caller reviews `dolt diff`
+and independently decides whether to commit or restore the working set.
+
+The local SQLite tests exercise the public mutation path and assert that
+dataset/table counts and identities do not change. Live PostgreSQL/MySQL/MariaDB and
+exact-version Dolt service evidence remains required before release execution
+claims for those engines.
+
+```text
+openstatspec apply-spss --database-url mysql+pymysql://user:password@host/database --dataset-id ... --actor agent@example.org --expected-branch feature/recode --expected-head ... --syntax-file transform.sps
+```

src/openstatspec/__init__.py

@@ -1,20 +1,30 @@
 """Public API for the OpenStatSpec Python reference implementation."""
 
 from .api import (
-    capabilities, capability_matrix, derive_sql_dataset, execute_sql_transformation,
-    export_sav, get_dataset, import_sav, inspect, list_datasets,
+    apply_spss_in_place, capabilities, capability_matrix, derive_sql_dataset,
+    execute_sql_transformation,
+    export_sav, get_dataset, import_sav, inspect,
+    install_in_place_transformation_schema, list_datasets,
     register_sql_transformation, reconcile_derived_removals,
     reconcile_sql_transformation_runs,
     remove_derived_physical_relation, retire_derived, validate, validate_derived,
 )
 from .core import CapabilityDeclaration, LossReport, UnsupportedOperationError
 from .sql.workflow import TransformationError
+from .transform import (
+    SpssFrontendCompilation, TransformationFrontendError,
+    VariableDefinition, VariableSchema, compile_spss_syntax,
+)
 
 __all__ = [
-    "CapabilityDeclaration", "LossReport", "TransformationError",
+    "CapabilityDeclaration", "LossReport", "SpssFrontendCompilation",
+    "TransformationError", "TransformationFrontendError",
+    "VariableDefinition", "VariableSchema",
     "UnsupportedOperationError", "capabilities", "capability_matrix",
-    "derive_sql_dataset", "execute_sql_transformation", "export_sav",
-    "get_dataset", "import_sav", "inspect", "list_datasets",
+    "apply_spss_in_place", "compile_spss_syntax", "derive_sql_dataset",
+    "execute_sql_transformation", "export_sav",
+    "get_dataset", "import_sav", "inspect",
+    "install_in_place_transformation_schema", "list_datasets",
     "register_sql_transformation", "reconcile_derived_removals",
     "reconcile_sql_transformation_runs",
     "remove_derived_physical_relation", "retire_derived", "validate",

src/openstatspec/api.py

@@ -20,6 +20,11 @@
     remove_derived_relation as _remove_derived_relation,
     retire_derived_dataset as _retire_derived_dataset,
 )
+from .sql.inplace_transform import (
+    apply_spss_in_place as _apply_spss_in_place,
+    in_place_transformation_capabilities,
+    install_in_place_transformation_schema as _install_in_place_schema,
+)
 from .sql.capabilities import (
     SPECIFICATION_COMMIT, SPECIFICATION_RELEASE, active_connection, catalog_binding,
 )
@@ -84,6 +89,9 @@ def capability_matrix(database_url: str | None = None) -> Mapping[str, Any]:
         "sql_profiles": declared_profiles(database_url),
         "optional_profiles": {
             "sql_transformation_workflow": transformation_capabilities(database_url),
+            "spss_in_place_transformation": (
+                in_place_transformation_capabilities()
+            ),
         },
     }
     return declaration
@@ -130,6 +138,27 @@ def derive_sql_dataset(*, database_url: Any, **options: Any) -> Mapping[str, Any
     return result(_derive_dataset(database_url=str(database_url), **options))
 
 
+def apply_spss_in_place(
+    *, database_url: Any, dataset_id: str, source_text: str,
+    actor: str, expected_branch: str | None = None,
+    expected_head: str | None = None,
+) -> Mapping[str, Any]:
+    """Apply supported SPSS-like syntax to the same SQL dataset/table."""
+    return result(_apply_spss_in_place(
+        database_url=str(database_url),
+        dataset_id=dataset_id,
+        source_text=source_text,
+        actor=actor,
+        expected_branch=expected_branch,
+        expected_head=expected_head,
+    ))
+
+
+def install_in_place_transformation_schema(*, database_url: Any) -> None:
+    """Install the compact apply-audit relation before the first apply."""
+    _install_in_place_schema(database_url=str(database_url))
+
+
 def validate_derived(*, database_url: Any, derived_dataset_id: str) -> Mapping[str, Any]:
     return result(_validate_derived_dataset(
         database_url=str(database_url), derived_dataset_id=derived_dataset_id,

src/openstatspec/cli.py

@@ -2,9 +2,11 @@
 import argparse
 import json
 from collections.abc import Sequence
+from pathlib import Path
 
 from .api import (
-    capability_matrix, derive_sql_dataset, execute_sql_transformation,
+    apply_spss_in_place, capability_matrix, derive_sql_dataset,
+    execute_sql_transformation,
     export_sav, get_dataset, import_sav, inspect, list_datasets,
     register_sql_transformation, validate, validate_derived,
 )
@@ -73,6 +75,19 @@ def main(argv: Sequence[str] | None = None) -> int:
     derive.add_argument("--dataset-name")
     derive.add_argument("--weight-variable")
 
+    apply_spss = commands.add_parser(
+        "apply-spss",
+        help="apply supported SPSS syntax in-place on a controlled Dolt branch",
+    )
+    apply_spss.add_argument("--database-url", required=True)
+    apply_spss.add_argument("--dataset-id", required=True)
+    apply_spss.add_argument("--actor", required=True)
+    apply_spss.add_argument("--expected-branch")
+    apply_spss.add_argument("--expected-head")
+    syntax_source = apply_spss.add_mutually_exclusive_group(required=True)
+    syntax_source.add_argument("--syntax")
+    syntax_source.add_argument("--syntax-file")
+
     derived_validator = commands.add_parser("validate-derived", help="validate a derived dataset")
     derived_validator.add_argument("--database-url", required=True)
     derived_validator.add_argument("--derived-dataset-id", required=True)
@@ -122,6 +137,20 @@ def main(argv: Sequence[str] | None = None) -> int:
             transformation_name=args.name, dataset_name=args.dataset_name,
             weight_variable=args.weight_variable,
         )
+    elif args.command == "apply-spss":
+        source_text = (
+            args.syntax
+            if args.syntax is not None
+            else Path(args.syntax_file).read_text(encoding="utf-8")
+        )
+        output = apply_spss_in_place(
+            database_url=args.database_url,
+            dataset_id=args.dataset_id,
+            source_text=source_text,
+            actor=args.actor,
+            expected_branch=args.expected_branch,
+            expected_head=args.expected_head,
+        )
     else:
         output = validate_derived(
             database_url=args.database_url, derived_dataset_id=args.derived_dataset_id,