Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
a2c2dbc
docs: design transformation and SPSS 0.2 support
TonisOrmisson Aug 17, 2026
a234bb6
docs: plan transformation and SPSS 0.2 support
TonisOrmisson Aug 17, 2026
9fc8667
test: load official transformation conformance manifests
TonisOrmisson Aug 17, 2026
5a33c99
feat: add official transformation plan 0.1 model
TonisOrmisson Aug 17, 2026
a8c006c
fix: validate transformation plan schema strings
TonisOrmisson Aug 17, 2026
1a90c93
feat: implement transformation plan 0.2 operations
TonisOrmisson Aug 17, 2026
d7e37b6
feat: add exact SPSS decimal binary64 conversion
TonisOrmisson Aug 17, 2026
d4af7af
feat: parse SPSS frontend 0.2 commands
TonisOrmisson Aug 17, 2026
ddbec07
fix: preserve SPSS frontend grammar edge cases
TonisOrmisson Aug 17, 2026
2426747
feat: compile official SPSS frontend 0.2 plans
TonisOrmisson Aug 17, 2026
26fad1b
fix: reject reserved existing SPSS targets
TonisOrmisson Aug 17, 2026
da006f6
feat: add official transformation apply audit
TonisOrmisson Aug 17, 2026
d605d87
fix: harden transformation audit migration
TonisOrmisson Aug 17, 2026
25fba28
feat: execute official plans atomically in place
TonisOrmisson Aug 17, 2026
502c590
fix: harden atomic official plan execution
TonisOrmisson Aug 17, 2026
f903c8d
fix: canonicalize SQLite physical ownership
TonisOrmisson Aug 17, 2026
b11f335
fix: canonicalize SQLite physical table names
TonisOrmisson Aug 17, 2026
cb3ed21
feat: enforce official SQL and Dolt apply context
TonisOrmisson Aug 17, 2026
dadb3f1
fix: harden MySQL transformation preflight
TonisOrmisson Aug 17, 2026
25f4a83
test: execute every official 0.2 backend case
TonisOrmisson Aug 17, 2026
bbeb55a
feat: claim official transformation and SPSS 0.2 conformance
TonisOrmisson Aug 17, 2026
eff7dd6
fix: require MySQL official transformation CI cases
TonisOrmisson Aug 17, 2026
f1c8596
test: align PostgreSQL slot diagnostic assertion
TonisOrmisson Aug 17, 2026
d5ad3f3
fix: close Codex preflight findings
TonisOrmisson Aug 17, 2026
d6f4a8a
fix: validate recode system missing and targets
TonisOrmisson Aug 17, 2026
c4ef4c0
fix: normalize physical schema preflight aliases
TonisOrmisson Aug 17, 2026
633e635
fix: fold MySQL-family physical table aliases when lower_case_table_n…
TonisOrmisson Aug 19, 2026
b680a08
fix: record transformation apply started_at before operations
TonisOrmisson Aug 19, 2026
79c740a
fix: rebuild MySQL-family audit migration via atomic RENAME TABLE
TonisOrmisson Aug 19, 2026
f148e81
docs: document the new brick/math runtime dependency
TonisOrmisson Aug 19, 2026
9b08fe6
fix: accept Microsoft-variant UUIDs in InPlaceApplyRequest
TonisOrmisson Aug 19, 2026
004a5d3
fix: cache MySQL @@lower_case_table_names per profile instance
TonisOrmisson Aug 19, 2026
fec9db0
fix: defensively assert ddlAtomic in createNumericTarget
TonisOrmisson Aug 19, 2026
a7a2b86
fix: validate SourceSpan offset and position ordering
TonisOrmisson Aug 19, 2026
5998fed
docs: document MySQL CAST AS BINARY choice in hashChecks
TonisOrmisson Aug 19, 2026
e27ff83
test: cover Binary64Value decimal locale normalization
TonisOrmisson Aug 19, 2026
f3907ab
docs: document SpssSyntaxException diagnostic-code contract
TonisOrmisson Aug 19, 2026
f88db5d
docs: clarify InPlaceOperationExecutor::value() return contract
TonisOrmisson Aug 19, 2026
0c55d73
refactor: share official fixture PDO accessors in a trait
TonisOrmisson Aug 19, 2026
2127617
test: drive the official Dolt context-change guard with a real concur…
TonisOrmisson Aug 19, 2026
6ac5ebd
fix: tidy whitespace concat and clarify doltConcurrentSession ownership
TonisOrmisson Aug 19, 2026
21c6896
test: cover SourceSpan ordering validation branches
TonisOrmisson Aug 19, 2026
2e0b241
fix: drop orphan MySQL archive table before rebuilding
TonisOrmisson Aug 19, 2026
9871aaf
docs: clarify when the orphan archive DROP actually fires
TonisOrmisson Aug 19, 2026
438423f
fix: handle MySQL schema aliases and stale staging tables
TonisOrmisson Aug 20, 2026
cf3f89d
fix: isolate staging constraint names and fold SQLite schema aliases
TonisOrmisson Aug 20, 2026
637a6e4
fix: make staging constraint suffixes unique per rebuild
TonisOrmisson Aug 20, 2026
5aa439c
fix: give the real Dolt concurrent commit a committed metadata change
TonisOrmisson Aug 20, 2026
278673e
fix: checkout the expected branch in the concurrent Dolt session
TonisOrmisson Aug 20, 2026
05a5dbb
fix: switch the concurrent Dolt session to an existing branch
TonisOrmisson Aug 20, 2026
83eb24b
test: preserve real concurrent Dolt history across rollback assertions
TonisOrmisson Aug 20, 2026
c729ad1
test: assert the exact Dolt commit observed by the guard
TonisOrmisson Aug 20, 2026
645a728
fix: close CodeRabbit conformance and Unicode edge cases
TonisOrmisson Aug 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ jobs:
coverage: none
extensions: pdo_pgsql, pdo_sqlite, bcmath, mbstring, zlib
- run: composer install --no-interaction --prefer-dist
- run: vendor/bin/phpunit --filter "PostgreSqlSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest"
- run: vendor/bin/phpunit --filter "PostgreSqlSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest|OfficialInPlaceTransformation01Test|OfficialInPlaceTransformation02Test"

mysql-integration:
name: MySQL ${{ matrix.mysql }} integration / PHP ${{ matrix.php }}
Expand All @@ -89,6 +89,7 @@ jobs:
MYSQL_USER: openstatspec
MYSQL_PASSWORD: openstatspec
MYSQL_ROOT_PASSWORD: root
MYSQL_ROOT_HOST: "%"
ports:
- 3306:3306
options: >-
Expand All @@ -100,6 +101,8 @@ jobs:
OPENSTATSPEC_MYSQL_DSN: mysql:host=127.0.0.1;port=3306;dbname=openstatspec;charset=utf8mb4
OPENSTATSPEC_MYSQL_USER: openstatspec
OPENSTATSPEC_MYSQL_PASSWORD: openstatspec
OPENSTATSPEC_MYSQL_ADMIN_USER: root
OPENSTATSPEC_MYSQL_ADMIN_PASSWORD: root
OPENSTATSPEC_EXPECTED_MYSQL_VERSION: ${{ matrix.mysql }}
steps:
- uses: actions/checkout@v7
Expand All @@ -115,7 +118,9 @@ jobs:
coverage: none
extensions: pdo_mysql, pdo_sqlite, bcmath, mbstring, zlib
- run: composer install --no-interaction --prefer-dist
- run: vendor/bin/phpunit --filter "MySqlSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest"
- run: vendor/bin/phpunit --filter "MySqlSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest|OfficialInPlaceTransformation01Test|OfficialInPlaceTransformation02Test"
- name: Require official MySQL 0.1 cases to execute
run: vendor/bin/phpunit tests/Integration/OfficialInPlaceTransformation01Test.php --filter '/mysql/' --fail-on-skipped

mariadb-integration:
name: MariaDB ${{ matrix.mariadb }} integration / PHP ${{ matrix.php }}
Expand Down Expand Up @@ -159,7 +164,7 @@ jobs:
coverage: none
extensions: pdo_mysql, pdo_sqlite, bcmath, mbstring, zlib
- run: composer install --no-interaction --prefer-dist
- run: vendor/bin/phpunit --filter "MariaDbSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest"
- run: vendor/bin/phpunit --filter "MariaDbSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest|OfficialInPlaceTransformation01Test|OfficialInPlaceTransformation02Test"

dolt-integration:
name: Dolt ${{ matrix.dolt.version }} integration / PHP ${{ matrix.php }}
Expand Down Expand Up @@ -213,4 +218,4 @@ jobs:
done
exit 1
- run: composer install --no-interaction --prefer-dist
- run: vendor/bin/phpunit --filter "DoltSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest"
- run: vendor/bin/phpunit --filter "DoltSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest|OfficialInPlaceTransformation01Test|OfficialInPlaceTransformation02Test"
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Task 6 report: bind and compile SPSS Frontend 0.2

## Status

Complete in the initial Task 6 commit plus one review-fix commit. Neither was pushed.

## Implementation

- Added strict `SpssFrontendRequest::fromArray()` validation for the exact Frontend 0.2 request shape, its ordered non-empty input schema, and every optional variable metadata field.
- Added source identity hashing that normalizes only CRLF and bare CR to LF before SHA-256 and remains available independently of parsing or binding success.
- Added an ASCII-case-insensitive evolving `SchemaState` that retains exact catalog spelling, resolves earlier created variables in later commands, and binds multi-variable `RECODE` sources and targets against pre-command state.
- Replaced the legacy binder/compiler boundary with typed bound nodes for recode, compute, conditional assignment, labels, formats, measurement levels, and execute.
- Compiled ordered bound statements one-for-one into the official Plan operation types and selected Plan 0.1 or 0.2 from each operation's minimum contract.
- Converted every decimal literal directly through `DecimalBinary64` into canonical `Binary64Value` bits, including exact rounding and positive-zero canonicalization, without a host-float parser intermediate.
- Added command, group, variable, expression, rule, value, result, and unsupported-command spans needed for stable binding diagnostics while preserving all source ordering.
- Added manifest-driven coverage for all 44 official Frontend 0.2 cases, exact source/plan hashes, exact canonical plans, referenced official Frontend 0.1 golden plans, plan contracts, stable failure codes, and non-null source spans.
- Added focused request, evolving-schema, simultaneous-recode, exact-binary64, diagnostic-code/span, invalid-empty-recode, and unsupported-legacy-command regressions.
- Review fix: existing reserved `__` COMPUTE and IF targets now fail in the binder with `reserved_target_name` at the exact target span, matching strict Plan validation; every successful official frontend plan is also decoded again through strict `PlanCodec` and compared canonically.

## TDD evidence

### RED

The official manifest test was added before the request/compiler implementation:

```text
vendor/bin/phpunit tests/Frontend/Spss/Conformance
44 errors: SpssFrontendRequest::fromArray() did not exist.
```

The focused compiler tests were then added before the new facade and binder:

```text
vendor/bin/phpunit tests/Frontend/Spss/SpssCompilerTest.php
25 errors: the strict request constructor and compile(request) boundary did not exist.
```

Further tests independently reproduced missing direct-constructor value-label validation, invalid ELSE-only RECODE output, loss of exact numeric source tokens, and zero-width unsupported-command spans before each corresponding fix.

The review regressions were added before the binder fix:

```text
vendor/bin/phpunit tests/Frontend/Spss/SpssCompilerTest.php --filter 'reserved existing'
FAILURES! Tests: 2, Assertions: 2, Failures: 2.
Both existing reserved targets unexpectedly compiled.
```

The root cause was that `validateTargetName()` ran only on the missing-target COMPUTE branch and never on IF; both existing-target paths could therefore emit a plan rejected by strict `PlanCodec`.

### GREEN

```text
vendor/bin/phpunit tests/Frontend/Spss/Conformance tests/Frontend/Spss/SpssCompilerTest.php
OK (76 tests, 301 assertions)
```

The 44 manifest cases are included in that count.

## Self-review

- **Request boundary:** exact root/schema/value object members are enforced; strings are valid UTF-8; ordered variables are non-empty; storage, typed values, formats, and measurement metadata are validated before parsing.
- **Source identity:** hashing is isolated from parsing and changes only line-ending bytes.
- **Resolution and ordering:** matching is ASCII-case-insensitive, emitted names preserve schema spelling, ambiguous names fail closed, new names retain source spelling, commands/groups/variables/rules/boolean operands remain ordered, and parallel RECODE targets are published only after the whole command binds.
- **Type and target semantics:** COMPUTE creates or replaces numeric targets; IF requires an existing numeric target; numeric-only expressions/formats, value-label types, ranges, result homogeneity, reserved names, fresh targets, and string limitations fail at the narrowest available source span.
- **Plan boundary:** emitted operations are official immutable Plan types; every successful official plan round-trips through strict `PlanCodec`; unchanged subsets stay Plan 0.1 and any 0.2 operation promotes the whole ordered plan to 0.2.
- **Failure behavior:** request, syntax, binding, and unsupported-command failures occur before mutation and carry the stable code/path/span contract.
- **Dolt/global constraints:** Task 6 changes only the pure frontend and tests. It adds no SQL mutation, dataset/table copy, derived dataset, staging/snapshot/rollback artifact, audit/version row, or Dolt operation; Dolt remains the sole version-history layer.
- **Scope:** the staged diff contains only Task 6 frontend source/tests and this report; unrelated changes were not included.

## Verification

```text
vendor/bin/phpunit tests/Transformation/Conformance tests/Frontend/Spss
OK (181 tests, 729 assertions)
```

```text
composer analyse
[OK] No errors
```

The CRLF-normalized staged-tree pre-commit hook ran the complete `composer check` gate successfully:

```text
composer validate --strict: valid
PHP lint: all source/test/tool files valid
php-cs-fixer: Found 0 of 213 files that can be fixed
PHPStan: [OK] No errors
PHPUnit: 408 tests, 2393 assertions, 30 skipped; exit 0
```

`git diff --cached --check` also passed.

The repository-wide working-tree `composer style` command was also run. It exits 8 because 133 of 213 repository files have pre-existing CRLF line endings; the normalized staged-tree hook runs the same style check against LF-normalized Git blobs and is the authoritative clean gate for the committed tree.

## Notes

- The Frontend 0.2 manifest's two `expected_plan_case_0_1` identifiers refer to cases in `spss-syntax-frontend-0.1.json`; they do not exist in `transformation-plan-0.1.json`. The conformance test therefore resolves those official references from their actual owning manifest rather than copying the golden plans.
- Verification used installed PHP 8.5.9 with Composer's declared PHP 8.4.1 platform floor.
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Task 7 report: explicit transformation audit migration and apply request

## Implementation

- Added `InPlaceApplyRequest` with fail-fast alias, canonical UUID, lowercase SHA-256, non-empty actor, and paired Dolt-context validation.
- Added explicit schema-v4 `TransformationAuditMigrator` for SQLite, PostgreSQL, MySQL, MariaDB, and Dolt. SQLite and MySQL-family profiles rebuild the one logical audit table through a staging relation; SQLite does so transactionally, while MySQL-family profiles use an atomic multi-table `RENAME TABLE` swap before dropping the archive.
- Added success-only `TransformationAuditWriter`, which requires the caller's open apply transaction and stores canonical plan identity, source identity, actor, target identity, operation count, timestamps, and nullable Dolt evidence.
- Wired `SpssAdapter::migrateCatalog()` to run the audit migration after normative table creation and before marking schema version 4.
- Preserved official 0.1 audit rows while allowing both 0.1 and 0.2 binding contracts. No dataset/data table, snapshot, staging, rollback, or recovery relation was added.
- MySQL-family hashes now use a MySQL 8.4/MariaDB/Dolt-compatible exact lowercase-hex check without `REGEXP BINARY`.
- MySQL-family contract upgrades copy rows into a uniquely constrained staging table and atomically swap it into place with `RENAME TABLE`. Stale staging/archive relations are removed before retry, so an interrupted rebuild remains recoverable without manual cleanup.
- Schema version 4 can be marked or accepted as ready only when the contiguous 1..4 migration history and readable compact audit relation both exist. Normative migration recording is private and limited to versions 1..3; the audit migrator alone records version 4.

## TDD evidence

- RED: `vendor/bin/phpunit tests/Transformation/Audit/TransformationAuditMigratorTest.php` reported 6 missing-class errors across 7 tests.
- GREEN: the final focused command `vendor/bin/phpunit tests/Transformation/Audit/TransformationAuditMigratorTest.php tests/Integration/ServerCatalogMigrationTest.php` passed 10 tests and 54 assertions; the only skip was the unconfigured server integration profile.
- Wiring mutation check: removing the adapter's explicit migrator call made `testAdapterMigrationRecordsFreshVersionsOneThroughFourExactlyOnce` fail because marker 4 was absent; restoring the call returned the suite to green.
- Review RED: the two new ownership regressions both failed because `markCurrentVersion()` and readiness accepted an audit-less v4 claim.
- Review RED: the configured MySQL 8.4 integration test failed during fresh migration with SQLSTATE 3995 from `REGEXP BINARY`.
- Review GREEN: ownership regressions cover both the reported `identity=4 / migrations=1..3 / audit=absent` state and a forged marker 4 with no audit table. The server test also rejects uppercase source/plan hashes and simulates recovery from a missing MySQL-family contract check.
- Review follow-up RED/GREEN: a second MySQL run against the same migrated schema first failed because the legacy fixture hard-coded the pre-upgrade check name; introspecting the current check made two consecutive runs pass.

## Verification

- `composer analyse`: `[OK] No errors`.
- Working-tree `composer style`: exits 8 because 126 pre-existing CRLF files are reported as fixable.
- CRLF-normalized staged-tree `composer check`: Composer validation, PHP lint, style, PHPStan, and PHPUnit all passed; PHPUnit ran 420 tests with 2,454 assertions and 30 environment skips.
- Focused SQLite/unconfigured-server command passed 35 tests and 144 assertions with the one expected unconfigured-server skip.
- Live MySQL 8.4.9 integration passed twice consecutively against one schema, each run with 1 test and 13 assertions, including fresh/idempotent DDL, legacy upgrade, exact hash rejection, and missing-check retry recovery.
- Live MariaDB 11.6.2 DDL probe passed fresh migration, lowercase acceptance, uppercase source/plan rejection, missing-check retry recovery, and exactly one v4 marker. This locally available version is outside the adapter's claimed MariaDB families, so the probe invoked the migrator directly rather than misrepresenting it as a supported full adapter run.
- `git diff --cached --check`: passed.

## Self-review

- SQLite v0.1 migration preserves the row and logical table name, verifies foreign keys before commit, leaves no `_v04` residue, and rolls back to the old table on copy failure.
- PostgreSQL changes the old contract check in one native transaction; MySQL-family/Dolt DDL uses an explicit staging/copy/rename path, is retryable and idempotent, and is never invoked by the audit writer or inside an apply transaction.
- The writer has no failure-row API and performs no migration, dataset copy, table copy, Dolt commit, branch operation, or recovery/version write.
- Fresh catalogs record versions 1 through 4 exactly once; v3 ownership recognition remains backward compatible, while v4 marker recognition requires the compact audit relation.

## Environment note

No PostgreSQL or Dolt service was available locally. Their configured-server integration branches remain covered by the same portable migration test; PostgreSQL's check replacement remains transactionally atomic, while Dolt follows the live-probed MySQL-family DDL path.
Loading