Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
uses: actions/checkout@v7
with:
repository: OpenStatSpec/specification
ref: cd8f198c68b849eb8ed018a894670a0904c2181d
ref: 864e84479f554b8ee250ffed44c4dfb963750d4a
path: openstatspec-specification
- uses: shivammathur/setup-php@v2
with:
Expand Down Expand Up @@ -63,7 +63,7 @@ jobs:
uses: actions/checkout@v7
with:
repository: OpenStatSpec/specification
ref: cd8f198c68b849eb8ed018a894670a0904c2181d
ref: 864e84479f554b8ee250ffed44c4dfb963750d4a
path: openstatspec-specification
- uses: shivammathur/setup-php@v2
with:
Expand Down Expand Up @@ -110,7 +110,7 @@ jobs:
uses: actions/checkout@v7
with:
repository: OpenStatSpec/specification
ref: cd8f198c68b849eb8ed018a894670a0904c2181d
ref: 864e84479f554b8ee250ffed44c4dfb963750d4a
path: openstatspec-specification
- uses: shivammathur/setup-php@v2
with:
Expand Down Expand Up @@ -156,7 +156,7 @@ jobs:
uses: actions/checkout@v7
with:
repository: OpenStatSpec/specification
ref: cd8f198c68b849eb8ed018a894670a0904c2181d
ref: 864e84479f554b8ee250ffed44c4dfb963750d4a
path: openstatspec-specification
- uses: shivammathur/setup-php@v2
with:
Expand Down Expand Up @@ -196,14 +196,17 @@ jobs:
OPENSTATSPEC_DOLT_PASSWORD: openstatspec
OPENSTATSPEC_DOLT_ADMIN_USER: root
OPENSTATSPEC_DOLT_ADMIN_PASSWORD: root
OPENSTATSPEC_DOLT_READ_ONLY_ADMIN_DSN: mysql:host=127.0.0.1;port=3306;charset=utf8mb4
OPENSTATSPEC_DOLT_READ_ONLY_ADMIN_USER: root
OPENSTATSPEC_DOLT_READ_ONLY_ADMIN_PASSWORD: root
OPENSTATSPEC_EXPECTED_DOLT_VERSION: ${{ matrix.dolt.version }}
steps:
- uses: actions/checkout@v7
- name: Checkout OpenStatSpec specification fixtures
uses: actions/checkout@v7
with:
repository: OpenStatSpec/specification
ref: cd8f198c68b849eb8ed018a894670a0904c2181d
ref: 864e84479f554b8ee250ffed44c4dfb963750d4a
path: openstatspec-specification
- uses: shivammathur/setup-php@v2
with:
Expand All @@ -218,4 +221,4 @@ jobs:
done
exit 1
- run: composer install --no-interaction --prefer-dist
- run: vendor/bin/phpunit --filter "DoltSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest|OfficialInPlaceTransformation01Test|OfficialInPlaceTransformation02Test"
- run: vendor/bin/phpunit --filter "DoltReadOnlyExportTest|DoltSpssRoundTripTest|ServerVersionEvidenceTest|ServerCatalogMigrationTest|StrictSpssDictionaryConformanceTest|OfficialSpssConformanceManifestTest|InPlaceTransformationServiceTest|OfficialInPlaceTransformation01Test|OfficialInPlaceTransformation02Test"
34 changes: 34 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,38 @@
# Changelog

## [0.7.0] - Unreleased

### Changed

- Pin capabilities and all CI fixtures to released OpenStatSpec specification
`v0.5.0` at immutable commit `864e84479f554b8ee250ffed44c4dfb963750d4a`.
Report `specification_status: released` and select the optional
`database_io_policy: openstatspec-database-io-v1` contract.
- **Breaking:** SAV/ZSAV export is database-read-only, including failures; it
no longer creates operation or fidelity audit records. Initialize or upgrade
the catalogue with `SpssAdapter::migrateCatalog()` on a write-capable deployment
connection before export. Unready catalogues fail with
`catalog_migration_required`; export never initializes or migrates them.
- Export authoritative normative metadata, including transformed labels and
numeric format defaults. Publish output only after a successful temporary-file
write, preserving an existing destination on writer failure. Injected engines
receive the temporary path and must use the Dataset's target format.
- **Breaking:** Default Dolt writes now support exactly 2.2.2 and 2.2.3, not
the previous patch range. No external declaration files are required; unknown
versions (including 2.2.4) fail before mutation. Read-only exports verify
server identity without requiring a supported write version.
- Keep Transformation Plan 0.1/0.2, SPSS Syntax Frontend 0.2, and In-Place
Transformation 0.1/0.2 claims unchanged. Optional 0.3 contracts in the new
specification are not implemented or claimed.
- Run the real SELECT-only Dolt export regression in CI on both exact supported
versions, checking SAV/ZSAV output, failure safety, and unchanged roots/history.

### Removed

- **Breaking:** Remove `SpssExportResult::operationId` and its constructor
argument. Export creates no `operation_id`; diagnostics and accepted loss
codes remain available in the result. Import operation IDs are unchanged.

## [0.6.0] - 2026-08-21

### Added
Expand Down Expand Up @@ -89,6 +122,7 @@
specification v0.1.0 at commit `d287c2cde9ade71f04e27dd012caec876901aed5`.

[Unreleased]: https://github.com/OpenStatSpec/php/compare/v0.6.0...HEAD
[0.7.0]: https://github.com/OpenStatSpec/php/compare/v0.6.0...v0.7.0
[0.6.0]: https://github.com/OpenStatSpec/php/compare/v0.5.0...v0.6.0
[0.5.0]: https://github.com/OpenStatSpec/php/compare/v0.4.0...v0.5.0
[0.4.0]: https://github.com/OpenStatSpec/php/compare/v0.3.0...v0.4.0
Expand Down
63 changes: 53 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,19 @@ It imports an unencrypted SPSS `.sav` or `.zsav` dataset into a relational datab

## Status

PHP v0.7.0 targets released OpenStatSpec specification `v0.5.0` at immutable
commit `864e84479f554b8ee250ffed44c4dfb963750d4a`
(`specification_status: released`). Composer derives the package version from its
Git tag; it is independent of the specification version.

This is an early reference implementation. Its round-trip contract is **semantic**, not byte-identical: supported cases, order, variables, values, dictionary metadata and technical metadata are preserved; compression layout, timestamps and other writer-specific bytes are not promised.

SQLite, PostgreSQL 17.x/18.x, MySQL 8.4.x/9.7.x, MariaDB
11.4.x/11.8.x/12.3.x and Dolt 2.2.x with the explicit
`>=2.2.2,<2.3.0` floor/range are implemented PDO profiles.
11.4.x/11.8.x/12.3.x and exact Dolt 2.2.2/2.2.3 are implemented PDO write profiles.
The SAV/ZSAV 1.0 capability declaration selects
`database_io_policy: openstatspec-database-io-v1`: exports are database-read-only,
and default Dolt writes use the packaged exact-version policy without external
declaration files. Unknown patches (including 2.2.4) fail before mutation.
Server-family claims are conservative compatibility policies; CI records exact
evidence at PostgreSQL 17.10/18.4, MySQL 8.4.11/9.7.2 and MariaDB
11.4.12/11.8.8/12.3.2, and Dolt 2.2.2/2.2.3. Each service job verifies its live normalized product
Expand All @@ -24,7 +32,7 @@ version before the run counts as evidence.
| PostgreSQL / `postgresql` | 17.x and 18.x | 17.10 and 18.4 |
| MySQL / `mysql` | 8.4.x and 9.7.x | 8.4.11 and 9.7.2 |
| MariaDB / `mariadb` | 11.4.x, 11.8.x, and 12.3.x | 11.4.12, 11.8.8, and 12.3.2 |
| Dolt / `dolt` | 2.2.x family with `>=2.2.2,<2.3.0` | 2.2.2 and 2.2.3 |
| Dolt / `dolt` | Writes: exact 2.2.2 and 2.2.3 only | 2.2.2 and 2.2.3 |

The PHP SQLite core profile remains `>=3.24.0,<4.0.0`. The Python adapter's
optional Transformation Workflow has its own narrower `>=3.35.0,<4.0.0`
Expand Down Expand Up @@ -66,9 +74,25 @@ $import = $adapter->import('/data/survey.zsav', 'survey_2026');
// SpssImportResult: operationId, datasetName, caseCount, diagnostics

$export = $adapter->export('survey_2026', '/data/survey-export.sav');
// SpssExportResult: operationId, datasetName, caseCount, diagnostics, allowLoss
// SpssExportResult: datasetName, targetPath, caseCount, diagnostics, allowLoss
```

Export needs only read access to an already migrated catalogue and its wide
tables. Initialize or upgrade the catalogue with `SpssAdapter::migrateCatalog()`
using a write-capable deployment connection before using a read-only exporter;
an uninitialized or outdated catalogue fails with `catalog_migration_required`.
It reads authoritative normative metadata directly and never initializes,
migrates, synchronizes compatibility tables, or writes database audit records—even
on failure. `SpssExportResult` no longer has `operationId`; export creates no
`operation_id`. Diagnostics and accepted loss codes remain in the result.
The destination is replaced only after successful writing to a temporary file in
its directory; a writer failure preserves an existing destination. Injected
engines receive that temporary path and must use the Dataset's target format.

Server identity is still checked for reads, but export does not require a packaged
write-version claim or any external support declaration. Import, migration and
transformation write safeguards and tested server versions are unchanged.

Use `GuardedImportSpssEngine` when an engine must read from an ephemeral
descriptor while the adapter and database receive only a logical source path:

Expand Down Expand Up @@ -110,13 +134,15 @@ $export = $adapter->export(
);
```

Pass only loss codes consciously accepted for that conversion. `operation_catalog` records successful and failed imports/exports; `fidelity_event_catalog` records emitted diagnostics. A failed preflight is therefore auditable even when it created no dataset. Each operation also records the selected SPSS engine package and Composer version in engine_details.
Pass only loss codes consciously accepted for that conversion. `operation_catalog` records journaled imports; `fidelity_event_catalog` records their emitted diagnostics. Import preflight failures after catalogue readiness are auditable even when no dataset was created. Each journaled import also records the selected SPSS engine package and Composer version in `engine_details`. Export returns diagnostics or throws without recording an operation.

## Transformation API

The adapter claims official Transformation Plan 0.1/0.2, SPSS Syntax Frontend
0.2, and In-Place Transformation 0.1/0.2 conformance. Compile an alias-based
frontend request, then bind that alias to the existing dataset at apply time:
frontend request, then bind that alias to the existing dataset at apply time.
The specification pin does not claim support for optional Transformation Plan,
SPSS Syntax Frontend, or In-Place Transformation 0.3.

```php
use OpenStatSpec\Frontend\Spss\Request\SpssFrontendRequest;
Expand Down Expand Up @@ -225,7 +251,8 @@ Run the local gate before committing:

```bash
composer install
composer check
# Checkout specification v0.5.0 at 864e84479f554b8ee250ffed44c4dfb963750d4a first.
OPENSTATSPEC_SPECIFICATION_DIR=/path/to/exact-specification-checkout composer check
```

Install the tracked pre-commit hook once per clone:
Expand All @@ -241,9 +268,25 @@ SAV and ZSAV integration round trips against exact PostgreSQL 17.10/18.4,
MySQL 8.4.11/9.7.2, MariaDB 11.4.12/11.8.8/12.3.2, and Dolt
2.2.2/2.2.3.
Those checks use their PDO drivers and php-spss V3 read/write paths, not only
DDL snapshots. Family policies remain runtime claims and exact patches are CI
evidence points; Dolt's 2.2.x family claim additionally has an explicit 2.2.2
minimum and 2.3.0 exclusive upper bound.
DDL snapshots. Other engines retain family policies; the packaged Dolt write
list is limited to the exact 2.2.2 and 2.2.3 versions tested by existing CI.
It does not infer evidence or support for any other patch.

CI runs `DoltReadOnlyExportTest` on both exact Dolt 2.2.2 and 2.2.3 with
root/root admin credentials. Locally opt-in SELECT-only export coverage
(also usable on Dolt 2.3.0) creates and
removes its own unique test database and user. It checks SAV/ZSAV output, failure
safety and unchanged working/staged roots and history, without expanding write
support:

```bash
OPENSTATSPEC_DOLT_READ_ONLY_ADMIN_DSN='mysql:host=127.0.0.1;port=13387;charset=utf8mb4' \
vendor/bin/phpunit tests/Integration/DoltReadOnlyExportTest.php
```

The test defaults to `root` with an empty password; optional
`OPENSTATSPEC_DOLT_READ_ONLY_ADMIN_USER` and
`OPENSTATSPEC_DOLT_READ_ONLY_ADMIN_PASSWORD` override these test credentials.

## Contributing

Expand Down
20 changes: 12 additions & 8 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ The public `SpssAdapter` chooses a profile from the PDO driver:
| `sqlite` | SQLite | in-memory unit and round-trip suite |
| `pgsql` | PostgreSQL | 17.x/18.x claim; live 17.10/18.4 SAV/ZSAV CI evidence |
| `mysql` | MySQL/MariaDB | MySQL 8.4.x/9.7.x and MariaDB 11.4.x/11.8.x/12.3.x claims; live 8.4.11/9.7.2 and 11.4.12/11.8.8/12.3.2 CI evidence |
| `mysql` | Dolt | 2.2.x claim with `>=2.2.2,<2.3.0` range; live 2.2.2/2.2.3 SAV/ZSAV CI evidence; detected by `@@version_comment` plus `DOLT_VERSION()` |
| `mysql` | Dolt | Packaged writes on exact 2.2.2/2.2.3 only under `openstatspec-database-io-v1` with SAV/ZSAV 1.0; live CI evidence for those exact versions; detected by `@@version_comment` plus `DOLT_VERSION()` |

Every profile creates the same logical strict-wide layout and metadata catalogue. Physical SQL types, identifier limits and capability preflight are profile-specific. A source that cannot be represented must be rejected before an incomplete substitute is created.

Expand All @@ -50,17 +50,21 @@ The catalogue is the semantic dictionary for the physical data table. It records
- typed ordered value labels and all supported user-missing forms;
- documents and technical file metadata;
- file/variable attributes, variable sets, multiple-response sets and roles; and
- import/export operation records and fidelity events.
- import operation records and fidelity events.

`SpssImportResult` carries an operation ID, dataset name, case count and diagnostics. `SpssExportResult` carries the same operational evidence plus explicitly accepted loss codes.
`SpssImportResult` carries an operation ID, dataset name, case count and diagnostics. `SpssExportResult` carries dataset name, target path, case count, diagnostics and explicitly accepted loss codes, but no operation ID.

Export reads the singular normative catalogue and existing physical wide table directly. The compatibility catalogue is not synchronized or used as an alternative semantic dictionary; only writer-supported technical provenance fields without normative counterparts are read from `file_technical_metadata`. No export path writes database state, including validation failures, missing catalogues and writer failures. Reads identify the server without requiring a packaged write-version claim. Write safeguards and the tested version matrix remain unchanged.

The writer receives a temporary filesystem path in the destination directory and the target format in the typed Dataset. Only a successful write is published by rename; failure leaves an existing destination intact and removes temporary output. No database staging, copies, temporary tables or audit entries are used. Export does not manage a read transaction; callers needing a consistent snapshot during concurrent edits must supply one.

## Round trip and fidelity policy

The objective is semantic equivalence for represented features, not byte identity. Original compression blocks, product-specific bytes, timestamps and other incidental writer representation are not promised.

An exporter may emit machine-readable `FidelityDiagnostic` values for a known loss boundary. Export is fail-closed: it refuses to write until each emitted diagnostic code is present in the caller's `allowLoss` list. That makes intentional lossy conversion explicit rather than silently producing a downgraded file.

The singular OpenStatSpec `operation` and `fidelity_event` tables are the authoritative audit record. Legacy `operation_catalog` and `fidelity_event_catalog` remain mirrored only for transition compatibility. A failed preflight creates a failed `operation` and at least one `fidelity_event` with a null dataset reference, direction, event code, source item and timestamp.
The singular OpenStatSpec `operation` and `fidelity_event` tables are the authoritative audit record. Legacy `operation_catalog` and `fidelity_event_catalog` remain mirrored only for transition compatibility. An import preflight failure after catalogue readiness creates a failed `operation` and at least one `fidelity_event` with a null dataset reference, direction, event code, source item and timestamp. Export never writes these audit tables.

## SAV/ZSAV boundary

Expand Down Expand Up @@ -120,9 +124,9 @@ This package requires neither Yii2 nor Laravel. Applications supply a PDO connec
Every normal import writes the singular OpenStatSpec catalogue tables from the
normative schema: `dataset`, `variable`, value-label and missing-rule tables,
attributes, `document`, variable/multiple-response sets, `operation`, and
`fidelity_event`. The older plural tables are a compatibility read model for
existing exports; they are not the standard contract for a newly imported
dataset.
`fidelity_event`. The older plural tables remain for compatibility and explicit
legacy migration; they are not the standard contract for a newly imported
dataset or the semantic source for public exports.

Before upgrading an existing database explicitly, run:

Expand All @@ -138,7 +142,7 @@ each exportable legacy dataset into the singular standard tables and provisions
the compact official transformation-apply audit relation.

A completely empty dedicated namespace is initialized automatically on its
first ordinary import or export attempt. An existing catalogue is never
first ordinary import attempt. Export never initializes even an empty namespace. An existing catalogue is never
upgraded implicitly: when its identity or validated pre-identity state is older,
ordinary use fails with `catalog_migration_required` before journal or schema
mutation, and deployment must call `migrateCatalog()` explicitly.
Expand Down
Loading