diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..60a50cb --- /dev/null +++ b/.gitattributes @@ -0,0 +1,8 @@ +# Vendored schemas and generated spec HTML are checked byte-for-byte: the drift +# gate compares schema/ with the pinned forge-cli release, and generate-docs.py +# --check verifies the "frozen" sha256 map in scripts/schema-versions.json. An +# end-of-line conversion on checkout (core.autocrlf on Windows) would change +# those bytes, so git must never convert them. +schema/** -text +specs/** -text +schema-diffs/** -text diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index 9eab28e..c3b9dca 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -1,14 +1,36 @@ # The gate that makes FLUID a standard rather than a schema with a docs site. # -# Three jobs, deliberately separate so a red build says WHICH promise broke: +# Seven jobs, deliberately separate so a red build says WHICH promise broke: # -# meta proves the other two gates can fail. Runs first: a green +# meta proves the gates below can fail. Runs first: a green # conformance run means nothing if the runner cannot go red. # conformance runs the vector corpus against the published schemas, and # again against the reference implementation (forge-cli) so # the corpus and the engine cannot drift apart unnoticed. # compat enforces the backward-compatibility promise the README and # every release note make. +# drift the vendored schemas are byte-identical to the ones in the +# forge-cli release PINNED in scripts/schema-versions.json +# (upstream.ref and upstream.commit). Pinned, so a new upstream +# release cannot turn an unrelated pull request red; moving +# the pin is a reviewed change. This used to live in +# schema-sync.yml, which compared against forge-cli `main`, +# failed every scheduled run from 22 June to 31 August 2026, +# and was then disabled by GitHub for inactivity, so later +# pull requests merged with no drift check at all. +# drift-latest ADVISORY. Warns when forge-cli has published a release after +# the pinned one. It cannot fail the run: its step is +# continue-on-error, and it is not a required check. +# generated specs/ (the HTML reference) and schema-diffs/ are what the +# generators produce from schema/ today, and every other file +# under schema/ and specs/ is frozen and unchanged. +# docs-links internal absolute links in docs/ carry the /fluid/ base. +# Moved here from link-check.yml for the same reason as drift. +# +# GitHub's inactivity rule disables only workflows that have a `schedule` +# trigger, and it disables the whole workflow, pull-request checks included. +# That is why this workflow has no schedule, and why per-PR gates live here +# rather than beside a weekly job. name: conformance @@ -19,8 +41,11 @@ on: - "schema/**" - "tests/**" - "conformance/**" - - "scripts/check-compat.py" - - "scripts/compat-waivers.txt" + - "scripts/**" + - "specs/**" + - "schema-diffs/**" + - "generate-docs.py" + - "generate-schema-diffs.py" - ".github/workflows/conformance.yml" # NO `paths:` filter here, and that is the point. A path-filtered check # cannot be a REQUIRED check: on a pull request that does not match, the @@ -84,11 +109,12 @@ jobs: # `run.py` answers "do the cases pass" — a corpus of zero cases answers # that perfectly. `mutation_coverage.py` answers the harder question: # delete a constraint from the schema and see whether any case notices. - # It has been in this repo, runnable, and wired into nothing, so the + # It used to sit in this repo, runnable and wired into nothing, so the # figure it produces could fall to zero between releases in silence. # - # The floor is the CURRENT number rounded down (40.7% -> 40), not a - # target. That is the convention mutation-testing gates converge on + # The floor is the CURRENT number rounded down, not a target: the + # lowest version is 0.7.5 at 41.0% (tests/README.md has the figures). + # That is the convention mutation-testing gates converge on # (Stryker, mutmut): set the break threshold where you actually are, so # it catches a regression on day one — an aspirational threshold goes red # immediately and gets deleted by day three, which is worse than no gate. @@ -136,15 +162,93 @@ jobs: # is what makes a local run and a CI run mean the same thing. - run: pip install "jsonschema[format]>=4.22" - # Pre-0.7.1 schemas predate the compatibility promise and already - # diverge from forge-cli at the byte level (see schema-sync.yml), so the - # gate runs over the range where the promise was actually made. + # The default run compares every adjacent pair from 0.7.1, the first + # version the compatibility promise covers, to the newest published + # schema, so a newly vendored version is covered without editing this + # file. Earlier pairs predate the promise (`--all-history` shows them). - name: every release keeps its backward-compatibility promise + run: python3 scripts/check-compat.py + + drift: + name: schemas match the reference release + runs-on: ubuntu-latest + needs: meta + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + # Compares every version listed in scripts/schema-versions.json byte for + # byte with the forge-cli commit recorded there (upstream.commit, the + # release upstream.ref), plus that release's preview set and its latest + # stable version. A pinned release, so the result depends only on this + # repository's contents: a new forge-cli release changes nothing here + # until someone edits the pin in a pull request. Not `main`: main carries + # unreleased schema edits nobody can install yet. + # The token only lifts the API rate limit; forge-cli is public. It is + # sent to api.github.com only. + - name: vendored schemas equal the pinned forge-cli release + env: + GITHUB_TOKEN: ${{ github.token }} + run: python3 scripts/check-schema-drift.py + + drift-latest: + name: newer forge-cli release (advisory) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + # Asks the one question the `drift` job deliberately does not: has + # forge-cli released since the pin? It prints what moving would involve + # and annotates the run with a warning. continue-on-error makes the job + # green whatever the script exits with (1 = a newer release exists, + # 2 = GitHub could not be reached), so this can never block a merge, + # and it is not, and should not become, a required check. + - name: has forge-cli released after the pinned release + continue-on-error: true + env: + GITHUB_TOKEN: ${{ github.token }} + run: python3 scripts/check-schema-drift.py --latest + + generated: + name: generated files are current + runs-on: ubuntu-latest + needs: meta + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + # The renderer is pinned exactly: the HTML is committed, so its version + # is part of the build's input (see scripts/requirements-docs.txt). It is + # installed from the hash-locked file, so a package replaced on PyPI + # after the lock was made is refused rather than run. + - run: pip install --require-hashes -r scripts/requirements-docs.lock + - name: specs// matches a fresh render of every synced schema, and no other file has moved + run: python3 generate-docs.py --check + - name: schema-diffs/ matches a fresh generation + run: python3 generate-schema-diffs.py --check + + docs-links: + name: docs links carry the site base + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Verify all internal absolute links carry the /fluid/ base run: | - set -euo pipefail - for pair in "0.7.1 0.7.2" "0.7.2 0.7.3" "0.7.3 0.7.4" "0.7.4 0.7.5"; do - set -- $pair - echo "::group::$1 -> $2" - python3 scripts/check-compat.py --from "$1" --to "$2" - echo "::endgroup::" - done + # VuePress base is /fluid/. Sidebar/navbar links auto-prefix, but + # body markdown links do NOT -- an absolute link like [x](/schema/foo) + # renders as and 404s on the live site. + PREFIXES='guide|concepts|schema|examples|how-to|releases|vision|contributing|deck|specs' + + if grep -rIn -E "\]\(/(${PREFIXES})[^)]*\)" docs/ --include="*.md" \ + | grep -v "(/fluid/" ; then + echo "" + echo "::error::Found internal absolute markdown links missing the /fluid/ base prefix." + echo "::error::These render as and 404 on the live site." + echo "::error::Fix: rewrite [text](/schema/foo) -> [text](/fluid/schema/foo) (or use a relative path)." + exit 1 + fi + echo "All internal absolute body links carry the /fluid/ base." diff --git a/.github/workflows/deploy-docs.yml b/.github/workflows/deploy-docs.yml index 59c4036..8340204 100644 --- a/.github/workflows/deploy-docs.yml +++ b/.github/workflows/deploy-docs.yml @@ -5,10 +5,9 @@ on: branches: [main] workflow_dispatch: +# Nothing is granted by default. Each job asks for exactly what it uses. permissions: contents: read - pages: write - id-token: write concurrency: group: pages @@ -17,15 +16,24 @@ concurrency: jobs: build: runs-on: ubuntu-latest + # configure-pages reads the Pages settings, which needs pages: read. The + # build job renders and uploads an artifact; it publishes nothing, so it + # has no write permission and no OIDC token. + permissions: + contents: read + pages: read steps: - name: Checkout uses: actions/checkout@v4 with: fetch-depth: 0 - # Regenerate the latest schema HTML so specs// exists before - # the asset-sync step mirrors it into the published site. The fluid - # generators expect a venv at /.venv (see schema-sync.yml). + # Render the HTML reference for every synced schema version (the list + # and the latest stable version live in scripts/schema-versions.json) + # with the exactly-pinned renderer, so specs/ is complete before the + # asset-sync step mirrors it into the published site. On a current + # main this reproduces the committed files byte for byte; the + # conformance workflow's `generated` job checks that on every PR. - name: Set up Python uses: actions/setup-python@v5 with: @@ -33,9 +41,7 @@ jobs: - name: Generate schema docs (specs/) run: | - venv_dir="$(dirname "$GITHUB_WORKSPACE")/.venv" - python3 -m venv "$venv_dir" - "$venv_dir/bin/pip" install --quiet json-schema-for-humans + pip install --require-hashes -r scripts/requirements-docs.lock python3 generate-docs.py - name: Setup Node.js @@ -64,6 +70,11 @@ jobs: path: docs/.vuepress/dist deploy: + # The only job that publishes. It does not check out or build anything, so + # it needs neither contents nor pages: read. + permissions: + pages: write + id-token: write environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} diff --git a/.github/workflows/link-check.yml b/.github/workflows/link-check.yml index 1936c19..aaea90f 100644 --- a/.github/workflows/link-check.yml +++ b/.github/workflows/link-check.yml @@ -1,25 +1,16 @@ name: Link Check -# Catches broken docs links before they ship. +# External URLs in docs/, checked weekly (lychee), opening an issue when any +# break. Scheduled only: it does not run on pull requests. # -# 1. base-prefix-check — fast shell regex catching the bug class where an -# absolute body-markdown link like [x](/schema/foo) is NOT auto-prefixed -# with the /fluid/ base by VuePress, so it 404s in the rendered HTML. -# 2. external — weekly cron (skipped on PRs) checking external URLs. +# The per-PR base-prefix check that used to live here (absolute body links +# missing the /fluid/ base) is now the `docs-links` job in conformance.yml. +# This workflow has a `schedule` trigger, so GitHub disables it after 60 days +# without repository activity -- and it was disabled that way, taking the +# pull-request check down with it. A disabled scheduled workflow is re-enabled +# from the Actions tab or with `gh workflow enable "Link Check"`. on: - push: - branches: [main] - paths: - - "docs/**" - - ".github/workflows/link-check.yml" - - ".lycheeignore" - pull_request: - branches: [main] - paths: - - "docs/**" - - ".github/workflows/link-check.yml" - - ".lycheeignore" schedule: - cron: "13 9 * * 2" workflow_dispatch: @@ -33,29 +24,6 @@ concurrency: cancel-in-progress: true jobs: - base-prefix-check: - name: Base-prefix check (catches /schema/ vs /fluid/schema/) - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - name: Verify all internal absolute links carry the /fluid/ base - run: | - # VuePress base is /fluid/. Sidebar/navbar links auto-prefix, but - # body markdown links do NOT — an absolute link like [x](/schema/foo) - # renders as and 404s on the live site. - PREFIXES='guide|concepts|schema|examples|how-to|releases|vision|contributing|deck|specs' - - if grep -rIn -E "\]\(/(${PREFIXES})[^)]*\)" docs/ --include="*.md" \ - | grep -v "(/fluid/" ; then - echo "" - echo "::error::Found internal absolute markdown links missing the /fluid/ base prefix." - echo "::error::These render as and 404 on the live site." - echo "::error::Fix: rewrite [text](/schema/foo) -> [text](/fluid/schema/foo) (or use a relative path)." - exit 1 - fi - echo "✅ All internal absolute body links carry the /fluid/ base." - external: name: External URLs runs-on: ubuntu-latest diff --git a/.github/workflows/schema-sync.yml b/.github/workflows/schema-sync.yml deleted file mode 100644 index 93bcf55..0000000 --- a/.github/workflows/schema-sync.yml +++ /dev/null @@ -1,114 +0,0 @@ -# Checks that schemas vendored into this repo still match their source of truth -# in Agenticstiger/forge-cli. The forge CLI is where the FLUID JSON schemas -# are authored; this repo is the public distribution point. If the two drift, -# consumers pointing at https://open-data-protocol.github.io/fluid/schema/... -# and consumers using forge-cli would see different contracts. -# -# Sync scope starts at 0.7.2. Older versions (0.0.1..0.7.1) already diverge -# between the two repos at the byte level and are intentionally excluded; -# extend SYNCED_VERSIONS when a new release is vendored. -# -# To re-vendor manually: -# curl -fsSL "$FORGE_RAW_BASE/fluid-schema-.json" \ -# -o schema/fluid-schema-.json -# python3 generate-schema-diffs.py -# python3 generate-docs.py -# -# Comparison is done on canonical JSON (sorted keys, no whitespace) rather -# than raw bytes so cosmetic formatting drift doesn't fail the build. - -name: schema-sync - -on: - push: - branches: [main] - pull_request: - branches: [main] - schedule: - # Weekly drift check on Monday at 06:00 UTC - - cron: '0 6 * * 1' - workflow_dispatch: - -jobs: - schema-sync: - name: schema ↔ forge-cli - runs-on: ubuntu-latest - env: - FORGE_RAW_BASE: https://raw.githubusercontent.com/Agenticstiger/forge-cli/main/fluid_build/schemas - SYNCED_VERSIONS: "0.7.2 0.7.3 0.7.4 0.7.5" - steps: - - name: Checkout fluid - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: '3.12' - - - name: Verify each synced schema matches forge-cli - run: | - set -euo pipefail - fail=0 - for version in $SYNCED_VERSIONS; do - local_file="schema/fluid-schema-${version}.json" - if [[ ! -f "$local_file" ]]; then - echo "::error ::$local_file missing from this repo" - fail=1 - continue - fi - - remote_url="${FORGE_RAW_BASE}/fluid-schema-${version}.json" - remote_file="$(mktemp)" - if ! curl -fsSL "$remote_url" -o "$remote_file"; then - echo "::error ::failed to fetch $remote_url" - fail=1 - continue - fi - - canon_local=$(python3 -c "import json,sys; print(json.dumps(json.load(open(sys.argv[1])), sort_keys=True, separators=(',',':')))" "$local_file") - canon_remote=$(python3 -c "import json,sys; print(json.dumps(json.load(open(sys.argv[1])), sort_keys=True, separators=(',',':')))" "$remote_file") - - if [[ "$canon_local" != "$canon_remote" ]]; then - echo "::error ::schema/fluid-schema-${version}.json drifted from forge-cli" - echo " expected source: $remote_url" - echo " to re-vendor: curl -fsSL $remote_url -o $local_file" - fail=1 - else - echo "✓ fluid-schema-${version}.json in sync with forge-cli" - fi - done - exit $fail - - docs-up-to-date: - name: specs/ ↔ schema/ (latest) - runs-on: ubuntu-latest - steps: - - name: Checkout fluid - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: '3.12' - - - name: Install json-schema-for-humans into expected venv path - run: | - # generate-docs.py expects a venv at /.venv - venv_dir="$(dirname "$GITHUB_WORKSPACE")/.venv" - python3 -m venv "$venv_dir" - "$venv_dir/bin/pip" install --quiet json-schema-for-humans - - - name: Regenerate specs for latest schema and verify no drift - run: | - set -euo pipefail - python3 generate-docs.py - # json-schema-for-humans emits one non-deterministic line — the - # generated-on timestamp in the footer. Ignore hunks whose only - # diff is that line so real content changes still fail the job. - if git diff --quiet -I 'generated-by-footer' -- specs/; then - echo "✓ specs/ is in sync with schema/" - else - echo "::error ::specs/ is out of date. Run 'python3 generate-docs.py' and commit the result." - git --no-pager diff --stat -I 'generated-by-footer' -- specs/ - exit 1 - fi diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7337074..62f7cf6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,35 +7,118 @@ request — it is the rule that most often surprises people. `schema/fluid-schema-*.json` is **vendored**. The FLUID JSON Schemas are authored in [`forge-cli`](https://github.com/Agenticstiger/forge-cli), the -reference implementation, and land here through -[`.github/workflows/schema-sync.yml`](.github/workflows/schema-sync.yml), which -fails the build if the two ever drift. This repository is the public -distribution point. - -**So a pull request that edits a file under `schema/` will be rejected**, not -because the change is wrong but because it will be silently reverted by the next -sync. Propose schema changes upstream in forge-cli. The sync currently covers -0.7.2 and later; earlier versions diverge historically and are excluded on -purpose. - -What you *can* change here: - -| Area | Notes | -|---|---| -| `tests/**` | The conformance corpus. This is where "FLUID-conformant" is defined — see [tests/README.md](tests/README.md). | -| `conformance/**` | The runner, the reference check, the coverage tool. | -| `scripts/check-compat.py`, `scripts/compat-waivers.txt` | The backward-compatibility gate. | -| `docs/**` | The documentation site. | -| `examples/**` | Example contracts. | +reference implementation, and copied here byte for byte. This repository is +the public distribution point: the schemas from 0.5.7 on declare an `$id` +under this site, and the site serves the copy in `schema/`. + +Nothing copies them here automatically. They are **checked by** the `drift` +job in [`.github/workflows/conformance.yml`](.github/workflows/conformance.yml), +which runs on every pull request and fails when: + +- a synced schema differs, by even one byte, from the copy in the forge-cli + release **pinned** in [`scripts/schema-versions.json`](scripts/schema-versions.json) + (`upstream.ref`, and `upstream.commit`, which is what the check fetches); +- that release bundles a schema version that is not published here; +- that release's preview versions, or its latest stable version, differ from + `scripts/schema-versions.json`. + +**So a pull request that edits a file under `schema/` by hand fails the `drift` +job**, not because the change is wrong but because the published schema must be +the one the reference implementation ships. The job fails that pull request's +checks; whether a failing check also blocks the merge is a branch-protection +setting, not something this file promises. Propose schema changes upstream in +forge-cli. + +The comparison is with the pinned release, not with whatever forge-cli +released last, so a new forge-cli release does not turn unrelated pull requests +red. Moving to one is a change to the pin, made in a pull request where a +reviewer sees which upstream commit is being trusted (see below). A separate, +advisory job, `drift-latest`, warns when forge-cli has released after the +pinned release; it cannot fail a run. + +Two more files are guarded, by the `generated` job. The versions before 0.7.2 +are not synced, so nothing regenerates them; instead `scripts/schema-versions.json` +records a sha256 for each of their schemas and HTML pages (`frozen`), and +`python3 generate-docs.py --check` fails when one changes, or when a file +appears under `schema/` or `specs/` that is neither generated for a synced +version nor listed there. + +[`scripts/schema-versions.json`](scripts/schema-versions.json) is the record of +which versions are synced, which one is the latest **stable** version, and +which are **previews**: a contract uses a preview only by naming it in +`fluidVersion`, and a preview can still change before it becomes stable. +`python3 generate-docs.py --print-latest-stable` prints the stable version. +Versions before 0.7.2 are not synced. The 0.7.1 published here differs from +the document forge-cli bundles under the same `$id`, and re-vendoring it would +change which documents it accepts — a decision for the maintainers, not a +sync. + +### Re-vendoring a schema + +When the drift check fails, or forge-cli releases a new schema version +(`python3 scripts/check-schema-drift.py --latest` says whether it has): + +1. Choose the release, a tag and never `main`, and find its commit: + + ```bash + TAG=v0.18.1 # the forge-cli release you are vendoring from + git ls-remote https://github.com/Agenticstiger/forge-cli "refs/tags/$TAG" "refs/tags/$TAG^{}" + ``` + + Take the commit from the **last** line printed. For an annotated tag that is + the `^{}` line; the line before it is the tag object, not a commit. + `python3 scripts/check-schema-drift.py --ref $TAG` shows what the tag would + change before you edit anything. + +2. Record it: set `upstream.ref` and `upstream.commit` in + [`scripts/schema-versions.json`](scripts/schema-versions.json). For a new + version, or a version whose status changed (a preview promoted to stable), + update `synced`, `preview` and `latestStable` in the same file, and for a new + version follow "Adding a version" in [tests/README.md](tests/README.md). +3. Copy each changed schema from that commit, byte for byte: + + ```bash + COMMIT=... # the commit from step 1, which is also upstream.commit + for v in 0.7.5 0.7.6; do + curl -fsSL "https://raw.githubusercontent.com/Agenticstiger/forge-cli/$COMMIT/fluid_build/schemas/fluid-schema-$v.json" \ + -o "schema/fluid-schema-$v.json" + done + ``` + +4. Regenerate what is derived from the schemas, with the locked renderer: + + ```bash + pip install --require-hashes -r scripts/requirements-docs.lock + python3 generate-docs.py # specs// for every synced version + python3 generate-schema-diffs.py # schema-diffs/ + ``` + +5. Run the gates below. `python3 scripts/check-schema-drift.py` now compares + with the commit you recorded in step 2, so it passes only if step 3 copied + exactly what that commit holds. A re-vendored schema can add constraints the + corpus does not pin yet, which lowers mutation coverage; add the cases in + the same pull request. +6. Name the release and the commit you vendored from in the pull request. ## Setup ```bash pip install "jsonschema[format]>=4.22" -python3 conformance/run.py # the corpus must be green -python3 tests/meta_test.py # the gates must be able to fail -python3 scripts/check-compat.py # no release may break its promise +python3 conformance/run.py # the corpus must be green +python3 conformance/mutation_coverage.py --min-coverage 40 # and still worth something +python3 tests/meta_test.py # the gates must be able to fail +python3 scripts/check-compat.py # no release may break its promise +python3 scripts/check-schema-drift.py # schema/ equals the pinned forge-cli release (needs network) +``` + +The generated files have their own gates and their own, hash-locked +requirements: + +```bash +pip install --require-hashes -r scripts/requirements-docs.lock +python3 generate-docs.py --check # specs/ is current; every other file in schema/ and specs/ is frozen +python3 generate-schema-diffs.py --check # schema-diffs/ is current ``` The `[format]` extra is not optional. Without it `jsonschema` registers no diff --git a/GENERATOR_DOCS.md b/GENERATOR_DOCS.md index 43e1b4c..8132f9b 100644 --- a/GENERATOR_DOCS.md +++ b/GENERATOR_DOCS.md @@ -1,259 +1,107 @@ -# FLUID Documentation Generator +# Generated files: the HTML reference and the schema diffs -This document explains how the `generate-docs.py` script works to automatically generate HTML documentation from FLUID JSON schema files. +Two directories in this repository are generated from `schema/` and committed: -## Overview +| Directory | Generator | What it holds | +|---|---|---| +| `specs//` | `generate-docs.py` | `fluid-spec.html`, the field-by-field HTML reference for one schema version, with its `schema_doc.css` and `schema_doc.min.js` | +| `schema-diffs/` | `generate-schema-diffs.py` | one Markdown diff per pair of consecutive schema versions, and an index | -The FLUID documentation generator is a Python script that: -1. Discovers JSON schema files in the project -2. Identifies the latest schema version -3. Generates beautiful HTML documentation using `json-schema-for-humans` -4. Outputs documentation to a versioned directory structure +Neither is edited by hand. The `generated` job in +`.github/workflows/conformance.yml` regenerates both on every pull request and +fails if the committed files differ from a fresh run. -## Prerequisites +Nothing else may sit beside them. `generate-docs.py --check` also accounts for +every file under `schema/` and `specs/`: each is either generated for a synced +version (or, under `schema/`, a synced schema, which the drift check compares +with the reference implementation) or listed with its sha256 in the `frozen` +map of `scripts/schema-versions.json`. A file that is neither, a frozen file +whose bytes changed, and a frozen file that is gone all fail the check. The +frozen files are the schemas before 0.7.2 and the HTML pages earlier renderer +releases produced for them; changing one is a deliberate edit to `frozen`. -- Python 3.6+ (tested with Python 3.12) -- Virtual environment (recommended) -- `json-schema-for-humans` package +## The version record -### Installing Dependencies +`scripts/schema-versions.json` says which schema versions are **synced** with +the reference implementation, which one is the latest **stable** version, and +which are **previews**, and it records which reference implementation release +they were vendored from (`upstream.ref` and `upstream.commit`, the release +`scripts/check-schema-drift.py` compares with) and the `frozen` files described +above. Both generators and the drift check read it. -```bash -# Activate your virtual environment -source .venv/bin/activate - -# Install the required package -pip install json-schema-for-humans -``` - -## How It Works - -### 1. Schema Discovery - -The generator automatically discovers schema files in the `schema/` directory: - -```python -def find_schema_files(): - candidates = list(SCHEMA_DIR.glob("fluid-schema-*.json")) -``` - -**Expected naming pattern**: `fluid-schema-X.Y.Z.json` where X.Y.Z is the semantic version. - -**Example files**: -- `fluid-schema-0.1.0.json` -- `fluid-schema-0.5.7.json` -- `fluid-schema-1.0.0.json` - -### 2. Version Extraction and Sorting - -The script extracts version numbers using regex and sorts them semantically: - -```python -def extract_version(filename: Path) -> str: - match = re.search(r"(\d+\.\d+\.\d+)", filename.name) - return match.group(1) -``` - -**Sorting logic**: Files are sorted by semantic version (e.g., 0.1.0 < 0.5.7 < 1.0.0), and the **latest version** is automatically selected for documentation generation. - -### 3. Documentation Generation - -The script uses the `json-schema-for-humans` library to convert JSON schemas into readable HTML documentation: - -```python -def generate_docs(schema_file: Path, version: str): - cmd = [ - str(venv_path), # Path to generate-schema-doc in venv - str(schema_file), - str(output_file), - "--config", - "default" - ] -``` - -**Key features**: -- Uses the `default` configuration for clean, readable output -- Automatically copies CSS and JavaScript assets -- Generates self-contained HTML files - -### 4. Output Structure - -Documentation is generated in a versioned directory structure: - -``` -specs/ -├── 0.5.7/ -│ ├── fluid-spec.html # Main documentation -│ ├── schema_doc.css # Styling -│ └── schema_doc.min.js # Interactive features -├── 0.6.0/ -│ ├── fluid-spec.html -│ ├── schema_doc.css -│ └── schema_doc.min.js -└── ... -``` - -## Directory Structure - -The generator relies on this project structure: - -``` -fluid/ -├── generate-docs.py # The generator script -├── schema/ # Source schema files -│ ├── fluid-schema-0.1.0.json -│ ├── fluid-schema-0.5.7.json -│ └── ... -└── specs/ # Generated documentation - ├── 0.5.7/ - ├── 0.6.0/ - └── ... -``` - -## Usage - -### Basic Usage - -Run the generator from the project root: +The latest stable version is recorded on its own, not derived from the +highest version number, because a preview has the higher number. Nothing +should link a preview as "latest": ```bash -# Using the virtual environment Python -/path/to/.venv/bin/python generate-docs.py - -# Or if virtual environment is activated -python generate-docs.py -``` - -### Example Output - +python3 generate-docs.py --print-latest-stable # prints the stable version ``` -[fluid-docs] Looking for schema files in: /path/to/schema -[fluid-docs] Found 7 schema files. -[fluid-docs] - fluid-schema-0.1.1.json -[fluid-docs] - fluid-schema-0.3.0.json -[fluid-docs] - fluid-schema-0.5.7.json -[fluid-docs] - ... -[fluid-docs] Using schema: fluid-schema-0.5.7.json (version 0.5.7) -[fluid-docs] Running command: generate-schema-doc ... -[fluid-docs] ✅ Docs generated at: /path/to/specs/0.5.7/fluid-spec.html -``` - -## Configuration - -The generator currently uses the `default` configuration from `json-schema-for-humans`. This can be customized by: - -1. **Modifying the config parameter**: - ```python - cmd = [..., "--config", "custom_config_name"] - ``` - -2. **Using a config file**: - ```python - cmd = [..., "--config-file", "path/to/config.json"] - ``` - -3. **Inline configuration**: - ```python - cmd = [..., "--config", "minify=false", "--config", "expand_buttons=true"] - ``` - -## Key Features - -### Robust Path Resolution -- All paths are resolved relative to the script location, not the current working directory -- Works regardless of where you run the script from - -### Error Handling -- Graceful handling of missing dependencies -- Clear error messages with actionable advice -- Validation of schema file naming conventions - -### Logging -- Detailed console output for debugging -- File and folder context in all messages -- Clear success/failure indicators -### Virtual Environment Support -- Automatically uses the correct Python environment -- Resolves the `generate-schema-doc` command path within the virtual environment +`generate-docs.py` refuses to run if the record is inconsistent: a version +listed as both stable and preview, a stable version that is not synced, or a +synced non-preview version higher than the recorded stable one. -## Troubleshooting +## `generate-docs.py` -### Common Issues - -1. **"generate-schema-doc command not found"** - - Install `json-schema-for-humans`: `pip install json-schema-for-humans` - - Ensure you're using the correct Python environment - -2. **"No schema files found"** - - Check that files are in the `schema/` directory - - Verify naming follows `fluid-schema-X.Y.Z.json` pattern - -3. **Permission errors** - - Ensure the `specs/` directory is writable - - Check file permissions on schema files - -### Debug Mode - -For additional debugging, you can modify the script to add more verbose logging: - -```python -def log(msg: str): - import datetime - timestamp = datetime.datetime.now().strftime("%H:%M:%S") - print(f"[{timestamp}] [fluid-docs] {msg}") +```bash +pip install --require-hashes -r scripts/requirements-docs.lock + +python3 generate-docs.py # every synced version +python3 generate-docs.py --all # the same, spelled out +python3 generate-docs.py --version 0.7.6 # one version; repeat the flag for more +python3 generate-docs.py --check # render to a temporary directory and + # exit 1 if specs/ differs, or if a + # file under schema/ or specs/ is not + # accounted for +python3 generate-docs.py --check-files # only the offline half of --check; + # needs no renderer ``` -## Extending the Generator - -### Adding New Output Formats +It renders with [json-schema-for-humans](https://github.com/coveooss/json-schema-for-humans), +in-process, using its default configuration with the footer timestamp turned +off, so the same schema and the same renderer produce the same bytes. The +renderer and the libraries that shape its output are pinned exactly in +`scripts/requirements-docs.txt`, and the script exits with an error naming the +mismatch if any other version is installed. That file is the input; what CI and +the docs deploy install is `scripts/requirements-docs.lock`, the same pins plus +every transitive dependency, each with its hashes, installed with +`--require-hashes` so that a package replaced on PyPI afterwards is refused. +`--check` fails if the two files disagree. To move a pin, change it in +`requirements-docs.txt`, recompile the lock, run `python3 generate-docs.py`, and +commit the lock and the regenerated `specs/` in the same pull request: -The generator can be extended to support additional output formats by modifying the `generate_docs` function: - -```python -def generate_docs(schema_file: Path, version: str, format_type="html"): - if format_type == "html": - # Current HTML generation - elif format_type == "markdown": - # Add markdown generation - elif format_type == "pdf": - # Add PDF generation +```bash +pip install pip-tools # under Python 3.12, the version CI uses +pip-compile --generate-hashes --strip-extras \ + --output-file=scripts/requirements-docs.lock scripts/requirements-docs.txt ``` -### Custom Schema Processing +A preview's page is labelled by its schema's own `title`; for 0.7.6 that reads +"FLUID 0.7.6 — Declarative Packaging Modes (preview)". -You can add custom schema processing before documentation generation: - -```python -def preprocess_schema(schema_file: Path) -> Path: - # Load, modify, and save schema - # Return path to processed schema - pass -``` +`--version` also renders a version that is not synced, if its schema is in +`schema/`. The pages for versions before 0.7.2 were rendered by earlier +releases of the renderer and are not regenerated; they are frozen, which is to +say checked by hash rather than by re-rendering. -## Integration with CI/CD +## `generate-schema-diffs.py` -The generator can be integrated into automated workflows: - -```yaml -# GitHub Actions example -- name: Generate Documentation - run: | - source .venv/bin/activate - python generate-docs.py - -- name: Deploy Documentation - # Upload or commit generated docs +```bash +python3 generate-schema-diffs.py # rewrite schema-diffs/ +python3 generate-schema-diffs.py --check # exit 1 if schema-diffs/ is stale ``` -## Dependencies - -- **Python 3.6+**: Core runtime -- **json-schema-for-humans**: Documentation generation -- **pathlib**: Path handling (built-in) -- **subprocess**: Command execution (built-in) -- **re**: Regular expressions (built-in) +It compares every pair of consecutive files in `schema/`, key by key, and +writes `diff--to-.md` plus `README.md`. Long values are truncated in +the output, so a diff file is a map of where to look, not a substitute for the +schema. A hand-written summary between `` and +`` in a diff file survives regeneration. The index marks +a pair whose newer version is a preview. -## License +## The docs site -This generator follows the same license as the FLUID project. \ No newline at end of file +`npm run docs:build` first runs `scripts/sync-public-assets.mjs`, which copies +`schema/` and `specs/` into `docs/.vuepress/public/`, so the site serves every +schema from 0.5.7 on at the URL its `$id` names, and each HTML reference beside it. The +deploy workflow (`.github/workflows/deploy-docs.yml`) runs +`python3 generate-docs.py` with the locked renderer before that build. diff --git a/GOVERNANCE.md b/GOVERNANCE.md index 49edce5..17b502f 100644 --- a/GOVERNANCE.md +++ b/GOVERNANCE.md @@ -64,6 +64,24 @@ One historical break is on record (0.7.1 → 0.7.2, `notification` became a clos object). It was found by the gate after the fact, and the release note that claimed otherwise has been corrected rather than quietly left standing. +### Version status + +Each vendored schema version is stable, preview or earlier, as recorded in +[`scripts/schema-versions.json`](scripts/schema-versions.json): + +- **Stable.** The latest stable version is the one the reference implementation + treats as current. It is recorded separately from the highest version number. +- **Preview.** A preview can change before it becomes stable. A contract uses it + only by naming it in `fluidVersion`. When a release of the reference implementation + changes a preview, the copy here is re-vendored to match. +- **Earlier.** Versions below the latest stable remain vendored. The + compatibility rule above covers them from 0.7.1 on. + +`scripts/check-schema-drift.py` fails when this record disagrees with the +reference implementation release it pins (`upstream.ref` and `upstream.commit` in +the same file), so promoting a preview is a change to that file made in the same +pull request as the re-vendor. + ## Trademarks "FLUID" as a name for this specification is reserved by the maintainers. The diff --git a/generate-docs.py b/generate-docs.py index fe21f05..08bd68c 100644 --- a/generate-docs.py +++ b/generate-docs.py @@ -1,108 +1,279 @@ #!/usr/bin/env python3 -""" -generate-docs.py ----------------- -A robust FLUID schema doc generator. - -Improvements: -- Resolves schema directory relative to script location, not cwd -- Better console logging with file/folder context -- Graceful exception handling -- Clearer output for debugging +"""Render the HTML schema reference: specs//fluid-spec.html. + + python3 generate-docs.py # every synced version + python3 generate-docs.py --all # the same, spelled out + python3 generate-docs.py --version 0.7.6 # one version (repeatable) + python3 generate-docs.py --check # render to a temp dir and + # fail if specs/ is stale, or + # holds a file it should not + python3 generate-docs.py --check-files # the offline half of --check + python3 generate-docs.py --print-latest-stable + +Which versions are "synced", and which one is the latest STABLE version, is +recorded once, in scripts/schema-versions.json. The latest stable version is +not the highest version number: a preview has the higher number and is never +"latest". This script used to render only the highest-numbered schema, so a +re-vendored older version kept its stale HTML, and vendoring a preview would +have made the preview the only page CI and the docs deploy ever refreshed. + +The renderer is json-schema-for-humans, run in-process, with the exact +versions pinned in scripts/requirements-docs.txt (the file a person edits) and +installed from scripts/requirements-docs.lock (the same pins plus their +transitive closure, hash-locked, which is what CI installs). The output is +committed and --check compares it byte for byte, so the renderer version is +part of the input; the script refuses to run with any other installed version. +The footer timestamp is switched off so that the same schema and the same +renderer give the same bytes. + +--check also accounts for every file under schema/ and specs/, offline. A file +is either generated (schema/fluid-schema-.json, which the drift check +compares with the reference implementation, and the OUTPUT_FILES of a synced +version under specs/) or "frozen": listed with its sha256 in +scripts/schema-versions.json. A file that is neither fails the check, as does a +frozen file whose bytes changed. Without this, only the synced versions' +outputs were compared, and the older pages and schemas -- and any extra file +dropped beside the generated ones -- could change with every gate green. +--check-files runs just this part and the lock check; it needs no renderer. """ -import os +from __future__ import annotations + +import argparse +import filecmp +import hashlib +import json import re -import subprocess import sys +import tempfile from pathlib import Path +from typing import Dict, List + +REPO = Path(__file__).resolve().parent +SCHEMA_DIR = REPO / "schema" +SPECS_DIR = REPO / "specs" +VERSIONS_FILE = REPO / "scripts" / "schema-versions.json" +REQUIREMENTS = REPO / "scripts" / "requirements-docs.txt" +LOCK = REPO / "scripts" / "requirements-docs.lock" +OUTPUT_FILES = ("fluid-spec.html", "schema_doc.css", "schema_doc.min.js") -# Resolve paths relative to this script file -SCRIPT_DIR = Path(__file__).resolve().parent -SCHEMA_DIR = SCRIPT_DIR / "schema" -SPECS_DIR = SCRIPT_DIR / "specs" -def log(msg: str): - """Simple logger""" +def log(msg: str) -> None: print(f"[fluid-docs] {msg}") -def find_schema_files(): - """Find all schema files in ./schema/ matching fluid-schema-x.x.x.json""" - log(f"Looking for schema files in: {SCHEMA_DIR}") - if not SCHEMA_DIR.exists(): - log("❌ Schema directory does not exist.") - return [] - - candidates = list(SCHEMA_DIR.glob("fluid-schema-*.json")) - log(f"Found {len(candidates)} schema files.") - for c in candidates: - log(f" - {c.name}") - return candidates - -def extract_version(filename: Path) -> str: - """Extract version x.x.x from schema filename""" - match = re.search(r"(\d+\.\d+\.\d+)", filename.name) - if not match: - raise ValueError(f"Could not extract version from filename: {filename}") - return match.group(1) - -def generate_docs(schema_file: Path, version: str): - """Run json-schema-for-humans to generate docs""" - output_dir = SPECS_DIR / version - output_dir.mkdir(parents=True, exist_ok=True) - - output_file = output_dir / "fluid-spec.html" - - # Use the full path to the generate-schema-doc command in the virtual environment - # On Windows: .venv\Scripts\, on Unix: .venv/bin/ - venv_bin_dir = "Scripts" if sys.platform == "win32" else "bin" - venv_cmd = "generate-schema-doc.exe" if sys.platform == "win32" else "generate-schema-doc" - venv_path = Path(__file__).resolve().parent.parent / ".venv" / venv_bin_dir / venv_cmd - - cmd = [ - str(venv_path), - str(schema_file), - str(output_file), - "--config", - "default" - ] - - log(f"Running command: {' '.join(cmd)}") - try: - subprocess.run(cmd, check=True) - except FileNotFoundError: - log("❌ Error: `generate-schema-doc` command not found. Did you install json-schema-for-humans?") - sys.exit(1) - except subprocess.CalledProcessError as e: - log(f"❌ Error running generate-schema-doc: {e}") - sys.exit(1) - - log(f"✅ Docs generated at: {output_file}") - -def main(): - try: - candidates = find_schema_files() - if not candidates: - log("❌ No schema files found. Make sure they are in ./schema/ and named like fluid-schema-x.x.x.json") - sys.exit(1) - - # Sort by version number - def version_key(f): - try: - return tuple(map(int, extract_version(f).split("."))) - except Exception: - return (0,0,0) - - candidates.sort(key=version_key) - latest = candidates[-1] - version = extract_version(latest) - - log(f"Using schema: {latest.name} (version {version})") - generate_docs(latest, version) - - except Exception as e: - log(f"Unexpected error: {e}") - sys.exit(1) + +def version_key(version: str): + return tuple(int(p) for p in version.split(".")) + + +def load_versions() -> Dict: + data = json.loads(VERSIONS_FILE.read_text()) + stable = data["latestStable"] + preview = set(data.get("preview", [])) + synced = data["synced"] + if stable in preview: + sys.exit(f"{VERSIONS_FILE.name}: latestStable {stable} is also listed as preview") + if stable not in synced: + sys.exit(f"{VERSIONS_FILE.name}: latestStable {stable} is not in synced") + highest_stable = max((v for v in synced if v not in preview), key=version_key) + if highest_stable != stable: + sys.exit( + f"{VERSIONS_FILE.name}: latestStable is {stable}, but {highest_stable} " + "is a higher synced version that is not a preview" + ) + return data + + +def check_renderer_pins() -> None: + """Refuse to render with anything but the pinned renderer.""" + from importlib.metadata import PackageNotFoundError, version + + pins = {} + for line in REQUIREMENTS.read_text().splitlines(): + line = line.split("#", 1)[0].strip() + m = re.fullmatch(r"([A-Za-z0-9_.-]+)==([^\s]+)", line) + if m: + pins[m.group(1)] = m.group(2) + wrong = [] + for name, wanted in pins.items(): + try: + got = version(name) + except PackageNotFoundError: + got = "not installed" + if got != wanted: + wrong.append(f"{name}: need {wanted}, found {got}") + if wrong: + sys.exit( + "generate-docs: the HTML renderer does not match its pins:\n " + + "\n ".join(wrong) + + f"\nInstall them with: pip install --require-hashes -r {LOCK.relative_to(REPO)}" + ) + + +def _normalise(name: str) -> str: + return re.sub(r"[-_.]+", "-", name).lower() + + +def check_lock() -> List[str]: + """requirements-docs.txt (edited) and requirements-docs.lock (installed) pin the same versions.""" + if not LOCK.is_file(): + return [f"{LOCK.relative_to(REPO)} is missing (pip-compile --generate-hashes; see its sibling .txt)"] + locked = {} + for line in LOCK.read_text().splitlines(): + m = re.match(r"([A-Za-z0-9_.-]+)==(\S+?)\s*\\?\s*$", line) + if m: + locked[_normalise(m.group(1))] = m.group(2) + problems = [] + for line in REQUIREMENTS.read_text().splitlines(): + m = re.fullmatch(r"([A-Za-z0-9_.-]+)==([^\s]+)", line.split("#", 1)[0].strip()) + if not m: + continue + name, wanted = m.groups() + got = locked.get(_normalise(name)) + if got != wanted: + problems.append( + f"{REQUIREMENTS.name} pins {name}=={wanted}, but {LOCK.name} has " + f"{'no entry' if got is None else got}; recompile the lock" + ) + return problems + + +def sha256_of(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def check_inventory(data: Dict) -> List[str]: + """Every file under schema/ and specs/ is generated output of a synced version, or frozen. + + The synced versions' outputs are compared with a fresh render and the synced + schemas with the reference implementation, so they are not hashed here. All + other files are hashed against the "frozen" map in scripts/schema-versions.json. + """ + synced = data["synced"] + frozen = data.get("frozen") + if not isinstance(frozen, dict): + return [f'{VERSIONS_FILE.name} has no "frozen" map, so nothing under schema/ or specs/ outside the synced versions can be accounted for'] + + generated = {f"specs/{v}/{name}" for v in synced for name in OUTPUT_FILES} + synced_schemas = {f"schema/fluid-schema-{v}.json" for v in synced} + problems: List[str] = [] + + for rel, digest in sorted(frozen.items()): + if rel in generated or rel in synced_schemas: + problems.append(f"{rel}: listed in \"frozen\", but it belongs to a synced version, which is compared instead") + continue + if not rel.startswith(("schema/", "specs/")) or ".." in rel.split("/"): + problems.append(f"{rel}: \"frozen\" lists only files under schema/ and specs/") + continue + path = REPO / rel + if path.is_symlink() or not path.is_file(): + problems.append(f"{rel}: listed in \"frozen\", but the file is missing") + elif sha256_of(path) != digest: + problems.append(f"{rel}: differs from its frozen sha256 (a frozen file is never edited; changing one is a deliberate edit to \"frozen\")") + + for root in (SCHEMA_DIR, SPECS_DIR): + for path in sorted(root.rglob("*")): + if path.is_dir() and not path.is_symlink(): + continue + rel = path.relative_to(REPO).as_posix() + if path.is_symlink(): + problems.append(f"{rel}: a symbolic link; schema/ and specs/ hold regular files only") + elif rel not in generated and rel not in synced_schemas and rel not in frozen: + problems.append( + f"{rel}: neither generated output of a synced version nor listed in \"frozen\" " + f"in scripts/{VERSIONS_FILE.name}" + ) + return problems + + +def render(version: str, out_dir: Path) -> None: + from json_schema_for_humans.generate import generate_from_filename + from json_schema_for_humans.generation_configuration import GenerationConfiguration + + schema_file = SCHEMA_DIR / f"fluid-schema-{version}.json" + if not schema_file.is_file(): + sys.exit(f"generate-docs: no schema for version {version}: {schema_file}") + out_dir.mkdir(parents=True, exist_ok=True) + config = GenerationConfiguration(footer_show_time=False) + generate_from_filename(str(schema_file), str(out_dir / "fluid-spec.html"), config=config) + + +def main(argv: List[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + which = ap.add_mutually_exclusive_group() + which.add_argument("--all", action="store_true", help="every synced version (the default)") + which.add_argument("--version", action="append", dest="versions", metavar="X.Y.Z") + ap.add_argument( + "--check", + action="store_true", + help="fail if specs/ differs from a fresh render, or schema/ or specs/ holds an unaccounted-for file", + ) + ap.add_argument( + "--check-files", + action="store_true", + help="only the offline half of --check (every file accounted for, lock matches pins); needs no renderer", + ) + ap.add_argument("--print-latest-stable", action="store_true") + args = ap.parse_args(argv) + + data = load_versions() + if args.print_latest_stable: + print(data["latestStable"]) + return 0 + + versions = args.versions or data["synced"] + + if args.check or args.check_files: + lock_problems = check_lock() + if lock_problems: + print("generate-docs: the lock does not match the pins:", file=sys.stderr) + for problem in lock_problems: + print(f" {problem}", file=sys.stderr) + inventory_problems = check_inventory(data) + if inventory_problems: + print("generate-docs: files under schema/ and specs/ that are not accounted for:", file=sys.stderr) + for problem in inventory_problems: + print(f" {problem}", file=sys.stderr) + file_problems = lock_problems + inventory_problems + if args.check_files: + if file_problems: + return 1 + log("every file under schema/ and specs/ is generated or frozen, and the lock matches the pins") + return 0 + else: + file_problems = [] + + check_renderer_pins() + + if not args.check: + for v in versions: + log(f"rendering {v} -> specs/{v}/fluid-spec.html") + render(v, SPECS_DIR / v) + log(f"latest stable: {data['latestStable']}; preview: {', '.join(data.get('preview', [])) or 'none'}") + return 0 + + stale = [] + with tempfile.TemporaryDirectory() as tmp: + for v in versions: + fresh = Path(tmp) / v + render(v, fresh) + for name in OUTPUT_FILES: + committed = SPECS_DIR / v / name + if not committed.is_file() or not filecmp.cmp(fresh / name, committed, shallow=False): + stale.append(f"specs/{v}/{name}") + if stale: + print("generate-docs: specs/ is out of date with schema/:", file=sys.stderr) + for path in stale: + print(f" {path}", file=sys.stderr) + print( + "Run `python3 generate-docs.py` and commit the result.", + file=sys.stderr, + ) + if stale or file_problems: + return 1 + log(f"specs/ is current for {', '.join(versions)}; every other file under schema/ and specs/ is frozen and unchanged") + return 0 + if __name__ == "__main__": - main() + sys.exit(main()) diff --git a/generate-schema-diffs.py b/generate-schema-diffs.py index ccc9228..b3d0894 100644 --- a/generate-schema-diffs.py +++ b/generate-schema-diffs.py @@ -11,8 +11,10 @@ - Value changes in a clear, readable format """ +import argparse import json import re +import sys from pathlib import Path from typing import Any, Dict, List, Tuple @@ -20,6 +22,7 @@ SCRIPT_DIR = Path(__file__).resolve().parent SCHEMA_DIR = SCRIPT_DIR / "schema" DIFFS_DIR = SCRIPT_DIR / "schema-diffs" +VERSIONS_FILE = SCRIPT_DIR / "scripts" / "schema-versions.json" def log(msg: str): @@ -228,33 +231,34 @@ def generate_diff_markdown(old_version: str, new_version: str, return "\n".join(md) -def generate_diff(old_version: str, old_path: Path, - new_version: str, new_path: Path): - """Generate a diff file between two schema versions""" - +def generate_diff(old_version: str, old_path: Path, + new_version: str, new_path: Path) -> Tuple[Path, str]: + """Render the diff file between two schema versions (not yet written).""" + # Load schemas with open(old_path, 'r', encoding='utf-8') as f: old_schema = json.load(f) - + with open(new_path, 'r', encoding='utf-8') as f: new_schema = json.load(f) - - # Write to file (preserving any hand-written HUMAN-NOTE block) - output_file = DIFFS_DIR / f"diff-{old_version}-to-{new_version}.md" - output_file.parent.mkdir(parents=True, exist_ok=True) - # Generate markdown + # Preserve any hand-written HUMAN-NOTE block already in the file + output_file = DIFFS_DIR / f"diff-{old_version}-to-{new_version}.md" preserved_note = extract_preserved_note(output_file) markdown = generate_diff_markdown(old_version, new_version, old_schema, new_schema, preserved_note) - - with open(output_file, 'w', encoding='utf-8') as f: - f.write(markdown) - - log(f"✅ Generated: {output_file.name}") + return output_file, markdown + + +def preview_versions() -> set: + """Versions scripts/schema-versions.json marks as preview.""" + if not VERSIONS_FILE.is_file(): + return set() + return set(json.loads(VERSIONS_FILE.read_text()).get("preview", [])) -def generate_index(versions: List[Tuple[str, Path]]): - """Generate an index file listing all diffs""" +def generate_index(versions: List[Tuple[str, Path]]) -> Tuple[Path, str]: + """Render the index file listing all diffs (not yet written).""" + preview = preview_versions() md = [ "# FLUID Schema Version History", "", @@ -263,50 +267,66 @@ def generate_index(versions: List[Tuple[str, Path]]): "## Version Progression", "" ] - + for i in range(len(versions) - 1): old_ver = versions[i][0] new_ver = versions[i + 1][0] - md.append(f"- [{old_ver} → {new_ver}](diff-{old_ver}-to-{new_ver}.md)") - + note = f" ({new_ver} is a preview and may still change)" if new_ver in preview else "" + md.append(f"- [{old_ver} → {new_ver}](diff-{old_ver}-to-{new_ver}.md){note}") + md.append("") md.append("---") md.append("") md.append(f"**Versions tracked:** {len(versions)}") md.append(f"**Diff files generated:** {len(versions) - 1}") md.append("") - - index_file = DIFFS_DIR / "README.md" - with open(index_file, 'w', encoding='utf-8') as f: - f.write("\n".join(md)) - - log(f"✅ Generated index: {index_file.name}") + + return DIFFS_DIR / "README.md", "\n".join(md) -def main(): +def main(argv=None) -> int: """Main entry point""" + ap = argparse.ArgumentParser(description="Generate schema-diffs/ from schema/.") + ap.add_argument("--check", action="store_true", + help="fail if schema-diffs/ differs from a fresh generation") + args = ap.parse_args(argv) + log("Starting schema diff generation...") - + # Find all schema versions versions = find_schema_files() - + if len(versions) < 2: log("❌ Need at least 2 schema versions to generate diffs") - return - - # Generate diffs for consecutive versions + return 1 + + outputs: List[Tuple[Path, str]] = [] for i in range(len(versions) - 1): old_ver, old_path = versions[i] new_ver, new_path = versions[i + 1] - log(f"Comparing {old_ver} -> {new_ver}") - generate_diff(old_ver, old_path, new_ver, new_path) - - # Generate index - generate_index(versions) - + outputs.append(generate_diff(old_ver, old_path, new_ver, new_path)) + outputs.append(generate_index(versions)) + + if args.check: + stale = [p for p, text in outputs + if not p.is_file() or p.read_text(encoding="utf-8") != text] + if stale: + log("❌ schema-diffs/ is out of date with schema/:") + for p in stale: + log(f" {p.relative_to(SCRIPT_DIR)}") + log("Run `python3 generate-schema-diffs.py` and commit the result.") + return 1 + log(f"✅ schema-diffs/ is current ({len(outputs) - 1} diff files)") + return 0 + + DIFFS_DIR.mkdir(parents=True, exist_ok=True) + for path, text in outputs: + path.write_text(text, encoding="utf-8") + log(f"✅ Generated: {path.name}") log(f"✅ Complete! Generated {len(versions) - 1} diff files in {DIFFS_DIR}") + return 0 if __name__ == "__main__": - main() + sys.exit(main()) diff --git a/schema-diffs/README.md b/schema-diffs/README.md index bc474e1..087b20f 100644 --- a/schema-diffs/README.md +++ b/schema-diffs/README.md @@ -15,8 +15,9 @@ This directory contains human-readable diffs between consecutive versions of the - [0.7.2 → 0.7.3](diff-0.7.2-to-0.7.3.md) - [0.7.3 → 0.7.4](diff-0.7.3-to-0.7.4.md) - [0.7.4 → 0.7.5](diff-0.7.4-to-0.7.5.md) +- [0.7.5 → 0.7.6](diff-0.7.5-to-0.7.6.md) (0.7.6 is a preview and may still change) --- -**Versions tracked:** 12 -**Diff files generated:** 11 +**Versions tracked:** 13 +**Diff files generated:** 12 diff --git a/schema-diffs/diff-0.7.4-to-0.7.5.md b/schema-diffs/diff-0.7.4-to-0.7.5.md index 47f2728..798b415 100644 --- a/schema-diffs/diff-0.7.4-to-0.7.5.md +++ b/schema-diffs/diff-0.7.4-to-0.7.5.md @@ -1,9 +1,22 @@ # Schema Changes: 0.7.4 → 0.7.5 -**Total changes:** 13 -- ✅ Added: 7 +**Total changes:** 26 +- ✅ Added: 18 - ❌ Removed: 0 -- 📝 Modified: 6 +- 📝 Modified: 8 + + +> **Note — 0.7.5 re-vendored.** The 0.7.5 schema first published here was the +> reference implementation's pre-GA copy (identical to forge-cli v0.10.0's, from +> while 0.7.5 was still a preview). It has been replaced, byte for byte, by the +> copy forge-cli v0.18.1 ships, and this diff regenerated. Against the earlier +> copy, the GA schema adds the `pgvector` platform, the `pgvector_table` format, +> `binding.vectorConfig`, and the Iceberg catalog-profile (`catalog`, +> `warehouse`, `uri`, `partitionBy`), Redshift Serverless (`namespace`, +> `workgroup`, `iam_role_arn`, `external_schema`, `glue_database`) and Kinesis +> (`stream`) location fields. `scripts/check-compat.py --from 0.7.4 --to 0.7.5` +> reports no narrowing. + --- @@ -67,6 +80,55 @@ } ``` +### `$defs.binding.properties.vectorConfig` +```json +{ + type: "object" + additionalProperties: false + description: "NEW in v0.7.5: vector / embeddings output-port configuration (platform 'pgvector', format 'pgvector_..." + required: ["dimensions"] + properties: { + dimensions: { + type: "integer" + minimum: 1 + maximum: 16000 + description: "Embedding vector width (e.g. 1536 for OpenAI text-embedding-3-small, 1024 for Cohere embed-english-l..." + } + embeddingModel: { + type: "string" + description: "Provenance: the embedding model that produces the vectors (e.g. 'text-embedding-3-small'). Recorded ..." + } + vectorType: { + type: "string" + enum: ["vector", "halfvec"] + default: "vector" + description: "pgvector column type: 'vector' (float32, up to 2000 dims) or 'halfvec' (float16, half the storage, u..." + } + indexType: { + type: "string" + enum: ["hnsw", "ivfflat", "none"] + default: "hnsw" + description: "ANN index built on the embedding column. 'hnsw' (higher recall / lower query latency, buildable befo..." + } + distanceMetric: { + type: "string" + enum: [...4 items...] + default: "cosine" + description: "Distance function the index optimises for → pgvector operator class: cosine→vector_cosine_ops, l2→ve..." + } + ... (4 more) + } +} +``` + +### `$defs.bindingLocation.properties.catalog` +```json +{ + type: "string" + description: "NEW in v0.7.5: Iceberg catalog kind/profile selector — maps to iceberg.catalog.type (e.g. glue, rest..." +} +``` + ### `$defs.bindingLocation.properties.confluent_role_arn` ```json { @@ -83,6 +145,30 @@ } ``` +### `$defs.bindingLocation.properties.external_schema` +```json +{ + type: "string" + description: "NEW in v0.7.5: Redshift external (Spectrum) schema name created over a Glue database via the redshif..." +} +``` + +### `$defs.bindingLocation.properties.glue_database` +```json +{ + type: "string" + description: "NEW in v0.7.5: Glue Data Catalog database that backs the Redshift external schema." +} +``` + +### `$defs.bindingLocation.properties.iam_role_arn` +```json +{ + type: "string" + description: "NEW in v0.7.5: ARN of the IAM role attached to the Redshift Serverless namespace / used by the redsh..." +} +``` + ### `$defs.bindingLocation.properties.kafka_cluster_id` ```json { @@ -91,8 +177,83 @@ } ``` +### `$defs.bindingLocation.properties.namespace` +```json +{ + type: "string" + description: "NEW in v0.7.5: Amazon Redshift Serverless namespace name \u2014 maps to aws_redshiftserverless_namespace." +} +``` + +### `$defs.bindingLocation.properties.partitionBy` +```json +{ + type: "array" + items: { + type: "string" + } + description: "NEW in v0.7.5: Iceberg partition columns — a FLUID abstraction over the Iceberg PARTITIONED BY / Par..." +} +``` + +### `$defs.bindingLocation.properties.stream` +```json +{ + type: "string" + description: "NEW in v0.7.5: AWS Kinesis Data Stream name — maps to aws_kinesis_stream in the streaming IaC emitte..." +} +``` + +### `$defs.bindingLocation.properties.uri` +```json +{ + type: "string" + description: "NEW in v0.7.5: REST Iceberg catalog endpoint URL — the canonical Iceberg REST 'uri' property (maps t..." +} +``` + +### `$defs.bindingLocation.properties.warehouse` +```json +{ + type: "string" + description: "NEW in v0.7.5: Iceberg warehouse — the canonical Iceberg REST 'warehouse' property (maps to iceberg...." +} +``` + +### `$defs.bindingLocation.properties.workgroup` +```json +{ + type: "string" + description: "NEW in v0.7.5: Amazon Redshift Serverless workgroup name \u2014 maps to aws_redshiftserverless_workgroup." +} +``` + ## 📝 Modified Properties +### `$defs.binding.properties.format.description` + +**Before:** +```json +"Binding format. NEW in v0.7.4: 'postgres_table' (PostgreSQL driver), 'athena_table' / 'glue_table' (..." +``` + +**After:** +```json +"Binding format. NEW in v0.7.5: 'pgvector_table' — a pgvector embeddings table target for the vector ..." +``` + +### `$defs.binding.properties.format.enum` + +**Before:** +```json +[...21 items...] +``` + +**After:** +```json +[...22 items...] +``` + ### `$defs.binding.properties.platform.description` **Before:** @@ -102,7 +263,7 @@ **After:** ```json -"Target platform. NEW in v0.7.5: 'confluent' for the Confluent Cloud Tableflow managed Kafka→Iceberg ..." +"Target platform. NEW in v0.7.5: 'pgvector' for the vector/embeddings output port (pgvector-on-Postgr..." ``` ### `$defs.binding.properties.platform.enum` @@ -114,7 +275,7 @@ **After:** ```json -[...11 items...] +[...12 items...] ``` ### `$id` diff --git a/schema-diffs/diff-0.7.5-to-0.7.6.md b/schema-diffs/diff-0.7.5-to-0.7.6.md new file mode 100644 index 0000000..a89ef4b --- /dev/null +++ b/schema-diffs/diff-0.7.5-to-0.7.6.md @@ -0,0 +1,462 @@ +# Schema Changes: 0.7.5 → 0.7.6 + +**Total changes:** 30 +- ✅ Added: 22 +- ❌ Removed: 0 +- 📝 Modified: 8 + + +> **Note — 0.7.6 is a preview.** 0.7.5 is the latest stable version. A contract +> opts in to 0.7.6 by declaring `fluidVersion: "0.7.6"`; the reference +> implementation (data-product-forge 0.18.1) never picks a preview as the +> default for a contract that omits `fluidVersion`. A preview can still change +> before it becomes stable, so this diff describes the copy vendored from +> forge-cli v0.18.1. `scripts/check-compat.py --from 0.7.5 +> --to 0.7.6` reports no narrowing. + + +--- + +## ✅ Added Properties + +### `$defs.binding.allOf` +```json +[{"if": {"properties": {"platform": {"const": "gcp"}}, "required": ["platform"]}, "then": {"properties": {"encryption": {"properties": {"kms": {"not": {"pattern": "^(alias/|arn:)"}}}}, "principals": {"additionalProperties": {"anyOf": [{"type": "string", "pattern": "^(user|group|serviceAccount|domain):.+$"}, {"type": "array", "items": {"type": "string", "pattern": "^(user|group|serviceAccount|domain):.+$"}}]}}}}}, {"if": {"properties": {"platform": {"const": "aws"}}, "required": ["platform"]}, "then": {"properties": {"encryption": {"properties": {"kms": {"not": {"pattern": "^projects/"}}}}, "principals": {"additionalProperties": {"anyOf": [{"type": "string", "pattern": "^arn:aws[a-z0-9-]*:iam::"}, {"type": "array", "items": {"type": "string", "pattern": "^arn:aws[a-z0-9-]*:iam::"}}]}}}}}] +``` + +### `$defs.binding.properties.encryption` +```json +{ + type: "object" + additionalProperties: false + description: "NEW in v0.7.6: Encryption at rest for an AWS S3 binding or a GCP BigQuery binding. Absent: nothing i..." + properties: { + kms: { + type: "string" + default: "product" + anyOf: [...4 items...] + description: "NEW in v0.7.6: The KMS key the bucket's objects are encrypted with (SSE-KMS with an S3 Bucket Key). ..." + } + } +} +``` + +### `$defs.binding.properties.packaging` +```json +{ + $ref: "#/$defs/packaging" + description: "NEW in v0.7.6: Per-exposure packaging override — key-wise precedence over the contract-wide top-leve..." +} +``` + +### `$defs.binding.properties.principals` +```json +{ + type: "object" + description: "NEW in v0.7.6: Maps the contract's LOGICAL principals (accessPolicy.grants[].principal and exposes[]..." + propertyNames: { + minLength: 1 + } + additionalProperties: { + anyOf: [{"type": "string", "minLength": 1}, {"type": "array", "items": {"type": "string", "minLength": 1}, "uniqueItems": true}] + } +} +``` + +### `$defs.bindingGovernance.properties.lakeFormation.properties.bucketPolicy` +```json +{ + type: "string" + enum: ["cross-account", "none", "all-grantees"] + default: "cross-account" + description: "NEW in v0.7.6: Which grants[] principals get a statement in the aws_s3_bucket_policy emitted beside ..." +} +``` + +### `$defs.consumeRef.dependentRequired` +```json +{ + upstreamWorkspace: ["upstreamDigest"] +} +``` + +### `$defs.consumeRef.properties.upstreamDigest` +```json +{ + type: "string" + pattern: "^sha256:[0-9a-f]{64}$" + description: "Cross-mesh only. The contract digest this product was composed against, as sha256:<64 hex>. `fluid a..." +} +``` + +### `$defs.consumeRef.properties.upstreamWorkspace` +```json +{ + type: "string" + minLength: 1 + description: "Cross-mesh only. Id of the federated workspace that owns this upstream, as declared in federation/up..." +} +``` + +### `$defs.consumerRef` +```json +{ + type: "object" + additionalProperties: false + required: ["name", "type"] + $comment: "dbt-exposures-shaped (name/label/type/owner/url/maturity/description); exposeIds is the FLUID-native..." + properties: { + name: { + $ref: "#/$defs/identifier" + description: "Stable identifier for this consumer (unique within the contract)." + } + label: { + type: "string" + description: "Human-facing display name, e.g. \"Weekly Revenue Dashboard\"." + } + type: { + type: "string" + enum: [...5 items...] + description: "What kind of artifact consumes this product." + } + owner: { + type: "object" + additionalProperties: false + $comment: "Mirrors metadata.owner's vocabulary so 'owner' means one thing across the standard: team is the rout..." + properties: { + team: { + type: "string" + description: "Owning team for this consumer (routable identity, as in metadata.owner)." + } + email: { + type: "string" + format: "email" + description: "Contact address for the consumer's owner." + } + } + } + url: { + type: "string" + description: "Link to the live artifact (dashboard URL, notebook, app)." + } + ... (3 more) + } +} +``` + +### `$defs.exposeLifecycle` +```json +{ + type: "object" + additionalProperties: false + properties: { + state: { + $ref: "#/$defs/lifecycleState" + description: "Unified lifecycle state vocabulary." + } + retention: { + $ref: "#/$defs/isoDuration" + description: "How long this expose's data is kept (ISO-8601 duration, e.g. P30D, P7Y). A declaration unless `expir..." + } + expire: { + type: "boolean" + default: false + description: "NEW in v0.7.6: Delete stored data once it is older than `retention`. Default false, so a contract th..." + } + deprecationPolicy: { + type: "object" + additionalProperties: false + properties: { + noticePeriod: { + $ref: "#/$defs/isoDuration" + } + contact: { + type: "string" + } + replacement: { + type: "string" + description: "Reference to replacement data product or expose." + } + tags: { + $ref: "#/$defs/tags" + description: "Deprecation policy tags." + } + labels: { + $ref: "#/$defs/labels" + description: "Deprecation policy labels." + } + } + } + tags: { + $ref: "#/$defs/tags" + description: "Lifecycle tags for automation." + } + ... (1 more) + } + description: "NEW in v0.7.6: An expose's lifecycle. The same fields as the contract-root lifecycle, plus `expire`,..." +} +``` + +### `$defs.exposePolicy.properties.authz.properties.columnRestrictions.items.properties.access.description` +```json +"deny: the principal may not read the columns. allow: only the principals an allow names may read the..." +``` + +### `$defs.exposePolicy.properties.authz.properties.columnRestrictions.items.properties.columns.description` +```json +"Columns of this expose's schema." +``` + +### `$defs.exposePolicy.properties.authz.properties.columnRestrictions.items.properties.principal.description` +```json +"Logical principal the restriction applies to, written as in accessPolicy (e.g. group:analysts@compan..." +``` + +### `$defs.exposePolicy.properties.privacy.properties.masking.description` +```json +"Columns to treat before they land. The DuckDB acquisition runner rewrites each named column inside t..." +``` + +### `$defs.exposePolicy.properties.privacy.properties.masking.items.properties.params.description` +```json +"Strategy parameters. The runner accepts exactly these keys and refuses any other, including a litera..." +``` + +### `$defs.exposePolicy.properties.privacy.properties.masking.items.properties.params.properties` +```json +{ + saltEnv: { + type: "string" + pattern: "^[A-Z_][A-Z0-9_]{0,127}$" + default: "FLUID_PII_HASH_SECRET" + description: "NEW in v0.7.6: hash only. The environment variable holding the salt, at least 16 bytes. Unset or emp..." + } + keyEnv: { + type: "string" + pattern: "^[A-Z_][A-Z0-9_]{0,127}$" + description: "NEW in v0.7.6: tokenize and encrypt. The environment variable holding the key. tokenize defaults to ..." + } + keepFirst: { + type: "integer" + minimum: 0 + maximum: 64 + default: 0 + description: "NEW in v0.7.6: mask only. Leading characters left as they are." + } + keepLast: { + type: "integer" + minimum: 0 + maximum: 64 + default: 4 + description: "NEW in v0.7.6: mask only. Trailing characters left as they are, so +46701234567 lands as ********456..." + } +} +``` + +### `$defs.exposePolicy.properties.privacy.properties.masking.items.properties.strategy.description` +```json +"hash: lowercase hex SHA-256 of salt || value, deterministic so joins work (64 hex characters). mask:..." +``` + +### `$defs.packaging` +```json +{ + type: "object" + additionalProperties: false + description: "NEW in v0.7.6: Declarative packaging mode — per-container ownership for the OpenTofu IaC emit (RFC-p..." + properties: { + mode: { + type: "string" + enum: ["isolated", "shared"] + description: "NEW in v0.7.6: Blanket ownership mode for every container kind. Default when the block is present: '..." + } + pool: { + type: "string" + minLength: 1 + description: "NEW in v0.7.6: Pool id of the platform-owned tenant pool this product writes into. REQUIRED (enforce..." + } + poolManifest: { + type: "string" + minLength: 1 + description: "NEW in v0.7.6: Optional path to the platform team's pool manifest file; snapshotted into the bundle ..." + } + containers: { + type: "object" + additionalProperties: false + description: "NEW in v0.7.6: Per-container-kind ownership overrides — each key wins over `mode` for that kind, yie..." + properties: { + bucket: { + $ref: "#/$defs/packagingContainerMode" + description: "NEW in v0.7.6: Ownership of the object-store bucket (aws_s3_bucket / google_storage_bucket)." + } + database: { + $ref: "#/$defs/packagingContainerMode" + description: "NEW in v0.7.6: Ownership of the database container — covers Snowflake `database` AND AWS `glue_datab..." + } + dataset: { + $ref: "#/$defs/packagingContainerMode" + description: "NEW in v0.7.6: Ownership of the BigQuery dataset (google_bigquery_dataset)." + } + schema: { + $ref: "#/$defs/packagingContainerMode" + description: "NEW in v0.7.6: Ownership of the Snowflake schema (snowflake_schema)." + } + warehouse: { + $ref: "#/$defs/packagingContainerMode" + description: "NEW in v0.7.6: Ownership of the Snowflake warehouse (snowflake_warehouse) — isolated gives per-produ..." + } + ... (1 more) + } + } + } +} +``` + +### `$defs.packagingContainerMode` +```json +{ + type: "string" + enum: ["isolated", "shared"] + description: "NEW in v0.7.6: Per-container ownership — 'isolated' (this product creates and owns it) or 'shared' (..." +} +``` + +### `$defs.semanticModel.properties.measures.items.properties.aggParams` +```json +{ + type: "object" + additionalProperties: false + description: "Aggregation parameters (mirrors dbt-semantic-interfaces agg_params). Currently used by agg=percentil..." + properties: { + percentile: { + type: "number" + minimum: 0 + maximum: 1 + description: "Percentile in [0, 1] for agg=percentile (0.5 = median). Consumers default to 0.5 when omitted." + } + useDiscretePercentile: { + type: "boolean" + description: "Use the discrete percentile (PERCENTILE_DISC / use_discrete_percentile) instead of continuous interp..." + } + } +} +``` + +### `properties.consumers` +```json +{ + type: "array" + description: "NEW in v0.7.6: Declared downstream consumers of this data product — the business artifacts built on ..." + items: { + $ref: "#/$defs/consumerRef" + } +} +``` + +### `properties.packaging` +```json +{ + $ref: "#/$defs/packaging" + description: "NEW in v0.7.6: Contract-wide packaging default — declarative container ownership (isolated = this pr..." +} +``` + +## 📝 Modified Properties + +### `$defs.bindingLocation.properties.partitionBy.description` + +**Before:** +```json +"NEW in v0.7.5: Iceberg partition columns — a FLUID abstraction over the Iceberg PARTITIONED BY / Par..." +``` + +**After:** +```json +"NEW in v0.7.5: Iceberg partition columns — a FLUID abstraction over the Iceberg PARTITIONED BY / Par..." +``` + +### `$defs.expose.properties.lifecycle.$ref` + +**Before:** +```json +"#/$defs/lifecycle" +``` + +**After:** +```json +"#/$defs/exposeLifecycle" +``` + +### `$defs.exposePolicy.properties.authz.properties.columnRestrictions.description` + +**Before:** +```json +"Column-level access control." +``` + +**After:** +```json +"Column-level access control. A column named in any restriction is restricted. 'deny': the principal ..." +``` + +### `$id` + +**Before:** +```json +"https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.5.json" +``` + +**After:** +```json +"https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.6.json" +``` + +### `description` + +**Before:** +```json +"FLUID Data Product contract (v0.7.5). Streaming Kafka→Iceberg Sink & Confluent Tableflow Release. + +🔥..." +``` + +**After:** +```json +"FLUID contract schema 0.7.6 — the current PREVIEW version, opened when 0.7.5 was promoted to stable...." +``` + +### `properties.fluidVersion.description` + +**Before:** +```json +"Contract schema version. Accepts '0.7.3' (source-aligned data products + acquisition pattern + inges..." +``` + +**After:** +```json +"Contract schema version. Accepts '0.7.3' (source-aligned data products + acquisition pattern + inges..." +``` + +### `properties.fluidVersion.enum` + +**Before:** +```json +["0.7.3", "0.7.4", "0.7.5"] +``` + +**After:** +```json +[...4 items...] +``` + +### `title` + +**Before:** +```json +"FLUID 0.7.5 \u2014 Streaming Kafka\u2192Iceberg Sink & Confluent Tableflow" +``` + +**After:** +```json +"FLUID 0.7.6 \u2014 Declarative Packaging Modes (preview)" +``` diff --git a/schema/fluid-schema-0.7.3.json b/schema/fluid-schema-0.7.3.json index a8f8f32..7f26de9 100644 --- a/schema/fluid-schema-0.7.3.json +++ b/schema/fluid-schema-0.7.3.json @@ -1377,7 +1377,7 @@ "properties": { "admins": { "type": "array", - "description": "IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. Idempotent: re-applying with the same set is a no-op; principals not listed are NOT removed (LF resource semantics preserve other admins).", + "description": "IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. AUTHORITATIVE, not additive: Lake Formation's PutDataLakeSettings replaces the account's admin list with the one sent, so an admin NOT listed here is REMOVED on apply. List every principal that must stay a data lake admin, including the role or user that runs the apply: left out, it loses Lake Formation admin on that apply, and the Lake Formation grants and registrations that need an admin can then fail. The terraform-provider-aws resource also clears the data lake settings its configuration omits, and this block sets only the admins, so the account's create-database and create-table default permissions (the IAMAllowedPrincipals default for new databases and tables), trusted resource owners and parameters are cleared too. Destroying the resource empties the admin list and resets CROSS_ACCOUNT_VERSION to 1. Re-applying the same list is a no-op.", "items": { "type": "string", "pattern": "^arn:aws[a-z0-9-]*:iam::" diff --git a/schema/fluid-schema-0.7.4.json b/schema/fluid-schema-0.7.4.json index bd99c54..613ea0a 100644 --- a/schema/fluid-schema-0.7.4.json +++ b/schema/fluid-schema-0.7.4.json @@ -5280,7 +5280,7 @@ "properties": { "admins": { "type": "array", - "description": "IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. Idempotent: re-applying with the same set is a no-op; principals not listed are NOT removed (LF resource semantics preserve other admins).", + "description": "IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. AUTHORITATIVE, not additive: Lake Formation's PutDataLakeSettings replaces the account's admin list with the one sent, so an admin NOT listed here is REMOVED on apply. List every principal that must stay a data lake admin, including the role or user that runs the apply: left out, it loses Lake Formation admin on that apply, and the Lake Formation grants and registrations that need an admin can then fail. The terraform-provider-aws resource also clears the data lake settings its configuration omits, and this block sets only the admins, so the account's create-database and create-table default permissions (the IAMAllowedPrincipals default for new databases and tables), trusted resource owners and parameters are cleared too. Destroying the resource empties the admin list and resets CROSS_ACCOUNT_VERSION to 1. Re-applying the same list is a no-op.", "items": { "type": "string", "pattern": "^arn:aws[a-z0-9-]*:iam::" diff --git a/schema/fluid-schema-0.7.5.json b/schema/fluid-schema-0.7.5.json index ccf8421..e21ece2 100644 --- a/schema/fluid-schema-0.7.5.json +++ b/schema/fluid-schema-0.7.5.json @@ -2,7 +2,7 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.5.json", "title": "FLUID 0.7.5 \u2014 Streaming Kafka\u2192Iceberg Sink & Confluent Tableflow", - "description": "FLUID Data Product contract (v0.7.5). Streaming Kafka\u2192Iceberg Sink & Confluent Tableflow Release.\n\n\ud83d\udd25 NEW in v0.7.5:\n\u2022 Streaming Iceberg sink \u2014 The `kafka-connect` acquisition pattern gains an opt-in Iceberg streaming sink: `iceberg_sink_enabled` (derive the sink connector config; off by default when a hand-written `sink_connector_config` is present), `streamingSink` (commit interval, dynamic routing, auto-create, schema-evolution tuning \u2014 RFC \u00a76.2), `sink_topics`, and `iceberg_catalog_overrides` (operator escape hatch merged last over the derived config).\n\u2022 Confluent Cloud Tableflow \u2014 `binding.platform: confluent` plus `binding.location` Tableflow fields (`environment_id`, `kafka_cluster_id`, `confluent_role_arn`) for the managed Kafka\u2192Iceberg emitter.\n\u2022 Backward compatible with v0.7.4 \u2014 All v0.7.4 contracts remain valid; the streaming-sink keys and the `confluent` binding are additive and optional.\n\n\ud83d\udccb Inherited from v0.7.4 (Runtime agentPolicy Enforcement at the MCP Gateway):\n\u2022 MCP output-port gateway \u2014 An `expose.mcp` block declares an expose as agent-consumable over MCP (sampling caps + classification); its presence opts the expose into `fluid mcp output-port serve`.\n\u2022 Runtime agentPolicy enforcement \u2014 When an `expose.mcp` block is present, `policy.agentPolicy.allowedModels` / `deniedModels` and `allowedUseCases` / `deniedUseCases` are enforced at the Fluid MCP gateway on every read.\n\n\ud83d\udccb Inherited from v0.7.3 (Source-Aligned Acquisition):\n\u2022 Acquisition build pattern \u2014 First-class support for source-aligned data products that ingest external systems (Salesforce, Postgres, Kafka, files) into the mesh.\n\u2022 Six ingestion engines \u2014 duckdb (zero-infra files/JDBC), airbyte (350+ SaaS connectors), meltano (600+ Singer taps), dlt (Python-native), kafka-connect (streaming), debezium (CDC).\n\u2022 Three deployment modes \u2014 embedded (in-process), bring-your-own (existing server), managed (Forge provisions via Helm/Compose/OpenTofu).\n\u2022 Capability-based negotiation \u2014 Runners declare {full_refresh, incremental_append, incremental_dedup, cdc, streaming, schema_discovery, dlp_scan, exactly_once}; contract asks validated against declarations.\n\u2022 Schema evolution semantics \u2014 schemaPolicy \u2208 {strict, discover_and_freeze, evolve_safe, evolve_all} with concrete per-change behavior.\n\u2022 Delivery guarantees \u2014 at_most_once | at_least_once | exactly_once with idempotency keys and DLQ semantics.\n\u2022 Source supply-chain security \u2014 Cosign image signature verification + SLSA provenance required in production.\n\u2022 Top-level retention block \u2014 runState, runLogs, lineage, dlq durations.\n\u2022 Extended sovereignty.dataResidency \u2014 region pinning + prohibitTransferTo for runtime enforcement.\n\n\ud83d\udccb Inherited from v0.7.2 (Semantic Layer):\n\u2022 Semantic model definitions \u2014 Optional `semantics` block on each expose for entities, measures, dimensions, and metrics.\n\u2022 Agent-consumable business logic \u2014 Eliminates semantic hallucination by providing machine-readable metric definitions.\n\u2022 Industry-aligned primitives \u2014 Compatible with dbt MetricFlow, Snowflake Semantic Views, and OSI-aligned formats.\n\n\ud83d\udccb Inherited from v0.7.1 (Provider-First Orchestration + Agentic Governance):\n\u2022 Agentic governance \u2014 AI/LLM usage policies (agentPolicy) declare which models can consume data and for what purposes (runtime enforcement added in v0.7.4 \u2014 see above).\n\u2022 Sovereignty constraints \u2014 Top-level jurisdiction and data residency requirements for compliance enforcement.\n\u2022 Orchestration philosophy \u2014 Pragmatic engine enumeration (v0.7.x) with path to capability-based abstraction (v0.8.0+).\n\u2022 Provider actions as first-class tasks \u2014 Direct invocation of provider actions (aws.s3.ensure_bucket, sf.table.ensure) without wrapper operators.\n\u2022 Unified task type taxonomy \u2014 provider_action, fluid_*, bash, python, sensor.\n\u2022 Data product references in datasets \u2014 Link trigger datasets to actual `exposes[]` with strong typing and auto-URI generation.\n\u2022 Task output \u2192 expose mapping \u2014 Formal `produces[]` declaration for automatic dataset registration and lineage.\n\u2022 Cross-provider workflow patterns \u2014 Native support for multi-cloud pipelines with XCom data passing.\n\u2022 Cost tracking \u2014 Track actual vs estimated costs with budget enforcement.\n\n\ud83c\udfaf Design Philosophy:\n\u2022 Provider actions = orchestration primitives.\n\u2022 Contracts drive both infrastructure and workflows.\n\u2022 Strong typing and validation.\n\u2022 Natural multi-cloud/multi-provider patterns.\n\u2022 Minimal abstraction layers.\n\u2022 Agentic governance: data owners control AI/LLM usage boundaries.\n\u2022 Sovereignty-first: jurisdiction constraints enforced at contract level.\n\n\u2699\ufe0f Orchestration Philosophy (v0.7.x):\n\u2022 Pragmatic approach: enumerate known engines (airflow, dagster, prefect) for immediate adoption.\n\u2022 Trade-off: implementation-specific vs pure abstraction \u2014 favouring real-world deployment patterns.\n\u2022 Future path: v0.8.0+ will evolve toward a capability-based orchestration model.\n\u2022 Escape hatch: `custom` engine allows experimentation without schema changes.", + "description": "FLUID Data Product contract (v0.7.5). Streaming Kafka\u2192Iceberg Sink & Confluent Tableflow Release.\n\n🔥 NEW in v0.7.5:\n\u2022 Streaming Iceberg sink \u2014 The `kafka-connect` acquisition pattern gains an opt-in Iceberg streaming sink: `iceberg_sink_enabled` (derive the sink connector config; off by default when a hand-written `sink_connector_config` is present), `streamingSink` (commit interval, dynamic routing, auto-create, schema-evolution tuning \u2014 RFC \u00a76.2), `sink_topics`, and `iceberg_catalog_overrides` (operator escape hatch merged last over the derived config).\n\u2022 Confluent Cloud Tableflow \u2014 `binding.platform: confluent` plus `binding.location` Tableflow fields (`environment_id`, `kafka_cluster_id`, `confluent_role_arn`) for the managed Kafka\u2192Iceberg emitter.\n\u2022 Backward compatible with v0.7.4 \u2014 All v0.7.4 contracts remain valid; the streaming-sink keys and the `confluent` binding are additive and optional.\n\n📋 Inherited from v0.7.4 (Runtime agentPolicy Enforcement at the MCP Gateway):\n\u2022 MCP output-port gateway \u2014 An `expose.mcp` block declares an expose as agent-consumable over MCP (sampling caps + classification); its presence opts the expose into `fluid mcp output-port serve`.\n\u2022 Runtime agentPolicy enforcement \u2014 When an `expose.mcp` block is present, `policy.agentPolicy.allowedModels` / `deniedModels` and `allowedUseCases` / `deniedUseCases` are enforced at the Fluid MCP gateway on every read.\n\n📋 Inherited from v0.7.3 (Source-Aligned Acquisition):\n\u2022 Acquisition build pattern \u2014 First-class support for source-aligned data products that ingest external systems (Salesforce, Postgres, Kafka, files) into the mesh.\n\u2022 Six ingestion engines \u2014 duckdb (zero-infra files/JDBC), airbyte (350+ SaaS connectors), meltano (600+ Singer taps), dlt (Python-native), kafka-connect (streaming), debezium (CDC).\n\u2022 Three deployment modes \u2014 embedded (in-process), bring-your-own (existing server), managed (Forge provisions via Helm/Compose/OpenTofu).\n\u2022 Capability-based negotiation \u2014 Runners declare {full_refresh, incremental_append, incremental_dedup, cdc, streaming, schema_discovery, dlp_scan, exactly_once}; contract asks validated against declarations.\n\u2022 Schema evolution semantics \u2014 schemaPolicy \u2208 {strict, discover_and_freeze, evolve_safe, evolve_all} with concrete per-change behavior.\n\u2022 Delivery guarantees \u2014 at_most_once | at_least_once | exactly_once with idempotency keys and DLQ semantics.\n\u2022 Source supply-chain security \u2014 Cosign image signature verification + SLSA provenance required in production.\n\u2022 Top-level retention block \u2014 runState, runLogs, lineage, dlq durations.\n\u2022 Extended sovereignty.dataResidency \u2014 region pinning + prohibitTransferTo for runtime enforcement.\n\n📋 Inherited from v0.7.2 (Semantic Layer):\n\u2022 Semantic model definitions \u2014 Optional `semantics` block on each expose for entities, measures, dimensions, and metrics.\n\u2022 Agent-consumable business logic \u2014 Eliminates semantic hallucination by providing machine-readable metric definitions.\n\u2022 Industry-aligned primitives \u2014 Compatible with dbt MetricFlow, Snowflake Semantic Views, and OSI-aligned formats.\n\n📋 Inherited from v0.7.1 (Provider-First Orchestration + Agentic Governance):\n\u2022 Agentic governance \u2014 AI/LLM usage policies (agentPolicy) declare which models can consume data and for what purposes (runtime enforcement added in v0.7.4 \u2014 see above).\n\u2022 Sovereignty constraints \u2014 Top-level jurisdiction and data residency requirements for compliance enforcement.\n\u2022 Orchestration philosophy \u2014 Pragmatic engine enumeration (v0.7.x) with path to capability-based abstraction (v0.8.0+).\n\u2022 Provider actions as first-class tasks \u2014 Direct invocation of provider actions (aws.s3.ensure_bucket, sf.table.ensure) without wrapper operators.\n\u2022 Unified task type taxonomy \u2014 provider_action, fluid_*, bash, python, sensor.\n\u2022 Data product references in datasets \u2014 Link trigger datasets to actual `exposes[]` with strong typing and auto-URI generation.\n\u2022 Task output \u2192 expose mapping \u2014 Formal `produces[]` declaration for automatic dataset registration and lineage.\n\u2022 Cross-provider workflow patterns \u2014 Native support for multi-cloud pipelines with XCom data passing.\n\u2022 Cost tracking \u2014 Track actual vs estimated costs with budget enforcement.\n\n🎯 Design Philosophy:\n\u2022 Provider actions = orchestration primitives.\n\u2022 Contracts drive both infrastructure and workflows.\n\u2022 Strong typing and validation.\n\u2022 Natural multi-cloud/multi-provider patterns.\n\u2022 Minimal abstraction layers.\n\u2022 Agentic governance: data owners control AI/LLM usage boundaries.\n\u2022 Sovereignty-first: jurisdiction constraints enforced at contract level.\n\n\u2699\ufe0f Orchestration Philosophy (v0.7.x):\n\u2022 Pragmatic approach: enumerate known engines (airflow, dagster, prefect) for immediate adoption.\n\u2022 Trade-off: implementation-specific vs pure abstraction \u2014 favouring real-world deployment patterns.\n\u2022 Future path: v0.8.0+ will evolve toward a capability-based orchestration model.\n\u2022 Escape hatch: `custom` engine allows experimentation without schema changes.", "type": "object", "additionalProperties": false, "required": [ @@ -1268,9 +1268,10 @@ "local", "kubernetes", "postgres", + "pgvector", "other" ], - "description": "Target platform. NEW in v0.7.5: 'confluent' for the Confluent Cloud Tableflow managed Kafka→Iceberg emitter. NEW in v0.7.4: 'postgres' for the MCP output-port PostgreSQL driver (AWS Athena uses platform 'aws')." + "description": "Target platform. NEW in v0.7.5: 'pgvector' for the vector/embeddings output port (pgvector-on-PostgreSQL RAG target — emits the embeddings table + ANN index DDL for the ai-embeddable columns). NEW in v0.7.5: 'confluent' for the Confluent Cloud Tableflow managed Kafka→Iceberg emitter. NEW in v0.7.4: 'postgres' for the MCP output-port PostgreSQL driver (AWS Athena uses platform 'aws')." }, "format": { "type": "string", @@ -1295,9 +1296,10 @@ "postgres_table", "athena_table", "glue_table", + "pgvector_table", "other" ], - "description": "Binding format. NEW in v0.7.4: 'postgres_table' (PostgreSQL driver), 'athena_table' / 'glue_table' (AWS Athena driver) for the MCP output port." + "description": "Binding format. NEW in v0.7.5: 'pgvector_table' — a pgvector embeddings table target for the vector output port. NEW in v0.7.4: 'postgres_table' (PostgreSQL driver), 'athena_table' / 'glue_table' (AWS Athena driver) for the MCP output port." }, "icebergConfig": { "type": "object", @@ -1404,6 +1406,93 @@ } } }, + "vectorConfig": { + "type": "object", + "additionalProperties": false, + "description": "NEW in v0.7.5: vector / embeddings output-port configuration (platform 'pgvector', format 'pgvector_table'). Consumed by fluid_build.output_ports.vector to emit the pgvector embeddings-table + ANN-index DDL and a RAG provenance manifest for the schema's ai-embeddable columns.", + "required": [ + "dimensions" + ], + "properties": { + "dimensions": { + "type": "integer", + "minimum": 1, + "maximum": 16000, + "description": "Embedding vector width (e.g. 1536 for OpenAI text-embedding-3-small, 1024 for Cohere embed-english-light). Maps to the pgvector column type vector()." + }, + "embeddingModel": { + "type": "string", + "description": "Provenance: the embedding model that produces the vectors (e.g. 'text-embedding-3-small'). Recorded in the RAG manifest and the embeddings-table's embedding_model column so a retriever can verify model parity." + }, + "vectorType": { + "type": "string", + "enum": [ + "vector", + "halfvec" + ], + "default": "vector", + "description": "pgvector column type: 'vector' (float32, up to 2000 dims) or 'halfvec' (float16, half the storage, up to 4000 dims)." + }, + "indexType": { + "type": "string", + "enum": [ + "hnsw", + "ivfflat", + "none" + ], + "default": "hnsw", + "description": "ANN index built on the embedding column. 'hnsw' (higher recall / lower query latency, buildable before data lands), 'ivfflat' (lower build cost, needs data first), or 'none' (no index — exact search only)." + }, + "distanceMetric": { + "type": "string", + "enum": [ + "cosine", + "l2", + "inner_product", + "l1" + ], + "default": "cosine", + "description": "Distance function the index optimises for → pgvector operator class: cosine→vector_cosine_ops, l2→vector_l2_ops, inner_product→vector_ip_ops, l1→vector_l1_ops." + }, + "table": { + "type": "string", + "description": "Target embeddings table name. Defaults to '_embeddings' when omitted." + }, + "sourceKeyColumn": { + "type": "string", + "description": "Provenance: the source primary-key column that ties each embedding row back to its origin row (written to the embeddings-table's source_key column). Enables update/delete propagation from the source product to the vector store." + }, + "hnsw": { + "type": "object", + "additionalProperties": false, + "description": "HNSW index tuning (used when indexType is 'hnsw').", + "properties": { + "m": { + "type": "integer", + "minimum": 1, + "description": "Max bi-directional links per node (pgvector default 16)." + }, + "efConstruction": { + "type": "integer", + "minimum": 1, + "description": "Build-time dynamic candidate-list size (pgvector default 64)." + } + } + }, + "ivfflat": { + "type": "object", + "additionalProperties": false, + "description": "IVFFlat index tuning (used when indexType is 'ivfflat').", + "properties": { + "lists": { + "type": "integer", + "minimum": 1, + "description": "Number of inverted lists (pgvector guidance: rows/1000 for < 1M rows)." + } + } + } + } + }, "location": { "$ref": "#/$defs/bindingLocation" }, @@ -1496,9 +1585,52 @@ "type": "string", "description": "NEW in v0.7.5: Confluent Cloud Kafka cluster id (lkc-xxxxx) the Tableflow topic belongs to." }, + "stream": { + "type": "string", + "description": "NEW in v0.7.5: AWS Kinesis Data Stream name — maps to aws_kinesis_stream in the streaming IaC emitter." + }, + "namespace": { + "type": "string", + "description": "NEW in v0.7.5: Amazon Redshift Serverless namespace name — maps to aws_redshiftserverless_namespace." + }, + "workgroup": { + "type": "string", + "description": "NEW in v0.7.5: Amazon Redshift Serverless workgroup name — maps to aws_redshiftserverless_workgroup." + }, + "iam_role_arn": { + "type": "string", + "description": "NEW in v0.7.5: ARN of the IAM role attached to the Redshift Serverless namespace / used by the redshift-data CREATE EXTERNAL SCHEMA bridge." + }, + "external_schema": { + "type": "string", + "description": "NEW in v0.7.5: Redshift external (Spectrum) schema name created over a Glue database via the redshift-data CREATE EXTERNAL SCHEMA bridge." + }, + "glue_database": { + "type": "string", + "description": "NEW in v0.7.5: Glue Data Catalog database that backs the Redshift external schema." + }, "confluent_role_arn": { "type": "string", "description": "NEW in v0.7.5: ARN of the pre-created AWS IAM role Confluent Tableflow assumes (byob_aws). Its trust policy is updated post-apply with the provider integration's external_id (two-phase IAM)." + }, + "catalog": { + "type": "string", + "description": "NEW in v0.7.5: Iceberg catalog kind/profile selector — maps to iceberg.catalog.type (e.g. glue, rest, nessie, hive). REST-fronted catalogs (polaris / unity / snowflake open catalog) map to 'rest'. Consumed by providers/_iceberg_catalog.py and the streaming-sink validator." + }, + "warehouse": { + "type": "string", + "description": "NEW in v0.7.5: Iceberg warehouse — the canonical Iceberg REST 'warehouse' property (maps to iceberg.catalog.warehouse): an object-store URI (s3://|gs://|abfss://) for object-store catalogs, or the catalog/warehouse name for a REST catalog." + }, + "uri": { + "type": "string", + "description": "NEW in v0.7.5: REST Iceberg catalog endpoint URL — the canonical Iceberg REST 'uri' property (maps to iceberg.catalog.uri)." + }, + "partitionBy": { + "type": "array", + "items": { + "type": "string" + }, + "description": "NEW in v0.7.5: Iceberg partition columns — a FLUID abstraction over the Iceberg PARTITIONED BY / PartitionSpec (maps to iceberg.tables.default-partition-by)." } } }, @@ -5341,7 +5473,7 @@ "properties": { "admins": { "type": "array", - "description": "IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. Idempotent: re-applying with the same set is a no-op; principals not listed are NOT removed (LF resource semantics preserve other admins).", + "description": "IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. AUTHORITATIVE, not additive: Lake Formation's PutDataLakeSettings replaces the account's admin list with the one sent, so an admin NOT listed here is REMOVED on apply. List every principal that must stay a data lake admin, including the role or user that runs the apply: left out, it loses Lake Formation admin on that apply, and the Lake Formation grants and registrations that need an admin can then fail. The terraform-provider-aws resource also clears the data lake settings its configuration omits, and this block sets only the admins, so the account's create-database and create-table default permissions (the IAMAllowedPrincipals default for new databases and tables), trusted resource owners and parameters are cleared too. Destroying the resource empties the admin list and resets CROSS_ACCOUNT_VERSION to 1. Re-applying the same list is a no-op.", "items": { "type": "string", "pattern": "^arn:aws[a-z0-9-]*:iam::" diff --git a/schema/fluid-schema-0.7.6.json b/schema/fluid-schema-0.7.6.json new file mode 100644 index 0000000..28ca63e --- /dev/null +++ b/schema/fluid-schema-0.7.6.json @@ -0,0 +1,6044 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://open-data-protocol.github.io/fluid/schema/fluid-schema-0.7.6.json", + "title": "FLUID 0.7.6 — Declarative Packaging Modes (preview)", + "description": "FLUID contract schema 0.7.6 — the current PREVIEW version, opened when 0.7.5 was promoted to stable.\n\n🔥 NEW in v0.7.6:\n• Declarative packaging modes — a top-level `packaging` block (plus a per-exposure `binding.packaging` override) declares per-container ownership for the IaC emit: `isolated` (this product creates and owns the container) vs `shared` (a pre-existing, platform-owned pool referenced via an OpenTofu data source), with a per-kind `containers` override map for hybrid tiers and a `pool` id for tenancy/cost attribution (RFC-packaging-modes.md).\n• Backward compatible with v0.7.5 — every 0.7.5 contract validates as 0.7.6 unchanged; an ABSENT packaging block keeps the legacy byte-identical emit.", + "type": "object", + "additionalProperties": false, + "required": [ + "fluidVersion", + "kind", + "id", + "name", + "metadata", + "exposes" + ], + "examples": [ + { + "fluidVersion": "0.5.7", + "kind": "DataProduct", + "id": "gold.customer.analytics_360_v1", + "name": "Customer 360 Analytics", + "description": "Unified customer profiles with ML-driven insights", + "domain": "Customer Experience", + "tags": [ + "customer-data", + "analytics", + "gold-layer" + ], + "labels": { + "team": "customer-analytics", + "criticality": "high", + "cost-center": "engineering" + }, + "metadata": { + "layer": "Gold", + "owner": { + "team": "customer-analytics", + "email": "customer-analytics@company.com" + }, + "businessContext": { + "domain": "Customer Experience", + "subdomain": "Customer Intelligence" + } + }, + "exposes": [ + { + "exposeId": "customer_profiles", + "title": "Unified Customer Profiles", + "version": "2.1.0", + "kind": "table", + "tags": [ + "pii", + "customer-facing", + "real-time" + ], + "labels": { + "sensitivity": "high", + "retention": "7-years", + "region": "us-central1" + }, + "contract": { + "schema": [ + { + "name": "customer_id", + "type": "STRING", + "required": true, + "description": "Unique customer identifier", + "sensitivity": "cleartext", + "tags": [ + "identifier", + "primary-key" + ], + "labels": { + "business-name": "Customer ID", + "data-category": "identifier" + } + } + ] + }, + "binding": { + "platform": "gcp", + "format": "bigquery_table", + "location": { + "project": "company-data", + "dataset": "gold_customer", + "table": "profiles_v1" + } + } + } + ] + } + ], + "properties": { + "extensions": { + "type": "object", + "additionalProperties": true, + "description": "Vendor / plugin-namespaced configuration. Each plugin claims a single sub-key (e.g. customScaffold, customCatalog) and validates its own schema there via the fluid_build.extension_validators entry-point." + }, + "fluidVersion": { + "type": "string", + "pattern": "^\\d+\\.\\d+(\\.\\d+)?$", + "enum": [ + "0.7.3", + "0.7.4", + "0.7.5", + "0.7.6" + ], + "description": "Contract schema version. Accepts '0.7.3' (source-aligned data products + acquisition pattern + ingestion engines) or '0.7.4' (adds the MCP output-port gateway + runtime agentPolicy enforcement). 0.7.4 is fully backward-compatible with 0.7.3 — every 0.7.3 contract validates as 0.7.4 unchanged. '0.7.6' is the current preview.", + "examples": [ + "0.7.4", + "0.7.3" + ] + }, + "kind": { + "type": "string", + "enum": [ + "DataProduct", + "MLPipeline" + ], + "description": "Kind of contract. 'MLPipeline' added for basic ML support." + }, + "id": { + "$ref": "#/$defs/identifier", + "description": "Stable product identifier, e.g. 'gold.hr.employee_360_v1'." + }, + "name": { + "type": "string", + "minLength": 1 + }, + "description": { + "type": "string" + }, + "domain": { + "type": "string", + "description": "Business domain/mesh domain (e.g., 'HR', 'Finance')." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Product-level tags for categorization and discovery." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Product-level labels for metadata and automation." + }, + "metadata": { + "type": "object", + "additionalProperties": false, + "required": [ + "owner" + ], + "properties": { + "provenance": { + "type": "object", + "description": "Generation envelope injected by `fluid forge` / `fluid init` — records how, when and by which tool/command the contract was generated. Machine-written (not user-authored); additive metadata that does not affect the data product's semantic contract.", + "properties": { + "schema_version": { + "type": "integer", + "description": "Envelope schema version." + }, + "kind": { + "type": "string", + "description": "Envelope discriminator, e.g. 'ContractMetadata'." + }, + "generated_at": { + "type": "string", + "format": "date-time", + "description": "ISO 8601 UTC timestamp of generation." + }, + "generated_by": { + "type": "object", + "description": "Tool, version and command that generated the contract.", + "properties": { + "tool": { + "type": "string" + }, + "version": { + "type": "string" + }, + "command": { + "type": "string" + } + } + } + } + }, + "layer": { + "type": "string", + "description": "Data layer label (e.g., Bronze/Silver/Gold). Free-form. When set together with `productType`, the two MUST be consistent under the canonical mapping (Bronze↔SDP, Silver↔ADP, Gold↔CDP)." + }, + "productType": { + "type": "string", + "enum": [ + "SDP", + "ADP", + "CDP" + ], + "description": "NEW in v0.7.3: Data Mesh data-product type. SDP = Source-Aligned Data Product (raw ingestion from external systems; corresponds to Bronze). ADP = Aggregated Data Product (cross-source joined / domain-modelled; corresponds to Silver). CDP = Consumption-Aligned Data Product (analytics- or product-shaped; corresponds to Gold). Either `metadata.layer` or `metadata.productType` is sufficient — the validator infers the missing one. When both are present they MUST agree." + }, + "classification": { + "type": "string", + "enum": [ + "public", + "internal", + "confidential", + "restricted" + ], + "description": "NEW in v0.7.3: Data classification label propagated to catalog + access policy enforcement." + }, + "experimental": { + "type": "array", + "description": "NEW in v0.7.3: Feature gates the contract opts into (e.g., 'acquisition' while Bronze acquisition is in beta).", + "items": { + "type": "string" + }, + "uniqueItems": true + }, + "owner": { + "type": "object", + "additionalProperties": false, + "properties": { + "team": { + "type": "string" + }, + "email": { + "type": "string", + "format": "email" + }, + "slack": { + "type": "string" + }, + "oncall": { + "type": "string", + "description": "Oncall rotation or contact for operational issues." + } + } + }, + "createdAt": { + "type": "string", + "format": "date-time" + }, + "businessContext": { + "type": "object", + "description": "Business alignment information for data mesh organization.", + "additionalProperties": false, + "properties": { + "domain": { + "type": "string", + "description": "Business domain this product belongs to." + }, + "subdomain": { + "type": "string", + "description": "Business subdomain for more granular organization." + }, + "businessCapability": { + "type": "string", + "description": "Business capability this product supports." + }, + "valueStream": { + "type": "string", + "description": "Value stream this product participates in." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Product-level tags on metadata for discovery and categorization." + } + } + }, + "exposes": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/expose" + } + }, + "consumes": { + "type": "array", + "items": { + "$ref": "#/$defs/consumeRef" + } + }, + "builds": { + "type": "array", + "description": "NEW in v0.5.5: Multiple build configurations for multi-modal data products (e.g., batch + streaming, SQL + ML).", + "items": { + "$ref": "#/$defs/build" + } + }, + "build": { + "$ref": "#/$defs/build", + "description": "Single build configuration (legacy). Use 'builds' array for multi-modal." + }, + "lineage": { + "$ref": "#/$defs/lineage", + "description": "Optional lineage tracking for better data governance." + }, + "schemaEvolution": { + "$ref": "#/$defs/schemaEvolution", + "description": "Optional schema evolution strategy for managing changes." + }, + "machineLearning": { + "$ref": "#/$defs/machineLearning", + "description": "Optional ML model specifications for MLPipeline kind." + }, + "environments": { + "type": "object", + "description": "Optional environment-specific overrides (dev, staging, prod).", + "additionalProperties": { + "$ref": "#/$defs/environmentConfig" + } + }, + "lifecycle": { + "$ref": "#/$defs/lifecycle" + }, + "docs": { + "$ref": "#/$defs/docs" + }, + "sovereignty": { + "$ref": "#/$defs/sovereignty", + "description": "NEW in v0.7.1: Jurisdiction and data residency requirements for compliance enforcement." + }, + "accessPolicy": { + "$ref": "#/$defs/accessPolicy", + "description": "Root-level access policy for automated IAM binding generation. Defines principals, permissions, and resources for data product access control." + }, + "orchestration": { + "$ref": "#/$defs/orchestration", + "description": "Top-level orchestration configuration for scheduling and workflow engines." + }, + "retention": { + "$ref": "#/$defs/retention", + "description": "NEW in v0.7.3: Retention durations for run state, run logs, lineage events, and DLQ records. Honored by the retention sweeper invoked from policy-apply." + }, + "consumers": { + "type": "array", + "description": "NEW in v0.7.6: Declared downstream consumers of this data product — the business artifacts built on it (dashboards, notebooks, analyses, ML apps). Shape follows dbt exposures so existing dbt exposure definitions map field-for-field. Purely declarative and additive: consumers never affect plan/apply. RESERVED, not yet consumed — no code reads this key today; it is carried verbatim into plan.json and the contract COMMENT so it survives round-trip, and is intended to feed lineage terminal nodes and impact statements ('Affects: Revenue dashboard') in a later release.", + "items": { + "$ref": "#/$defs/consumerRef" + } + }, + "governance": { + "$ref": "#/$defs/governance", + "description": "Contract-level governance: account/project-wide settings (e.g. AWS Lake Formation admins and LF-tag definitions). Per-resource grants and tag associations live under binding.governance." + }, + "packaging": { + "$ref": "#/$defs/packaging", + "description": "NEW in v0.7.6: Contract-wide packaging default — declarative container ownership (isolated = this product creates and owns its infrastructure containers; shared = containers are pre-existing, platform-owned pools the product writes into but does not own). Overridable per exposure via binding.packaging. An ABSENT block means legacy behavior: the IaC emit is byte-identical to pre-packaging releases." + } + }, + "$defs": { + "packaging": { + "type": "object", + "additionalProperties": false, + "description": "NEW in v0.7.6: Declarative packaging mode — per-container ownership for the OpenTofu IaC emit (RFC-packaging-modes.md). 'isolated' emits owned resources (today's exact emit); 'shared' emits a data-source reference to a platform-owned pool plus leaf-only owned resources (tables, prefixed objects, scoped grants). Hybrid tiers (e.g. Snowflake shared-database / owned-schema / owned-warehouse) come from the per-kind containers override map. Prior art: Terraform's resource-vs-data-source ownership split, Unity Catalog isolation-mode OPEN|ISOLATED, Kubernetes shared-vs-dedicated node pools.", + "properties": { + "mode": { + "type": "string", + "enum": [ + "isolated", + "shared" + ], + "description": "NEW in v0.7.6: Blanket ownership mode for every container kind. Default when the block is present: 'isolated'. Per-kind exceptions go in `containers`." + }, + "pool": { + "type": "string", + "minLength": 1, + "description": "NEW in v0.7.6: Pool id of the platform-owned tenant pool this product writes into. REQUIRED (enforced by the resolver) whenever any container resolves 'shared' — a pool must be addressable. Propagated as the `fluid_pool` label/tag for cost attribution." + }, + "poolManifest": { + "type": "string", + "minLength": 1, + "description": "NEW in v0.7.6: Optional path to the platform team's pool manifest file; snapshotted into the bundle at stage 1 so bundleDigest covers it (any pool-file edit invalidates downstream plans)." + }, + "containers": { + "type": "object", + "additionalProperties": false, + "description": "NEW in v0.7.6: Per-container-kind ownership overrides — each key wins over `mode` for that kind, yielding hybrid tiers. Kind↔platform mapping: bucket → aws_s3_bucket / google_storage_bucket; database → snowflake_database AND aws_glue_catalog_database; dataset → google_bigquery_dataset; schema → snowflake_schema; warehouse → snowflake_warehouse; cluster → Confluent environment/cluster (v1: shared only).", + "properties": { + "bucket": { + "$ref": "#/$defs/packagingContainerMode", + "description": "NEW in v0.7.6: Ownership of the object-store bucket (aws_s3_bucket / google_storage_bucket)." + }, + "database": { + "$ref": "#/$defs/packagingContainerMode", + "description": "NEW in v0.7.6: Ownership of the database container — covers Snowflake `database` AND AWS `glue_database`." + }, + "dataset": { + "$ref": "#/$defs/packagingContainerMode", + "description": "NEW in v0.7.6: Ownership of the BigQuery dataset (google_bigquery_dataset)." + }, + "schema": { + "$ref": "#/$defs/packagingContainerMode", + "description": "NEW in v0.7.6: Ownership of the Snowflake schema (snowflake_schema)." + }, + "warehouse": { + "$ref": "#/$defs/packagingContainerMode", + "description": "NEW in v0.7.6: Ownership of the Snowflake warehouse (snowflake_warehouse) — isolated gives per-product cost attribution." + }, + "cluster": { + "$ref": "#/$defs/packagingContainerMode", + "description": "NEW in v0.7.6: Ownership of the Confluent environment/cluster. v1 accepts 'shared' only — dedicated-cluster provisioning is not yet supported (the resolver rejects 'isolated')." + } + } + } + } + }, + "packagingContainerMode": { + "type": "string", + "enum": [ + "isolated", + "shared" + ], + "description": "NEW in v0.7.6: Per-container ownership — 'isolated' (this product creates and owns it) or 'shared' (pre-existing platform-owned pool, referenced not created)." + }, + "retention": { + "type": "object", + "description": "NEW in v0.7.3: Retention durations (ISO-8601). Defaults: runState=P30D, runLogs=P90D, lineage=P365D, dlq=P180D.", + "additionalProperties": false, + "properties": { + "runState": { + "$ref": "#/$defs/isoDuration" + }, + "runLogs": { + "$ref": "#/$defs/isoDuration" + }, + "lineage": { + "$ref": "#/$defs/isoDuration" + }, + "dlq": { + "$ref": "#/$defs/isoDuration" + } + } + }, + "identifier": { + "type": "string", + "pattern": "^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$", + "description": "Unified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle. Both casing conventions accepted (snake_case / dotted lowercase / PascalCase / UPPER) so existing catalog ids round-trip unchanged." + }, + "semver": { + "type": "string", + "pattern": "^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(?:-[0-9A-Za-z.-]+)?(?:\\+[0-9A-Za-z.-]+)?$", + "description": "Semantic version (semver.org)." + }, + "semverRange": { + "type": "string", + "description": "Semver range (e.g., ^1.2, ~1.4, >=1.0.0). Not strictly validated by regex; tooling enforces." + }, + "isoDuration": { + "type": "string", + "pattern": "^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?(T(\\d+H)?(\\d+M)?(\\d+S)?)?$", + "description": "ISO-8601 duration (e.g., P1D, PT15M, P2Y6M)." + }, + "availabilityPct": { + "type": "string", + "pattern": "^(100(\\.0+)?|\\d{2}(\\.\\d+)?|\\d{1}\\d(\\.\\d+)?)%$", + "description": "Availability percentage (e.g., 99.9%)." + }, + "lifecycleState": { + "type": "string", + "enum": [ + "preview", + "active", + "deprecated", + "retired" + ], + "description": "Unified lifecycle state vocabulary." + }, + "sensitivityLevel": { + "type": "string", + "enum": [ + "none", + "internal", + "confidential", + "restricted", + "pii", + "phi", + "cleartext", + "treated", + "anonymized", + "pseudonymized", + "tokenized", + "encrypted" + ], + "description": "Unified data sensitivity classification." + }, + "tags": { + "type": "array", + "description": "Simple string tags for categorization, discovery, and automation. Consistent pattern used throughout FLUID objects.", + "items": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$", + "description": "Tag format: lowercase alphanumeric with hyphens, e.g., 'customer-data', 'pii', 'real-time'" + }, + "uniqueItems": true + }, + "labels": { + "type": "object", + "description": "Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.", + "additionalProperties": { + "type": "string", + "description": "Label values are always strings for consistency and tooling compatibility." + }, + "examples": [ + { + "team": "customer-analytics", + "criticality": "high", + "cost-center": "engineering", + "retention": "7-years", + "region": "us-central1", + "business-name": "Customer Profiles", + "data-category": "analytical" + } + ] + }, + "expose": { + "type": "object", + "additionalProperties": false, + "required": [ + "exposeId", + "kind", + "binding", + "contract" + ], + "examples": [ + { + "exposeId": "customer_profiles", + "title": "Customer Profiles Table", + "kind": "table", + "tags": [ + "pii", + "customer-facing", + "analytical" + ], + "labels": { + "sensitivity": "high", + "retention": "7-years", + "team": "customer-analytics" + }, + "contract": { + "schema": [ + { + "name": "customer_id", + "type": "STRING", + "required": true, + "sensitivity": "cleartext", + "tags": [ + "identifier", + "primary-key" + ], + "labels": { + "business-name": "Customer ID", + "data-category": "identifier" + } + } + ] + }, + "binding": { + "platform": "gcp", + "format": "bigquery_table", + "location": { + "project": "company-data", + "dataset": "customer", + "table": "profiles" + } + } + } + ], + "properties": { + "exposeId": { + "$ref": "#/$defs/identifier", + "description": "Stable interface handle (used by consumers & build.outputs)." + }, + "title": { + "type": "string" + }, + "description": { + "type": "string", + "description": "Detailed description of the exposed resource." + }, + "version": { + "$ref": "#/$defs/semver" + }, + "kind": { + "type": "string", + "enum": [ + "table", + "view", + "api", + "file", + "stream", + "topic", + "feature_store", + "model", + "vector", + "graph", + "time_series", + "other" + ] + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Expose-level tags for categorization and routing." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Expose-level labels for metadata and automation." + }, + "contract": { + "$ref": "#/$defs/exposeContract" + }, + "qos": { + "$ref": "#/$defs/exposeQoS" + }, + "policy": { + "$ref": "#/$defs/exposePolicy" + }, + "binding": { + "$ref": "#/$defs/binding" + }, + "lifecycle": { + "$ref": "#/$defs/exposeLifecycle" + }, + "observability": { + "$ref": "#/$defs/observability" + }, + "docs": { + "$ref": "#/$defs/docs" + }, + "semantics": { + "$ref": "#/$defs/semanticModel", + "description": "NEW in v0.7.2: Semantic model definition mapping physical columns to business concepts. Defines entities, measures, dimensions, and metrics for agent-consumable business logic. Eliminates semantic hallucination by providing machine-readable KPI definitions. Compatible with dbt MetricFlow, Snowflake Semantic Views, and OSI-aligned formats." + }, + "crawler": { + "type": "object", + "description": "AWS Glue crawler configuration for auto-discovery of schema from S3 paths.", + "properties": { + "name": { + "type": "string", + "description": "Glue crawler name." + }, + "role": { + "type": "string", + "description": "IAM role ARN for the crawler." + }, + "schedule": { + "type": "string", + "description": "Cron schedule expression (e.g., 'cron(0 6 * * ? *)')." + }, + "classifiers": { + "type": "array", + "description": "Custom classifier names to apply.", + "items": { + "type": "string" + } + }, + "schemaChangePolicy": { + "type": "object", + "description": "Policy for schema changes detected by the crawler.", + "properties": { + "updateBehavior": { + "type": "string", + "enum": [ + "UPDATE_IN_DATABASE", + "LOG" + ] + }, + "deleteBehavior": { + "type": "string", + "enum": [ + "LOG", + "DELETE_FROM_DATABASE", + "DEPRECATE_IN_DATABASE" + ] + } + } + } + } + }, + "iceberg": { + "type": "object", + "description": "AWS Glue Iceberg table management configuration (snapshots, compaction).", + "properties": { + "writeFormat": { + "type": "string", + "enum": [ + "parquet", + "orc", + "avro" + ], + "description": "Underlying file format for Iceberg data files." + }, + "snapshotRetention": { + "type": "object", + "description": "Snapshot retention policy.", + "properties": { + "maxSnapshotAgeMs": { + "type": "integer", + "description": "Maximum age of snapshots in milliseconds." + }, + "minSnapshotsToKeep": { + "type": "integer", + "description": "Minimum number of snapshots to retain." + } + } + }, + "compaction": { + "type": "object", + "description": "Iceberg compaction settings.", + "properties": { + "enabled": { + "type": "boolean" + }, + "targetFileSizeMb": { + "type": "integer", + "description": "Target file size in MB after compaction." + } + } + } + } + }, + "mcp": { + "$ref": "#/$defs/exposeMcp", + "description": "NEW in v0.7.4: Declares this expose as agent-consumable over MCP, with per-expose overrides for the consumer-side output-port server (`fluid mcp output-port serve`). Its presence opts the expose into the Fluid MCP gateway, where `policy.agentPolicy.allowedModels` / `allowedUseCases` are enforced at every read." + } + } + }, + "exposeContract": { + "type": "object", + "additionalProperties": false, + "properties": { + "schemaSignature": { + "type": "string", + "pattern": "^sha256:[0-9a-fA-F]{64}$", + "description": "Hash of canonical schema/OpenAPI. Used by contract-tests gates." + }, + "schemaPolicy": { + "type": "string", + "enum": [ + "strict", + "discover_and_freeze", + "evolve_safe", + "evolve_all" + ], + "default": "strict", + "description": "NEW in v0.7.3: Schema evolution policy. strict = fail on any schema change; discover_and_freeze = capture from source on first run, fail thereafter; evolve_safe = accept additive changes; evolve_all = accept all changes (cast on type narrow, route failures to DLQ)." + }, + "schema": { + "type": "array", + "items": { + "$ref": "#/$defs/column" + }, + "description": "Tabular schema (for table/view/file)." + }, + "openapiRef": { + "type": "string", + "format": "uri", + "description": "OpenAPI document URL (for APIs)." + }, + "guarantees": { + "type": "object", + "additionalProperties": false, + "properties": { + "compatibility": { + "type": "string", + "enum": [ + "none", + "patch", + "minor", + "backward", + "forward", + "strict" + ], + "description": "Compatibility promise for non-breaking evolution." + }, + "evolution": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "additive_columns", + "additive_fields", + "additive_endpoints", + "nullable_to_required_disallowed", + "field_removal_with_notice", + "type_widening_allowed" + ] + } + } + } + }, + "dq": { + "$ref": "#/$defs/dqSpec", + "description": "Enhanced data quality rules including anomaly detection." + }, + "quality": { + "type": "array", + "description": "Simplified SQL-expression data quality rules (alternate form of dq.rules for inline assertions).", + "items": { + "type": "object", + "required": [ + "rule", + "expression", + "severity" + ], + "properties": { + "rule": { + "type": "string", + "description": "Rule identifier." + }, + "expression": { + "type": "string", + "description": "SQL or boolean expression that must evaluate to true." + }, + "severity": { + "type": "string", + "enum": [ + "error", + "warning", + "info" + ], + "description": "Severity when the rule fails." + } + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Contract-level tags for data quality and governance." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Contract-level labels for automation and metadata." + } + }, + "anyOf": [ + { + "required": [ + "schema" + ] + }, + { + "required": [ + "openapiRef" + ] + } + ] + }, + "column": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "type" + ], + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "type": { + "description": "Column data type. Accepts a canonical FLUID/SQL type name (case-insensitive) or a parameterized form such as decimal(18,4) or varchar(255). The enum lists the canonical bare types; the pattern additionally permits case variants and parameter suffixes. Intentionally generous — it rejects obvious garbage (e.g. MYSTERY_TYPE) while admitting the cross-dialect superset.", + "anyOf": [ + { + "type": "string", + "enum": [ + "string", + "text", + "varchar", + "varchar2", + "nvarchar", + "char", + "nchar", + "character", + "clob", + "int", + "integer", + "int2", + "int4", + "int8", + "int16", + "int32", + "int64", + "tinyint", + "smallint", + "mediumint", + "bigint", + "long", + "longint", + "serial", + "bigserial", + "float", + "float4", + "float8", + "float32", + "float64", + "double", + "real", + "decimal", + "dec", + "numeric", + "number", + "bignumeric", + "money", + "boolean", + "bool", + "bit", + "date", + "time", + "datetime", + "datetime2", + "smalldatetime", + "timestamp", + "timestamptz", + "timestamp_tz", + "timestamp_ntz", + "timestamp_ltz", + "timestampntz", + "interval", + "year", + "variant", + "object", + "array", + "struct", + "map", + "record", + "row", + "json", + "jsonb", + "super", + "binary", + "varbinary", + "bytes", + "blob", + "bytea", + "raw", + "geography", + "geometry", + "geom", + "point", + "uuid", + "uniqueidentifier", + "guid", + "enum", + "hll" + ] + }, + { + "type": "string", + "pattern": "(?i)^\\s*(string|text|varchar|varchar2|nvarchar|char|nchar|character|clob|int|integer|int2|int4|int8|int16|int32|int64|tinyint|smallint|mediumint|bigint|long|longint|serial|bigserial|float|float4|float8|float32|float64|double|real|double\\s+precision|decimal|dec|numeric|number|bignumeric|money|boolean|bool|bit|date|time|datetime|datetime2|smalldatetime|timestamp|timestamptz|timestamp_tz|timestamp_ntz|timestamp_ltz|timestampntz|timestamp\\s+with(out)?\\s+time\\s+zone|interval|year|variant|object|array|struct|map|record|row|json|jsonb|super|binary|varbinary|bytes|blob|bytea|raw|geography|geometry|geom|point|uuid|uniqueidentifier|guid|enum|hll)\\s*(\\(\\s*[0-9A-Za-z_,\\s'\\\"-]*\\s*\\))?\\s*$" + } + ] + }, + "required": { + "type": "boolean" + }, + "description": { + "type": "string" + }, + "sensitivity": { + "$ref": "#/$defs/sensitivityLevel", + "description": "Field-level sensitivity classification for improved policy ergonomics." + }, + "semanticType": { + "type": "string", + "description": "Semantic meaning (e.g., 'email', 'phone', 'identifier')." + }, + "businessName": { + "type": "string", + "description": "Business-friendly name for this field." + }, + "businessDefinition": { + "type": "string", + "description": "Business definition and meaning of this field." + }, + "validationRules": { + "type": "array", + "description": "Field-level validation constraints.", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "type": { + "type": "string", + "enum": [ + "range", + "regex", + "enum", + "custom" + ] + }, + "constraint": { + "type": "string" + }, + "message": { + "type": "string" + } + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Field-level tags for categorization and data governance." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Field-level labels for metadata and automation." + } + } + }, + "dqSpec": { + "type": "object", + "additionalProperties": false, + "properties": { + "rules": { + "type": "array", + "items": { + "$ref": "#/$defs/dqRule" + } + }, + "monitoring": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "window": { + "$ref": "#/$defs/isoDuration" + }, + "owner": { + "type": "string" + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Data quality specification tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Data quality specification labels." + } + } + }, + "dqRule": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "severity" + ], + "properties": { + "id": { + "type": "string" + }, + "type": { + "type": "string", + "enum": [ + "freshness", + "completeness", + "uniqueness", + "valid_values", + "accuracy", + "schema", + "anomaly_detection", + "drift_detection" + ] + }, + "selector": { + "type": "string" + }, + "threshold": { + "type": "number" + }, + "operator": { + "type": "string", + "enum": [ + ">=", + ">", + "<=", + "<", + "==", + "!=" + ] + }, + "window": { + "$ref": "#/$defs/isoDuration" + }, + "severity": { + "type": "string", + "enum": [ + "info", + "warn", + "error", + "critical" + ] + }, + "description": { + "type": "string" + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "DQ rule tags for categorization and automation." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "DQ rule labels for metadata and routing." + } + } + }, + "exposeQoS": { + "type": "object", + "additionalProperties": false, + "properties": { + "availability": { + "$ref": "#/$defs/availabilityPct" + }, + "freshnessSLO": { + "$ref": "#/$defs/isoDuration" + }, + "dataLossSLO": { + "type": "string", + "description": "e.g., '0 rows'" + }, + "latencyP95": { + "$ref": "#/$defs/isoDuration" + }, + "completenessTarget": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Target completeness ratio (0.0 to 1.0)." + }, + "errorBudget": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Acceptable error rate (0.0 to 1.0)." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "QoS-level tags for SLA management." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "QoS-level labels for automation and reporting." + } + } + }, + "exposePolicy": { + "type": "object", + "additionalProperties": false, + "properties": { + "authn": { + "type": "string", + "enum": [ + "oidc", + "oauth2", + "api_key", + "none", + "custom", + "iam", + "jwt" + ] + }, + "authz": { + "type": "object", + "additionalProperties": false, + "properties": { + "readers": { + "type": "array", + "items": { + "type": "string" + } + }, + "writers": { + "type": "array", + "items": { + "type": "string" + } + }, + "columnRestrictions": { + "type": "array", + "description": "Column-level access control. A column named in any restriction is restricted. 'deny': the principal may not read the columns. 'allow': the columns are readable only by the principals an allow names. A deny beats an allow, and a restriction never grants access: the readers are the expose's readers. Principals are logical and resolve through binding.principals. NEW in v0.7.6 (the field itself is older): `fluid apply` enforces it. GCP BigQuery: a Data Catalog taxonomy per product and dataset with fine-grained access control, a policy tag per set of restricted columns that share their readers, attached through the table schema's policyTags, and roles/datacatalog.categoryFineGrainedReader for exactly the expose's readers allowed to read them: the accessPolicy read grantees and the expose's own policy.authz.readers (a denied principal gets an access error on the columns). A restriction on an expose with no reader on the binding is refused, as it would lock the columns for everyone. A restriction's tags and labels are written into the policy tag's description; AWS has no place for them. AWS: each governance.lakeFormation grant with SELECT excludes the restricted columns its principal may not read (table_with_columns.excluded_column_names); a grant's hand-written excludedColumns must agree, and a binding with no Lake Formation grants is refused. `fluid verify` checks the policy tags and their readers on GCP, and the principals' Lake Formation permissions on AWS.", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "principal": { + "type": "string", + "description": "Logical principal the restriction applies to, written as in accessPolicy (e.g. group:analysts@company.example); binding.principals maps it per cloud." + }, + "columns": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Columns of this expose's schema." + }, + "access": { + "type": "string", + "enum": [ + "allow", + "deny" + ], + "description": "deny: the principal may not read the columns. allow: only the principals an allow names may read them." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Access control rule tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Access control rule labels." + } + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Authorization policy tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Authorization policy labels." + } + } + }, + "privacy": { + "type": "object", + "additionalProperties": false, + "properties": { + "masking": { + "type": "array", + "description": "Columns to treat before they land. The DuckDB acquisition runner rewrites each named column inside the write, so the file, S3 object or BigQuery load and the DLQ receive the treated value; quality gates judge the source value first. A treated column lands as a string, so declare it with a string type (string, or VARCHAR under a schemaPolicy that compares source type names). A rule the build cannot apply (an unset salt or key, k_anonymity, a column the stream lacks, a non-string declared type, the incremental cursor) refuses the build rather than landing the column untreated. fluid verify checks every non-null value of a masked column for its strategy's shape, on a local file and, through Athena, on an S3+Glue table. The MCP output-port gateway drops masked columns from what it serves.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "column", + "strategy" + ], + "properties": { + "column": { + "type": "string" + }, + "strategy": { + "type": "string", + "description": "hash: lowercase hex SHA-256 of salt || value, deterministic so joins work (64 hex characters). mask: every character but the first params.keepFirst and the last params.keepLast replaced by '*', length preserved. tokenize: HMAC-SHA256 token, 32 hex characters, the token the tokenize_pii preLand hook makes. encrypt: AES-GCM, written as 'aesgcm:v1:' + base64url(nonce || ciphertext || tag) with the column name as associated data; reversible with the key, not deterministic. k_anonymity: accepted by the schema but refused at landing: it is a property of a whole table over its quasi-identifiers, not of one value.", + "enum": [ + "mask", + "hash", + "tokenize", + "encrypt", + "k_anonymity" + ] + }, + "params": { + "type": "object", + "additionalProperties": true, + "description": "Strategy parameters. The runner accepts exactly these keys and refuses any other, including a literal salt or key: secrets come only from the environment, never the contract, which is published with the plan, the bundle and the Glue table's fluid_contract parameter. Contracts on 0.7.5 and earlier may use the same keys; their schemas leave params untyped, and the runner checks them at build time.", + "properties": { + "saltEnv": { + "type": "string", + "pattern": "^[A-Z_][A-Z0-9_]{0,127}$", + "default": "FLUID_PII_HASH_SECRET", + "description": "NEW in v0.7.6: hash only. The environment variable holding the salt, at least 16 bytes. Unset or empty, the build is refused: an unsalted SHA-256 of a phone number is reversed by enumerating the numbering plan." + }, + "keyEnv": { + "type": "string", + "pattern": "^[A-Z_][A-Z0-9_]{0,127}$", + "description": "NEW in v0.7.6: tokenize and encrypt. The environment variable holding the key. tokenize defaults to FLUID_PII_TOKENIZATION_KEY and needs at least 32 bytes; encrypt defaults to FLUID_PII_ENCRYPTION_SECRET_KEY and needs the base64 of a 16, 24 or 32-byte AES key (openssl rand -base64 32). Unset or empty, the build is refused." + }, + "keepFirst": { + "type": "integer", + "minimum": 0, + "maximum": 64, + "default": 0, + "description": "NEW in v0.7.6: mask only. Leading characters left as they are." + }, + "keepLast": { + "type": "integer", + "minimum": 0, + "maximum": 64, + "default": 4, + "description": "NEW in v0.7.6: mask only. Trailing characters left as they are, so +46701234567 lands as ********4567. A value no longer than keepFirst + keepLast is masked entirely." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Privacy masking rule tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Privacy masking rule labels." + } + } + } + }, + "rowLevelPolicy": { + "type": "object", + "additionalProperties": false, + "required": [ + "expression" + ], + "properties": { + "expression": { + "type": "string", + "description": "Provider-specific predicate" + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Row-level policy tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Row-level policy labels." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Privacy policy tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Privacy policy labels." + } + } + }, + "classification": { + "type": "string", + "enum": [ + "Public", + "Internal", + "Confidential", + "Restricted" + ] + }, + "agentPolicy": { + "$ref": "#/$defs/agentPolicy", + "description": "NEW in v0.7.1: AI/LLM usage governance - controls which AI models can consume this data and for what purposes." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Policy-level tags for governance." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Policy-level labels for automation." + } + } + }, + "binding": { + "type": "object", + "additionalProperties": false, + "required": [ + "platform", + "format", + "location" + ], + "properties": { + "platform": { + "type": "string", + "enum": [ + "gcp", + "aws", + "azure", + "snowflake", + "databricks", + "kafka", + "confluent", + "local", + "kubernetes", + "postgres", + "pgvector", + "other" + ], + "description": "Target platform. NEW in v0.7.5: 'pgvector' for the vector/embeddings output port (pgvector-on-PostgreSQL RAG target — emits the embeddings table + ANN index DDL for the ai-embeddable columns). NEW in v0.7.5: 'confluent' for the Confluent Cloud Tableflow managed Kafka→Iceberg emitter. NEW in v0.7.4: 'postgres' for the MCP output-port PostgreSQL driver (AWS Athena uses platform 'aws')." + }, + "format": { + "type": "string", + "enum": [ + "bigquery_table", + "snowflake_table", + "snowflake_view", + "gcs_file", + "s3_file", + "http_api", + "grpc_api", + "pubsub_topic", + "kafka_topic", + "delta_table", + "iceberg", + "parquet", + "csv", + "json", + "redshift_table", + "redshift_serverless", + "redshift_external_schema", + "postgres_table", + "athena_table", + "glue_table", + "pgvector_table", + "other" + ], + "description": "Binding format. NEW in v0.7.5: 'pgvector_table' — a pgvector embeddings table target for the vector output port. NEW in v0.7.4: 'postgres_table' (PostgreSQL driver), 'athena_table' / 'glue_table' (AWS Athena driver) for the MCP output port." + }, + "icebergConfig": { + "type": "object", + "description": "Apache Iceberg table format configuration", + "additionalProperties": false, + "properties": { + "writeVersion": { + "type": "integer", + "enum": [ + 1, + 2 + ], + "description": "Iceberg table format version" + }, + "fileFormat": { + "type": "string", + "enum": [ + "parquet", + "orc", + "avro" + ], + "description": "Underlying file format for data files" + }, + "partitionSpec": { + "type": "array", + "description": "Partition transformation specifications", + "items": { + "type": "object", + "required": [ + "sourceColumn", + "transform" + ], + "properties": { + "name": { + "type": "string", + "description": "Partition field name" + }, + "sourceColumn": { + "type": "string", + "description": "Source column for partitioning" + }, + "transform": { + "type": "string", + "enum": [ + "identity", + "year", + "month", + "day", + "hour", + "bucket", + "truncate" + ], + "description": "Partition transform function" + }, + "numBuckets": { + "type": "integer", + "description": "Number of buckets (for bucket transform)" + }, + "width": { + "type": "integer", + "description": "Truncate width (for truncate transform)" + } + } + } + }, + "sortOrder": { + "type": "array", + "description": "Sort order specifications for data files", + "items": { + "type": "object", + "required": [ + "column" + ], + "properties": { + "column": { + "type": "string", + "description": "Column to sort by" + }, + "direction": { + "type": "string", + "enum": [ + "asc", + "desc" + ], + "description": "Sort direction" + }, + "nullOrder": { + "type": "string", + "enum": [ + "nulls-first", + "nulls-last" + ], + "description": "NULL value ordering" + } + } + } + }, + "properties": { + "type": "object", + "description": "Iceberg table properties", + "additionalProperties": { + "type": "string" + } + } + } + }, + "vectorConfig": { + "type": "object", + "additionalProperties": false, + "description": "NEW in v0.7.5: vector / embeddings output-port configuration (platform 'pgvector', format 'pgvector_table'). Consumed by fluid_build.output_ports.vector to emit the pgvector embeddings-table + ANN-index DDL and a RAG provenance manifest for the schema's ai-embeddable columns.", + "required": [ + "dimensions" + ], + "properties": { + "dimensions": { + "type": "integer", + "minimum": 1, + "maximum": 16000, + "description": "Embedding vector width (e.g. 1536 for OpenAI text-embedding-3-small, 1024 for Cohere embed-english-light). Maps to the pgvector column type vector()." + }, + "embeddingModel": { + "type": "string", + "description": "Provenance: the embedding model that produces the vectors (e.g. 'text-embedding-3-small'). Recorded in the RAG manifest and the embeddings-table's embedding_model column so a retriever can verify model parity." + }, + "vectorType": { + "type": "string", + "enum": [ + "vector", + "halfvec" + ], + "default": "vector", + "description": "pgvector column type: 'vector' (float32, up to 2000 dims) or 'halfvec' (float16, half the storage, up to 4000 dims)." + }, + "indexType": { + "type": "string", + "enum": [ + "hnsw", + "ivfflat", + "none" + ], + "default": "hnsw", + "description": "ANN index built on the embedding column. 'hnsw' (higher recall / lower query latency, buildable before data lands), 'ivfflat' (lower build cost, needs data first), or 'none' (no index — exact search only)." + }, + "distanceMetric": { + "type": "string", + "enum": [ + "cosine", + "l2", + "inner_product", + "l1" + ], + "default": "cosine", + "description": "Distance function the index optimises for → pgvector operator class: cosine→vector_cosine_ops, l2→vector_l2_ops, inner_product→vector_ip_ops, l1→vector_l1_ops." + }, + "table": { + "type": "string", + "description": "Target embeddings table name. Defaults to '_embeddings' when omitted." + }, + "sourceKeyColumn": { + "type": "string", + "description": "Provenance: the source primary-key column that ties each embedding row back to its origin row (written to the embeddings-table's source_key column). Enables update/delete propagation from the source product to the vector store." + }, + "hnsw": { + "type": "object", + "additionalProperties": false, + "description": "HNSW index tuning (used when indexType is 'hnsw').", + "properties": { + "m": { + "type": "integer", + "minimum": 1, + "description": "Max bi-directional links per node (pgvector default 16)." + }, + "efConstruction": { + "type": "integer", + "minimum": 1, + "description": "Build-time dynamic candidate-list size (pgvector default 64)." + } + } + }, + "ivfflat": { + "type": "object", + "additionalProperties": false, + "description": "IVFFlat index tuning (used when indexType is 'ivfflat').", + "properties": { + "lists": { + "type": "integer", + "minimum": 1, + "description": "Number of inverted lists (pgvector guidance: rows/1000 for < 1M rows)." + } + } + } + } + }, + "location": { + "$ref": "#/$defs/bindingLocation" + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Binding configuration tags for infrastructure." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Binding configuration labels for automation." + }, + "properties": { + "type": "object", + "description": "Provider-specific binding properties (e.g., cluster_by, comment for Snowflake).", + "additionalProperties": true + }, + "governance": { + "$ref": "#/$defs/bindingGovernance", + "description": "Per-resource governance: registerLocation, principal grants, tag associations, row/column filters. Account-wide governance (admins, tag definitions) lives at the contract top-level governance block." + }, + "packaging": { + "$ref": "#/$defs/packaging", + "description": "NEW in v0.7.6: Per-exposure packaging override — key-wise precedence over the contract-wide top-level `packaging` block (binding.packaging > packaging > absent-legacy)." + }, + "encryption": { + "type": "object", + "additionalProperties": false, + "description": "NEW in v0.7.6: Encryption at rest for an AWS S3 binding or a GCP BigQuery binding. Absent: nothing is emitted, and the cloud's default applies: SSE-S3 for new S3 objects, Google-managed keys for BigQuery.", + "properties": { + "kms": { + "type": "string", + "default": "product", + "anyOf": [ + { + "enum": [ + "product", + "none" + ] + }, + { + "pattern": "^alias/[A-Za-z0-9/_-]{1,250}$" + }, + { + "pattern": "^arn:aws[a-z-]*:kms:[a-z0-9-]+:[0-9]{12}:(key|alias)/[A-Za-z0-9/_-]{1,250}$" + }, + { + "pattern": "^projects/[a-z0-9.:-]{1,100}/locations/[a-z0-9-]{1,63}/keyRings/[A-Za-z0-9_-]{1,63}/cryptoKeys/[A-Za-z0-9_-]{1,63}$" + } + ], + "description": "NEW in v0.7.6: The KMS key the bucket's objects are encrypted with (SSE-KMS with an S3 Bucket Key). 'product' (the default): `fluid apply` creates a customer managed key for each bucket this product owns (aws_kms_key, rotation on, alias alias/fluid//) and makes it the bucket's default encryption (aws_s3_bucket_server_side_encryption_configuration). Its key policy lets the account's IAM policies decide who may use it (the KMS default statement for the account root) and, when the binding carries governance.lakeFormation, lets the Lake Formation service-linked role AWSServiceRoleForLakeFormationDataAccess encrypt and decrypt, matched by aws:PrincipalArn so the role need not exist yet. A principal querying through Athena with credentials Lake Formation vends then needs no KMS permission of its own; a grantee in another account that the bucket policy lets read the objects directly (governance.lakeFormation.bucketPolicy) is also allowed kms:Decrypt, through S3 only. The writer (the identity the build runs as) needs kms:GenerateDataKey and kms:Decrypt from IAM. `tofu destroy` deletes the alias and schedules the key for deletion after 7 days, the minimum KMS allows; kms:CancelKeyDeletion recovers it until then. Dropping the block, or naming another key, removes the product key while the bucket keeps the objects encrypted under it, which become unreadable once the deletion window ends unless they are rewritten or the deletion is cancelled; `fluid apply` refuses that plan without --allow-data-loss. On a shared (pool) bucket the bucket's default encryption is the pool owner's, so 'product' is refused there. 'alias/' or a key or alias ARN: an existing key, looked up at plan time (data aws_kms_key, so the identity running `fluid apply` needs kms:DescribeKey on it) and made the default encryption of a bucket this product owns by its key ARN, since S3 resolves an alias in the account of whoever writes; the plan fails unless the key is Enabled and a symmetric encryption key (SYMMETRIC_DEFAULT). On a pool bucket it is only checked. Its key policy is its owner's, and must let the Lake Formation role use it when the location is registered: an AWS managed key cannot be used with that role, so alias/aws/s3 is refused with registerLocation, and the plan fails for any key that is not customer managed. 'none': nothing is emitted. `fluid verify` checks that the key is Enabled and that the objects under the binding's prefix are SSE-KMS with it. On a gcp binding (fluid-schema 0.7.6, BigQuery tables): 'product' creates a Cloud KMS key ring and crypto key for each dataset this product writes (google_kms_key_ring fluid-- and google_kms_crypto_key 'bigquery', in the dataset's location, EU and US mapping to the europe and us key locations, rotated every 90 days), grants the project's BigQuery service agent roles/cloudkms.cryptoKeyEncrypterDecrypter on it, and makes it the dataset's default_encryption_configuration and the table's encryption_configuration. The project must have the Cloud KMS API (cloudkms.googleapis.com) enabled. A key ring and key cannot be deleted on GCP: `tofu destroy` removes them from state and schedules the key's versions for destruction (30 days by default), and a later apply adopts the same names. 'projects/

/locations//keyRings//cryptoKeys/': an existing key in the dataset's location, used as given; its owner grants the BigQuery service agent. 'none': Google-managed keys. An alias/... or arn:... value is refused on a gcp binding, and a Cloud KMS name on an aws binding. Adding a key to a table that exists replaces the table (BigQuery cannot re-key it in place), so `fluid apply` refuses that plan without --allow-data-loss; the next build lands the data again. `fluid verify` checks kmsKeyName on the dataset and the table." + } + } + }, + "principals": { + "type": "object", + "description": "NEW in v0.7.6: Maps the contract's LOGICAL principals (accessPolicy.grants[].principal and exposes[].policy.authz.columnRestrictions[].principal, keyed exactly as the contract writes them) to the identities they stand for on this binding's platform, so the base contract stays cloud-neutral and each environment's overlay binds it. On gcp an identity is an IAM member (user:, group:, serviceAccount: or domain:); on aws an IAM principal ARN. A value is one identity or a list; [] says the principal has no identity on this cloud, and nothing is granted to it there. When the block is present, every principal the contract names for this expose must be a key: an unmapped one is refused, never emitted as written. Without the block the principals are used as written, except that a GCP principal in a reserved top-level domain (.example, .test, .invalid, .localhost) is refused as a placeholder. Modelled on ODCS v3's roles[] bound per server (servers[].roles) and on dbt grants resolved per target.", + "propertyNames": { + "minLength": 1 + }, + "additionalProperties": { + "anyOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + }, + "uniqueItems": true + } + ] + } + } + }, + "allOf": [ + { + "if": { + "properties": { + "platform": { + "const": "gcp" + } + }, + "required": [ + "platform" + ] + }, + "then": { + "properties": { + "encryption": { + "properties": { + "kms": { + "not": { + "pattern": "^(alias/|arn:)" + } + } + } + }, + "principals": { + "additionalProperties": { + "anyOf": [ + { + "type": "string", + "pattern": "^(user|group|serviceAccount|domain):.+$" + }, + { + "type": "array", + "items": { + "type": "string", + "pattern": "^(user|group|serviceAccount|domain):.+$" + } + } + ] + } + } + } + } + }, + { + "if": { + "properties": { + "platform": { + "const": "aws" + } + }, + "required": [ + "platform" + ] + }, + "then": { + "properties": { + "encryption": { + "properties": { + "kms": { + "not": { + "pattern": "^projects/" + } + } + } + }, + "principals": { + "additionalProperties": { + "anyOf": [ + { + "type": "string", + "pattern": "^arn:aws[a-z0-9-]*:iam::" + }, + { + "type": "array", + "items": { + "type": "string", + "pattern": "^arn:aws[a-z0-9-]*:iam::" + } + } + ] + } + } + } + } + } + ] + }, + "bindingLocation": { + "type": "object", + "description": "Provider-native addressing with tightened structure for better interop.", + "additionalProperties": false, + "properties": { + "account": { + "type": "string", + "description": "Cloud account ID" + }, + "project": { + "type": "string", + "description": "GCP project ID" + }, + "dataset": { + "type": "string", + "description": "BigQuery dataset or database name" + }, + "database": { + "type": "string", + "description": "Database name" + }, + "schema": { + "type": "string", + "description": "Schema name" + }, + "table": { + "type": "string", + "description": "Table name" + }, + "bucket": { + "type": "string", + "description": "Storage bucket name" + }, + "path": { + "type": "string", + "description": "File or object path" + }, + "gateway": { + "type": "string", + "description": "API gateway endpoint" + }, + "baseUrl": { + "type": "string", + "format": "uri", + "description": "Base URL for API endpoints" + }, + "topic": { + "type": "string", + "description": "Pub/Sub or Kafka topic name" + }, + "subscription": { + "type": "string", + "description": "Pub/Sub subscription name" + }, + "region": { + "type": "string", + "description": "Cloud region" + }, + "zone": { + "type": "string", + "description": "Cloud zone" + }, + "environment_id": { + "type": "string", + "description": "NEW in v0.7.5: Confluent Cloud environment id (env-xxxxx) for the Tableflow emitter." + }, + "kafka_cluster_id": { + "type": "string", + "description": "NEW in v0.7.5: Confluent Cloud Kafka cluster id (lkc-xxxxx) the Tableflow topic belongs to." + }, + "stream": { + "type": "string", + "description": "NEW in v0.7.5: AWS Kinesis Data Stream name — maps to aws_kinesis_stream in the streaming IaC emitter." + }, + "namespace": { + "type": "string", + "description": "NEW in v0.7.5: Amazon Redshift Serverless namespace name — maps to aws_redshiftserverless_namespace." + }, + "workgroup": { + "type": "string", + "description": "NEW in v0.7.5: Amazon Redshift Serverless workgroup name — maps to aws_redshiftserverless_workgroup." + }, + "iam_role_arn": { + "type": "string", + "description": "NEW in v0.7.5: ARN of the IAM role attached to the Redshift Serverless namespace / used by the redshift-data CREATE EXTERNAL SCHEMA bridge." + }, + "external_schema": { + "type": "string", + "description": "NEW in v0.7.5: Redshift external (Spectrum) schema name created over a Glue database via the redshift-data CREATE EXTERNAL SCHEMA bridge." + }, + "glue_database": { + "type": "string", + "description": "NEW in v0.7.5: Glue Data Catalog database that backs the Redshift external schema." + }, + "confluent_role_arn": { + "type": "string", + "description": "NEW in v0.7.5: ARN of the pre-created AWS IAM role Confluent Tableflow assumes (byob_aws). Its trust policy is updated post-apply with the provider integration's external_id (two-phase IAM)." + }, + "catalog": { + "type": "string", + "description": "NEW in v0.7.5: Iceberg catalog kind/profile selector — maps to iceberg.catalog.type (e.g. glue, rest, nessie, hive). REST-fronted catalogs (polaris / unity / snowflake open catalog) map to 'rest'. Consumed by providers/_iceberg_catalog.py and the streaming-sink validator." + }, + "warehouse": { + "type": "string", + "description": "NEW in v0.7.5: Iceberg warehouse — the canonical Iceberg REST 'warehouse' property (maps to iceberg.catalog.warehouse): an object-store URI (s3://|gs://|abfss://) for object-store catalogs, or the catalog/warehouse name for a REST catalog." + }, + "uri": { + "type": "string", + "description": "NEW in v0.7.5: REST Iceberg catalog endpoint URL — the canonical Iceberg REST 'uri' property (maps to iceberg.catalog.uri)." + }, + "partitionBy": { + "type": "array", + "items": { + "type": "string" + }, + "description": "NEW in v0.7.5: Iceberg partition columns — a FLUID abstraction over the Iceberg PARTITIONED BY / PartitionSpec (maps to iceberg.tables.default-partition-by). On a gcp BigQuery table with exposes[].lifecycle.expire: true (fluid-schema 0.7.6), one DATE, TIMESTAMP or DATETIME column to partition by (time_partitioning.field); without it the table is partitioned by ingestion time. With a column, retention counts from the date in that column, not from landing: BigQuery deletes a partition `retention` after its date, so a backfill of older rows is deleted at once. Leave it out for the S3 rule's semantics (age since written)." + } + } + }, + "observability": { + "type": "object", + "additionalProperties": false, + "properties": { + "metrics": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "source" + ], + "properties": { + "name": { + "type": "string" + }, + "source": { + "type": "string" + }, + "sli": { + "type": "string", + "description": "Human/machine-readable SLI check description" + } + } + } + }, + "onBreach": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "type" + ], + "properties": { + "type": { + "type": "string", + "enum": [ + "slack", + "email", + "webhook", + "pagerduty", + "custom" + ] + }, + "channel": { + "type": "string" + }, + "target": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + }, + "severity": { + "type": "string", + "enum": [ + "info", + "warning", + "critical" + ] + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Alert configuration tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Alert configuration labels." + } + } + } + }, + "defaultSLIs": { + "$ref": "#/$defs/defaultSLIs", + "description": "Default SLI trio (freshness, completeness, latency) by kind." + }, + "alert": { + "type": "object", + "additionalProperties": false, + "description": "Alert dispatcher config consumed by the acquisition Alerter (DLQ overflow, schema-fingerprint changes, anomaly signals).", + "properties": { + "channels": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind" + ], + "properties": { + "kind": { + "type": "string", + "enum": [ + "log", + "file", + "webhook" + ] + }, + "path": { + "type": "string", + "description": "For kind=file: NDJSON sink path (env-vars expanded)." + }, + "url": { + "type": "string", + "format": "uri", + "description": "For kind=webhook: Slack-compatible incoming webhook URL. Validated against the SSRF guard at construction time." + } + } + } + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Observability configuration tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Observability configuration labels." + } + } + }, + "defaultSLIs": { + "type": "object", + "description": "Default SLI trio that CLI can auto-generate based on expose kind.", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "default": true, + "description": "Enable auto-generation of default SLIs." + }, + "freshness": { + "type": "object", + "description": "Default freshness SLI configuration.", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "threshold": { + "$ref": "#/$defs/isoDuration", + "description": "Freshness threshold (auto-set by kind: table=PT6H, stream=PT1M, api=PT1S)" + }, + "severity": { + "type": "string", + "enum": [ + "info", + "warning", + "critical" + ], + "default": "warning" + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Freshness SLI tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Freshness SLI labels." + } + } + }, + "completeness": { + "type": "object", + "description": "Default completeness SLI configuration.", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "threshold": { + "type": "number", + "minimum": 0, + "maximum": 1, + "default": 0.95, + "description": "Completeness threshold (auto-set by kind: table=0.95, stream=0.99)" + }, + "severity": { + "type": "string", + "enum": [ + "info", + "warning", + "critical" + ], + "default": "warning" + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Completeness SLI tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Completeness SLI labels." + } + } + }, + "latency": { + "type": "object", + "description": "Default latency SLI configuration.", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "threshold": { + "$ref": "#/$defs/isoDuration", + "description": "P95 latency threshold (auto-set by kind: api=PT200MS, table=PT5S, stream=PT100MS)" + }, + "percentile": { + "type": "number", + "minimum": 0, + "maximum": 100, + "default": 95, + "description": "Latency percentile to measure." + }, + "severity": { + "type": "string", + "enum": [ + "info", + "warning", + "critical" + ], + "default": "warning" + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Latency SLI tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Latency SLI labels." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Default SLI configuration tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Default SLI configuration labels." + } + } + }, + "lifecycle": { + "type": "object", + "additionalProperties": false, + "properties": { + "state": { + "$ref": "#/$defs/lifecycleState", + "description": "Unified lifecycle state vocabulary." + }, + "retention": { + "$ref": "#/$defs/isoDuration" + }, + "deprecationPolicy": { + "type": "object", + "additionalProperties": false, + "properties": { + "noticePeriod": { + "$ref": "#/$defs/isoDuration" + }, + "contact": { + "type": "string" + }, + "replacement": { + "type": "string", + "description": "Reference to replacement data product or expose." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Deprecation policy tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Deprecation policy labels." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Lifecycle tags for automation." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Lifecycle labels for governance." + } + } + }, + "exposeLifecycle": { + "type": "object", + "additionalProperties": false, + "properties": { + "state": { + "$ref": "#/$defs/lifecycleState", + "description": "Unified lifecycle state vocabulary." + }, + "retention": { + "$ref": "#/$defs/isoDuration", + "description": "How long this expose's data is kept (ISO-8601 duration, e.g. P30D, P7Y). A declaration unless `expire` is true. With `expire: true` on an AWS S3 binding, `fluid apply` expires the objects under the binding's prefix this long after they are written; on a GCP BigQuery table it deletes each daily partition this long after its day ends. Years and months are counted as 365 and 30 days, and a part of a day rounds up to a whole one." + }, + "expire": { + "type": "boolean", + "default": false, + "description": "NEW in v0.7.6: Delete stored data once it is older than `retention`. Default false, so a contract that declares a retention period does not start deleting data when forge-cli is upgraded. Honoured by the AWS emitter on an S3 binding and by the GCP emitter on a BigQuery table; on a gcp binding that is not a BigQuery table (GCS, Pub/Sub, Iceberg storage) it is refused rather than dropped. AWS: `fluid apply` writes one rule per expose into the bucket's aws_s3_bucket_lifecycle_configuration, filtered to the binding's prefix (location.path, else {database}/{table}/; a binding with neither is refused, so no rule ever expires a whole bucket). The rule expires the current version `retention` after it is written, removes a noncurrent version (on a versioning-enabled bucket) one day after it stops being current, and aborts an incomplete multipart upload after 7 days or the retention period if shorter. The lifecycle configuration is AUTHORITATIVE for the whole bucket: S3 keeps one per bucket, so rules that `fluid apply` did not write are replaced. It is therefore written only for a bucket this product owns; on a shared (pool) bucket (packaging) nothing is written, the pool's owner must hold the rule, and `fluid verify` checks that an enabled rule covering the prefix expires objects after exactly this period. Objects already older than the period when the rule is first applied are expired on S3's next lifecycle run, typically within a day. Removing `expire` later deletes the lifecycle configuration, which `fluid apply` refuses without --allow-data-loss (it refuses every plan that removes a resource holding data or policy; removing an access grant or a policy tag is a revocation and is not gated); `tofu destroy` deletes it. GCP (fluid-schema 0.7.6): the BigQuery table is partitioned by day (time_partitioning type DAY, on binding.location.partitionBy when it names one date or timestamp column, else by ingestion time) with expiration_ms = `retention` in days, so BigQuery deletes each partition that long after its day ends. BigQuery counts from the partition's date, not from when its rows were written: by ingestion time (no partitionBy, the default and the S3 rule's semantics) no row is deleted sooner than `retention` after it landed; with partitionBy, retention is the age of the date in that column, so a backfill of rows whose date is already older than `retention` lands in expired partitions and is deleted at once. It is never a whole-table expiration, which would delete the product. BigQuery cannot partition a table that exists, so adding `expire` to a live table replaces it: `fluid apply` refuses that plan without --allow-data-loss, and the next build lands the data again. A later change of `retention` is an in-place update. Removing `expire` from a live table removes the partitioning's replacement trigger, which `fluid apply` also refuses without --allow-data-loss; BigQuery cannot un-partition a table, so to keep data longer set a longer `retention` instead. `fluid verify` checks the live table's partition expiration." + }, + "deprecationPolicy": { + "type": "object", + "additionalProperties": false, + "properties": { + "noticePeriod": { + "$ref": "#/$defs/isoDuration" + }, + "contact": { + "type": "string" + }, + "replacement": { + "type": "string", + "description": "Reference to replacement data product or expose." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Deprecation policy tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Deprecation policy labels." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Lifecycle tags for automation." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Lifecycle labels for governance." + } + }, + "description": "NEW in v0.7.6: An expose's lifecycle. The same fields as the contract-root lifecycle, plus `expire`, which is only meaningful where there is storage to expire." + }, + "docs": { + "type": "object", + "additionalProperties": false, + "properties": { + "homepage": { + "type": "string", + "format": "uri" + }, + "runbook": { + "type": "string", + "format": "uri" + }, + "dictionary": { + "type": "string", + "format": "uri" + }, + "changeLog": { + "type": "string", + "format": "uri" + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Documentation tags for organization." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Documentation labels for automation." + } + } + }, + "agentPolicy": { + "type": "object", + "description": "NEW in v0.7.1: AI/LLM usage governance. Controls which AI models can consume data and defines usage boundaries. Enables data owners to enforce 'reporting yes, training no' policies.", + "additionalProperties": false, + "properties": { + "allowedModels": { + "type": "array", + "description": "Whitelist of AI models permitted to consume this data. Examples: ['gpt-4', 'claude-3-opus', 'gemini-pro']. Empty array = no AI access.", + "items": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9-_.]*[a-z0-9]$|^[a-z0-9]$", + "examples": [ + "gpt-4", + "claude-3-opus", + "claude-3-sonnet", + "gemini-pro", + "gemini-ultra", + "llama-3", + "mixtral-8x7b" + ] + }, + "uniqueItems": true + }, + "deniedModels": { + "type": "array", + "description": "Blacklist of AI models explicitly prohibited from consuming this data. Takes precedence over allowedModels.", + "items": { + "type": "string" + }, + "uniqueItems": true + }, + "maxTokensPerRequest": { + "type": "integer", + "minimum": 1, + "description": "Maximum tokens per AI request. Prevents excessive data exposure in single queries.", + "examples": [ + 4096, + 8192, + 16384 + ] + }, + "maxTokensPerDay": { + "type": "integer", + "minimum": 1, + "description": "Daily token limit for AI consumption. Enforces usage quotas." + }, + "allowedUseCases": { + "type": "array", + "description": "Permitted AI use cases. Controls whether data can be used for training, inference, reasoning, etc.", + "items": { + "type": "string", + "enum": [ + "inference", + "reasoning", + "analysis", + "summarization", + "classification", + "embedding", + "search", + "qa", + "code_generation", + "fine_tuning", + "training", + "rag" + ] + }, + "uniqueItems": true, + "examples": [ + [ + "inference", + "reasoning", + "summarization" + ], + [ + "search", + "qa", + "rag" + ] + ] + }, + "deniedUseCases": { + "type": "array", + "description": "Prohibited AI use cases. Example: Allow reporting but deny model training.", + "items": { + "type": "string", + "enum": [ + "inference", + "reasoning", + "analysis", + "summarization", + "classification", + "embedding", + "search", + "qa", + "code_generation", + "fine_tuning", + "training", + "rag" + ] + }, + "uniqueItems": true, + "examples": [ + [ + "training", + "fine_tuning" + ] + ] + }, + "canReason": { + "type": "boolean", + "default": false, + "description": "Whether AI agents can use this data for multi-step reasoning/chain-of-thought. False = simple retrieval only." + }, + "canStore": { + "type": "boolean", + "default": false, + "description": "Whether AI systems can cache/store this data. False = ephemeral processing only." + }, + "retentionPolicy": { + "type": "object", + "description": "Data retention requirements for AI systems consuming this data.", + "properties": { + "maxRetentionDays": { + "type": "integer", + "minimum": 0, + "description": "Maximum days AI system can retain this data. 0 = no retention allowed." + }, + "requireDeletion": { + "type": "boolean", + "default": true, + "description": "Whether AI system must delete data after use." + } + } + }, + "auditRequired": { + "type": "boolean", + "default": true, + "description": "Whether AI consumption must be logged/audited." + }, + "purposeLimitation": { + "type": "string", + "description": "Free-text description of allowed purposes. Example: 'Customer service chatbot only, not for marketing'." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Agent policy tags for categorization." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Agent policy labels for automation." + } + } + }, + "sovereignty": { + "type": "object", + "description": "NEW in v0.7.1: Data sovereignty and jurisdiction requirements. Declares WHERE data must reside and enables compliance enforcement before deployment.", + "additionalProperties": false, + "properties": { + "jurisdiction": { + "type": "string", + "description": "Required legal jurisdiction for data storage and processing. Used to validate binding.location matches sovereignty intent.", + "enum": [ + "EU", + "US", + "UK", + "CA", + "AU", + "JP", + "CN", + "IN", + "BR", + "Global", + "Multi-Region" + ], + "examples": [ + "EU", + "US", + "Global" + ] + }, + "allowedRegions": { + "type": "array", + "description": "Explicit list of allowed cloud regions. Example: ['eu-west-1', 'eu-central-1'] for EU-only.", + "items": { + "type": "string" + }, + "uniqueItems": true, + "examples": [ + [ + "eu-west-1", + "eu-central-1" + ], + [ + "us-east-1", + "us-west-2" + ] + ] + }, + "deniedRegions": { + "type": "array", + "description": "Explicit list of prohibited cloud regions. Takes precedence over allowedRegions.", + "items": { + "type": "string" + }, + "uniqueItems": true + }, + "dataResidency": { + "type": "boolean", + "default": true, + "description": "Whether data must remain within jurisdiction boundaries at rest and in transit. True = strict residency." + }, + "crossBorderTransfer": { + "type": "boolean", + "default": false, + "description": "Whether cross-border data transfer is permitted. False = data never leaves jurisdiction." + }, + "transferMechanisms": { + "type": "array", + "description": "Allowed legal mechanisms for cross-border transfer (if permitted). Example: ['SCCs', 'BCRs', 'Adequacy'].", + "items": { + "type": "string", + "enum": [ + "SCCs", + "BCRs", + "Adequacy", + "DPF", + "Consent", + "Derogation" + ] + }, + "uniqueItems": true + }, + "regulatoryFramework": { + "type": "array", + "description": "Applicable regulatory frameworks. Example: ['GDPR', 'CCPA', 'HIPAA'].", + "items": { + "type": "string", + "enum": [ + "GDPR", + "CCPA", + "CPRA", + "HIPAA", + "PIPEDA", + "LGPD", + "PDPA", + "POPIA", + "DPA", + "APPI" + ] + }, + "uniqueItems": true, + "examples": [ + [ + "GDPR" + ], + [ + "GDPR", + "CCPA" + ], + [ + "HIPAA" + ] + ] + }, + "enforcementMode": { + "type": "string", + "enum": [ + "strict", + "advisory", + "audit" + ], + "default": "strict", + "description": "How sovereignty violations are handled. strict = block deployment, advisory = warn, audit = log only." + }, + "validationRequired": { + "type": "boolean", + "default": true, + "description": "Whether binding.location must be validated against sovereignty rules before deployment." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Sovereignty policy tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Sovereignty policy labels." + } + }, + "examples": [ + { + "jurisdiction": "EU", + "allowedRegions": [ + "eu-west-1", + "eu-central-1" + ], + "dataResidency": true, + "crossBorderTransfer": false, + "regulatoryFramework": [ + "GDPR" + ], + "enforcementMode": "strict" + }, + { + "jurisdiction": "US", + "allowedRegions": [ + "us-east-1", + "us-west-2" + ], + "dataResidency": true, + "regulatoryFramework": [ + "CCPA", + "HIPAA" + ], + "enforcementMode": "strict" + } + ] + }, + "accessPolicy": { + "type": "object", + "description": "Root-level access control policy for automated IAM binding generation. Defines who can access this data product and with what permissions.", + "additionalProperties": false, + "properties": { + "grants": { + "type": "array", + "description": "List of access grants to principals (users, groups, service accounts).", + "items": { + "type": "object", + "required": [ + "principal" + ], + "additionalProperties": false, + "properties": { + "principal": { + "type": "string", + "description": "Identity receiving access. Format: 'user:email@domain.com', 'group:name@domain.com', 'serviceAccount:name@project.iam.gserviceaccount.com'", + "examples": [ + "group:data-analytics@company.com", + "serviceAccount:pipeline@project.iam.gserviceaccount.com", + "user:analyst@company.com" + ] + }, + "permissions": { + "type": "array", + "description": "Permissions to grant. Will be mapped to provider-specific roles (e.g., BigQuery roles).", + "items": { + "type": "string", + "enum": [ + "read", + "select", + "query", + "write", + "insert", + "update", + "delete", + "create", + "admin", + "manage" + ] + }, + "uniqueItems": true, + "examples": [ + [ + "read", + "select" + ], + [ + "write", + "insert", + "update" + ] + ] + }, + "resources": { + "type": "array", + "description": "Resources this grant applies to. JSONPath expressions referencing exposes[] items. If omitted, applies to all exposed resources.", + "items": { + "type": "string" + }, + "examples": [ + "$.exposes[?(@.exposeId=='customer_table')]", + "$.exposes[?(@.kind=='table')]" + ] + }, + "conditions": { + "type": "object", + "description": "Optional conditions for conditional access (e.g., IP restrictions, time windows).", + "additionalProperties": true + } + } + } + } + }, + "examples": [ + { + "grants": [ + { + "principal": "group:data-analytics@company.com", + "permissions": [ + "read", + "select", + "query" + ] + }, + { + "principal": "serviceAccount:pipeline@project.iam.gserviceaccount.com", + "permissions": [ + "write", + "insert", + "update", + "delete" + ] + } + ] + } + ] + }, + "consumeRef": { + "type": "object", + "additionalProperties": false, + "required": [ + "productId", + "exposeId" + ], + "$comment": "productId identifies data product (renamed from 'provider' for clarity)", + "properties": { + "productId": { + "$ref": "#/$defs/identifier", + "description": "Upstream data product identifier (e.g., 'silver.hr.people_v2'). Renamed from 'provider' for clarity." + }, + "exposeId": { + "$ref": "#/$defs/identifier", + "description": "The upstream exposeId being consumed." + }, + "versionConstraint": { + "$ref": "#/$defs/semverRange" + }, + "qosExpectations": { + "type": "object", + "additionalProperties": false, + "properties": { + "freshnessMax": { + "$ref": "#/$defs/isoDuration" + }, + "maxStaleness": { + "$ref": "#/$defs/isoDuration", + "description": "Maximum acceptable staleness for consumed data." + }, + "minCompleteness": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Minimum completeness expectation." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "QoS expectation tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "QoS expectation labels." + } + } + }, + "requiredPolicies": { + "type": "array", + "items": { + "type": "string" + } + }, + "purpose": { + "type": "string", + "description": "Business purpose for consuming this data." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Consumption relationship tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Consumption relationship labels." + }, + "upstreamWorkspace": { + "type": "string", + "minLength": 1, + "description": "Cross-mesh only. Id of the federated workspace that owns this upstream, as declared in federation/upstreams.yaml. Set it when the upstream lives in another mesh rather than this workspace; omit it for a local upstream, which is resolved by walking the workspace." + }, + "upstreamDigest": { + "type": "string", + "pattern": "^sha256:[0-9a-f]{64}$", + "description": "Cross-mesh only. The contract digest this product was composed against, as sha256:<64 hex>. `fluid apply` re-fetches the upstream's live digest and compares: a mismatch means the upstream changed since composition. Required whenever upstreamWorkspace is set." + } + }, + "dependentRequired": { + "upstreamWorkspace": [ + "upstreamDigest" + ] + } + }, + "build": { + "type": "object", + "description": "Enhanced build configuration supporting v0.4.0 patterns with v0.5.5 improvements.", + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/identifier", + "description": "Build identifier for multi-build scenarios (e.g., 'batch-processing', 'ml-training')." + }, + "description": { + "type": "string", + "description": "Human-readable description of this build configuration." + }, + "pattern": { + "type": "string", + "enum": [ + "hybrid-reference", + "embedded-logic", + "multi-stage", + "acquisition" + ], + "default": "hybrid-reference", + "description": "Build pattern: hybrid-reference (dbt-style), embedded-logic (raw SQL/code), multi-stage (complex pipelines), or acquisition (source-aligned ingestion from external systems)." + }, + "engine": { + "type": "string", + "anyOf": [ + { + "enum": [ + "dbt", + "sql", + "python", + "spark", + "glue", + "custom", + "duckdb", + "airbyte", + "meltano", + "dlt", + "kafka-connect", + "debezium" + ] + }, + { + "pattern": "^dbt-[a-z0-9]+([_-][a-z0-9]+)*$" + } + ], + "default": "dbt", + "description": "Build/ingestion engine. Transformation engines: dbt, sql, python, spark, glue, custom. Adapter-qualified dbt engines (dbt-, e.g. dbt-glue, dbt-snowflake, dbt-bigquery) are accepted too and route through the dbt build runner, which derives the adapter from the suffix. Ingestion engines (acquisition pattern only): duckdb (zero-infra files/JDBC), airbyte (350+ SaaS), meltano (Singer 600+ taps), dlt (Python-native), kafka-connect (streaming), debezium (CDC). AWS Glue: glue (Spark ETL jobs managed by Glue)." + }, + "capabilities": { + "type": "array", + "description": "Capabilities the build asks for (acquisition pattern). Validation enforces capabilities ⊆ runner.declared_capabilities.", + "items": { + "type": "string", + "enum": [ + "full_refresh", + "incremental_append", + "incremental_dedup", + "incremental_merge", + "cdc", + "streaming", + "schema_discovery", + "schema_evolution", + "dlp_scan", + "at_most_once", + "at_least_once", + "exactly_once" + ] + }, + "uniqueItems": true + }, + "repository": { + "type": "string" + }, + "properties": { + "$ref": "#/$defs/buildProperties", + "description": "Pattern-specific build properties with conditional validation." + }, + "execution": { + "$ref": "#/$defs/buildExecution", + "description": "Build execution configuration (triggers, runtime, retries)." + }, + "outputs": { + "type": "array", + "description": "List of exposeIds produced by this build.", + "items": { + "$ref": "#/$defs/identifier" + } + }, + "dependencies": { + "type": "array", + "description": "Dependencies on other builds (for multi-build orchestration).", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "buildId": { + "$ref": "#/$defs/identifier" + }, + "condition": { + "type": "string", + "enum": [ + "success", + "completion", + "always" + ] + } + } + } + }, + "transformations": { + "type": "array", + "description": "Legacy transformations array for backward compatibility.", + "items": { + "$ref": "#/$defs/transformation" + } + }, + "tags": { + "$ref": "#/$defs/tags" + }, + "labels": { + "$ref": "#/$defs/labels" + } + }, + "allOf": [ + { + "if": { + "properties": { + "pattern": { + "const": "hybrid-reference" + } + } + }, + "then": { + "properties": { + "properties": { + "$ref": "#/$defs/hybridReferencePattern" + } + } + } + }, + { + "if": { + "properties": { + "pattern": { + "const": "embedded-logic" + } + } + }, + "then": { + "properties": { + "properties": { + "$ref": "#/$defs/embeddedLogicPattern" + } + } + } + }, + { + "if": { + "properties": { + "pattern": { + "const": "multi-stage" + } + } + }, + "then": { + "properties": { + "properties": { + "$ref": "#/$defs/multiStagePattern" + } + } + } + }, + { + "if": { + "properties": { + "pattern": { + "const": "acquisition" + } + } + }, + "then": { + "properties": { + "properties": { + "$ref": "#/$defs/acquisitionPattern" + } + } + } + } + ] + }, + "buildProperties": { + "type": "object", + "description": "Base properties for all build patterns." + }, + "hybridReferencePattern": { + "type": "object", + "description": "Hybrid-reference pattern from v0.4.0 (dbt-style).", + "additionalProperties": false, + "properties": { + "model": { + "type": "string" + }, + "target": { + "type": "string", + "description": "dbt profile target to run against. Forwarded to dbt as `--target ` by the build runner (allowlisted against the profile's real targets; falls back to the profile default when the requested target is unavailable)." + }, + "select": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ], + "description": "dbt node selector(s) forwarded to `dbt --select`. Accepts a single selector string or an array of selector strings; the build runner drops flag-like values (those starting with '-')." + }, + "models": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ], + "description": "Legacy alias for `select` (dbt's older `--models` flag). Same string-or-array shape; `select` takes precedence when both are set." + }, + "vars": { + "type": "object", + "additionalProperties": true + }, + "materializations": { + "type": "object", + "additionalProperties": { + "type": "string", + "enum": [ + "table", + "view", + "incremental", + "ephemeral" + ] + } + }, + "tags": { + "$ref": "#/$defs/tags" + }, + "labels": { + "$ref": "#/$defs/labels" + } + }, + "required": [ + "model" + ] + }, + "embeddedLogicPattern": { + "type": "object", + "description": "Embedded logic pattern from v0.4.0.", + "additionalProperties": false, + "properties": { + "sql": { + "type": "string" + }, + "language": { + "type": "string", + "enum": [ + "sql", + "flink_sql", + "pyspark", + "scala", + "python", + "r" + ] + }, + "parameters": { + "type": "object", + "additionalProperties": true + }, + "tags": { + "$ref": "#/$defs/tags" + }, + "labels": { + "$ref": "#/$defs/labels" + } + }, + "required": [ + "sql" + ] + }, + "multiStagePattern": { + "type": "object", + "description": "NEW in v0.5.5: Multi-stage orchestration pattern.", + "additionalProperties": false, + "properties": { + "stages": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "name": { + "type": "string" + }, + "pattern": { + "type": "string", + "enum": [ + "hybrid-reference", + "embedded-logic" + ], + "default": "hybrid-reference" + }, + "properties": { + "$ref": "#/$defs/buildProperties" + }, + "dependsOn": { + "type": "array", + "items": { + "type": "string" + } + }, + "outputs": { + "type": "array", + "items": { + "$ref": "#/$defs/identifier" + } + }, + "tags": { + "$ref": "#/$defs/tags" + }, + "labels": { + "$ref": "#/$defs/labels" + } + } + } + }, + "orchestration": { + "type": "object", + "properties": { + "parallelism": { + "type": "integer", + "minimum": 1 + }, + "retryPolicy": { + "$ref": "#/$defs/retryPolicy" + } + } + }, + "tags": { + "$ref": "#/$defs/tags" + }, + "labels": { + "$ref": "#/$defs/labels" + } + } + }, + "acquisitionPattern": { + "type": "object", + "description": "NEW in v0.7.3: Source-aligned acquisition pattern. The build ingests data from an external system (no transformation). Engine ∈ {duckdb, airbyte, meltano, dlt, kafka-connect, debezium}.", + "additionalProperties": false, + "required": [ + "source" + ], + "properties": { + "source": { + "$ref": "#/$defs/acquisitionSource" + }, + "sink": { + "$ref": "#/$defs/acquisitionSink" + }, + "delivery": { + "$ref": "#/$defs/acquisitionDelivery" + }, + "schemaEvolution": { + "$ref": "#/$defs/acquisitionSchemaEvolution" + }, + "preLand": { + "type": "array", + "description": "Pre-land hook chain - runs on each batch before destination write.", + "items": { + "type": "string", + "enum": [ + "dlp_scan", + "tokenize_pii", + "quality_gate", + "emit_lineage_input" + ] + } + }, + "quality": { + "$ref": "#/$defs/acquisitionQuality" + }, + "cost": { + "$ref": "#/$defs/acquisitionCost" + }, + "catalog": { + "$ref": "#/$defs/acquisitionCatalog" + }, + "concurrency": { + "$ref": "#/$defs/acquisitionConcurrency" + }, + "lineage": { + "type": "object", + "additionalProperties": false, + "properties": { + "emit": { + "type": "boolean", + "default": true + } + } + }, + "duckdb": { + "type": "object", + "additionalProperties": true, + "properties": { + "extensions": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "airbyte": { + "type": "object", + "additionalProperties": true, + "properties": { + "connector_image": { + "type": "string" + }, + "version": { + "type": "string" + }, + "normalization": { + "type": "boolean" + }, + "image_signature": { + "$ref": "#/$defs/acquisitionImageSignature" + }, + "deployment": { + "$ref": "#/$defs/acquisitionDeployment" + } + } + }, + "meltano": { + "type": "object", + "additionalProperties": true, + "properties": { + "tap": { + "type": "string" + }, + "project_dir": { + "type": "string" + }, + "deployment": { + "$ref": "#/$defs/acquisitionDeployment" + } + } + }, + "dlt": { + "type": "object", + "additionalProperties": true, + "properties": { + "source_module": { + "type": "string" + }, + "pipeline_name": { + "type": "string" + } + } + }, + "kafka-connect": { + "type": "object", + "additionalProperties": true, + "properties": { + "connector_class": { + "type": "string" + }, + "connector_name": { + "type": "string" + }, + "tasks_max": { + "type": "integer", + "minimum": 1, + "default": 1 + }, + "schema_registry": { + "type": "object", + "additionalProperties": false, + "description": "Confluent-compatible Schema Registry config. When ``url`` is set, the runner injects ``key.converter``/``value.converter`` for Avro into the emitted connector config.", + "properties": { + "url": { + "type": "string", + "format": "uri", + "description": "Schema Registry HTTP endpoint, e.g. http://schema-registry:8081" + }, + "auth": { + "type": "object", + "additionalProperties": false, + "properties": { + "secretRef": { + "type": "string" + } + } + } + } + }, + "sink_connector_name": { + "type": "string" + }, + "sink_connector_config": { + "type": "object", + "additionalProperties": true + }, + "image_signature": { + "$ref": "#/$defs/acquisitionImageSignature" + }, + "deployment": { + "$ref": "#/$defs/acquisitionDeployment" + }, + "iceberg_sink_enabled": { + "type": "boolean", + "description": "Opt-in: derive the Iceberg sink connector config. Defaults OFF when a hand-written sink_connector_config is present (RFC §6.2)." + }, + "sink_topics": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Explicit topics the derived Iceberg sink consumes (else derived from source streams)." + }, + "iceberg_catalog_overrides": { + "type": "object", + "additionalProperties": { + "type": "string" + }, + "description": "Operator escape hatch: connector config keys merged LAST over the derived Iceberg sink config." + }, + "streamingSink": { + "type": "object", + "additionalProperties": false, + "description": "Optional Iceberg streaming-sink tuning (RFC §6.2). All keys optional.", + "properties": { + "commitIntervalMs": { + "type": "integer", + "minimum": 1 + }, + "routeField": { + "type": "string" + }, + "dynamicEnabled": { + "type": "boolean" + }, + "autoCreate": { + "type": "boolean" + }, + "evolveSchema": { + "type": "boolean" + }, + "upsertMode": { + "type": "boolean" + }, + "controlTopic": { + "type": "string" + } + } + } + } + }, + "debezium": { + "type": "object", + "additionalProperties": true, + "properties": { + "connector_class": { + "type": "string" + }, + "snapshot_mode": { + "type": "string", + "enum": [ + "initial", + "schema_only", + "never", + "when_needed", + "always" + ] + }, + "deployment": { + "$ref": "#/$defs/acquisitionDeployment" + } + } + } + } + }, + "acquisitionSource": { + "type": "object", + "description": "External source system specification.", + "additionalProperties": false, + "required": [ + "kind", + "mode" + ], + "properties": { + "kind": { + "type": "string", + "description": "Source system kind: filesystem, postgres, mysql, sqlite, http, salesforce, stripe, github, kafka, etc. Engine-specific catalog of supported kinds." + }, + "connection": { + "type": "object", + "description": "Connection details. Use ${VAR} placeholders for env values; use secretRef for credentials. Inline secrets are forbidden by validator.", + "additionalProperties": true, + "properties": { + "uri": { + "type": "string" + }, + "host": { + "type": "string" + }, + "port": { + "oneOf": [ + { + "type": "integer" + }, + { + "type": "string", + "description": "Allows ${ENV_VAR} placeholders that resolve to an integer at runtime." + } + ] + }, + "database": { + "type": "string" + }, + "instance_url": { + "type": "string" + }, + "secretRef": { + "type": "string", + "pattern": "^[a-z][a-z0-9_+.-]*://.+", + "description": "Reference to a secret in the configured backend (vault://, aws://, gcp://, azure://, env://)." + }, + "user": { + "type": "string", + "description": "NEW in v0.7.3: Username/principal for SQL- or REST-style sources. Required by most database drivers (Postgres/MySQL/Oracle/etc.) — without it the underlying client falls back to the OS-user identity and typically fails authentication. Goes alongside ``secretRef`` (or an inline ``password``) for the credential pair." + }, + "schema": { + "type": "string", + "description": "NEW in v0.7.3: Default schema/namespace within the source database to read from. Maps to engine-specific config: dlt sql_database schema=, Meltano tap-postgres filter_schemas=[], Airbyte source-postgres schemas=[], Debezium schema.include.list, Kafka Connect JDBC schema.pattern. Use ``schemas`` (array) to scope to multiple schemas at once." + }, + "schemas": { + "type": "array", + "items": { + "type": "string" + }, + "uniqueItems": true, + "description": "NEW in v0.7.3: Multi-schema variant of ``schema``. When both are set, ``schemas`` wins and ``schema`` is treated as a deprecated alias. Engines that don't natively support multi-schema reads will iterate the list." + } + } + }, + "mode": { + "type": "string", + "enum": [ + "full_refresh", + "incremental_append", + "incremental_dedup", + "incremental_merge", + "cdc", + "streaming" + ] + }, + "cursor_field": { + "type": "string", + "description": "Column used for incremental cursor (e.g., updated_at, SystemModstamp)." + }, + "watermark": { + "type": "object", + "additionalProperties": false, + "properties": { + "strategy": { + "type": "string", + "enum": [ + "high_water_mark", + "log_position", + "lsn" + ] + }, + "allowedLateness": { + "$ref": "#/$defs/isoDuration" + } + } + }, + "streams": { + "type": "array", + "description": "List of streams/tables/objects to ingest.", + "items": { + "type": "string" + } + }, + "reader": { + "type": "object", + "additionalProperties": true, + "properties": { + "format": { + "type": "string", + "enum": [ + "csv", + "parquet", + "json", + "ndjson", + "avro", + "orc" + ] + }, + "options": { + "type": "object", + "additionalProperties": true + } + } + } + } + }, + "acquisitionSink": { + "type": "object", + "description": "Destination format for ingested data. binding.platform on the expose remains the source of truth for where; sink.format describes how.", + "additionalProperties": false, + "properties": { + "format": { + "type": "string", + "enum": [ + "iceberg", + "delta", + "parquet", + "csv", + "json", + "snowflake_table", + "bigquery_table", + "redshift_table", + "duckdb_table" + ] + }, + "catalog": { + "type": "string", + "enum": [ + "rest", + "glue", + "nessie", + "unity", + "snowflake-managed", + "hive" + ] + }, + "partitionBy": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "acquisitionDelivery": { + "type": "object", + "description": "Delivery semantics + idempotency + DLQ configuration.", + "additionalProperties": false, + "properties": { + "guarantee": { + "type": "string", + "enum": [ + "at_most_once", + "at_least_once", + "exactly_once" + ], + "default": "at_least_once" + }, + "idempotencyKey": { + "type": "string", + "description": "Template for the idempotency key. Default: {run_id}:{stream}:{record_pk}", + "default": "{run_id}:{stream}:{record_pk}" + }, + "dlq": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "sink": { + "type": "object", + "additionalProperties": false, + "properties": { + "format": { + "type": "string", + "enum": [ + "parquet", + "json", + "ndjson" + ] + }, + "location": { + "type": "string" + } + } + }, + "maxRecordsBeforeAbort": { + "type": "integer", + "minimum": 0, + "default": 10000 + }, + "alertOn": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "pii_classification_failed", + "schema_violation", + "destination_write_failed", + "quality_gate_failed" + ] + } + } + } + } + } + }, + "acquisitionSchemaEvolution": { + "type": "object", + "description": "Schema evolution behavior. Stricter-wins between policy and per-change overrides.", + "additionalProperties": false, + "properties": { + "policy": { + "type": "string", + "enum": [ + "strict", + "discover_and_freeze", + "evolve_safe", + "evolve_all" + ], + "default": "strict" + }, + "onAddedColumn": { + "type": "string", + "enum": [ + "include", + "warn", + "fail" + ] + }, + "onRemovedColumn": { + "type": "string", + "enum": [ + "drop", + "warn", + "fail" + ] + }, + "onTypeChange": { + "type": "string", + "enum": [ + "cast", + "warn", + "fail" + ] + }, + "sourceFingerprint": { + "type": "string", + "enum": [ + "required", + "optional", + "disabled" + ], + "default": "required" + } + } + }, + "acquisitionQuality": { + "type": "object", + "description": "Pre-land quality gates + anomaly signals.", + "additionalProperties": false, + "properties": { + "gates": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": true, + "required": [ + "rule", + "severity" + ], + "properties": { + "rule": { + "type": "string", + "enum": [ + "not_null", + "unique", + "regex", + "range", + "row_count_anomaly", + "freshness" + ] + }, + "columns": { + "type": "array", + "items": { + "type": "string" + } + }, + "column": { + "type": "string" + }, + "pattern": { + "type": "string" + }, + "min": { + "type": "number" + }, + "max": { + "type": "number" + }, + "algorithm": { + "type": "string", + "enum": [ + "ewma", + "iqr", + "exact" + ] + }, + "sensitivity": { + "type": "number" + }, + "severity": { + "type": "string", + "enum": [ + "info", + "warn", + "error" + ] + } + } + } + }, + "onError": { + "type": "string", + "enum": [ + "route_to_dlq", + "abort_run", + "best_effort" + ], + "default": "route_to_dlq" + }, + "anomalies": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "signal", + "severity" + ], + "properties": { + "signal": { + "type": "string", + "enum": [ + "record_count_drop", + "record_count_surge", + "bytes_per_record_outlier", + "latency_outlier", + "dlq_rate_spike", + "cursor_stalled", + "schema_fingerprint_changed" + ] + }, + "algorithm": { + "type": "string", + "enum": [ + "ewma", + "iqr", + "exact" + ] + }, + "sensitivity": { + "type": "number" + }, + "severity": { + "type": "string", + "enum": [ + "info", + "warn", + "error" + ] + } + } + } + } + } + }, + "acquisitionCost": { + "type": "object", + "description": "Cost tracking + budget enforcement.", + "additionalProperties": false, + "properties": { + "budget": { + "type": "object", + "additionalProperties": false, + "properties": { + "monthly": { + "type": "object", + "additionalProperties": false, + "properties": { + "rows": { + "type": "integer", + "minimum": 0 + }, + "bytes": { + "type": "string", + "description": "Human-readable size with unit suffix (e.g., '50GB', '500MB')." + }, + "computeMinutes": { + "type": "integer", + "minimum": 0 + } + } + }, + "onExceed": { + "type": "string", + "enum": [ + "warn", + "abort" + ], + "default": "warn" + } + } + }, + "chargeback": { + "type": "object", + "additionalProperties": true, + "properties": { + "team": { + "type": "string" + }, + "project": { + "type": "string" + }, + "costCenter": { + "type": "string" + } + } + } + } + }, + "acquisitionCatalog": { + "type": "object", + "description": "Catalog auto-registration on first apply.", + "additionalProperties": false, + "properties": { + "register": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "datahub", + "openmetadata", + "datamesh_manager", + "unity", + "glue", + "snowflake_horizon" + ] + } + }, + "documentation": { + "type": "string", + "enum": [ + "auto", + "manual", + "none" + ], + "default": "auto" + } + } + }, + "acquisitionConcurrency": { + "type": "object", + "description": "Single-flight concurrency control.", + "additionalProperties": false, + "properties": { + "lock": { + "type": "object", + "additionalProperties": false, + "properties": { + "scope": { + "type": "string", + "enum": [ + "product", + "build" + ], + "default": "product" + }, + "timeout": { + "$ref": "#/$defs/isoDuration" + }, + "onContended": { + "type": "string", + "enum": [ + "abort", + "queue", + "replace" + ], + "default": "abort" + } + } + } + } + }, + "acquisitionImageSignature": { + "type": "object", + "description": "Connector image supply-chain verification.", + "additionalProperties": false, + "properties": { + "verifier": { + "type": "string", + "enum": [ + "cosign" + ], + "default": "cosign" + }, + "publicKey": { + "type": "string" + }, + "slsaProvenance": { + "type": "string", + "enum": [ + "required", + "optional", + "disabled" + ], + "default": "optional" + } + } + }, + "acquisitionDeployment": { + "type": "object", + "description": "Engine deployment mode and target.", + "additionalProperties": false, + "required": [ + "mode" + ], + "properties": { + "mode": { + "type": "string", + "enum": [ + "embedded", + "bring-your-own", + "managed" + ], + "default": "embedded" + }, + "server_url": { + "type": "string", + "description": "URL for bring-your-own mode." + }, + "auth": { + "type": "object", + "additionalProperties": true, + "properties": { + "secretRef": { + "type": "string" + } + } + }, + "managed": { + "type": "object", + "additionalProperties": false, + "required": [ + "target" + ], + "properties": { + "target": { + "type": "string", + "enum": [ + "docker", + "kubernetes", + "terraform", + "opentofu" + ] + }, + "profile": { + "type": "string", + "enum": [ + "small", + "medium", + "large" + ], + "default": "small" + }, + "chart": { + "type": "object", + "additionalProperties": true, + "properties": { + "repo": { + "type": "string" + }, + "name": { + "type": "string" + }, + "version": { + "type": "string" + } + } + }, + "values_overlay": { + "type": "object", + "additionalProperties": true + }, + "secrets": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "name": { + "type": "string" + }, + "ref": { + "type": "string" + } + } + } + }, + "network": { + "type": "object", + "additionalProperties": false, + "properties": { + "egressAllowList": { + "type": "array", + "items": { + "type": "string" + } + } + } + } + } + } + } + }, + "buildExecution": { + "type": "object", + "description": "Build execution configuration from v0.4.0 enhanced.", + "additionalProperties": false, + "properties": { + "trigger": { + "$ref": "#/$defs/trigger" + }, + "runtime": { + "$ref": "#/$defs/runtime" + }, + "retries": { + "$ref": "#/$defs/retryPolicy" + }, + "notifications": { + "type": "array", + "items": { + "$ref": "#/$defs/notification" + } + }, + "orchestration": { + "type": "object", + "required": [ + "engine" + ], + "properties": { + "engine": { + "type": "string", + "enum": [ + "airflow", + "dagster", + "prefect", + "kubeflow", + "custom", + "none" + ] + }, + "mode": { + "type": "string", + "enum": [ + "generated", + "manual", + "hybrid" + ], + "default": "generated" + }, + "generateOnChange": { + "type": "boolean", + "default": true + }, + "airflow": { + "$ref": "#/$defs/airflowOrchestration" + }, + "dagster": { + "type": "object" + }, + "prefect": { + "type": "object" + } + } + } + } + }, + "trigger": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "schedule", + "event", + "manual", + "dependency", + "dataset", + "schedule_and_dataset", + "timetable" + ] + }, + "schedule": { + "type": "string" + }, + "event": { + "type": "string" + }, + "condition": { + "type": "string" + }, + "datasets": { + "type": "array", + "description": "NEW in v0.7.0: Can reference data products directly via exposeRef for strong typing and auto-URI generation. Backward compatible with raw URIs.", + "items": { + "oneOf": [ + { + "type": "string", + "description": "Raw dataset URI (legacy, e.g., 'dataset://gcp/project/dataset/table')" + }, + { + "type": "object", + "description": "NEW in v0.7.0: Data product reference with strong typing", + "required": [ + "productId", + "exposeId" + ], + "properties": { + "productId": { + "$ref": "#/$defs/identifier", + "description": "Data product ID to depend on" + }, + "exposeId": { + "$ref": "#/$defs/identifier", + "description": "Expose ID within the data product" + }, + "versionConstraint": { + "$ref": "#/$defs/semverRange", + "description": "Optional version constraint (e.g., '^2.0.0')" + } + } + } + ] + } + }, + "datasetsOperator": { + "type": "string", + "enum": [ + "any", + "all" + ], + "default": "all" + }, + "cron": { + "type": "string", + "description": "Cron expression for schedule trigger" + }, + "timezone": { + "type": "string", + "default": "UTC" + }, + "timetable": { + "type": "object", + "description": "Custom timetable configuration" + } + } + }, + "runtime": { + "type": "object", + "properties": { + "resources": { + "type": "object", + "properties": { + "cpu": { + "type": "string" + }, + "memory": { + "type": "string" + }, + "disk": { + "type": "string" + } + } + }, + "timeout": { + "type": "string" + }, + "environment": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "platform": { + "type": "string", + "enum": [ + "gcp", + "aws", + "azure", + "local", + "kubernetes", + "composer", + "mwaa", + "databricks", + "snowflake", + "athena", + "glue", + "redshift" + ] + }, + "executor": { + "type": "string", + "enum": [ + "LocalExecutor", + "CeleryExecutor", + "KubernetesExecutor", + "SequentialExecutor" + ] + }, + "image": { + "type": "string", + "description": "Container image for execution" + }, + "serviceAccount": { + "type": "string", + "description": "Service account for cloud execution" + } + } + }, + "retryPolicy": { + "type": "object", + "properties": { + "maxAttempts": { + "type": "integer", + "minimum": 1 + }, + "backoffStrategy": { + "type": "string", + "enum": [ + "fixed", + "exponential", + "linear" + ] + }, + "initialDelay": { + "type": "string" + }, + "maxDelay": { + "type": "string" + } + } + }, + "notification": { + "type": "object", + "additionalProperties": false, + "properties": { + "type": { + "type": "string", + "enum": [ + "email", + "slack", + "webhook", + "pagerduty" + ] + }, + "target": { + "type": "string" + }, + "condition": { + "type": "string", + "enum": [ + "success", + "failure", + "always" + ] + } + } + }, + "transformation": { + "type": "object", + "additionalProperties": false, + "required": [ + "name" + ], + "properties": { + "name": { + "type": "string" + }, + "model": { + "type": "string", + "description": "Path/script identifier (dbt model, sql file, etc.)" + }, + "outputs": { + "type": "array", + "description": "List of exposeIds materialized by this step. VALIDATION: Tools must validate that every output exists in exposes[].exposeId.", + "items": { + "$ref": "#/$defs/identifier" + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Transformation step tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Transformation step labels." + } + } + }, + "lineage": { + "type": "object", + "additionalProperties": false, + "properties": { + "granularity": { + "type": "string", + "enum": [ + "table_level", + "field_level" + ], + "default": "table_level", + "description": "Level of lineage tracking granularity." + }, + "upstream": { + "type": "array", + "description": "Upstream dependencies and their field mappings.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "productId", + "exposeId" + ], + "properties": { + "productId": { + "$ref": "#/$defs/identifier", + "description": "Upstream data product ID." + }, + "exposeId": { + "$ref": "#/$defs/identifier", + "description": "Upstream expose ID being consumed." + }, + "fieldMappings": { + "type": "array", + "description": "Field-level lineage mappings (optional).", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "sourceField", + "targetField" + ], + "properties": { + "sourceField": { + "type": "string" + }, + "targetField": { + "type": "string" + }, + "transformation": { + "type": "object", + "additionalProperties": false, + "properties": { + "type": { + "type": "string", + "enum": [ + "direct_mapping", + "aggregation", + "calculation", + "lookup" + ] + }, + "expression": { + "type": "string", + "description": "Transformation logic or SQL expression." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Transformation tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Transformation labels." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Field mapping tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Field mapping labels." + } + } + } + }, + "relationship": { + "type": "string", + "enum": [ + "direct_consumption", + "lookup_enrichment", + "aggregation" + ], + "description": "Type of data relationship." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Upstream relationship tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Upstream relationship labels." + } + } + } + }, + "downstream": { + "type": "array", + "description": "Known downstream consumers (optional).", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "consumer": { + "$ref": "#/$defs/identifier", + "description": "Downstream data product ID." + }, + "impact": { + "type": "string", + "enum": [ + "critical", + "high", + "medium", + "low" + ], + "description": "Impact level if this data product changes." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Downstream relationship tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Downstream relationship labels." + } + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Lineage configuration tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Lineage configuration labels." + } + } + }, + "schemaEvolution": { + "type": "object", + "additionalProperties": false, + "properties": { + "strategy": { + "type": "string", + "enum": [ + "semantic_versioning", + "date_based", + "sequential" + ], + "description": "Versioning strategy for schema changes." + }, + "compatibility": { + "type": "string", + "enum": [ + "backward_compatible", + "forward_compatible", + "full_compatible", + "breaking" + ], + "description": "Compatibility approach for schema evolution." + }, + "changePolicy": { + "type": "object", + "additionalProperties": false, + "properties": { + "changeWindowDays": { + "type": "integer", + "minimum": 0, + "description": "Advance notice period for breaking changes." + }, + "approvalRequired": { + "type": "boolean", + "description": "Whether schema changes require approval." + }, + "approvers": { + "type": "array", + "items": { + "type": "string" + }, + "description": "List of approvers for schema changes." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Change policy tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Change policy labels." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Schema evolution tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Schema evolution labels." + } + } + }, + "machineLearning": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "default": false + }, + "framework": { + "type": "string", + "enum": [ + "mlflow", + "kubeflow", + "sagemaker", + "vertex_ai", + "custom" + ] + }, + "models": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "version", + "type" + ], + "properties": { + "name": { + "type": "string" + }, + "version": { + "$ref": "#/$defs/semver" + }, + "type": { + "type": "string", + "enum": [ + "classification", + "regression", + "clustering", + "recommendation", + "other" + ] + }, + "algorithm": { + "type": "string" + }, + "features": { + "type": "array", + "description": "List of feature names used by this model.", + "items": { + "type": "string" + } + }, + "target": { + "type": "string", + "description": "Target variable for supervised learning." + }, + "validation": { + "type": "object", + "additionalProperties": false, + "properties": { + "metrics": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Validation metrics (e.g., accuracy, auc_roc)." + }, + "thresholds": { + "type": "object", + "additionalProperties": { + "type": "number" + }, + "description": "Minimum acceptable metric values." + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Model validation tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Model validation labels." + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "ML model tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "ML model labels." + } + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "ML configuration tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "ML configuration labels." + } + } + }, + "environmentConfig": { + "type": "object", + "additionalProperties": false, + "properties": { + "metadata": { + "type": "object", + "description": "Environment-specific metadata overrides.", + "additionalProperties": false, + "properties": { + "owner": { + "type": "object", + "additionalProperties": false, + "properties": { + "team": { + "type": "string" + }, + "email": { + "type": "string" + }, + "slack": { + "type": "string" + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Environment-specific metadata tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Environment-specific metadata labels." + } + } + }, + "exposes": { + "type": "array", + "description": "Environment-specific expose overrides.", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "exposeId": { + "$ref": "#/$defs/identifier" + }, + "binding": { + "$ref": "#/$defs/binding" + }, + "qos": { + "$ref": "#/$defs/exposeQoS" + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Environment-specific expose tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Environment-specific expose labels." + } + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Environment configuration tags." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Environment configuration labels." + } + } + }, + "orchestration": { + "type": "object", + "required": [ + "engine" + ], + "properties": { + "engine": { + "type": "string", + "enum": [ + "airflow", + "dagster", + "prefect", + "kubeflow", + "custom", + "none" + ] + }, + "mode": { + "type": "string", + "enum": [ + "generated", + "manual", + "hybrid" + ], + "default": "generated" + }, + "generateOnChange": { + "type": "boolean", + "default": true + }, + "airflow": { + "$ref": "#/$defs/airflowOrchestration" + }, + "dagster": { + "type": "object" + }, + "prefect": { + "type": "object" + } + } + }, + "airflowOrchestration": { + "type": "object", + "required": [ + "dagId" + ], + "properties": { + "dagId": { + "type": "string", + "pattern": "^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$" + }, + "dagConfig": { + "type": "object", + "properties": { + "description": { + "type": "string" + }, + "schedule": { + "type": "string" + }, + "startDate": { + "type": "string" + }, + "endDate": { + "type": "string" + }, + "catchup": { + "type": "boolean", + "default": false + }, + "maxActiveRuns": { + "type": "integer", + "minimum": 1 + }, + "dagRunTimeout": { + "$ref": "#/$defs/isoDuration" + }, + "defaultArgs": { + "type": "object" + }, + "tags": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Airflow DAG tags (different from FLUID tags)" + }, + "onSuccessCallback": { + "type": "string" + }, + "onFailureCallback": { + "type": "string" + }, + "slaCallback": { + "type": "string" + } + } + }, + "taskDefaults": { + "type": "object", + "description": "Default configuration for all tasks (can be overridden per task)", + "properties": { + "retries": { + "type": "integer", + "minimum": 0 + }, + "retryDelay": { + "$ref": "#/$defs/isoDuration" + }, + "executionTimeout": { + "$ref": "#/$defs/isoDuration" + }, + "pool": { + "type": "string" + }, + "queue": { + "type": "string" + }, + "priority": { + "type": "integer" + }, + "triggerRule": { + "type": "string", + "enum": [ + "all_success", + "all_failed", + "all_done", + "one_success", + "one_failed", + "none_failed", + "none_skipped" + ] + }, + "errorHandling": { + "type": "object", + "description": "NEW in v0.7.0: Enhanced error categorization and handling", + "properties": { + "strategy": { + "type": "string", + "enum": [ + "fixed", + "exponential_backoff", + "linear_backoff" + ], + "default": "exponential_backoff" + }, + "retryableErrors": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Error types that should trigger retries", + "examples": [ + [ + "TemporaryFailure", + "RateLimitExceeded", + "ConnectionTimeout", + "DeadlineExceeded" + ] + ] + }, + "nonRetryableErrors": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Error types that should fail immediately", + "examples": [ + [ + "ValidationError", + "SchemaViolation", + "PermissionDenied", + "InvalidCredentials" + ] + ] + }, + "onError": { + "type": "array", + "items": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": [ + "notify", + "create_incident", + "log", + "custom" + ] + }, + "channel": { + "type": "string" + }, + "severity": { + "type": "string", + "enum": [ + "low", + "medium", + "high", + "critical" + ] + }, + "when": { + "type": "string", + "enum": [ + "always", + "maxAttemptsExceeded", + "nonRetryableError" + ] + } + } + } + } + } + } + } + }, + "tasks": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/airflowTask" + } + }, + "sensors": { + "type": "array", + "items": { + "$ref": "#/$defs/airflowSensor" + } + }, + "taskGroups": { + "type": "array", + "items": { + "type": "object", + "required": [ + "groupId" + ], + "properties": { + "groupId": { + "type": "string" + }, + "description": { + "type": "string" + }, + "tasks": { + "type": "array", + "items": { + "$ref": "#/$defs/airflowTask" + } + } + } + } + } + } + }, + "airflowTask": { + "type": "object", + "required": [ + "taskId" + ], + "properties": { + "taskId": { + "type": "string", + "pattern": "^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$" + }, + "type": { + "type": "string", + "description": "NEW in v0.7.0: Simplified task type taxonomy. Replaces 'operator' field (backward compatible). Types: provider_action (NEW!), fluid_validate, fluid_plan, fluid_apply, fluid_execute, fluid_verify, fluid_dq_check, bash, python, sensor, or provider-specific operators.", + "enum": [ + "provider_action", + "fluid_validate", + "fluid_plan", + "fluid_apply", + "fluid_execute", + "fluid_verify", + "fluid_dq_check", + "bash", + "python", + "branch_python", + "sensor", + "email", + "http", + "snowflake_query", + "bigquery_query", + "bigquery_job", + "glue_job", + "databricks_job", + "custom" + ] + }, + "operator": { + "type": "string", + "description": "DEPRECATED in v0.7.0: Use 'type' instead. Maintained for backward compatibility with v0.6.1 contracts. Maps to 'type' automatically (FluidExecuteOperator → fluid_execute, BashOperator → bash)." + }, + "provider": { + "type": "string", + "description": "NEW in v0.7.0: Provider name when type=provider_action. Examples: aws, gcp, azure, snowflake, databricks.", + "enum": [ + "aws", + "gcp", + "azure", + "snowflake", + "databricks", + "kafka", + "kubernetes", + "local", + "custom" + ] + }, + "action": { + "type": "string", + "pattern": "^[a-z0-9_]+\\.[a-z0-9_]+$", + "description": "NEW in v0.7.0: Provider action when type=provider_action. Format: service.operation (e.g., s3.ensure_bucket, table.ensure, kinesis.create_stream).", + "examples": [ + "s3.ensure_bucket", + "table.ensure", + "database.ensure", + "kinesis.create_stream", + "glue.create_job", + "bigquery.create_table" + ] + }, + "buildStepRef": { + "type": "string", + "description": "NEW in v0.7.0: Reference to transformation step in builds[].transformations[]. Links orchestration tasks to build steps for clearer mapping." + }, + "dependencies": { + "type": "array", + "items": { + "type": "string" + } + }, + "params": { + "type": "object", + "additionalProperties": true + }, + "produces": { + "type": "array", + "description": "NEW in v0.7.0: Declares what exposes[] this task produces. Enables automatic dataset registration and lineage tracking.", + "items": { + "type": "object", + "required": [ + "exposeId" + ], + "properties": { + "exposeId": { + "$ref": "#/$defs/identifier", + "description": "The exposeId from exposes[] that this task produces" + }, + "dataset": { + "type": "string", + "description": "Optional: Airflow dataset URI. Auto-generated from binding.location if not specified." + }, + "updateStrategy": { + "type": "string", + "enum": [ + "replace", + "append", + "merge", + "upsert" + ], + "default": "replace", + "description": "How this task updates the data" + } + } + } + }, + "dataProductDependencies": { + "type": "array", + "description": "NEW in v0.7.0: Tasks can declare dependencies on other data products with version constraints and freshness requirements.", + "items": { + "type": "object", + "required": [ + "productId", + "exposeId" + ], + "properties": { + "productId": { + "$ref": "#/$defs/identifier", + "description": "Data product ID to depend on" + }, + "exposeId": { + "$ref": "#/$defs/identifier", + "description": "Specific expose within the product" + }, + "versionConstraint": { + "$ref": "#/$defs/semverRange", + "description": "Required version (e.g., '^2.0.0', '>=1.5.0')" + }, + "awaitFreshness": { + "$ref": "#/$defs/isoDuration", + "description": "Maximum age of data to wait for (e.g., PT6H = 6 hours)" + } + } + } + }, + "overrides": { + "type": "object", + "description": "Task-specific overrides for taskDefaults" + }, + "doc": { + "type": "string" + }, + "docMd": { + "type": "string" + }, + "uiColor": { + "type": "string" + }, + "uiFgcolor": { + "type": "string" + }, + "xcom": { + "type": "object", + "description": "XCom configuration for inter-task communication (from v0.6.1)", + "properties": { + "push": { + "type": "array", + "items": { + "type": "object", + "properties": { + "key": { + "type": "string" + }, + "value": {}, + "serializer": { + "type": "string", + "enum": [ + "json", + "pickle", + "cloudpickle", + "custom" + ] + } + } + } + }, + "pull": { + "type": "array", + "items": { + "type": "object", + "properties": { + "key": { + "type": "string" + }, + "task_id": { + "type": "string" + }, + "dag_id": { + "type": "string" + }, + "map_indexes": { + "type": "array" + } + } + } + } + } + }, + "accessControl": { + "type": "object", + "description": "Task-level access control (from v0.6.1)", + "properties": { + "requiredRoles": { + "type": "array", + "items": { + "type": "string" + } + }, + "approvalRequired": { + "type": "boolean" + }, + "approvers": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "costEstimate": { + "type": "object", + "description": "Cost tracking (enhanced in v0.7.0)", + "properties": { + "compute": { + "type": "object", + "properties": { + "units": { + "type": "number" + }, + "pricePerUnit": { + "type": "number" + }, + "currency": { + "type": "string", + "default": "USD" + } + } + }, + "storage": { + "type": "object", + "properties": { + "units": { + "type": "number" + }, + "pricePerUnit": { + "type": "number" + } + } + }, + "total": { + "type": "number" + }, + "trackActual": { + "type": "boolean", + "default": false, + "description": "NEW in v0.7.0: Track actual costs vs estimate" + }, + "alertThreshold": { + "type": "number", + "description": "NEW in v0.7.0: Alert if actual > threshold * estimate (e.g., 1.5 = 150%)" + }, + "budget": { + "type": "object", + "description": "NEW in v0.7.0: Budget enforcement", + "properties": { + "monthly": { + "type": "number" + }, + "stopOnExceed": { + "type": "boolean", + "default": false + }, + "alertOnExceed": { + "type": "boolean", + "default": true + } + } + } + } + } + }, + "allOf": [ + { + "description": "Backward compatibility: If 'operator' is specified but not 'type', map operator to type", + "if": { + "required": [ + "operator" + ], + "not": { + "required": [ + "type" + ] + } + }, + "then": { + "description": "Operator field is deprecated but still works. Maps: FluidExecuteOperator→fluid_execute, BashOperator→bash, etc." + } + }, + { + "description": "NEW in v0.7.0: Validate provider_action tasks have required fields", + "if": { + "properties": { + "type": { + "const": "provider_action" + } + }, + "required": [ + "type" + ] + }, + "then": { + "required": [ + "provider", + "action", + "params" + ], + "errorMessage": "Tasks with type=provider_action must specify provider, action, and params" + } + }, + { + "description": "Validate fluid_execute tasks", + "if": { + "anyOf": [ + { + "properties": { + "type": { + "const": "fluid_execute" + } + } + }, + { + "properties": { + "operator": { + "pattern": "FluidExecute" + } + } + } + ] + }, + "then": { + "properties": { + "params": { + "type": "object", + "required": [ + "contract_path" + ], + "properties": { + "contract_path": { + "type": "string" + }, + "build_id": { + "type": "string" + } + } + } + } + } + }, + { + "description": "Validate bash tasks", + "if": { + "anyOf": [ + { + "properties": { + "type": { + "const": "bash" + } + } + }, + { + "properties": { + "operator": { + "pattern": "Bash" + } + } + } + ] + }, + "then": { + "properties": { + "params": { + "anyOf": [ + { + "required": [ + "bash_command" + ] + }, + { + "required": [ + "command" + ] + } + ] + } + } + } + }, + { + "description": "Validate python tasks", + "if": { + "anyOf": [ + { + "properties": { + "type": { + "const": "python" + } + } + }, + { + "properties": { + "operator": { + "pattern": "Python" + } + } + } + ] + }, + "then": { + "properties": { + "params": { + "anyOf": [ + { + "required": [ + "python_callable" + ] + }, + { + "required": [ + "callable" + ] + } + ] + } + } + } + } + ] + }, + "airflowSensor": { + "type": "object", + "required": [ + "taskId", + "sensorType" + ], + "properties": { + "taskId": { + "type": "string" + }, + "sensorType": { + "type": "string", + "enum": [ + "FileSensor", + "ExternalTaskSensor", + "TimeSensor", + "HttpSensor", + "S3KeySensor", + "GCSObjectSensor", + "custom" + ] + }, + "params": { + "type": "object" + }, + "pokInterval": { + "type": "integer", + "minimum": 1, + "default": 60 + }, + "timeout": { + "type": "integer", + "minimum": 1, + "default": 3600 + }, + "mode": { + "type": "string", + "enum": [ + "poke", + "reschedule" + ], + "default": "poke" + }, + "dependencies": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "semanticModel": { + "type": "object", + "description": "NEW in v0.7.2: Semantic model definition for an exposed interface. Maps physical columns to business concepts (entities, measures, dimensions, metrics) enabling AI agents and BI tools to generate correct queries without semantic hallucination. Compatible with dbt MetricFlow, Snowflake Semantic Views, and OSI-aligned formats.", + "additionalProperties": false, + "properties": { + "name": { + "type": "string", + "description": "Human-readable name for this semantic model." + }, + "description": { + "type": "string", + "description": "Business context explaining what this semantic model represents." + }, + "defaultAggTimeDimension": { + "type": "string", + "description": "Default time dimension used for measure aggregation. Can be overridden per-measure." + }, + "entities": { + "type": "array", + "description": "Join keys with type annotations. Entities connect this table to others in the semantic graph.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "type" + ], + "properties": { + "name": { + "type": "string", + "description": "Entity name (e.g., 'order', 'customer')." + }, + "type": { + "type": "string", + "enum": [ + "primary", + "foreign", + "unique", + "natural" + ], + "description": "Entity key type: primary (unique, complete), foreign (references another table), unique (unique, possibly partial), natural (real-world identifier)." + }, + "expr": { + "type": "string", + "description": "Column expression. Defaults to entity name if omitted." + }, + "description": { + "type": "string" + } + } + } + }, + "measures": { + "type": "array", + "description": "Aggregatable column expressions. Measures are the atomic building blocks from which metrics are composed.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "agg" + ], + "properties": { + "name": { + "type": "string", + "description": "Measure name (e.g., 'total_amount', 'order_count')." + }, + "description": { + "type": "string" + }, + "agg": { + "type": "string", + "enum": [ + "sum", + "avg", + "count", + "count_distinct", + "min", + "max", + "median", + "percentile" + ], + "description": "Aggregation function applied to the expression." + }, + "expr": { + "type": "string", + "description": "SQL expression to aggregate. Defaults to measure name if omitted." + }, + "aggParams": { + "type": "object", + "additionalProperties": false, + "description": "Aggregation parameters (mirrors dbt-semantic-interfaces agg_params). Currently used by agg=percentile.", + "properties": { + "percentile": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Percentile in [0, 1] for agg=percentile (0.5 = median). Consumers default to 0.5 when omitted." + }, + "useDiscretePercentile": { + "type": "boolean", + "description": "Use the discrete percentile (PERCENTILE_DISC / use_discrete_percentile) instead of continuous interpolation." + } + } + }, + "aggTimeDimension": { + "type": "string", + "description": "Override the model-level default time dimension for this measure." + }, + "nonAdditiveDimension": { + "type": "object", + "additionalProperties": false, + "description": "Dimension over which this measure cannot be naively aggregated (e.g., account balances).", + "properties": { + "name": { + "type": "string" + }, + "windowChoice": { + "type": "string", + "enum": [ + "min", + "max" + ], + "description": "Which value to pick when the non-additive dimension is present." + } + } + }, + "createMetric": { + "type": "boolean", + "default": false, + "description": "If true, automatically create a simple metric from this measure." + } + } + } + }, + "dimensions": { + "type": "array", + "description": "Categorical and time-based grouping axes. Dimensions define how data can be sliced in queries.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "type" + ], + "properties": { + "name": { + "type": "string", + "description": "Dimension name (e.g., 'order_date', 'region', 'product_category')." + }, + "type": { + "type": "string", + "enum": [ + "categorical", + "time" + ], + "description": "Dimension type: categorical (text/enum grouping) or time (temporal axis)." + }, + "expr": { + "type": "string", + "description": "SQL expression. Defaults to dimension name if omitted." + }, + "description": { + "type": "string" + }, + "typeParams": { + "type": "object", + "additionalProperties": false, + "description": "Type-specific parameters.", + "properties": { + "timeGranularity": { + "type": "string", + "enum": [ + "day", + "week", + "month", + "quarter", + "year", + "hour", + "minute" + ], + "description": "Default time granularity for this dimension." + } + } + } + } + } + }, + "metrics": { + "type": "array", + "description": "Composable KPI definitions built from measures. Metrics are the named business calculations that agents and tools resolve.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "type" + ], + "properties": { + "name": { + "type": "string", + "description": "Metric name (e.g., 'net_revenue', 'customer_churn_rate')." + }, + "description": { + "type": "string", + "description": "Business definition of this metric." + }, + "type": { + "type": "string", + "enum": [ + "simple", + "derived", + "ratio" + ], + "description": "Metric type: simple (single measure + filter), derived (expression over other metrics), ratio (numerator/denominator)." + }, + "measure": { + "type": "string", + "description": "Reference to a measure name (for simple metrics)." + }, + "filter": { + "type": "string", + "description": "SQL WHERE clause filter (e.g., \"status = 'completed' AND is_refunded = FALSE\")." + }, + "inputMetrics": { + "type": "array", + "items": { + "type": "string" + }, + "description": "References to other metric names (for derived/ratio metrics)." + }, + "expr": { + "type": "string", + "description": "Mathematical expression combining input metrics (for derived metrics)." + }, + "numerator": { + "type": "string", + "description": "Numerator metric name (for ratio metrics)." + }, + "denominator": { + "type": "string", + "description": "Denominator metric name (for ratio metrics)." + }, + "owner": { + "type": "string", + "description": "Business owner of this metric definition." + } + } + } + }, + "tags": { + "$ref": "#/$defs/tags", + "description": "Semantic model tags for discovery and categorization." + }, + "labels": { + "$ref": "#/$defs/labels", + "description": "Semantic model labels for automation and metadata." + } + }, + "examples": [ + { + "name": "orders_revenue_model", + "description": "Canonical semantic model for revenue analytics", + "defaultAggTimeDimension": "order_date", + "entities": [ + { + "name": "order", + "type": "primary", + "expr": "order_id" + }, + { + "name": "customer", + "type": "foreign", + "expr": "customer_id" + } + ], + "measures": [ + { + "name": "order_amount", + "agg": "sum", + "expr": "amount", + "description": "Sum of all order amounts" + }, + { + "name": "order_count", + "agg": "count_distinct", + "expr": "order_id", + "createMetric": true + } + ], + "dimensions": [ + { + "name": "order_date", + "type": "time", + "expr": "completed_at", + "typeParams": { + "timeGranularity": "day" + } + }, + { + "name": "region", + "type": "categorical" + } + ], + "metrics": [ + { + "name": "net_revenue", + "description": "GAAP-compliant net revenue: completed orders minus discounts, excluding refunds", + "type": "derived", + "inputMetrics": [ + "gross_revenue", + "total_discounts" + ], + "expr": "gross_revenue - total_discounts", + "owner": "finance-data@company.com" + }, + { + "name": "gross_revenue", + "description": "Total value of completed orders", + "type": "simple", + "measure": "order_amount", + "filter": "status = 'completed'" + } + ] + } + ] + }, + "exposeMcp": { + "type": "object", + "additionalProperties": false, + "description": "NEW in v0.7.4: Per-expose overrides for the MCP output-port server — sampling caps + classification. The presence of this (non-empty) block opts the expose into the Fluid MCP gateway, where policy.agentPolicy is enforced at every read.", + "properties": { + "sampling": { + "type": "object", + "additionalProperties": false, + "properties": { + "maxRows": { + "type": "integer", + "minimum": 1, + "description": "Hard cap for the `sample` MCP tool. Defends against an over-curious agent asking for petabyte-scale row counts. Default: 100." + } + } + }, + "classification": { + "type": "object", + "additionalProperties": false, + "properties": { + "dataClass": { + "type": "string", + "enum": [ + "public", + "internal", + "confidential", + "restricted" + ], + "description": "Surfaced on the `describe` tool so consumer agents can declare downstream handling. Advisory; the real gate is `policy.agentPolicy`." + } + } + } + } + }, + "governance": { + "type": "object", + "description": "Account/project-wide governance settings. Today only the lakeFormation block is supported; future cloud-specific blocks may live alongside it (e.g. snowflake masking-policy registries, gcp organization-policies).", + "additionalProperties": false, + "properties": { + "lakeFormation": { + "type": "object", + "additionalProperties": false, + "description": "AWS Lake Formation account-wide settings: data-lake admins and LF-tag (TBAC) definitions. Applies to the AWS account the contract's IaC apply targets.", + "properties": { + "admins": { + "type": "array", + "description": "IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. AUTHORITATIVE, not additive: Lake Formation's PutDataLakeSettings replaces the account's admin list with the one sent, so an admin NOT listed here is REMOVED on apply. List every principal that must stay a data lake admin, including the role or user that runs the apply: left out, it loses Lake Formation admin on that apply, and the Lake Formation grants and registrations that need an admin can then fail. The terraform-provider-aws resource also clears the data lake settings its configuration omits, and this block sets only the admins, so the account's create-database and create-table default permissions (the IAMAllowedPrincipals default for new databases and tables), trusted resource owners and parameters are cleared too. Destroying the resource empties the admin list and resets CROSS_ACCOUNT_VERSION to 1. Re-applying the same list is a no-op.", + "items": { + "type": "string", + "pattern": "^arn:aws[a-z0-9-]*:iam::" + }, + "uniqueItems": true + }, + "tagDefinitions": { + "type": "object", + "description": "LF-TBAC tag-key definitions. Each entry is a tag key mapped to its allowed string values. Emitted as one aws_lakeformation_lf_tag per key; values are the tag's allowed-values list. Field idiom borrowed from OpenMetadata's classification-label model — see ODCS roles + OpenMetadata data contracts spec.", + "additionalProperties": { + "type": "array", + "items": { + "type": "string" + }, + "uniqueItems": true, + "minItems": 1 + } + } + } + } + } + }, + "bindingGovernance": { + "type": "object", + "description": "Per-resource (per-exposure) governance. Currently scoped to AWS Lake Formation; future cloud-specific blocks may sit alongside (e.g. gcpIamPolicy, snowflakeMaskingPolicy).", + "additionalProperties": false, + "properties": { + "lakeFormation": { + "type": "object", + "additionalProperties": false, + "description": "Per-resource LF settings: location registration, principal grants, LF-tag associations, row/column filters. Applies only to formats backed by S3 + Glue catalog (iceberg/parquet/csv/json/avro/orc/delta).", + "properties": { + "registerLocation": { + "type": "boolean", + "default": false, + "description": "When true, emit aws_lakeformation_resource that registers the binding's S3 path with Lake Formation. Required before LF can manage access to objects under that path." + }, + "bucketPolicy": { + "type": "string", + "enum": [ + "cross-account", + "none", + "all-grantees" + ], + "default": "cross-account", + "description": "NEW in v0.7.6: Which grants[] principals get a statement in the aws_s3_bucket_policy emitted beside the grants (s3:ListBucket and s3:GetBucketLocation on the bucket, s3:GetObject under location.path). A statement lets its principal read the objects straight from S3, which skips Lake Formation's column, row and cell filters and its revocations. 'cross-account' (the default): only principals whose ARN names an AWS account other than the one running the apply, decided at plan time against the caller identity; same-account grantees get no statement, since on a registered location Lake Formation vends them credentials. 'none': no bucket policy at all; the tightest setting when the location is registered (registerLocation) and every reader, cross-account ones included, queries through a Lake Formation integrated engine, or when the bucket policy is managed elsewhere. 'all-grantees': every grantee, same-account included, which is the output before this field existed and reopens the direct S3 read path. An emitted aws_s3_bucket_policy is authoritative: it replaces every other statement on the bucket. When no grantee is left none is emitted." + }, + "grants": { + "type": "array", + "description": "Principal-based LF grants on this exposure's database/table. Each entry maps a principal to a permission set (and optional column projection). Emitted as aws_lakeformation_permissions resources. Field idiom adapted from ODCS v3 roles[] (principal + access-type), bound to IAM ARNs since LF is cloud-specific.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "principal", + "permissions" + ], + "properties": { + "principal": { + "type": "string", + "description": "IAM principal ARN (role or user) receiving the grant.", + "pattern": "^arn:aws[a-z0-9-]*:iam::" + }, + "permissions": { + "type": "array", + "description": "LF permissions granted. Valid values per the hashicorp/aws aws_lakeformation_permissions resource.", + "items": { + "type": "string", + "enum": [ + "ALL", + "SELECT", + "ALTER", + "DROP", + "DELETE", + "INSERT", + "DESCRIBE", + "CREATE_TABLE", + "CREATE_DATABASE", + "DATA_LOCATION_ACCESS" + ] + }, + "uniqueItems": true, + "minItems": 1 + }, + "permissionsWithGrantOption": { + "type": "array", + "description": "Subset of permissions the principal can re-grant. Maps to aws_lakeformation_permissions.permissions_with_grant_option.", + "items": { + "type": "string" + }, + "uniqueItems": true + }, + "columns": { + "type": "array", + "description": "Optional column-level restriction. When set, the grant is emitted against aws_lakeformation_permissions.table_with_columns instead of .table, allowing the principal to read only the listed columns.", + "items": { + "type": "string" + }, + "uniqueItems": true + }, + "excludedColumns": { + "type": "array", + "description": "Opposite of columns: every column EXCEPT these is granted. Maps to table_with_columns.excluded_column_names. Mutually exclusive with columns.", + "items": { + "type": "string" + }, + "uniqueItems": true + } + } + } + }, + "tags": { + "type": "object", + "description": "LF-tag (TBAC) associations applied to this table. Keys must reference tag definitions declared in the contract's top-level governance.lakeFormation.tagDefinitions. Emitted as aws_lakeformation_resource_lf_tags.", + "additionalProperties": { + "type": "string" + } + }, + "rowFilter": { + "type": "object", + "additionalProperties": false, + "description": "Row-level (and optionally column-level) filter applied to all reads of this table via LF. Emitted as aws_lakeformation_data_cells_filter. The filter must then be granted to principals via grants[].permissions including SELECT — LF enforces both layers.", + "properties": { + "name": { + "type": "string", + "description": "Filter name (must be unique per table)." + }, + "rowExpression": { + "type": "string", + "description": "PartiQL-flavoured row predicate. ``ALL`` lets every row through (use when only column filtering is intended)." + }, + "columnNames": { + "type": "array", + "description": "Columns visible through the filter. Mutually exclusive with allColumns / excludedColumnNames.", + "items": { + "type": "string" + }, + "uniqueItems": true + }, + "excludedColumnNames": { + "type": "array", + "description": "Columns hidden through the filter (everything else is visible).", + "items": { + "type": "string" + }, + "uniqueItems": true + }, + "allColumns": { + "type": "boolean", + "default": false, + "description": "Convenience for the LF column-wildcard form (every column visible). When true, columnNames / excludedColumnNames must be omitted." + } + }, + "required": [ + "name", + "rowExpression" + ] + } + } + } + } + }, + "consumerRef": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "type" + ], + "$comment": "dbt-exposures-shaped (name/label/type/owner/url/maturity/description); exposeIds is the FLUID-native refinement tying a consumer to specific output ports.", + "properties": { + "name": { + "$ref": "#/$defs/identifier", + "description": "Stable identifier for this consumer (unique within the contract)." + }, + "label": { + "type": "string", + "description": "Human-facing display name, e.g. \"Weekly Revenue Dashboard\"." + }, + "type": { + "type": "string", + "enum": [ + "dashboard", + "notebook", + "analysis", + "ml", + "application" + ], + "description": "What kind of artifact consumes this product." + }, + "owner": { + "type": "object", + "additionalProperties": false, + "$comment": "Mirrors metadata.owner's vocabulary so 'owner' means one thing across the standard: team is the routable identity, email the contact. A dbt exposure's owner.name maps to team on import.", + "properties": { + "team": { + "type": "string", + "description": "Owning team for this consumer (routable identity, as in metadata.owner)." + }, + "email": { + "type": "string", + "format": "email", + "description": "Contact address for the consumer's owner." + } + } + }, + "url": { + "type": "string", + "description": "Link to the live artifact (dashboard URL, notebook, app)." + }, + "maturity": { + "type": "string", + "enum": [ + "high", + "medium", + "low" + ], + "description": "How production-hardened the consumer is (dbt exposures semantics)." + }, + "description": { + "type": "string" + }, + "exposeIds": { + "type": "array", + "items": { + "$ref": "#/$defs/identifier" + }, + "description": "Which output ports this consumer reads. Absent = all exposes." + } + } + } + } +} \ No newline at end of file diff --git a/scripts/check-compat.py b/scripts/check-compat.py index 2f67969..71c55e3 100644 --- a/scripts/check-compat.py +++ b/scripts/check-compat.py @@ -22,7 +22,8 @@ pass exists alongside it. Usage: - python3 scripts/check-compat.py # every adjacent pair + python3 scripts/check-compat.py # every adjacent pair from 0.7.1 on + python3 scripts/check-compat.py --all-history # ... and the pre-promise pairs too python3 scripts/check-compat.py --from 0.7.4 --to 0.7.5 python3 scripts/check-compat.py --format json @@ -53,6 +54,12 @@ VERSION_RE = re.compile(r"^fluid-schema-(\d+\.\d+\.\d+)\.json$") +# The first version the compatibility promise covers. Earlier schemas predate +# it, and 0.4.0 -> 0.5.7 is a large, deliberate break (accessPolicy, governance, +# operations, security and slo removed), so a default run starting at 0.0.1 +# would exit 1 on history no one can change. --all-history still reports it. +PROMISE_FLOOR = "0.7.1" + # Keywords whose numeric value narrowing the instance set is breaking. LOWER_BOUNDS = ("minimum", "exclusiveMinimum", "minItems", "minLength", "minProperties") UPPER_BOUNDS = ("maximum", "exclusiveMaximum", "maxItems", "maxLength", "maxProperties") @@ -400,12 +407,20 @@ def main(argv: Optional[Sequence[str]] = None) -> int: ap.add_argument( "--show-additive", action="store_true", help="also list additive changes" ) + ap.add_argument( + "--all-history", + action="store_true", + help=f"also compare the pairs before {PROMISE_FLOOR}, which predate the promise", + ) args = ap.parse_args(argv) if bool(args.old) != bool(args.new): ap.error("--from and --to must be given together") versions = published_versions() + if not args.all_history: + floor = tuple(int(p) for p in PROMISE_FLOOR.split(".")) + versions = [v for v in versions if tuple(int(p) for p in v.split(".")) >= floor] pairs = [(args.old, args.new)] if args.old else adjacent_pairs(versions) if not pairs: print("check-compat: fewer than two published schemas; nothing to compare") diff --git a/scripts/check-schema-drift.py b/scripts/check-schema-drift.py new file mode 100644 index 0000000..32e3000 --- /dev/null +++ b/scripts/check-schema-drift.py @@ -0,0 +1,432 @@ +#!/usr/bin/env python3 +"""Check that the vendored schemas equal the reference implementation's release. + + python3 scripts/check-schema-drift.py # the PINNED release + python3 scripts/check-schema-drift.py --latest # advisory: is there a newer one? + python3 scripts/check-schema-drift.py --ref v0.18.1 # an arbitrary tag (re-vendoring) + python3 scripts/check-schema-drift.py --source-dir DIR # offline: a directory + # holding fluid-schema-*.json + # (and, optionally, + # schema_manager.py) + +The FLUID schemas are authored in forge-cli and vendored into schema/ byte for +byte. This script is the check that they still match; it lands nothing. It +compares against a RELEASE, not forge-cli's main branch: a release is what +`pip install data-product-forge` gives people, while main carries unreleased +schema edits that would turn this check red before anyone could validate a +contract against them. + +WHICH release is recorded in scripts/schema-versions.json ("upstream.ref" and +"upstream.commit"), not looked up at run time. The default run is therefore +deterministic: it compares the working tree with that one commit and nothing +upstream can change the result. It fetches by the commit, which cannot move, +and only reports (as a warning) if the tag now points somewhere else. A new +forge-cli release does not turn a pull request red; moving to it is a change +to those two fields, in a pull request, where a reviewer sees which upstream +commit is being trusted. `--latest` is the separate, advisory question "has +forge-cli released since?"; it exits 1 when it has, and CI runs it as a +non-blocking job. + +What the default run checks, against scripts/schema-versions.json: + + 1. every version listed in "synced" is byte-identical to the release's copy; + 2. the release bundles no version, at or above the oldest synced one, that + is missing from "synced" -- so a new schema cannot ship unpublished; + 3. the release's preview set (PREVIEW_VERSIONS in schema_manager.py) equals + "preview", and "latestStable" is the release's highest non-preview + version -- so a promotion to stable is not missed. + +Exit 0 when everything matches; 1 on drift (with --latest: when a newer +release exists); 2 when the release cannot be read or the pin is malformed. The +output says how to re-vendor. + +Set GITHUB_TOKEN to avoid the unauthenticated API rate limit; it is sent only +to api.github.com and is never forwarded across a redirect. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sys +import urllib.error +import urllib.request +from pathlib import Path +from typing import Dict, List, Optional, Set, Tuple + +REPO = Path(__file__).resolve().parent.parent +SCHEMA_DIR = REPO / "schema" +VERSIONS_FILE = REPO / "scripts" / "schema-versions.json" +SCHEMA_RE = re.compile(r"^fluid-schema-(\d+\.\d+\.\d+)\.json$") +PREVIEW_RE = re.compile(r"PREVIEW_VERSIONS\b[^=\n]*=\s*frozenset\(\s*\{([^}]*)\}\s*\)") + +# A release tag, and nothing else. A ref is interpolated into URLs and into the +# re-vendor hint this script prints (which people paste into a shell), so a +# value read from the network or from a file is checked before it is used. +REF_RE = re.compile(r"v?\d+\.\d+\.\d+([.+-][0-9A-Za-z.]+)?") +COMMIT_RE = re.compile(r"[0-9a-f]{40}") +REPOSITORY_RE = re.compile(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+") +REPO_PATH_RE = re.compile(r"[A-Za-z0-9_.][A-Za-z0-9_./-]*") + + +class SourceError(RuntimeError): + pass + + +def vkey(v: str): + return tuple(int(p) for p in v.split(".")) + + +def validate_ref(ref: object, what: str) -> str: + """Return `ref` if it is a release tag; otherwise fail closed.""" + if not isinstance(ref, str) or not REF_RE.fullmatch(ref): + raise SourceError(f"{what} is not a release tag (expected something like v1.2.3): {ref!r}") + return ref + + +def validate_commit(commit: object, what: str) -> str: + if not isinstance(commit, str) or not COMMIT_RE.fullmatch(commit): + raise SourceError(f"{what} is not a full 40-character lowercase commit SHA: {commit!r}") + return commit + + +def validate_repo_fields(repository: object, schema_dir: object, preview_source: object) -> None: + if not isinstance(repository, str) or not REPOSITORY_RE.fullmatch(repository): + raise SourceError(f"upstream.repository is not an owner/name pair: {repository!r}") + for name, value in (("upstream.schemaDir", schema_dir), ("upstream.previewSource", preview_source)): + if not isinstance(value, str) or not REPO_PATH_RE.fullmatch(value) or ".." in value.split("/"): + raise SourceError(f"{name} is not a plain repository path: {value!r}") + + +# -------------------------------------------------------------------------- +# sources +# -------------------------------------------------------------------------- + + +def _get(url: str, accept: Optional[str] = None) -> bytes: + req = urllib.request.Request(url, headers={"User-Agent": "fluid-schema-drift-check"}) + if accept: + req.add_header("Accept", accept) + token = os.environ.get("GITHUB_TOKEN") + if token and url.startswith("https://api.github.com/"): + # Unredirected: urllib copies ordinary request headers onto a redirected + # request, so a redirect away from api.github.com would carry the token + # with it. Unredirected headers are sent to the original host only. + req.add_unredirected_header("Authorization", f"Bearer {token}") + try: + with urllib.request.urlopen(req, timeout=30) as r: + return r.read() + except (urllib.error.URLError, TimeoutError) as exc: + raise SourceError(f"could not fetch {url}: {exc}") from exc + + +def latest_release_tag(repository: str) -> str: + data = json.loads(_get(f"https://api.github.com/repos/{repository}/releases/latest")) + tag = data.get("tag_name") + if not tag: + raise SourceError(f"{repository} reports no latest release") + return validate_ref(tag, f"{repository}'s latest release tag") + + +class GitHubRelease: + """A forge-cli release, read from GitHub. + + With `commit`, every file is fetched by that commit rather than by the tag, + so the content compared is immutable; `ref` is then only the human-readable + name for it. + """ + + def __init__( + self, + repository: str, + schema_dir: str, + preview_source: str, + ref: str, + commit: Optional[str] = None, + ): + validate_repo_fields(repository, schema_dir, preview_source) + self.repository = repository + self.schema_dir = schema_dir + self.preview_source = preview_source + self.ref = validate_ref(ref, "the release ref") + self.commit = validate_commit(commit, "the pinned commit") if commit else None + self.fetch_ref = self.commit or self.ref + self.label = f"{repository}@{self.ref}" + (f" (commit {self.commit[:12]})" if self.commit else "") + + def tag_warning(self) -> Optional[str]: + """A warning when the tag no longer points at the pinned commit. + + Not a failure: the content compared is fetched by commit, so a moved + tag changes nothing this script decides. It does mean the recorded tag + name no longer describes the recorded commit, which a person should look at. + """ + if not self.commit: + return None + try: + now = _get( + f"https://api.github.com/repos/{self.repository}/commits/{self.ref}", + accept="application/vnd.github.sha", + ).decode("ascii", "replace").strip() + except SourceError as exc: + return f"could not confirm that tag {self.ref} still points at {self.commit}: {exc}" + if now != self.commit: + return ( + f"tag {self.ref} now points at {now[:12]}, but {VERSIONS_FILE.name} pins {self.commit[:12]}; " + "the comparison used the pinned commit, but the tag has been moved or the pin is wrong" + ) + return None + + def _raw(self, path: str) -> bytes: + return _get(f"https://raw.githubusercontent.com/{self.repository}/{self.fetch_ref}/{path}") + + def versions(self) -> List[str]: + listing = json.loads( + _get(f"https://api.github.com/repos/{self.repository}/contents/{self.schema_dir}?ref={self.fetch_ref}") + ) + names = [entry["name"] for entry in listing if entry.get("type") == "file"] + return sorted((m.group(1) for n in names if (m := SCHEMA_RE.match(n))), key=vkey) + + def schema(self, version: str) -> bytes: + return self._raw(f"{self.schema_dir}/fluid-schema-{version}.json") + + def preview_text(self) -> Optional[str]: + return self._raw(self.preview_source).decode("utf-8") + + def url(self, version: str) -> str: + return f"https://raw.githubusercontent.com/{self.repository}/{self.fetch_ref}/{self.schema_dir}/fluid-schema-{version}.json" + + +class LocalDir: + def __init__(self, path: Path, preview_source: Optional[Path]): + if not path.is_dir(): + raise SourceError(f"not a directory: {path}") + self.path = path + self.preview_path = preview_source + self.label = str(path) + + def versions(self) -> List[str]: + return sorted((m.group(1) for p in self.path.iterdir() if (m := SCHEMA_RE.match(p.name))), key=vkey) + + def schema(self, version: str) -> bytes: + return (self.path / f"fluid-schema-{version}.json").read_bytes() + + def preview_text(self) -> Optional[str]: + if self.preview_path and self.preview_path.is_file(): + return self.preview_path.read_text() + return None + + def url(self, version: str) -> str: + return str(self.path / f"fluid-schema-{version}.json") + + +def parse_preview(text: str) -> Set[str]: + m = PREVIEW_RE.search(text) + if not m: + raise SourceError( + "could not find `PREVIEW_VERSIONS = frozenset({...})` in the reference " + "implementation's schema manager; update PREVIEW_RE in this script" + ) + return set(re.findall(r"[\"'](\d+\.\d+\.\d+)[\"']", m.group(1))) + + +# -------------------------------------------------------------------------- +# checks +# -------------------------------------------------------------------------- + + +def canonical(raw: bytes): + return json.loads(raw) + + +def check(source, record: Dict) -> List[str]: + problems: List[str] = [] + synced: List[str] = record["synced"] + preview: Set[str] = set(record.get("preview", [])) + stable: str = record["latestStable"] + floor = min(synced, key=vkey) + + upstream = source.versions() + print(f"reference: {source.label}") + print(f" bundles: {', '.join(upstream) or '(none)'}") + + for version in synced: + local = SCHEMA_DIR / f"fluid-schema-{version}.json" + if version not in upstream: + problems.append(f"{version}: listed as synced, but {source.label} does not bundle it") + continue + if not local.is_file(): + problems.append(f"{version}: listed as synced, but schema/{local.name} is missing") + continue + theirs = source.schema(version) + ours = local.read_bytes() + if ours == theirs: + print(f" ok {version} byte-identical") + continue + kind = "formatting only (same JSON, different bytes)" if canonical(ours) == canonical(theirs) else "content differs" + problems.append( + f"{version}: schema/{local.name} drifted from {source.label}: {kind}\n" + f" re-vendor: curl -fsSL {source.url(version)} -o schema/{local.name}" + ) + + for version in upstream: + if vkey(version) >= vkey(floor) and version not in synced: + problems.append( + f"{version}: {source.label} bundles it, but it is not vendored here " + f"(add it to schema/ and to \"synced\" in {VERSIONS_FILE.name})" + ) + + text = source.preview_text() + if text is None: + print(" preview set: not checked (no schema manager source given)") + else: + theirs = parse_preview(text) + if theirs != preview: + problems.append( + f"preview versions: {source.label} has {sorted(theirs, key=vkey)}, " + f"{VERSIONS_FILE.name} says {sorted(preview, key=vkey)}" + ) + stable_upstream = [v for v in upstream if v not in theirs] + if stable_upstream and stable_upstream[-1] != stable: + problems.append( + f"latest stable: {source.label}'s highest non-preview version is " + f"{stable_upstream[-1]}, {VERSIONS_FILE.name} says {stable}" + ) + if not problems: + print(f" ok preview {sorted(theirs, key=vkey)}, latest stable {stable}") + return problems + + +# -------------------------------------------------------------------------- +# modes +# -------------------------------------------------------------------------- + + +def warn(message: str) -> None: + """A warning on stderr, and an annotation on the run page under GitHub Actions.""" + print(f"check-schema-drift: warning: {message}", file=sys.stderr) + if os.environ.get("GITHUB_ACTIONS") == "true": + print(f"::warning title=forge-cli release::{message}") + + +def read_upstream(record: Dict) -> Tuple[str, str, str, str, str]: + """The pinned upstream, validated. Anything missing or malformed fails closed.""" + up = record.get("upstream") + if not isinstance(up, dict): + raise SourceError(f"{VERSIONS_FILE.name} has no \"upstream\" object") + for key in ("repository", "schemaDir", "previewSource", "ref", "commit"): + if key not in up: + raise SourceError( + f"{VERSIONS_FILE.name}: upstream.{key} is missing. The drift check compares against a " + "pinned release (upstream.ref and upstream.commit); it does not guess one." + ) + validate_repo_fields(up["repository"], up["schemaDir"], up["previewSource"]) + validate_ref(up["ref"], "upstream.ref") + validate_commit(up["commit"], "upstream.commit") + return up["repository"], up["schemaDir"], up["previewSource"], up["ref"], up["commit"] + + +def newer_than(candidate: str, pinned: str) -> Optional[bool]: + """True/False when both tags order numerically; None when they do not.""" + def numeric(tag: str): + m = re.match(r"v?(\d+)\.(\d+)\.(\d+)", tag) + return tuple(int(p) for p in m.groups()) if m else None + + a, b = numeric(candidate), numeric(pinned) + return None if a is None or b is None else a > b + + +def run_latest(record: Dict) -> int: + """Advisory: has forge-cli published a release after the pinned one?""" + repository, schema_dir, preview_source, pinned_ref, _ = read_upstream(record) + latest = latest_release_tag(repository) + if latest == pinned_ref: + print(f"check-schema-drift: {repository}'s latest release, {latest}, is the pinned one.") + return 0 + + ordering = newer_than(latest, pinned_ref) + if ordering is False: + print( + f"check-schema-drift: {repository}'s latest release is {latest}, which is older than the " + f"pinned {pinned_ref}. Nothing to do." + ) + return 0 + + print(f"check-schema-drift: a newer forge-cli release exists: {latest} (this repository vendors {pinned_ref}).") + print("What moving to it would involve, from a byte comparison with its schemas:") + problems = check(GitHubRelease(repository, schema_dir, preview_source, latest), record) + if problems: + for p in problems: + print(f" - {p}") + else: + print(f" nothing: the vendored schemas already equal {latest}'s; only the pin in {VERSIONS_FILE.name} would move.") + warn( + f"forge-cli {latest} is released; this repository vendors {pinned_ref}. " + f"To move: set upstream.ref and upstream.commit in scripts/{VERSIONS_FILE.name}, re-vendor, and " + "regenerate (see CONTRIBUTING.md). This does not fail pull requests." + ) + return 1 + + +def main(argv: Optional[List[str]] = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + which = ap.add_mutually_exclusive_group() + which.add_argument( + "--latest", + action="store_true", + help="advisory: report whether forge-cli has released after the pinned release (exit 1 if so)", + ) + which.add_argument("--ref", help="compare against this forge-cli tag instead of the pinned release") + which.add_argument("--source-dir", type=Path, help="compare against this directory instead of GitHub") + ap.add_argument( + "--schema-manager", + type=Path, + help="with --source-dir: the schema_manager.py whose PREVIEW_VERSIONS to compare", + ) + args = ap.parse_args(argv) + if args.schema_manager and not args.source_dir: + ap.error("--schema-manager is only meaningful with --source-dir") + + record = json.loads(VERSIONS_FILE.read_text()) + try: + if args.latest: + return run_latest(record) + + warnings: List[str] = [] + if args.source_dir: + source = LocalDir(args.source_dir, args.schema_manager) + elif args.ref: + repository, schema_dir, preview_source, _, _ = read_upstream(record) + source = GitHubRelease(repository, schema_dir, preview_source, validate_ref(args.ref, "--ref")) + else: + repository, schema_dir, preview_source, ref, commit = read_upstream(record) + source = GitHubRelease(repository, schema_dir, preview_source, ref, commit) + tag_note = source.tag_warning() + if tag_note: + warnings.append(tag_note) + problems = check(source, record) + for w in warnings: + warn(w) + except SourceError as exc: + print(f"check-schema-drift: {exc}", file=sys.stderr) + return 2 + + if problems: + print("\ncheck-schema-drift: the published schemas no longer match the reference release:", file=sys.stderr) + for p in problems: + print(f" - {p}", file=sys.stderr) + print( + "\nRe-vendor byte for byte (never edit a schema here), update " + f"scripts/{VERSIONS_FILE.name}, then run `python3 generate-schema-diffs.py` " + "and `python3 generate-docs.py` and commit the results. CONTRIBUTING.md " + "lists the steps.", + file=sys.stderr, + ) + return 1 + print("\ncheck-schema-drift: every synced schema matches the reference release.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/compat-waivers.txt b/scripts/compat-waivers.txt index b00303d..bc550e8 100644 --- a/scripts/compat-waivers.txt +++ b/scripts/compat-waivers.txt @@ -9,8 +9,9 @@ # can be recorded rather than silently tolerated. # # Pre-0.7.1 pairs are not waived individually. Those schemas predate the -# compatibility promise and, per .github/workflows/schema-sync.yml, already -# diverge from forge-cli at the byte level; CI runs the gate from 0.7.1 only. +# compatibility promise and are not synced with forge-cli (see +# scripts/schema-versions.json); check-compat.py starts at 0.7.1 by default +# (PROMISE_FLOOR) and reports the earlier pairs only with --all-history. # --------------------------------------------------------------------------- # 0.7.1 -> 0.7.2 -- REAL BREAK, SHIPPED. Not benign; waived only because it @@ -53,11 +54,11 @@ # --------------------------------------------------------------------------- # 0.7.3 -> 0.7.4 -- INTENTIONAL, and outside what the promise covers. # -# properties/fluidVersion gained an enum (0.7.3 constrained the member by -# pattern alone, so any \d+\.\d+(\.\d+)? was accepted; 0.7.4 restricts it to -# ["0.7.3", "0.7.4"]). This is the version gate doing its job: a document -# declares the schema version it targets, and that member is expected to move -# with the release. The empirical pass in check-compat.py normalises +# properties/fluidVersion changed from const "0.7.3" (with a pattern) to enum +# ["0.7.3", "0.7.4"]. The gate reports "enum constraint added" because 0.7.3 +# had no enum. Every value 0.7.3 accepted is still accepted. This is the +# version gate doing its job: a document declares the schema version it +# targets, and that member is expected to move with the release. The empirical pass in check-compat.py normalises # fluidVersion across versions for exactly this reason. # # Re-examine if the enum ever DROPS a version that a supported release still diff --git a/scripts/requirements-docs.lock b/scripts/requirements-docs.lock new file mode 100644 index 0000000..bb7690a --- /dev/null +++ b/scripts/requirements-docs.lock @@ -0,0 +1,418 @@ +# +# This file is autogenerated by pip-compile with Python 3.12 +# by the following command: +# +# pip-compile --generate-hashes --output-file=scripts/requirements-docs.lock --strip-extras scripts/requirements-docs.txt +# +certifi==2026.7.22 \ + --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \ + --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55 + # via requests +charset-normalizer==3.5.2 \ + --hash=sha256:01077390b03f7988f11d700a2194e69b119741a86b1a638b1db88891e3eced8e \ + --hash=sha256:01b0c0d2262a9e28e8484a278c7e1b5d650e3ac8cf2683d2967e25899f208bdf \ + --hash=sha256:04851f73ae72b8413dddadb16a49dfee95263553741fd42d546f7d66907e6be5 \ + --hash=sha256:0521c5665880b33d603717defa76c094048900010897909952397feb3039da56 \ + --hash=sha256:0774bf9bf620249fee3e0b8b9fd3065de213be30f3aa94ce2494b3b638949e26 \ + --hash=sha256:0891b9d3903c5571c03771ca669a4b0ec5618ca722a5c957d3d29cd4e5062848 \ + --hash=sha256:0c951d5e6dd9c2ff60609476752bee49da4206adde960ebc247766937f72e718 \ + --hash=sha256:0fed1d06615f022ee3b13caf5e8b180cfea32bb2c5aded8a9d44277afc040f93 \ + --hash=sha256:114e4d0c92d618409ed82a99e22b5c5e768fe995f2973f78265f4524f49d4640 \ + --hash=sha256:11912e4bb14baae7c5d8791aa55ba0a3a03ec6729073307b0f57270abaa713d3 \ + --hash=sha256:11a4d68a6ecda3292cb1e50239e111543ba5d709bb62a6b4ea1afcfa729d8875 \ + --hash=sha256:124fbf1a8ff966d87ae05bb8bd45a71f966055ed8bba320d0c7cf450bc5f4d0e \ + --hash=sha256:1461ac396c4fdb983a675f20aa555624f0ee18ac83d832b9244ffff3d8055275 \ + --hash=sha256:1503bccbeb36d5527790c3930327704c39af22de3112f1b1666a9f3ce15ee204 \ + --hash=sha256:15bb4005af6320d259dc7593ca84a38d7fe06a421dbcf7b910ae23979101e787 \ + --hash=sha256:15c44f7edfd477b06f517a5cc317fc1707edb9de2c865f43d4b6513907473234 \ + --hash=sha256:16fa0eccf81304b79c5cd87f9271c3b85dd9dd99245e4422ae9c0dd45e0f99d3 \ + --hash=sha256:183b88127acdb4fabe59d951ab424faf1af7b63cdbb5f776186c1ea2ffcaed98 \ + --hash=sha256:195c26fb65950f8fce54e26349852b7bdd7c5f120aeefbcc440b8a20faaed4a3 \ + --hash=sha256:1afb975bd5d68d5ce9f6b6d44fdf2f7e34b895a35e95708a7a91b20a3b51d187 \ + --hash=sha256:1b4cbc7c3491ccb4aa17fcd8165649d01cf39f76de1696da8631b5f71b85401d \ + --hash=sha256:1bc0baf5ef96b6ede57d47f4b8fe4d9d84019c3bfcbeb20a41edc6a6ee341f1f \ + --hash=sha256:1c50fe28bbc2ced33386f298650d91218076c05420e6cbd790b913adc41659e7 \ + --hash=sha256:1db38f4c5496827c1a501846d64d14c3b80c7e6714e406cd7dc36a9899fa1011 \ + --hash=sha256:211d5a3eb6af8f513b8d4ca19a8c1b7accab1b5f0d3175f9826b03c1a920dc1f \ + --hash=sha256:23851fb4e1b85ed3f6c2a27b777cdfe2e19fb5b38429a8faf38c7542b7665869 \ + --hash=sha256:254eb48b9fa5ee9898a3c445825a1f340fe53712a098904b39b0bddba8ea3cb1 \ + --hash=sha256:2625388c6c754520c37abaf3b41eb34d1cc4a373f457898f08606c8e362b891d \ + --hash=sha256:281cb91036248400f4cc957495cccd44c275c2e0c5854f7e45ac5cf7dc193847 \ + --hash=sha256:28a15fdad492a99b6eccfaaed66ef3f74050680545ea61ec8b2f4c538f1f1320 \ + --hash=sha256:28b4f0d66fb834ff90f28209ac7bce77868c45d8c93e26f906709d9b7c2e1af9 \ + --hash=sha256:2a925889534b3748302dae5dead07cc13480de1dac3aea80a941b729b471ef93 \ + --hash=sha256:2b7b3bbfb4fe8ef40600792d762fbaa9057559f9d3fad209525b7a22b99e91fd \ + --hash=sha256:2c9ad19a6cfcd5ea5c0d41161d22f9df1dcc277e9bef2751391334546a314c00 \ + --hash=sha256:2cc961b171b3f3440f410489ab3573e86aea8736134ebbb40ea1338b7f0831bc \ + --hash=sha256:2ce45c6627b22c47e390bc91a41c3d13032192e699fa0bea96e9671b373d69b0 \ + --hash=sha256:2e06a3a98f916dd41d27f3105e02e7a40181c98c94b9158733d03a6f80506c09 \ + --hash=sha256:304d5463e65a35d7bb0850550e0780395395f6fcf452f04db7d5ca7cecc425ac \ + --hash=sha256:304d8e4d493af723536393eee0c689eb7813f4a474c8b479dee63f1fdd98f621 \ + --hash=sha256:30fcd120b732aa79317f08dee04d7de0847822e4cf7ee0e9f445bb958832252c \ + --hash=sha256:31f3930700408d211f13378ccbe1c40845d8da54bd0681fac3a9b5aae81c7aa8 \ + --hash=sha256:34276fd796040bf0993ab33a369aa572e6979c7aab225a88893667ad8eac8f7a \ + --hash=sha256:355ad8011081dec5412240c087a9a0c9d4d5039f3ed11a3f13e18c2b29b56c51 \ + --hash=sha256:38a873987f3be698494da8b2e3085e29da02da7b633dce73e79c699a113d7bf0 \ + --hash=sha256:39de2a259fc954455c57274dc94c79d5842774e1247a016aff30bc0efed0f4ef \ + --hash=sha256:3d14b50de6bf4d0edf857a9386836846f982b8f524e188e2e68b96d702bcf4aa \ + --hash=sha256:3d21b8b13c7592db2ac5e544a6d83187b995257472b0c9e8351b6d507ae37ed6 \ + --hash=sha256:3d31298449090ab8d47b7b1b2a555ff73cac7ed438a08b7ac160980c7ebed649 \ + --hash=sha256:3ddacd27458c45bdacd6bd6db644bfb730efbf9e830310186e3045c9c5be8fb2 \ + --hash=sha256:3df041de8887954562c9b261cba85ca0e9ded74048daf125f45edcfaa4832229 \ + --hash=sha256:40ab6bffa02ae10a0581e6c198be7d2d8ca5c2a0c64e4ed3465d766df457573e \ + --hash=sha256:4275811936e2f06feff5e598fb42a1b7ae852da8e39605211892b56b81a34efd \ + --hash=sha256:443eae2bf318abeaf6f15d785138f71fd6de770e99a92158b8b814265e079115 \ + --hash=sha256:447441e76ec720b15e64418d32e092297340387053047c7c694f579efb0ee1d9 \ + --hash=sha256:4495c5002a7b28557e7e222e77e0b661183e432b7d6d2e788101e3f240e05b8c \ + --hash=sha256:44bd4fbb29dfbeba60e7d2bd000c59e4b21ddb3cc53912b14048d37092706d7c \ + --hash=sha256:4685902cf26edf013ed7a3da0f426ebba7a00ebb9541386d835afbf002c11cab \ + --hash=sha256:498dc3188ca05a68231ac3fdbfc7f57eb67e1343c30e0fea17f8218c1599b253 \ + --hash=sha256:4c2b5031f63e331e3839b40aed2dd6f191e9c07edbde303e7876846ea1946995 \ + --hash=sha256:4d48f2d08b9de5864e2c8744d4461b862fb149a18274abc8b698c45975573438 \ + --hash=sha256:4f87960d57feabfb618e4e0af6e7371645fa26a277860739d6e5d6e0012c92f0 \ + --hash=sha256:50e3adfb96fc189eb27b1cf62d3b598b89b4bb0420d93a3d3e42e137409011be \ + --hash=sha256:51cf45226a9b588d0d2b4880c62d686934b63ab0bd79ca23ab0e9762eb27441b \ + --hash=sha256:52aa6992700996af31f375de0c6bacd402b0097fe40b53c426b9f51a90ebabc7 \ + --hash=sha256:55ea99acb17b9325618de155a0cd6a2e8f5d10be008113e1d433bbb58db543b2 \ + --hash=sha256:56bc200a365efb37383b7852e4cc5898d3b2da5987289b543956cf8cad71018a \ + --hash=sha256:588461c2e8384d309bd63e5826019b6977bc66d629b99ac8737bb795d7b2cb5a \ + --hash=sha256:58ca3755ee7ff7f59b57789ec9833c9de9ea275405cdd240eda1f193112e398a \ + --hash=sha256:58f361dcbab699cf8f42db3f47c8e7fd1036f138c23a5d08de9fde5f425a730c \ + --hash=sha256:598a11a2c7ebaa5334bf698bf29568c9c390abac6a154d8170fedecd1cea38c5 \ + --hash=sha256:59f63901b0031c3136cf64704dcb21de0bbae62ce2c9529bc39d27665463de37 \ + --hash=sha256:5cde776b7cc66e4f6c99612cea4aa7269aa65863f7a15841b2c264f103822f4e \ + --hash=sha256:5e2b6b57e9733d39f0c9fd3185efa6b8e29652c4cd8fe94180272cf6ed9a78c4 \ + --hash=sha256:5fb29fb8cd1a46c27a1bf9613ad5ec2599310d46b4025d9556404a6b6a292800 \ + --hash=sha256:6045373d5a89a5ec71afde535db987ca28e76dfa276c2d4c818265b375d4b055 \ + --hash=sha256:619799369eeef6366ed3e8755a5670f4f2f0fb6b30a0fd7264dc0fdc2357058e \ + --hash=sha256:62588a277bfb59def052abd940703fa35107152bf479781a878617d60faf8fb5 \ + --hash=sha256:62603db9a7caa0802eaa28c1c46fecd7b3a263a774069c24c3c28c302448721c \ + --hash=sha256:65cd72beeeca9d3aaea1201e5923859f308f952f9c71de93f06063c79f0f7a3b \ + --hash=sha256:68eb192d85ab8e5f6ec69c2bc6ac0179fbf04a5ac1569d12fbef74883fe102d0 \ + --hash=sha256:6bd128f206a7752ae1f2ab6c61bf8a24ba28913a10df8b14c2637b973ff97a80 \ + --hash=sha256:6be488a102b8cf28d0391d8c4ba7748938ae28b78ad901f8585520fca33ead1a \ + --hash=sha256:7218e8f32b0956cfcd048fd42d9d5779809745ca1d86113ca56f66e7ae1549c4 \ + --hash=sha256:7441d755b7ab94f8d4eb3e43ec05482d760842fd263d003a99102d742cd835e2 \ + --hash=sha256:749e97e1b32313717a565abbe321bc2190bc8b35f1a67e4cdbc7c56c8d8ffe58 \ + --hash=sha256:75a3ceed0724d625d64b86ca20aba182e4df462e04c2414fc941c0f523f06aac \ + --hash=sha256:780fbe7cab297b81dad9fb8dc5eb003c0468ffb0d9e5f65068c53a34661a96bc \ + --hash=sha256:78456a747de8dc58360ffa581f30a002baf5aa28cb262536545e91f113ed7639 \ + --hash=sha256:7967d08cf06dee78443b874f98c98036f624f3a4e73e11f9f64f5be4d25393cf \ + --hash=sha256:7a881931aa470808df94a8c380eed2bbbc76cd9dc622310f99665658c821eb6d \ + --hash=sha256:7dcd882da75ef9adf94903b1e3b9419e8aa8fb4c7396822b834b9ef7fb96954f \ + --hash=sha256:7e841fb9010836c992c9f12fcbd43a831de93a5f726fc1ccd8ca1d0268c5014c \ + --hash=sha256:7fdde2c9fd9e3eca40631e024664cf2584272cc8f96308cbe5fdfc930f51d8bc \ + --hash=sha256:8024d00c3faf3fc0c16e07a69f4405e8eac7cc0ab15f65fe6cf43827c4cf72b4 \ + --hash=sha256:80d02b6f04e92601a081dd97b23d3128033098bff5d35d392ddcc0476ea11253 \ + --hash=sha256:838dcc90063569a0448120554591a1d6c4a4ffe11babf048908793154ab86ade \ + --hash=sha256:849df64e889b2e17230d58410a03dba311a65b163508fd33679b2b737d4b7858 \ + --hash=sha256:87475fabc8d9996fd9c27debb395e642e8c838d78a00b6e932227a0e06b81e26 \ + --hash=sha256:87e50a3e7cb90af586b6c5faf23e302a970415ac73bd7bd90a515a04b427ef96 \ + --hash=sha256:89b53f3cda69831909888e0494f4fa0bcd3537e3e138dabeb620bd6ad946bae8 \ + --hash=sha256:8a893cc101149f80a653f82062ebc95b34525a2614382e1da5458fe7c6997249 \ + --hash=sha256:8b2bfab86aa71ae13aa41a6a26aab338e0db2b8bc75434b05aea89e011ff35a4 \ + --hash=sha256:8d86d6fc60743dc916eb79e2eb1ec4818e21e427731543af40a3021851174a13 \ + --hash=sha256:915563965d418f986e7e145accc592eae9e1a1be3566ff98a05d7a9ec42a76e1 \ + --hash=sha256:92888bb3187c5ba50500b00b3b310c9f2c651709d28036077680cb5255450a03 \ + --hash=sha256:93223adc95033dd47133a46ccfc316a0139176fd79085762e27202ec56018f03 \ + --hash=sha256:9373ad13ef0d2c0fb761e04e55bfdee5a08b52cef2c882c8fbe9935b1517152e \ + --hash=sha256:9409a8bf35cf78353942504b24a57de3d75b708997a1e4bd8db71ac8633ce364 \ + --hash=sha256:9b7f416ff0978e2f2249330527f0ad6fa02f4932e6199692d3b52da2048c19e4 \ + --hash=sha256:9bde855991b7e362c146535e3136a50bfaffc0487d38b33ca7e5edefc6e23849 \ + --hash=sha256:9cae88599c7219005d879f98e5ed53341e9a122af585e1091200358a3003d2a0 \ + --hash=sha256:9cf9b1a857e25c4baceeb3624e92a56df3668f398c4acba74e174d81fb4d1d3a \ + --hash=sha256:9f56f72050826f63dcee7a7f55b0a77168cb3bfc553fd405e7f8f9ece75a4036 \ + --hash=sha256:a090bb2c68df85450502e3e20d665e3a5af9c65a84d6508ed477badd49166fd3 \ + --hash=sha256:a192e2c40070d92c3ccf777e3a5c4ff515573cd2bb7ed0c537fdadbbec5bbf21 \ + --hash=sha256:a19a731138fc27d5682277d3b9df22855cea1239bce7fcec5f78f42ef2d1f3c3 \ + --hash=sha256:a66c3bc5ab1f0ff2164fc9965ddd611ff0802173f4b9d24554c563f6ab7e1d6e \ + --hash=sha256:a815775b6c38d4e0ff7bcffbeba67feded90202bb6a226b8dd35f1c855217413 \ + --hash=sha256:a89012d6d5476ee112d20d998570ed58df2260a852afb1758809cd6900411d21 \ + --hash=sha256:ae4f5fea5b8b8ccff88238cc8569303e5ee95efae67fa62922a311397a71f346 \ + --hash=sha256:b6856554c4f44d79fc2307d5768854310a8f0096e501c75637542c82292b0429 \ + --hash=sha256:b6b751274acb69d77b3323d6b7dbaa3c7fdfc1eb829b7eb61d262f32e1af9685 \ + --hash=sha256:b736353c0a625bbd5fcec108576e2385db3496f4f771f785ff32e108d3c3bc45 \ + --hash=sha256:b7fd005a73d9e657273b7a10dc71a9e03c8fb9ee6999798d6918ce095b81ac7f \ + --hash=sha256:b91363207bd9dc966a691e959bb47f64b30f7ac4b072be9968b366982f7db77c \ + --hash=sha256:ba0b1d2620edf869789c3879223f52bf2afc5d31b3cb47cc57b3a12c05e2aa9d \ + --hash=sha256:bbbfc8e28816f19d7c0f1816664980c0a9875d01b27cdf8eedddb639d9e108ad \ + --hash=sha256:bd16aabe4a02a297c23417aa17ac6299dbd8c49f673bcd645b4929b11f5a4400 \ + --hash=sha256:c0afc6800ba57ccc350374c5bd6150419915d95ce93cdbab2d783d75eaf30ecb \ + --hash=sha256:c6708715abcf3c73b99508253e961a9967f02fe536532834149574eda6de0d1c \ + --hash=sha256:c7c9ab723cde841fefb34efbad91e87f00a674b1fe1cd0784fde742bf2c154dc \ + --hash=sha256:c8f3d67aeaf55f017982b73683f0e7342ba2f6635a78f69ce89ebb26aa411e5c \ + --hash=sha256:c9790464842f85f437dbbb54417eda1e0e6bfc52dd8d22d6fd1c994b73b2dc74 \ + --hash=sha256:ca403d7e4798f525fdfc78e258820419cbbd0f0ecbab9de7840e3c017cf6b8cf \ + --hash=sha256:d008d90a7f2471519aef0c90dfbe73b3e6e4d5e66ac48e19154c17e89e98b604 \ + --hash=sha256:d19fbd981a488e22cd04883659ca6b08f50b5974f9fd7c95655ef6a043e5893f \ + --hash=sha256:d1befeed746d247c81127bb14de9dc3d30edb6e5976d34f83f86ed262b1d9105 \ + --hash=sha256:d2374b62878abb00cd8309b32af6c0b715cd02dec0ca74ef12e5069bdc64144a \ + --hash=sha256:d376bbd28b3a8999db1a103b3b388aee6f1ddeb3e51bc2172993efdcd86e064d \ + --hash=sha256:d4a7319f304a774bed22115bc891618e45f85065ab44ea6acd07d274e750519a \ + --hash=sha256:d6734d2ef8a50fbf8445c139477da401f50d62a0606bf00e20ec6d87773fefb1 \ + --hash=sha256:d760fe2a4d7c3b226cb9026d6a842868d52a7901bd98420e1baf14e80da85cf5 \ + --hash=sha256:d913de495d90407cd859d263bee2e5d1a4ed3eb6573c04e70d9ec619a7cbed7f \ + --hash=sha256:db19d07e2e0129e974a0e65d0064fc222a446cd5122c2fd4184d2af9fc734a9e \ + --hash=sha256:dca9ab98072a5a54ebacebdc45f53e645336b320c667410b061be1ca588ae709 \ + --hash=sha256:ddc7dacc8ece3a182e7f15cb862d1fd616b46d076cb1ae9dd232b2c38b655874 \ + --hash=sha256:ddf19c062bea7a0cc80f519243d2c01dd091be0cf952a0750d4ad576709559f5 \ + --hash=sha256:def79fa35ef0cef8d2accec024f4fdc7ead3012ff02f5215c783f39f03ef8cfc \ + --hash=sha256:df29a0a7107f7011e77f4eebdddec4c7331e24d787a0b21a46d63bdf7445da95 \ + --hash=sha256:e09a3942ecbdee5cce73ea9d42da82b81b72ac1bf031ce069b93b5adf4eac8cd \ + --hash=sha256:e242bb1c5e76e97dfa9e7f209a71e93a01d7f19ffdd5cfbb2e2d55b4f08f8ab0 \ + --hash=sha256:e243bd13217235fc7290c621941c3f5cc8b66e4872495be821d7436ba2fb838d \ + --hash=sha256:e2af3aad578aa6bd1384bcf4750fc285e5a9de53f40b7d41e5a0bf748edeb2b3 \ + --hash=sha256:e4e81e09c1578b8df602e3db08b0b3ea0a6947ad612f52bf8dc5ea8d47691f0c \ + --hash=sha256:e54da4baf05720032d527874d40b65fa4d7e5c6c6a43d0c3adbeffcaf275a2b3 \ + --hash=sha256:e80e6c2f55656b4824d72065abb4ddd6a525c74bd78a0aab5d9fc2cf4fb5af50 \ + --hash=sha256:ed2a239c0ea213acc1908150a3037257083c7c083128f1a4cec2ec4b97dca491 \ + --hash=sha256:ed905975ab14056a2e5eb1c376cb2e1ebc5396baf84163939c518556fccde9f5 \ + --hash=sha256:ee21e28f0430bd6dc9086c6e525d5e818a44a5ad19720c8a0ef766792f3eb5e5 \ + --hash=sha256:ee43c17b173d46a3212baa6ead3ae258eeabdae48c263a01ccf0218c366dd655 \ + --hash=sha256:ef4fcbf3327382cd4c9f540babd61248208af7b93eec4de397b4d5f58a09e288 \ + --hash=sha256:eff0ac9dbe711a4aee69bf04a83896aa9b85f19641264053a9f6d48573abb7dd \ + --hash=sha256:f0aa869112ef88429ae17820d99c3dd9504c9e9c671d3c246f3d7442cb051084 \ + --hash=sha256:f3c96f633825733f735c5a9cf21d21a257d8e1edf0b1cee0a064b9c424ca0f7d \ + --hash=sha256:f5833ad231be5eb6553de524a70f48d71b2c8563101750531e0b80184e175cd4 \ + --hash=sha256:f5ec61164adcec446f8969a3358ec3f9b26bbda3b9213e5586d219afa8df2915 \ + --hash=sha256:f7d486c83842422badd511868fd8a9a20e9407ace71564b6af47ce7e60a336c1 \ + --hash=sha256:fb9e68df06293761f9fe66ade60a9bc6d0f5e42b8acf2939a9158af86ab0e5bd \ + --hash=sha256:fc14a032f813bf5fe624d991960ea83e9715adc27e4c1830a2361eb1d02ac341 \ + --hash=sha256:fcff63213e8e6e47770541a4607175404f47cbb3ebea7b6058cc82d524a0e424 \ + --hash=sha256:fd1fbe0f116b6e55da77aca2c6ddcddcfac2186cbf78bdebf40fc156efca389d \ + --hash=sha256:fe9753dfee015c570d73df76f899f18444d41388bffcde097deba51c4fadbb9f + # via requests +click==8.5.0 \ + --hash=sha256:255bc9599cf7748b4b1a446ccc735421bd08a2ae529a8b88597d3de5664ee360 \ + --hash=sha256:ba0d2089de75ea0310e2dde03160e6ca10009947fb95a182f9b54021bb272e34 + # via json-schema-for-humans +dataclasses-json==0.6.7 \ + --hash=sha256:0dbf33f26c8d5305befd61b39d2b3414e8a407bedc2834dea9b8d642666fb40a \ + --hash=sha256:b6b3e528266ea45b9535223bc53ca645f5208833c29229e847b3f26a1cc55fc0 + # via json-schema-for-humans +idna==3.20 \ + --hash=sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44 \ + --hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c + # via requests +jinja2==3.1.6 \ + --hash=sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d \ + --hash=sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67 + # via + # -r scripts/requirements-docs.txt + # json-schema-for-humans +json-schema-for-humans==1.5.1 \ + --hash=sha256:6b5bb65c8ccfb46219352f3da32312c9d904c08790bb96f43f109338c2141478 \ + --hash=sha256:a43023f6c1b99f8e75126cc824c22992d8f55e941060aa3fddfc7c38de3cb973 + # via -r scripts/requirements-docs.txt +markdown2==2.5.5 \ + --hash=sha256:001547e68f6e7fcf0f1cb83f7e82f48aa7d48b2c6a321f0cd20a853a8a2d1664 \ + --hash=sha256:be798587e09d1f52d2e4d96a649c4b82a778c75f9929aad52a2c95747fa26941 + # via + # -r scripts/requirements-docs.txt + # json-schema-for-humans +markupsafe==3.0.3 \ + --hash=sha256:0303439a41979d9e74d18ff5e2dd8c43ed6c6001fd40e5bf2e43f7bd9bbc523f \ + --hash=sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a \ + --hash=sha256:0bf2a864d67e76e5c9a34dc26ec616a66b9888e25e7b9460e1c76d3293bd9dbf \ + --hash=sha256:0db14f5dafddbb6d9208827849fad01f1a2609380add406671a26386cdf15a19 \ + --hash=sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf \ + --hash=sha256:0f4b68347f8c5eab4a13419215bdfd7f8c9b19f2b25520968adfad23eb0ce60c \ + --hash=sha256:1085e7fbddd3be5f89cc898938f42c0b3c711fdcb37d75221de2666af647c175 \ + --hash=sha256:116bb52f642a37c115f517494ea5feb03889e04df47eeff5b130b1808ce7c219 \ + --hash=sha256:12c63dfb4a98206f045aa9563db46507995f7ef6d83b2f68eda65c307c6829eb \ + --hash=sha256:133a43e73a802c5562be9bbcd03d090aa5a1fe899db609c29e8c8d815c5f6de6 \ + --hash=sha256:1353ef0c1b138e1907ae78e2f6c63ff67501122006b0f9abad68fda5f4ffc6ab \ + --hash=sha256:15d939a21d546304880945ca1ecb8a039db6b4dc49b2c5a400387cdae6a62e26 \ + --hash=sha256:177b5253b2834fe3678cb4a5f0059808258584c559193998be2601324fdeafb1 \ + --hash=sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce \ + --hash=sha256:1b4b79e8ebf6b55351f0d91fe80f893b4743f104bff22e90697db1590e47a218 \ + --hash=sha256:1b52b4fb9df4eb9ae465f8d0c228a00624de2334f216f178a995ccdcf82c4634 \ + --hash=sha256:1ba88449deb3de88bd40044603fafffb7bc2b055d626a330323a9ed736661695 \ + --hash=sha256:1cc7ea17a6824959616c525620e387f6dd30fec8cb44f649e31712db02123dad \ + --hash=sha256:218551f6df4868a8d527e3062d0fb968682fe92054e89978594c28e642c43a73 \ + --hash=sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c \ + --hash=sha256:2713baf880df847f2bece4230d4d094280f4e67b1e813eec43b4c0e144a34ffe \ + --hash=sha256:2a15a08b17dd94c53a1da0438822d70ebcd13f8c3a95abe3a9ef9f11a94830aa \ + --hash=sha256:2f981d352f04553a7171b8e44369f2af4055f888dfb147d55e42d29e29e74559 \ + --hash=sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa \ + --hash=sha256:3524b778fe5cfb3452a09d31e7b5adefeea8c5be1d43c4f810ba09f2ceb29d37 \ + --hash=sha256:3537e01efc9d4dccdf77221fb1cb3b8e1a38d5428920e0657ce299b20324d758 \ + --hash=sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f \ + --hash=sha256:38664109c14ffc9e7437e86b4dceb442b0096dfe3541d7864d9cbe1da4cf36c8 \ + --hash=sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d \ + --hash=sha256:3b562dd9e9ea93f13d53989d23a7e775fdfd1066c33494ff43f5418bc8c58a5c \ + --hash=sha256:457a69a9577064c05a97c41f4e65148652db078a3a509039e64d3467b9e7ef97 \ + --hash=sha256:4bd4cd07944443f5a265608cc6aab442e4f74dff8088b0dfc8238647b8f6ae9a \ + --hash=sha256:4e885a3d1efa2eadc93c894a21770e4bc67899e3543680313b09f139e149ab19 \ + --hash=sha256:4faffd047e07c38848ce017e8725090413cd80cbc23d86e55c587bf979e579c9 \ + --hash=sha256:509fa21c6deb7a7a273d629cf5ec029bc209d1a51178615ddf718f5918992ab9 \ + --hash=sha256:5678211cb9333a6468fb8d8be0305520aa073f50d17f089b5b4b477ea6e67fdc \ + --hash=sha256:591ae9f2a647529ca990bc681daebdd52c8791ff06c2bfa05b65163e28102ef2 \ + --hash=sha256:5a7d5dc5140555cf21a6fefbdbf8723f06fcd2f63ef108f2854de715e4422cb4 \ + --hash=sha256:69c0b73548bc525c8cb9a251cddf1931d1db4d2258e9599c28c07ef3580ef354 \ + --hash=sha256:6b5420a1d9450023228968e7e6a9ce57f65d148ab56d2313fcd589eee96a7a50 \ + --hash=sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698 \ + --hash=sha256:729586769a26dbceff69f7a7dbbf59ab6572b99d94576a5592625d5b411576b9 \ + --hash=sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b \ + --hash=sha256:795e7751525cae078558e679d646ae45574b47ed6e7771863fcc079a6171a0fc \ + --hash=sha256:7be7b61bb172e1ed687f1754f8e7484f1c8019780f6f6b0786e76bb01c2ae115 \ + --hash=sha256:7c3fb7d25180895632e5d3148dbdc29ea38ccb7fd210aa27acbd1201a1902c6e \ + --hash=sha256:7e68f88e5b8799aa49c85cd116c932a1ac15caaa3f5db09087854d218359e485 \ + --hash=sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f \ + --hash=sha256:8485f406a96febb5140bfeca44a73e3ce5116b2501ac54fe953e488fb1d03b12 \ + --hash=sha256:8709b08f4a89aa7586de0aadc8da56180242ee0ada3999749b183aa23df95025 \ + --hash=sha256:8f71bc33915be5186016f675cd83a1e08523649b0e33efdb898db577ef5bb009 \ + --hash=sha256:915c04ba3851909ce68ccc2b8e2cd691618c4dc4c4232fb7982bca3f41fd8c3d \ + --hash=sha256:949b8d66bc381ee8b007cd945914c721d9aba8e27f71959d750a46f7c282b20b \ + --hash=sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a \ + --hash=sha256:9a1abfdc021a164803f4d485104931fb8f8c1efd55bc6b748d2f5774e78b62c5 \ + --hash=sha256:9b79b7a16f7fedff2495d684f2b59b0457c3b493778c9eed31111be64d58279f \ + --hash=sha256:a320721ab5a1aba0a233739394eb907f8c8da5c98c9181d1161e77a0c8e36f2d \ + --hash=sha256:a4afe79fb3de0b7097d81da19090f4df4f8d3a2b3adaa8764138aac2e44f3af1 \ + --hash=sha256:ad2cf8aa28b8c020ab2fc8287b0f823d0a7d8630784c31e9ee5edea20f406287 \ + --hash=sha256:b8512a91625c9b3da6f127803b166b629725e68af71f8184ae7e7d54686a56d6 \ + --hash=sha256:bc51efed119bc9cfdf792cdeaa4d67e8f6fcccab66ed4bfdd6bde3e59bfcbb2f \ + --hash=sha256:bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581 \ + --hash=sha256:bdd37121970bfd8be76c5fb069c7751683bdf373db1ed6c010162b2a130248ed \ + --hash=sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b \ + --hash=sha256:c0c0b3ade1c0b13b936d7970b1d37a57acde9199dc2aecc4c336773e1d86049c \ + --hash=sha256:c47a551199eb8eb2121d4f0f15ae0f923d31350ab9280078d1e5f12b249e0026 \ + --hash=sha256:c4ffb7ebf07cfe8931028e3e4c85f0357459a3f9f9490886198848f4fa002ec8 \ + --hash=sha256:ccfcd093f13f0f0b7fdd0f198b90053bf7b2f02a3927a30e63f3ccc9df56b676 \ + --hash=sha256:d2ee202e79d8ed691ceebae8e0486bd9a2cd4794cec4824e1c99b6f5009502f6 \ + --hash=sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e \ + --hash=sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d \ + --hash=sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d \ + --hash=sha256:de8a88e63464af587c950061a5e6a67d3632e36df62b986892331d4620a35c01 \ + --hash=sha256:df2449253ef108a379b8b5d6b43f4b1a8e81a061d6537becd5582fba5f9196d7 \ + --hash=sha256:e1c1493fb6e50ab01d20a22826e57520f1284df32f2d8601fdd90b6304601419 \ + --hash=sha256:e1cf1972137e83c5d4c136c43ced9ac51d0e124706ee1c8aa8532c1287fa8795 \ + --hash=sha256:e2103a929dfa2fcaf9bb4e7c091983a49c9ac3b19c9061b6d5427dd7d14d81a1 \ + --hash=sha256:e56b7d45a839a697b5eb268c82a71bd8c7f6c94d6fd50c3d577fa39a9f1409f5 \ + --hash=sha256:e8afc3f2ccfa24215f8cb28dcf43f0113ac3c37c2f0f0806d8c70e4228c5cf4d \ + --hash=sha256:e8fc20152abba6b83724d7ff268c249fa196d8259ff481f3b1476383f8f24e42 \ + --hash=sha256:eaa9599de571d72e2daf60164784109f19978b327a3910d3e9de8c97b5b70cfe \ + --hash=sha256:ec15a59cf5af7be74194f7ab02d0f59a62bdcf1a537677ce67a2537c9b87fcda \ + --hash=sha256:f190daf01f13c72eac4efd5c430a8de82489d9cff23c364c3ea822545032993e \ + --hash=sha256:f34c41761022dd093b4b6896d4810782ffbabe30f2d443ff5f083e0cbbb8c737 \ + --hash=sha256:f3e98bb3798ead92273dc0e5fd0f31ade220f59a266ffd8a4f6065e0a3ce0523 \ + --hash=sha256:f42d0984e947b8adf7dd6dde396e720934d12c506ce84eea8476409563607591 \ + --hash=sha256:f71a396b3bf33ecaa1626c255855702aca4d3d9fea5e051b41ac59a9c1c41edc \ + --hash=sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a \ + --hash=sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50 + # via + # -r scripts/requirements-docs.txt + # jinja2 +marshmallow==3.26.2 \ + --hash=sha256:013fa8a3c4c276c24d26d84ce934dc964e2aa794345a0f8c7e5a7191482c8a73 \ + --hash=sha256:bbe2adb5a03e6e3571b573f42527c6fe926e17467833660bebd11593ab8dfd57 + # via dataclasses-json +mypy-extensions==1.1.0 \ + --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \ + --hash=sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558 + # via typing-inspect +packaging==26.3 \ + --hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \ + --hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c + # via marshmallow +pygments==2.21.0 \ + --hash=sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9 \ + --hash=sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c + # via + # -r scripts/requirements-docs.txt + # json-schema-for-humans +pytz==2026.5 \ + --hash=sha256:e658af3757f9e26a9d25dd2aff38335acd92bc9104f890a894b2c1ba28311b03 \ + --hash=sha256:fa23724b9c486543b9ff54a327ee7569ac83ade54bb9afd0fc18676620401c86 + # via json-schema-for-humans +pyyaml==6.0.3 \ + --hash=sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c \ + --hash=sha256:0150219816b6a1fa26fb4699fb7daa9caf09eb1999f3b70fb6e786805e80375a \ + --hash=sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3 \ + --hash=sha256:02ea2dfa234451bbb8772601d7b8e426c2bfa197136796224e50e35a78777956 \ + --hash=sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6 \ + --hash=sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c \ + --hash=sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65 \ + --hash=sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a \ + --hash=sha256:1ebe39cb5fc479422b83de611d14e2c0d3bb2a18bbcb01f229ab3cfbd8fee7a0 \ + --hash=sha256:214ed4befebe12df36bcc8bc2b64b396ca31be9304b8f59e25c11cf94a4c033b \ + --hash=sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1 \ + --hash=sha256:22ba7cfcad58ef3ecddc7ed1db3409af68d023b7f940da23c6c2a1890976eda6 \ + --hash=sha256:27c0abcb4a5dac13684a37f76e701e054692a9b2d3064b70f5e4eb54810553d7 \ + --hash=sha256:28c8d926f98f432f88adc23edf2e6d4921ac26fb084b028c733d01868d19007e \ + --hash=sha256:2e71d11abed7344e42a8849600193d15b6def118602c4c176f748e4583246007 \ + --hash=sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310 \ + --hash=sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4 \ + --hash=sha256:3c5677e12444c15717b902a5798264fa7909e41153cdf9ef7ad571b704a63dd9 \ + --hash=sha256:3ff07ec89bae51176c0549bc4c63aa6202991da2d9a6129d7aef7f1407d3f295 \ + --hash=sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea \ + --hash=sha256:418cf3f2111bc80e0933b2cd8cd04f286338bb88bdc7bc8e6dd775ebde60b5e0 \ + --hash=sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e \ + --hash=sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac \ + --hash=sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9 \ + --hash=sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7 \ + --hash=sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35 \ + --hash=sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb \ + --hash=sha256:5cf4e27da7e3fbed4d6c3d8e797387aaad68102272f8f9752883bc32d61cb87b \ + --hash=sha256:5e0b74767e5f8c593e8c9b5912019159ed0533c70051e9cce3e8b6aa699fcd69 \ + --hash=sha256:5ed875a24292240029e4483f9d4a4b8a1ae08843b9c54f43fcc11e404532a8a5 \ + --hash=sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b \ + --hash=sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c \ + --hash=sha256:6344df0d5755a2c9a276d4473ae6b90647e216ab4757f8426893b5dd2ac3f369 \ + --hash=sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd \ + --hash=sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824 \ + --hash=sha256:66291b10affd76d76f54fad28e22e51719ef9ba22b29e1d7d03d6777a9174198 \ + --hash=sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065 \ + --hash=sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c \ + --hash=sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c \ + --hash=sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764 \ + --hash=sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196 \ + --hash=sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b \ + --hash=sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00 \ + --hash=sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac \ + --hash=sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8 \ + --hash=sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e \ + --hash=sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28 \ + --hash=sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3 \ + --hash=sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5 \ + --hash=sha256:9c57bb8c96f6d1808c030b1687b9b5fb476abaa47f0db9c0101f5e9f394e97f4 \ + --hash=sha256:9c7708761fccb9397fe64bbc0395abcae8c4bf7b0eac081e12b809bf47700d0b \ + --hash=sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf \ + --hash=sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5 \ + --hash=sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702 \ + --hash=sha256:b30236e45cf30d2b8e7b3e85881719e98507abed1011bf463a8fa23e9c3e98a8 \ + --hash=sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788 \ + --hash=sha256:b865addae83924361678b652338317d1bd7e79b1f4596f96b96c77a5a34b34da \ + --hash=sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d \ + --hash=sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc \ + --hash=sha256:bdb2c67c6c1390b63c6ff89f210c8fd09d9a1217a465701eac7316313c915e4c \ + --hash=sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba \ + --hash=sha256:c2514fceb77bc5e7a2f7adfaa1feb2fb311607c9cb518dbc378688ec73d8292f \ + --hash=sha256:c3355370a2c156cffb25e876646f149d5d68f5e0a3ce86a5084dd0b64a994917 \ + --hash=sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5 \ + --hash=sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26 \ + --hash=sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f \ + --hash=sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b \ + --hash=sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be \ + --hash=sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c \ + --hash=sha256:efd7b85f94a6f21e4932043973a7ba2613b059c4a000551892ac9f1d11f5baf3 \ + --hash=sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6 \ + --hash=sha256:fa160448684b4e94d80416c0fa4aac48967a969efe22931448d853ada8baf926 \ + --hash=sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0 + # via json-schema-for-humans +requests==2.34.2 \ + --hash=sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0 \ + --hash=sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed + # via json-schema-for-humans +typing-extensions==4.16.0 \ + --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ + --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 + # via typing-inspect +typing-inspect==0.9.0 \ + --hash=sha256:9ee6fc59062311ef8547596ab6b955e1b8aa46242d854bfc78f4f6b0eff35f9f \ + --hash=sha256:b23fc42ff6f6ef6954e4852c1fb512cdd18dbea03134f91f856a95ccc9461f78 + # via dataclasses-json +urllib3==2.8.0 \ + --hash=sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3 \ + --hash=sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63 + # via requests diff --git a/scripts/requirements-docs.txt b/scripts/requirements-docs.txt new file mode 100644 index 0000000..60d601b --- /dev/null +++ b/scripts/requirements-docs.txt @@ -0,0 +1,32 @@ +# The HTML renderer behind specs//fluid-spec.html, pinned exactly. +# +# The rendered HTML is committed, and CI regenerates it to prove it is current +# (generate-docs.py --check), so the renderer is part of the build's input. +# Re-rendering the unchanged 0.7.2 schema with a later release than the one +# that produced the committed page rewrote its quote escaping throughout and +# added sections the earlier release had not rendered. generate-docs.py +# refuses to run with any other installed version of these packages. +# +# THIS FILE IS THE INPUT. Install from scripts/requirements-docs.lock, which +# holds these pins plus every transitive dependency, each with its hashes: +# +# pip install --require-hashes -r scripts/requirements-docs.lock +# +# so that a package replaced on PyPI after the fact is refused rather than run. +# generate-docs.py --check fails if the two files disagree. +# +# Move a pin deliberately: bump it here, recompile the lock (from the repository +# root, with the Python version CI uses, 3.12, because the lock is resolved for +# the interpreter that runs pip-compile): +# +# pip install pip-tools +# pip-compile --generate-hashes --strip-extras \ +# --output-file=scripts/requirements-docs.lock scripts/requirements-docs.txt +# +# then run `python3 generate-docs.py`, and commit the regenerated specs/ in the +# same pull request. +json-schema-for-humans==1.5.1 +Jinja2==3.1.6 +MarkupSafe==3.0.3 +markdown2==2.5.5 +Pygments==2.21.0 diff --git a/scripts/schema-versions.json b/scripts/schema-versions.json new file mode 100644 index 0000000..f4bc56c --- /dev/null +++ b/scripts/schema-versions.json @@ -0,0 +1,50 @@ +{ + "$comment": [ + "The one record of which vendored schema versions are kept in sync with the reference implementation, and of their status.", + "Read by generate-docs.py (which versions to render, and which one is the latest STABLE) and by scripts/check-schema-drift.py (which versions must equal the reference implementation's latest release, byte for byte).", + "latestStable is deliberately separate from the highest version number: a preview has the higher number and is never 'latest'.", + "check-schema-drift.py fails when this file disagrees with the release it compares against (a new bundled version, a preview promoted to stable), so editing it is part of every re-vendor, not a separate chore.", + "upstream.ref and upstream.commit are the forge-cli release the synced schemas were vendored from, and the one check-schema-drift.py compares them with. It fetches by the commit, so the result does not change when forge-cli publishes or moves a tag. Moving to a newer release is a change to these two fields, reviewed in a pull request. commit is the peeled commit of the tag (git ls-remote refs/tags/^{}), not the annotated tag object.", + "frozen maps every file under schema/ and specs/ that is NOT generated from a synced schema to its sha256: the versions before 0.7.2, which are not synced, and the HTML pages that earlier renderer releases produced for them. generate-docs.py --check verifies the hashes offline and fails on any file under schema/ or specs/ that is neither synced output nor listed here, so these cannot be edited, and nothing can be added beside them, unnoticed. Changing an entry is a deliberate edit to this file." + ], + "upstream": { + "repository": "Agenticstiger/forge-cli", + "schemaDir": "fluid_build/schemas", + "previewSource": "fluid_build/schema_manager.py", + "ref": "v0.18.1", + "commit": "93e78d4386c70b9e040283f9c5d98a89f901c986" + }, + "latestStable": "0.7.5", + "preview": ["0.7.6"], + "synced": ["0.7.2", "0.7.3", "0.7.4", "0.7.5", "0.7.6"], + "notSynced": { + "0.7.1": "The reference implementation bundles a different document under the same $id (it adds orchestration, icebergConfig, binding.properties and more). Re-vendoring it would change which documents the published 0.7.1 accepts, so it is a decision for the maintainers rather than a sync, and the published copy is left as it is.", + "0.0.1-0.5.7": "Predate the compatibility promise; the reference implementation no longer bundles them." + }, + "frozen": { + "schema/fluid-schema-0.0.1.json": "7415076c34e6faf3b523cc512d0afa270d1402dfe05377933eab82331db0b7f6", + "schema/fluid-schema-0.1.0.json": "6c9fccfd10f6f9c285a04d292f75f6a0c0f7ca419e5991b7509af31182bf6348", + "schema/fluid-schema-0.1.1.json": "5e1b5109b665c4e7f5d0501d9ac314ccc02379f391ad81dcf902f3d28a1de556", + "schema/fluid-schema-0.2.0.json": "3b48623e0e50db833ee55fdefe85b2b03791a84d88aad0d1985ba4c33cfe3c75", + "schema/fluid-schema-0.3.0.json": "b385967b11c9a63e18ea67a159a8785e2131c96d1c463ee1a7c6601cc3a49ec6", + "schema/fluid-schema-0.4.0.json": "5e762a15432a1a2d0d96a3a5740ce7fd75ba8d616b5c4d88e8e825d8b9cbc9d8", + "schema/fluid-schema-0.5.7.json": "2f6f775ff87692548f89567c38b831e2c9f7344e0a6deb7eb6cdc290712b2d46", + "schema/fluid-schema-0.7.1.json": "b9ed5f201a9462f3267e1ea2b51a4e51066540b965149849aacce2f0e20d9df5", + "specs/0.2.0/fluid-schema-0.2.0.json": "3b48623e0e50db833ee55fdefe85b2b03791a84d88aad0d1985ba4c33cfe3c75", + "specs/0.2.0/fluid-spec.html": "7f4a551399471ca1b457467b75b59546465d43fea54e4408de32c87f82d3d0a9", + "specs/0.2.0/schema_doc.css": "2768fd01a875d1f58273e76fef9b421986824d2ab02251853840bca777b625c0", + "specs/0.2.0/schema_doc.min.js": "165a29291d7d66653e46fd1ac5868f63e18f9d67c06ea9ebcb95d4830a528e68", + "specs/0.3.0/fluid-spec.html": "beb79323e1add68b06dfeefe4e47cd55488d76695ae53b4af8f04e71c14eb681", + "specs/0.3.0/schema_doc.css": "2768fd01a875d1f58273e76fef9b421986824d2ab02251853840bca777b625c0", + "specs/0.3.0/schema_doc.min.js": "165a29291d7d66653e46fd1ac5868f63e18f9d67c06ea9ebcb95d4830a528e68", + "specs/0.4.0/fluid-spec.html": "a2cbbed1f6a98c262e4e2714fe15dc459659ae72d63ef08d2bef222699ea12e4", + "specs/0.4.0/schema_doc.css": "2768fd01a875d1f58273e76fef9b421986824d2ab02251853840bca777b625c0", + "specs/0.4.0/schema_doc.min.js": "165a29291d7d66653e46fd1ac5868f63e18f9d67c06ea9ebcb95d4830a528e68", + "specs/0.5.7/fluid-spec.html": "696b60b249a3445d553f561c3e8e74639c3295c0de979709c40946b79f6592db", + "specs/0.5.7/schema_doc.css": "2768fd01a875d1f58273e76fef9b421986824d2ab02251853840bca777b625c0", + "specs/0.5.7/schema_doc.min.js": "165a29291d7d66653e46fd1ac5868f63e18f9d67c06ea9ebcb95d4830a528e68", + "specs/0.7.1/fluid-spec.html": "5ca45312c66eb6cdceb624dafaca88ced2d042b1f6b111b5f31bd81062884476", + "specs/0.7.1/schema_doc.css": "2768fd01a875d1f58273e76fef9b421986824d2ab02251853840bca777b625c0", + "specs/0.7.1/schema_doc.min.js": "165a29291d7d66653e46fd1ac5868f63e18f9d67c06ea9ebcb95d4830a528e68" + } +} diff --git a/scripts/sync-public-assets.mjs b/scripts/sync-public-assets.mjs index 23923db..75aa98b 100644 --- a/scripts/sync-public-assets.mjs +++ b/scripts/sync-public-assets.mjs @@ -8,7 +8,7 @@ // byte-for-byte. // // The canonical copies stay at the repo root (untouched by the Python -// generators and the schema-sync CI). These public/ mirrors are +// generators and the schema drift check). These public/ mirrors are // git-ignored and rebuilt on every `npm run docs:dev` / `docs:build` // via the npm "sync:assets" prebuild step. // ===================================================================== diff --git a/specs/0.7.2/fluid-spec.html b/specs/0.7.2/fluid-spec.html index 06b31b7..f563ab5 100644 --- a/specs/0.7.2/fluid-spec.html +++ b/specs/0.7.2/fluid-spec.html @@ -98,75 +98,75 @@

Example:

{
-    "fluidVersion": "0.5.7",
-    "kind": "DataProduct",
-    "id": "gold.customer.analytics_360_v1",
-    "name": "Customer 360 Analytics",
-    "description": "Unified customer profiles with ML-driven insights",
-    "domain": "Customer Experience",
-    "tags": [
-        "customer-data",
-        "analytics",
-        "gold-layer"
+    "fluidVersion": "0.5.7",
+    "kind": "DataProduct",
+    "id": "gold.customer.analytics_360_v1",
+    "name": "Customer 360 Analytics",
+    "description": "Unified customer profiles with ML-driven insights",
+    "domain": "Customer Experience",
+    "tags": [
+        "customer-data",
+        "analytics",
+        "gold-layer"
     ],
-    "labels": {
-        "team": "customer-analytics",
-        "criticality": "high",
-        "cost-center": "engineering"
+    "labels": {
+        "team": "customer-analytics",
+        "criticality": "high",
+        "cost-center": "engineering"
     },
-    "metadata": {
-        "layer": "Gold",
-        "owner": {
-            "team": "customer-analytics",
-            "email": "customer-analytics@company.com"
+    "metadata": {
+        "layer": "Gold",
+        "owner": {
+            "team": "customer-analytics",
+            "email": "customer-analytics@company.com"
         },
-        "businessContext": {
-            "domain": "Customer Experience",
-            "subdomain": "Customer Intelligence"
+        "businessContext": {
+            "domain": "Customer Experience",
+            "subdomain": "Customer Intelligence"
         }
     },
-    "exposes": [
+    "exposes": [
         {
-            "exposeId": "customer_profiles",
-            "title": "Unified Customer Profiles",
-            "version": "2.1.0",
-            "kind": "table",
-            "tags": [
-                "pii",
-                "customer-facing",
-                "real-time"
+            "exposeId": "customer_profiles",
+            "title": "Unified Customer Profiles",
+            "version": "2.1.0",
+            "kind": "table",
+            "tags": [
+                "pii",
+                "customer-facing",
+                "real-time"
             ],
-            "labels": {
-                "sensitivity": "high",
-                "retention": "7-years",
-                "region": "us-central1"
+            "labels": {
+                "sensitivity": "high",
+                "retention": "7-years",
+                "region": "us-central1"
             },
-            "contract": {
-                "schema": [
+            "contract": {
+                "schema": [
                     {
-                        "name": "customer_id",
-                        "type": "STRING",
-                        "required": true,
-                        "description": "Unique customer identifier",
-                        "sensitivity": "cleartext",
-                        "tags": [
-                            "identifier",
-                            "primary-key"
+                        "name": "customer_id",
+                        "type": "STRING",
+                        "required": true,
+                        "description": "Unique customer identifier",
+                        "sensitivity": "cleartext",
+                        "tags": [
+                            "identifier",
+                            "primary-key"
                         ],
-                        "labels": {
-                            "business-name": "Customer ID",
-                            "data-category": "identifier"
+                        "labels": {
+                            "business-name": "Customer ID",
+                            "data-category": "identifier"
                         }
                     }
                 ]
             },
-            "binding": {
-                "platform": "gcp",
-                "format": "bigquery_table",
-                "location": {
-                    "project": "company-data",
-                    "dataset": "gold_customer",
-                    "table": "profiles_v1"
+            "binding": {
+                "platform": "gcp",
+                "format": "bigquery_table",
+                "location": {
+                    "project": "company-data",
+                    "dataset": "gold_customer",
+                    "table": "profiles_v1"
                 }
             }
         }
@@ -205,7 +205,7 @@ 

 
         

 
Example:

-
"0.7.2"
+
"0.7.2"
 

 

             

@@ -484,13 +484,13 @@ 

         

 
Example:

 
{
-    "team": "customer-analytics",
-    "criticality": "high",
-    "cost-center": "engineering",
-    "retention": "7-years",
-    "region": "us-central1",
-    "business-name": "Customer Profiles",
-    "data-category": "analytical"
+    "team": "customer-analytics",
+    "criticality": "high",
+    "cost-center": "engineering",
+    "retention": "7-years",
+    "region": "us-central1",
+    "business-name": "Customer Profiles",
+    "data-category": "analytical"
 }
 

 

@@ -1604,44 +1604,44 @@ 

         

 
Example:

 
{
-    "exposeId": "customer_profiles",
-    "title": "Customer Profiles Table",
-    "kind": "table",
-    "tags": [
-        "pii",
-        "customer-facing",
-        "analytical"
+    "exposeId": "customer_profiles",
+    "title": "Customer Profiles Table",
+    "kind": "table",
+    "tags": [
+        "pii",
+        "customer-facing",
+        "analytical"
     ],
-    "labels": {
-        "sensitivity": "high",
-        "retention": "7-years",
-        "team": "customer-analytics"
+    "labels": {
+        "sensitivity": "high",
+        "retention": "7-years",
+        "team": "customer-analytics"
     },
-    "contract": {
-        "schema": [
+    "contract": {
+        "schema": [
             {
-                "name": "customer_id",
-                "type": "STRING",
-                "required": true,
-                "sensitivity": "cleartext",
-                "tags": [
-                    "identifier",
-                    "primary-key"
+                "name": "customer_id",
+                "type": "STRING",
+                "required": true,
+                "sensitivity": "cleartext",
+                "tags": [
+                    "identifier",
+                    "primary-key"
                 ],
-                "labels": {
-                    "business-name": "Customer ID",
-                    "data-category": "identifier"
+                "labels": {
+                    "business-name": "Customer ID",
+                    "data-category": "identifier"
                 }
             }
         ]
     },
-    "binding": {
-        "platform": "gcp",
-        "format": "bigquery_table",
-        "location": {
-            "project": "company-data",
-            "dataset": "customer",
-            "table": "profiles"
+    "binding": {
+        "platform": "gcp",
+        "format": "bigquery_table",
+        "location": {
+            "project": "company-data",
+            "dataset": "customer",
+            "table": "profiles"
         }
     }
 }
@@ -8247,19 +8247,19 @@ 

 
         

 
Examples:

-
"gpt-4"
+
"gpt-4"
 

-
"claude-3-opus"
+
"claude-3-opus"
 

-
"claude-3-sonnet"
+
"claude-3-sonnet"
 

-
"gemini-pro"
+
"gemini-pro"
 

-
"gemini-ultra"
+
"gemini-ultra"
 

-
"llama-3"
+
"llama-3"
 

-
"mixtral-8x7b"
+
"mixtral-8x7b"
 

 

         

@@ -8637,15 +8637,15 @@ 
Must be one of:

     


 
Examples:

 
[
-    "inference",
-    "reasoning",
-    "summarization"
+    "inference",
+    "reasoning",
+    "summarization"
 ]
 

 
[
-    "search",
-    "qa",
-    "rag"
+    "search",
+    "qa",
+    "rag"
 ]
 

 

@@ -8772,8 +8772,8 @@ 
Must be one of:

     


 
Example:

 
[
-    "training",
-    "fine_tuning"
+    "training",
+    "fine_tuning"
 ]
 

 

@@ -15435,67 +15435,67 @@ 

         

 
Example:

 
{
-    "name": "orders_revenue_model",
-    "description": "Canonical semantic model for revenue analytics",
-    "defaultAggTimeDimension": "order_date",
-    "entities": [
+    "name": "orders_revenue_model",
+    "description": "Canonical semantic model for revenue analytics",
+    "defaultAggTimeDimension": "order_date",
+    "entities": [
         {
-            "name": "order",
-            "type": "primary",
-            "expr": "order_id"
+            "name": "order",
+            "type": "primary",
+            "expr": "order_id"
         },
         {
-            "name": "customer",
-            "type": "foreign",
-            "expr": "customer_id"
+            "name": "customer",
+            "type": "foreign",
+            "expr": "customer_id"
         }
     ],
-    "measures": [
+    "measures": [
         {
-            "name": "order_amount",
-            "agg": "sum",
-            "expr": "amount",
-            "description": "Sum of all order amounts"
+            "name": "order_amount",
+            "agg": "sum",
+            "expr": "amount",
+            "description": "Sum of all order amounts"
         },
         {
-            "name": "order_count",
-            "agg": "count_distinct",
-            "expr": "order_id",
-            "createMetric": true
+            "name": "order_count",
+            "agg": "count_distinct",
+            "expr": "order_id",
+            "createMetric": true
         }
     ],
-    "dimensions": [
+    "dimensions": [
         {
-            "name": "order_date",
-            "type": "time",
-            "expr": "completed_at",
-            "typeParams": {
-                "timeGranularity": "day"
+            "name": "order_date",
+            "type": "time",
+            "expr": "completed_at",
+            "typeParams": {
+                "timeGranularity": "day"
             }
         },
         {
-            "name": "region",
-            "type": "categorical"
+            "name": "region",
+            "type": "categorical"
         }
     ],
-    "metrics": [
+    "metrics": [
         {
-            "name": "net_revenue",
-            "description": "GAAP-compliant net revenue: completed orders minus discounts, excluding refunds",
-            "type": "derived",
-            "inputMetrics": [
-                "gross_revenue",
-                "total_discounts"
+            "name": "net_revenue",
+            "description": "GAAP-compliant net revenue: completed orders minus discounts, excluding refunds",
+            "type": "derived",
+            "inputMetrics": [
+                "gross_revenue",
+                "total_discounts"
             ],
-            "expr": "gross_revenue - total_discounts",
-            "owner": "finance-data@company.com"
+            "expr": "gross_revenue - total_discounts",
+            "owner": "finance-data@company.com"
         },
         {
-            "name": "gross_revenue",
-            "description": "Total value of completed orders",
-            "type": "simple",
-            "measure": "order_amount",
-            "filter": "status = 'completed'"
+            "name": "gross_revenue",
+            "description": "Total value of completed orders",
+            "type": "simple",
+            "measure": "order_amount",
+            "filter": "status = 'completed'"
         }
     ]
 }
@@ -20404,18 +20404,18 @@ 

         

     

 

-

+

     

-        

+        

             

-                
+                
             

         

 
-        

+        

             

 
     
root
@@ -20467,27 +20467,32 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    vars
Type: object

-
+    target
Type: string

+
dbt profile target to run against. Forwarded to dbt as --target <name> by the build runner (allowlisted against the profile's real targets; falls back to the profile default when the requested target is unavailable).

+
         
 
         
         
 
         
-

+            

+        

+    

+

+
-

-

-    

-        

-            

-                
-            

-        

-
-        

-            

+    select


+
dbt node selector(s) forwarded to dbt --select. Accepts a single selector string or an array of selector strings; the build runner drops flag-like values (those starting with '-').

+

+  
+

+

+            
 
     
root
         
@@ -20624,7 +20611,21 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    materializations
Type: object

+    select
+        
+            
+        
+    oneOf
+        
+            
+        
+    item 0
Type: string

 
         
 
@@ -20632,19 +20633,9 @@ 

         
 
         
-

-    

-        

-            

-                
-            

-        

-
-        

-            
Each additional property must conform to the following schema

+        

+            
 
     
root
         
@@ -20695,45 +20686,31 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    materializations
+    select
         
             
         
-    additionalProperties
Type: enum (of string)

-

-            
Must be one of:

-            
  • "table"
  • "view"
  • "incremental"
  • "ephemeral"

-            

+    oneOf
+        
+            
+        
+    item 1
Type: array of string

+
         
 
         
         
 
-        
-            

-        

-    

-

-            

-        

-    

-

-

+         No Additional Items
Each item of this array must be:

     

-        

-            

-                
-            

-        

-
-        

-            

+        

+            
 
     
root
         
@@ -20784,25 +20761,60 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    tags
Type: array of string

-
Simple string tags for categorization, discovery, and automation. Consistent pattern used throughout FLUID objects.

-Same definition as tags
+    select
+        
+            
+        
+    oneOf
+        
+            
+        
+    item 1
+        
+            
+        
+    item 1 items
Type: string

+
+        
+
+        
+        
+
+        
+        

+    

+        

+        
+
+        
+        
+
+        
             

         

     

 

-

+

     

-        

+        

             

-                
+                
             

         

 
-        

+        

             

 
     
root
@@ -20854,26 +20866,23 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    labels
Type: object

-
Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.

-Same definition as labels
-            

-        

-    

-

-            

-        

-    

-

-        

-        
-
-        
-        
-
-        
-        

+    models


+
Legacy alias for select (dbt's older --models flag). Same string-or-array shape; select takes precedence when both are set.

+

+  
+

+

             
 
     
root
@@ -20904,72 +20913,42 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
Type: object

-

-    
-

-
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected.
-    Otherwise, the conditions in the "Else" tab should be respected.

-
-
-

-    
-    

-        
-
-    
root
+    item 0
         
             
         
-    builds
+    then
         
             
         
-    builds items
+    properties
         
             
         
-    allOf
+    models
         
             
         
-    item 1
+    oneOf
         
             
         
-    if
Type: object

+    item 0
Type: string

 
         
 
@@ -20977,19 +20956,9 @@ 

         
 
         
-

-    

-        

-            

-                
-            

-        

-
-        

-            

+        

+            
 
     
root
         
@@ -21019,38 +20988,51 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
+    item 0
         
             
         
-    if
+    then
         
             
         
-    pattern
Type: const

-Specific value: "embedded-logic"
-        
+    properties
+        
+            
+        
+    models
+        
+            
+        
+    oneOf
+        
+            
+        
+    item 1
Type: array of string

 
         
-        
 
         
-            

-        

-    

-

-    

-
-    
         
-        

+
+         No Additional Items
Each item of this array must be:

+    

+        

             
 
     
root
@@ -21081,100 +21063,81 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
-        
-            
-        
-    then
Type: object

-
-        
-
-        
-        
-
-        
-

-    

-        

-            

-                
-            

-        

-
-        

-            

-
-    
root
+    item 0
         
             
         
-    builds
+    then
         
             
         
-    builds items
+    properties
         
             
         
-    allOf
+    models
         
             
         
-    item 1
+    oneOf
         
             
         
-    then
+    item 1
         
             
         
-    properties
Type: object

-
Embedded logic pattern from v0.4.0.

-
+    item 1 items
Type: string

 
-     No Additional Properties
         
 
         
         
 
         
-

+        

+    

+        

+        
+
+        
+        
+
+        
+            

+        

+    

+

+

     

-        

+        

             

-                
+                
             

         

 
-        

+        

             

 
     
root
@@ -21205,28 +21168,28 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
+    item 0
         
             
         
-    then
+    then
         
             
         
-    properties
+    properties
         
             
         
-    sql
Type: string

+    vars
Type: object

 
         
 
@@ -21234,23 +21197,19 @@ 

         
 
         
-            

-        

-    

-

-

+

     

-        

+        

             

-                
+                
             

         

 
-        

-            

+        

+            
Additional Properties of any type are allowed.

 
     
root
         
@@ -21280,32 +21239,36 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
+    item 0
         
             
         
-    then
+    then
         
             
         
-    properties
+    properties
         
             
         
-    language
Type: enum (of string)

-

-            
Must be one of:

-            
  • "sql"
  • "flink_sql"
  • "pyspark"
  • "scala"
  • "python"
  • "r"

-            

+    vars
+        
+            
+        
+    additionalProperties
Type: object

+
         
 
         
@@ -21316,18 +21279,22 @@ 
Must be one of:

         

     

 

-

+            

+        

+    

+

+

     

-        

+        

             

-                
+                
             

         

 
-        

+        

             

 
     
root
@@ -21358,28 +21325,28 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
+    item 0
         
             
         
-    then
+    then
         
             
         
-    properties
+    properties
         
             
         
-    parameters
Type: object

+    materializations
Type: object

 
         
 
@@ -21387,19 +21354,19 @@ 

         
 
         
-

+

     

-        

+        

             

-                
+                
             

         

 
-        

-            
Additional Properties of any type are allowed.

+        

+            
Each additional property must conform to the following schema

 
     
root
         
@@ -21429,36 +21396,39 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
+    item 0
         
             
         
-    then
+    then
         
             
         
-    properties
+    properties
         
             
         
-    parameters
+    materializations
         
             
         
-    additionalProperties
Type: object

-
+    additionalProperties
Type: enum (of string)

+

+            
Must be one of:

+            
  • "table"
  • "view"
  • "incremental"
  • "ephemeral"

+            

         
 
         
@@ -21473,18 +21443,18 @@ 

         

     

 

-

+

     

-        

+        

             

-                
+                
             

         

 
-        

+        

             

 
     
root
@@ -21515,46 +21485,46 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
+    item 0
         
             
         
-    then
+    then
         
             
         
-    properties
+    properties
         
             
         
-    tags
Type: array of string

+    tags
Type: array of string

 
Simple string tags for categorization, discovery, and automation. Consistent pattern used throughout FLUID objects.

 Same definition as tags
             

         

     

 

-

+

     

-        

+        

             

-                
+                
             

         

 
-        

+        

             

 
     
root
@@ -21585,28 +21555,28 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 1
+    item 0
         
             
         
-    then
+    then
         
             
         
-    properties
+    properties
         
             
         
-    labels
Type: object

+    labels
Type: object

 
Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.

 Same definition as labels
             

@@ -21625,7 +21595,7 @@ 

 
         
         

+             id="tab-pane_builds_items_allOf_i1" role="tabpanel">
             
 
     
root
@@ -21656,18 +21626,18 @@ 

                 d="M4 8a.5.5 0 0 1 .5-.5h5.793L8.146 5.354a.5.5 0 1 1 .708-.708l3 3a.5.5 0 0 1 0 .708l-3 3a.5.5 0 0 1-.708-.708L10.293 8.5H4.5A.5.5 0 0 1 4 8z"
             />
         
-    item 2
Type: object

+    item 1
Type: object

 

     
 

 
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected.
     Otherwise, the conditions in the "Else" tab should be respected.

-