feat(databricks): fluid genie apply, one Genie space over several data products

.github/workflows/iac-tests.yml

@@ -214,13 +214,15 @@ jobs:
               tests/iac/test_iac_databricks_fake_e2e.py
               tests/iac/test_iac_databricks_jobs_fake_e2e.py
               tests/iac/test_iac_databricks_genie_fake_e2e.py
+              tests/iac/test_iac_databricks_genie_estate_fake_e2e.py
               tests/iac/test_iac_databricks_metrics_fake_e2e.py
             pytest-args: "-p no:randomly"
             junit: databricks-fake.xml
             require: |
               Databricks fake tier=tests/iac/test_iac_databricks_fake_e2e.py
               Databricks jobs fake tier=tests/iac/test_iac_databricks_jobs_fake_e2e.py
               Databricks Genie fake tier=tests/iac/test_iac_databricks_genie_fake_e2e.py
+              Databricks Genie estate fake tier=tests/iac/test_iac_databricks_genie_estate_fake_e2e.py
               Databricks metric views fake tier=tests/iac/test_iac_databricks_metrics_fake_e2e.py
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1

AGENTS.md

@@ -381,6 +381,27 @@ changing `parentPath`, leaves the space in the old folder with a note naming it.
 `databricks_genie_space` action, and `fluid generate iac` notes that its module holds none.
 Code: `fluid_build/iac/providers/databricks_genie.py`.
 
+### One Genie space over several products (an estate)
+
+```bash
+fluid genie apply tower-management/ --estate tower-management --title "Tower Management"
+fluid genie remove --estate tower-management
+```
+
+`fluid genie apply` takes contract files, globs or folders (searched for `*.fluid.yaml`), and
+`--env` and `--dry-run` as `fluid apply` does; each contract passes the same schema and
+version gate. Apply the products first: the run checks every table exists and names the
+product of one that does not. Genie adds a join of its own for each Unity Catalog foreign key
+that touches a table entering a space; the run reads each write back and takes those out, so
+a second run changes nothing. The space holds each product's tables
+as a product space does, and a join for every foreign key the contracts declare between them
+(a column tagged `foreign-key` whose `labels.references` is `<product id>.<column>`), the same
+keys `fluid apply` declares in Unity Catalog (`databricks_keys.py`). `CAN RUN` goes only to the
+principals that read every table of the estate, because the space describes them all. The space
+is found again by `[fluid-genie-estate:<id>]` at the end of its description, in the deploying
+identity's own folder, and converges to exactly the contracts given. Code:
+`fluid_build/cli/genie.py`, `fluid_build/iac/providers/databricks_genie_estate.py`.
+
 ---
 
 ## For AI-Assisted Project Creation (Forge)

CHANGELOG.md

@@ -9,6 +9,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **`fluid genie apply`: one Databricks Genie space over several data products (an estate).**
+  `fluid genie apply <contracts|globs|folders...> --estate <id>` creates or updates one AI/BI
+  Genie space over the Databricks tables of every contract given, so a question can join the
+  products: `fluid genie apply tower-management/ --estate tower-management`. Each table carries
+  what a product's own space says of it (column descriptions, allowed values, synonyms, example
+  questions, one instruction section from its semantics, metrics and `agentPolicy`), and the space
+  gets a join for every foreign key the contracts declare between its tables: a column tagged
+  `foreign-key` whose `labels.references` is `<product id>.<column>`, resolved as `fluid apply`
+  resolves it for Unity Catalog. A join is MANY_TO_ONE from the referencing table, ONE_TO_ONE
+  when the foreign key is that table's own primary key, a composite key is one join, and a key
+  to its own table joins a second alias. A reference to a product the estate does not hold is
+  said and left out. `CAN RUN` goes only to the principals that read every table of the estate,
+  since the space describes them all; the others are named. The products are applied first: the
+  run checks that every table exists before it writes and names the product of one that does
+  not. Like a product's space it is found again by a marker, `[fluid-genie-estate:<id>]`, in the
+  deploying identity's own folder only, changes nothing on a second run, and converges to
+  exactly the contracts given. `--title`, `--env` (contract overlays) and `--dry-run`;
+  `fluid genie remove --estate <id>` trashes it. Each contract passes the schema and version
+  gate `fluid apply` runs, and an unreadable one is `contract_load_failed`, naming the file. A
+  product whose `agentPolicy` rules out question answering refuses the estate, and so do more
+  than 50 tables, Genie's cap (measured). Whenever a table enters a space, Genie adds a join of
+  its own for each Unity Catalog foreign key that touches it (measured): a run reads each
+  write back, takes those duplicates out, and says so, so the next run finds no change. This
+  holds for a product's own space too. A product space's update now names a table that
+  leaves the space as a table, not as a metric view.
+
 - **A Databricks AI/BI Genie space for a data product.** With `extensions.databricksGenie: {}`
   (or `{title: ...}`) in a contract with Databricks table exposes, `fluid apply` creates the
   product's Genie space once its tables, masks and row filters are in place. The space holds
@@ -28,7 +54,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   with `parentPath` (below): a space another identity made in its own folder is left alone.
   `fluid apply --mode dry-run` prints what would change. The space is made through the Genie
   REST API (the `[databricks]` extra), since the databricks/databricks provider has no Genie
-  space resource. One space per product; a space over several products is a follow-up.
+  space resource. One space per product; `fluid genie apply` makes one over several products.
 
 - **One Genie space for a product deployed from more than one identity.**
   `extensions.databricksGenie.parentPath` names a workspace folder, such as `/genie/<domain>`.

HONESTLY_TESTED.md

@@ -102,6 +102,9 @@ Markers declared in `pyproject.toml`.
 | Stage 1 real tofu, stand-in workspace: Genie space | `tests/iac/test_iac_databricks_genie_fake_e2e.py` | ✅ 4. The real `fluid` CLI and `tofu`: validate, dry-run, apply (the space after the tables, shared with the readers), a second apply that changes nothing, the block removed (trashed); a Genie failure fails the apply after the tables with its fix and the next apply converges; an invalid block refused before `tofu`; `parentPath`: a missing folder stops the apply before `tofu` writes a table, then the space made in the folder and a second identity's apply changing nothing | The stand-in is modelled on the measured API, not the service itself |
 | Manual live, Free Edition, Genie space, 2026-11-12 | the real `fluid` CLI, no test file: branch `feat/databricks-genie-space`, databricks-sdk 0.162.0, a copy of industry-models tower-management `dp019_site_360` in the scratch catalog `fluid_iactest_genie` (no dbt build, no MENA sovereignty block: the workspace is in us-west-2), 6 synthetic rows | ✅ dry-run (would create, would share), apply `+73` then the space created and shared `CAN_RUN` with two persona service principals, a second apply `+0 ~0 -0` and "no change", the question "Which on-air sites in Amman are colocation-ready with hybrid power and no standing risk?" answered through the Genie Conversation API with the 2 expected sites (its SQL took the latest `snapshot_date` and the inverse of the `at_risk_site_count` filter from the instruction), a metric view created by hand picked up ("updated: metric views", then "no change"), the block removed (trashed). Two defects found and fixed in the run: the metric-view read failed on a schema not created yet, and the service keeps a metric view among the tables. A re-check after the security review (the plugin's own hook over `samples.nyctaxi.trips`, nothing created in Unity Catalog) found the list shows a new space only ~9 s later (a run straight after made a second space) and that a taken title is suffixed on create and 409 on PATCH; fixed, then create, an immediate second run ("no change", same space), and the block removed (trashed, via the folder check) passed live. Swept: the space trashed, `DROP CATALOG fluid_iactest_genie CASCADE`, 0 left | A reader without `SELECT` asking (no persona credential here; the access model is Genie's documented one); a service principal as the deploying identity (its folder is `/Users/<application id>`, the one Databricks Asset Bundles build from `current_user.userName`; a create that lands anywhere else is trashed and stops the apply); a space over several products |
 | Manual live, Free Edition, Genie `parentPath`, 2026-11-12 | the real `fluid` CLI, no test file: branch `feat/databricks-genie-shared-folder` on main 97811add, databricks-sdk 0.162.0, a one-table product in the scratch catalog `fluid_iactest_genie_pp`, its folder `/fluid_iactest_genie_pp` at the workspace root | ✅ A probe measured the API first. A create accepts `/Users/<me>`, `/Workspace/Users/<me>` (accepted this time; an earlier probe that day had a 400), `/Shared/x`, `/Workspace/Shared/x`, a trailing `/` and a root-level folder, and a GET returns the bare form. A missing folder is 404, a relative path 400, and `/shared/x` 404 (paths are case-sensitive). `/Shared` grants `users` `CAN_MANAGE` directly, and every folder and space under it inherits that; the root grants only admins. A space's permissions all come from its folder, and `PATCH {parent_path}` moves it. Then through the CLI: validate refuses `/Workspace/Shared/...`; an apply with the folder missing stops after `tofu plan`, before `tofu apply` (no schema made); a space made without `parentPath` in the user's folder, a dry run says it would move, the apply moves it (same id), and a second apply changes nothing; `users` with `CAN_MANAGE` on the folder is refused before `tofu apply`, and `users` with `CAN_RUN` is a note; `fluid plan` lists `genie_space_<id>`, `generate iac` prints the note, and `diff --exit-on-drift` exits 0 with the space among `added`; with the block removed, the folder's space is left with a note. Swept: the space trashed, the folder deleted, the catalog dropped, 0 left | A second identity, live (no service-principal secret here; the stand-in e2e covers it); a folder a non-admin made inside someone's home |
+| Stage 1 unit: Databricks Genie estate (`fluid genie apply`) | `tests/iac/test_iac_databricks_genie_estate.py` (the compile, `converge_estate` and `remove_estate` against the stand-in, `fluid genie` through the real entry point) | ✅ 40. Every product's table in one space, sorted; a join per declared foreign key between them (MANY_TO_ONE from the referencing table, ONE_TO_ONE when it is that table's primary key, one join for a composite key, a second alias for a key to its own table, every column qualified, 32-hex ids sorted); the questions once each; one instruction with a section per product; no measure snippets; CAN_RUN only for the readers of every table, the others named; a reference outside the estate, to a keyless product or to no single primary key said, and a nullable key column or a garbled reference said as `fluid apply` says them; the refusals (an id the marker cannot carry, no contract, no id, a product twice, no Databricks table, a table in two products, an `agentPolicy` against Genie, more than 50 tables, a text past 25,000 characters, a refusal every contract shares said once); create and share, then no change; the joins Genie adds from Unity Catalog keys on a create and on an update that adds a table taken out at once; a product left out updates the space; a table not applied yet named with its product before any write; dry run; remove trashes only the estate's space and a failed one names `fluid genie remove`; a product space and an estate space never touch; two marker copies stop the run; a planted copy left alone; the estate's own command in its messages and the product's wording kept; the stand-in refusing an unknown relationship tag and a 51st table; contract globs (one level) and folders once each, past no symlink or ignored folder; a schema-invalid, pre-0.7 or unreadable contract a typed error naming its file | Genie's answers (only the live row asks one) |
+| Stage 1 real tofu, stand-in workspace: Genie estate | `tests/iac/test_iac_databricks_genie_estate_fake_e2e.py` | ✅ 1. Two products applied with the real `fluid apply` and `tofu` (their tables and the Unity Catalog key between them); `fluid genie apply` before the second exists refused, naming it; dry run; the space with the join from the declared foreign key, shared with the reader of both tables only; a second run changes nothing; `fluid genie remove` | The stand-in is modelled on the measured API, not the service itself |
+| Manual live, Free Edition, Genie estate, 2026-11-12/13 | the real `fluid` CLI, no test file: branch `feat/databricks-genie-estate`, databricks-sdk 0.162.0, the 19 tower-management products as fluid-demo-lab-run's `databricks_only_forge` track derives them (overlays to us-west-2 and the persona principals; builds left out; the analysts taken off 3 restricted products), each applied with `fluid apply` into the scratch catalog `fluid_iactest_estate` (19 runs, rc 0, the Unity Catalog keys declared), 7 synthetic rows | ✅ dry run (19 tables, 57 sample questions, 24 joins; CAN_RUN for the steward only, the analyst named, 16 of 19), create and share, a question through the Genie Conversation API ("Which site name has the most unbilled kWh?": Basra Port, 1500, by a JOIN of `tenant_billing` to `site_master` on `site_id`), the next run "no change", a product left out (updated: description, sample questions, tables, instructions, joins) and put back, remove, and remove again (nothing to trash). Found and fixed in the run: whenever a table enters a space Genie adds a join of its own for each Unity Catalog foreign key that touches it (24 sent, 48 held; 1 more when the product came back), so the second run updated; each write is now read back and set back. Probes, each space trashed at once: join_specs kept as sent by an update with the same tables, a self-join under two aliases and ONE_TO_ONE taken, an unknown relationship tag refused, 51 tables refused ("A space can have at most 50 tables."), 60 sample questions and a 60,000-character instruction taken. Swept: `DROP CATALOG fluid_iactest_estate CASCADE`; 0 `fluid_iactest_*` catalogs and 0 such spaces left | A reader without SELECT asking (no persona credential here); a service principal as the deploying identity; Genie's answers over more questions; an estate of more than 19 tables |
 | Stage 2 emulator (nightly, non-gating) | `tests/iac/test_iac_databricks_minilake.py` (job `nightly-databricks-minilake`) | ⚠️ 1 passed, 4 strict xfails (manual dispatch, run 42267640334, 2026-11-06, minilake v1.9.0). Schema, table and grant creates work, the replace gate holds, adding a column plans no replace, and a grant revoke is classified not gated. The four tests about those resources apply the module without its tag assignments (the `untagged` fixture). An xfail is satisfied only by its own gap's failure (`MinilakeGap`), and a pass fails the run, so an image that closes a gap turns the job red until the mark comes off. Earlier: 1/4 from the first run (2026-10-30) to the 2026-11-04 nightly, then 0/4 after #831, every apply stopping at the first tag. | A second plan is not empty and no in-place table update applies: minilake drops a CREATE TABLE's `TBLPROPERTIES` and `COMMENT`, and DuckDB cannot parse the `ALTER TABLE ... SET TBLPROPERTIES` the provider sends on every update of a table that sets properties. No tag is applied: minilake has no entity-tag-assignments API (it answers 501). v1.9.3 (2026-11-05) has both gaps, so the pin stays; an upstream issue for each is drafted, not filed. Jobs and workspace files are not in this flow yet. |
 | Stage 1 unit: `fluid test` on Databricks, results table, dashboard | `tests/providers/test_databricks_validation.py`, `tests/providers/test_databricks_statements.py`, `tests/providers/test_quality_engine_column_rules.py`, `tests/cli/test_contract_validation_schema_rules.py`, `tests/cli/test_fluid_test_results_table.py`, `tests/cli/test_fluid_test_dashboard.py` | ✅ Against a stubbed `api_client.do`: the Unity Catalog columns and the contract's types through the emitter's type table (`VARCHAR` is `string`), a missing table (404 → `missing_resource`, and every rule reported as not run with no statement), the row count, the warehouse check; the statements over `databricks_enforce.SqlRunner` with `:name` parameters, polling, timeout cancel, chunks, type conversion and scrubbed errors; every rule kind in the Databricks dialect (backticked identifiers, `\'` literals, `timestampdiff` freshness, `RLIKE`), comma-listed columns, values from the column's enum, regex or range rule, references resolved by the `fluid_product_id` tag (same catalog first, ambiguity refused, cached); `schema` rules from the column comparison; the result rows (one per JUnit case, `rule_type` `structure` for the structural ones, expose version, UTC run time), the batched parameterised `INSERT`, the refusals (bad name, a product's own schema, a table without the columns), the fix-its; the dashboard definition's structure and the create, unchanged, re-publish, update-with-etag and create-race paths. | What Databricks accepts: the manual live row below. |
 | Manual live, Free Edition (founder-approved), 2026-11-12: `fluid test` results | the real `fluid` CLI, no test file: `fluid test <contract> --env databricks --output junit --results-table fluid_iactest_dqr.dq.fluid_test_results --results-dashboard '/Workspace/Shared/fluid_iactest_dqr/Data quality'` on the 19 tower-management products in `northwind_towers` (read only: `SELECT` and metadata reads), contracts derived from industry-models main | ✅ 19 runs, 17-34 s each, 18 exit 0; `dp019_site_360` exits 1 because its table is not deployed in the demo workspace (`missing_resource`; its rules are reported as not run, and no statement is sent). 168 rule cases: the 155 on deployed tables all ran (146 pass, 9 freshness rules fail at `warn`: the bronze data is hours old against a 15-minute window), every rule kind the pack uses among them: enum, regex and range values (#4's range rule over five columns included), 23 references (3 of them bronze references outside `consumes[]`, resolved by the `fluid_product_id` tag), `schema` rules and comma-listed completeness rules. 396 rows appended, one per JUnit case; the dashboard was created by the first run, found unchanged by every later one, and updated (PATCH, re-publish) once when its definition changed; in a browser it shows the counters (88.1% of rules passing, 19 products, 20 failing rules at that point), the per-product bars by layer, the trend and the failing-checks table. Afterwards the catalog `fluid_iactest_dqr` was dropped CASCADE and the dashboard deleted. | Lakeview validates widget specs only when the dashboard renders, so a widget shape Databricks changes would show as "Invalid widget definition" without failing the command (the table widget did, until it carried every column key). |