feat(databricks): fluid genie apply, one Genie space over several data products by fas89 · Pull Request #870 · Agenticstiger/forge-cli

fas89 · 2026-11-12T23:36:30Z

What

fluid genie apply <contracts|globs|folders...> --estate <id> makes one Databricks AI/BI Genie
space over the Databricks tables of a set of data products, so one question can join them:

fluid genie apply tower-management/ --estate tower-management --title "Tower Management" --env databricks
fluid genie remove --estate tower-management

It follows the per-product space (#863) and reuses its compile helpers, its identity rule and
its converge:

Tables. Each table carries what a product space says of it: the column descriptions,
allowed values and synonyms, the product's example questions as sample questions, and an
instruction section with its semantics, metrics and agentPolicy.
Joins. The space adds instructions.join_specs, one per foreign key the contracts
declare between its tables. A foreign key is a column tagged foreign-key whose
labels.references is <product id>.<column>. feat(databricks): Unity Catalog primary and foreign keys from the contract #860's databricks_keys.foreign_keys
resolves it among the estate's own tables, so each join is the Unity Catalog FOREIGN KEY
fluid apply declares.
- A join is MANY_TO_ONE from the referencing table, or ONE_TO_ONE when the key is that
  table's own primary key.
- A composite key gives one join, and a key to its own table joins a second alias.
- Every column is qualified by its alias.
- A reference with no join (outside the estate, a keyless or nullable parent, a garbled
  label) is reported.
Who may ask (founder decision). CAN RUN goes only to the principals that read every
table of the estate, because the space describes all of them. The others are named.
Identity. The space is found by the marker [fluid-genie-estate:<id>], only in the
deploying identity's own folder. It changes nothing on a second run and converges to
exactly the contracts given. fluid genie remove trashes it.
Order of work. Products are applied first: each run checks every table exists before it
writes, and names the product to apply.
Refusals. A product whose agentPolicy rules out question answering refuses the estate,
and so does a set of more than 50 tables.
Product space fix. An update now names a table that leaves the space as a table, not
as a metric view.

Why

This is the follow-up #863 deferred: an estate space over several products, with joins from
the products' declared keys. The founder's design calls of 2026-11-12 were this flag shape
(not a manifest file or a pack reader), and CAN RUN only for the readers of every table.

Borrowed

databricks/databricks-agent-skills skills/databricks-genie-agents/references/create-genie-agent.md:
the join_specs shape (a two-element sql with a --rt=FROM_RELATIONSHIP_TYPE_*-- tag),
qualified columns, id and sort rules. Its "30 tables" is stale: 51 was refused live and
50 was taken.
apache/ossie (OSI) core-spec relationships[]: the many-to-one orientation (from is the
many side).
In repo: databricks_genie.py (feat(databricks): a Genie space for a data product (extensions.databricksGenie) #863), databricks_keys.py (feat(databricks): Unity Catalog primary and foreign keys from the contract #860), and fluid publish's
contract globbing.
Not used: Databricks Asset Bundles genie_spaces (the Go CLI's direct engine and its own
state) and terraform-provider-databricks databricks_genie_agent (v1.141.0, Private
Preview, a typed shape that refuses serialized_space).

Validation

Gates: whole-tree ruff, the dead-code gate, black 24.10.0, license headers,
import-linter (6 contracts kept), product guardrail, and detect-secrets 1.5.0 on the changed
files are all clean.
Unit: the Databricks unit tier, the help-style lint and the startup budget: 1,308
passed. Two tests in tests/cli/test_lazy_core_commands.py fail identically on the base
commit; their -I subprocess imports the shared checkout's fluid_build. The estate alone:
40 tests.
Stand-in tier: real tofu (offline provider mirror) and the real databricks-sdk 0.162.0
against _fake_databricks: on this tip the estate, product Genie and quality tiers passed (10). The full Databricks set (estate, Genie, Databricks, jobs, quality) passed 66 one rebase earlier, before feat(test): Databricks live checks, a Unity Catalog results table and an AI/BI dashboard #867 (fluid test live checks) merged.
Live, Free Edition (2026-11-12/13):
- Setup: the 19 tower-management products, as fluid-demo-lab-run's databricks_only_forge
  track derives them, were applied with fluid apply into the scratch catalog
  fluid_iactest_estate (19 runs, rc 0, Unity Catalog keys declared).
- Dry run: 19 tables, 57 sample questions, 24 joins. CAN RUN went to the steward only;
  the analyst, who reads 16 of 19 tables, was named.
- Create and share passed.
- One question through the Genie Conversation API: "Which site name has the most unbilled
  kWh?" was answered Basra Port, 1500, by a JOIN of tenant_billing to site_master on
  site_id.
- The next run reported "no change". A product left out and put back was handled. Remove
  trashed the space, and a second remove found nothing to trash.
- Found and fixed in the run: whenever a table enters a space, Genie adds a join of its own
  for each Unity Catalog foreign key touching it (24 sent, 48 held). The second run therefore
  updated; each write is now read back and set back.
- Probes, each space trashed at once: an unknown relationship tag is refused, 51 tables are
  refused, and a self-join and ONE_TO_ONE are taken.
- Swept with DROP CATALOG fluid_iactest_estate CASCADE: 0 catalogs and 0 spaces left.
ripwire --quality-delta: exit 2 on 35 rows, judged and not gamed. They are test-helper
idioms, the ContextVar idiom iac/source.py already uses, parallel tests, and
short-horizon churn on the feat(databricks): a Genie space for a data product (extensions.databricksGenie) #863 code this follow-up had to extend. The new functions that
went over the complexity bar were split.

Docs

Docs PR linked
No docs needed
Docs TODO: forge_docs needs a "Genie space over several products" section next to the product space's on the Databricks page. It should cover:
- the fluid genie apply <contracts|globs|folders> --estate <id> [--title] [--env] [--dry-run] and fluid genie remove --estate <id> commands;
- the joins built from declared foreign keys (foreign-key tag plus labels.references);
- who gets CAN RUN (readers of every table);
- how the space is found again (the marker, the own-folder rule);
- the 50-table cap, and that Genie's own joins from Unity Catalog keys are taken out.
It is not written yet; the needs-docs label tracks it.

Reviews

security-review: no HIGH or MEDIUM finding. An Opus judge confirmed this independently.
Both fix rounds were re-checked: no HIGH or MEDIUM.
world-class-inspection (diff mode): CONCERNS, from two P2 message defects: a nullable or
garbled key was reported wrongly, and the share note named the wrong command. The CLI
journey walk found three UX defects (fluid --help did not list genie, one
overlay_applied line per contract, and zero counts before a refusal). All five are fixed
and re-verified.
defect-scan (diff mode): 1 Medium and 5 Low, every one confirmed by a skeptic.
- The Medium: fluid genie apply skipped fluid apply's contract gate.
- The Lows: untyped loader errors, a glob that hung on symlink loops, a matched ignored
  folder that was walked, remove naming apply, and one product message reworded.
- All six are fixed, and re-verified 11 of 11.
Rounds: 3, the limit. Round 1 fixed the UX and delivery findings; round 2 fixed the defect findings; round 3 fixed a defect found in the live run and split the complex new functions. An auditor and a security re-check verified each round. Nothing is open at Medium or above.

Follow-ups (not in this PR)

forge_docs: a section for fluid genie (the docs live there).
A product space over several exposes could carry join_specs for its own foreign keys. This
PR takes out the joins Genie adds, as it does for an estate.
Live: a service principal as the deploying identity, and a reader without SELECT asking.

…t a metric view An update named every data source the space no longer holds as a metric view, since anything other than the space's own tables counted as one: taking an expose out of a product printed "updated: metric views". Each data source leaving the space is now read in Unity Catalog. One that is still a table is said as a table; a metric view, or one that is gone, as a metric view, as before. The read only names the change, so a failed read never fails the update.

… a marker of its own So a space kept by another command can reuse the converge. The messages that named `fluid apply`, the contract's block or "one space per product" now read them from a Voice, set for the converge's duration (a ContextVar, reset after); the product's wording is unchanged byte for byte. Space.token overrides the marker, Compiled.voice carries the voice, ACCESS_LINE and require_warehouse are shared, and a found space whose marker is not a product's is named by its own marker in the left-alone note. The summary counts joins, and an update names a change of instructions.join_specs as "joins".

…a products `fluid genie apply <contracts|globs|folders...> --estate <id>` creates or updates one AI/BI Genie space over the Databricks tables of every contract given (founder decisions 2026-11-12: this flag shape, and CAN_RUN only for the principals that read every table of the estate, since the space describes them all). Each table carries what a product space says of it. The space adds a join_spec for each foreign key the contracts declare between its tables, resolved by databricks_keys.foreign_keys among the estate's own tables, so each join is the Unity Catalog FOREIGN KEY fluid apply declares: MANY_TO_ONE from the referencing table, ONE_TO_ONE when the key is that table's primary key, one join for a composite key, a second alias for a key to its own table, and every column qualified by its alias. A reference outside the estate, or to no single primary key, is said. One text instruction with a section per product; no measure snippets. The space reuses the product space's identity rule and converge (databricks_genie.py): it is found by [fluid-genie-estate:<id>] in the deploying identity's own folder, updated with its etag only when it differs, and converges to exactly the contracts given. Each run checks that every table exists before it writes, and names the product to apply first. `fluid genie remove --estate <id>` trashes it. Options: --title, --env (overlays), --dry-run. Refusals and API failures are typed CLI errors (genie_estate_refused, genie_estate_failed) with the fix. Measured on Free Edition 2026-11-12 and modelled in the stand-in: join_specs are kept as sent; a self-join under two aliases and ONE_TO_ONE are accepted; an unknown relationship tag is refused ("Failed to parse export proto"), and so is a 51st table ("A space can have at most 50 tables": databricks-agent-skills' 30 is stale); 60 sample questions and an instruction of 60,000 characters are accepted. Borrowed: the join_spec shape and its rules from databricks/databricks-agent-skills (skills/databricks-genie-agents/references/create-genie-agent.md); the many-to-one orientation of apache/ossie (OSI core-spec relationships[], whose `from` is the many side); contract-file globbing as `fluid publish` does it.

…ENTS.md and CHANGELOG The command, what the space holds beyond a product's own (the joins from the declared foreign keys), who gets CAN RUN, how it is found again, and removal; the product space's update now names a leaving table as a table.

… test helpers Prompted by ripwire --quality-delta on the estate branch. differences names a change from two small tables of serialized_space parts instead of a ladder of comparisons (complexity 18 -> under the bar), and parses the live space once (_held). The stand-in's whole-space limits (one text instruction, a known relationship tag, 50 tables) move out of _genie_check into _genie_limit_problem. Estate.joins replaces the CLI's own count. The Genie tests read their writes through _fake_databricks.writes_since, and the estate tests write contracts through _databricks_cli.write_contract, which now takes a path below the root.

…ne overlay line per contract Found walking the command's journeys. `fluid --help` did not list `genie`; it now sits under Integrations next to publish and market, and the help-style lint covers it. A pack read with --env printed the loader's "overlay_applied" once per contract (19 lines for the tower pack): the header now names the overlay once, a missing overlay still warns, and --log-level DEBUG shows every line. The header no longer prints "0 table(s), 0 join(s)" before a refusal: the counts are shown only when the space compiled.

… note names its command Found by the diff review. The estate dropped the warnings databricks_keys.compile_keys gives (a primary key on a column without required: true, a labels.references that names no product and column), so a parent whose key column is nullable was reported as tagging no primary key, and a garbled reference made no join without a word. The estate now says both, as fluid apply says them of its Unity Catalog keys, and the no-key note names both conditions. The note on direct CAN_EDIT/CAN_MANAGE holders named `fluid apply` in an estate run; it reads the Voice.

…; remove names itself Found by the diff defect scan. - `fluid genie apply` skipped the gate `fluid apply` runs, so a schema-invalid contract died as "Unexpected error" (exit 2) and a pre-0.7 one was not refused. Each contract now passes plan.py's _gate_contract_for_plan_or_apply (genie_contract_invalid, contract_version_unsupported), and a file the loader cannot read is contract_load_failed; both name the file. - A glob followed symlinked folders: with two symlink loops in a pack, `**` never ended. Globs now expand as `fluid publish` expands them (one level per `**`); a folder argument is the recursive search, past no symlink. A glob whose match is an ignored folder itself (`*` on `target/`) is skipped as the docstring says. - A failed `fluid genie remove` said to re-run `fluid genie apply`; it names itself. - The product's "an apply in the next minute" note had become "another apply": the default Voice now keeps it word for word, and a test pins it.

… Genie adds its own Measured on Free Edition 2026-11-13 over the 19 tower-management tables, whose Unity Catalog foreign keys fluid apply declares. Whenever a table enters a space (a create, or an update that adds one), Genie adds a join for each foreign key that touches it, with its own id and the parent aliased <table>_2, beside the joins it was sent: 24 sent, 48 held. An update with the same tables keeps exactly what it is sent. So the first run made the space, and the second "updated: joins" it. Each write is now read back, and a space the service added to is set back once and said ("Genie added N join(s) of its own ..."), so the next run finds no change. The stand-in adds the same joins on a create and on an update that adds a table.

…s in the docs HONESTLY_TESTED gains the estate's unit (40), stand-in (1) and live Free Edition rows: the 19 tower-management products applied into fluid_iactest_estate, a cross-product question answered through the Genie Conversation API by the join the contracts declare, a second run with no change once Genie's own joins are taken out, removal, and the sweep (0 left). CHANGELOG and AGENTS.md say each contract passes fluid apply's gate and that a run takes out the joins Genie adds from Unity Catalog foreign keys.

…port in the estate tests

…contract paths ripwire --quality-delta flagged the estate's own new functions over the complexity bar: join_specs (27), contract_paths (18) and _refusals (16). Each now delegates one concern: _with_a_parent and _say_unjoined (which references can join, and what is said of the rest), _member_refusals (what one contract makes the estate refuse), and _matches and _folder_contracts (what one argument names). No behaviour change; the 40 estate tests pin it.

github-actions · 2026-11-12T23:36:46Z

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

Link a docs PR — check the "Docs PR linked" box and paste the URL
Mark as no docs needed — check "No docs needed" with a justification
Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.

Flagged by the code-quality bot on #870: remove() and Run(...)() ran inside assert statements, which python -O skips. Each call now runs on its own line and the assert checks its result (5 places; the bot named 3).

iac-tests.yml: the Databricks fake lane keeps both new tiers, the Genie estate's and the metric views'.

…ut keys After main's metric views (#865) the stand-in keeps a metric view among its tables with no constraints list, and _with_uc_joins read one for every data source of a new space: a KeyError on create (test_the_products_genie_space_holds_the_metric_view_the_apply_made). It reads constraints only where a table has them. The secrets baseline's line numbers move with the merge.

databricks_genie.py: #869's folder logic (_placed, _at_most_one, _check_folder, the move path, _found_again(space), _title_kept(space)) with the estate's Voice, Space.token and the read-back after every write (_keep_as_sent, on the move path's update too). #869's new messages read the Voice: one space per {scope}, the title setting, and the parentPath hint only where a space has that setting (Voice.folder_setting, empty for an estate). remove_estate takes #869's _placed(...)[0]. Docs and HONESTLY_TESTED keep both sides; the secrets baseline is main's with the line numbers refreshed.

fas89 added 12 commits November 13, 2026 00:30

chore: refresh the secrets baseline's line numbers; drop an unused im…

4eb05bc

…port in the estate tests

fas89 enabled auto-merge (squash) November 12, 2026 23:42

github-code-quality Bot found potential problems Nov 12, 2026

View reviewed changes

Comment thread tests/iac/test_iac_databricks_genie_estate.py Fixed

Comment thread tests/iac/test_iac_databricks_genie_estate.py Fixed

Comment thread tests/iac/test_iac_databricks_genie_estate.py Fixed

github-actions Bot removed the needs-docs Pull request needs a linked docs update or justification label Nov 12, 2026

fas89 added 4 commits November 13, 2026 00:46

test(databricks): no side effect inside an assert in the estate tests

0005f62

Flagged by the code-quality bot on #870: remove() and Run(...)() ran inside assert statements, which python -O skips. Each call now runs on its own line and the assert checks its result (5 places; the bot named 3).

Merge main (#865 metric views) into feat/databricks-genie-estate

c7097b0

iac-tests.yml: the Databricks fake lane keeps both new tiers, the Genie estate's and the metric views'.

fas89 merged commit f9c1a8c into main Nov 13, 2026
50 checks passed

fas89 added a commit that referenced this pull request Nov 13, 2026

Merge main (#868, #870) into feat/databricks-dbt-dq-tests

d50084b

This was referenced Nov 13, 2026

feat(databricks): a build can be a Lakeflow pipeline whose expectations are the contract's DQ rules #872

Merged

chore(release): cut v0.30.0 changelog entry #874

Merged

Navigation Menu

Repository navigation

feat(databricks): fluid genie apply, one Genie space over several data products - #870

feat(databricks): fluid genie apply, one Genie space over several data products#870
fas89 merged 16 commits into
mainfrom
feat/databricks-genie-estate

fas89 commented Nov 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fas89 commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Borrowed

Validation

Docs

Reviews

Follow-ups (not in this PR)

Uh oh!

github-actions Bot commented Nov 12, 2026

📄 Documentation Reminder

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fas89 commented Nov 12, 2026 •

edited

Loading