Repository navigation
Docs for 2.3 - #34
Merged
Merged
Docs for 2.3#34
Conversation
added 17 commits
November 10, 2026 00:14
Signed-off-by: The Architect <hello@ingestron.io>
Signed-off-by: The Architect <hello@ingestron.io>
… limits - Start from a published standard, Move to a newer one and the CLI reference: the name alone takes the newest version that is not a preview among the installed versions and the catalogue's tags, or the newest installed with a note when the catalogue cannot be reached. - The Databricks kit's page: project.yaml holds the two groups. - Share a template: reading tags for a name alone uses the network; git and ssh never ask for a password; the 15 and 60 second limits. - Tests: the note as init prints it, and each group's line in project.yaml. Signed-off-by: The Architect <hello@ingestron.io>
Signed-off-by: The Architect <hello@ingestron.io>
Signed-off-by: The Architect <hello@ingestron.io>
…rning The intent vocabulary describes the quality and cleansing paths: rule kinds and table checks are written in the contract, actions in the standard, a layer or an expectation, and each cleansing operation where its column is cleansed. The DuckDB starter's required order_id is now NOT_APPLIED in every plan, since its layers say nothing of checking, so the first project, adding a source and their tests show and explain that warning. The quality table says where DuckDB and Synapse check rows and what is NOT_APPLIED. Signed-off-by: The Architect <hello@ingestron.io>
The quality example is one contract's checks on Databricks, DuckDB and Synapse, an environment each, and Check data quality is rewritten around it: where checks run by layer, warn, drop and fail, table checks, quarantine and results, and the engine table from each pack's coverage. New pages: Not applied here (what the state means, where it shows, how it differs from other warnings and CAPABILITY_OFF, and the does hint); Cleanse data in the right layer, on the DuckDB published standard, with a transform without code and timestamps to UTC; and Accept a warning, for every environment or through a value. Each pack's page gains Quality and cleansing from its coverage, with the layers each value needs and its sources; Synapse lists the zones it cannot convert. The errors page says how each code is reported and marks CONTRACT_FIELD_UNUSED removed. Layers, the intent and documents reference, contracts, transforms, the platform pages, troubleshooting and the kits say what layer duties, cleansing and accepted warnings change. Signed-off-by: The Architect <hello@ingestron.io>
…rnings The 2.3 notes say that passports rewrite their Quality section, that DuckDB and Synapse check rows only where a layer's does says so and need nothing turned on for table checks, which old codes became NOT_APPLIED or CAPABILITY_OFF, and that a warning can be accepted. The package notes list the quality and cleansing additions. Signed-off-by: The Architect <hello@ingestron.io>
duckdb-local, data-products and synapse-views give each layer its does: bronze keeps raw rows and counts its checks, silver and gold check. Their plans no longer list NOT_APPLIED for required columns, the data-products passport says what checks each table, and Synapse writes checks.sql, which its platform and pack pages now list. Not applied here describes the passport in words, and the test that shows the does hint on Synapse takes the duties away first. Signed-off-by: The Architect <hello@ingestron.io>
A table Lakeflow Connect writes still runs its table checks after each refresh, and the event log keeps results; only its row checks, actions, quarantine, cleansing and deduplicate are not applied. Check data quality shows the VS Code way to plan again; Accept a warning says entries for build-only warnings are judged only when the packs build; a custom rule is one only another engine could run. Signed-off-by: The Architect <hello@ingestron.io>
Describe the governance part of the streaming lakehouse kit, move the line number its NOT_APPLIED sample shows, and expect the kit's new REPLACE_ME values. Signed-off-by: The Architect <hello@ingestron.io>
A guide to granting access, masking columns and filtering rows, the governed-databricks example, roles and grants in synapse-views, and an accepted NOT_APPLIED in data-products. Each pack's page lists its access coverage, Synapse's page its platform section, and the intent reference the access paths. The upgrade notes list the new warnings and output changes. Signed-off-by: The Architect <hello@ingestron.io>
Filter wording, Synapse masks and filters in the upgrade notes, who the policies exempt, links to each pack's access, roles and masks on the contract page, when to run the SQL after the first deploy, and a unique order for the new example. Signed-off-by: The Architect <hello@ingestron.io>
The ask-the-platform-team page names the Synapse switch beside the Databricks ones, as the request lists it. Signed-off-by: The Architect <hello@ingestron.io>
The Databricks apply task makes its policies before it grants, and the pages say so. Masks hide values in the tables, not in the files ADF lands. Synapse exact grants keep the permissions of the user that runs views.sql. The upgrade page lists the per-pack request and the single classification tag, and the platform team page gives a Synapse example of a switch that is off. Signed-off-by: The Architect <hello@ingestron.io>
…ers filled in Signed-off-by: The Architect <hello@ingestron.io>
Signed-off-by: The Architect <hello@ingestron.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documentation for 2.3: intent, published standards, quality and cleansing, accepting warnings, and the how-to Grant access, mask columns and filter rows with the
governed-databricksexample; the Upgrade to 2.3 notes.pnpm validatepasses under Node 22 against the local packages.One release across eight repositories
This change ships together with the
release/2.3pull requests iningestron,pack-databricks,pack-adf,pack-synapse,pack-duckdb,vscodeanddocs, and the newstandardsrepository. Packages arepublished before the ones that depend on them:
@ingestron/config1.7.0,@ingestron/definitions1.13.0 (@ingestron/generate1.1.0 is unchanged)@ingestron/pack-databricks2.3.0,@ingestron/pack-adf2.3.0,@ingestron/pack-synapse2.2.0,@ingestron/pack-duckdb2.2.0@ingestron/standards1.0.0@ingestron/core1.1.0,@ingestron/authoring2.3.0,@ingestron/cli2.3.0 andingestron2.3.0Checks that install from npm pass once the packages they need are published;
each repository then gets its lockfile refreshed on this branch.