Repository navigation
Docs for ingestron 2.0: one output folder per platform, deploy through environments - #28
Merged
Merged
Conversation
GitHub Actions (deploy.yml and deploy-environment.yml) and Azure DevOps (azure-pipelines.yml and templates/deploy-environment.yml) check the generated files, build the Data Factory template once with Microsoft's package 1.0.3, and deploy the same files to each environment in turn, gated by that environment's approvals. Deploy steps use only the platforms' tools: the pre- and post-deployment script and an incremental ARM deployment, the Databricks CLI v1.23.0, and sqlcmd (Go) v1.10.0 checked by its SHA-256. The check templates install ingestron@2. Signed-off-by: The Architect <contact@intentlabs.dev>
…nment Each standard names databricks@2, adf@2, synapse@2 or duckdb@2. One bundle or factory now serves every environment, so connection-first-adf names its ADF pipeline and trigger without the environment, and neither it nor standards-and-conventions names a pipeline per environment. Signed-off-by: The Architect <contact@intentlabs.dev>
Each README names the folder ingestron 2.0 writes: one bundle in build/databricks/, one factory in build/adf/ with a parameter file per environment, build/synapse/ with a script per environment, and build/duckdb/<environment>/ and build/products/<environment>/. Signed-off-by: The Architect <contact@intentlabs.dev>
….0 and the four packs 2.0.0 Signed-off-by: The Architect <contact@intentlabs.dev>
…e names The first project, the home page and Add a source show what ingestron 2.0 prints and where it writes. The starter's sample customer is Ana. The home page says each platform gets one build for every environment. Signed-off-by: The Architect <contact@intentlabs.dev>
…loying
Generated files and ownership holds the layout, build/<pack>/, in one
place. Environments and YAML references say that an ${envvar:…} value
in a Databricks, Data Factory or Synapse project is supplied by the
deploying pipeline, under the name ingestron request gives. Add an
environment generates every environment together and shows the new
target.
Signed-off-by: The Architect <contact@intentlabs.dev>
The platform defaults table lists the Platform fields a pack declares it reads, in its order, keeping those whose description states a default, instead of a list kept in the docs. Signed-off-by: The Architect <contact@intentlabs.dev>
Databricks and Azure Data Factory are rewritten in the platform pages' structure on the 2.0 files: one bundle with a target per environment, and one factory with an ARM parameter file per environment, deployed by target or by template with the pre- and post-deployment script. Synapse deploys with sqlcmd from build/synapse, and DuckDB in Azure uploads the environment's folder. Each deploy section renders its step of the CI templates. Shape ADF pipelines and Land files only render the pack's settings in place of hand-written tables. Old section ids stay as anchors. Signed-off-by: The Architect <contact@intentlabs.dev>
Upgrade Ingestron and its packs gains the 2.0 upgrade notes: every pack at @2 through PACK_VERSION's fix, the folders 1.x leaves, values supplied when deploying, each platform's first deployment, VS Code, and the interfaces that changed for code that calls the packages. Name tables and resources says pipeline, job and ADF names are the same in every environment, and shows the renames 2.0 plans. Signed-off-by: The Architect <contact@intentlabs.dev>
…oject The errors page lists PACK_LOAD and PACK_NOTICE from definitions 1.10, and says that PACK_LOAD stops every command with exit 2. The upgrade page is reformatted. Signed-off-by: The Architect <contact@intentlabs.dev>
Signed-off-by: The Architect <contact@intentlabs.dev>
A new how-to, after Check generated files in CI, is the one home of promotion: what each environment's stage runs per platform, the GitHub Actions and Azure DevOps templates rendered from ci-templates/, the set up of environments, approvals, identities and variables, values supplied when deploying, and adding an environment. Check generated files in CI renders its templates. Plan, apply and review plans every environment together. Signed-off-by: The Architect <contact@intentlabs.dev>
The selected environment is for problems, previews and the platform request; Plan, Apply and Check work on every environment together, and auto-apply logs a line for each output folder. The screenshots are retaken from the extension at its current head. Signed-off-by: The Architect <contact@intentlabs.dev>
Troubleshooting gains entries for --env refused, a pack that cannot be loaded, PACK_VERSION, ENVIRONMENTS_DIFFER, DEPLOY_VALUE, the runtime notice and sqlcmd's working folder, and its entries quote 2.0 output. The command line reference describes one folder per platform and exit 2 for a pack that cannot be loaded. Generated files lists the layout as a table. The tests read the 2.0 folders. Signed-off-by: The Architect <contact@intentlabs.dev>
The pack reference pages name @2 and link each pack's files on its platform page; How Ingestron works describes one folder per platform for every environment; the how-to pages show generated files from their 2.0 paths; Write a transform says what can differ in a PySpark transform. Signed-off-by: The Architect <contact@intentlabs.dev>
Signed-off-by: The Architect <contact@intentlabs.dev>
…s the templates pin Signed-off-by: The Architect <contact@intentlabs.dev>
…ron, and Azure DevOps fixes Every GitHub action moves to its current major (checkout 7, setup-node 7, upload-artifact 7, download-artifact 8, azure/login 3, azure/powershell 3, all on Node 24), pinned by full commit SHA with its version beside it, and a test checks every uses: line. The checks install ingestron@2.1.0. Only the deploy jobs get id-token: write. The build runs Microsoft's export alone, which validates too. The headers give both GitHub OIDC subject formats and say required reviewers on a private repository need GitHub Enterprise Cloud. Azure DevOps: the Azure CLI steps set useGlobalConfig, as Microsoft Learn requires for databricks bundle; each stage sets lockBehavior: sequential, with an exclusive lock check on each environment; the check template uses UseNode@1. Signed-off-by: The Architect <contact@intentlabs.dev>
Databricks and Azure Data Factory passed the platform pages' length for a split, so each gains a sibling page, Deploy the Databricks bundle and Deploy the factory, with what must exist first, the deployment by hand and from CI, and running it. Each platform page keeps one paragraph in their place, with the old anchors. Deploy the factory is the one home of one factory per project, and names both sign-ins. Deploy the Databricks bundle says to deploy each target as one identity, with a shared root_path, and how to bind what another identity deployed. The ADF schedule settings move to Shape ADF pipelines. Links, routes and redirects point at the new pages, and the From CI sections leave sign-in to Deploy through environments. Signed-off-by: The Architect <contact@intentlabs.dev>
… and Azure DevOps set-up The upgrade notes say the first Databricks deploy updates in place only as the identity that deployed 1.x, and to set a shared root_path and bind before CI deploys; the schema unbind runs from the 1.x bundle's folder before it is deleted. The templates and pages pin ingestron@2.1.0, tied by a test. Deploy through environments gives both GitHub OIDC subject formats, says required reviewers on a private repository need GitHub Enterprise and what to do without them, and adds Azure DevOps' exclusive lock check. Layout sentences outside Generated files become links. Signed-off-by: The Architect <contact@intentlabs.dev>
…Enterprise Signed-off-by: The Architect <contact@intentlabs.dev>
…gestron exactly Signed-off-by: The Architect <contact@intentlabs.dev>
…/<target> Signed-off-by: The Architect <contact@intentlabs.dev>
… wording for the subject formats, and the shared-schedule anchor On GitHub plans without required reviewers for a private repository, the page and the template say test and prod deploy after dev with no approval. The OIDC subject formats name repositories created, renamed or transferred since 15 July 2026, or opted in. Azure Data Factory keeps tables-that-share-a-schedule as an old anchor beside its link to Shape ADF pipelines. Signed-off-by: The Architect <contact@intentlabs.dev>
… and pack-databricks 2.1.0 The ingestron bundle the templates and the upgrade page pin, 2.1.0, is now the installed cli's version, and the test says so. Signed-off-by: The Architect <contact@intentlabs.dev>
…itten connection-first-files is what init 2.1 writes, REPLACE_ME and comments included. connection-first-adf is init 2.1's PostgreSQL project with a --set for each value, then the ADF names set by hand. New guides retail-filled and private-sql-server-filled are the retail and private-sql-server starters created with a --set for each value, so the platform pages show a project that deploys. Signed-off-by: The Architect <contact@intentlabs.dev>
…s Values to replace Signed-off-by: The Architect <contact@intentlabs.dev>
init asks only what picks the project and writes REPLACE_ME, with a comment, where a value is yours to give. Create a project says so, with README.md, --set by name and per environment, the warnings, and check failing until each is written; Make it yours becomes Write the values it cannot know, keeping its old anchor. Starters and templates, Share templates and standards and the built-in starters' table no longer speak of questions. The first project names its README.md and the VS Code wizard's folder step. Ask the platform team quotes request 2.1's Values to provide and names Values to replace. Signed-off-by: The Architect <contact@intentlabs.dev>
…s written Databricks and Azure Data Factory create their starter, say what it leaves as REPLACE_ME and init's count line, then show the same init with a --set for each value; their files come from the new samples, which the tests check against that command. The other platform pages say what init leaves to write and drop the wizard's questions. The tests join a command's continuation lines, and bind Create a project's count line and its --set example to the CLI. Signed-off-by: The Architect <contact@intentlabs.dev>
… and the 2.1 upgrade notes VS Code's New project asks only the start, the source or starter, and the folder, then opens README.md; the pages that sent readers to its standard box say to use --standard instead. Check generated files in CI, the command line reference and the errors page say check fails while a value is REPLACE_ME. The upgrade page gains Upgrade to 2.1. The home page shows the quality example's table, whose path is written, instead of the minimal starter's REPLACE_ME. Signed-off-by: The Architect <contact@intentlabs.dev>
Signed-off-by: The Architect <contact@intentlabs.dev>
Signed-off-by: The Architect <contact@intentlabs.dev>
Signed-off-by: The Architect <contact@intentlabs.dev>
The check that nothing built or downloadable holds a regional name the public repositories leave out builds its pattern from parts, and its messages and success line name no country. A brand test's sample row uses the starter's own name. Signed-off-by: The Architect <contact@intentlabs.dev>
…d step Create a project shows plan's REPLACE_ME line and says it counts places where init counts names, bound by test. Its diagram drops the Standard step, which init no longer asks. Connector pages say init asks only how the source is reached, and the route helpers' comments match. Signed-off-by: The Architect <contact@intentlabs.dev>
A starter's download is now built from ingestron init --template --yes, with the starter's own id: its README.md lists what to replace, its .gitignore is in place, and the template's own README and template.yaml are gone. The download pages and the connector pages say so. Signed-off-by: The Architect <contact@intentlabs.dev>
Signed-off-by: The Architect <contact@intentlabs.dev>
Create it on the Databricks and ADF pages gave the --set command as a second step, which fails after the first. It is now an alternative, run from the same parent folder, and the page says what plan prints for each. The tests run the commands in page order. Signed-off-by: The Architect <contact@intentlabs.dev>
pack-adf 2.1 writes Binary connections as Anonymous and refuses a login on them, and SASL and HTTP need both user and password. The notes say so, and how to change a 2.0 project, instead of saying no generated file changes. Signed-off-by: The Architect <contact@intentlabs.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The site moves to cli 2.0 and the 2.0 packs.
ci-templates/(actions pinned by commit, tools pinned by version and checksum, tests that the templates parse and pin).@2; the errors page addsPACK_LOADandPACK_NOTICE; tools/vscode describes the 2.0 extension, with its screenshots retaken.Draft: merges with ingestron 2.1.0 (init defaults); the init pages and New project screenshots are finished against it.