Ingestron

Documentation  ·  First project  ·  Source  ·  npm

You need Node.js 22.13 or later. No cloud account and no data source are needed.

npm install -g ingestron
ingestron init shop --template duckdb --yes
cd shop
ingestron plan     # what would be written, with diffs
ingestron apply    # write it

That creates a project from the DuckDB starter and writes its SQL. Run your first project on your laptop runs that SQL with DuckDB, explains each file, then changes the project and breaks it on purpose. Install a tool covers a global install and the VS Code extension.

Ingestron writes the files a data engineer would otherwise write by hand. You describe sources, tables and rules in YAML your team reviews; Ingestron applies your organisation's standard and writes ordinary platform files. You deploy them with your platform's own tools, from your own CI/CD pipeline.

• One build for every environment. Each platform gets one set of files with only what differs between dev and prod varying, so a reviewer sees exactly what changes.
• Nothing of Ingestron at run time. The generated files need only your platform, and you edit them yourself if you stop using Ingestron.
• Your edits are kept. A file someone changed by hand is reported as a conflict and never overwritten.

The platform team writes one standard: the packs, names, layers, types, default settings, the settings no project may change, and what every data product must declare. VS Code marks a break as you type, ingestron plan reports it, and ingestron check fails the CI build when the generated files are out of date. See Set your organisation's standard.

• It does not connect to your sources or your cloud.
• It does not deploy anything. Your platform's tools do.
• It does not read secret values. A ${secret:…} reference carries only the secret's name.
• It does not send telemetry, from the command line or from VS Code.

Ingestron is licensed under Apache-2.0. The packs and the command are open source, and the documentation is built from them, so what it says matches what the code does.

Questions and feedback: hello@ingestron.io. Report vulnerabilities privately to security@ingestron.io; please do not include live secrets or customer data.

_{Ingestron is an independent project made by IntentLabs. Original code is licensed under Apache-2.0.}