Skip to content

About

Examples of operational data systems built with DataSQRL: real-time pipelines, data APIs, and data products.

Topics

Resources

Stars

10 stars

Watchers

1 watching

Forks

Latest commit

 

History

205 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DataSQRL Examples: Operational Data Systems

DataSQRL is an open-source harness for agentic data engineering on operational data: agents build real-time pipelines, data APIs, and data products; a compiler verifies them.

Each example is an operational data system: a real-time pipeline, a data API, or a data product.

None of these is a dbt-style model inside a warehouse. Several land data in Iceberg for a warehouse or query engine to read.

Pick an example by what you need

Example What it builds Triggers
Finance Credit Card Chatbot Data for AI agents: enriched credit card transactions, a rewards program, and spending views, served to a chatbot Kafka, real-time, data API (GraphQL, MCP), AI agents, Iceberg
Oil & Gas Agent Automation A data product over a shared catalog of wells: a monitoring API for an agent and an operations backend with alerts Kafka, real-time, data API, AI agents
Clickstream AI Recommendation Personalized recommendations from clickstream data and LLM vector embeddings Kafka, real-time, data API
Healthcare Study One catalog feeding a data API, analytics tables in Iceberg, and an enriched Kafka stream Kafka, data API, Iceberg
Logistics Shipping Real-time shipment tracking served to customers real-time, data API
IoT Sensor Metrics A sensor API that ingests readings and serves metrics and alerts data API, real-time
Law Enforcement An integrated view of drivers, vehicles, warrants, and BOLOs from databases and event streams multiple source systems, data API
Usage Analytics Per-user token usage with hourly aggregates and an over-limit alert data API
Getting Started Examples Seven minimal pipelines: Kafka to console, Kafka to Kafka, joins, Kafka and files to Iceberg, Schema Registry Kafka, Iceberg, AWS Glue
Iceberg Data Deduplication Compaction and deletion jobs that keep Iceberg tables written by operational pipelines deduplicated Iceberg, batch
User Defined Function A custom Java function shipped with JBang or Maven extension point

The examples

  • Finance Credit Card Chatbot: A data product for AI agents. A real-time pipeline enriches and analyzes credit card transactions and serves them through a data API (GraphQL and MCP) to a GenAI chatbot that answers customers' questions about their spending. The extended example adds a credit card rewards program, and a batch variant writes spending views to Iceberg.
  • Oil & Gas Agent Automation: A data product built on a shared data catalog. A real-time pipeline enriches well readings and triggers an agent to analyze abnormal events for automated troubleshooting.
  • Clickstream AI Recommendation: A real-time pipeline and data API that serve personalized recommendations from clickstream data and vector content embeddings generated by an LLM.
  • Healthcare Study: A real-time pipeline that enriches healthcare study data, serves it through a data API, lands it in Iceberg for a warehouse or query engine to read, and publishes it to Kafka.
  • Logistics Shipping: A real-time pipeline and data API for shipment tracking served to customers.
  • IoT Sensor Metrics: A data API that ingests sensor metrics, processes them in real-time, and serves alerts and metrics to users.
  • Law Enforcement: A real-time pipeline and data API that integrate driver and vehicle databases with warrant and BOLO streams.
  • Usage Analytics: A data API that tracks token usage per user and alerts when a user goes over the hourly limit.
  • Getting Started Examples: Seven small real-time pipelines that show one connector pattern each, from Kafka to Iceberg on AWS Glue.
  • Iceberg Data Deduplication: Lake maintenance for operational pipelines: batch jobs that compact and deduplicate Iceberg tables.
  • User Defined Function: A small tutorial that shows how to call a custom function in your SQRL script, for custom logic inside an operational pipeline.

Supporting folders:

  • data-generator generates larger datasets for experiments and benchmarks.
  • util holds Python helpers for loading data into Kafka and reading it back.
  • test-jobs is CI material, not an example: regression jobs that CI runs against the DataSQRL version in sqrl.Dockerfile.

Build this with the DataSQRL plugin

Every example can be built by a coding agent. Install the DataSQRL plugin in Claude Code, Codex, Cursor, or Copilot, then describe the system you want. For example:

Build a DataSQRL pipeline that reads credit card transactions from Kafka, enriches them with
customer and merchant data, and serves each customer's spending by category through a GraphQL
and MCP API for a chatbot.

Each example README has a prompt like this one for that example.

Running the Examples

Prerequisites

To compile and run these examples with DataSQRL you need to have a recent version of Docker installed on your machine.

DataSQRL version

The commands in this repository use datasqrl/cmd:latest. CI tests every example against the version pinned in sqrl.Dockerfile. If an example does not run on latest, replace latest with the tag from that file.

Running & Compiling Examples

To run the examples on Linux or MacOS, open a terminal and run the following command:

docker run -it --rm -p 8081:8081 -p 8888:8888 -p 9092:9092 -v $PWD:/workspace datasqrl/cmd:latest run [PACKAGE FILES]

If you are on Windows using PowerShell, you need to reference the local directory with a slightly different syntax:

docker run -it --rm -p 8081:8081 -p 8888:8888 -p 9092:9092 -v ${PWD}:/workspace datasqrl/cmd:latest run [PACKAGE FILES]

Replace [PACKAGE FILES] with the package files of the example, base first and overlay second; later files override earlier ones. Examples that share a data catalog also pass -r <project> to name the project. Check the README.md in the respective directory for the exact command. The DataSQRL command reference lists every option. We will be using the Unix syntax, so keep in mind that you have to adjust the commands slightly on Windows machines by using ${PWD} instead.

To compile an example (without running it), use this command:

docker run --rm -v $PWD:/workspace datasqrl/cmd:latest compile [PACKAGE FILES]

What is DataSQRL?

DataSQRL is an open-source harness for agentic data engineering on operational data. It extends coding agents with a SQL compiler, validator, and event-time simulator. Agents build real-time pipelines, data APIs, and data products on Kafka, Flink, Postgres, Iceberg, Spark, Snowflake, and DuckDB, and the compiler verifies them across every system before deployment. Apache 2.0 licensed, it runs on your infrastructure.

About

Examples of operational data systems built with DataSQRL: real-time pipelines, data APIs, and data products.

Topics

Resources

Stars

10 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages