Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`dbt docs generate`.
- Id-first database addressing: pin `database_id` in the profile, or let the
first run create a database and print its id.
- Cross-database macros for DataFusion's SQL surface: `dateadd`, `datediff`,
- Cross-database macros for HotSQL: `dateadd`, `datediff`,
`convert_timezone`.
- Transient API errors (409/429/5xx) retry for ~42s via the shared
`hotdata-framework` client; terminal errors fail the node immediately.
Expand Down
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

Transform data in [Hotdata](https://hotdata.dev) instant databases with [dbt](https://www.getdbt.com) — the companion to [hotdata-dlt-destination](https://github.com/hotdata-dev/hotdata-dlt-destination) for the T in ELT.

Hotdata is a managed analytics engine (Apache DataFusion, Postgres-dialect SQL) with **no DDL surface**: tables are created by loading data, not by `CREATE TABLE`. This adapter embraces that. Every model runs as the Chain pattern, entirely against the API:
Hotdata is a managed analytics engine speaking [HotSQL](https://www.hotdata.dev/docs/sql) (standard SQL with analytics extensions — if you know Postgres, you already know most of it) with **no DDL surface**: tables are created by loading data, not by `CREATE TABLE`. This adapter embraces that. Every model runs as the Chain pattern, entirely against the API:

1. the model's compiled `SELECT` executes **server-side**,
2. the result streams back as Arrow,
Expand All @@ -28,7 +28,7 @@ No local database engine, no driver, no version matching — pure Python over HT

## Requirements

- Python **3.11+**, dbt-core **1.10+**
- Python **3.11+**, dbt-core **1.10+** (tested through 1.12)
- A [Hotdata](https://hotdata.dev) workspace, an API key, and its workspace ID — from your Hotdata dashboard or the [Hotdata CLI](https://github.com/hotdata-dev/sdk-python).

## Install
Expand Down Expand Up @@ -119,6 +119,10 @@ Tests, `dbt show`, analyses, and source freshness all run as plain SELECTs on th

Schema evolution is additive and automatic: a model that starts producing a new column just includes it in the next load — existing data is never touched, and types can widen but never silently shrink. (`on_schema_change` is therefore ignored.)

### SQL dialect

Write models in [HotSQL](https://www.hotdata.dev/docs/sql). It is Postgres-familiar, so SQL written for Postgres mostly runs unchanged, and the adapter overrides the cross-database macros (`dateadd`, `datediff`, `convert_timezone`) where HotSQL differs. Hotdata's query API also accepts the Postgres, DuckDB, and Snowflake dialects (translated to HotSQL server-side), but this adapter always submits model SQL as native HotSQL.

## Feature support

| Feature | Support | Notes |
Expand All @@ -129,11 +133,12 @@ Schema evolution is additive and automatic: a model that starts producing a new
| Tests (generic + singular) | ✅ | Run server-side; `store_failures` supported |
| `dbt docs generate` | ✅ | Catalog from the managed-table API |
| Source freshness | ✅ | `loaded_at_field` queries run server-side |
| Cross-database macros | ✅ | `dateadd`, `datediff`, `convert_timezone` implemented for [HotSQL](https://www.hotdata.dev/docs/sql) (`convert_timezone` is DST-aware) |
Comment thread
eddietejeda marked this conversation as resolved.
| Hooks (`pre-hook`/`post-hook`, `on-run-*`) | ⚠️ | Run server-side — SELECT-shaped SQL only (no DDL exists) |
| Python models | ❌ | |
| Model contracts / constraints | ❌ | No DDL; dbt warns they are unenforced |
| Grants | ❌ | Ignored with a warning — access is governed by workspace API keys |
| Transactions | ❌ | `begin`/`commit` are no-ops (DataFusion has no transactions) |
| Transactions | ❌ | `begin`/`commit` are no-ops (Hotdata has no transactions) |
| Query cancellation | ❌ | An in-flight HTTPS query can't be interrupted client-side |

## Configuration
Expand Down
2 changes: 1 addition & 1 deletion dbt/adapters/hotdata/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
``dbtRunner.invoke()`` in the same process gets fresh credentials and
resolves fresh — nothing can serve a stale or differently-configured record.

Every SQL statement is executed server-side (Apache DataFusion, Postgres
Every SQL statement is executed server-side (HotSQL, a Postgres-familiar
dialect) scoped to the resolved database, and results come back as Arrow.
There is no DDL surface: tables are created by declaring them and loading
parquet (``replace`` / ``append`` / ``upsert`` / ``delete`` modes).
Expand Down
6 changes: 3 additions & 3 deletions dbt/adapters/hotdata/column.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@


def dtype_from_arrow(arrow_type: pa.DataType) -> str:
"""Render an Arrow type as the Postgres-surface name DataFusion presents.
"""Render an Arrow type as the SQL type name HotSQL presents.

Used when describing relations: the engine returns Arrow schemas, and dbt
(docs, `{{ col.data_type }}`, schema tests) expects SQL type names.
Expand Down Expand Up @@ -43,7 +43,7 @@ def dtype_from_arrow(arrow_type: pa.DataType) -> str:
if (
pa.types.is_string(arrow_type)
or pa.types.is_large_string(arrow_type)
# DataFusion reads loaded string columns back as Utf8View; the SQL
# The engine reads loaded string columns back as Utf8View; the SQL
# name must still be a castable one, never "string_view".
or pa.types.is_string_view(arrow_type)
):
Expand All @@ -57,7 +57,7 @@ def dtype_from_arrow(arrow_type: pa.DataType) -> str:
class HotdataColumn(Column):
@classmethod
def string_type(cls, size: int) -> str:
# The base class renders "character varying(256)", which DataFusion
# The base class renders "character varying(256)", which HotSQL
# rejects in casts; strings are unbounded here.
return "varchar"

Expand Down
4 changes: 2 additions & 2 deletions dbt/adapters/hotdata/connections.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
"""dbt connection manager for Hotdata.

There is no database driver here: a "connection" is an HTTPS client
(:class:`HotdataDbtClient`), SQL executes server-side on Apache DataFusion
(Postgres dialect) scoped to the run's instant database, and results come
(:class:`HotdataDbtClient`), SQL executes server-side as HotSQL
(Postgres-familiar) scoped to the run's instant database, and results come
back as Arrow. Consequences for the dbt contract:

* ``begin``/``commit`` are no-ops — the engine has no transactions.
Expand Down
4 changes: 2 additions & 2 deletions dbt/adapters/hotdata/impl.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
"""Hotdata dbt adapter.

Hotdata has no DDL surface — tables are created by declaring them on the
instant database and loading parquet, and queries run server-side (Apache
DataFusion, Postgres dialect) returning Arrow. The adapter therefore keeps
instant database and loading parquet, and queries run server-side (HotSQL,
a Postgres-familiar dialect) returning Arrow. The adapter therefore keeps
all metadata and materialization work in Python:

* Relation listing, columns, and the docs catalog come from the
Expand Down
6 changes: 3 additions & 3 deletions dbt/include/hotdata/macros/utils.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{#
Cross-database macro overrides for DataFusion's Postgres surface.
Cross-database macro overrides for HotSQL.

dbt's default__ implementations of dateadd/datediff emit Snowflake-style
function calls (`dateadd('day', 5, x)`, `datediff('day', a, b)`) that
DataFusion does not have. These rebuild them from constructs it does have:
HotSQL does not have. These rebuild them from constructs it does have:
string-built interval casts (dynamic-safe: `x + cast(concat(n, ' days') as
interval)`) and integer date subtraction (`date - date` -> whole days).
#}
Expand All @@ -16,7 +16,7 @@
{#-
dbt_date dispatch hook (packages need a root-project shim or dispatch
config to reach it). NOT the Postgres double-AT TIME ZONE + cast-to-naive
pattern: on DataFusion that final cast re-renders the UTC instant,
pattern: on HotSQL that final cast re-renders the UTC instant,
silently losing the shift. Here AT TIME ZONE on a naive timestamp
localizes it as the source zone's wall clock (12:00 @ America/New_York
-> 12:00-05:00, verified — non-UTC sources convert correctly), and
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ API-shaped reality.

## The constraint that shapes everything

Hotdata is a managed engine (Apache DataFusion, Postgres-dialect SQL over
HTTPS) with **no DDL surface**:
Hotdata is a managed engine ([HotSQL](https://www.hotdata.dev/docs/sql),
Postgres-familiar SQL over HTTPS) with **no DDL surface**:

- Queries: `POST /query` scoped to an instant database (`X-Database-Id`),
polled to completion, result fetched as Arrow. SELECT only — no
Expand Down
Loading