feat(databases): load csv/json, keyed load modes, and table declaration

README.md

@@ -42,17 +42,20 @@ PostgreSQL-dialect SQL. Everything else builds on that.
 
 ## Getting your data in
 
-**Upload a parquet file** directly (convert CSV/JSON first):
+**Upload a file** directly — csv, newline-delimited json, or parquet:
 
 ```sh
-hotdata databases load --catalog demo --table listings --file ./listings.parquet
+hotdata databases load --catalog demo --table listings --file ./listings.csv
 ```
 
-A load **replaces** the table by default. Add `--append` to add rows to an
+The format comes from the extension; pass `--format csv|json|parquet` when the
+extension is missing or misleading.
+
+A load **replaces** the table by default. `--mode append` adds rows to an
 existing table instead:
 
 ```sh
-hotdata databases load --catalog demo --table listings --file ./more-listings.parquet --append
+hotdata databases load --catalog demo --table listings --file ./more-listings.parquet --mode append
 ```
 
 **Import from an external source** — Postgres/MySQL, S3/GCS buckets, Iceberg,
@@ -163,10 +166,11 @@ The full command surface. The top level has nine groups — `auth`, `workspaces`
 | `databases use` | Set the current (default) database |
 | `databases unset` | Clear the current database |
 | `databases remove` | Delete a database and all its tables |
-| `databases load` | Load a parquet file or saved result into a table (replace, or `--append`) |
+| `databases load` | Load a csv/json/parquet file or saved result into a table (`--mode replace\|append\|delete\|update\|upsert`) |
+| `databases tables add` | Declare a table with its key and storage layout |
 | `databases tables list` | List tables in a database |
 | `databases tables show` | Show column definitions for a table |
-| `databases tables load` | Load parquet/result into a table (replace, or `--append`) |
+| `databases tables load` | Same as `databases load`, addressed by database instead of catalog |
 | `databases tables remove` | Delete a table from a database |
 | `databases context list` | List named contexts in a database |
 | `databases context show` | Print context content to stdout |

skills/hotdata/SKILL.md

@@ -87,7 +87,7 @@ Returns workspaces with `public_id`, `name`, `active`, `favorite`, `provision_st
 
 **Instant databases** are Hotdata-owned catalogs you create and populate yourself — no remote source to sync. Query them in SQL as **`<database_id>.<schema>.<table>`**. Prefer **`hotdata databases`** for this workflow.
 
-**Parquet only:** `databases tables load` accepts **parquet** files (local `--file`, remote `--url`, or a pre-staged `--upload-id`).
+**File formats:** `databases tables load` accepts **csv**, **newline-delimited json**, and **parquet** (local `--file`, remote `--url`, or a pre-staged `--upload-id`). The format is read from the file's extension; `--format csv|json|parquet` overrides it, and an unrecognised extension is left to the server to resolve from the bytes.
 
 **Active database:** `hotdata databases use <id>` saves the active database to config. `databases tables list`/`load`/`remove`, `databases queries`/`results`, and all `databases context` commands default to the active database; pass **`--database <id>`** to override per-command. (`databases tables show` instead takes a fully-qualified `catalog.schema.table`.)
 
@@ -108,12 +108,14 @@ hotdata databases remove <id> [--workspace-id <workspace_id>]
 hotdata databases attach <catalog|name> [--database <id>] [--alias <alias>]
 hotdata databases detach <catalog|name|alias> [--database <id>]
 
-# Preferred: load by catalog alias (auto-declares table if needed). --append adds rows instead of replacing.
-hotdata databases load --catalog <alias> --table <table> [--schema public] (--file <path> | --url <url> | --upload-id <id> | --result-id <id>) [--append] [--workspace-id <workspace_id>]
+# Preferred: load by catalog alias (server declares the table/schema if missing).
+# Loads csv, newline-delimited json, or parquet — format read from the extension.
+hotdata databases load --catalog <alias> --table <table> [--schema public] (--file <path> | --url <url> | --upload-id <id> | --result-id <id>) [--mode replace|append|delete|update|upsert] [--append] [--format csv|json|parquet] [--key <col>]... [--workspace-id <workspace_id>]
 
 # Also available via tables subcommand
 hotdata databases tables list [--database <id>] [--schema <name>] [--workspace-id <workspace_id>] [--output table|json|yaml]
-hotdata databases tables load <table> [--database <id>] [--schema public] (--file <path> | --url <url> | --upload-id <id> | --result-id <id>) [--append] [--workspace-id <workspace_id>]
+hotdata databases tables add <table|schema.table> [--database <id>] [--schema public] [--key <col>]... [--key-determines <col>]... [--sorted-by <col>[=asc|desc]]... [--partition-by <col>[=identity|year|month|day|hour]]... [--output table|json|yaml]
+hotdata databases tables load <table> [--database <id>] [--schema public] (--file <path> | --url <url> | --upload-id <id> | --result-id <id>) [--mode replace|append|delete|update|upsert] [--append] [--format csv|json|parquet] [--key <col>]... [--workspace-id <workspace_id>]
 hotdata databases tables remove <table> [--database <id>] [--schema public] [--workspace-id <workspace_id>]
 ```
 
@@ -126,9 +128,11 @@ hotdata databases tables remove <table> [--database <id>] [--schema public] [--w
 - `unset` — clears the active database from config.
 - `<id>` — inspect one database (returns id, catalog, name, expires_at; a fork also shows its `forked_from` record).
 - `remove` — removes the instant database; clears the active-database config if it matched.
-- `load` (top-level shorthand) — loads parquet into `--catalog.--schema.--table`. Accepts `--file`, `--url`, `--upload-id`, or `--result-id` (load a saved query result by id — from `hotdata databases results` or a query's `[result-id: …]` footer — instead of a file; the result must belong to the target database). Replaces the table by default; pass `--append` to add rows to the existing table instead. If the table was not declared at create time, the CLI automatically deletes and recreates the database with the table declared, then retries the load.
+- `load` (top-level shorthand) — loads a file into `--catalog.--schema.--table`. Accepts `--file`, `--url`, `--upload-id`, or `--result-id` (load a saved query result by id — from `hotdata databases results` or a query's `[result-id: …]` footer — instead of a file; the result must belong to the target database). **Formats:** csv, newline-delimited json (`.json`/`.jsonl`/`.ndjson`), and parquet; the format comes from the file's extension, and `--format` overrides it (needed when the extension is absent or misleading). An unrecognised extension is not rejected — the server reads the bytes. A table or schema that was never declared is declared by the server as part of the load, so no up-front `--table` is required.
+- **Load modes** (`--mode`, default `replace`) — `replace` supersedes the table's contents; `append` adds rows; `delete`, `update`, and `upsert` match existing rows **by key**. `--append` is the old shorthand for `--mode append` and still works, but the two cannot be combined. The keyed modes need a key: declare one with `databases tables add --key`, or name it per-load with `--key` (repeat for a composite key). `delete` uploads only the key columns; `update` replaces matching rows and ignores unmatched ones; `upsert` inserts the unmatched instead. Keyed modes are not available with `--result-id`.
 - `tables list` — lists tables with `TABLE` (`<catalog>.<schema>.<table>`), `SYNCED`, `LAST_SYNC`. Uses active database when `--database` is omitted.
-- `tables load` — publishes to an instant-database table from a local parquet file (`--file`), a remote parquet URL (`--url`), a pre-staged upload (`--upload-id`), or a saved query result (`--result-id`, must belong to the target database). Defaults to **replace** mode; pass `--append` to add rows to the existing table instead.
+- `tables add` — declares a table **with its key and storage layout**, which a load cannot infer. `--key` (repeatable) is what enables the `delete`/`update`/`upsert` load modes on that table. `--sorted-by <col>` or `<col>=desc` sets sort order; `--partition-by <col>` partitions on the value, `<col>=month` (or `year`/`day`/`hour`) on a calendar part — one partition per calendar month needs **both** `<col>=year` and `<col>=month`, or every March shares a partition. Sort and partition are fixed once the table exists. `--key-determines` (repeatable) asserts a column's value is fixed by the key: it prunes keyed loads harder, and is **correctness-affecting** — declare it only where the invariant really holds, or a keyed load can leave a duplicate key behind. Re-adding an existing table is a conflict (409), and `tables remove` does not clear the declaration — the table leaves the listing but the name stays declared and still conflicts. So **a key cannot be retrofitted onto a table declared without one**: declare it with `--key` up front, or use a new table name.
+- `tables load` — publishes to an instant-database table from a local file (`--file`), a remote URL (`--url`), a pre-staged upload (`--upload-id`), or a saved query result (`--result-id`, must belong to the target database). Same `--mode`, `--format`, and `--key` flags as the top-level `load` above.
 - `tables remove` — drops a table from the instant database.
 - `attach` — attaches a **catalog** to an instant database, so the catalog's **live** tables become visible inside that database's query scope. Defaults to the active database; target another with `--database`. `--alias` sets the SQL name the catalog answers to (defaults to the catalog's name). This is how you query an attached catalog's tables and **join across catalogs** — see [Querying across catalogs](#querying-across-catalogs-attach).
 - `detach` — removes an attached catalog. Accepts the catalog name/id **or** the alias you attached it under. Defaults to the active database.
@@ -142,6 +146,20 @@ hotdata databases load --catalog airbnb --table listings --url https://example.c
 hotdata query "SELECT count(*) FROM airbnb.public.listings"
 ```
 
+Keeping a table in sync by key. Declare the key **before the table's first
+load** — `tables add` on a table that already exists returns 409, and a key
+cannot be added afterwards:
+
+```
+hotdata databases tables add bookings --key booking_id --sorted-by updated_at=desc
+hotdata databases load --catalog airbnb --table bookings --file bookings.csv
+hotdata databases load --catalog airbnb --table bookings --file changed.csv --mode upsert
+hotdata databases load --catalog airbnb --table bookings --file removed.csv --mode delete
+```
+
+`removed.csv` carries only the key columns. On a table already declared without
+a key, name one per load instead: `--mode upsert --key booking_id`.
+
 #### Querying across catalogs (attach)
 
 **A `hotdata query` runs inside exactly one instant database** — the active database (`hotdata databases use <id>`) or the one named by `--database`. With none set, the query fails with *"a database is required."* That database's query scope sees **only its own catalog plus any catalogs explicitly attached to it** — a workspace catalog is **not** visible just because it exists. Referencing an unattached catalog fails with *"table '\<catalog\>.\<schema\>.\<table\>' not found."*

skills/hotdata/references/WORKFLOWS.md

@@ -11,7 +11,7 @@ The `hotdata` skill is always loaded first (auth and workspace setup). The three
 | User goal | Skill | Key commands |
 |-----------|--------|----------------|
 | Login, workspaces, datasources, tables, context | **`hotdata`** | `auth`, `workspaces`, `ingest sources`, `ingest`, `databases tables`, `databases context` |
-| Load parquet files into an instant database | **`hotdata`** | `databases create` + `databases load` |
+| Load csv/json/parquet files into an instant database | **`hotdata`** | `databases create` + `databases tables add` + `databases load` |
 | SQL analytics, aggregations, history, Chain | **`hotdata-analytics`** (`subskills/analytics/SKILL.md`) | `query`, `databases queries`, `databases results` |
 | BM25 / vector search, retrieval indexes | **`hotdata-search`** (`subskills/search/SKILL.md`) | `search`, `search create`, `search embeddings` |
 | Geospatial / PostGIS-style SQL | **`hotdata-geospatial`** (`subskills/geospatial/SKILL.md`) | `query` with `ST_*`, WKB columns |
@@ -94,33 +94,56 @@ A `hotdata query` runs inside **one** instant database; its scope sees that data
 
 | | **Instant databases** |
 |---|------------------------|
-| **Best for** | Parquet files you own; catalog-style `alias.schema.table` |
+| **Best for** | Files you own (csv, newline-delimited json, parquet); catalog-style `alias.schema.table` |
 | **SQL prefix** | `<catalog>.<schema>.<table>` where catalog = `--catalog` alias |
 | **CLI** | `hotdata databases create --catalog` + `databases load` |
-| **Declare schema up front** | Yes — `--table` on create (auto-declared on first `databases load`) |
-| **Parquet file uploads** | `databases load --file` / `--url` / `--upload-id` |
-| **Refresh** | Replace via `databases load` again |
+| **Declare schema up front** | Optional — the load declares a missing table/schema. Declare with `databases tables add --key` when you need the keyed load modes; a key cannot be added later |
+| **File uploads** | `databases load --file` / `--url` / `--upload-id`; format from the extension, or `--format` |
+| **Refresh** | `databases load` again — `--mode replace` (default) or `append`, or `delete`/`update`/`upsert` against a declared key |
 
-**Rule of thumb:** Parquet files you control as **`mydb.public.orders`** → **instant databases**.
+**Rule of thumb:** Files you control as **`mydb.public.orders`** → **instant databases**.
 
-### Workflow: instant database (parquet)
+### Workflow: instant database (file upload)
 
 1. Create the database with a catalog alias:
 
    ```bash
    hotdata databases create --catalog sales
    ```
 
-2. Load parquet per table (tables are auto-declared if needed):
+2. **Before the first load**, declare any table that needs a key, a sort order,
+   or partitioning — the load cannot infer these, and a key cannot be added to
+   a table that already exists:
 
    ```bash
-   hotdata databases load --catalog sales --table orders --file ./orders.parquet
+   hotdata databases tables add orders --key order_id --sorted-by created_at=desc
+   ```
+
+   Skip this for a table you will only ever replace or append to.
+
+3. Load a file per table. A table or schema not declared above is declared as
+   part of the load, and the format comes from the file's extension:
+
+   ```bash
+   hotdata databases load --catalog sales --table orders --file ./orders.csv
    hotdata databases load --catalog sales --table customers --url https://example.com/customers.parquet
    ```
 
-   > Auto-declaring a *new* table recreates the database (no add-table API), which **changes its `id`** — the id returned by `databases create` goes stale after the next `load` of an undeclared table. Declare tables up front (`databases create --table orders --table customers`) to avoid the recreate, and don't cache ids across loads: re-read the current id from `databases list` at time of use. (Selection is still always by id — names and catalogs are not unique.)
+   > The database keeps its `id` and its other tables across a load into an
+   > undeclared table — nothing is recreated. Selection is still always by id,
+   > since names and catalogs are not unique.
+
+4. Keep a keyed table in sync by loading only what changed:
+
+   ```bash
+   hotdata databases load --catalog sales --table orders --file ./changed.csv --mode upsert
+   hotdata databases load --catalog sales --table orders --file ./removed.csv --mode delete
+   ```
+
+   `removed.csv` carries only the key columns. This needs the key declared in
+   step 2, or named per load with `--key order_id`.
 
-3. Confirm and query:
+5. Confirm and query:
 
    ```bash
    hotdata databases tables list

skills/hotdata/subskills/search/SKILL.md

@@ -37,6 +37,7 @@ hotdata search "<query>" --in <name>
 | **`vector`** | Pass plain-text query; name the **source text column** (e.g. `title`). Server embeds using the same provider/metric/dimensions as the index. SQL uses `vector_distance(col, 'text')`. Results sort by distance (ascending). |
 
 - **Index name:** the index carries its own type, column, and provider — you only name the index. Use `search list` to see available index names.
+- **`--select`:** defaults to the table's own columns. An auto-embed vector index materialises a `{column}_embedding` column on the table, and a search leaves it out — it is a 1536-float list in every row, tens of kilobytes per search nobody asked for. Ask for it back with `--select '*'`, or name it (`--select 'id,body_embedding'`); pair either with `--output json`, since `--output table` abbreviates long lists for display.
 - **Custom embedding model, raw query vector, or no vector index?** Use `hotdata query` directly (e.g. `cosine_distance(col, [<vec>])`) — `search` only auto-embeds the query text via the index's own provider.
 - **Before search:** create the right index (`search create <name> --type text` or `--type vector`). See [references/INDEXES.md](references/INDEXES.md).
 - Default `--limit` is 10.
@@ -69,7 +70,7 @@ hotdata search remove <name> [-d <db-id>]
 - **`--type` is required** on create: `text` (BM25; one or more text columns, comma-separated in `--column`) or `vector` (exactly one column; often embeddings or auto-embedded text). (`sorted` is also a valid `--type`, covered in **`hotdata-analytics`** — [`../analytics/SKILL.md`](../analytics/SKILL.md).)
 - **`sorted`** indexes (range/equality for OLAP filters) are documented in **`hotdata-analytics`** ([`../analytics/SKILL.md`](../analytics/SKILL.md)) — this skill focuses on retrieval types.
 - **`--async`:** poll with `hotdata jobs <job_id>` (see **`hotdata`** skill **Jobs**).
-- **Auto-embedding:** `--type vector` on a **text** column generates embeddings server-side. Optional `--provider`; default output column `{column}_embedding` (override with `--output-column`).
+- **Auto-embedding:** `--type vector` on a **text** column generates embeddings server-side. Optional `--provider`; default output column `{column}_embedding` (override with `--output-column`). The generated column becomes part of the table, so it shows up in `information_schema.columns` and in a hand-written `SELECT *` — `hotdata search` leaves it out of its own projection (see `--select` above).
 
 Full workflow (gather workload → compare existing → create → verify): [references/INDEXES.md](references/INDEXES.md).
 

src/cli.rs

@@ -119,7 +119,9 @@ pub enum Commands {
         #[arg(long, short = 'd')]
         database: Option<String>,
 
-        /// Columns to display (comma-separated, defaults to all)
+        /// Columns to display (comma-separated). Defaults to the table's own
+        /// columns; a vector index's generated embedding column is left out.
+        /// Pass `--select '*'` to include it.
         #[arg(long)]
         select: Option<String>,