feat(databases): load a json file in whatever shape it holds

Cargo.toml

@@ -39,7 +39,14 @@ serde = { version = "1", features = ["derive"] }
 # intercepts and serde_yaml does not — so every `-o yaml` output carrying a
 # `Value` number turns into a nested mapping. That breaks commands unrelated to
 # this one, which costs more than the rounding it buys.
-serde_json = "1"
+#
+# `raw_value` is on, for the json load's reshape (src/commands/json_rows.rs):
+# it hands each row over as its own JSON text, so a row is rewritten
+# byte-for-byte instead of round-tripping through `Value` — which would sort
+# its keys (`BTreeMap`, since `preserve_order` is off) and round a number
+# wider than i64/u64/f64. Unlike `arbitrary_precision` above, this feature
+# changes nothing for code that does not name `RawValue`.
+serde_json = { version = "1", features = ["raw_value"] }
 arrow = { version = "59", default-features = false, features = [
   "ipc",
   # Timestamps carrying a named IANA zone ("UTC", "America/New_York")

README.md

@@ -42,14 +42,15 @@ PostgreSQL-dialect SQL. Everything else builds on that.
 
 ## Getting your data in
 
-**Upload a file** directly — csv, newline-delimited json, or parquet:
+**Upload a file** directly — csv, json, or parquet:
 
 ```sh
 hotdata databases load --catalog demo --table listings --file ./listings.csv
 ```
 
 The format comes from the extension; pass `--format csv|json|parquet` when the
-extension is missing or misleading.
+extension is missing or misleading. json is read whatever shape it arrives in —
+an array of objects, a pretty-printed document, or one object per line.
 
 A load **replaces** the table by default. `--mode append` adds rows to an
 existing table instead:

skills/hotdata/SKILL.md

@@ -87,7 +87,7 @@ Returns workspaces with `public_id`, `name`, `active`, `favorite`, `provision_st
 
 **Instant databases** are Hotdata-owned catalogs you create and populate yourself — no remote source to sync. Query them in SQL as **`<database_id>.<schema>.<table>`**. Prefer **`hotdata databases`** for this workflow.
 
-**File formats:** `databases tables load` accepts **csv**, **newline-delimited json**, and **parquet** (local `--file`, remote `--url`, or a pre-staged `--upload-id`). The format is read from the file's extension; `--format csv|json|parquet` overrides it, and an unrecognised extension is left to the server to resolve from the bytes.
+**File formats:** `databases tables load` accepts **csv**, **json**, and **parquet** (local `--file`, remote `--url`, or a pre-staged `--upload-id`). The format is read from the file's extension; `--format csv|json|parquet` overrides it, and an unrecognised extension is left to the server to resolve from the bytes. **json is read in any shape** — an array of objects, a pretty-printed document, or one object per line: `--file`/`--url` reshape it to newline-delimited json before upload, so no `jq` step is needed. Every row must be an object, and the reshape is byte-faithful per row — key order and number digits are preserved. Two inputs are refused locally, before anything uploads: a row that is not an object, and a source with no rows at all (`[]`, or an empty file). Note the **load itself** reads json numbers as f64, whatever the shape: an integer wider than i64/u64 or a decimal past ~17 significant digits lands rounded — use parquet, or a string column, where the exact digits matter.
 
 **Active database:** `hotdata databases use <id>` saves the active database to config. `databases tables list`/`load`/`remove`, `databases queries`/`results`, and all `databases context` commands default to the active database; pass **`--database <id>`** to override per-command. (`databases tables show` instead takes a fully-qualified `catalog.schema.table`.)
 
@@ -109,7 +109,7 @@ hotdata databases attach <catalog|name> [--database <id>] [--alias <alias>]
 hotdata databases detach <catalog|name|alias> [--database <id>]
 
 # Preferred: load by catalog alias (server declares the table/schema if missing).
-# Loads csv, newline-delimited json, or parquet — format read from the extension.
+# Loads csv, json, or parquet — format read from the extension.
 hotdata databases load --catalog <alias> --table <table> [--schema public] (--file <path> | --url <url> | --upload-id <id> | --result-id <id>) [--mode replace|append|delete|update|upsert] [--append] [--format csv|json|parquet] [--key <col>]... [--workspace-id <workspace_id>]
 
 # Also available via tables subcommand
@@ -128,7 +128,7 @@ hotdata databases tables remove <table> [--database <id>] [--schema public] [--w
 - `unset` — clears the active database from config.
 - `<id>` — inspect one database (returns id, catalog, name, expires_at; a fork also shows its `forked_from` record).
 - `remove` — removes the instant database; clears the active-database config if it matched.
-- `load` (top-level shorthand) — loads a file into `--catalog.--schema.--table`. Accepts `--file`, `--url`, `--upload-id`, or `--result-id` (load a saved query result by id — from `hotdata databases results` or a query's `[result-id: …]` footer — instead of a file; the result must belong to the target database). **Formats:** csv, newline-delimited json (`.json`/`.jsonl`/`.ndjson`), and parquet; the format comes from the file's extension, and `--format` overrides it (needed when the extension is absent or misleading). An unrecognised extension is not rejected — the server reads the bytes. A table or schema that was never declared is declared by the server as part of the load, so no up-front `--table` is required.
+- `load` (top-level shorthand) — loads a file into `--catalog.--schema.--table`. Accepts `--file`, `--url`, `--upload-id`, or `--result-id` (load a saved query result by id — from `hotdata databases results` or a query's `[result-id: …]` footer — instead of a file; the result must belong to the target database). **Formats:** csv, json (`.json`/`.jsonl`/`.ndjson`), and parquet; the format comes from the file's extension, and `--format` overrides it (needed when the extension is absent or misleading). An unrecognised extension is not rejected — the server reads the bytes, and a file that plainly opens a json array is taken as json even without an extension. A json source in any shape (array, pretty-printed, one object per line) is reshaped locally to newline-delimited json before upload; an already-newline-delimited file is uploaded untouched. A table or schema that was never declared is declared by the server as part of the load, so no up-front `--table` is required.
 - **Load modes** (`--mode`, default `replace`) — `replace` supersedes the table's contents; `append` adds rows; `delete`, `update`, and `upsert` match existing rows **by key**. `--append` is the old shorthand for `--mode append` and still works, but the two cannot be combined. The keyed modes need a key: declare one with `databases tables add --key`, or name it per-load with `--key` (repeat for a composite key). `delete` uploads only the key columns; `update` replaces matching rows and ignores unmatched ones; `upsert` inserts the unmatched instead. Keyed modes are not available with `--result-id`.
 - `tables list` — lists tables with `TABLE` (`<catalog>.<schema>.<table>`), `SYNCED`, `LAST_SYNC`. Uses active database when `--database` is omitted.
 - `tables add` — declares a table **with its key and storage layout**, which a load cannot infer. `--key` (repeatable) is what enables the `delete`/`update`/`upsert` load modes on that table. `--sorted-by <col>` or `<col>=desc` sets sort order; `--partition-by <col>` partitions on the value, `<col>=month` (or `year`/`day`/`hour`) on a calendar part — one partition per calendar month needs **both** `<col>=year` and `<col>=month`, or every March shares a partition. Sort and partition are fixed once the table exists. `--key-determines` (repeatable) asserts a column's value is fixed by the key: it prunes keyed loads harder, and is **correctness-affecting** — declare it only where the invariant really holds, or a keyed load can leave a duplicate key behind. Re-adding an existing table is a conflict (409), and `tables remove` does not clear the declaration — the table leaves the listing but the name stays declared and still conflicts. So **a key cannot be retrofitted onto a table declared without one**: declare it with `--key` up front, or use a new table name.
@@ -146,6 +146,15 @@ hotdata databases load --catalog airbnb --table listings --url https://example.c
 hotdata query "SELECT count(*) FROM airbnb.public.listings"
 ```
 
+csv and json load on the same command, with nothing to convert first —
+`reviews.json` may hold `[{…}, {…}]`, one object per line, or a single
+pretty-printed object:
+
+```
+hotdata databases load --catalog airbnb --table hosts --file hosts.csv
+hotdata databases load --catalog airbnb --table reviews --file reviews.json
+```
+
 Keeping a table in sync by key. Declare the key **before the table's first
 load** — `tables add` on a table that already exists returns 409, and a key
 cannot be added afterwards:

skills/hotdata/references/WORKFLOWS.md

@@ -94,7 +94,7 @@ A `hotdata query` runs inside **one** instant database; its scope sees that data
 
 | | **Instant databases** |
 |---|------------------------|
-| **Best for** | Files you own (csv, newline-delimited json, parquet); catalog-style `alias.schema.table` |
+| **Best for** | Files you own (csv, json, parquet); catalog-style `alias.schema.table` |
 | **SQL prefix** | `<catalog>.<schema>.<table>` where catalog = `--catalog` alias |
 | **CLI** | `hotdata databases create --catalog` + `databases load` |
 | **Declare schema up front** | Optional — the load declares a missing table/schema. Declare with `databases tables add --key` when you need the keyed load modes; a key cannot be added later |

src/commands.rs

@@ -9,6 +9,7 @@ pub mod indexes;
 pub mod ingest;
 pub mod ingest_common;
 pub mod jobs;
+pub mod json_rows;
 pub mod prompt;
 pub mod queries;
 pub mod query;