feat(databases): load csv/json, keyed load modes, and table declaration

[anoop-narang]

The CLI offered less than the load API accepts. Four capabilities the API documents were unreachable from the command line, and a vector search returned a column nobody asked for.

Loads are no longer parquet-only

--file and --url rejected anything not ending in .parquet. The load API takes csv, newline-delimited json, and parquet, so the gate is gone:

hotdata databases load --catalog demo --table listings --file ./listings.csv

The format comes from the extension (.json/.jsonl/.ndjson all mean newline-delimited json), and --format csv|json|parquet overrides it when the extension is missing or misleading. --format is rejected with --result-id, which is always parquet. An unrecognised extension is no longer a client-side failure — the upload announces application/octet-stream and the server resolves the format from the bytes.

Dropping the gate alone would not have worked. The upload's recorded content type was pinned to the parquet MIME type, and the load treats a confidently wrong content type as authoritative rather than sniffing past it, so a csv announced as parquet fails with failed to parse parquet metadata: Corrupt footer. The content type now follows the file. For --url it is derived from the URL rather than from the staged temp file, whose generated name says nothing about the bytes inside.

All five load modes

--mode replace|append|delete|update|upsert. Only replace and append were reachable before, through --append.

The keyed modes match existing rows by key, so they need one: declare it on the table (below), or name it per load with --key, repeatable for a composite key.

hotdata databases load --catalog demo --table listings --file changed.csv --mode upsert
hotdata databases load --catalog demo --table listings --file removed.csv --mode delete --key listing_id

--append still works as the shorthand for --mode append; the two cannot be combined. Keyed modes are refused with --result-id, which carries whole rows rather than a key-shaped upload.

databases tables add

Declares a table with the key and storage layout a load cannot infer:

hotdata databases tables add events \
  --key event_id \
  --sorted-by created_at=desc \
  --partition-by created_at=year --partition-by created_at=month

Declaring --key is what enables the keyed load modes on that table. --sorted-by and --partition-by are fixed once the table exists. --partition-by takes a calendar transform after = (year, month, day, hour) and defaults to the value itself — one partition per calendar month needs both year and month, or every March shares a partition. --key-determines asserts a column's value is fixed by the key; it prunes keyed loads harder and is correctness-affecting, so it is documented as such.

Declare the key before the table's first load. tables add on a table that already exists returns 409, and tables remove does not clear the declaration — the table leaves tables list but the name stays declared and still conflicts. So a key cannot be retrofitted onto a table declared without one; the CLI's no-key hint and the docs both say so rather than suggesting a recovery that fails. This is a server-side constraint, not something the CLI can work around.

This also removes a delete-and-recreate path from the load. A load into a table that was never declared used to delete the entire database and recreate it with the table declared, warning that loaded data would be lost and returning a new database id. The load endpoint now declares a missing table (and a missing schema) itself, so that branch could not be reached; it is removed rather than rewritten.

Vector search no longer dumps the embedding column

An auto-embed vector index materialises a {column}_embedding column on the table, and hotdata search issued SELECT * — so every result row carried a 1536-float list, tens of kilobytes per search, unrequested.

Before and after on the same index:

columns: [id, title, body, body_embedding, dist]     # ~6 KB of floats per row
columns: [id, title, body, dist]

It is excluded with a wildcard EXCLUDE, taken from the index's own metadata rather than guessed from the column name. This changes hotdata search output for anyone reading the embedding out of it: --select '*' asks for it back, and naming the column (--select 'id,body_embedding') still reaches it. Pair either with --output json, since --output table abbreviates long lists for display.

--output csv no longer abbreviates lists

The csv writer shared the terminal's abbreviator, so a long list value came out as [1, 2, 3, ..., 9] (1536 items) in a csv cell — lossy in a format read by programs. Csv now renders list values whole. The table renderer still abbreviates, where a human is reading and terminal width is the constraint.

Testing

Unit and CLI-surface tests cover the request bodies, the extension-to-format mapping, the sort/partition parsing and its rejections, the flag conflicts, and the search projection including the quoting of user-named index columns.

Every path was also exercised against the API by hand, asserting destination row counts rather than trusting status codes:

• csv, .jsonl, and extension-free (.dat) uploads via --file; csv via --url; csv misnamed .parquet reproducing Corrupt footer, then succeeding with --format csv
• replace, append, --append, upsert (rows updated and inserted), delete (rows removed), update (matched row changed, unmatched ignored)
• tables add with key, key_determines, two calendar partitions on one column, and a sort direction, followed by loads into the resulting table
• a load into a never-declared table and a never-declared schema, confirming the database id and a sibling table's rows both survive
• a real auto-embed index, diffed against the released CLI, plus both --select escape hatches

Later commits added: the two hand-rolled rejections clap cannot express (a keyed --mode with --result-id, and --key-determines without --key), the layout parse rejections, the extension parser's URL cases, and the sort/partition renderers.

Both worked examples in the docs were run verbatim against the API — declare with key and sort, load, upsert, delete, read back — which is how the ordering bug in them was found.

The integration job needs HOTDATA_SDK_TEST_* credentials and skips without them, so none of the live checks re-run automatically; the tests added here are unit and parse-time.

[claude]

Reviewed the full diff. Four non-blocking comments inline: --format silently dropped with --result-id, URL query strings defeating the extension lookup, a missing test for the keyed-mode guard, and the omitted layout in tables add output.

CI was still queued or in progress at review time, so the test results are unknown.

[claude]

Cycle 3: prior blocking items are none, and the four cycle-2 nits are addressed or answered. One new nit inline on the no-key hint. CI / test was still running at review time, so this approval rests on reading the diff, not on a passing test run.

Note: the .suffix(".parquet") thread (#3959092816) has no reply yet and stays open.

[codecov]

Codecov Report

❌ Patch coverage is 75.10730% with 116 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/commands/databases.rs	67.51%	102 Missing ⚠️
src/commands/indexes.rs	0.00%	6 Missing ⚠️
src/main.rs	80.00%	6 Missing ⚠️
src/commands/query.rs	0.00%	1 Missing ⚠️
src/commands/search.rs	98.52%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[claude]

The full diff was omitted from the review prompt. Reviewed the patch fetched with gh pr diff 293, plus src/commands/query.rs, src/commands/indexes.rs, and src/commands/databases.rs from the checkout.

All four prior threads are resolved in code:

• --format now carries conflicts_with = "result_id" on both load commands, with a surface test.
• extension_of truncates at ? and #, and both content_type_for_path and format_for_path share it.
• databases_load_rejects_a_keyed_mode_with_result_id and databases_tables_add_rejects_key_determines_without_a_key cover the two hand-rolled guards.
• tables add echoes sorted: and parts:.
• download_to_temp takes the suffix from extension_of(url).

Checked two things that the removal of the delete-and-recreate branch could have broken: collect_tables, create_database_request, and CreateDatabaseResponse all still have live callers, and value_to_string in src/commands/query.rs:102 reaches only the csv arm of print_result, so the table renderer keeps its abbreviation.

No blocking issues. CI was still queued or in progress at review time, so the test results are unknown.

[claude]

The full diff was omitted from the review prompt. Fetched it with gh pr diff 293 and read it in full, plus src/commands/query.rs, src/commands/indexes.rs, and README.md from the checkout.

All four prior threads are resolved in code: --format now conflicts with --result-id on both load commands, extension_of truncates at ?/#, the staged temp file takes the source URL's extension, the keyed-mode and --key-determines guards have surface tests, and tables add echoes sorted: and parts:.

Verified separately: is_parquet_path has no remaining callers, truncate_array is still used by the table renderer, and value_to_string reaches only the -o csv branch (src/commands/query.rs:650), so dropping abbreviation there does not change table output.

One non-blocking docs nit left inline.

[claude]

Reviewed the full patch. The diff was omitted from the review context, so it was fetched with gh pr diff; src/commands/query.rs and src/commands/databases.rs were also read from the checkout.

All prior threads are resolved in code. No blocking issues.

CI is still queued or in progress for every check except CI / changelog. This approval does not assert that the test job passes.