feat(cli): resolve database names in flags, mark default workspace, name catalog ids as catalog - #280
feat(cli): resolve database names in flags, mark default workspace, name catalog ids as catalog#280eddietejeda wants to merge 8 commits into
Conversation
Connections are an internal concept now that ingest creates catalogs, but `databases show` still printed a raw `conn…` id as "catalog id:" and listed attachments by connection id, and show/create/fork exposed `default_connection_id` in -o json/yaml. - `databases show` (table): drop the "catalog id:" line; render attachments by the catalog name they're reachable as (alias, else the connection's name via one connections-list call, skipped when every attachment is aliased), never the raw id - `databases show`/`create`/`fork` (json/yaml): stop serializing `default_connection_id` and `attachments[].connection_id`; attachments expose `catalog` instead - keep the fields internally (skip_serializing) — the managed load/delete paths, the detach-by-alias fallback, and the connection resolver still need them
…kes an id The `-d/--database` flags on query, query status, queries, results, context, and search show/remove passed their value raw into the X-Database-Id header (or a database-scoped path), so `-d <name>` failed with a server "not found" while `databases show <name>` resolved it fine. `databases use <name>` failed the same way. - new `resolve_database_flag`: a `dbid…` value passes through untouched (no extra round trip on the common path); a catalog or name resolves to its id first, so names work everywhere ids do - extract load's active-database disambiguation into `try_resolve_database_preferring_active` and use it for every catalog/name lookup: when several databases share a catalog (e.g. `default` across create/fork), the active one wins instead of erroring as ambiguous - `search create --from <catalog>.…` now applies that same preference — it previously errored on an ambiguous catalog even when the active database's own catalog matched (and its error suggested setting an active database, which didn't actually help) - `databases use <name>` resolves names/catalogs; ids keep the 403-tolerant existence check so database API tokens still work - error-message grammar: "pass an instant database's catalog"
`workspaces list -o json` had no field identifying the workspace the CLI acts on: the table view marks it with a DEFAULT column, but the JSON only carried `active` — a server-side workspace-state flag that is true for every usable workspace, and a footgun for scripts that read it as "the current workspace". Add `default: bool` (exactly one true), stamped from the same source the table marker uses (HOTDATA_WORKSPACE, else the front of the configured list), and derive the table marker from it so the two views can't drift.
@eddietejeda - don't do this. connection ids will become catalog ids which should be exposed. they just haven't been fully updated yet. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
| pub fn resolve_database_flag(workspace_id: &str, database: Option<&str>) -> Option<String> { | ||
| let key = database?; | ||
| if is_database_id(key) { | ||
| return Some(key.to_string()); | ||
| } | ||
| let api = Api::new(Some(workspace_id)); | ||
| Some(resolve_database_preferring_active(&api, key).id) | ||
| } |
There was a problem hiding this comment.
nit: builds a second Api that every caller then builds again (not blocking).
Each call site is resolve_database_flag(workspace_id, database) followed by Api::new(Some(workspace_id)), so a name/catalog flag pays two Api::new runs — two config::loads and two ensure_access_token preflights. Taking &Api (and letting callers construct once, then .scoped_to_database_opt(...)) would avoid it and would also let the resolver reuse the caller's already-resolved ambient scope rather than re-deriving it. The Option<&str> → Option<String> shape can stay the same.
There was a problem hiding this comment.
Reviewed the full diff plus surrounding context in databases.rs, search.rs, workspace.rs, connections.rs, indexes.rs, and main.rs.
The three changes hold up:
skip_serializingonly affects theSerializepath, so the managed load/delete paths (managed_table_load_path),detach's alias→connection lookup,indexes::locate_by_name, andsearch create'sconn_idall still readdefault_connection_id/connection_idunchanged.resolve_database_flagshort-circuits on thedbidprefix, so the database-scoped-token path thatsetexplicitly guards is not regressed at the other call sites either.search::runfeedsquery::executealoc.database_id, which takes the passthrough branch. Keepingseton plainresolve_databaserather than the preferring-active variant is the right call — preferring the active database while changing it would be circular.- The
tables_loadextraction is behavior-preserving and slightly better:api.database_id()also honorsHOTDATA_DATABASE, which the old inlineload_current_databasedid not, and the active-match branch now reuses the detail response instead of re-fetching by id. workspaces listno longer loads config itself, butfetch_workspaces→Api::new(None)still exits on a config error beforedefault_workspace_idruns, so that path is preserved.
Note that databases show/create/fork -o json dropping default_connection_id is a user-visible output change for any script reading it; the conventional-commit prefixes should surface it in the generated changelog.
Three non-blocking comments inline. CI was still queued when this review started, so I have not seen the test or fmt results.
…ent display - an unresolved attachment keeps `catalog: null` in -o json/yaml instead of disguising the internal connection id as a SQL catalog name (an FQN built from it wouldn't be queryable); the table view falls back to the raw id itself, which detach still accepts - resolve_database_flag takes the caller's &Api instead of constructing a second one (a name/catalog flag paid two config loads + token preflights) - update the stale `--database` help on search show/remove and the top-level search command: the flag now takes an id, catalog, or name
|
Addressed the three review nits in 1b383ff: unresolved attachments now serialize |
| // `--database` accepts a catalog or name as well as an id, like every | ||
| // other database flag; locate_by_name needs the id. The Api is built only | ||
| // when there's a non-id flag to resolve — the id fast-path never needs it. |
There was a problem hiding this comment.
nit: the comment says the opposite of what the code does (not blocking).
"The Api is built only when there's a non-id flag to resolve — the id fast-path never needs it" isn't what happens: Api::new(Some(workspace_id)) is an argument, so it's evaluated for every Some(d), id or not. resolve_database_flag then returns key.to_string() without touching it. So search show -d dbid… now pays a config::load + ensure_access_token preflight to build an Api that is immediately discarded — and locate_by_name builds its own right after. Before 1b383ff the id path built nothing here, so this is a small regression on the fast path, at the one call site that can't reuse a caller-held Api.
Either make the construction lazy (e.g. expose is_database_id as pub(crate) and guard on it, or have the resolver take impl FnOnce() -> Api), or just drop the claim from the comment.
There was a problem hiding this comment.
Review
All three nits from the last cycle are addressed in 1b383ff — unresolved attachments now serialize catalog: null with the table view falling back to the raw id, resolve_database_flag takes the caller's &Api, and the three --database help strings read "id, catalog, or name". Not re-raising any of them.
Blocking Issues
-
src/commands/search.rs:332-334— formatting will failCI / fmt. The new closure block is a single expression that rustfmt collapses to|d| expr; at 93 columns it fits, so rustfmt emits the break-after-=form instead of what's committed. There is norustfmt.toml, somax_widthis the default 100.CI / fmthad not reported when this review started, so this comes from the formatting rules, not from a log. Suggestion posted inline. -
Unresolved maintainer objection to commit b14e54a. @zfarrell asked for the connection-id hiding to be dropped — "connection ids will become catalog ids which should be exposed. they just haven't been fully updated yet." That's a direction call on the first of the three changes here, not a code finding of mine, but it is unanswered and it governs roughly half the diff (
skip_serializingondefault_connection_id, theDatabaseAttachment.catalogfield,resolve_attachment_catalogs, and thegetdisplay changes). The other two changes stand on their own and could ship without it.
Action Required
- Reformat
locate_or_exitper the inline suggestion (or runcargo fmt). - Settle the direction question on commit 1 with @zfarrell — keep, drop, or split it out.
One non-blocking nit inline on the locate_or_exit comment, which claims an id fast-path the code does not actually take.
I reviewed the full diff as provided; I did not run tests, linters, or the binary.
Maintainer feedback on the PR: connection ids are becoming catalog ids and should stay exposed, so drop the hiding from b14e54a (and the catalog-resolution follow-up from 1b383ff) -- databases show prints the catalog id line and raw attachment ids again, and -o json/yaml keeps default_connection_id and attachments[].connection_id. Also reformat locate_or_exit (the closure block didn't survive cargo fmt) and correct its comment: the Api is built for id flags too; it's the resolve that short-circuits.
|
@zfarrell dropped the connection-id hiding in 5c8b23d — |
There was a problem hiding this comment.
Review
Blocking Issues
src/commands/workspace.rs:55—default_workspace_idusesconfig.workspaces.first(). Commands resolve their workspace throughcredentials::default_workspace_id(src/client/credentials.rs:75) instead. The two disagree for--api-keyandHOTDATA_API_KEYcredentials. A database token scoped to one workspace makesworkspaces list -o jsonmark zero workspacesdefault: true, so the new field breaks the invariant it promises. See the inline comment for the full scenario and the fix.
Action Required
- Load the profile and call
crate::client::credentials::default_workspace_id(&profile), keeping theHOTDATA_WORKSPACEcheck first.src/commands/auth.rsalready calls that helper. - Add a test covering an
ApiKeySource::Envprofile whose config cache front is not the workspace the credential authorizes.
Note (not blocking)
Commits b14e54a and 5c8b23d cancel each other. cliff.toml parses each commit and gives ^revert its own group (cliff.toml:83), so the release changelog gets a "Hide internal connection ids" line and a "Keep connection ids" line for a net no-op. Squash both away at merge. The PR title also still claims the connection ids are hidden.
Items 2 and 3 of the prior review are addressed. The revert of item 1 is clean: no references to resolve_attachment_catalogs or the attachment catalog field remain.
The new `default` field in `workspaces list` resolved from `config.workspaces.first()`, but commands resolve their workspace through credentials::default_workspace_id, which pins an env/flag api key's own authorized workspace. With HOTDATA_API_KEY set to a database token, the listing only contains the key's workspace while the config cache front names another -- so nothing was marked default and the "exactly one true" contract broke. Route the marker through the same helper resolve_workspace uses (HOTDATA_WORKSPACE still wins), and add a test for an Env-source profile whose cache front is not a workspace the credential authorizes.
|
Addressed the review blocker in 008643d: the |
There was a problem hiding this comment.
Prior blocking issue is resolved. The default marker now resolves through credentials::default_workspace_id, matching resolve_workspace in src/main.rs. The new test covers the env-credential case. No new blocking issues.
Note: CI / test had not reported when this review ran, so test results are unverified here.
runtimedb removed connections as a concept (the engine is push-only: create_connection rejects every source type but `managed`, and v53 migrated external catalogs), so a connection id IS a catalog id. Expose it under that name in `-o json`/`-o yaml`: `default_connection_id` becomes `default_catalog_id` and `attachments[].connection_id` becomes `catalog_id`. The table view already said "catalog id:"; the wire and the internal field names still say connection, so CreateDatabaseResponse renames on serialize only. BREAKING: scripts reading `default_connection_id` from `databases show/create/fork -o json` must switch to `default_catalog_id`.
|
@zfarrell following up on your note: runtimedb's connections removal has landed (push-only engine, |
There was a problem hiding this comment.
Prior blocking issue on the workspace default marker is resolved in 008643d. The serde renames in c92464b are serialize-only: Database and DatabaseAttachment derive no Deserialize, and wire parsing goes through DatabaseDetailResponse. Table output and JSON output now use the same catalog-id name.
CI / test was still in progress at review time, so this approval cites no test result.
Conflicts were confined to the databases.rs test module, where both sides appended tests at the same anchors: main's forked_from and lineage-tree tests now sit alongside this branch's catalog-id and flag-resolution tests. One semantic fixup: main added forked_from to Database, so this branch's literal-construction test gains forked_from: None. 410 unit + integration tests pass, clippy and fmt clean.
There was a problem hiding this comment.
Approved. The prior blocking finding is fixed: workspace.rs now resolves the default marker through crate::client::credentials::default_workspace_id, with HOTDATA_WORKSPACE checked first, and default_follows_the_credential_not_the_config_cache_front covers the credential-vs-cache case.
Checked the catalog-id rename for a deserialization hazard. Database and DatabaseAttachment derive Serialize only, so the plain #[serde(rename)] on those two cannot break wire parsing. CreateDatabaseResponse derives both and correctly uses rename(serialize = ...). No repo file reads default_connection_id back out of CLI JSON.
CI has not reported fmt or test yet at review time, so this approval does not assert those checks pass.
Live e2e after the conflict-resolution mergeAll against production (throwaway databases, cleaned up, config restored):
A finding that validates this PR's design choice: the Also: 410 unit + integration tests pass on the merged head, clippy zero warnings, fmt clean. |
Summary
Two UX fixes that came out of a full end-to-end sweep of every CLI command against production (plus adversarial re-verification of each finding against the source), plus the catalog-id rename that replaces the withdrawn connection-id hiding:
1.
-d <name>works everywhere-d <id>doesquery,query status,queries,results,context,search show/remove, anddatabases usepassed the flag value raw intoX-Database-Id(or a scoped path), so names failed with a server "not found" whiledatabases show <name>resolved fine. Newresolve_database_flagresolves catalogs/names to ids;dbid…values pass through with zero extra round trips. Also extractsload's active-database disambiguation into a shared helper and applies it tosearch create --from <catalog>.…, which previously errored on an ambiguous catalog even when the active database's own catalog matched.2.
workspaces list -o jsonmarks the default workspaceThe JSON only carried
active— a server-side state flag that is true for every usable workspace and a footgun for scripts reading it as "the current workspace" (all 11 workspaces in the test account reportactive: true). Newdefault: bool(exactly one true), same source as the table'sDEFAULT *marker, which now derives from it.3. Name catalog ids as catalog in json/yaml output
Replaces the withdrawn connection-id hiding (reverted in 5c8b23d per @zfarrell: "connection ids will become catalog ids which should be exposed"). runtimedb has since removed connections as a concept — the engine is push-only,
create_connectionrejects every source type butmanaged, and v53 migrated external catalogs — so a connection id is a catalog id, and the CLI now exposes it under that name:databases show/create/fork -o json/yamlemitdefault_catalog_id(wasdefault_connection_id) andattachments[].catalog_id(wasconnection_id). The table view already saidcatalog id:. Wire protocol and internal names are untouched (the server still sendsdefault_connection_id; ids keep theconnprefix until runtimedb renames them).Breaking: scripts reading
default_connection_idfrom-o jsonmust switch todefault_catalog_id.Testing
query/queries/results/context -d littlesis,databases use littlesis,search create --from default.…with active DB set (previously the exact ambiguity error),workspaces list -o jsonshows exactly onedefault: true, id paths unchanged,databases show -o jsonemitsdefault_catalog_idand the table view still printscatalog id: