Repository navigation
feat: the cost check holds over-limit dbt runs on Snowflake and Databricks - #39
Conversation
…s from the planner's estimate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lan estimates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| const heldNow = stored && stored.command === e.command ? { label: stored.target, bytes: stored.bytes, display: stored.display } : overNow.get(e.command) | ||
| if (over.length === 0 && failed && heldNow && heldNow.bytes > limitBytes) { | ||
| over.push({ label: heldNow.label, bytes: heldNow.bytes, point: heldNow.bytes, display: heldNow.display }) | ||
| over.push({ label: heldNow.label, bytes: heldNow.bytes, point: heldNow.bytes, display: heldNow.display, basis: 'basis' in heldNow ? heldNow.basis : undefined }) |
There was a problem hiding this comment.
SUGGESTION: The recorded hold loses where the estimate came from
A hold read back from CostState.held (line 378-379) has no basis, so this fallback keeps the number but drops the qualifier this PR adds. Concretely, on Snowflake: the first pass denies with ≈ 20.0 GiB would be scanned (Snowflake plan estimate), over the 10.0 GiB limit. …; a retry whose EXPLAIN now fails (failed, over empty, stored matching the command) denies with ≈ 20.0 GiB would be scanned, over the 10.0 GiB limit. … — the source silently disappears on the path that is already explaining why no fresh estimate exists. The same gap leaves bandLine (line 205) and the /allow-scan confirmation without the source.
Carrying basis (or the warehouse name, rendered with basisOf) through held in the two update calls that record it would keep every surface consistent and remove the 'basis' in heldNow narrowing here, since stored and overNow would then have the same shape. Not a correctness problem — the hold and the limit check are unchanged — but the message this PR is standardising is the one place it is missing.
There was a problem hiding this comment.
Fixed in c04a446. The recorded hold now keeps planner (CostState.held.planner, types/index.d.ts), and every surface renders it through basisOf: the deny for a hold kept while the retry's estimate fails, the band line, and /allow-scan (orders (≈ 20.0 GiB, Snowflake plan estimate)). The 'basis' in heldNow narrowing is gone, since stored and overNow now have the same shape. Regression test: "on Snowflake, a hold kept while the estimate fails on the retry still names the plan estimate, and so does /allow-scan" in cost.test.tsx. It fails on the previous head.
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Incremental review of Files Reviewed (2 files)
Previous Review Summaries (2 snapshots, latest commit c04a446)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit c04a446)Status: No Issues Found | Recommendation: Merge Incremental review of Files Reviewed (3 files)
Previous review (commit 914ce2f)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (5 files)
Verified against HEAD Reviewed by deepseek-v4.1-flash-1017 · Input: 51.3K · Output: 9.8K · Cached: 835.6K |
|
review-loop: APPROVED at 914ce2f (reviewers: gpt-6-astra high, gpt-6.1-sol high) |
The deny for a hold kept while the estimate fails on a retry, the band and /allow-scan now name the Snowflake or Databricks plan estimate too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at could not be recorded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
review-loop: APPROVED at e71c6ef (reviewers: gpt-6-astra high, gpt-6.1-sol high) |
sahrizvi
left a comment
There was a problem hiding this comment.
Consensus review (Claude as lead, Codex as an independent second reviewer): APPROVE.
No critical or major findings remain. On all 17 result shapes that altimate-code #1768 really returns, the gate makes the right call. Over the limit is held, and the deny, note, band, /scan-cost and /allow-scan text name the planner. Exactly at the limit, under it, or 0 bytes runs, with the source named. Each success: false reason (no statistics, failed EXPLAIN, unreadable plan, script refused, file read refused) runs, with one cost · no estimate for orders: … line. BigQuery behaves exactly as on main: same decisions and same text, checked side by side at base and head. claude plugin test: 1144 pass, 0 fail.
How this was tested. No Snowflake or Databricks warehouse was used, and no warehouse credentials were read. #1768's head (c3e8f88e03) was built locally. Its dbt_cost_estimate was run with a fake dbt adapter to record each result shape. Those answers were then fed through the plugin's test kit at head and at base, in place of the altimate tool run process. The npm 0.12.11 CLI was run for real. There was no live claude --plugin-dir session: an isolated config stops at the login screen.
Five minors remain, each as an inline comment. None blocks:
- Databricks warehouse start not mentioned in the setting text: wording only. Base already makes the same estimate call with the new CLI, and the call comes just before a command that starts the same warehouse anyway.
- The text doesn't say that only a single named model or
dbt show --inlineis estimated: wording, about behaviour that predates this PR. - A hold kept while the estimate keeps failing hides the failure, and
/allow-scan's "held again" isn't always true: an edge case, and the logic predates this PR. /scan-costwith no argument drops the planner name: cosmetic.- No test pins "exactly at the limit runs": test-only.
The dependency on unreleased altimate-code is a merge-order note, not a block (details in the PR comment). Nits, the merge with #38, and the test tables are in that same PR comment.
| "title": "Largest BigQuery scan without asking (not used yet)", | ||
| "description": "Not used yet: the BigQuery scan check isn't in this version, so no run is held. When it ships, a dbt run, build or show that BigQuery estimates would scan more bytes than this waits for your approval (0 never waits; other warehouses are never held).", | ||
| "title": "Largest dbt scan without asking", | ||
| "description": "A dbt run, build or show estimated to scan more bytes than this is held until you approve it with /allow-scan; 0 never holds. BigQuery's estimate is its dry run; Snowflake's and Databricks' is the query planner's (EXPLAIN), which can differ from what the run scans. A query with no estimate, and other warehouses, are never held. Optional; default: 10737418240 (10 GiB).", |
There was a problem hiding this comment.
Minor (wording, real): on Databricks the estimate is an EXPLAIN COST on the SQL warehouse the profile connects to. altimate-code #1768's tool description says this starts the warehouse if it is stopped, and it is billed while up. Neither this description nor the README row tells the user.
It doesn't block. The estimate runs only just before a dbt run, build or show of one named model (or show --inline) on the default target, which would start the same warehouse anyway. Only a hold, or an explicit /scan-cost, starts it without the command following. It runs one at a time per project, within a 120 s budget. Base already makes the same call once the new CLI is installed.
Suggest adding a short clause here and in the README row, e.g. "On Databricks the estimate runs on the SQL warehouse, which starts it if stopped." Codex rated this major at first, then accepted minor on that evidence.
| | --- | --- | --- | | ||
| | `dbt_mods` | on | Turns all of the dbt helpers (above) off when false. | | ||
| | `maxScanBytes` | 10 GiB | The largest BigQuery scan a dbt run, build or show may make before it waits for your approval (0 never waits; at least 0). Other warehouses are never held. | | ||
| | `maxScanBytes` | 10 GiB | A dbt run, build or show estimated to scan more bytes than this is held until you approve it with `/allow-scan` (0 never holds; at least 0). On BigQuery the estimate is a dry run; on Snowflake and Databricks it is the query planner's (`EXPLAIN`): Snowflake's bytes after pruning, an upper bound, and Databricks' table sizes, which a run usually reads less of. A query with no estimate (a Databricks table without statistics) runs, with a note saying why; other warehouses are never held. | |
There was a problem hiding this comment.
Minor (wording, real; the behaviour predates this PR): only a single plain model selector (-s orders, -m orders, --select=orders) or dbt show --inline is estimated. A plain dbt run or dbt build is never estimated or held, and gets the note no estimate for this selection (no model named; selection resolves with dbt ls). So on a 50-model project, a dbt build makes zero estimates. The same goes for +orders, two selectors, --target, --full-refresh, and dbt after cd …&& or echo x;.
This row and plugin.json read as if any run, build or show is checked. Suggest one sentence: "Only a command that names one model (-s orders) or dbt show --inline is estimated; others run with a note." Base has the same gate code (38 command forms gave identical results at base and head).
| // Or from an over-limit answer a parallel pass for the same command has in hand and has not recorded yet. | ||
| const stored = over.length === 0 && failed ? (await read($, costAtom))?.held : null | ||
| const heldNow = stored && stored.command === e.command ? { label: stored.target, bytes: stored.bytes, display: stored.display } : overNow.get(e.command) | ||
| const heldNow = stored && stored.command === e.command ? { label: stored.target, bytes: stored.bytes, display: stored.display, planner: stored.planner } : overNow.get(e.command) |
There was a problem hiding this comment.
Minor (edge case; the logic predates this PR): with the #1768 shapes, a model held at 40.0 GiB on Databricks whose retry gets "no statistics" stays held. The deny repeats ≈ 40.0 GiB would be scanned (Databricks plan estimate)…, but notes.delete drops the line with the failure reason and the ANALYZE TABLE … NOSCAN advice. So neither Claude nor the user learns that the current estimate failed.
After /allow-scan, the command runs once. The next retry, still with no estimate, also runs, even though /allow-scan said "after that it is held again".
plugin.json:34 ("A query with no estimate … never held") and README:161 ("runs, with a note") also leave out this kept-hold case. Suggest keeping the failure line in the deny (e.g. "… (last estimate; the estimate now fails: )"), and qualifying the two setting texts.
| /** The warehouse with no estimate (not BigQuery, Snowflake or Databricks, or an altimate-code too old for it), once the tool said so: nothing is estimated after that. */ | ||
| unsupportedAdapter: string | null | ||
| /** The newest estimate: what was estimated, in bytes, as the dry run wrote it. */ | ||
| /** The newest estimate: what was estimated, in bytes, as the dry run or the planner gave it. */ |
There was a problem hiding this comment.
Minor (cosmetic, real): lastEstimate keeps no planner. So /scan-cost with no argument (cost.tsx:159) shows Last estimate: orders would scan 12.0 GiB., without the source, right after /scan-cost orders said (Snowflake plan estimate). Adding planner? here and passing it to rememberEstimate would make it match the other surfaces.
There was a problem hiding this comment.
Fixed in #41: the last estimate keeps its planner (cost.tsx and types/index.d.ts), and /scan-cost with no model says "(Snowflake plan estimate)". Test added.
| await rememberEstimate($, label, answer.bytes, answer.display) | ||
| if (answer.maxBytes <= limitBytes) lines.push(`cost · ${label} would scan ${answer.display}`) | ||
| else over.push({ label, bytes: answer.maxBytes, point: answer.bytes, display: answer.display }) | ||
| if (answer.maxBytes <= limitBytes) lines.push(`cost · ${label} would scan ${answer.display}${basisOf(answer)}`) |
There was a problem hiding this comment.
Minor (test-only): changing this <= to < (which would hold a planner estimate of exactly maxScanBytes) still passes the whole cost suite (115/0). Dropping planner from the hold that refuse() records (around line 288, the pending/unreadable path) also passes.
These mutants are caught: turning off the explain branch (10 fail), dropping planner from the final hold record (4) or from heldNow (3), and dropping it from the /allow-scan text (2).
The code is right: a real #1768 answer of bytes: 10737418240 runs, and 10737418241 is held as "just over 10.0 GiB". It just isn't pinned by a test. Suggest an at-limit test for a planner estimate.
|
Consensus review notes (Claude as lead, Codex as an independent second reviewer). These go with the approval on e71c6ef. Gate decisions on altimate-code #1768's real result shapes (limit 10 GiB,
|
| Result shape | Warehouse | Head: held or runs | Text Claude gets | Base | Correct |
|---|---|---|---|---|---|
source: explain, 12.0 GiB |
Snowflake | held | ≈ 12.0 GiB would be scanned (Snowflake plan estimate), over the 10.0 GiB limit. … |
held, no source named | yes |
| 40.0 GiB | Databricks | held | ≈ 40.0 GiB would be scanned (Databricks plan estimate), … |
held | yes |
| exactly 10737418240 | Snowflake | runs | cost · orders would scan 10.0 GiB (Snowflake plan estimate) |
runs | yes |
| 10737418241 | Snowflake | held | just over 10.0 GiB would be scanned (Snowflake plan estimate), … |
held | yes |
| 2.0 GiB / 3.0 GiB | Snowflake / Databricks | runs | cost · orders would scan 2.0 GiB (Snowflake plan estimate) |
runs | yes |
bytes: 0 |
Snowflake | runs | … would scan 0.0 Bytes (Snowflake plan estimate) |
runs | yes |
| 5e18 bytes | Snowflake | held | ≈ 4440.9 PiB would be scanned (Snowflake plan estimate), … |
held | yes |
success: false: no statistics, failed EXPLAIN, unreadable plan, script refused, file read refused |
Databricks / Snowflake | runs | cost · no estimate for orders: <the tool's reason> |
runs | yes |
source but no bytes; bytes as a string |
Snowflake | runs | cost · no estimate for orders: altimate-code gave no estimate |
runs | yes |
no bound |
Snowflake | held | same as the 12.0 GiB row | held | yes |
dry run 12.0 GiB with maxBytes / 2.0 GiB / 0 with lowerBound |
BigQuery | held / runs / runs | the same text as main | the same | yes |
supported: false |
postgres, spark | runs | cost · no estimate on postgres: scans are estimated on BigQuery, Snowflake and Databricks only |
runs | yes |
supported: false (a BigQuery-only CLI) |
Snowflake, Databricks | runs | … this altimate-code estimates BigQuery only; Snowflake needs a newer altimate-code (the release with Snowflake and Databricks estimates) |
runs | yes |
Unknown tool (npm 0.12.11, run for real) |
any | runs | cost · no estimate for orders: this altimate-code has no cost estimate: it needs a newer altimate-code (the release with dbt_cost_estimate) |
runs | yes |
A held command when the estimate can't be made again
- Held, then fixed. Held at 12.0 GiB, the model is edited, and the retry estimates 2.0 GiB: the command runs and the band clears.
- Held, then no estimate. Held at 40.0 GiB (Databricks), and the retry gets "no statistics": the command stays held, with the old number. The band and
/allow-scanname the planner./allow-scanlets it run once, and the next retry (still no estimate) runs too. See the inline comment on cost.tsx:379.
Which command forms are held (orders over the limit, other under)
| Command | Held | What was estimated |
|---|---|---|
dbt run -s orders, -m orders, --select=orders, -s "orders", dbt build -s orders, dbt show -s orders --limit 5, uv run dbt run -s orders, dbt run -s orders | tee log |
yes | orders |
dbt show --inline "select …" |
yes | the inline query |
dbt run --select other |
no, runs with a note | other only |
dbt run, dbt build (no selection) |
no, a note: "no model named" | nothing |
-s +orders, -s orders other, --target prod, DBT_TARGET=prod …, --full-refresh, cd /repo && …, git status && …, echo x; … |
no, a note says why | nothing |
bash -c "dbt run -s orders", sh -c …, eval "…", timeout 600 dbt run …, python -m dbt …, npx dbt … |
no, no note | nothing |
dbt compile, dbt test, dbt docs generate, dbt seed, dbt snapshot, dbt retry, dbt clone |
not held, by design | nothing |
All 38 forms gave the same result at base and at head. The gaps belong to the gate code this PR builds on, not to this PR. Claude can get past the check by rewording (bash -c, timeout), but the check is a guard against accidental cost, as before.
Test results (run one after another)
| Check | base e014863 | head e71c6ef | head merged onto #38's head 3f40e70 |
|---|---|---|---|
claude plugin validate plugins/altimate-code |
pass | pass | pass |
claude plugin validate --strict . |
pass | pass | pass |
claude plugin validate --strict plugins/altimate-code |
fail: the plugin has no icon (ICON_MISSING), plus TS2304 setTimeout warnings in two test files |
the same | the same |
claude plugin test plugins/altimate-code |
1127 pass, 1 fail | 1144 pass, 0 fail | 1186 pass, 1 fail |
tests/*.sh (8) |
all pass | all pass | all pass |
python3 -m pytest tests/ |
65 pass | 65 pass | 65 pass |
The one failure at base and on the merge is the runs.test.tsx test "a dbt call that starts while another call reads its artifact". It depends on timing: run alone, it fails in 1 or 2 of 3 runs at base, at head and on the merge. CONTRIBUTING.md names no separate tsc step.
Head's cost.test.tsx with main's cost.tsx gives 101 pass, 14 fail.
Mutation checks, where a mutant is "caught" when at least one test fails:
- Caught: turning off the explain branch, and dropping
plannerfrom the final hold, fromheldNowor from the/allow-scantext. - Not caught: the at-limit comparison, and the hold
refuse()records (see the inline comment on cost.tsx:357).
Nits
- The body says "11 fail with main's cost.tsx". It is now 14: the band and hold tests added in c04a446 and e71c6ef also fail on main.
- The body says that until the release is installed, Snowflake and Databricks get "a line saying a newer altimate-code is needed". With the released 0.12.11, that line is the "has no cost estimate … (the release with dbt_cost_estimate)" one. The "estimates BigQuery only" line shows only with an unreleased altimate-code main build made after #1732 and before #1768. The messages name no version, which is right.
- The pending deny (cost.tsx:372) says "the dbt adapter may be starting". On Databricks the wait is often the SQL warehouse starting.
- The
--strictfailure on the plugin folder and the flakyrunstest happen on main too. This PR didn't cause them.
Merge order and the conflict with #38
- Dependency. No released altimate-code has
dbt_cost_estimate: npm 0.12.11 answersUnknown tool. The BigQuery version (#1732) and this one (#1768) are both only on altimate-code main or in an open PR. Until a release carries both, the check holds nothing on any warehouse. Each estimated command gets the "needs a newer altimate-code" line. This is a release-order note, not a block. The settings text that callsmaxScanBytesused comes from fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38 as well. - Conflict with fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38 (3f40e70).
git merge-treefinds conflicts inplugin.json(themaxScanBytestitle and description) andREADME.md. In the README the conflicting lines are the settings intro, themaxScanBytesrow and thehealthBlocksPushrow next to it, which fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38 also rewrote. - What the rebase must keep:
- fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38's intro: "seven settings", "All are optional …", the comments settings shown as used, and
show_edit_impact. - fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38's default text
10737418240 (10 GiB). - fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38's
healthBlocksPushrow. - This PR's
maxScanBytestitle, description and row text. - The intro's "by the BigQuery scan check" changed to "by the scan check (BigQuery, Snowflake, Databricks)".
- It must also drop this PR's sentence "The comments settings are not used yet". Resolved that way, the merge gives the results in the table above.
- fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38's intro: "seven settings", "All are optional …", the comments settings shown as used, and
…lake-databricks # Conflicts: # plugins/altimate-code/.claude-plugin/plugin.json # plugins/altimate-code/README.md
|
review-loop: APPROVED at f35afbe (reviewers: gpt-6-astra high, gpt-6.1-sol high) This head merges main (with #38) into the branch. Conflicts were in One MINOR from gpt-6-astra, not fixed here: after a Snowflake or Databricks estimate, |
What
The scan check (
hooks/features/cost.tsx) holds a dbtrun,buildorshowwhose estimated scan is overmaxScanBytes, but it only worked on BigQuery. Everywhere else it said "only BigQuery has a dry run" and held nothing. It now uses altimate-code's new Snowflake and Databricks estimates, from the query planner'sEXPLAIN.Depends on AltimateAI/altimate-code#1768 (
dbt_cost_estimatereadsEXPLAIN USING JSONon Snowflake andEXPLAIN COSTon Databricks). Until that release is installed, Snowflake and Databricks projects get a line saying a newer altimate-code is needed, and nothing is held.source: "explain", the gate checksbytesagainst the limit, since a planner estimate has no rounding bound. The deny, the note under the Bash row,/scan-cost, the "allowed once" note, the band's held line and/allow-scanall add where the number came from. The recorded hold keeps the planner (CostState.held.planner), so a hold kept while the retry's estimate fails still names it:≈ 20.0 GiB would be scanned (Snowflake plan estimate), over the 10.0 GiB limit. …cost · orders would scan 2.0 GiB (Databricks plan estimate)cost · no estimate for <model>: <reason>, and the command runs. This is per query, not per session, so the next model is asked again. As before, a command held earlier stays held while its estimate can't be made again.supported: falsefor snowflake or databricks getsthis altimate-code estimates BigQuery only; Snowflake needs a newer altimate-code (the release with Snowflake and Databricks estimates). Other warehouses getscans are estimated on BigQuery, Snowflake and Databricks only.sourceis a dry run: samemaxBytes/lowerBoundhandling, same text. A test pins the BigQuery note.maxScanBytesinplugin.json) and README: the limit stays in bytes. On BigQuery it is checked against the dry run; on Snowflake and Databricks against the planner's estimate, which can differ from what the run scans. Snowflake's figure is an upper bound (bytes after compile-time pruning). Databricks' figure is table sizes, of which a run usually reads less. Command descriptions no longer say BigQuery only.Accuracy (from the altimate-code PR)
GlobalStats.bytesAssigned, documented by Snowflake as an upper-bound estimate. Runtime pruning can scan less, and a cached result scans nothing.sizeInBytesover the plan's scanned relations. These are file and table sizes before column pruning and file skipping. A table without statistics gives no estimate.Verification
claude plugin test plugins/altimate-code: 1144 pass, 0 fail.cost.tsx: 1129 pass, 11 fail. The 11 are the new behaviours: hold and pass on Snowflake and Databricks, older-CLI wording on each, the allowed-once note, re-asking after a no-estimate model, both/scan-costtexts, and the duckdb wording. The two new tests that also pass on main are guards: BigQuery output unchanged, and Databricks no-stats runs with its reason.claude plugin validate --strict .passes.bash tests/*.sh: all 8 pass.Merge order and conflicts
fix/cost-runs-together(test: cost and runs tests pass with both features on #37, already on main): no conflict.fix/plugin-launch-blockers(fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38):git merge-treeconflicts inplugins/altimate-code/.claude-plugin/plugin.jsonandplugins/altimate-code/README.md, on the samemaxScanBytestext that fix: launch blockers — no false zero in the column summary, no start-up nags, old CLI shown, narrow-terminal line, settings text (1.8.1) #38 also rewrites.cost.tsx, its tests andtypes/index.d.tsmerge cleanly.🤖 Generated with Claude Code