Repository navigation
feat(tool): dbt_run_results reads the last dbt run's id, timings and per-node results #1729
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
b7cfc04
feat: foundation for the dbt Power User tools
hkaltai 1eb722e
Merge remote-tracking branch 'origin/main' into feat/pu-mods-foundation
hkaltai 1ee47f3
feat(tool): dbt_run_results reads the last dbt run's id, timings and …
hkaltai 21d0494
fix: harden the dbt Power User tool foundation
hkaltai 3728f95
fix(tool): dbt_run_results reads one coherent run, bounds its answer …
hkaltai 2c46312
fix(tool): dbt_run_results looks for the project above the folder it …
hkaltai 60bca0f
test(tool): name the nested-folder case as what it checks
hkaltai 2e84742
Merge branch 'feat/pu-mods-foundation' into feat/dbt-run-results-tool
hkaltai 6d497bb
Merge remote-tracking branch 'origin/main' into feat/dbt-run-results-…
hkaltai 1aec5ac
fix(tool): dbt_run_results reads each run from one parse, takes a tar…
hkaltai b1e36d6
fix(validators): a run_results.json rewritten during the read is no e…
hkaltai 8c67725
fix(validators): detect a rewrite during the run_results.json read by…
hkaltai a609c63
fix(validators): run_results.json keeps the filesystem's own mtime ro…
hkaltai 4845d65
fix(tool): review follow-ups — falsey API bodies, one stat snapshot f…
hkaltai d8c3469
fix: send falsy API bodies, follow cancellation in apiRequest, check …
hkaltai dace1c2
Merge remote-tracking branch 'origin/main' into feat/pu-mods-foundation
hkaltai 769a7ac
Merge remote-tracking branch 'origin/main' into feat/dbt-run-results-…
hkaltai 18a3fc4
fix: keep the stored API key out of API errors, bound the dbt config …
hkaltai 6024ac9
fix: redact the API key when a refused header races the caller's abor…
hkaltai f5159c5
docs: say which tools use the dbt adapter helper
hkaltai f57de3d
docs: rewrap and correct the adapter helper comments
hkaltai e65a30f
Merge origin/feat/pu-mods-foundation into feat/dbt-run-results-tool
hkaltai aaff414
fix: never echo a send failure's text, so a trimmed key can't leak
hkaltai 3a98dff
fix: a rejection with no reason is a send failure, not an abort
hkaltai 4773866
Merge remote-tracking branch 'origin/feat/pu-mods-foundation' into fe…
hkaltai d0d99c7
Merge remote-tracking branch 'origin/main' into feat/dbt-run-results-…
hkaltai 32848a3
test: abort the in-flight request only after the fetch started
hkaltai 2025857
Merge origin/main: take main's foundation files, keep only dbt_run_re…
hkaltai cccd13f
Merge remote-tracking branch 'origin/main' into feat/dbt-run-results-…
hkaltai 910f061
fix(tool): dbt_run_results reads a run written with --target-path fro…
hkaltai 2576266
fix(tool): dbt_run_results parses the artifact once and bounds the ec…
hkaltai 79bddf3
Merge origin/main into feat/dbt-run-results-tool
hkaltai File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
147 changes: 141 additions & 6 deletions
147
packages/opencode/src/altimate/tools/dbt-run-results.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,19 +1,154 @@ | ||
| import path from "node:path" | ||
| import z from "zod" | ||
| import { Tool } from "../../tool/tool" | ||
| import { projectRoot } from "../native/connections/registry" | ||
| import { isFailedRunStatus, readRunResults } from "../validators/validator-utils" | ||
|
|
||
| /** A message longer than this is cut: a build of thousands of nodes must stay readable and small. */ | ||
| const MAX_MESSAGE = 2000 | ||
| /** An id, name, status or command longer than this is cut: the artifact decides how long they are, the answer must not follow. */ | ||
| const MAX_IDENTIFIER = 300 | ||
| /** The rows one answer carries: a plugin reads at most 4 MiB of it, so a bigger build is counted, not listed. */ | ||
| const MAX_ROWS = 200 | ||
|
|
||
| export interface DbtRunResultRow { | ||
| uniqueId: string | ||
| name: string | ||
| status: string | ||
| executionTimeSec: number | null | ||
| message: string | null | ||
| } | ||
|
|
||
| export interface DbtRunResultsView { | ||
| found: boolean | ||
| /** The dbt project folder the artifact was read from. */ | ||
| projectDir: string | null | ||
| invocationId: string | null | ||
| command: string | null | ||
| elapsedSec: number | null | ||
| /** The artifact's modification time, ISO 8601. */ | ||
| completedAt: string | null | ||
| /** Every node dbt reported; `results` holds at most MAX_ROWS of them, the failures first. */ | ||
| total: number | ||
| passed: number | ||
| failed: number | ||
| skipped: number | ||
| results: DbtRunResultRow[] | ||
| } | ||
|
|
||
| const NONE: DbtRunResultsView = { | ||
| found: false, | ||
| projectDir: null, | ||
| invocationId: null, | ||
| command: null, | ||
| elapsedSec: null, | ||
| completedAt: null, | ||
| total: 0, | ||
| passed: 0, | ||
| failed: 0, | ||
| skipped: 0, | ||
| results: [], | ||
| } | ||
|
|
||
| /** A test's unique id ends in a hash (`test.shop.not_null_orders_id.f3a1c`): its name is the part before it. */ | ||
| function displayName(uniqueId: string, name: string): string { | ||
| const parts = uniqueId.split(".") | ||
| return parts[0] === "test" && parts.length === 4 ? (parts[2] ?? name).toLowerCase() : name | ||
| } | ||
|
|
||
| /** Statuses of a node dbt did not run. */ | ||
| const NOT_RUN = new Set(["skipped", "no-op"]) | ||
|
|
||
| function cut(text: string | null, max = MAX_MESSAGE): string | null { | ||
| if (text === null) return null | ||
| return text.length > max ? `${text.slice(0, max)}…` : text | ||
| } | ||
|
|
||
| /** | ||
| * The last dbt run of a project: `readRunResults` gives the path, the command and each row's status and | ||
| * message, the invocation id, the elapsed time and the execution times, all from one parse of the file. | ||
| */ | ||
| export async function readRunView(dbtRoot: string, targetPath?: string): Promise<DbtRunResultsView> { | ||
| const artifact = await readRunResults(dbtRoot, targetPath) | ||
| if (!artifact) return NONE | ||
| const all = artifact.results.map( | ||
| (node): DbtRunResultRow => ({ | ||
| uniqueId: cut(node.uniqueId, MAX_IDENTIFIER) ?? "", | ||
| name: cut(displayName(node.uniqueId, node.name), MAX_IDENTIFIER) ?? "", | ||
| status: cut(node.status, MAX_IDENTIFIER) ?? "", | ||
| executionTimeSec: node.executionTimeSec, | ||
| message: cut(node.message), | ||
| }), | ||
| ) | ||
| const failed = all.filter((row) => isFailedRunStatus(row.status) && !NOT_RUN.has(row.status)) | ||
| const skipped = all.filter((row) => NOT_RUN.has(row.status)).length | ||
| const failing = new Set(failed) | ||
| // The failures first, then the rest in the file's order, whatever the size: a reader that stops early still sees them. | ||
| const listed = [...failed, ...all.filter((row) => !failing.has(row))].slice(0, MAX_ROWS) | ||
| return { | ||
| found: true, | ||
| projectDir: dbtRoot, | ||
| invocationId: cut(artifact.invocationId, MAX_IDENTIFIER), | ||
| command: cut(artifact.command, MAX_IDENTIFIER), | ||
| elapsedSec: artifact.elapsedSec, | ||
| completedAt: new Date(artifact.mtimeMs).toISOString(), | ||
| total: all.length, | ||
| passed: all.length - failed.length - skipped, | ||
| failed: failed.length, | ||
| skipped, | ||
| results: listed, | ||
| } | ||
| } | ||
|
|
||
| export const DbtRunResultsTool = Tool.define("dbt_run_results", { | ||
| description: "The latest dbt run's results from target/run_results.json: command, duration and each node's status.", | ||
| description: | ||
| "Read the last dbt run of a project from its target/run_results.json (or the folder `target_path` names): the invocation id, the dbt command, the elapsed seconds, when it finished, the counts, and one row per node (the failures first, at most 200) with its status, execution seconds and message. Read-only; runs nothing. A compile or parse writes success rows without building anything, so look at `command`.", | ||
| parameters: z.object({ | ||
| project_dir: z | ||
| .string() | ||
| .optional() | ||
| .describe("dbt project folder (default: the dbt project found from the working directory)"), | ||
| .describe("A folder in the dbt project (its dbt_project.yml is looked for there and above; default: the project the session's directory is in, else the one the dbt adapter is configured for)"), | ||
| target_path: z | ||
| .string() | ||
| .optional() | ||
| .describe("The folder the dbt run wrote its artifacts to when it was not the project's own target folder (a relative one is from the project)"), | ||
| }), | ||
| async execute() { | ||
| async execute(args) { | ||
| const root = args.project_dir ? await projectAbove(path.resolve(projectRoot(), args.project_dir)) : await defaultRoot() | ||
| const view = await readRunView(root, args.target_path) | ||
| if (!view.found) { | ||
| return { | ||
| title: "dbt run results: none", | ||
| metadata: { ...view }, | ||
| output: `No readable run_results.json for the dbt project at ${root}${args.target_path?.trim() ? ` (in ${cut(args.target_path.trim(), MAX_IDENTIFIER)})` : ""}: no dbt command has written one there yet, or the file is not a run artifact.`, | ||
| } | ||
| } | ||
| const command = view.command ? `dbt ${view.command}` : "dbt (unknown command)" | ||
| return { | ||
| title: "dbt_run_results: not implemented", | ||
| metadata: { success: false, error: "not implemented" }, | ||
| output: "dbt_run_results is not implemented in this build of altimate-code yet.", | ||
| title: `${command}: ${view.passed}/${view.total} ok`, | ||
| metadata: { ...view }, | ||
| output: [ | ||
| `${command} · invocation ${view.invocationId ?? "unknown"} · ${view.elapsedSec ?? "?"}s · ${view.completedAt}`, | ||
| ...view.results.map( | ||
| (row) => `${row.status} ${row.name} ${row.executionTimeSec ?? "-"}s${row.message ? ` · ${row.message}` : ""}`, | ||
| ), | ||
| ...(view.total > view.results.length ? [`… and ${view.total - view.results.length} more nodes not listed`] : []), | ||
| ].join("\n"), | ||
| } | ||
| }, | ||
| }) | ||
|
|
||
| /** The dbt project `dir` is in: the closest folder at or above it holding dbt_project.yml, else `dir`. */ | ||
| async function projectAbove(dir: string): Promise<string> { | ||
| const { findProjectRoot } = await import("../../../../dbt-tools/src/config") | ||
| return findProjectRoot(dir) ?? dir | ||
| } | ||
|
|
||
| /** The dbt project the session's directory is in, else the one the dbt adapter is configured for (as the other dbt tools choose). */ | ||
| async function defaultRoot(): Promise<string> { | ||
| const { findProjectRoot } = await import("../../../../dbt-tools/src/config") | ||
| const here = findProjectRoot(projectRoot()) | ||
| if (here) return here | ||
| const { getDbtProjectRoot } = await import("../native/connections/register") | ||
| return (await getDbtProjectRoot()) ?? projectRoot() | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.