Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions docs/docs/data-engineering/tools/finops-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,37 @@ Classic (sql warehouse, workspace 525779153077873) · Migrate Classic/Pro to Ser
$29.29 in DBUs over the window; serverless would bill $4.44 for the busy time alone
```


---

## finops_snowflake_savings

Check a Snowflake account for savings with Altimate's rules. It lists warehouses with `SHOW WAREHOUSES` and, to report on seven days, reads up to eleven days of `SNOWFLAKE.ACCOUNT_USAGE` (`WAREHOUSE_METERING_HISTORY`, `WAREHOUSE_EVENTS_HISTORY`, `QUERY_HISTORY`), read-only, on the connection's warehouse. The role needs `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database, and sees only the warehouses it has access to; warehouses that billed credits but cannot be listed are named as not checked.
Comment thread
shreyastelkar marked this conversation as resolved.
Resolved
Comment thread
shreyastelkar marked this conversation as resolved.
Resolved

- **High Idle Time:** on days a warehouse sat idle for more than 40% of its active time, the idle share of that day's cost.
- **Downsize Warehouse:** nearly all queries finish within 5 seconds, or nearly all active periods last 80 seconds or less, so one size smaller would do.
- **Underutilized Cluster:** the highest cluster of a multi-cluster warehouse is barely needed. It is reported without a dollar estimate.
- **Frequent Restart Overcharge:** a warehouse that suspends and resumes within one billed minute pays the overlap twice.

The tool measures each warehouse's usage; altimate-core decides what qualifies and what it saves. Credits are priced from `ORGANIZATION_USAGE.RATE_SHEET_DAILY` when the role can read it in dollars, otherwise at $3.00 (also for an account billed in another currency), and the output says which. Each warehouse is counted once in the total.

**What it does not assess.** These are four of Altimate's warehouse checks. Storage, serverless features and individual query tuning are not covered. A warehouse that never suspended in the window has no completed active period, so it gets no idle finding. Days are UTC, and the window ends at 00:00 UTC on the latest day at least six hours past, because `ACCOUNT_USAGE` lags by up to that long.

**Asks first, and stops long queries.** The check runs only when called with `confirm: true`; without it, the agent is told to ask you first, since a run can resume a suspended warehouse that then bills. It runs six queries and stops after 9 minutes, connecting included, cancelling any query still running on Snowflake (or sooner, at an `altimate run --time-budget` cutoff); stopping the session cancels the running query too.
Comment thread
shreyastelkar marked this conversation as resolved.
Resolved

**Assessment mode.** `ALTIMATE_ASSESSMENT_MODE=true` works as it does for `finops_databricks_savings`: the run contacts your Snowflake account, and turns off the Snowflake driver's cloud-platform detection. If a result is large enough for Snowflake to deliver it from its own cloud storage, the driver fetches it from there too.

```
> finops_snowflake_savings --warehouse prod_sf --confirm

Snowflake savings: 12 warehouses checked, the 7 days to 2026-11-18, at $3.00 a credit from the account's rate sheet.
Total: $54.00–$90.00 a month on 1 warehouse, each counted once at its largest saving.

BI_WH (warehouse) · Downsize Warehouse · $54.00–$90.00 a month (medium)
Large could be Medium: 95.0% of its active periods lasted 80 seconds or less
The saving is a share of $42.00 over the window
```

---

## finops_role_grants
Expand Down
112 changes: 102 additions & 10 deletions packages/drivers/src/snowflake.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
import * as fs from "fs"
import type { ConnectionConfig, Connector, ConnectorResult, ExecuteOptions, SchemaColumn } from "./types"
import { toPkcs8Pem } from "./snowflake-key"
import { QueryTimeoutError } from "./types"
import { MAX_TIMER_MS } from "./query-timeout"
import { loadOptionalDriver } from "./resolve"
import { emitBrowserSignIn, openInBrowser } from "./sign-in"
import { emitReconnect, type ReconnectEvent } from "./reconnect-events"
Expand Down Expand Up @@ -54,6 +56,26 @@ export function suppressSnowflakeLogging(snowflake: any): void {
})
}

/**
* In assessment mode nothing but the warehouse may be contacted. When it loads, snowflake-sdk otherwise probes
* the AWS, Azure and GCP metadata and identity services to report the platform it runs on. Its certificate
* revocation requests need no switch here: they start from an HTTPS agent that Bun, which runs the packaged
* CLI, does not use.
*/
export function keepSdkOffCloudMetadata(env: NodeJS.ProcessEnv = process.env): void {
const mode = env.ALTIMATE_ASSESSMENT_MODE?.trim().toLowerCase()
if (mode && mode !== "0" && mode !== "false") env.SNOWFLAKE_DISABLE_PLATFORM_DETECTION = "true"
}

/** How long to wait for Snowflake to acknowledge a cancel before reporting the statement as possibly still running. */
const CANCEL_WAIT_MS = 30_000

/**
* How a caller's `cancelAfterMs` reaches what it covers: why it was stopped, and every statement sent. A read
* retried after a closed-session error leaves two, and the first may still be running.
*/
type StatementStop = { stopped?: "deadline" | "aborted"; statements: any[] }

/**
* Errors meaning the session is gone: 407002 is raised client-side for any
* request on a terminated connection; 390111 (session no longer exists),
Expand Down Expand Up @@ -183,10 +205,12 @@ function describeAccount(account: unknown): string {
export async function connect(
config: ConnectionConfig,
sdk?: unknown,
seams?: { connectTimeoutMs?: number },
seams?: { connectTimeoutMs?: number; cancelWaitMs?: number },
): Promise<Connector> {
const connectTimeoutMs = seams?.connectTimeoutMs ?? CONNECT_TIMEOUT_MS
const cancelWaitMs = seams?.cancelWaitMs ?? CANCEL_WAIT_MS
let snowflake: any
keepSdkOffCloudMetadata()
snowflake = sdk ?? (await loadOptionalDriver("snowflake", "snowflake-sdk"))
snowflake = snowflake.default || snowflake

Expand Down Expand Up @@ -468,7 +492,11 @@ export async function connect(
* reopened and a read-only statement is retried once; a write is not, because
* it may already have run (see `isRetrySafe`).
*/
async function executeQuery(sql: string, binds?: any[]): Promise<{ columns: string[]; rows: any[][] }> {
async function executeQuery(
sql: string,
binds?: any[],
stop?: StatementStop,
): Promise<{ columns: string[]; rows: any[][] }> {
// Captured before any wait: a statement queued behind a reconnect started on the old session's assumptions.
const started = generation
const startedInEpoch = epoch
Expand All @@ -477,7 +505,7 @@ export async function connect(
const used = connection
const ranIn = generation
try {
const result = await runTracked(used, sql, binds, ranIn)
const result = await runTracked(used, sql, binds, ranIn, stop)
Comment thread
shreyastelkar marked this conversation as resolved.
Resolved
await noteSession(sql, binds, ranIn, startedInEpoch)
return result
} catch (err) {
Expand All @@ -496,7 +524,7 @@ export async function connect(
)
}
const ranAgainIn = generation
const result = await runTracked(connection, sql, binds, ranAgainIn)
const result = await runTracked(connection, sql, binds, ranAgainIn, stop)
await noteSession(sql, binds, ranAgainIn, startedInEpoch)
return result
}
Expand All @@ -511,20 +539,27 @@ export async function connect(

/** `runQuery`, counting a statement that may hold session state against the session it was sent to (`ran`) for
* as long as it runs. Counted only once sent: one still waiting on a reconnect will run on the new session. */
async function runTracked(conn: any, sql: string, binds: any[] | undefined, ran: number) {
if (!mayHoldState(sql, binds)) return runQuery(conn, sql, binds)
async function runTracked(conn: any, sql: string, binds: any[] | undefined, ran: number, stop?: StatementStop) {
if (!mayHoldState(sql, binds)) return runQuery(conn, sql, binds, stop)
stateStatementsRunning.set(ran, (stateStatementsRunning.get(ran) ?? 0) + 1)
try {
return await runQuery(conn, sql, binds)
return await runQuery(conn, sql, binds, stop)
} finally {
const left = (stateStatementsRunning.get(ran) ?? 1) - 1
if (left > 0) stateStatementsRunning.set(ran, left)
else stateStatementsRunning.delete(ran)
}
}

function runQuery(conn: any, sql: string, binds?: any[]): Promise<{ columns: string[]; rows: any[][] }> {
function runQuery(
conn: any,
sql: string,
binds?: any[],
stop?: StatementStop,
): Promise<{ columns: string[]; rows: any[][] }> {
return new Promise((resolve, reject) => {
// Stopped while waiting on a reconnect: the statement is never sent.
if (stop?.stopped) return reject(new Error("Snowflake statement stopped before it was sent"))
const options: Record<string, any> = {
sqlText: sql,
complete(err: Error | null, _stmt: any, rows: any[]) {
Expand All @@ -541,10 +576,63 @@ export async function connect(
},
}
if (binds && binds.length > 0) options.binds = binds
conn.execute(options)
const statement = conn.execute(options)
stop?.statements.push(statement)
})
}

function cancelStatement(statement: any): Promise<boolean> {
return new Promise((resolve) => {
const timer = setTimeout(() => resolve(false), cancelWaitMs)
const finish = (cancelled: boolean) => {
clearTimeout(timer)
resolve(cancelled)
}
try {
statement.cancel((err: Error | null | undefined) => finish(!err))
} catch {
finish(false)
}
})
}

/**
* `cancelAfterMs` is an opt-in deadline: past it, or when `signal` aborts, the statement is cancelled on
* Snowflake. One still waiting on a reconnect is never sent, so nothing is left running.
*/
async function executeWithCancel(
sql: string,
binds: any[] | undefined,
deadlineMs: number,
signal?: AbortSignal,
): Promise<{ columns: string[]; rows: any[][] }> {
signal?.throwIfAborted()
const stop: StatementStop = { statements: [] }
const STOPPED = Symbol("stopped")
let wake: () => void = () => {}
const stopping = new Promise<typeof STOPPED>((resolve) => (wake = () => resolve(STOPPED)))
const halt = (why: "deadline" | "aborted") => {
stop.stopped ??= why
wake()
}
const timer = setTimeout(() => halt("deadline"), Math.min(deadlineMs, MAX_TIMER_MS))
const onAbort = () => halt("aborted")
signal?.addEventListener("abort", onAbort, { once: true })
const work = executeQuery(sql, binds, stop)
let outcome: { columns: string[]; rows: any[][] } | typeof STOPPED
try {
outcome = await Promise.race([work, stopping])
} finally {
clearTimeout(timer)
signal?.removeEventListener("abort", onAbort)
}
if (outcome !== STOPPED) return outcome
work.catch(() => {})
const cancelled = (await Promise.all(stop.statements.map(cancelStatement))).every(Boolean)
if (stop.stopped === "aborted") throw signal!.reason
throw new QueryTimeoutError({ engine: "snowflake", timeoutMs: deadlineMs, cancelled })
}

return {
async connect() {
closed = false
Expand Down Expand Up @@ -710,7 +798,11 @@ export async function connect(
query = `${sql.replace(/;\s*$/, "")} LIMIT ${effectiveLimit + 1}`
}

const result = await executeQuery(query, binds)
const deadlineMs = options?.cancelAfterMs
Comment thread
shreyastelkar marked this conversation as resolved.
Resolved
const result =
deadlineMs && deadlineMs > 0
? await executeWithCancel(query, binds, deadlineMs, options?.signal)
: await executeQuery(query, binds)
const truncated = effectiveLimit > 0 && result.rows.length > effectiveLimit
const rows = truncated
? result.rows.slice(0, effectiveLimit)
Expand Down
2 changes: 1 addition & 1 deletion packages/drivers/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ export interface ExecuteOptions {
queryTimeoutMs?: number
/** Cancel the statement on the server once this many milliseconds pass, or when
* `signal` aborts. Opt-in: `executeQuery` fills in `queryTimeoutMs` for every
* call, so only callers that set this get the server-side cancel. Databricks only. */
* call, so only callers that set this get the server-side cancel. Databricks and Snowflake. */
cancelAfterMs?: number
/** DuckDB only: called once after handing this query to its helper; callback errors do not affect execution. */
onDispatch?: () => void
Expand Down
Loading
Loading