Feat/focus ledger schema - #33
Merged
Merged
Conversation
The DuckDB file was a cache, not a ledger: two per-account tables of display-shaped JSON, plus a cost_data table nothing wrote. Sankey, attribution, anomaly detection and month-end freezing all need a fact table to build on, so this adds one. billing.duckdb is a separate file from the application state, because accounts and caches are user-entered or re-fetchable while the ledger is the record that has to survive. It holds fct_charge, ingest_batch, fct_balance_snapshot and dim_fx_rate, named after FOCUS columns so a later ingest of a real CUR or bill export needs no schema change. Writes are whole-period replacement in one transaction: providers re-issue a bill in full mid-month and correct prior months, so a row-by-row upsert would leave behind rows the provider has since deleted and the total would stop matching theirs. Charge ids are a hash of the row's natural key, which is what makes "run ingest twice, get identical results" checkable; rows whose natural keys collide get an occurrence suffix rather than being folded together, so no money goes missing. Nothing normalizes into the ledger yet — PR4 and PR5 do — so the cache tables stay for now. Dropping them before then would mean paying Cost Explorer for a fetch on every launch. The application database is versioned and rebuilt at v1 along the way: cost_data is gone, provider is now source_id, and the credential columns are dropped after any secret still in them is moved to the OS keyring. It is a rebuild rather than a sequence of ALTERs because DuckDB will not alter a table a foreign key points at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fetching and interpreting were one step: get_cost_summary() called Cost Explorer and returned a display-shaped struct, so the response existed only long enough to be reshaped. Cost Explorer bills per request, which made every mapping change cost another round of paid calls, and there was no way to assert on billing logic without a live account. fetch now returns what the provider sent, unchanged, and nothing else. The payloads are written as Hive-partitioned Parquet under raw/provider=/account=/billing_period=/batch=, the layout a bill export bucket already uses, so P1's S3/OSS channel replaces only the fetch half. normalize is a pure function from a stored batch to FOCUS rows: no clock, no network, no database — the fetch time it needs rides along on the batch. Each source is now tested against a recorded response. The two Cost Explorer methods were near-identical copies differing in one field of the request body; they collapse into one signed call plus a request builder. What the normalizers do not do yet is the mapping detail PR4 and PR5 own. AWS asks only for UnblendedCost and files every row as Usage, because without RECORD_TYPE in the grouping the payload cannot tell a credit from a charge and guessing would put refunds on the wrong side of the total. Alibaba Cloud keeps the discount visible as the gap between billed_cost and list_cost rather than inventing Credit rows. DeepSeek writes balance snapshots and derives no top-ups. ingest_period ties the three steps to one batch id, shared by the raw partition and the ledger batch; renormalize_period replays the newest stored batch without fetching, which is the whole reason the raw store exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cost Explorer was asked for UnblendedCost grouped by service, which is one unlabelled amount per service per day. A credit, a refund, a tax line and a support fee all arrived looking like usage, and there was no amortized figure at all, so a reserved instance paid for up front showed as a spike in the month it was bought and as free capacity for a year afterwards. The ingest request now carries UnblendedCost, AmortizedCost and UsageQuantity in one call — Cost Explorer bills per request, not per metric — and groups by RECORD_TYPE as well as SERVICE. The record type maps to charge_category, so the ledger can tell a credit from a charge. Amounts keep the sign Cost Explorer gives them, so a period total is a plain sum. An unrecognized record type becomes an Adjustment with a warning naming it: money moved, and filing it as Usage would quietly inflate what reads as consumption. A row is dropped only when both cost metrics are zero. Usage covered by a savings plan is zero unblended and non-zero amortized; dropping it would lose what the commitment bought. Grouping by service also mixes usage types, which Cost Explorer signals by returning the unit N/A — a quantity like that is not stored, because it cannot be added to anything. Which key is which is read from the response's GroupDefinitions rather than assumed from the request this build would have sent, so payloads already in the raw store still normalize; they normalize as Usage, which is what they always were. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Alibaba Cloud reports gross and net on the same line, so a voucher was only visible as the difference between two numbers. Each deduction — InvoiceDiscount, coupons, cash coupons, a stored-value card — is now a Credit row of its own beside a usage charge carrying the gross amount. Putting the net amount on the usage row and the deductions beside it would have counted them twice; AWS bills a discount as a separate line, Alibaba Cloud does not. Decomposed this way a product's rows sum to PretaxAmount, which is what was actually charged, so a period total stays a plain sum. When the deductions we can name do not close the gap between gross and net, the remainder becomes one Adjustment row and a warning naming the product: the bill accounts for that money even where this parser cannot, and dropping it would leave the ledger disagreeing with the invoice. DeepSeek publishes a balance, which is state. It says what is left, never what was bought, so a purchase can only be inferred from movement: a rise in the topped-up balance between two consecutive observations is money that went in. Those rows are derived at ingest rather than written once, because replacing a period clears whatever was there before — recomputing them keeps a re-ingest idempotent. The first observation of an account yields nothing: a balance that was simply there the first time it was looked at was not witnessed being paid for, and inventing a purchase for it would drop the whole opening balance into whichever month the account happened to be added. Balances that differ only in currency never derive a purchase from each other, and a falling topped-up balance is consumption, not a top-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard added AWS dollars to Alibaba Cloud yuan and printed the result with a dollar sign. It also asked each provider for a display-shaped summary and kept a copy in two cache tables, so the numbers on screen were whatever the last API call happened to return. Charges are now read through v_charge_normalized, which converts each one at a rate dated no later than the charge itself. Conversion lives in a view rather than at write time because rates get corrected and the reporting currency is the user's to change; either would otherwise mean rewriting the fact table. Switching currency in Settings replaces a view and rewrites nothing. A charge already in the reporting currency converts at 1.0 without needing a row in the rate table. One whose currency no rate covers keeps a NULL converted amount: it is counted nowhere and reported on the dashboard as an explicit "not included" line, because a total that quietly folds in an unconverted amount at par is worse than one that admits what it is missing. The cache tables go with it (application schema v2). ingest_batch already records when each period was written, so that is what the freshness window checks now. The trend chart reads the daily rows the refresh already stored instead of paying for a third Cost Explorer call, and a source's trend window becomes how far back its own rows are worth charting — Alibaba Cloud's monthly rows get two billing periods rather than seven empty days. With nothing left reading them, get_cost_summary, get_cost_trend and get_cost_data are gone from the trait and from all three sources, along with the parsers and aggregators behind them. A source fetches and normalizes; what the numbers mean afterwards is the ledger's business. The clients also stop carrying an account id and name they no longer use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.