From 051d0582940c5f9fd12f05bc0f3fdd69ff280677 Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Tue, 6 Oct 2026 10:22:40 +0530 Subject: [PATCH] Document the billing adapters and align the docs with current terminology The docs site gets a Billing Adapters section with full guides for Stripe, Metronome, Lago and a custom billing adapter, and Getting Started no longer assumes Lago: choose a billing platform, install, configure pricing and packaging, verify. Wording follows how vCluster Labs talks about the product: tenant cluster, control plane cluster, AI Cloud, billing platform and billing adapter, node capacity tracking, and "reserved" only as a capacity type. Also fixed: the AI Clouds section layout on the landing page, the Metronome pricing example (pricing and presentation keys must form the metric's group key), v0.1 log lines quoted in troubleshooting, a note that billing states need a Stripe or Metronome webhook secret, the admission policy's reach for tenant clusters with their own nodes, horizontal scrolling at phone width, and gitignored key file names. --- .gitignore | 8 + README.md | 38 +- deploy/helm/vbilling/templates/NOTES.txt | 4 +- deploy/helm/vbilling/values.yaml | 4 +- docs/configuration.html | 39 +- docs/custom-adapter.html | 613 +++++++++++++++++++++++ docs/example-gke.html | 15 +- docs/getting-started.html | 392 ++++----------- docs/index.html | 60 +-- docs/lago.html | 425 ++++++++++++++++ docs/metrics.html | 25 +- docs/metronome.html | 519 +++++++++++++++++++ docs/private-nodes.html | 21 +- docs/stripe.html | 588 ++++++++++++++++++++++ docs/troubleshooting.html | 49 +- examples/gke-gpu/01-prometheus.yaml | 2 +- scripts/demo.sh | 24 +- 17 files changed, 2426 insertions(+), 400 deletions(-) create mode 100644 docs/custom-adapter.html create mode 100644 docs/lago.html create mode 100644 docs/metronome.html create mode 100644 docs/stripe.html diff --git a/.gitignore b/.gitignore index 573638f..014f4a5 100644 --- a/.gitignore +++ b/.gitignore @@ -12,3 +12,11 @@ deploy/lago/lago_rsa* deploy/lago/.env deploy/lago/entrypoint.sh .claude/ + +# Key files the docs tell you to create next to the chart +stripe-*-key +stripe-key +stripe-webhook-secret +metronome-token +lago-api-key +webhook-secret diff --git a/README.md b/README.md index 3c4c1f4..250ef14 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ vBilling is the **metering pipe**, not the billing engine. It turns what runs in ## What's new in v0.2 - **Stripe and Metronome adapters.** Use Stripe Billing Meters on their own, or Metronome for rating with Stripe for payment, invoicing and tax. vBilling creates the Stripe customer and links it in Metronome. -- **Durable, hash-chained ledger.** Every window is committed atomically to disk before any backend sees it. If vBilling restarts or a backend has an outage, no usage is lost, and missed windows are backfilled. +- **Durable, hash-chained ledger.** Every window is committed atomically to disk before any billing platform sees it. If vBilling restarts or a billing platform has an outage, no usage is lost, and missed windows are backfilled. - **Several destinations at once.** Each destination has its own cursor, so a down billing API never blocks your data platform feed. - **Deterministic event IDs.** Retries, replays and two replicas all deduplicate downstream. - **Second-level allocation metering.** GPU time is metered from container start to finish, including fractional GPUs (MIG mixed and single strategy, GKE GPU partitions, time-slicing including GKE time-sharing) and GPUs allocated through DRA ResourceClaims. @@ -124,7 +124,7 @@ Every event carries `tenant`, `region`, `project`, `sku`, `resource_id`, `tenant | Tenancy model | Where workloads run | How vBilling meters it | |---|---|---| | Shared nodes | Control plane cluster nodes, pods synced by vCluster | From the control plane cluster: pods, volumes, load balancers (`billing_mode=shared`) | -| Dedicated nodes | Control plane cluster nodes reserved for one tenant cluster | Whole node: node-hours, whole GPUs, CPU and memory capacity (`billing_mode=dedicated_node`) | +| Dedicated nodes | Control plane cluster nodes assigned to one tenant cluster | Whole node: node-hours, whole GPUs, CPU and memory capacity (`billing_mode=dedicated_node`) | | Private Nodes, Auto Nodes | Machines joined to the tenant cluster itself; nothing is synced to the control plane cluster | Through the tenant cluster's own API: each private node billed whole (`billing_mode=private_node`), plus the tenant cluster's volumes and load balancers. `PRIVATE_NODE_BILLING=usage` bills the pods on them instead | | Standalone, or any cluster elsewhere | No control plane cluster | Through a kubeconfig listed in `TENANT_CLUSTERS_FILE`, like private nodes | @@ -165,9 +165,9 @@ vCluster exports kubeconfigs for `localhost`; vBilling connects to the tenant cl **Outages.** If a tenant cluster's API cannot be read, the rest of the fleet is still metered, and the missed windows are filled as soon as it answers (up to `MAX_BACKFILL`). Fills are late ledger events deduplicated by ID, so a repeated fill never bills twice. -## Billing backends +## Billing adapters -Select one or more with `ADAPTERS` (Helm: `adapters`). +Select one or more with `ADAPTERS` (Helm: `adapters`). Each has a full guide on the docs site: [Stripe](https://vclusterlabs-experiments.github.io/vbilling/stripe.html), [Metronome](https://vclusterlabs-experiments.github.io/vbilling/metronome.html), [Lago](https://vclusterlabs-experiments.github.io/vbilling/lago.html) and the [Custom Billing Adapter](https://vclusterlabs-experiments.github.io/vbilling/custom-adapter.html) (webhook, ingest API, or your own Go adapter). | Adapter | What vBilling does | Pricing lives in | |---|---|---| @@ -175,10 +175,10 @@ Select one or more with `ADAPTERS` (Helm: `adapters`). | `metronome` | Creates customers with the tenant ID as ingest alias. Creates SUM billable metrics with one compound group key (`region, sku, capacity_type, billing_mode, tenant_cluster`). Ingests events and optionally creates a contract per tenant from a rate card. Reconciles via `/v1/usage`. | Metronome rate cards (pricing group keys) | | `metronome` + `METRONOME_STRIPE_LINK=true` | Also creates the tenant's Stripe customer and links it in Metronome, so **Metronome rates and Stripe collects payment, invoices and tax**. | Metronome | | `lago` | Customers, subscriptions and the default plan, plus events whose dimensions are properties for Lago charge filters. | Lago plans and charges | -| `webhook` | Signed CloudEvents 1.0 batches (`application/cloudevents-batch+json`), including tenant and catalog events. Use it to feed a data platform, Kafka bridge or homegrown rating engine. | wherever you send it | +| `webhook` | Signed CloudEvents 1.0 batches (`application/cloudevents-batch+json`), including tenant, offboarding and catalog events. Use it to feed a data platform, Kafka bridge or homegrown rating engine. | wherever you send it | | `noop` | Logs events (dry run). | n/a | -Stripe now steers new usage-based integrations toward Metronome, which Stripe acquired in January 2026. Both paths are supported here: Billing Meters for straightforward pay-as-you-go, Metronome for dimensional pricing, commits, credits and multi-currency rate cards. +Stripe acquired Metronome in January 2026. Both paths are supported here: Billing Meters for straightforward pay-as-you-go, Metronome for dimensional pricing, commits, credits and multi-currency rate cards. ## Quick start @@ -214,11 +214,11 @@ helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \ --set region=ap-southeast-2 ``` -In Metronome, add products on the vBilling billable metrics with `pricing_group_key: ["region","sku","capacity_type"]` to price H100 differently from L40S, Sydney differently from Melbourne, and spot or preemptible at a flat discount. Rate cards carry the currency (AUD and NZD are supported natively). +In Metronome, add products on the vBilling billable metrics with `pricing_group_key: ["region","sku","capacity_type","billing_mode"]` and `presentation_group_key: ["tenant_cluster"]` (together they must form the metric's group key) to price H100 differently from L40S, one region differently from another, and spot or preemptible at a flat discount. Rate cards carry the currency. **Shared and private GPU nodes on GKE**: [`examples/gke-gpu`](examples/gke-gpu) is a complete setup, from an empty project to Stripe invoices: a time-sliced L4 tenant cluster, an A100 joined as a vCluster private node, DCGM through Prometheus, and the manifests, values and commands for each step. Also on the [docs site](https://vclusterlabs-experiments.github.io/vbilling/example-gke.html). -**Lago (local demo)**: `deploy/lago/docker-compose.yml` runs Lago locally. See [docs/getting-started.html](docs/getting-started.html). +**Lago (local demo)**: `deploy/lago/docker-compose.yml` runs Lago locally. See the [Lago guide](https://vclusterlabs-experiments.github.io/vbilling/lago.html), which also runs the whole pipeline on a laptop with vind. Then: @@ -238,7 +238,7 @@ By default every tenant cluster is its own billing customer, with ID `vcluster-< | Key (`vbilling.vcluster.com/...`) | Effect | |---|---| | `tenant` | Billing customer ID. Several tenant clusters can share one customer. | -| `display-name`, `email`, `currency` | Customer details sent to the backend | +| `display-name`, `email`, `currency` | Customer details sent to the billing platform | | `project` | Cost attribution (defaults to the vCluster Platform project) | | `tenant-class` | e.g. `public`, `enterprise`, `government`, `dev`. Stamped on events for pricing. | | `plan` | Lago plan, Stripe plan tag, or Metronome rate card alias/ID | @@ -253,16 +253,16 @@ On nodes, `vbilling.vcluster.com/capacity-type` (`on-demand`, `spot`, `preemptib | Guarantee | How | |---|---| -| No usage lost on restart, upgrade or backend outage | Windows are committed to an fsynced, append-only ledger on a PVC before delivery. Each destination has a persisted cursor. On restart, missed windows (up to `MAX_BACKFILL`, default 6h) are backfilled from object lifetimes. | +| No usage lost on restart, upgrade or billing platform outage | Windows are committed to an fsynced, append-only ledger on a PVC before delivery. Each destination has a persisted cursor. On restart, missed windows (up to `MAX_BACKFILL`, default 6h) are backfilled from object lifetimes. | | No duplicates | Event IDs are a hash of the event's identity (tenant, metric, window, dimensions), never of its quantity. Stripe dedupes on `identifier`, Metronome on `transaction_id` (34 days), Lago on `transaction_id`. A window can never be committed twice. | | Half-written windows never escape | A window is a single write plus fsync ending in a commit record. A torn or uncommitted tail is truncated on startup and collected again. | | Tamper evidence | Every ledger record is SHA-256 chained to the previous one. `GET /api/v1/ledger/verify` re-hashes the whole retained ledger. | -| Bad events don't block good ones | Payloads a backend rejects are isolated (partial-batch errors, or bisection) and parked as dead letters. Auth errors, rate limits and outages are retried with backoff; they are never dead-lettered. | +| Bad events don't block good ones | Payloads a billing platform rejects are isolated (partial-batch errors, or bisection) and parked as dead letters. Auth errors, rate limits and outages are retried with backoff; they are never dead-lettered. | | Provable totals | `GET /api/v1/reconcile` recomputes each tenant's usage from the ledger and compares it with what Stripe or Metronome recorded. Dead letters are accounted for. | | Tenant API outages do not lose usage | A tenant cluster whose own API is unreachable is recorded as a gap (persisted under `DATA_DIR`). Once it answers, exactly the missed windows are metered and appended as late events, deduplicated by ID. | | A wrong customer is never billed | Customer search results must carry the tenant's exact metadata. Discovery errors never offboard tenants. A tenant must be gone for `OFFBOARD_GRACE` before `RemoveTenant` runs, and customers are never deleted. | -Running two replicas (each with its own PVC) is safe for delivery: both compute the same deterministic IDs, so backends keep the first copy. +Running two replicas (each with its own PVC) is safe for delivery: both compute the same deterministic IDs, so billing platforms keep the first copy. ## Regions and data sovereignty @@ -278,19 +278,19 @@ Point Stripe webhooks at `/webhooks/stripe` (`STRIPE_WEBHOOK_SECRET`) and Metron | Event | State | |---|---| -| `invoice.payment_failed`, subscription `past_due` | `delinquent` | +| `invoice.payment_failed`, subscription `past_due`, Metronome `payment_gate.payment_status` failed | `delinquent` | | subscription `unpaid` / `canceled` / deleted | `suspended` | | `billing.alert.triggered`, Metronome `alerts.*` (spend, usage, low credit or commit balance) | `warning` | -| `invoice.paid`, subscription `active` | `active` | +| `invoice.paid`, subscription `active`, Metronome `payment_gate.payment_status` paid | `active` | Override any mapping with `ENFORCEMENT_RULES`, for example `alerts.spend_threshold_reached=suspended` for a hard spend cap. `ENFORCEMENT_MODE` controls what happens: - **`observe`** (default): record the state only. It appears in the API, metrics and dashboard. - **`annotate`**: also annotate the tenant cluster, label its namespace `vbilling.vcluster.com/billing-state`, and emit a Kubernetes event. -- **`enforce`**: also label suspended namespaces `vbilling.vcluster.com/suspended=true`. With `enforcement.admissionPolicy.enabled=true` (Kubernetes 1.30+), a ValidatingAdmissionPolicy then denies **new** pods in those namespaces. Running workloads keep running, the tenant sees the reason as a sync error inside their tenant cluster, and a paid invoice lifts it automatically. +- **`enforce`**: also label suspended namespaces `vbilling.vcluster.com/suspended=true`. With `enforcement.admissionPolicy.enabled=true` (Kubernetes 1.30+), a ValidatingAdmissionPolicy then denies **new** pods in those namespaces. Running workloads keep running, the tenant sees the reason as a sync error inside their tenant cluster, and a paid invoice lifts it automatically. The policy acts in the control plane cluster, so it holds tenant clusters on shared or dedicated nodes; tenant clusters with their own nodes (Private Nodes, Auto Nodes, Standalone) run their pods only inside the tenant cluster, so act on their billing state with your own automation. -Metronome does not emit webhooks for Stripe payment failures, so in Metronome + Stripe mode, send Stripe's webhooks to vBilling as well. +Payment failures on invoices that Stripe collects arrive through Stripe's webhooks, so in Metronome + Stripe mode, send Stripe's webhooks to vBilling as well. Billing states are enabled once `STRIPE_WEBHOOK_SECRET` or `METRONOME_WEBHOOK_SECRET` is set. ## Custom usage: tokens, Slurm, storage, network @@ -303,7 +303,7 @@ curl -X POST localhost:8080/api/v1/events -H "Authorization: Bearer $INGEST_TOKE "dimensions": {"model": "llama-3.3-70b"}}]}' ``` -Events are validated, stamped with the region, deduplicated (send your own `id`, or let vBilling derive one from the identity), committed to the ledger, and fanned out like collector events. Billing backends create meters and metrics for custom metrics at bootstrap. +Events are validated, stamped with the region, deduplicated (send your own `id`, or let vBilling derive one from the identity), committed to the ledger, and fanned out like collector events. The billing adapters create meters and metrics for custom metrics at bootstrap. ### Metrics from Prometheus @@ -349,7 +349,7 @@ vCluster Platform 4.11+ collects metrics from every tenant cluster, private node | `GET /api/v1/usage?from&to&group_by=tenant,sku,region,day&format=csv` | Unrated line items (filters: `tenant`, `metric`, `project`, `region`, `sku`, `cluster`) | | `GET /api/v1/events?from&to&format=jsonl\|csv` | Raw event export | | `POST /api/v1/events` | Ingest custom usage (`INGEST_TOKEN`) | -| `GET /api/v1/reconcile?from&to&destination` | Ledger vs. backend totals (default: previous UTC day) | +| `GET /api/v1/reconcile?from&to&destination` | Ledger vs. billing platform totals (default: previous UTC day) | | `GET /api/v1/ledger/verify` | Hash-chain verification | | `GET /api/v1/destinations`, `GET …/{name}/dead-letters`, `POST …/{name}/dead-letters/replay`, `POST …/{name}/cursor` | Delivery operations | | `GET /api/v1/tenants`, `GET /api/v1/billing-states`, `PUT /api/v1/billing-states/{tenant}` | Tenants and billing state | @@ -403,7 +403,7 @@ Alert on rising lag, dead letters, or a stale last window. ## Upgrading from v0.1 - **Workload:** the chart now deploys a **StatefulSet** with a PVC (the ledger). The unused `pricing.*` values were removed. -- **Discounts:** quantities are no longer discounted for spot nodes (`SPOT_DISCOUNT_PERCENT` is ignored). Price `capacity_type=spot|preemptible` in your backend instead, so metered quantities stay physical and reconcilable. +- **Discounts:** quantities are no longer discounted for spot nodes (`SPOT_DISCOUNT_PERCENT` is ignored). Price `capacity_type=spot|preemptible` in your billing platform instead, so metered quantities stay physical and reconcilable. - **Double counting fixed:** v0.1 billed GPUs and CPU/memory on dedicated nodes twice (pod usage plus node capacity). It also over-counted egress about 5x with 60s windows. - **Timestamps:** events use the window start, and identical windows now produce identical IDs. - **Dashboard:** it is served by vBilling itself and reads its API (the Lago-only `dashboard/index.html` remains for reference). diff --git a/deploy/helm/vbilling/templates/NOTES.txt b/deploy/helm/vbilling/templates/NOTES.txt index c2bfa3f..66f032f 100644 --- a/deploy/helm/vbilling/templates/NOTES.txt +++ b/deploy/helm/vbilling/templates/NOTES.txt @@ -4,12 +4,12 @@ Destinations: {{ include "vbilling.adapters" . }} kubectl -n {{ .Release.Namespace }} port-forward svc/{{ include "vbilling.fullname" . }} {{ .Values.api.service.port }} curl -s localhost:{{ .Values.api.service.port }}/api/v1/status # destinations, lag, last window curl -s localhost:{{ .Values.api.service.port }}/api/v1/usage?group_by=tenant,sku - curl -s localhost:{{ .Values.api.service.port }}/api/v1/reconcile # ledger vs billing backend, previous UTC day + curl -s localhost:{{ .Values.api.service.port }}/api/v1/reconcile # ledger vs billing platform, previous UTC day open http://localhost:{{ .Values.api.service.port }}/ # usage dashboard {{- if not .Values.persistence.enabled }} WARNING: persistence is disabled. The ledger is the durable outbox for usage that -has not reached your billing backend yet; without a volume it is lost on restart. +has not reached your billing platform yet; without a volume it is lost on restart. {{- end }} {{- if not .Values.api.token }} diff --git a/deploy/helm/vbilling/values.yaml b/deploy/helm/vbilling/values.yaml index b30c5a0..70ee7d7 100644 --- a/deploy/helm/vbilling/values.yaml +++ b/deploy/helm/vbilling/values.yaml @@ -9,8 +9,8 @@ image: pullPolicy: IfNotPresent # Destinations usage fans out to. Built-in: stripe, metronome, lago, webhook, noop. -# Each destination has its own cursor into the ledger, so one backend being -# down never blocks another. +# Each destination has its own cursor into the ledger, so one billing platform +# being down never blocks another. adapters: - lago # Deprecated single-adapter form from v0.1; used only when `adapters` is empty. diff --git a/docs/configuration.html b/docs/configuration.html index 00d3341..03bb2d6 100644 --- a/docs/configuration.html +++ b/docs/configuration.html @@ -4,7 +4,7 @@ Configuration - vBilling Docs - + @@ -96,7 +96,7 @@ .sidebar-back:hover { border-color: var(--primary); color: var(--primary-dark); } .content { - flex: 1; margin-left: var(--sidebar-width); + flex: 1; min-width: 0; margin-left: var(--sidebar-width); padding: 48px 64px 80px; max-width: 900px; } .content h1 { @@ -125,7 +125,7 @@ .content code { background: var(--primary-light-2); padding: 2px 7px; border-radius: 4px; font-family: 'Roboto Mono', 'SF Mono', 'Fira Code', monospace; - font-size: 0.85rem; color: var(--primary-darker); + font-size: 0.85rem; color: var(--primary-darker); overflow-wrap: anywhere; } .content a { color: var(--primary-dark); text-decoration: none; font-weight: 500; } .content a:hover { text-decoration: underline; } @@ -232,17 +232,26 @@ Example: GKE with GPUs Troubleshooting +
+
Billing Adapters
+ Stripe + Metronome + Lago + Custom Billing Adapter +
Resources
GitHub Repository - Lago Docs vCluster Docs + Stripe Billing Docs + Metronome Docs + Lago Docs

Configuration

-

Every setting for vBilling v0.2: environment variables, Helm chart values, and pricing setup in your billing backend.

+

Every setting for vBilling v0.2: environment variables, Helm chart values, and pricing setup in your billing platform.

Environment Variables

@@ -436,7 +445,7 @@

Metering

PROMETHEUS_METRICS_FILE - Metrics defined by PromQL, evaluated per tenant cluster at every window end (billable ones are created in the billing backends). + Metrics defined by PromQL, evaluated per tenant cluster at every window end (billable ones are created in the billing platforms). (none) @@ -498,7 +507,8 @@

API and enforcement

-

Stripe

+

Stripe

+

Setup, meters, prices and webhooks are in the Stripe guide.

@@ -548,7 +558,8 @@

Stripe

-

Metronome

+

Metronome

+

Setup, rate cards, the Stripe link and webhooks are in the Metronome guide.

@@ -588,7 +599,8 @@

Metronome

-

Lago

+

Lago

+

Running Lago and pricing with charge filters are in the Lago guide.

@@ -618,7 +630,8 @@

Lago

-

Webhook

+

Webhook

+

Event types, signature verification and retries are in the Custom Billing Adapter guide.

@@ -652,7 +665,7 @@

Deprecated

- + @@ -868,7 +881,7 @@

Charges to configure

Example: AI Cloud pricing

-

For a GPU-focused AI Cloud selling compute to AI teams, you might set higher GPU prices and include a premium for dedicated infrastructure:

+

For an AI Cloud selling GPU compute to AI teams, you might set higher GPU prices and include a premium for dedicated infrastructure:

# AI Cloud pricing example CPU Core-Hours: $0.08 # higher due to bare-metal premium Memory GB-Hours: $0.015 # ECC memory cost @@ -901,7 +914,7 @@

Example: Internal platform team pricing

Stripe and Metronome Pricing

Stripe: vBilling creates a Billing Meter per metric and, because Stripe meters cannot price on dimensions, a meter per SKU as new SKUs appear (for example vcluster_gpu_hours__nvidia_h100_80gb_hbm3). Create a metered price on each meter you charge for. To subscribe new tenants automatically, tag the prices metadata[vbilling_plan]=vcluster-standard (or the tenant's plan annotation) and enable stripe.autoSubscribe.

-

Metronome: vBilling creates SUM billable metrics named <Metric> [<code>], each with one compound group key: region, sku, capacity_type, billing_mode, tenant_cluster for GPU hours. On your rate card, add a usage product per metric with pricing_group_key set to the dimensions you price on (for example [region, sku, capacity_type]) and presentation_group_key: [tenant_cluster] for invoice lines per tenant cluster. Rate cards carry the currency, so use one rate card per currency (for example AUD and NZD). Point metronome.rateCard or a tenant's plan annotation at it.

+

Metronome: vBilling creates SUM billable metrics named <Metric> [<code>], each with one compound group key: region, sku, capacity_type, billing_mode, tenant_cluster for GPU hours. On your rate card, add a usage product per metric with pricing_group_key set to the dimensions you price on and presentation_group_key: [tenant_cluster] for invoice lines per tenant cluster; together the two must form the metric's group key (for GPU hours, pricing_group_key: [region, sku, capacity_type, billing_mode]). Rate cards carry the currency, so use one rate card per currency. Point metronome.rateCard or a tenant's plan annotation at it.

diff --git a/docs/custom-adapter.html b/docs/custom-adapter.html new file mode 100644 index 0000000..6800698 --- /dev/null +++ b/docs/custom-adapter.html @@ -0,0 +1,613 @@ + + + + + + Custom Billing Adapter - vBilling Docs + + + + + + + + + + +
+ + +
+

Custom Billing Adapter

+

Connect a billing platform vBilling has no built-in adapter for: stream signed CloudEvents to it with the webhook adapter, push usage vBilling cannot measure through the ingest API, or write a billing adapter in Go.

+
+
i
+
Stripe, Metronome and Lago have built-in billing adapters: see Stripe, Metronome and Lago. Every adapter, built-in or custom, has its own cursor into the ledger, so several can run at once and one billing platform being down never blocks another.
+
+
+
SPOT_DISCOUNT_PERCENTIgnored since v0.2. Price capacity_type in your billing backend instead, so quantities stay physical and reconcilable.Ignored since v0.2. Price capacity_type in your billing platform instead, so quantities stay physical and reconcilable. (none)
+ + + + + + +
ApproachCode to writeUse it for
Webhook adapterNone in vBilling, an HTTPS receiver on your sideA data platform, a Kafka bridge or a homegrown rating engine
Ingest APIAn HTTP client in your serviceUsage vBilling cannot see: inference tokens, Slurm accounting, storage
Go adapterA Go package in your fork of vBillingA billing platform with its own API for customers, meters and usage
+
+ +

Webhook adapter

+

The webhook adapter POSTs CloudEvents 1.0 batches in structured mode to one endpoint, up to 500 events per request. Keep the signing secret in a Secret under the key secret, and list webhook next to any other adapter (for example '{metronome,webhook}'):

+
kubectl -n vbilling-system create secret generic vbilling-webhook \ + --from-file=secret=./webhook-secret +helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \ + --set adapters='{webhook}' \ + --set webhook.url=https://billing.example.com/vbilling \ + --set webhook.existingSecret=vbilling-webhook \ + --set region=us-east-1 --set clusterName=us-east-1a
+
+ + + + + + + + +
Helm valueEnvironment variableEffect
webhook.urlWEBHOOK_URLEndpoint that receives the batches. Required when webhook is in adapters.
webhook.existingSecret (key secret)WEBHOOK_SECRETSigns every request with HMAC-SHA256. Without it, requests are not signed.
webhook.headersWEBHOOK_HEADERS (k=v,k=v)Extra request headers. The chart sets them as a plain environment value, not from a Secret.
region, clusterNameREGION, CLUSTER_NAMEForm the CloudEvents source: vbilling/<region>/<clusterName>.
+
+ +

Event types

+
+ + + + + + + + +
typeSentdata
com.vcluster.vbilling.usage.v1For every event committed to the ledger, collector and ingested alike: every metric, including informational (vcluster_gpu_utilization, vcluster_gpu_downtime_hours) and custom onesOne usage event
com.vcluster.vbilling.tenant.v1Before the tenant's first usage event, again when its details change, and after a restartThe tenant: id, display_name, email, currency, region, project, class, plan, clusters, metadata, provider_ids, created_at
com.vcluster.vbilling.tenant.offboarded.v1Once the tenant has had no tenant clusters for OFFBOARD_GRACE (default 1h)The tenant
com.vcluster.vbilling.catalog.v1Each time vBilling starts, before any usageEvery metric definition: code, name, unit, description, billable, group_keys, custom
+
+

A usage event's CloudEvent id is its usage event ID (vb1_...); the others are tenant-<tenant>-<unix time>, offboard-<tenant>-<unix time> and catalog-<unix time>. Each envelope also carries specversion 1.0, source, subject (the tenant ID; the catalog has none), time (a usage event's window_end), datacontenttype application/json and, on usage events, dataschema. A usage batch, formatted and trimmed to one private node's GPU-hours for one minute:

+
[{ + "specversion": "1.0", + "id": "vb1_5d0f8e2c9a7b4e1f3c6a8d2b0e9f7a41", + "source": "vbilling/us-east-1/us-east-1a", + "type": "com.vcluster.vbilling.usage.v1", + "subject": "acme-ai", + "time": "2026-10-05T10:01:00Z", + "datacontenttype": "application/json", + "dataschema": "https://github.com/vClusterLabs-Experiments/vbilling/blob/main/docs/schema/usage-event.v1.json", + "data": { + "schema_version": "vbilling.usage/v1", + "id": "vb1_5d0f8e2c9a7b4e1f3c6a8d2b0e9f7a41", + "tenant": "acme-ai", "metric": "vcluster_gpu_hours", + "quantity": 0.133333333, "unit": "gpu-hours", + "window_start": "2026-10-05T10:00:00Z", "window_end": "2026-10-05T10:01:00Z", + "region": "us-east-1", "sku": "NVIDIA-H100-80GB-HBM3", "resource_id": "gpu-node-1", + "dimensions": {"billing_mode": "private_node", "capacity_type": "reserved", + "gpu_profile": "full", "gpu_type": "NVIDIA-H100-80GB-HBM3", + "node": "gpu-node-1", "tenant_cluster": "vcluster-acme-ai-train"}, + "properties": {"gpu_devices": 8, "physical_gpus": 8}, + "source": "collector", "recorded_at": "2026-10-05T10:01:04Z" + } +}]
+ +

Headers and signature

+
POST /vbilling HTTP/1.1 +Content-Type: application/cloudevents-batch+json +User-Agent: vbilling/0.2 +X-VBilling-Signature: t=1791194464,v1=<hex HMAC-SHA256>
+

The signature uses the t=...,v1=... scheme of Stripe's Stripe-Signature: an HMAC-SHA256, keyed with your secret, over the timestamp, a period and the raw request body. Verify it on the raw bytes before parsing JSON and reject stale timestamps. A retry is signed again with a new timestamp.

+
import hashlib, hmac, time + +def verify(body: bytes, header: str, secret: str, tolerance: int = 300) -> bool: + fields = dict(p.split("=", 1) for p in header.split(",") if "=" in p) + ts, sig = fields.get("t", ""), fields.get("v1", "") + if not ts.isdigit() or abs(time.time() - int(ts)) > tolerance: + return False + mac = hmac.new(secret.encode(), ts.encode() + b"." + body, hashlib.sha256).hexdigest() + return hmac.compare_digest(mac, sig)
+ +

Responses, retries and dead letters

+
+ + + + + + + +
Your endpoint returnsvBilling
2xxCounts the batch as delivered and moves the cursor on. Return it only once the batch is stored.
400, 413 or 422Treats the payload as rejected for good: it splits the batch until each rejected event is alone, parks those as dead letters and delivers the rest.
Anything else, or no answer within 30sRetries the whole batch with exponential backoff (1s, doubling up to 5 min) while later events wait. Auth errors, 404, 409, 429 and 5xx never dead-letter an event.
+
+
+
!
+
Deduplicate on id. A batch can arrive twice: after a timeout, a restart before the cursor was saved, a dead-letter replay or a cursor rewind. A usage event keeps its ID for the same window, so upsert on it. Tenant and catalog IDs carry a timestamp: keep the latest. Return 2xx for types you ignore, because vBilling holds all delivery to the endpoint until it accepts the catalog event.
+
+

Dead letters stay on the ledger volume until a replay clears them. Add -H "Authorization: Bearer $API_TOKEN" when the API is protected:

+
curl -s localhost:8080/api/v1/destinations/webhook/dead-letters \ + | jq '.dead_letters[] | {reason, id: .event.id}' +curl -s -X POST localhost:8080/api/v1/destinations/webhook/dead-letters/replay +# send the retained ledger (RETENTION, default 7 days) to a new endpoint +curl -s -X POST localhost:8080/api/v1/destinations/webhook/cursor -d '{"seq": 0}'
+ +

Ingest API

+

Usage vBilling cannot measure itself, such as inference tokens from a model gateway or Slurm accounting, goes in through POST /api/v1/events. Accepted events are committed to the same ledger as collector events and fan out to every adapter, the webhook and your own adapter included. Declare the metric, enable ingest with a token, and send closed windows:

+
helm upgrade vbilling deploy/helm/vbilling -n vbilling-system --reuse-values \ + --set metering.customMetrics='inference_output_tokens:tokens:model' \ + --set api.existingSecret=vbilling-api # keys: api-token, ingest-token + +curl -X POST localhost:8080/api/v1/events -H "Authorization: Bearer $INGEST_TOKEN" \ + -H "X-VBilling-Client: model-gateway" -d '{"events": [{ + "tenant": "acme-ai", "metric": "inference_output_tokens", "quantity": 1250000, + "window_start": "2026-10-05T10:00:00Z", "window_end": "2026-10-05T10:01:00Z", + "dimensions": {"model": "llama-3.3-70b"}}]}'
+
    +
  • Ingest is off until INGEST_TOKEN (or API_TOKEN) is set. Up to 5,000 events or 10 MiB per request, as {"events": [...]} or a bare array.
    • +
  • The metric must be built in or declared in CUSTOM_METRICS (code:unit[:group_key|group_key]). Custom metrics are billable, so billing adapters create them at bootstrap.
    • +
  • window_end may be at most 5 minutes in the future and window_start at most 34 days old. region defaults to REGION, and source becomes ingest:<X-VBilling-Client>.
    • +
  • Send a stable id, or let vBilling derive one from the event's identity, which excludes the quantity. An ID seen in the last 48 hours is reported as a duplicate and dropped, so send each window once, after it closes.
    • +
  • Events are validated one by one: the response lists accepted IDs, duplicates and rejections with their index and reason, and is a 400 only when no event passed.
    • +
+

See Configuration and Custom metrics for the full reference.

+ +

Go adapter

+

For a billing platform with its own API, implement the Destination interface in a Go package. vBilling's packages are under internal/, so the adapter lives in your fork of the repository, at internal/destinations/<name>, and ships in your own image.

+
// internal/destinations/destination.go (package destinations) +type Destination interface { + Name() string + Bootstrap(ctx context.Context, metrics []usage.MetricDef) error + EnsureTenant(ctx context.Context, t usage.Tenant) error + RemoveTenant(ctx context.Context, t usage.Tenant) error + SendEvents(ctx context.Context, events []usage.Event) error +}
+
+ + + + + + + + + +
MethodCalledContract
NameWherever the adapter is named: status, logs, /api/v1/destinations/<name>Return the name you register, the one operators list in adapters (ADAPTERS). A name may be listed once.
BootstrapWhen delivery starts, with the whole catalog (built-in, informational and custom metrics)Create meters or billable metrics. No event is delivered until it succeeds; failures are retried with backoff.
EnsureTenantBefore a tenant's first event reaches the adapter, when its details change, and after a restartCreate or update the customer. A Permanent error is logged and delivery continues; any other error retries the batch.
RemoveTenantOnce a tenant has had no tenant clusters for OFFBOARD_GRACEDecide what offboarding means. Never delete a customer with open invoices. An error is retried on the next pass.
SendEventsWith batches of up to 500 events from the ledger, and with every parked dead letter on a replayDeliver them. The error you return decides what happens next (below).
+
+

vBilling retries every method freely, so each must be idempotent. Three optional interfaces add capabilities:

+
+ + + + + + + +
InterfaceMethodPurpose
EventFilterAccepts(metric string) boolSkip metrics the platform should not see. destinations.BillableOnly drops the informational ones. Skipped events are counted, never retried.
ReconcilerRecordedTotals(ctx context.Context, q TotalsQuery) ([]Total, error)Report what the platform recorded per tenant and metric, so GET /api/v1/reconcile can compare it with the ledger.
CustomerResolverTenantForCustomer(ctx context.Context, customerID string) (string, bool)Map a platform customer ID back to a tenant for inbound billing webhooks. Only the Stripe and Metronome adapters' webhooks (/webhooks/stripe, /webhooks/metronome) consult it today.
+
+ +

Errors decide what happens to events

+
+ + + + + + + + +
SendEvents returnsvBilling
nilCounts every event as delivered and moves the cursor on.
&destinations.PartialError{Rejected: rejected}, mapping event IDs to reasonsDead-letters the listed events and counts the rest as delivered. Return it only when nothing in the batch needs a retry.
destinations.Permanent(err)Splits the batch until each rejected event is alone, then dead-letters it.
Any other errorRetries the whole batch with backoff while later events wait. Use it for auth errors, rate limits and outages: dead-lettering usage because a key expired would lose revenue.
+
+

destinations.StatusError applies the same rule to HTTP responses: 400, 413 and 422 are permanent, everything else is retried.

+

Idempotency. Event IDs are deterministic: vb1_ plus a SHA-256 of the event's identity (tenant, metric, window, region, project, SKU, resource ID and dimensions), never of its quantity. A window keeps its ID through restarts, backfills, replays and a second replica. Send ev.ID as the platform's idempotency key and treat "already exists" as success. A retried batch repeats events you already delivered; destinations.SentCache skips them without a request. Events can also arrive late (backfill up to MAX_BACKFILL, default 6h, tenant API gap fills, replays), so accept backdated timestamps, or reject too-old events one by one with a PartialError, as the Stripe adapter does for Stripe's 35-day limit.

+ +

A minimal adapter

+

internal/destinations/acme/adapter.go, for a platform that takes JSON over HTTPS. Its settings come from the environment in the factory:

+
// Package acme ships usage to the Acme billing API. +package acme + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "io" + "net/http" + "os" + + "github.com/vclusterlabs-experiments/vbilling/internal/config" + "github.com/vclusterlabs-experiments/vbilling/internal/destinations" + "github.com/vclusterlabs-experiments/vbilling/internal/usage" +) + +func init() { destinations.Register("acme", factory) } + +func factory(cfg *config.Config) (destinations.Destination, error) { + url, key := os.Getenv("ACME_URL"), os.Getenv("ACME_API_KEY") + if url == "" || key == "" { + return nil, errors.New("ACME_URL and ACME_API_KEY must be set") + } + return New(url, key), nil +} + +type Adapter struct { + url, key string + http *http.Client + sent *destinations.SentCache +} + +func New(url, key string) *Adapter { + return &Adapter{url: url, key: key, http: destinations.NewHTTPClient(), + sent: destinations.NewSentCache(100_000)} +} + +func (a *Adapter) Name() string { return "acme" } + +// Accepts skips informational metrics (GPU utilization, GPU downtime). +func (a *Adapter) Accepts(metric string) bool { return destinations.BillableOnly(metric) } + +func (a *Adapter) Bootstrap(ctx context.Context, metrics []usage.MetricDef) error { + for _, m := range metrics { + if !m.Billable { + continue + } + meter := map[string]any{"name": m.Name, "unit": m.Unit, "group_by": m.GroupKeys} + if err := a.post(ctx, "/meters/"+m.Code, meter); err != nil { + return err + } + } + return nil +} + +func (a *Adapter) EnsureTenant(ctx context.Context, t usage.Tenant) error { + return a.post(ctx, "/customers/"+t.ID, map[string]any{ + "name": t.DisplayName, "email": t.Email, "currency": t.Currency, + }) +} + +// RemoveTenant keeps the customer: its final usage still has to be invoiced. +func (a *Adapter) RemoveTenant(ctx context.Context, t usage.Tenant) error { return nil } + +func (a *Adapter) SendEvents(ctx context.Context, events []usage.Event) error { + rejected := map[string]string{} + for _, ev := range events { + if a.sent.Has(ev.ID) { + continue // delivered before this batch was retried + } + err := a.post(ctx, "/usage", map[string]any{ + "idempotency_key": ev.ID, "customer": ev.Tenant, "meter": ev.Metric, + "quantity": ev.Quantity, "timestamp": ev.WindowStart, "sku": ev.SKU, + "region": ev.Region, "dimensions": ev.Dimensions, + }) + switch { + case err == nil: + a.sent.Add(ev.ID) + case destinations.IsPermanent(err): + rejected[ev.ID] = err.Error() // dead-lettered; the rest is delivered + default: + return err // the batch is retried with backoff + } + } + if len(rejected) > 0 { + return &destinations.PartialError{Rejected: rejected} + } + return nil +} + +func (a *Adapter) post(ctx context.Context, path string, body any) error { + b, err := json.Marshal(body) + if err != nil { + return err + } + url := a.url + path + req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(b)) + if err != nil { + return err + } + req.Header.Set("Authorization", "Bearer "+a.key) + req.Header.Set("Content-Type", "application/json") + resp, err := a.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + msg, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10)) + if resp.StatusCode >= 200 && resp.StatusCode < 300 { + return nil + } + return destinations.StatusError(http.MethodPost, url, resp.StatusCode, msg) +} + +var ( + _ destinations.Destination = (*Adapter)(nil) + _ destinations.EventFilter = (*Adapter)(nil) +)
+

Register it with a blank import next to the built-in adapters in cmd/vbilling/main.go:

+
// Register the remaining built-in adapters. + _ "github.com/vclusterlabs-experiments/vbilling/internal/destinations/lago" + _ "github.com/vclusterlabs-experiments/vbilling/internal/destinations/noop" + _ "github.com/vclusterlabs-experiments/vbilling/internal/destinations/webhook" + _ "github.com/vclusterlabs-experiments/vbilling/internal/destinations/acme"
+

Test it the way internal/destinations/webhook/adapter_test.go does: point the adapter at an httptest server and check that a 422 returns a PartialError and a 503 a retryable error. Then build and deploy your image:

+
go test -race ./internal/destinations/acme/ +make test lint +make docker-push IMAGE=registry.example.com/vbilling TAG=acme-1 +helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \ + --set adapters='{acme}' \ + --set image.repository=registry.example.com/vbilling --set image.tag=acme-1
+

The chart has no generic extra environment value: add ACME_URL, and ACME_API_KEY from a Secret, to the container's env in deploy/helm/vbilling/templates/statefulset.yaml in your fork.

+ +

Dry runs with noop

+

--set adapters='{noop}' delivers nowhere and logs every event, for example [noop] event tenant=acme-ai metric=vcluster_gpu_hours qty=0.133333333 gpu-hours sku=NVIDIA-H100-80GB-HBM3 region=us-east-1 window=10:00:00. Use it to check metering before a billing platform is connected, or list it next to an adapter you are building.

+
+ + + + + + + diff --git a/docs/example-gke.html b/docs/example-gke.html index 4fc206b..395e5d0 100644 --- a/docs/example-gke.html +++ b/docs/example-gke.html @@ -96,7 +96,7 @@ .sidebar-back:hover { border-color: var(--primary); color: var(--primary-dark); } .content { - flex: 1; margin-left: var(--sidebar-width); + flex: 1; min-width: 0; margin-left: var(--sidebar-width); padding: 48px 64px 80px; max-width: 900px; } .content h1 { @@ -128,7 +128,7 @@ .content code { background: var(--primary-light-2); padding: 2px 7px; border-radius: 4px; font-family: 'Roboto Mono', 'SF Mono', 'Fira Code', monospace; - font-size: 0.85rem; color: var(--primary-darker); + font-size: 0.85rem; color: var(--primary-darker); overflow-wrap: anywhere; } .content a { color: var(--primary-dark); text-decoration: none; font-weight: 500; } .content a:hover { text-decoration: underline; } @@ -256,11 +256,20 @@ Example: GKE with GPUs Troubleshooting +
+
Billing Adapters
+ Stripe + Metronome + Lago + Custom Billing Adapter +
Resources
GitHub Repository - Lago Docs vCluster Docs + Stripe Billing Docs + Metronome Docs + Lago Docs
diff --git a/docs/getting-started.html b/docs/getting-started.html index 62952c0..6b183b1 100644 --- a/docs/getting-started.html +++ b/docs/getting-started.html @@ -4,7 +4,7 @@ Getting Started - vBilling Docs - + @@ -122,7 +122,7 @@ /* ── Content ── */ .content { - flex: 1; margin-left: var(--sidebar-width); + flex: 1; min-width: 0; margin-left: var(--sidebar-width); padding: 48px 64px 80px; max-width: 900px; } .content h1 { @@ -153,7 +153,7 @@ .content code { background: var(--primary-light-2); padding: 2px 7px; border-radius: 4px; font-family: 'Roboto Mono', 'SF Mono', 'Fira Code', monospace; - font-size: 0.85rem; color: var(--primary-darker); + font-size: 0.85rem; color: var(--primary-darker); overflow-wrap: anywhere; } .content a { color: var(--primary-dark); text-decoration: none; font-weight: 500; } .content a:hover { text-decoration: underline; } @@ -222,6 +222,25 @@ box-shadow: var(--shadow-lg); } + .config-table-wrap { + overflow-x: auto; border-radius: var(--radius); border: 1px solid var(--border); + box-shadow: var(--shadow-sm); margin-bottom: 24px; + } + .config-table { width: 100%; border-collapse: collapse; font-size: 0.88rem; } + .config-table thead { background: var(--primary-light); } + .config-table th { + padding: 12px 16px; text-align: left; font-weight: 700; + color: var(--primary-darker); font-size: 0.78rem; + text-transform: uppercase; letter-spacing: 0.5px; + border-bottom: 2px solid var(--primary-light-2); + } + .config-table td { + padding: 10px 16px; border-bottom: 1px solid var(--border); + color: var(--text-gray); vertical-align: top; + } + .config-table tr:last-child td { border-bottom: none; } + .config-table tr:hover td { background: var(--primary-light); } + /* ── Responsive ── */ @media (max-width: 900px) { .sidebar { @@ -268,320 +287,123 @@ Example: GKE with GPUs Troubleshooting +
+
Billing Adapters
+ Stripe + Metronome + Lago + Custom Billing Adapter +
Resources
GitHub Repository - Lago Docs vCluster Docs + Stripe Billing Docs + Metronome Docs + Lago Docs

Getting Started

-

Go from zero to metered vCluster billing in four steps. This guide walks you through deploying Lago, installing vBilling, configuring pricing, and verifying the full pipeline.

+

Install vBilling in your control plane cluster, connect the billing platform you run, and see the first usage arrive. vBilling meters; your billing platform prices, invoices and collects.

Prerequisites

-

Before you begin, make sure you have the following tools installed:

    -
  • Docker and Docker Compose (for running Lago locally)
    • -
  • kubectl configured to talk to a Kubernetes cluster
    • -
  • vCluster CLI (vcluster) with at least one vCluster running
    • -
  • Helm 3 (for the Helm installation path)
    • -
  • Go 1.22+ (only if building from source)
    • -
  • metrics-server installed in the cluster (for CPU/memory metrics)
    • +
  • A control plane cluster with at least one tenant cluster. vBilling meters every tenancy model: tenant clusters with their own nodes (vCluster Private Nodes, Auto Nodes, Standalone), dedicated nodes and shared nodes.
    • +
  • kubectl and Helm 3
    • +
  • metrics-server in the control plane cluster, for CPU and memory usage
    • +
  • An account on your billing platform (a Stripe sandbox, a Metronome account), or Docker to run Lago locally
i
-
vBilling auto-discovers tenant clusters by scanning for StatefulSets/Deployments with the app=vcluster label, or via the vCluster Platform API if available. Make sure you have at least one vCluster running before starting.
+
vBilling finds tenant clusters by their StatefulSets or Deployments (label app=vcluster), and through the vCluster Platform API when it is available. No cluster yet? Run everything on a laptop with vind and Lago, or follow the GKE example with real GPUs.
- -

1Deploy Lago

-

Lago is the open-source billing engine that vBilling sends usage events to. You can run it locally with Docker Compose or deploy it to your cluster.

- -

Clone the Lago repository

-
git clone https://github.com/getlago/lago.git -cd lago
- -

Generate RSA keys for Lago API authentication

-
# Generate RSA keypair for JWT signing -openssl genrsa -out lago.key 2048 -openssl rsa -in lago.key -pubout -out lago.pub - -# Base64-encode for environment variables -export LAGO_RSA_PRIVATE_KEY=$(base64 -w0 lago.key 2>/dev/null || base64 lago.key) -echo "RSA_PRIVATE_KEY=$LAGO_RSA_PRIVATE_KEY" >> .env
- -

Start Lago with Docker Compose

-
docker compose up -d - -# Wait for services to be healthy (takes ~60 seconds) -docker compose ps - -# Expected output: -# NAME STATUS -# lago-api-1 running (healthy) -# lago-front-1 running -# lago-db-1 running (healthy) -# lago-redis-1 running (healthy) -# lago-worker-1 running
- -

Create an organization via GraphQL

-

Lago needs an organization before you can use the API. Run this GraphQL mutation against the Lago API:

-
curl -X POST http://localhost:3000/graphql \ - -H "Content-Type: application/json" \ - -d '{ - "query": "mutation { registerUser(input: { email: \"admin@example.com\", password: \"ILoveLago99!\", organizationName: \"vBilling\" }) { token user { id email } } }" - }' - -# Expected output: -# {"data":{"registerUser":{"token":"eyJhb...","user":{"id":"...","email":"admin@example.com"}}}}
- -

Get the API key from the database

-

The API key is stored in the Lago database. Extract it with:

-
# Connect to the Lago database and get the API key -docker compose exec db psql -U lago -d lago \ - -c "SELECT value FROM api_keys LIMIT 1;" - -# Expected output: -# value -# --------------------------------- -# a1b2c3d4-e5f6-7890-abcd-ef1234567890 -# (1 row) - -# Save it for later -export LAGO_API_KEY="<your-lago-api-key>"
- -
-
!
-
Keep this API key safe. You will need it when configuring vBilling. In production, store it in a Kubernetes Secret.
+ +

1Choose a billing platform

+

vBilling is the pipe, not the billing engine: prices, invoices and payments stay in the billing platform you choose. Each platform has a billing adapter in vBilling, and several adapters can run at once (for example adapters='{stripe,webhook}'), each with its own delivery cursor.

+
+ + + + + + + + +
Billing adapterChoose it for
StripePay-as-you-go prices per metric and SKU, with Stripe invoices, payments and tax
MetronomePricing on many dimensions, commits, credits and rate cards, optionally with Stripe collecting payment
LagoOpen source, self-hosted billing that also runs on a laptop with Docker Compose
Custom Billing AdapterYour own rating engine or data platform, through signed CloudEvents webhooks or an adapter written in Go
- +

2Install vBilling

-

You can install vBilling via Helm (recommended for production) or run it locally during development.

- -

Option A: Helm chart (recommended)

- -

The chart defaults to the published image ghcr.io/vclusterlabs-experiments/vbilling:v0.2.0 (linux/amd64 and linux/arm64). Build your own only to run local changes:

-
# Build multi-arch image -docker buildx build \ - --platform linux/amd64,linux/arm64 \ - -t <your-registry>/vbilling:dev \ - --push . - -# Or single-arch for local testing: -# docker build -t vbilling:latest .
- -

Install with Helm:

-
helm upgrade --install vbilling deploy/helm/vbilling \ - --namespace vbilling-system --create-namespace \ - --set lago.apiURL="http://lago-api.lago-system.svc.cluster.local:3000" \ - --set lago.apiKey="$LAGO_API_KEY" - -# Expected output: -# Release "vbilling" does not exist. Installing it now. -# NAME: vbilling -# NAMESPACE: vbilling-system -# STATUS: deployed
- -

Or use an existing Kubernetes secret for the API key:

-
# Create the secret first -kubectl create secret generic vbilling-lago-secret \ - -n vbilling-system \ - --from-literal=api-key="$LAGO_API_KEY" - -# Install with secret reference -helm upgrade --install vbilling deploy/helm/vbilling \ - --namespace vbilling-system --create-namespace \ - --set lago.apiURL="http://lago-api.lago-system.svc.cluster.local:3000" \ - --set lago.existingSecret="vbilling-lago-secret"
- -

Option B: Run locally (development)

-
# Clone the repository -git clone https://github.com/vClusterLabs-Experiments/vbilling.git -cd vbilling - -# Build the binary -make build - -# Expected output: -# CGO_ENABLED=0 go build -o bin/vbilling ./cmd/vbilling - -# Run with environment variables -export LAGO_API_URL="http://localhost:3000" -export LAGO_API_KEY="<your-lago-api-key>" -./bin/vbilling - -# Expected output: -# vBilling - vCluster Billing Controller -# ======================================= -# Lago API: http://localhost:3000 -# Plan: vcluster-standard | Currency: USD -# Collection: 1m0s | Reconcile: 30s -# Bootstrapping Lago billing configuration... -# [bootstrap] setting up Lago billing configuration... -# [bootstrap] created metric "vcluster_cpu_core_hours" (id=...) -# [bootstrap] created metric "vcluster_memory_gb_hours" (id=...) -# ... -# [bootstrap] created plan "vcluster-standard" (id=...) with 9 charges -# [bootstrap] Lago billing configuration complete -# Starting billing controller...
- -
-
i
-
When running locally, vBilling uses your ~/.kube/config to connect to the cluster. Make sure your kubeconfig context points to the cluster with your tenant clusters.
-
- - -

3Configure Pricing in Lago

-

vBilling bootstraps all 9 billable metrics and a default plan (vcluster-standard) on startup. However, all charges default to $0.00. You must set pricing in the Lago UI.

- -

Open the Lago dashboard

-
# Lago UI is available at: -open http://localhost:80 - -# Log in with the credentials you used in the GraphQL mutation: -# Email: admin@example.com -# Password: ILoveLago99!
- -

Edit the plan charges

-
    -
  1. Navigate to Plans in the left sidebar
    2. -
  2. Click on vCluster Standard (code: vcluster-standard)
    3. -
  3. Scroll to the Charges section
    4. -
  4. For each charge, click the edit icon and set the price per unit:
    5. -
- -
-
i
-
- Recommended pricing (adjust to your costs):
- CPU Core-Hours: $0.05 | Memory GB-Hours: $0.01 | Storage GB-Hours: $0.001
- Instance Hours: $0.10 | GPU Hours: $2.50 (default) | Network Egress/GB: $0.05
- LoadBalancer Hours: $0.025 | Private Node Hours: $1.00 -
-
- - -

4Verify It Works

-

Once vBilling is running, it will auto-discover your tenant clusters and start sending usage events within one reconcile cycle (default: 30 seconds).

- -

Check vBilling logs

-
# If running via Helm: -kubectl logs -n vbilling-system -l app=vbilling -f - -# Expected output: -# [controller] starting billing controller -# [controller] reconcile interval: 30s, collection interval: 1m0s -# [controller] new vCluster discovered: vcluster-team-alpha/my-vcluster -# [controller] ensured customer vcluster-vcluster-team-alpha-my-vcluster -# [controller] created subscription sub-vcluster-vcluster-team-alpha-my-vcluster -> plan vcluster-standard -# [metrics] vcluster-team-alpha: CPU=0.250 cores, Memory=0.51 GB -# [metrics] vcluster-team-alpha: Storage=10.00 GB (2 PVCs) -# [controller] sent 4 billing events to Lago
- -

Check Lago for customers

-

Open the Lago UI and navigate to Customers. You should see one customer per discovered vCluster:

-
# Or verify via the API: -curl -s http://localhost:3000/api/v1/customers \ - -H "Authorization: Bearer $LAGO_API_KEY" | jq '.customers[].external_id' - -# Expected output: -# "vcluster-vcluster-team-alpha-my-vcluster" -# "vcluster-default-dev-cluster"
- -

Check current usage

-
curl -s http://localhost:3000/api/v1/customers/vcluster-vcluster-team-alpha-my-vcluster/current_usage\?external_subscription_id=sub-vcluster-vcluster-team-alpha-my-vcluster \ - -H "Authorization: Bearer $LAGO_API_KEY" | jq - -# Expected output: -# { -# "customer_usage": { -# "from_datetime": "2025-01-01T00:00:00Z", -# "to_datetime": "2025-01-31T23:59:59Z", -# "amount_cents": 1250, -# "amount_currency": "USD", -# ... -# } -# }
- -
-
✓
-
If you see customers and usage data, vBilling is working. Usage events are sent every collection interval (default: 60 seconds). Invoices are generated monthly by Lago.
-
- -

Try It Locally with vind

-

You can test the full pipeline on your laptop using vind (vCluster in Docker) -- no cloud cluster needed.

- -

Create a vind control plane cluster

-
# Set Docker driver and create a control plane cluster -vcluster use driver docker -vcluster create vbilling-host --connect=true - -# Install metrics-server -kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -kubectl patch deployment metrics-server -n kube-system --type=json \ - -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
- -

Create nested tenant clusters

-
vcluster use driver helm -vcluster create team-alpha --namespace vcluster-team-alpha --connect=false -vcluster create team-beta --namespace vcluster-team-beta --connect=false -vcluster create team-gpu --namespace vcluster-team-gpu --connect=false
- -

Deploy vBilling as a pod in the vind cluster

-
# Build the image -docker build --build-arg TARGETARCH=arm64 -t vbilling:test . - -# Load into vind's containerd -docker save vbilling:test | docker exec -i vcluster.cp.vbilling-host \ - ctr -n k8s.io images import --all-platforms - - -# Create secret and install via Helm +

The chart defaults to the published image ghcr.io/vclusterlabs-experiments/vbilling:v0.2.0 (linux/amd64 and linux/arm64). Keep billing keys in Secrets, never in --set flags or values files. From a clone of the repository, with Stripe as the example:

+
git clone https://github.com/vClusterLabs-Experiments/vbilling.git && cd vbilling kubectl create namespace vbilling-system -kubectl create secret generic lago-credentials \ - --namespace vbilling-system \ - --from-literal=api-key="$LAGO_API_KEY" - -helm upgrade --install vbilling deploy/helm/vbilling \ - --namespace vbilling-system \ - --set image.repository=vbilling \ - --set image.tag=test \ - --set image.pullPolicy=Never \ - --set lago.apiURL=http://host.docker.internal:3000 \ - --set lago.existingSecret=lago-credentials
+kubectl -n vbilling-system create secret generic vbilling-stripe --from-file=api-key=./stripe-test-key +helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \ + --set adapters='{stripe}' \ + --set stripe.existingSecret=vbilling-stripe \ + --set region=us-east-1
+

The other adapters take the same shape:

+
+ + + + + + + + + +
AdapterSecret keysHelm values
stripeapi-key, webhook-secretstripe.existingSecret
metronomeapi-token, webhook-secretmetronome.existingSecret, metronome.rateCard
lagoapi-keylago.existingSecret, lago.apiURL
webhooksecretwebhook.existingSecret, webhook.url
noopNone: logs every event instead of sending it, for a dry run
+
+

Run one vBilling per control plane cluster and set region: it is stamped on every event, and every ledger record stays in that cluster.

-

Verify

-
kubectl logs -n vbilling-system statefulset/vbilling +

Run from source (development)

+
make build +ADAPTERS=noop DATA_DIR=./data ./bin/vbilling
+

Outside a cluster, vBilling uses your current kubeconfig context. With noop, each event is logged ([noop] event tenant=... metric=... qty=...) instead of sent.

-# Expected output (v0.2): -# Using in-cluster Kubernetes config -# [controller] starting (adapters=[lago], window=1m0s, reconcile=30s, region=default) -# API listening on :8080 -# [bootstrap] setting up Lago billing configuration... -# [controller] window 10:18:20: 17 events for 3 tenant cluster(s) (11 pods, 5 GPU pods, 0 nodes down) -# [lago] subscribed tenant acme to plan vcluster-standard
+ +

3Configure pricing and packaging

+

vBilling creates the metrics in your billing platform; the prices are yours. Every tenant is on the default plan vcluster-standard unless its tenant cluster carries vbilling.vcluster.com/plan.

+
    +
  • Stripe: a metered price on each meter you charge for, tagged with the plan for automatic subscriptions. See Prices and subscriptions.
    • +
  • Metronome: rate card products on the billable metrics, priced on dimensions such as SKU, region and capacity type. See Pricing on dimensions.
    • +
  • Lago: the plan's charges, with charge filters on sku or capacity_type. See Configure pricing.
    • +
+

Every event carries the dimensions to price on, among them sku, region, capacity_type (on-demand, spot, preemptible or reserved), billing_mode and tenant_class. See the Metrics Reference.

+ + +

4Verify it works

+

vBilling meters closed windows (60 seconds by default), commits them to its ledger, and delivers them to each adapter.

+
kubectl -n vbilling-system logs statefulset/vbilling +Using in-cluster Kubernetes config +[controller] starting (adapters=[stripe], window=1m0s, reconcile=30s, region=us-east-1) +API listening on :8080 +[controller] window 10:18:00: ... events for 3 tenant cluster(s) (...)
+
kubectl -n vbilling-system port-forward svc/vbilling 8080 +# dashboard: http://localhost:8080/ +curl -s localhost:8080/api/v1/status | jq '.destinations' # delivery health per adapter +curl -s 'localhost:8080/api/v1/usage?group_by=tenant,sku' | jq # line items from the ledger +curl -s localhost:8080/api/v1/reconcile | jq '.results[] | {destination, ok}' # Stripe and Metronome
✓
-
This is output from a verified end-to-end test. vBilling runs as a StatefulSet with a persistent ledger and an in-cluster ServiceAccount, discovers tenant clusters via RBAC, and delivers events to Lago. Check delivery with kubectl -n vbilling-system port-forward svc/vbilling 8080 and curl localhost:8080/api/v1/status.
+
Once the destinations report deliveries, the usage is in your billing platform: meters and customers in Stripe, customers and usage in Metronome, or customers and subscriptions in Lago. A platform outage never loses usage: the ledger keeps it until the adapter catches up.
-

Clean up

-
vcluster use driver docker -vcluster delete vbilling-host -cd deploy/lago && docker compose down -v
- -

Next Steps

+

Next steps

diff --git a/docs/index.html b/docs/index.html index 9e73796..f66edd1 100644 --- a/docs/index.html +++ b/docs/index.html @@ -558,7 +558,7 @@ } .qs-step { display: grid; - grid-template-columns: 40px 1fr; + grid-template-columns: 40px minmax(0, 1fr); gap: 24px; align-items: start; } @@ -732,13 +732,13 @@ .section-dark .section-title { color: var(--text-reversed); } .section-dark .section-body { color: rgba(255,255,255,0.65); } - .AI Cloud-grid { + .ai-cloud-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 64px; align-items: center; } - .AI Cloud-copy .section-body { max-width: 100%; margin-bottom: 32px; } + .ai-cloud-copy .section-body { max-width: 100%; margin-bottom: 32px; } .checklist { list-style: none; display: flex; @@ -893,7 +893,7 @@ .section-title { font-size: 32px; } .steps { grid-template-columns: repeat(2, 1fr); } .features-grid { grid-template-columns: repeat(2, 1fr); } - .AI Cloud-grid { grid-template-columns: 1fr; gap: 40px; } + .ai-cloud-grid { grid-template-columns: 1fr; gap: 40px; } .main-nav { display: none; } .footer-top { grid-template-columns: 1fr 1fr; gap: 40px; } } @@ -995,7 +995,7 @@

Usage-based billing for AI Clouds running vCluster or vMetalRead the docs View on GitHub -

New in v0.2: Stripe and Metronome, a verifiable ledger, vCluster Private Nodes, and DCGM through your own Prometheus. See it on GKE with shared and private GPUs →

+

New in v0.2: Stripe and Metronome, a verifiable ledger, vCluster Private Nodes, and DCGM through your own Prometheus. See it end to end on GKE with real GPUs →

@@ -1051,12 +1051,12 @@

Usage-based billing for AI Clouds running vCluster or vMetal

Adapter-agnostic

-

Ships to your billing backend

+

Ships to your billing platform

vBilling is the pipe, not the billing engine. Meter tenant clusters once and route events to whichever billing adapter you run.

+ aria-label="vCluster and vMetal feed vBilling which routes usage events to Lago, Metronome, Stripe, or a custom billing adapter"> @@ -1102,7 +1102,7 @@

Ships to your billing backend

- Webhook / Custom + Custom Adapter
@@ -1128,7 +1128,7 @@

Three steps to tenant billing

Choose Billing Platform

-

Pick the billing backend you already run: Stripe, Metronome (with Stripe for payments), Lago, or your own pipeline through a signed webhook. Send to several at once.

+

Pick the billing platform you already run: Stripe, Metronome (with Stripe for payments), Lago, or your own pipeline through a signed webhook. Send to several at once.

02 · Configure

@@ -1148,7 +1148,7 @@

Configure Pricing & Packaging

Connect vBilling Adapter

-

Deploy vBilling and point it at your backend. Tenant clusters are auto-discovered, and usage starts flowing from the next closed window.

+

Deploy vBilling and point it at your billing platform. Tenant clusters are auto-discovered, and usage starts flowing from the next closed window.

@@ -1160,7 +1160,7 @@

Connect vBilling Adapter

Features

Everything you need to bill tenants

-

vBilling meters shared and dedicated GPU capacity, commits it to a durable ledger, and delivers it to every billing backend you run, exactly once.

+

vBilling meters GPU capacity in every tenancy model, commits it to a durable ledger, and delivers it to every billing platform you run, exactly once.

@@ -1192,7 +1192,7 @@

GPU SKU-aware

grid_view

Fractional GPUs

-

MIG slices and time-sliced GPUs are metered at their fraction, so shared GPUs can be sold by the slice.

+

MIG slices and time-sliced GPUs are metered at their fraction of a GPU, so every slice can carry its own price.

@@ -1203,8 +1203,8 @@

Fair metering

admin_panel_settings -

Dedicated nodes

-

Bill whole bare-metal nodes by SKU or instance type, without double-billing the pods that run on them.

+

Node capacity tracking

+

Meter the full capacity of dedicated nodes (CPU, memory and GPU SKUs), billed by SKU or instance type, without double-billing the pods that run on them.

@@ -1252,7 +1252,7 @@

Regions and sovereignty

autorenew

Auto-discovery

-

Finds tenant clusters and vCluster Platform projects. Customers, meters and metrics are created in your backend automatically.

+

Finds tenant clusters and vCluster Platform projects. Customers, meters and metrics are created in your billing platform automatically.

@@ -1265,11 +1265,11 @@

Auto-discovery

Architecture

How the pieces fit together

-

vBilling runs as a controller in your Control Plane Cluster. It watches tenant clusters, collects capacity and usage metrics, and streams events to the billing adapter you choose.

+

vBilling runs as a controller in your control plane cluster. It watches tenant clusters, collects capacity and usage metrics, and streams events to the billing adapter you choose.

-
Control Plane Cluster
+
control plane cluster
─────────────────────────────────────────────
 
tenant cluster team-alpha · private · 4x A100
@@ -1297,7 +1297,7 @@

How the pieces fit together

Quick start

Up and running in 3 steps

-

Pick your billing adapter below. Each tab walks through deploy, install, and pricing for that platform.

+

Pick your billing adapter below. Each tab walks through deploy, install, and pricing for that platform; the full guides cover Stripe, Metronome, Lago and a custom billing adapter.

@@ -1452,11 +1452,11 @@

Install vBilling with Metronome

2

Price on dimensions

-

vBilling creates SUM billable metrics grouped on region, sku, capacity_type, billing_mode and tenant_cluster. Add rate card products with pricing group keys.

+

vBilling creates SUM billable metrics, each with one compound group key (for GPU hours: region, sku, capacity_type, billing_mode, tenant_cluster). Add rate card products whose pricing and presentation group keys together form it.

# Metronome product on vcluster_gpu_hours
-pricing_group_key:      [region, sku, capacity_type]
+pricing_group_key:      [region, sku, capacity_type, billing_mode]
 presentation_group_key: [tenant_cluster]
@@ -1480,8 +1480,8 @@

Reconcile

@@ -1525,21 +1525,21 @@

Default events, zero manual tagging

-
+
-
+

For AI Clouds

Built for AI Clouds

-

If you run an AI cloud on Kubernetes, vBilling gives you the billing pipe, without forcing a billing backend on you.

+

If you run an AI Cloud on Kubernetes, vBilling gives you the billing pipe, without forcing a billing platform on you.


-

AI Clouds give each customer a tenant cluster with dedicated GPU nodes. vBilling detects the hardware, meters node capacity by GPU SKU, and streams events into whichever billing adapter you run.

+

AI Clouds give each customer an isolated tenant cluster with its own GPU nodes. vBilling detects the hardware, meters node capacity by GPU SKU, and streams events into whichever billing adapter you run.

    -
  • Shared, fractional (MIG, time-sliced) and dedicated GPUs, metered to the second
    • +
  • Whole GPU nodes (Private Nodes, Standalone, dedicated nodes) and fractional GPUs (MIG, time-slicing), metered to the second
  • SKU, region and capacity-type dimensions priced in your billing platform
  • No charges for provisioning, image pulls, or unhealthy hardware
  • Durable ledger, backfill and reconciliation: metering held to financial-data standards
    • -
  • Stripe, Metronome, Lago or a webhook: keep your existing billing backend
    • +
  • Stripe, Metronome, Lago or a custom billing adapter: keep your existing billing platform
@@ -1557,7 +1557,7 @@

Built for AI Clouds

+ private nodes (8x H100)
↓
vBilling auto-discovers - detects dedicated nodes + reads its private nodes
↓
Meters every 60s window, per hour: 8 GPU-hours (H100) @@ -1613,7 +1613,7 @@

Company

- © 2026 vCluster. All rights reserved. + © 2026 vCluster Labs. All rights reserved.
Privacy Policy Terms of Service diff --git a/docs/lago.html b/docs/lago.html new file mode 100644 index 0000000..5f1dd0c --- /dev/null +++ b/docs/lago.html @@ -0,0 +1,425 @@ + + + + + + Lago - vBilling Docs + + + + + + + + + + +
+ + +
+

Lago

+

Usage-based billing on open source from end to end. Lago rates and invoices, vBilling meters your tenant clusters and sends Lago the events, and both can run on a laptop next to your tenant clusters.

+ +

What vBilling sets up

+
+ + + + + + + + + +
In LagoCreated by vBilling
Billable metricsOne per vBilling metric (vcluster_gpu_hours, vcluster_cpu_core_hours, ...), summing a field named after it (gpu_hours, cpu_core_hours, ...)
Planvcluster-standard (billing.defaultPlanCode) with one charge per metric at $0, for you to price
CustomersOne per tenant: external_id is the tenant ID, with its name, email, currency and metadata
Subscriptionssub-<tenant ID> on the tenant's plan (vbilling.vcluster.com/plan, or the default plan)
EventsBatches of up to 100. transaction_id is the vBilling event ID, so a resent event is never counted twice.
+
+

Prices, invoices, taxes and the payment provider stay in Lago.

+ +

Run Lago locally

+

The repository ships a Docker Compose file for Lago 1.17, with the UI on :8081 and the API on :3000 (vBilling's own dashboard uses :8080). Lago signs its tokens with an RSA key: generate one, and never commit it. From the repository root:

+
git clone https://github.com/vClusterLabs-Experiments/vbilling.git && cd vbilling +openssl genrsa 2048 > deploy/lago/lago_rsa.key +echo "LAGO_RSA_PRIVATE_KEY=$(base64 -i deploy/lago/lago_rsa.key | tr -d '\n')" > deploy/lago/.env +docker compose -f deploy/lago/docker-compose.yml --env-file deploy/lago/.env up -d
+

Open http://localhost:8081 and sign up to create the organization. Then copy the API key from Developers, API keys into a file (paste it, press Enter, then Ctrl-D):

+
cat > ./lago-api-key
+

For a shared or production setup, run Lago in Kubernetes with its Helm chart (see the Lago docs) and point lago.apiURL at its API Service.

+ +

Install vBilling with Lago

+
kubectl create namespace vbilling-system +kubectl -n vbilling-system create secret generic vbilling-lago --from-file=api-key=./lago-api-key +helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \ + --set adapters='{lago}' \ + --set lago.apiURL=http://lago-api.lago-system.svc.cluster.local:3000 \ + --set lago.existingSecret=vbilling-lago
+
+ + + + + + + + + +
Helm valueEnvironment variableDefaultEffect
lago.apiURLLAGO_API_URLhttp://lago-api.lago-system.svc.cluster.local:3000Lago API, reachable from the vBilling pod. Use http://host.docker.internal:3000 for Lago in Docker on the same machine as a vind control plane cluster.
lago.existingSecretSecret with the key api-key. Use it instead of lago.apiKey.
lago.apiKeyLAGO_API_KEYLago API key.
billing.defaultPlanCodeDEFAULT_PLAN_CODEvcluster-standardPlan that vBilling creates, and that tenants without a plan annotation subscribe to.
billing.currencyBILLING_CURRENCYUSDCurrency of the plan and of tenants without a currency annotation.
+
+

Check that vBilling bootstrapped Lago and subscribed the first tenants:

+
kubectl -n vbilling-system logs statefulset/vbilling | grep -E '\[(bootstrap|lago)\]' +[bootstrap] created metric "vcluster_gpu_hours" (id=...) +[bootstrap] created plan "vcluster-standard" (id=...) with ... charges +[lago] subscribed tenant vcluster-team-a-team-a to plan vcluster-standard +curl -s http://localhost:3000/api/v1/customers -H "Authorization: Bearer $(cat ./lago-api-key)" | jq '.customers[].external_id'
+ +

Configure pricing

+

Every charge on vcluster-standard starts at $0, so invoices stay at $0.00 until you price them. In the Lago UI, open Plans, vCluster Standard, and set a price per unit on each charge. vBilling sends every dimension as an event property, so charge filters can price one metric several ways:

+
    +
  • sku (or gpu_type): H100 hours differently from L40S hours
    • +
  • capacity_type: a discount for spot and preemptible, or a reserved rate
    • +
  • region, billing_mode, tenant_class and tenant_cluster
    • +
+

To price tenants differently, create more plans and set vbilling.vcluster.com/plan on their tenant clusters. Lago pricing lists the charges to configure, with example price lists.

+ +

Events in Lago

+

Each event carries the quantity under the metric's field name, which the billable metric sums, and every dimension as a property (trimmed here):

+
{ + "transaction_id": "vb1_3f9c...", + "external_subscription_id": "sub-vcluster-team-a-team-a", + "code": "vcluster_gpu_hours", + "timestamp": 1791216000, + "properties": { + "gpu_hours": 0.25, + "sku": "NVIDIA-H100-80GB-HBM3", + "capacity_type": "on-demand", + "billing_mode": "shared", + "region": "us-east-1", + "tenant_cluster": "vcluster-team-a-team-a", + "vcluster": "vcluster-team-a-team-a" + } +}
+

The timestamp is the window start. vcluster repeats tenant_cluster so that charge filters written for v0.1 keep working.

+ +

Offboarding

+

When a tenant has been gone for billing.offboardGrace (default 1h), vBilling terminates its subscription and Lago issues the final invoice.

+ +

Good to know

+
    +
  • GET /api/v1/reconcile covers Stripe and Metronome. For Lago, compare the ledger (GET /api/v1/usage) with Lago's current usage for the subscription.
    • +
  • Billing states and enforcement come from Stripe and Metronome webhooks, and are only enabled when one of their webhook secrets is set. A Lago-only setup has no billing states.
    • +
  • Lago also receives vcluster_gpu_utilization, an informational metric that v0.1 created in Lago and existing plans may still reference. Leave it unpriced.
    • +
+ +

Run it all on a laptop with vind

+

vind runs vCluster in Docker, so a control plane cluster, three tenant clusters, vBilling and Lago fit on one machine. Start Lago as above, then, from the repository root:

+
# a control plane cluster in Docker, with metrics-server +vcluster use driver docker +vcluster create vbilling-host --connect=true +kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml +kubectl patch deployment metrics-server -n kube-system --type=json \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' + +# three tenant clusters inside it +vcluster use driver helm +vcluster create team-alpha --namespace vcluster-team-alpha --connect=false +vcluster create team-beta --namespace vcluster-team-beta --connect=false +vcluster create team-gpu --namespace vcluster-team-gpu --connect=false + +# vBilling, sending to Lago on the host +kubectl create namespace vbilling-system +kubectl -n vbilling-system create secret generic vbilling-lago --from-file=api-key=./lago-api-key +helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \ + --set adapters='{lago}' \ + --set lago.apiURL=http://host.docker.internal:3000 \ + --set lago.existingSecret=vbilling-lago
+

Within a minute or two the log shows a window line for the three tenant clusters and a subscription per tenant:

+
kubectl -n vbilling-system logs statefulset/vbilling +[controller] starting (adapters=[lago], window=1m0s, reconcile=30s, region=default) +API listening on :8080 +[controller] window ...: ... events for 3 tenant cluster(s) (...) +[lago] subscribed tenant vcluster-vcluster-team-alpha-team-alpha to plan vcluster-standard
+

Open the dashboard with kubectl -n vbilling-system port-forward svc/vbilling 8080 and http://localhost:8080/, and the customers in Lago at http://localhost:8081. To clean up:

+
vcluster use driver docker +vcluster delete vbilling-host +docker compose -f deploy/lago/docker-compose.yml down -v
+ +

Troubleshooting

+ +
+
+ + + + + + diff --git a/docs/metrics.html b/docs/metrics.html index 0293d7c..2777e19 100644 --- a/docs/metrics.html +++ b/docs/metrics.html @@ -4,7 +4,7 @@ Metrics Reference - vBilling Docs - + @@ -96,7 +96,7 @@ .sidebar-back:hover { border-color: var(--primary); color: var(--primary-dark); } .content { - flex: 1; margin-left: var(--sidebar-width); + flex: 1; min-width: 0; margin-left: var(--sidebar-width); padding: 48px 64px 80px; max-width: 920px; } .content h1 { @@ -128,7 +128,7 @@ .content code { background: var(--primary-light-2); padding: 2px 7px; border-radius: 4px; font-family: 'Roboto Mono', 'SF Mono', 'Fira Code', monospace; - font-size: 0.85rem; color: var(--primary-darker); + font-size: 0.85rem; color: var(--primary-darker); overflow-wrap: anywhere; } .content a { color: var(--primary-dark); text-decoration: none; font-weight: 500; } .content a:hover { text-decoration: underline; } @@ -257,17 +257,26 @@ Example: GKE with GPUs Troubleshooting
+
+
Billing Adapters
+ Stripe + Metronome + Lago + Custom Billing Adapter +
Resources
GitHub Repository - Lago Docs vCluster Docs + Stripe Billing Docs + Metronome Docs + Lago Docs

Metrics Reference

-

Every metric vBilling v0.2 emits, how it is measured, and the dimensions billing backends can price on.

+

Every metric vBilling v0.2 emits, how it is measured, and the dimensions billing platforms can price on.

i
@@ -293,7 +302,7 @@

vcluster_gpu_hours

Billing starts at the first container start (image pulls and scheduling are free) and stops when the last container finishes. Pods that started or finished mid-window are billed for exactly the seconds they ran.

Fractional GPUs: a MIG slice counts as slices/7 of a GPU (A30: /4), e.g. mig-1g.10gb = 1/7. This works for the MIG mixed strategy (nvidia.com/mig-* resources), the single strategy (slices exposed as nvidia.com/gpu, product label ending in -MIG-<profile>) and GKE GPU partitions (cloud.google.com/gke-gpu-partition-size). On time-sliced nodes (nvidia.com/gpu.replicas > 1, or GKE cloud.google.com/gke-max-shared-clients-per-gpu with time-sharing or MPS) a device counts as 1/replicas. A DRA claim counts the devices allocated to it (a MIG device at its profile's fraction), split across the pods it is reserved for. The SKU becomes <model>-mig-<profile> or <model>-timeslice-<n>, always against the base GPU model (the -SHARED and -MIG- label suffixes are dropped).

-

Dedicated and private nodes: the node's GPU capacity is billed in whole GPUs with billing_mode=dedicated_node (control plane cluster nodes reserved for the tenant) or billing_mode=private_node (vCluster Private Nodes, Auto Nodes and Standalone, metered through the tenant cluster's own API): MIG slices and time-slicing replicas are converted back, so 8 GPUs shared 4 ways bill as 8, not 32. Pods on those nodes are not billed again. See Dedicated and Private Nodes.

+

Dedicated and private nodes: the node's GPU capacity is billed in whole GPUs with billing_mode=dedicated_node (control plane cluster nodes assigned to the tenant) or billing_mode=private_node (vCluster Private Nodes, Auto Nodes and Standalone, metered through the tenant cluster's own API): MIG slices and time-slicing replicas are converted back, so 8 GPUs shared 4 ways bill as 8, not 32. Pods on those nodes are not billed again. See Dedicated and Private Nodes.

Fair metering: GPU time on a node that is NotReady or carries an UNHEALTHY_NODE_TAINTS taint is emitted as vcluster_gpu_downtime_hours instead. So is a dedicated or private node's GPU time while its GPUs are installed (GPU Feature Discovery's nvidia.com/gpu.count) but not allocatable, for example while the driver rebuilds after a reboot or a MIG change.

Dimensions

@@ -544,7 +553,7 @@

vcluster_gpu_downtime_hours

Quantity
as GPU hours
-

An auditable record of the time tenants were not charged for, never sent to billing backends.

+

An auditable record of the time tenants were not charged for, never sent to billing platforms.

Dimensions

@@ -557,7 +566,7 @@

Dimensions

Custom metrics

-

Declare operator metrics with CUSTOM_METRICS="code:unit[:key|key],..." (for example inference_output_tokens:tokens:model|region) and push closed windows to POST /api/v1/events. Billing backends create meters and billable metrics for them at startup.

+

Declare operator metrics with CUSTOM_METRICS="code:unit[:key|key],..." (for example inference_output_tokens:tokens:model|region) and push closed windows to POST /api/v1/events. The billing adapters create meters and billable metrics for them at startup.

diff --git a/docs/metronome.html b/docs/metronome.html new file mode 100644 index 0000000..48b000a --- /dev/null +++ b/docs/metronome.html @@ -0,0 +1,519 @@ + + + + + + Metronome - vBilling Docs + + + + + + + + + + +
+ + +
+

Metronome

+

The Metronome billing adapter ships vBilling usage to Metronome for rating and invoicing. vBilling creates the customers, billable metrics and contracts; your rate cards set the prices. With the Stripe link on, Stripe collects payment, invoices and tax.

+ +
+
!
+
The adapter's tests validate every request body against Metronome's published OpenAPI schemas and check webhook verification against Metronome's documented test vector. They do not run against a live Metronome account, so run vBilling against a Metronome sandbox before you invoice from it.
+
+ +

When to choose Metronome

+

Both billing platforms receive the same usage. They differ in where pricing lives.

+
+
+

Stripe Billing Meters

+

One meter per metric, split per SKU, with a metered price on each. A good fit for a pay-as-you-go price list. See Stripe.

+
+
+

Metronome

+

One billable metric per metric, priced on dimensions such as region, SKU and capacity type. Contracts add commits, credits and rate cards in the currency you invoice in.

+
+
+
+
KeyMeaning
+ + + + + + +
ModeSettingsWho does what
Metronomeadapters: [metronome]Metronome rates usage and produces invoices.
Metronome + Stripeadapters: [metronome]
metronome.stripeLink: true		Metronome rates usage. vBilling creates each tenant's Stripe customer and links it, so Stripe collects payment, invoices and tax. The stripe adapter is not needed.
Plus a data feedadapters: [metronome, webhook]The same events also stream as signed CloudEvents to your data platform. Each destination has its own cursor, so one outage never blocks the other.
+
+ +

Install

+

You need a Metronome API token. For Metronome + Stripe, also connect your Stripe account in Metronome and give vBilling a key for the same Stripe account: the customer IDs it links must exist there. Keep both in Secrets; values passed with --set are stored in the Helm release.

+
kubectl create namespace vbilling-system +kubectl -n vbilling-system create secret generic vbilling-metronome --from-file=api-token=./metronome-token +helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \ + --set adapters='{metronome}' \ + --set metronome.existingSecret=vbilling-metronome \ + --set metronome.rateCard=payg-usd \ + --set region=us-east-1
+

For Metronome + Stripe, add a Stripe Secret and turn on the link:

+
kubectl -n vbilling-system create secret generic vbilling-stripe --from-file=api-key=./stripe-key +helm upgrade vbilling deploy/helm/vbilling -n vbilling-system --reuse-values \ + --set metronome.stripeLink=true \ + --set stripe.existingSecret=vbilling-stripe
+ +

Settings

+
+ + + + + + + + + + + + +
Helm valueEnvironment variableDefaultDescription
adaptersADAPTERS[lago]Include metronome.
metronome.existingSecretMETRONOME_API_TOKEN, METRONOME_WEBHOOK_SECRET(none)Secret with the key api-token and, for webhooks, webhook-secret.
metronome.rateCardMETRONOME_RATE_CARD(none)Rate card alias or ID. Each tenant gets a contract on it; a tenant's vbilling.vcluster.com/plan overrides it.
metronome.stripeLinkMETRONOME_STRIPE_LINKfalseCreate each tenant's Stripe customer and link it in Metronome. Needs stripe.existingSecret.
metronome.stripeCollectionMethodMETRONOME_STRIPE_COLLECTION_METHODcharge_automaticallyOr send_invoice.
stripe.existingSecretSTRIPE_API_KEY, STRIPE_WEBHOOK_SECRET(none)Secret with the key api-key and, for Stripe webhooks, webhook-secret.
metronome.apiBaseMETRONOME_API_BASEhttps://api.metronome.comAPI base URL.
metronome.apiToken, metronome.webhookSecret(none)Inline alternatives to the Secret. Avoid them.
+
+ +

What vBilling creates in Metronome

+

Customers

+

Each tenant becomes one customer whose ingest alias is the tenant ID: vcluster-<namespace>-<name> by default, or the value of vbilling.vcluster.com/tenant. vBilling looks the customer up by alias before creating it, and a 409 on the alias resolves to the existing customer, so a tenant never gets two.

+
    +
  • The customer's name comes from vbilling.vcluster.com/display-name and is set at creation. Rename it in Metronome afterwards.
    • +
  • To use a customer that already exists, set vbilling.vcluster.com/metronome-customer-id on the tenant cluster's StatefulSet, its namespace or its VirtualClusterInstance.
    • +
  • Offboarded tenants keep their customer and contract, so final usage still invoices.
    • +
+ +

Billable metrics

+

At startup vBilling creates one SUM billable metric per billable metric code. It sums the value property, filters on the event type (the metric code) and requires value and every group-key property to be present. Each metric gets one compound group key: the metric's pricing dimensions plus tenant_cluster. Existing metrics are matched by ID or by name and never duplicated; an archived metric is created again at the next start.

+
+ + + + + + + + + + + + +
Billable metricGroup key
GPU Hours [vcluster_gpu_hours]region, sku, capacity_type, billing_mode, tenant_cluster
CPU Core Hours [vcluster_cpu_core_hours]region, capacity_type, billing_mode, tenant_cluster
Memory GiB Hours [vcluster_memory_gb_hours]region, capacity_type, billing_mode, tenant_cluster
Storage GiB Hours [vcluster_storage_gb_hours]region, sku, tenant_cluster
Tenant Cluster Hours [vcluster_instance_hours]region, tenant_class, tenant_cluster
Dedicated Node Hours [vcluster_private_node_hours]region, sku, capacity_type, tenant_cluster
Network Egress GiB [vcluster_network_egress_gb]region, tenant_cluster
Load Balancer Hours [vcluster_lb_hours]region, tenant_cluster
+
+

Custom metrics, from CUSTOM_METRICS or defined in PromQL, get a billable metric too, grouped on the keys you declare. The informational metrics vcluster_gpu_utilization and vcluster_gpu_downtime_hours are never sent to Metronome.

+ +

Contracts

+

When a tenant has a rate card, vBilling creates one contract per tenant and rate card. The rate card is the tenant's vbilling.vcluster.com/plan, or metronome.rateCard when the tenant has none. A UUID is used as the rate card ID, anything else as its alias.

+
    +
  • The contract starts at UTC midnight on the day vBilling creates it, with monthly usage statements on the first of the month.
    • +
  • Its uniqueness key is vbilling-<tenant>-<rate card>, so a retry or a second replica never creates a second contract.
    • +
  • Moving a tenant to another rate card creates a contract on the new one. vBilling never ends contracts, so end the old one in Metronome.
    • +
  • Without a rate card, vBilling creates the customer only and you manage its contracts in Metronome.
    • +
+ + +

With metronome.stripeLink=true, vBilling finds or creates the tenant's Stripe customer the same way the Stripe adapter does (by the metadata vbilling_tenant_id, or the customer pinned with vbilling.vcluster.com/stripe-customer-id). It then creates the Metronome customer with Stripe as its billing provider, delivery method direct_to_billing_provider and your collection method. Contracts carry the same billing provider. No Stripe meters or prices are created in this mode.

+
+
i
+
The link is set when vBilling creates the Metronome customer. A customer that already exists in Metronome, found by its alias or pinned by ID, is used as it is: link it to Stripe in Metronome yourself.
+
+ +

Pricing on dimensions

+

Add a usage product per billable metric to your rate card. A product prices on a pricing group key and splits invoice lines with a presentation group key. Metronome requires the two together to form one compound group key of the billable metric, and vBilling defines exactly one, so split its keys between them:

+
# Usage product on "GPU Hours [vcluster_gpu_hours]" +pricing_group_key: [region, sku, capacity_type, billing_mode] +presentation_group_key: [tenant_cluster] + +# Usage product on "Dedicated Node Hours [vcluster_private_node_hours]" +pricing_group_key: [region, sku, capacity_type] +presentation_group_key: [tenant_cluster]
+

Then give each combination you sell its own rate:

+
    +
  • GPU model: sku carries the node's SKU label or GPU model, so H100 and L40S hours get different rates.
    • +
  • Region: region is the vBilling instance's region (or each node's topology.kubernetes.io/region with REGION_FROM_NODE=true), so two regions can be priced differently.
    • +
  • Capacity type: capacity_type is on-demand, spot, preemptible or reserved. A discount is a lower rate; quantities stay physical.
    • +
  • Billing mode: billing_mode is shared, dedicated_node or private_node, so GPUs on whole nodes can carry their own rate.
    • +
+

Price whole nodes either by node-hours or by their GPU, CPU and memory hours, not both (see what gets billed). Group-key properties an event does not carry are sent as none, so every event still matches its metric. vBilling sends quantities only: the rate card sets prices and currency.

+ +

Delivery

+

vBilling posts events to /v1/ingest in batches of 100, one event per tenant, metric, window and set of dimensions. customer_id is the Metronome customer ID once vBilling knows it, and the ingest alias before that:

+
{ + "transaction_id": "vb1_7c4e9a0d5b...", + "customer_id": "vcluster-team-a-team-a", + "event_type": "vcluster_gpu_hours", + "timestamp": "2026-10-05T10:00:00Z", + "properties": { + "value": "0.133333333", "unit": "gpu-hours", + "window_start": "2026-10-05T10:00:00Z", "window_end": "2026-10-05T10:01:00Z", + "schema_version": "vbilling.usage/v1", "source": "collector", + "region": "us-east-1", "sku": "NVIDIA-H100-80GB-HBM3", "resource_id": "gpu-node-1", + "capacity_type": "on-demand", "billing_mode": "private_node", + "gpu_type": "NVIDIA-H100-80GB-HBM3", "gpu_profile": "full", + "tenant_cluster": "vcluster-team-a-team-a" + } +}
+
+ + + + + + + + +
CaseWhat happens
Retries and replaysMetronome ignores a repeated transaction_id for 34 days, so nothing counts twice. vBilling sends no Idempotency-Key, because Metronome would cache an error response under it.
Events older than 34 daysDead-lettered as older than Metronome's 34-day ingest window (vBilling keeps an hour of margin).
HTTP 400, 413, 422The batch is split until the rejected events are isolated. Those are dead-lettered; the rest are delivered.
HTTP 401, 403, 404, 408, 409, 429, 5xx, network errorsRetried with backoff, never dead-lettered. Only Metronome waits: other destinations keep delivering from their own cursors.
+
+

Check delivery, list dead letters and replay them (add your API token as a bearer token if you set one):

+
kubectl -n vbilling-system port-forward svc/vbilling 8080 +curl -s localhost:8080/api/v1/destinations | jq '.destinations[] | select(.name == "metronome")' +curl -s localhost:8080/api/v1/destinations/metronome/dead-letters | jq +curl -s -X POST localhost:8080/api/v1/destinations/metronome/dead-letters/replay
+

Adding Metronome to a running vBilling starts delivery at the next committed window. To send the retained ledger as well (persistence.retention, default 168h), rewind its cursor; transaction IDs keep the overlap from counting twice:

+
curl -s -X POST localhost:8080/api/v1/destinations/metronome/cursor -d '{"seq": 0}'
+ +

Reconciliation

+

GET /api/v1/reconcile?destination=metronome recomputes each tenant's usage from the ledger and compares it with what Metronome's usage API (/v1/usage) recorded. Metronome reports whole UTC days only, so from and to must be UTC midnights (YYYY-MM-DD works). Without them, the previous full UTC day is checked. Narrow it with tenant, metric, project, region, sku or cluster.

+
curl -s 'localhost:8080/api/v1/reconcile?destination=metronome&from=2026-10-04&to=2026-10-05' | jq +{ + "from": "2026-10-04T00:00:00Z", + "to": "2026-10-05T00:00:00Z", + "results": [{ + "destination": "metronome", + "ok": true, + "rows": [{ + "tenant": "vcluster-team-a-team-a", "metric": "vcluster_gpu_hours", + "ledger": 192, "dead_lettered": 0, "expected": 192, + "recorded": 192, "difference": 0, "match": true + }] + }] +}
+

expected is the ledger total minus dead letters, and a row matches within 0.1%. Reconcile once a day has closed and lag_records on /api/v1/destinations is back to 0, so no events are still in flight.

+ +

Webhooks and billing state

+

Create a webhook in Metronome pointing at https://<your vBilling host>/webhooks/metronome and add its secret to the Secret as webhook-secret. The Service is ClusterIP; expose only the /webhooks/ paths to the internet. Billing-state enforcement turns on when a webhook secret is set.

+
kubectl -n vbilling-system create secret generic vbilling-metronome \ + --from-file=api-token=./metronome-token --from-file=webhook-secret=./metronome-webhook-secret \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl -n vbilling-system rollout restart statefulset/vbilling
+

vBilling verifies Metronome-Webhook-Signature: hex HMAC-SHA256 with the secret over X-Metronome-Date (or Date), a newline and the raw body, within 5 minutes of that date. A bad signature gets a 400. Verified events that carry no billing signal, belong to a customer that is not a vBilling tenant, or repeat an event already applied get a 200 with "applied": false; other failures return a 500 so Metronome retries.

+
+ + + + + + + +
Metronome eventBilling state
alerts.*, for example alerts.spend_threshold_reached or alerts.low_remaining_contract_credit_balance_reachedwarning, with the alert's name as the reason
payment_gate.payment_status with payment_status: faileddelinquent
payment_gate.payment_status with payment_status: paidactive
+
+

Map any event type to another state with enforcement.rules (ENFORCEMENT_RULES). A hard spend cap suspends the tenant when its spend alert fires:

+
# values.yaml +enforcement: + mode: enforce + rules: "alerts.spend_threshold_reached=suspended" + admissionPolicy: + enabled: true
+
+ + + + + + + +
enforcement.modeEffect
observe (default)Records the state. It appears in GET /api/v1/billing-states, the metrics and the dashboard.
annotateAlso annotates the tenant cluster, labels its namespace vbilling.vcluster.com/billing-state and emits a Kubernetes event.
enforceAlso labels suspended namespaces vbilling.vcluster.com/suspended=true. With enforcement.admissionPolicy.enabled=true (Kubernetes 1.30+), a ValidatingAdmissionPolicy denies new pods in those namespaces of the control plane cluster. Running workloads keep running.
+
+

Metronome + Stripe: payment failures on invoices Stripe collects reach vBilling through Stripe's webhooks, not Metronome's, so point Stripe's webhooks at /webhooks/stripe as well and add webhook-secret to the vbilling-stripe Secret. vBilling resolves them through the linked Stripe customers. See Stripe webhooks. To override a state by hand, use PUT /api/v1/billing-states/<tenant> with {"state": "active", "reason": "..."}.

+ +

Troubleshooting

+

The pod exits at startup

+

initialize adapters: METRONOME_API_TOKEN is required when the metronome adapter is enabled means the token did not reach the pod. The chart mounts Secret keys as optional, so a wrong Secret name or key shows up here: the key must be api-token in the Secret named by metronome.existingSecret. METRONOME_STRIPE_LINK needs STRIPE_API_KEY to create Stripe customers is the same problem with the key api-key in stripe.existingSecret.

+

Delivery to Metronome is paused

+

[pipeline] metronome: bootstrap failed (delivery paused, will retry): list billable metrics: metronome GET /v1/billable-metrics: HTTP 401 means Metronome rejected the token. Usage keeps accumulating in the ledger and other destinations keep delivering. Fix the token, then restart vBilling, which reads Secrets at startup.

+

Tenants get no contract

+

Contract errors name the tenant and rate card: create contract for <tenant> on <rate card>. Check that the alias or ID exists in Metronome, and check the tenant's vbilling.vcluster.com/plan if it has one. When Metronome answers 400, the log reads [pipeline] metronome: tenant <tenant> rejected and usage keeps flowing to the customer without a contract; other errors pause delivery and retry.

+

Dead letters older than 34 days

+

older than Metronome's 34-day ingest window comes from a cursor rewind or an outage longer than Metronome accepts backdated events. Metronome cannot take them. They stay in the dead-letter list, and reconciliation reports them as dead_lettered.

+

Reconciliation fails

+

metronome reconciliation needs whole UTC days (got ... to ...) means from or to is not a UTC midnight. Pass dates as YYYY-MM-DD, or RFC 3339 times at 00:00:00Z.

+
+
+ + + + + + diff --git a/docs/private-nodes.html b/docs/private-nodes.html index 5988fdc..33a31da 100644 --- a/docs/private-nodes.html +++ b/docs/private-nodes.html @@ -4,7 +4,7 @@ Dedicated and Private Nodes - vBilling Docs - + @@ -96,7 +96,7 @@ .sidebar-back:hover { border-color: var(--primary); color: var(--primary-dark); } .content { - flex: 1; margin-left: var(--sidebar-width); + flex: 1; min-width: 0; margin-left: var(--sidebar-width); padding: 48px 64px 80px; max-width: 900px; } .content h1 { @@ -128,7 +128,7 @@ .content code { background: var(--primary-light-2); padding: 2px 7px; border-radius: 4px; font-family: 'Roboto Mono', 'SF Mono', 'Fira Code', monospace; - font-size: 0.85rem; color: var(--primary-darker); + font-size: 0.85rem; color: var(--primary-darker); overflow-wrap: anywhere; } .content a { color: var(--primary-dark); text-decoration: none; font-weight: 500; } .content a:hover { text-decoration: underline; } @@ -256,11 +256,20 @@ Example: GKE with GPUs Troubleshooting
+
+
Billing Adapters
+ Stripe + Metronome + Lago + Custom Billing Adapter +
Resources
GitHub Repository - Lago Docs vCluster Docs + Stripe Billing Docs + Metronome Docs + Lago Docs
@@ -269,10 +278,10 @@

Dedicated and Private Node Billing

How vBilling bills nodes that belong to one tenant cluster for their full capacity, whether they sit in the control plane cluster (dedicated nodes) or join the tenant cluster itself (vCluster Private Nodes, Auto Nodes, Standalone), without ever billing the workloads on them twice.

What dedicated nodes are

-

A dedicated node is a physical server, VM or bare-metal machine reserved for one tenant cluster. AI Clouds typically sell them by SKU (for example bm.gpu.h100.8), billing the whole machine regardless of utilization. Shared nodes, by contrast, are billed by what each pod allocates.

+

A dedicated node is a physical server, VM or bare-metal machine assigned to one tenant cluster. AI Clouds typically sell them by SKU (for example bm.gpu.h100.8), billing the whole machine regardless of utilization. Shared nodes, by contrast, are billed by what each pod allocates.

i
-
Dedicated nodes are control plane cluster nodes reserved through labels. With vCluster Private Nodes, nodes join the tenant cluster directly and the control plane cluster never sees them or their pods; see Private Nodes, Auto Nodes and Standalone.
+
Dedicated nodes are control plane cluster nodes assigned through labels. With vCluster Private Nodes, nodes join the tenant cluster directly and the control plane cluster never sees them or their pods; see Private Nodes, Auto Nodes and Standalone.

How vBilling detects them

diff --git a/docs/stripe.html b/docs/stripe.html new file mode 100644 index 0000000..8e4bcba --- /dev/null +++ b/docs/stripe.html @@ -0,0 +1,588 @@ + + + + + + Stripe - vBilling Docs + + + + + + + + + + +
+ + +
+

Stripe

+

Bill tenant clusters with Stripe Billing Meters. vBilling creates the meters and customers, sends every usage window exactly once, keeps subscriptions in step with the prices you tag, and turns Stripe's payment events into billing states for your tenant clusters.

+ +

How it works

+ 
tenant clusters ──► vBilling ──► ledger (fsynced, hash-chained)
+                                   │ stripe cursor
+                                   ▼
+                          Stripe adapter
+                          ├─ Billing Meters   one per metric, one per SKU
+                          ├─ Customers        one per tenant, metadata vbilling_tenant_id
+                          ├─ Meter events     identifier = event ID, timestamp = window start
+                          └─ Subscriptions    optional: prices tagged vbilling_plan
+
+Stripe ──► /webhooks/stripe ──► billing state ──► tenant cluster
+invoices, subscriptions,          active, warning,
+usage alerts                      delinquent, suspended
+
+
+

vBilling does

+

Meters, customers and meter events; retries and dead letters; optional subscriptions; reconciliation against Stripe; billing states from Stripe webhooks.

+
+
+

You do in Stripe

+

Products and prices, invoice and tax settings, payment methods, Smart Retries and dunning emails, and the customer portal.

+
+
+
+
i
+
A Stripe meter prices one quantity. To price several dimensions at once (region, SKU and capacity type), or to sell commits and credits, rate usage in Metronome and let Stripe collect payment.
+
+ +

Install in test mode

+

Start with a sandbox or test-mode key: a secret key (sk_test_...) or a restricted key (rk_test_...). Save it to a file and keep it in a Secret, never in --set flags or values files. From a clone of the repository:

+
kubectl create namespace vbilling-system +kubectl -n vbilling-system create secret generic vbilling-stripe \ + --from-file=api-key=./stripe-test-key +helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \ + --set adapters='{stripe}' \ + --set stripe.existingSecret=vbilling-stripe \ + --set region=us-east-1
+

vBilling creates one meter per billable metric at startup, and a customer for each tenant as its usage arrives. Check the log and the destination's health:

+
kubectl -n vbilling-system logs statefulset/vbilling | grep '\[stripe\]' +[stripe] created meter vcluster_gpu_hours (mtr_...) +[stripe] created customer cus_... for tenant vcluster-team-a-team-a +kubectl -n vbilling-system port-forward svc/vbilling 8080 & +curl -s localhost:8080/api/v1/status | jq '.destinations'
+

For CI, run stripe-mock and set stripe.apiBase to it: every request is validated against Stripe's OpenAPI spec, as in scripts/e2e-kind.sh.

+ +

Settings

+
+ + + + + + + + + + + + + + + +
Helm valueEnvironment variableDefaultEffect
stripe.existingSecretSecret with the keys api-key and, for webhooks, webhook-secret. Use it instead of the two values below.
stripe.apiKeySTRIPE_API_KEYSecret or restricted key. Required when stripe is in adapters.
stripe.webhookSecretSTRIPE_WEBHOOK_SECRETSigning secret (whsec_...). Enables /webhooks/stripe.
stripe.splitMetersBySTRIPE_SPLIT_METERS_BY[sku]Event fields that get their own meter. See Meters.
stripe.autoSubscribeSTRIPE_AUTO_SUBSCRIBEfalseSubscribe each tenant to the prices tagged with its plan.
stripe.cancelOnRemoveSTRIPE_CANCEL_ON_REMOVEfalseCancel an offboarded tenant's subscription at the end of the period.
stripe.maxRPSSTRIPE_MAX_RPS0Request rate cap. 0 picks 20 requests/s for test keys and 200 for live keys.
stripe.apiVersionSTRIPE_API_VERSION2026-09-30.endiveStripe API version sent on every request.
stripe.apiBaseSTRIPE_API_BASEhttps://api.stripe.comAPI base URL, for stripe-mock or a proxy.
billing.defaultPlanCodeDEFAULT_PLAN_CODEvcluster-standardPlan of tenants without a plan annotation.
billing.currencyBILLING_CURRENCYUSDCurrency of tenants without a currency annotation.
+
+

Restricted keys. A restricted key works if it can write customers and Billing Meters (meters and meter events) and read prices, plus write subscriptions when autoSubscribe or cancelOnRemove is on.

+ +

Meters

+

At startup vBilling creates, or reactivates, one meter per billable metric: sum aggregation, customers mapped by stripe_customer_id, the quantity in value. Informational metrics (vcluster_gpu_utilization, vcluster_gpu_downtime_hours) are never sent to Stripe.

+

Because a meter cannot price on dimensions, each event goes to a meter named after its metric plus the value of every field in splitMetersBy, created the first time that value appears. Values are lowercased and every other character becomes _; a name longer than Stripe's 100 characters ends in a short hash.

+
+ + + + + + + + +
MetricMeters that receive events (split on sku)SKU comes from
GPU hoursvcluster_gpu_hours__nvidia_h100_80gb_hbm3
vcluster_gpu_hours__nvidia_a100_sxm4_40gb_mig_1g_5gb
vcluster_gpu_hours__nvidia_l4_timeslice_4		The GPU model or the node's SKU label, plus the MIG profile or time-slicing replicas for fractional GPUs
Dedicated node hoursvcluster_private_node_hours__bm_gpu_h100_8The SKU label, the instance type, or <n>x<GPU model>
Storage GiB-hoursvcluster_storage_gb_hours__standard_rwoThe storage class
CPU core-hours, memory GiB-hours, tenant cluster hours, LoadBalancer hours, network egressvcluster_cpu_core_hours, vcluster_memory_gb_hours, and so onNo SKU: the base meter
+
+
+
!
+
Put prices on the meters that receive events. GPU usage lands on vcluster_gpu_hours__<sku>, so the base vcluster_gpu_hours meter stays empty and a price on it invoices nothing.
+
+
STRIPE_KEY=$(cat ./stripe-test-key) +curl -s -G https://api.stripe.com/v1/billing/meters -u "$STRIPE_KEY:" -d limit=100 | jq -r '.data[] | "\(.id) \(.event_name)"'
+ +

Pricing on more dimensions

+

Add fields to the split to give them their own meters and prices. Any event field works: sku, region, project, or a dimension such as capacity_type, billing_mode or tenant_class. Suffixes follow the order of the list:

+
--set stripe.splitMetersBy='{sku,capacity_type}' +# vcluster_gpu_hours__nvidia_h100_80gb_hbm3__on_demand +# vcluster_gpu_hours__nvidia_h100_80gb_hbm3__spot
+

Every field multiplies the meters and prices you manage. Settle the split before you go live: a new split sends new usage to new meters, and earlier usage stays on the old ones.

+ +

Customers

+

Each vBilling tenant gets one Stripe customer. By default a tenant is one tenant cluster, with the ID vcluster-<namespace>-<name>; the vbilling.vcluster.com/tenant key groups several tenant clusters under one customer, and TENANT_SOURCE=project gives each vCluster Platform project one. vBilling finds the customer in this order:

+
    +
  1. A customer ID pinned with vbilling.vcluster.com/stripe-customer-id.
    2. +
  2. The customer vBilling recorded earlier in state-stripe.json on its volume.
    3. +
  3. A Stripe customer whose metadata[vbilling_tenant_id] is exactly the tenant ID. vBilling uses customer search, and lists customers instead (by email first) for accounts without search.
    4. +
  4. A new customer, created with an idempotency key so that a retry never creates a second one.
    5. +
+

Customer details come from labels or annotations on the tenant cluster's StatefulSet, its namespace, or its vCluster Platform VirtualClusterInstance, and are updated when they change:

+
kubectl annotate namespace team-a \ + vbilling.vcluster.com/tenant=acme-ai \ + vbilling.vcluster.com/display-name="Acme AI" \ + vbilling.vcluster.com/email=billing@acme-ai.example \ + vbilling.vcluster.com/plan=vcluster-standard + +# reuse a customer that already exists in Stripe +kubectl annotate namespace team-a vbilling.vcluster.com/stripe-customer-id=cus_EXISTING
+
+ + + + + + + + + +
Customer fieldValue
name, emailThe display-name (or the tenant ID) and email keys
metadata[vbilling_tenant_id]The tenant ID. vBilling only uses a customer whose value matches exactly.
metadata[vbilling_plan]The tenant's plan
metadata[vbilling_region], [vbilling_project], [vbilling_tenant_class]The region, vCluster Platform project and tenant class
metadata[vbilling_clusters]The tenant clusters billed to this customer
+
+

vBilling never deletes customers. Do not delete them in Stripe either: to move a tenant to another customer, pin it.

+ +

Prices and subscriptions

+

vBilling never creates prices: what a GPU-hour costs is your decision. Create a recurring, metered price on each meter you charge for. Quantities are hours (GPU-hours, node-hours, GiB-hours), so unit prices are per hour. Fractional GPUs are already counted in GPU-equivalents (a 1g.5gb slice of an A100 is 1/7), so one per-GPU-hour price fits every variant of a SKU, or the slices can carry a premium.

+
METER=$(curl -s -G https://api.stripe.com/v1/billing/meters -u "$STRIPE_KEY:" -d limit=100 \ + | jq -r '.data[] | select(.event_name=="vcluster_gpu_hours__nvidia_h100_80gb_hbm3") | .id') +curl -s https://api.stripe.com/v1/prices -u "$STRIPE_KEY:" \ + -d currency=usd -d unit_amount_decimal=250 \ + -d "recurring[interval]=month" -d "recurring[usage_type]=metered" \ + -d "recurring[meter]=$METER" \ + -d "product_data[name]=H100 GPU-hours" \ + -d "metadata[vbilling_plan]=vcluster-standard"
+

unit_amount_decimal is in cents: 250 is $2.50 per GPU-hour.

+ +

Automatic subscriptions

+

Without autoSubscribe, usage accrues on the meters and is invoiced once you subscribe the customer to prices yourself. With --set stripe.autoSubscribe=true, vBilling subscribes every tenant to the prices of its plan:

+
    +
  • The plan is the tenant's vbilling.vcluster.com/plan, or billing.defaultPlanCode (vcluster-standard).
    • +
  • A plan's prices are the active metered prices on vBilling's meters tagged metadata[vbilling_plan]=<plan>, in the tenant's currency (vbilling.vcluster.com/currency, or billing.currency).
    • +
  • A plan holds up to 20 prices, Stripe's limit per subscription.
    • +
  • An existing vBilling subscription gets missing prices added. Prices are never removed from it.
    • +
  • New prices reach new tenants within five minutes, and existing tenants when vBilling restarts or their labels and annotations change.
    • +
+

Plans let each tenant pay for what it buys. A tenant that rents whole GPU machines (dedicated or private nodes) can be on a plan with node-hour prices only, and a tenant that pays per GPU-hour on a plan with GPU-hour prices. Stripe only invoices a customer for the meters its subscription has prices on.

+ +

Invoices

+

Stripe sums each meter over the billing period and adds a line per price to the subscription's invoice; payment collection, tax, retries and dunning stay in Stripe. To see what a tenant owes so far, open the customer's upcoming invoice in the Dashboard, or preview it:

+
curl -s https://api.stripe.com/v1/invoices/create_preview -u "$STRIPE_KEY:" -d customer=cus_... \ + | jq '.lines.data[] | {description, amount}'
+

Stripe processes meter events asynchronously, so new usage can take a few minutes to show up.

+ +

Delivery and idempotency

+

Every closed window is committed to the ledger first. The Stripe destination sends it from its own cursor, so a Stripe outage never loses usage and never holds up other destinations.

+
+ + + + + + + + + + +
ConcernWhat vBilling does
DuplicatesThe meter event identifier is the vBilling event ID: a hash of the event's identity (tenant, metric, window, dimensions), never of its quantity. A resend after a crash is rejected by Stripe as a duplicate and counted as delivered.
Event timeThe meter event timestamp is the window start, so late deliveries and backfills land in the period they belong to.
PrecisionQuantities are plain decimals with at most 15 significant digits and 9 decimals.
RateA token bucket at maxRPS, and four workers partitioned by customer and meter, so a customer's meter never gets concurrent calls.
RetriesAuthentication errors (401, 403), 404, conflicts (409), rate limits (429), 5xx and network errors are retried with backoff and never dead-lettered. Stripe's Stripe-Should-Retry header wins when present.
Dead lettersEvents Stripe rejects as invalid (400, 413, 422) are isolated one by one and parked with the reason; the rest of the batch goes through.
+
+
+
!
+
Stripe enforces identifier uniqueness for at least 24 hours and accepts events up to 35 days old. vBilling never resends acknowledged events, and dead-letters events older than 35 days minus an hour of margin. Do not rewind the Stripe destination's cursor by more than 24 hours, or Stripe may count those events again; rewinding is meant for a newly added destination.
+
+

After fixing what caused a rejection (often by upgrading vBilling), replay the parked events. Events Stripe still rejects stay parked (still_rejected). With API_TOKEN set, add -H "Authorization: Bearer $API_TOKEN".

+
curl -s localhost:8080/api/v1/destinations/stripe/dead-letters | jq +curl -s -X POST localhost:8080/api/v1/destinations/stripe/dead-letters/replay +{"replayed":12}
+ +

Reconciliation

+

GET /api/v1/reconcile recomputes each tenant's usage per metric from the ledger, subtracts what was dead-lettered, and compares it with Stripe's meter event summaries for the tenant's customer, summed over every split meter of the metric. A row matches within 0.1%. The default range is the previous full UTC day; from and to take RFC 3339 times or dates, and tenant and metric narrow it down.

+
curl -s 'localhost:8080/api/v1/reconcile?destination=stripe&from=2026-10-05&to=2026-10-06' | jq +{ + "from": "2026-10-05T00:00:00Z", + "to": "2026-10-06T00:00:00Z", + "results": [{ + "destination": "stripe", + "ok": true, + "rows": [ + {"tenant": "acme-ai", "metric": "vcluster_gpu_hours", "ledger": 2.75, "dead_lettered": 0, + "expected": 2.75, "recorded": 2.75, "difference": 0, "match": true} + ] + }] +}
+

Stripe's summaries are aligned to minutes and trail ingestion by a few minutes, so reconcile closed periods rather than the current hour. Run it daily and alert when ok is false.

+ +

Webhooks and billing states

+

Stripe's payment events tell vBilling which tenants are paid up. Add a webhook endpoint in Stripe for https://<your-host>/webhooks/stripe with the events below, then add its signing secret to the Secret and restart vBilling:

+
+ + + + + + + + + +
Stripe eventBilling state
invoice.payment_faileddelinquent
invoice.paid, invoice.payment_succeededactive
customer.subscription.created, .updated, .resumedBy subscription status: past_due is delinquent; unpaid, canceled and incomplete_expired are suspended; active and trialing are active
customer.subscription.deletedsuspended
billing.alert.triggered (a usage alert on a meter)warning
+
+
kubectl -n vbilling-system create secret generic vbilling-stripe \ + --from-file=api-key=./stripe-test-key --from-file=webhook-secret=./stripe-webhook-secret \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl -n vbilling-system rollout restart statefulset/vbilling
+

vBilling verifies the Stripe-Signature header (HMAC-SHA256 over the timestamp and the body, with the five-minute tolerance of Stripe's libraries) and applies each event once. It answers 400 to a bad signature, 200 with "applied": false for events it ignores, repeats, or customers it does not manage, and 500 when it could not record the state, so that Stripe retries.

+

Only the webhook path needs to be reachable from Stripe. Keep /api on the internal Service, for example with an Ingress for that one path:

+
apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: vbilling-webhooks + namespace: vbilling-system +spec: + ingressClassName: nginx + tls: [{hosts: [billing-hooks.example.com], secretName: billing-hooks-tls}] + rules: + - host: billing-hooks.example.com + http: + paths: + - path: /webhooks/stripe + pathType: Exact + backend: + service: {name: vbilling, port: {number: 8080}}
+

To try it locally, let the Stripe CLI forward events. It prints a signing secret for the session to use as webhook-secret:

+
stripe listen --forward-to localhost:8080/webhooks/stripe
+

Events fired with stripe trigger belong to new test customers that vBilling does not manage, so they come back with "applied": false. To see a state change, act on one of vBilling's customers, for example by paying its invoice with one of Stripe's declining test cards.

+ +

What a billing state does

+

Billing states are on once webhook-secret is set. They are recorded in every mode and shown by GET /api/v1/billing-states, the metrics and the dashboard. enforcement.mode decides what else happens:

+
+ + + + + + + + +
ModeEffect
observe (default)Record the state only.
annotateAlso annotate the tenant cluster, label its namespace vbilling.vcluster.com/billing-state, and emit a Kubernetes event.
enforceAlso label suspended namespaces vbilling.vcluster.com/suspended=true. With enforcement.admissionPolicy.enabled=true (Kubernetes 1.30+), new pods are denied there while running workloads keep running. A paid invoice lifts it.
The admission policy acts in the control plane cluster, so it holds tenant clusters on shared or dedicated nodes. Tenant clusters with their own nodes (Private Nodes, Auto Nodes, Standalone) run their pods only inside the tenant cluster: act on their billing state with your own automation.
+
+

Override a mapping with enforcement.rules (ENFORCEMENT_RULES), for example invoice.payment_failed=suspended to suspend on the first failed payment, or billing.alert.triggered=suspended to turn a Stripe usage alert into a hard cap. Operators can also set a state by hand:

+
curl -s -X PUT localhost:8080/api/v1/billing-states/acme-ai -d '{"state": "active", "reason": "paid by bank transfer"}'
+ +

Offboarding

+

When a tenant has been gone for billing.offboardGrace (default 1h), vBilling offboards it. The Stripe customer and its subscription stay, so final usage is still invoiced. With stripe.cancelOnRemove=true, the subscription is set to cancel at the end of the period.

+ +

Stripe limits

+
+ + + + + + + + + + + +
Stripe limitWhat vBilling does
Meter events at most 35 days oldOlder events are dead-lettered as older than Stripe's 35-day meter event window. Normal backfills (billing.maxBackfill, default 6h) stay far inside it.
Identifiers unique for at least 24 hoursAcknowledged events are never resent. Keep cursor rewinds within 24 hours.
Event names up to 100 charactersLonger split meter names end in a hash.
20 prices per subscriptionAutomatic subscriptions refuse a plan with more than 20 prices: split it into several plans.
Request rate (sandboxes share 25 requests/s)20 requests/s for test keys and 200 for live keys, adjustable with maxRPS. 429s are retried.
No pricing on dimensions within a meterMeters are split by splitMetersBy. For pricing on many dimensions, use Metronome.
Customer search is not offered to every accountvBilling lists customers instead, still requiring an exact metadata[vbilling_tenant_id] match.
+
+ +

Troubleshooting

+

The invoice shows no usage

+
    +
  • The price is on a meter that receives nothing, usually the base meter instead of the SKU meter. List the meters and compare with the event names in the log.
    • +
  • The tenant is not subscribed. With autoSubscribe, the log says [stripe] no active metered prices tagged vbilling_plan=<plan> in <currency>; tenant <id> not subscribed: check that the price is active and metered, carries the plan in its metadata, and is in the tenant's currency.
    • +
  • The usage is still being processed: Stripe aggregates meter events asynchronously.
    • +
+

Deliveries stop with 401 or 403

+

The key is wrong or revoked, or a restricted key lacks a permission. Nothing is lost: events wait in the ledger and are retried. Fix the Secret and restart vBilling; /api/v1/status shows each destination's failures and its oldest pending event.

+

Events dead-lettered as older than 35 days

+

Stripe no longer accepts them. They stay in the ledger, and reconciliation reports them under dead_lettered. Bill them with a one-off invoice item if needed.

+

Webhooks fail with 400 or 404

+

400: the signing secret does not match the endpoint (every endpoint, and every stripe listen session, has its own), the clock is more than five minutes off, or a proxy rewrote the body. 404: webhook-secret is not set.

+

"customer search unavailable"

+

[stripe] customer search unavailable (...); finding customers by listing them is expected for accounts without customer search. vBilling switches to listing customers on its own.

+ +

See also

+ +
+
+ + + + + + diff --git a/docs/troubleshooting.html b/docs/troubleshooting.html index 7eeee02..2428251 100644 --- a/docs/troubleshooting.html +++ b/docs/troubleshooting.html @@ -4,7 +4,7 @@ Troubleshooting - vBilling Docs - + @@ -96,7 +96,7 @@ .sidebar-back:hover { border-color: var(--primary); color: var(--primary-dark); } .content { - flex: 1; margin-left: var(--sidebar-width); + flex: 1; min-width: 0; margin-left: var(--sidebar-width); padding: 48px 64px 80px; max-width: 900px; } .content h1 { @@ -125,7 +125,7 @@ .content code { background: var(--primary-light-2); padding: 2px 7px; border-radius: 4px; font-family: 'Roboto Mono', 'SF Mono', 'Fira Code', monospace; - font-size: 0.85rem; color: var(--primary-darker); + font-size: 0.85rem; color: var(--primary-darker); overflow-wrap: anywhere; } .content a { color: var(--primary-dark); text-decoration: none; font-weight: 500; } .content a:hover { text-decoration: underline; } @@ -231,11 +231,20 @@ Example: GKE with GPUs Troubleshooting
+
+
Billing Adapters
+ Stripe + Metronome + Lago + Custom Billing Adapter +
Resources
GitHub Repository - Lago Docs vCluster Docs + Stripe Billing Docs + Metronome Docs + Lago Docs
@@ -247,11 +256,11 @@

Troubleshooting

No Tenant Clusters Discovered

-

"[discovery] found 0 vCluster(s)"

+

"[controller] window ...: 0 events for 0 tenant cluster(s)"

Symptoms

-

vBilling starts but reports zero tenant clusters. No customers or events appear in Lago. The controller log shows [discovery] found 0 vCluster(s) on every reconcile cycle.

+

vBilling starts but meters no tenant clusters. No customers or events appear in your billing platform, and every window logs [controller] window ...: 0 events for 0 tenant cluster(s).

Cause

vBilling discovers tenant clusters by scanning for StatefulSets and Deployments with the label app=vcluster. If your tenant clusters use different labels, or if RBAC prevents listing these resources, discovery will find nothing.

@@ -277,10 +286,10 @@

Fix Check RBAC permissions

# If RBAC is the issue, check the ClusterRole kubectl get clusterrolebinding | grep vbilling
-

Fix Verify Platform API access (optional)

-

If using vCluster Platform, vBilling tries the Platform API first (management.loft.sh/v1 VirtualClusterInstance resources). If this CRD does not exist, vBilling falls back to workload scanning. Check the logs for:

-
[discovery] Platform API not available, falling back to StatefulSet scanning: ...
-

This is normal if you are not running vCluster Platform -- the fallback to StatefulSet scanning is the expected behavior.

+

Fix Check vCluster Platform enrichment (optional)

+

Workloads are the source of truth: vBilling finds tenant clusters by their StatefulSets and Deployments. With vCluster Platform, it also reads VirtualClusterInstance resources (management.loft.sh/v1) to add the project and display name, and logs:

+
[discovery] enriched 3 tenant cluster(s) from vCluster Platform
+

Without vCluster Platform this line never appears. That is expected: discovery does not depend on it.

Fix Check namespace filter

# If WATCH_NAMESPACES is set, only those namespaces are scanned @@ -377,7 +386,7 @@

Cause

The most common cause is a label value mismatch between the node label and the vCluster namespace name. vBilling tries multiple label values, but they must correspond to the namespace.

Fix Understand the label matching logic

-

For a vCluster in namespace vcluster-team-gpu, vBilling searches for nodes with these selectors:

+

For a tenant cluster in namespace vcluster-team-gpu, vBilling searches for nodes with these selectors:

# Selector 1: full namespace name vcluster.loft.sh/managed-by=vcluster-team-gpu vcluster.loft.sh/cluster=vcluster-team-gpu @@ -428,7 +437,7 @@

Fix Export the read-only kubeconfig

kubectl -n <tenant-namespace> get secret vbilling-kubeconfig

Fix The tenant cluster's control plane exits

-

you are trying to use a vCluster pro feature 'Private Nodes' (vcp-distro-private-nodes) that is not available means the tenant cluster has no license: register it with vCluster Platform (vcluster platform add vcluster <name> -n <namespace>). denied by loft access key scope means the CLI's access key is scoped and cannot register tenant clusters; log in with a key that can. vCluster 0.36.2 and later need Platform 4.11.3 or later.

+

An error naming the feature 'Private Nodes' (vcp-distro-private-nodes) as not available means the tenant cluster has no license: register it with vCluster Platform (vcluster platform add vcluster <name> -n <namespace>). denied by loft access key scope means the CLI's access key is scoped and cannot register tenant clusters; log in with a key that can. vCluster 0.36.2 and later need Platform 4.11.3 or later.

Fix GPU-hours missing right after a node reboot

While a driver container rebuilds (after a reboot, a spot restart or a MIG change), a Ready node advertises no GPUs. vBilling bills the node and records its installed GPUs (GPU Feature Discovery's nvidia.com/gpu.count) as vcluster_gpu_downtime_hours with the reason "GPUs installed but not allocatable" until they are allocatable again.

@@ -503,7 +512,7 @@

Symptoms

vBilling is successfully sending events (you can see them in the Lago UI under the customer's events tab), but the current usage and invoices all show $0.00.

Cause

-

When vBilling bootstraps the vcluster-standard plan, all charge amounts default to $0.00 per unit. This is by design -- vBilling creates the plan structure, but you must configure the actual pricing.

+

When vBilling bootstraps the vcluster-standard plan, all charge amounts default to $0.00 per unit. This is by design: vBilling creates the plan structure, and the prices are yours to set.

Fix Configure pricing in Lago

    @@ -533,7 +542,7 @@

    Symptoms

    After restarting the vBilling controller, you see warnings about subscription creation failing, or duplicate subscriptions appear in Lago.

    Cause

    -

    vBilling is designed to be idempotent on restart. When it discovers a vCluster, it first probes for an existing subscription using the GetCurrentUsage API call. If the subscription already exists, Lago returns usage data and vBilling reuses the subscription. If the probe fails (404), vBilling creates a new subscription.

    +

    vBilling is designed to be idempotent on restart. When it discovers a tenant cluster, it first probes for an existing subscription using the GetCurrentUsage API call. If the subscription already exists, Lago returns usage data and vBilling reuses the subscription. If the probe fails (404), vBilling creates a new subscription.

    How it works

    # vBilling's subscription detection logic: @@ -599,11 +608,13 @@

    General Debugging Tips

    Enable verbose logging: vBilling logs to stdout with the [component] prefix convention. Key prefixes to filter:
      -
    • [discovery] -- vCluster discovery events
      • -
    • [controller] -- Reconciliation and event sending
      • -
    • [metrics] -- Resource collection (CPU, memory, GPU, etc.)
      • -
    • [bootstrap] -- Lago metric and plan setup
      • -
    • [lago] -- Raw API calls to Lago
      • +
    • [controller]: metering windows, backfills and tenant changes
      • +
    • [discovery]: vCluster Platform enrichment
      • +
    • [metrics]: collection (CPU, memory, GPUs, Prometheus)
      • +
    • [tenantapi]: tenant clusters read through their own API (Private Nodes, Standalone)
      • +
    • [pipeline]: delivery to each adapter, retries and dead letters
      • +
    • [stripe], [metronome], [lago], [bootstrap], [noop]: the billing adapters
      • +
    • [enforcement], [api]: billing states, webhooks and the HTTP API
    diff --git a/examples/gke-gpu/01-prometheus.yaml b/examples/gke-gpu/01-prometheus.yaml index aebf877..ace6eff 100644 --- a/examples/gke-gpu/01-prometheus.yaml +++ b/examples/gke-gpu/01-prometheus.yaml @@ -1,5 +1,5 @@ # Central Prometheus for the control plane cluster ("bring your own Prometheus"). -# - Scrapes GKE's managed DCGM exporter on the shared GPU nodes. Its target +# - Scrapes GKE's managed DCGM exporter on the control plane cluster's GPU nodes. Its target # labels win, so DCGM's namespace/pod labels become exported_namespace/exported_pod. # - Accepts remote writes from tenant clusters with private nodes, whose DCGM # never reaches this cluster otherwise (see 06-prometheus-agent.yaml). diff --git a/scripts/demo.sh b/scripts/demo.sh index 0545e92..fa5a768 100755 --- a/scripts/demo.sh +++ b/scripts/demo.sh @@ -5,7 +5,7 @@ set -euo pipefail # Sets up a complete demo environment using vind (vCluster in Docker): # - vind cluster as the "host" Kubernetes cluster (no kind needed!) # - Lago (billing engine) via docker-compose -# - Nested vClusters inside the vind cluster +# - Tenant clusters nested inside the vind cluster # - vBilling controller metering everything SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -26,8 +26,8 @@ for cmd in docker kubectl vcluster; do fi done -# --- Step 1: Create vind cluster (host cluster via Docker) --- -echo ">>> Step 1: Creating vind host cluster '$VIND_CLUSTER'..." +# --- Step 1: Create vind cluster (control plane cluster via Docker) --- +echo ">>> Step 1: Creating vind control plane cluster '$VIND_CLUSTER'..." echo " This replaces kind - a full K8s cluster runs in Docker via vCluster." echo "" @@ -54,7 +54,7 @@ EOF fi echo "" -echo " vind host cluster is ready!" +echo " vind control plane cluster is ready!" kubectl cluster-info 2>/dev/null || true kubectl get nodes 2>/dev/null || true echo "" @@ -199,12 +199,12 @@ echo " For this demo, you can set it via:" echo " export LAGO_API_KEY=" echo "" -# --- Step 5: Switch to kubernetes driver and create nested vClusters --- -echo ">>> Step 5: Creating nested vClusters inside the vind host cluster..." -echo " These are vClusters-inside-vCluster - billing targets for vBilling." +# --- Step 5: Switch to kubernetes driver and create nested tenant clusters --- +echo ">>> Step 5: Creating tenant clusters inside the vind control plane cluster..." +echo " These tenant clusters run inside vind: the billing targets for vBilling." echo "" -# Switch driver back to kubernetes for creating nested vClusters +# Switch driver back to kubernetes for creating nested tenant clusters vcluster use driver kubernetes 2>/dev/null || true # Team Alpha - a development team @@ -226,14 +226,14 @@ vcluster create team-gpu \ --connect=false 2>/dev/null || echo " team-gpu may already exist" echo "" -echo " Nested vClusters created inside vind host:" +echo " Tenant clusters created inside the vind control plane cluster:" kubectl get statefulsets -A -l app=vcluster 2>/dev/null || true echo "" # --- Step 6: Deploy some workloads to generate metrics --- echo ">>> Step 6: Deploying sample workloads for billing..." -# Deploy a simple workload directly in each vCluster's host namespace +# Deploy a simple workload directly in each tenant cluster's namespace of the control plane cluster # so metrics-server picks it up immediately for ns in vcluster-team-alpha vcluster-team-beta vcluster-team-gpu; do kubectl create namespace "$ns" 2>/dev/null || true @@ -277,7 +277,7 @@ echo "" echo " Architecture:" echo "" echo " Docker" -echo " ├── vind host cluster ($VIND_CLUSTER)" +echo " ├── vind control plane cluster ($VIND_CLUSTER)" echo " │ ├── vCluster: team-alpha (web workloads)" echo " │ ├── vCluster: team-beta (data processing)" echo " │ └── vCluster: team-gpu (ML training)" @@ -298,7 +298,7 @@ echo " export LAGO_API_URL=http://localhost:3000" echo " ./bin/vbilling" echo "" echo "3. Watch the magic:" -echo " - vBilling discovers 3 vClusters automatically" +echo " - vBilling discovers 3 tenant clusters automatically" echo " - Creates billing customers in Lago" echo " - Meters CPU, memory, storage every 60s" echo " - Check Lago UI > Customers for live billing data"