Per-container usage metrics: OTLP collector + live per-owner report (#440) - #452
Open
cmyers-mieweb wants to merge 6 commits into
Open
Per-container usage metrics: OTLP collector + live per-owner report (#440)#452cmyers-mieweb wants to merge 6 commits into
cmyers-mieweb wants to merge 6 commits into
Conversation
usage-collector.js is a third long-running service (beside server.js and
job-runner.js) that polls one clusterResources('lxc') call per Proxmox
cluster and exports per-container resource metrics via OTLP http/protobuf
using the standard OTEL_* environment variables. It is a no-op (exits 0)
unless OTEL_EXPORTER_OTLP_ENDPOINT is configured, so the observability
backend stays out of this repo.
Metrics (all attributed with container.id, site.id, proxmox.node, owner,
container.name, container.status):
- container.cpu.usage / container.cpu.limit ({cpu})
- container.memory.usage / container.memory.limit (By)
- container.disk.usage / container.disk.limit (By)
- container.disk.io (By, disk.io.direction=read|write)
- container.network.io (By, network.io.direction=receive|transmit)
- container.uptime (s)
Owner attribution uses the Proxmox owner tag (written at container
creation) as the source of truth, cross-checked against the manager DB;
drift and unattributed containers are logged. Pure mapping logic lives in
utils/usage-sample.js with unit tests. DummyApi now returns simulated
usage numbers and owner tags so the collector works in dev.
Ships with a systemd unit (usage-collector.service), Makefile/postinstall
wiring, an npm script, and a 'make dev' process.
GET /api/v1/sites/:siteId/usage computes the per-owner report from the same one-call-per-cluster sampling cycle the OTLP collector uses, extracted into utils/usage-collection.js so both consumers see identical data. Aggregation (utils/usage-report.js) is pure and unit-tested; every metric shows allocated alongside used, never used alone. Visibility mirrors the containers list: admins see every owner on the site (plus attribution findings — tag/DB drift, unattributed containers, unregistered cluster nodes); other users see their own containers plus ones shared with them. Client: /sites/:siteId/usage page (sidebar link for all users) renders a DataVis grid of owners — each row expands to per-container detail — with an admin-only attribution warning banner. Byte formatting moved to lib/format.ts, shared with ResourceBar.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds per-container usage accounting that can be aggregated by owner, with two delivery paths: (1) an OTLP metrics exporter service (usage-collector.js) for external observability backends, and (2) a live, on-demand per-owner usage report exposed via a new API endpoint and UI page—without persisting time-series data in the manager DB.
Changes:
- Introduces shared usage sampling/aggregation utilities plus unit tests (
utils/usage-sample.js,utils/usage-report.js,utils/usage-collection.js). - Adds OTLP metrics collector service (
usage-collector.js), wiring for dev/prod (Makefile, systemd unit, packaging scripts, env docs). - Adds live usage report API + UI (new
/api/v1/sites/:siteId/usageendpoint, “Usage” page, grid + detail table, client types/queries).
Reviewed changes
Copilot reviewed 29 out of 30 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| mie-opensource-landing/docs/developers/release-pipeline.md | Documents the new usage-collector systemd unit being shipped/enabled. |
| mie-opensource-landing/docs/developers/development-workflow.md | Updates dev workflow docs to include the usage collector process. |
| create-a-container/utils/usage-sample.js | Pure parsing/mapping + owner attribution for a single /cluster/resources row. |
| create-a-container/utils/usage-report.js | Pure per-owner aggregation (sums + sorting) for the live report. |
| create-a-container/utils/usage-collection.js | Shared “one cycle” collector that polls Proxmox/Dummy/Docker and produces normalized samples + findings. |
| create-a-container/utils/dummy-api.js | Extends DummyApi to emit tags + simulated usage fields for dev. |
| create-a-container/utils/tests/usage-sample.test.js | Unit tests for tag parsing, cpu conversion, and sample building + drift/unattributed cases. |
| create-a-container/utils/tests/usage-report.test.js | Unit tests for per-owner aggregation behavior and sorting. |
| create-a-container/usage-collector.js | New long-running OTLP metrics exporter using OpenTelemetry metrics SDK. |
| create-a-container/routers/api/v1/usage.js | New API endpoint GET /api/v1/sites/:siteId/usage for live per-owner report. |
| create-a-container/routers/api/v1/sites.js | Mounts the new /usage nested router under sites. |
| create-a-container/README.md | Updates local dev + packaging/runtime docs to include the usage collector and usage report route. |
| create-a-container/package.json | Adds usage-collector script + OpenTelemetry dependencies. |
| create-a-container/package-lock.json | Locks the new OpenTelemetry dependency set. |
| create-a-container/openapi.v1.yaml | Adds OpenAPI path + tag for the new usage endpoint. |
| create-a-container/Makefile | Runs usage collector in make dev; ships usage-collector.js in packages; installs new systemd unit. |
| create-a-container/example.env | Documents OTLP env vars + collector interval knob. |
| create-a-container/contrib/systemd/usage-collector.service | New systemd unit to run usage-collector.js. |
| create-a-container/contrib/preremove.sh | Disables usage-collector.service on real removal. |
| create-a-container/contrib/postinstall.sh | Enables usage-collector.service post-install (alongside existing units). |
| create-a-container/client/src/pages/usage/UsagePage.tsx | New UI page that polls the usage API and renders warnings + grid. |
| create-a-container/client/src/lib/types.ts | Adds client types for usage report payload (owners/containers/findings). |
| create-a-container/client/src/lib/queries.ts | Adds query key + API call wrapper for /usage. |
| create-a-container/client/src/lib/format.ts | Centralizes formatBytes for shared usage and node views. |
| create-a-container/client/src/components/usage/UsageDataGrid.tsx | Owner-level DataVis grid with expandable per-container detail. |
| create-a-container/client/src/components/usage/OwnerContainersTable.tsx | Per-container detail table for an owner row expansion. |
| create-a-container/client/src/components/usage/AttributionWarnings.tsx | Admin-only warning banner for drift/unattributed/unknown-node findings. |
| create-a-container/client/src/components/nodes/ResourceBar.tsx | Refactors to reuse shared formatBytes. |
| create-a-container/client/src/app/Sidebar.tsx | Adds “Usage” link to the site navigation. |
| create-a-container/client/src/app/router.tsx | Adds route for /sites/:siteId/usage. |
Files not reviewed (1)
- create-a-container/package-lock.json: Generated file
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+14
to
+41
| const { Site, ContainerCollaborator } = require('../../../models'); | ||
| const { apiAuth, asyncHandler, ok, ApiError } = require('../../../middlewares/api'); | ||
| const { collectUsage } = require('../../../utils/usage-collection'); | ||
| const { aggregateByOwner } = require('../../../utils/usage-report'); | ||
|
|
||
| const router = express.Router({ mergeParams: true }); | ||
|
|
||
| router.use(apiAuth); | ||
|
|
||
| router.get( | ||
| '/', | ||
| asyncHandler(async (req, res) => { | ||
| const site = await Site.findByPk(parseInt(req.params.siteId, 10)); | ||
| if (!site) throw new ApiError(404, 'site_not_found', 'Site not found'); | ||
|
|
||
| const { samples, findings, unknownNodeRows } = await collectUsage({ siteId: site.id }); | ||
|
|
||
| let visible = samples; | ||
| if (!req.session.isAdmin) { | ||
| const shared = await ContainerCollaborator.findAll({ | ||
| where: { username: req.session.user }, | ||
| attributes: ['containerId'], | ||
| }); | ||
| const sharedIds = new Set(shared.map((row) => row.containerId)); | ||
| visible = samples.filter( | ||
| (s) => s.owner === req.session.user || (s.containerDbId && sharedIds.has(s.containerDbId)), | ||
| ); | ||
| } |
Collaborator
Author
There was a problem hiding this comment.
Resolved unintentionally by making the usages tab administrator only
Comment on lines
+49
to
+66
| async function collectUsage({ siteId = null } = {}) { | ||
| const nodeWhere = db.Node.provisionableWhere(); | ||
| if (siteId != null) nodeWhere.siteId = siteId; | ||
| const nodes = await db.Node.findAll({ where: nodeWhere }); | ||
| if (nodes.length === 0) return { samples: [], findings: [], unknownNodeRows: 0 }; | ||
|
|
||
| const containerIndex = await loadContainerIndex(siteId); | ||
|
|
||
| // Node lookup by `${siteId}:${name}` — cluster-resources rows carry the | ||
| // Proxmox node name, and node names are only meaningful within a site. | ||
| const nodesByName = new Map(nodes.map((n) => [`${n.siteId}:${n.name}`, n])); | ||
| const covered = new Set(); | ||
|
|
||
| const samples = []; | ||
| const findings = []; | ||
| let unknownNodeRows = 0; | ||
|
|
||
| for (const node of nodes) { |
The Usage page now leads with cluster-wide stacked bars — memory and CPU used vs physical capacity, one colored segment per owner (stable colors across bars, top consumers named, remainder bucketed as 'others', red tick marks capacity on over-commit). Capacity comes from the node rows of the same /cluster/resources call family (2 bulk calls per cluster). PSI (pressure stall information) is now collected: each cycle probes rrddata for the highest-utilization running containers, capped by USAGE_PSI_PROBE_LIMIT (default 16) so the fleet is never swept — raw stats miss exactly what PSI catches (a container at 99% memory can be healthy; one thrashing shows psiMemFull > 40 while raw memory looks identical). Candidate selection and RRD parsing are pure and unit-tested (utils/usage-psi.js). Surfaced as: - container.cpu/memory/io.pressure OTLP gauges (attr pressure.kind: some|full) on the collector - a Pressure column on the owner grid (worst PSI across their containers) and per-container badges with full readings in the tooltip; green/amber/red at 10/40 per the #440 incident evidence DummyApi simulates node capacity rows and PSI series for dev parity; ProxmoxApi gains rrdData().
Strangler-fig migration (docs/mvc-manifesto.md section 6): the /sites/:siteId/usage report moves from routers/api/v1/usage.js into a full resources/usage/ vertical slice (router/controller/service/repository/serializer/validator) with unit tests for the service and supertest integration tests pinning the wire contract. Only intentional contract change: a non-numeric siteId is now a 400 invalid_request instead of reaching the database (same policy as the migrated apikeys resource).
The collector was disabled by default (no-op without OTEL_EXPORTER_OTLP_ENDPOINT) and the usage dashboard never depended on it - GET /sites/:siteId/usage computes its report live via utils/usage-collection.js. Removes usage-collector.js, its systemd unit and packaging/dev-runner wiring, the @opentelemetry dependencies, and the OTLP env vars; keeps collectUsage, usage-sample/report/psi, and USAGE_PSI_PROBE_LIMIT, which the dashboard uses.
Collaborator
Author
Recent changesDropped the OTLP usage-collector servicePer review feedback: the collector was disabled by default (a no-op without
Kept everything the dashboard uses: Migrated the usage route to
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #440
What this adds
Per-container resource metrics that can be grouped by owner, delivered in two layers: OTLP export for durable collection/retention in a dedicated observability stack, and a live report in the manager for immediate visibility. No time-series data is stored in the manager database.
1. Collection — OTLP usage collector
usage-collector.jsruns as a third long-running service besideserver.jsandjob-runner.js(systemd unit,make devprocess, npm script included). Each cycle (default 5 min,USAGE_COLLECTOR_INTERVAL_MS):/cluster/resourcescalls per cluster (LXCs + node capacity) — cluster peers are deduplicated from the response, never one call per containerContainer.usernamerrddata) for the highest-utilization running containers, capped byUSAGE_PSI_PROBE_LIMIT(default 16) — the issue's two-tier pattern, so the fleet is never sweptOTEL_*environment variables; a no-op (exits 0) unlessOTEL_EXPORTER_OTLP_ENDPOINTis configured, so the observability backend stays out of this repoMetrics (all carry
owner,container.id,container.name,container.status,proxmox.node,site.idattributes, so any OTel backend cansum by (owner)):container.cpu.usage/container.cpu.limit{cpu}container.memory.usage/container.memory.limitBycontainer.disk.usage/container.disk.limitBycontainer.disk.io(disk.io.direction: read|write)Bycontainer.network.io(network.io.direction: receive|transmit)Bycontainer.uptimescontainer.cpu.pressure/container.memory.pressure/container.io.pressure(pressure.kind: some|full)%2. Display — live per-owner usage in the manager
GET /api/v1/sites/:siteId/usagecomputes the report on demand from the same sampling cycle (sharedutils/usage-collection.js), so the API and the OTLP export always agree; includes physical cluster capacity from the node rows/sites/:siteId/usage, sidebar link for all users):3. Tagging & attribution reporting
DummyApisimulates usage numbers, owner tags, node capacity, and PSI series so the whole path works in devWhy PSI matters more than raw stats
The #440 evidence shows why: 6 containers sit at 91–99% of their memory cap with
psiMemFull = 0.00and are perfectly healthy, while CT 284 at the same raw utilization hadpsiMemFull = 99and was degrading all four nodes. Raw usage says who holds resources; PSI says who is suffering/thrashing. The report carries both.Design note (pivot from the ticket's SQL ledger)
The issue's draft plan proposed a
ContainerUsageSampleledger in Postgres. This PR pivots to OTLP export instead: the manager's relational DB isn't a good fit for time-series data, and a dedicated OTel stack handles retention/rollups natively and opens the door to log aggregation later. The parts of the plan that depended on the ledger (daily rollups, org-hierarchy rollups, digests) are superseded by backend-side queries; IOPS (node-resident) and detection/intervention remain follow-on work under #436.Testing
usage-sample.js), aggregation (usage-report.js), and PSI selection/parsing (usage-psi.js) are unit-tested — 21 testsGET /sites/:siteId/usageexercised via the dev dummy node