feat: add container restart count and build duration metrics by thegdsks · Pull Request #560 · glincker/levelrail

thegdsks · 2026-09-22T04:00:58Z

Summary

Container restart count is now a real per-app metric: persists alerting.RestartTracker's existing Docker-event-stream restart signal (used for crashloop detection) as a container_restart_count telemetry sample instead of relying on Docker's own restart-policy counter, which never increments for levelrail-managed containers (restart policy is disabled by design; the reconciler is the sole authority on restarting a dead container).
Build duration (build_duration_seconds) was already recorded backend-side; this surfaces it on the per-app metrics dashboard as a real line chart.
Request rate, latency percentiles, and error rate remain out of scope for this pass: they need Caddy ingress instrumentation (access logs or a metrics module) that doesn't exist yet.

Test plan

go build ./..., go vet ./..., golangci-lint run ./... (0 issues)
go test ./... full backend suite green, including live Docker tests
npm run build, npm run lint, npx tsc -b, npx vitest run (244 tests passed, 46 files)
Pre-push hook verification: changed-line coverage 95.0% (threshold 70%), internal/alerting/crashloop.go 94.7% coverage, internal/telemetry/deploy_metrics.go 100% coverage
Real end-to-end verification against a live running instance (isolated ports): deployed a real app via a real BuildKit build from a local git repo, recorded a real build_duration_seconds of 0.810703292 seconds (displays as "1s" on dashboard), verified via the metrics API
Killed the real running container twice via docker kill; confirmed the real reconciler brought it back in place (same container name); container_restart_count recorded exactly 2 real samples, each Value=1, timestamps matching the kill times to the second
Logged into the real running dashboard UI in a real browser session and confirmed the app's Metrics page renders "Build duration" as a chart with current reading "1s" and header shows "Deploy frequency: 1 in this range. Restarts: 2 in this range." matching the real events triggered

Docker's own container-inspect RestartCount field is unusable here: every levelrail-managed container disables Docker's restart policy by design, so it never increments. Wires the real signal that already existed for crashloop detection (alerting.RestartTracker's Docker event-stream watcher) through as a persisted container_restart_count metric instead. Build duration was already recorded backend-side; this surfaces it on the per-app metrics dashboard as a real chart. Request rate, latency percentiles, and error rate are skipped this pass: they need Caddy ingress instrumentation that doesn't exist yet.

coderabbitai · 2026-09-22T04:01:08Z

Warning

Review limit reached

Next included review available in 39 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 3c33ef79-6044-4c0e-ad03-a8fe30a28237

📥 Commits

Reviewing files that changed from the base of the PR and between 59e4888 and 9ca171f.

📒 Files selected for processing (10)

cmd/levelrail/main.go
internal/alerting/crashloop.go
internal/alerting/crashloop_test.go
internal/telemetry/deploy_metrics.go
internal/telemetry/deploy_metrics_test.go
web/src/components/MetricsDashboard.tsx
web/src/components/metricName.ts
web/src/lib/metricChart.test.ts
web/src/lib/metricChart.ts
web/src/types/metrics.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

sonarqubecloud · 2026-09-22T04:02:05Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud

greptile-apps · 2026-09-22T04:03:55Z

Confidence Score: 2/5

The PR is not safe to merge until restart totals, process-start accounting, and restart-alert semantics are corrected.

Findings

Restart totals count buckets ▶
Restart thresholds cannot count ▶
First restart is discarded ▶
Telemetry blocks event processing ▶

Summary

This PR persists reconciler-observed container restarts as telemetry, displays restart totals and build-duration charts on the application metrics dashboard, and makes the new metrics available to alert-rule forms.

Adds restart-event persistence through RestartTracker.
Adds dashboard presentation and duration formatting for the new metrics.
Extends frontend metric names and alert-picker options.
The current implementation undercounts aggregated and post-restart events, and the restart metric is incompatible with generic threshold evaluation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  D[Docker start event] --> T[RestartTracker]
  T --> M[In-memory crashloop history]
  T --> W[Telemetry database write]
  W --> A[App metrics API]
  A --> B[Range aggregation]
  B --> U[Metrics dashboard]
  W --> E[Threshold evaluator]

_{Reviews (1) · Last reviewed commit: "feat: add container restart count and bu..."}

greptile-apps · 2026-09-22T04:03:58Z

+    'container_restart_count',
+    range,
+  )
+  const restartCount = restartCountQuery.data?.points.length ?? null


Restart totals count buckets

For the 6h, 24h, and 7d ranges, the API groups restart samples into time buckets and exposes the number of samples in each point's count field. Using points.length counts non-empty buckets instead, so multiple restarts in one bucket are displayed as a single restart.

Suggested change

const restartCount = restartCountQuery.data?.points.length ?? null

const restartCount =

restartCountQuery.data?.points.reduce(

(total, point) => total + point.count,

0,

) ?? null

greptile-apps · 2026-09-22T04:03:58Z

  'network_tx_bytes',
  'disk_read_bytes',
  'disk_write_bytes',
+  'container_restart_count',


Restart thresholds cannot count

This exposes container_restart_count in the generic threshold-rule picker, but every restart is stored as a separate sample with value 1 and threshold evaluation checks only the latest sample. A rule such as “container restarts > 3” therefore cannot fire even when a container restarts repeatedly.

greptile-apps · 2026-09-22T04:03:58Z

 	if !t.seen[containerName] {
 		t.seen[containerName] = true
+		t.mu.Unlock()
 		return
 	}


First restart is discarded

The tracker starts with an empty in-memory seen map and does not seed it from containers that are already running. After Levelrail starts, the first real restart of a pre-existing container is therefore treated as that container's initial startup and returns before telemetry is recorded, causing restart totals to be undercounted across control-plane restarts.

greptile-apps · 2026-09-22T04:03:58Z

+	if !ok || serviceName == "" {
+		return
+	}
+	if err := recorder.RecordContainerRestart(context.Background(), serviceName, at); err != nil {


Telemetry blocks event processing

The telemetry write runs synchronously inside the sole restart-event consumer and uses context.Background(). If SQLite is busy, the write can stall subsequent crashloop observations and cannot be canceled with RestartTracker.Run, delaying event processing and shutdown until the database operation finishes.

thegdsks enabled auto-merge (squash) September 22, 2026 04:01

github-actions Bot added size/l 200-499 lines changed type/feature New capability or ergonomic improvement area/frontend web/ area/alerting internal/alerting area/telemetry internal/telemetry (metrics, logs) labels Sep 22, 2026

greptile-apps Bot reviewed Sep 22, 2026

View reviewed changes

thegdsks mentioned this pull request Sep 22, 2026

docs: secret age tracking and rotation reminders #573

Merged

thegdsks merged commit 1b278fa into main Sep 23, 2026
27 of 29 checks passed

github-actions Bot mentioned this pull request Sep 22, 2026

chore(main): release 0.2.0-beta.6 #544

Merged

thegdsks deleted the feat/extended-app-metrics branch September 24, 2026 03:59

Navigation Menu

feat: add container restart count and build duration metrics - #560

feat: add container restart count and build duration metrics#560
thegdsks merged 1 commit into
mainfrom
feat/extended-app-metrics

thegdsks commented Sep 22, 2026

Uh oh!

coderabbitai Bot commented Sep 22, 2026

Review limit reached

Uh oh!

sonarqubecloud Bot commented Sep 22, 2026

Uh oh!

greptile-apps Bot commented Sep 22, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Sep 22, 2026

Uh oh!

greptile-apps Bot Sep 22, 2026

Uh oh!

greptile-apps Bot Sep 22, 2026

Uh oh!

greptile-apps Bot Sep 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

-  const restartCount = restartCountQuery.data?.points.length ?? null
+  const restartCount =
+    restartCountQuery.data?.points.reduce(
+      (total, point) => total + point.count,
+,
+    ) ?? null

Conversation

thegdsks commented Sep 22, 2026

Summary

Test plan

Uh oh!

coderabbitai Bot commented Sep 22, 2026

Review limit reached

Uh oh!

sonarqubecloud Bot commented Sep 22, 2026

Quality Gate passed

Uh oh!

greptile-apps Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 2/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Sep 22, 2026 •

edited

Loading