Skip to content

fix(alerts): budget sparse workqueue warnings #240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rldyourmnd merged 1 commit into from
Aug 27, 2026
+8 −8
Merged

fix(alerts): budget sparse workqueue warnings #240

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions config/observability-rules.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,21 +179,21 @@ rules:
action: Preserve slab, audit and AppArmor evidence; close member admission and roll to the fixed kernel after jobs drain.
recovery: Every fleet host remains below two GiB unreclaimable slab after workload churn.
- id: kernel_workqueue_hog
severity: page
severity: ticket
query_language: promql
stream_name: gha_fleet_host_signal_events
expression: sum(max_over_time(gha_fleet_host_signal_events{signal_class="kernel_workqueue_hog",aggregation_temporality="AGGREGATION_TEMPORALITY_CUMULATIVE"}[15m]))
expression: max(max_over_time(gha_fleet_host_signal_events{signal_class="kernel_workqueue_hog",aggregation_temporality="AGGREGATION_TEMPORALITY_CUMULATIVE"}[1h]) - min_over_time(gha_fleet_host_signal_events{signal_class="kernel_workqueue_hog",aggregation_temporality="AGGREGATION_TEMPORALITY_CUMULATIVE"}[1h]))
operator: ">"
threshold: 0
evaluation_seconds: 60
hold_seconds: 60
threshold: 10
evaluation_seconds: 300
hold_seconds: 600
destination_ref: fleet_oncall
enabled: false
owner: fleet-operations
runbook: https://github.com/NDDev-OpenNetwork/github-actions/blob/main/docs/runbooks/fleet-alerts.md
summary: A fleet kernel workqueue reported a CPU hog event.
action: Preserve the kernel message and host pressure state, then close admission on the affected member if latency or PSI is elevated.
recovery: No workqueue-hog event is present in the rolling fifteen-minute observation window.
summary: A fleet member repeatedly reports kernel workqueue CPU-hog events.
action: Correlate the affected member with workqueue identity, host pressure and workload churn before changing admission or kernel policy.
recovery: No member records more than ten new workqueue-hog notices in a rolling hour for ten minutes.
- id: lifecycle_assigned_stall
severity: page
query_language: promql
Expand Down