Skip to content

fix(health): stop inflating service CPU by 100/TotalCores - #89

Merged
iyashjayesh merged 1 commit into
mainfrom
fix/service-cpu-units
Aug 29, 2026
Merged

iyashjayesh merged 1 commit into
mainfrom
fix/service-cpu-units

Conversation

@iyashjayesh

Copy link
Copy Markdown
Owner

Closes #87. Uses the percent-of-one-core convention, matching what the dashboard already displays.

The same measurement, two different numbers

One /metrics response, 10-core host:

load_statistics.service_cpu_load : 1.68%
service_health.icon_msg          : CPU Usage 17.66% / 95.00%

Ratio: 10.51. Cores: 10.

getServiceCPUUsage returns gopsutil's Process.CPUPercent — percent of one core, the top convention where a process saturating two cores reads 200%. The health path then did:

cpuUsagePercentage := (cpuUsage / float64(totalAvailableCores)) * 100

Dividing by the core count converts to share-of-machine. The * 100 then multiplied a value that was already a percentage.

It gets worse as machines get smaller

Cores Inflation
1 100×
4 25×
10 10×
100 1× (correct by coincidence)

An idle test process, measured:

cores=1: service CPU reads 360.39%, over the whole allowance, for a test process doing nothing
cores=2: service CPU reads 178.20%
cores=4: service CPU reads 131.96%

All three exceed a 95% limit. And since #86 made a breach any single resource over its own limit, that was enough on its own to report a perfectly healthy service as unhealthy — on exactly the small hosts where people are most likely to be watching.

System health was checked, and is fine

calculateSystemHealth uses GetCPUPrecent, which returns a machine-wide total and is used with no division. I verified this rather than assuming it — the issue flagged it as an open question.

A test that was shaped around the defect

healthyStats() set TotalCores = 100000, with a comment explaining that service CPU "is divided by TotalCores, so a small core count makes the result depend on how busy the machine running the tests happens to be." The fixture was compensating for the bug rather than exposing it. It is 8 now.

TestServiceCPUDoesNotScaleWithCoreCount fails on the old arithmetic with the numbers above.

Behaviour note

MaxCPUUsage is now read in the same units, so a service saturating more than one core exceeds 100% by design — as top reports it. Anyone who tuned a threshold against the old inflated figure will want to revisit it; the numbers get smaller, so the failure mode is a threshold that is now too loose rather than too tight.

Verification

go test -race ./... green across all 12 packages, go vet clean.

getServiceCPUUsage returns gopsutil's Process.CPUPercent, which is
percent of ONE core -- the `top` convention, where a process saturating
two cores reads 200%. The health calculation then did:

    cpuUsagePercentage := (cpuUsage / float64(totalAvailableCores)) * 100

Dividing by the core count converts to share-of-machine. The `* 100`
then multiplied a value that was already a percentage, inflating the
result by 100/TotalCores.

The same measurement therefore appeared twice with different values: the
dashboard showed 1.68% under SERVICE CPU LOAD while the health message
reported 17.66% for the same quantity on a 10-core host.

The error is worst on the smallest machines -- 25x on four cores, 100x
on one -- which are the machines most likely to be near a real limit. An
idle test process measures 360% on one core and 132% on four. Since #86
made a breach any single resource over its own limit, that alone was
enough to report a healthy service as unhealthy.

Service CPU now reports percent of one core, matching what
load_statistics already shows, so the two agree. MaxCPUUsage is read in
the same units, which means a service saturating more than one core
exceeds 100% by design, exactly as top would report it.

System health was checked and is unaffected: calculateSystemHealth uses
GetCPUPrecent, which returns a machine-wide total and is used with no
division.

healthyStats set TotalCores to 100000 to stop the inflated reading
swamping the fixture -- a test shaped around the defect. It is 8 now.

TestServiceCPUDoesNotScaleWithCoreCount fails on the old arithmetic by
quoting the numbers above.

Closes #87.
@iyashjayesh
iyashjayesh merged commit 364c077 into main Aug 29, 2026
5 checks passed
@iyashjayesh
iyashjayesh deleted the fix/service-cpu-units branch August 29, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Service-health CPU is inflated by 100/TotalCores

1 participant