Repository navigation
fix(health): stop inflating service CPU by 100/TotalCores - #89
Merged
Merged
Conversation
getServiceCPUUsage returns gopsutil's Process.CPUPercent, which is
percent of ONE core -- the `top` convention, where a process saturating
two cores reads 200%. The health calculation then did:
cpuUsagePercentage := (cpuUsage / float64(totalAvailableCores)) * 100
Dividing by the core count converts to share-of-machine. The `* 100`
then multiplied a value that was already a percentage, inflating the
result by 100/TotalCores.
The same measurement therefore appeared twice with different values: the
dashboard showed 1.68% under SERVICE CPU LOAD while the health message
reported 17.66% for the same quantity on a 10-core host.
The error is worst on the smallest machines -- 25x on four cores, 100x
on one -- which are the machines most likely to be near a real limit. An
idle test process measures 360% on one core and 132% on four. Since #86
made a breach any single resource over its own limit, that alone was
enough to report a healthy service as unhealthy.
Service CPU now reports percent of one core, matching what
load_statistics already shows, so the two agree. MaxCPUUsage is read in
the same units, which means a service saturating more than one core
exceeds 100% by design, exactly as top would report it.
System health was checked and is unaffected: calculateSystemHealth uses
GetCPUPrecent, which returns a machine-wide total and is used with no
division.
healthyStats set TotalCores to 100000 to stop the inflated reading
swamping the fixture -- a test shaped around the defect. It is 8 now.
TestServiceCPUDoesNotScaleWithCoreCount fails on the old arithmetic by
quoting the numbers above.
Closes #87.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #87. Uses the percent-of-one-core convention, matching what the dashboard already displays.
The same measurement, two different numbers
One
/metricsresponse, 10-core host:Ratio: 10.51. Cores: 10.
getServiceCPUUsagereturns gopsutil'sProcess.CPUPercent— percent of one core, thetopconvention where a process saturating two cores reads 200%. The health path then did:Dividing by the core count converts to share-of-machine. The
* 100then multiplied a value that was already a percentage.It gets worse as machines get smaller
An idle test process, measured:
All three exceed a 95% limit. And since #86 made a breach any single resource over its own limit, that was enough on its own to report a perfectly healthy service as unhealthy — on exactly the small hosts where people are most likely to be watching.
System health was checked, and is fine
calculateSystemHealthusesGetCPUPrecent, which returns a machine-wide total and is used with no division. I verified this rather than assuming it — the issue flagged it as an open question.A test that was shaped around the defect
healthyStats()setTotalCores = 100000, with a comment explaining that service CPU "is divided by TotalCores, so a small core count makes the result depend on how busy the machine running the tests happens to be." The fixture was compensating for the bug rather than exposing it. It is8now.TestServiceCPUDoesNotScaleWithCoreCountfails on the old arithmetic with the numbers above.Behaviour note
MaxCPUUsageis now read in the same units, so a service saturating more than one core exceeds 100% by design — astopreports it. Anyone who tuned a threshold against the old inflated figure will want to revisit it; the numbers get smaller, so the failure mode is a threshold that is now too loose rather than too tight.Verification
go test -race ./...green across all 12 packages,go vetclean.