Repository navigation
fix(health): make the verdict mean what the card says it means - #86
Merged
Merged
Conversation
Two defects in the same place, one found while fixing the other. The badge contradicted its own explanation. Healthy came from `Percent > 50` -- a composite score -- while IconMsg came from a threshold comparison. Nothing kept them in step, so the dashboard rendered "Degraded" beside the sentence "System usage is within limits", with every number in the card inside its limit. It also flapped. A binary taken from a continuously drifting score oscillates near the cutoff: sampled every 3s on an idle machine it crossed the boundary twice in 45 seconds while the bars beside it sat still. A health badge that changes twice a minute for no reason trains people to stop reading it. The verdict now comes from the same check that writes the message. The two are set at the same point, so they cannot drift apart. Percent is untouched and still drives the ring and the graded status text -- it just no longer decides a binary it was never designed to answer, which also removes the flapping without needing hysteresis, because thresholds do not sit near the boundary. The second defect showed up in the first test run. The breach test ran on the average of the usage ratios, so an idle resource cancelled a saturated one: memory at 104% of its allowance beside CPU at 12% averages to 58 and reported "within limits" while memory was over. Left alone, this fix would have made the badge faithfully report a wrong verdict. A breach is now any single resource over its own limit; averaging stays where it belongs, in the graded score. Both regressions are pinned. Restoring `Percent > 50` fails TestTheVerdictAgreesWithItsOwnExplanation by quoting the contradiction; restoring the averaged breach fails TestOneBlownResourceIsABreach with the 104%-but-"within limits" message. Closes #84.
pull Bot
pushed a commit
to bhardwajRahul/monigo
that referenced
this pull request
Aug 29, 2026
getServiceCPUUsage returns gopsutil's Process.CPUPercent, which is
percent of ONE core -- the `top` convention, where a process saturating
two cores reads 200%. The health calculation then did:
cpuUsagePercentage := (cpuUsage / float64(totalAvailableCores)) * 100
Dividing by the core count converts to share-of-machine. The `* 100`
then multiplied a value that was already a percentage, inflating the
result by 100/TotalCores.
The same measurement therefore appeared twice with different values: the
dashboard showed 1.68% under SERVICE CPU LOAD while the health message
reported 17.66% for the same quantity on a 10-core host.
The error is worst on the smallest machines -- 25x on four cores, 100x
on one -- which are the machines most likely to be near a real limit. An
idle test process measures 360% on one core and 132% on four. Since iyashjayesh#86
made a breach any single resource over its own limit, that alone was
enough to report a healthy service as unhealthy.
Service CPU now reports percent of one core, matching what
load_statistics already shows, so the two agree. MaxCPUUsage is read in
the same units, which means a service saturating more than one core
exceeds 100% by design, exactly as top would report it.
System health was checked and is unaffected: calculateSystemHealth uses
GetCPUPrecent, which returns a machine-wide total and is used with no
division.
healthyStats set TotalCores to 100000 to stop the inflated reading
swamping the fixture -- a test shaped around the defect. It is 8 now.
TestServiceCPUDoesNotScaleWithCoreCount fails on the old arithmetic by
quoting the numbers above.
Closes iyashjayesh#87.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #84. Two defects in the same place — the second found while fixing the first.
1. The badge contradicted its own explanation
Healthycame fromPercent > 50(a composite score) whileIconMsgcame from a threshold comparison. Nothing kept them in step, so one payload produced one card saying both things at once:Every number inside its limit, the sentence saying so, and the badge saying the opposite.
It also flapped. A binary taken from a continuously drifting score oscillates near its cutoff. Sampled every 3s on an idle machine:
Two crossings in 45 seconds, while the bars beside it did not move.
The verdict now comes from the same check that writes the message, set at the same point so they cannot drift.
Percentis untouched — it still drives the ring and the graded status text. It just no longer answers a binary it was never designed for, which removes the flapping without hysteresis, because thresholds do not sit near the boundary.2. An idle resource masked a saturated one
This surfaced in the first test run. The breach test ran on the average of the usage ratios:
Memory at 104% of its allowance — over the limit — reported as within limits, because
(11.69 + 104.21) / 2 = 57.95.Left alone, fix #1 would have made the badge faithfully report a wrong verdict. A breach is now any single resource over its own limit. Averaging stays where it belongs, in the graded score.
Both regressions are pinned
Restoring
Percent > 50:Restoring the averaged breach test:
Behaviour change worth knowing
Services that were quietly reporting healthy while one resource was over its limit will now report unhealthy. That is the point, but it is a change in what existing deployments display.
Not in this PR
While reading the same file I found the service-health CPU normalisation looks wrong —
(cpuUsage / totalCores) * 100on a gopsutil value that is already a percentage of one core, which inflates by100/TotalCores(10× on a 10-core machine, 25× on a 4-core one). Cross-checked againstload_statistics.service_cpu_load: 1.68% reported there, 17.66% in the health message for the same quantity. It changes every user's numbers, so it wants its own issue and its own PR rather than riding along here.Verification
go test -race ./...green across all 12 packages,go vetclean.