Skip to content

fix(health): make the verdict mean what the card says it means - #86

Merged
iyashjayesh merged 1 commit into
mainfrom
fix/health-verdict
Aug 29, 2026
Merged

iyashjayesh merged 1 commit into
mainfrom
fix/health-verdict

Conversation

@iyashjayesh

Copy link
Copy Markdown
Owner

Closes #84. Two defects in the same place — the second found while fixing the first.

1. The badge contradicted its own explanation

Healthy came from Percent > 50 (a composite score) while IconMsg came from a threshold comparison. Nothing kept them in step, so one payload produced one card saying both things at once:

SYSTEM HEALTH   Degraded
host cpu        27.79% / 95.00%
host memory     83.88% / 95.00%

System usage is within limits: CPU Usage 18.06% / 95.00%, Memory Usage 81.25% / 95.00%

Every number inside its limit, the sentence saying so, and the badge saying the opposite.

It also flapped. A binary taken from a continuously drifting score oscillates near its cutoff. Sampled every 3s on an idle machine:

55.66%  healthy=True   -> Healthy
48.91%  healthy=False  -> Degraded
44.93%  healthy=False  -> Degraded
51.39%  healthy=True   -> Healthy

Two crossings in 45 seconds, while the bars beside it did not move.

The verdict now comes from the same check that writes the message, set at the same point so they cannot drift. Percent is untouched — it still drives the ring and the graded status text. It just no longer answers a binary it was never designed for, which removes the flapping without hysteresis, because thresholds do not sit near the boundary.

2. An idle resource masked a saturated one

This surfaced in the first test run. The breach test ran on the average of the usage ratios:

"within limits: CPU Usage 11.69% / 95.00%, Memory Usage 104.21% / 95.00%"

Memory at 104% of its allowance — over the limit — reported as within limits, because (11.69 + 104.21) / 2 = 57.95.

Left alone, fix #1 would have made the badge faithfully report a wrong verdict. A breach is now any single resource over its own limit. Averaging stays where it belongs, in the graded score.

Both regressions are pinned

Restoring Percent > 50:

--- FAIL: TestTheVerdictAgreesWithItsOwnExplanation
    system reports Healthy=false while its own message says the usage is within limits.
      message: System usage is within limits: CPU Usage 11.69% / 95.00%, Memory Usage 104.21% / 95.00%

Restoring the averaged breach test:

--- FAIL: TestOneBlownResourceIsABreach
    system: a fixture that exceeds a limit reports Breached=false
    system: message claims the usage is within limits while a resource is over it

Behaviour change worth knowing

Services that were quietly reporting healthy while one resource was over its limit will now report unhealthy. That is the point, but it is a change in what existing deployments display.

Not in this PR

While reading the same file I found the service-health CPU normalisation looks wrong — (cpuUsage / totalCores) * 100 on a gopsutil value that is already a percentage of one core, which inflates by 100/TotalCores (10× on a 10-core machine, 25× on a 4-core one). Cross-checked against load_statistics.service_cpu_load: 1.68% reported there, 17.66% in the health message for the same quantity. It changes every user's numbers, so it wants its own issue and its own PR rather than riding along here.

Verification

go test -race ./... green across all 12 packages, go vet clean.

Two defects in the same place, one found while fixing the other.

The badge contradicted its own explanation. Healthy came from
`Percent > 50` -- a composite score -- while IconMsg came from a
threshold comparison. Nothing kept them in step, so the dashboard
rendered "Degraded" beside the sentence "System usage is within limits",
with every number in the card inside its limit.

It also flapped. A binary taken from a continuously drifting score
oscillates near the cutoff: sampled every 3s on an idle machine it
crossed the boundary twice in 45 seconds while the bars beside it sat
still. A health badge that changes twice a minute for no reason trains
people to stop reading it.

The verdict now comes from the same check that writes the message. The
two are set at the same point, so they cannot drift apart. Percent is
untouched and still drives the ring and the graded status text -- it
just no longer decides a binary it was never designed to answer, which
also removes the flapping without needing hysteresis, because thresholds
do not sit near the boundary.

The second defect showed up in the first test run. The breach test ran
on the average of the usage ratios, so an idle resource cancelled a
saturated one: memory at 104% of its allowance beside CPU at 12%
averages to 58 and reported "within limits" while memory was over. Left
alone, this fix would have made the badge faithfully report a wrong
verdict. A breach is now any single resource over its own limit;
averaging stays where it belongs, in the graded score.

Both regressions are pinned. Restoring `Percent > 50` fails
TestTheVerdictAgreesWithItsOwnExplanation by quoting the contradiction;
restoring the averaged breach fails TestOneBlownResourceIsABreach with
the 104%-but-"within limits" message.

Closes #84.
@iyashjayesh
iyashjayesh merged commit e41540f into main Aug 29, 2026
5 checks passed
@iyashjayesh
iyashjayesh deleted the fix/health-verdict branch August 29, 2026 18:44
pull Bot pushed a commit to bhardwajRahul/monigo that referenced this pull request Aug 29, 2026
getServiceCPUUsage returns gopsutil's Process.CPUPercent, which is
percent of ONE core -- the `top` convention, where a process saturating
two cores reads 200%. The health calculation then did:

    cpuUsagePercentage := (cpuUsage / float64(totalAvailableCores)) * 100

Dividing by the core count converts to share-of-machine. The `* 100`
then multiplied a value that was already a percentage, inflating the
result by 100/TotalCores.

The same measurement therefore appeared twice with different values: the
dashboard showed 1.68% under SERVICE CPU LOAD while the health message
reported 17.66% for the same quantity on a 10-core host.

The error is worst on the smallest machines -- 25x on four cores, 100x
on one -- which are the machines most likely to be near a real limit. An
idle test process measures 360% on one core and 132% on four. Since iyashjayesh#86
made a breach any single resource over its own limit, that alone was
enough to report a healthy service as unhealthy.

Service CPU now reports percent of one core, matching what
load_statistics already shows, so the two agree. MaxCPUUsage is read in
the same units, which means a service saturating more than one core
exceeds 100% by design, exactly as top would report it.

System health was checked and is unaffected: calculateSystemHealth uses
GetCPUPrecent, which returns a machine-wide total and is used with no
division.

healthyStats set TotalCores to 100000 to stop the inflated reading
swamping the fixture -- a test shaped around the defect. It is 8 now.

TestServiceCPUDoesNotScaleWithCoreCount fails on the old arithmetic by
quoting the numbers above.

Closes iyashjayesh#87.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

System health flaps between Healthy and Degraded, and contradicts its own explanation

1 participant