[MitchDrage]

Description

Fixes #14397
Fixes #14411

This PR makes each management server collect VM stats only from the hosts connected to it. Previously every management server polled every host, so with 3 management servers each VM got 3 rows in vm_stats per interval.

• VmStatsCollector now filters its host search on managementServerId.
• HostCollector is unchanged and still collects from all hosts.
• Added unit tests for both collectors' host searches.

When vm.stats.increment.metrics is on, the network and disk values returned are the sum of all stored rows for the VM. What this change does to that sum depends on the hypervisor:

• KVM: no change. The agent returns a delta since the last poll by any management server, so the extra rows were slices of the same interval and added up to the right total.
• VMware and XenServer: totals drop to the correct value. Each poll returns an average over a fixed window, so every management server stored roughly the same value and the sum was N times too high (Accumulated VM stats are multiplied by the number of management servers on VMware and XenServer #14411). Operators with this setting on will see these totals fall by a factor of N after upgrading. That's the bug being fixed.

With the setting off (the default), the API returns the latest row. On KVM each row now covers the full vm.stats.interval, so single samples are larger and more consistent. On VMware and XenServer single samples are unchanged.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

• StatsCollectorTest passes (60 tests), including the two new ones.
• With the fix removed, the new VmStatsCollector test fails.
• The VMware and XenServer behaviour is from reading the code (VmwareResource.getVmStats, CitrixResourceBase.getVmStats); I haven't tested on either.

How did you try to break this feature and the system with this change?

• Checked that nothing reads vm_stats.mgmt_server_id. listVirtualMachines uses the newest row (or the sum), the UI charts plot every row in the time range, and the cleanup job deletes by timestamp, so they all still see every VM's stats.
• Checked that every Up host has an owning management server on all hypervisors: agentStatusTransitTo stamps the host with the msid on each status change, and VMware/XenServer hosts are claimed by one management server when they load. No host is left without stats.
• If a host reconnects to a different management server, at most one or two samples are missed.

[Damans227]

thanks @MitchDrage, fix looks right. LGTM.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 16.42%. Comparing base (376c1b4) to head (556df61).

Additional details and impacted files

@@            Coverage Diff            @@
##               4.20   #14398   +/-   ##
=========================================
  Coverage     16.41%   16.42%           
- Complexity    13657    13658    +1     
=========================================
  Files          5669     5669           
  Lines        501628   501629    +1     
  Branches      60942    60942           
=========================================
+ Hits          82336    82372   +36     
+ Misses       410064   410025   -39     
- Partials       9228     9232    +4     

Flag	Coverage Δ
uitests	4.16% <ø> (ø)
unittests	17.28% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[DaanHoogland]

it looks good but I will do a quick manual verification once smoke tests pass.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19647

[DaanHoogland]

@blueorangutan test keepEnv

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests