Improved support for CGroups to fix CPU usage values - #180
Merged
Merged
Conversation
The current implementation of the CpuMetricsProvider uses the `OperatingSystemMXBean`'s cpu usage properties to report them to Platform. However, its internal implementation divides cpu usage by the CFS time, instead of the wall time, thus reporting only the CPU usage in the time intervals it has any cpu usage. This specially becomes significant at 0.2 cpu request limit or lower, or applications that run idle more (no event processors constantly ticking). So, this PR changes the CpuMetricsProvider to use wall-clock time and /proc/stat when available. It will fall back to the old method if unavailable. This issue also surfaced that it's very useful for us to know the configuration of the jvm and container. As such, the setup payload was extended with many (non-sensitive) system properties, such as os and jvm vendors and versions, available processors, garbage collection, and more. Disclaimer: Most of this has been created with the help of Claude. I have some basic Linux knowledge, and through this story I've read up on it. I've confirmed CGroup and /proc/stat format against our development containers, and it will work. In cases it doesn't, I've wrapped everything in a `runCatching`, so it doesn't affect the functionality. Once merged, I will run Axoniq Platform with this first, to see the outcome in a real environment. Then, we will port it to the Axoniq Framework 5 main branch.
CodeDrivenMitch
requested review from
Andrew-deVillier and
stefanmirkovic
and removed request for
a team
September 22, 2026 11:57
|
| val versions: Versions, | ||
| val upcasters: List<String>, | ||
| val features: SupportedFeatures = SupportedFeatures(), | ||
| val runtime: RuntimeInformation? = null, |
Contributor
There was a problem hiding this comment.
Non-blocking question: Is inspector's SetupPayload deserialization confirmed to tolerate unknown fields from newer clients, or is that worth a quick check before the client release goes out?
Collaborator
Author
There was a problem hiding this comment.
Yes, this is set up this way by design due to supporting multiple versions
stefanmirkovic
approved these changes
Sep 22, 2026
Collaborator
Author
|
Ported to the AF5 line in #182. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The current implementation of the CpuMetricsProvider uses the
OperatingSystemMXBean's cpu usage properties to report them to Platform. However, its internal implementation divides cpu usage by the CFS time, instead of the wall time, thus reporting only the CPU usage in the time intervals it has any cpu usage. This specially becomes significant at 0.2 cpu request limit or lower, or applications that run idle more (no event processors constantly ticking).So, this PR changes the CpuMetricsProvider to use wall-clock time and /proc/stat when available. It will fall back to the old method if unavailable.
This issue also surfaced that it's very useful for us to know the configuration of the jvm and container. As such, the setup payload was extended with many (non-sensitive) system properties, such as os and jvm vendors and versions, available processors, garbage collection, and more.
Disclaimer: Most of this has been created with the help of Claude. I have some basic Linux knowledge, and through this story I've read up on it. I've confirmed CGroup and /proc/stat format against our development containers, and it will work. In cases it doesn't, I've wrapped everything in a
runCatching, so it doesn't affect the functionality.Once merged, I will run Axoniq Platform with this first, to see the outcome in a real environment. Then, we will port it to the Axoniq Framework 5 main branch.