Conversation
11 tasks
9 tasks
…re data loss The Neuron per-core data-loss defect fixed in aws/amazon-cloudwatch-agent#2263 only appears on a node running two or more Neuron runtimes, and no fixture here produced that shape -- so nothing in this repo could have caught it. neuron-burn-core is replicas: 1 holding one of inf2.xlarge's two NeuronCores, so its node has exactly one runtime. It also carries a self-referential requiredDuringScheduling podAntiAffinity on neuron-test=true, which actively prevents a second Neuron pod from joining it. Raising replicas to 2 does not help: the anti-affinity pushes the second replica onto the idle node, giving one runtime per node again and destroying the idle-node fixture. Two existing tests look like they should catch the defect and do not: - TestNeuronRuntimeTagInResourceScope passes on the broken agent. It asserts the tag exists, is non-empty, and is absent from the datapoint; all three hold, because the collapsed resource still carries one (wrong) tag. - TestNeuronNoDuplicateSeries passes because the collision happens in-agent. Duplicate identities are merged before export, so the surface shows too FEW series rather than duplicated ones. Cardinality is the signal, not duplication. Fixture (terraform/eks/daemon/otel-neuron/main.tf): adds a neuron-burn-peer Deployment taking the second core, co-located with neuron-burn-core via podAffinity, so one node runs two runtimes burning different cores. Three constraints, each of which breaks an existing test if changed: not named neuron-burn-core-* TestNeuronBurnWorkloadLabels and TestNeuronBurnCorePodColor match HasPrefix(pod, "neuron-burn-core") and assert workload.name / pod-color on it no neuron-test: "true" label neuron-burn-core's podAntiAffinity targets it, which would make the two mutually exclusive on a host and push neuron-burn-core onto the idle node podAffinity, not nodeSelector both node groups use var.instance_type, so a nodeSelector alone could land it on the idle node, breaking the uncorrelated-results fixture wait_neuron_monitor waits for both burn deployments to be Available and fails if they did not co-locate -- a silent split would make every multi-runtime test vacuously skip. The validator's burn-loop readiness wait also gates on neuron-burn-peer, symmetric with neuron-burn-core: Available only means the container is Running, and the trace compile that starts it emitting runtime_tag takes about a minute more. peer carries no neuron-test label, so every `-l neuron-test=true` diagnostic dump gets an explicit `-l app=neuron-burn-peer` companion query. Tests (test/otel/neuron/multi_runtime_test.go): 8 tests. The value-level one requires >=2 busy cores reported by >=2 runtime tags; the structural one requires cores x tags series, which catches a collapse even on an idle node where values cannot distinguish it. Also a tripwire asserting no series carries runtime_tag="DEFAULT", which is what the in-agent-aggregation alternative stamps -- changing that trade-off should mean changing the test, not a silent behaviour swap. Deliberately no assertion on distinct pod count: two runtimes need not be two pods, and one pod can host both, so requiring it would encode this fixture's shape rather than the invariant. That over-specification was caught by running against a real cluster. Validated against a live 2-runtime inf2.xlarge node in both directions: - With the agent fix: 7 of 8 pass. The 8th, FixtureIsBothBurnWorkloads, asserts the neuron-burn-core / neuron-burn-peer pod names and therefore needs this terraform cluster; the validation cluster used a differently-named workload. - With the fix reverted on the live agent: all 8 fail. The defect was confirmed to reproduce first -- 2 series instead of 4, one surviving runtime tag, core 0 reporting 0 while pinned at ~75% -- then restored and re-run to green. That negative control also exposed a diagnostic problem, now fixed: all 8 failed via the shared fixture guard rather than their own assertions, because the collapse destroys the runtime tag itself, so ">= 2 distinct tags" never holds. The guard's message previously blamed the fixture only. It now names both causes -- the defect and a missing fixture -- with the kubectl command to tell them apart, because a collapsed node is structurally indistinguishable from a genuine single-runtime node on the metrics surface. gofmt clean, go vet -tags integration ./test/otel/neuron/ clean, terraform fmt -check and terraform validate clean.
spanaik
added a commit
to aws/amazon-cloudwatch-agent
that referenced
this pull request
Sep 22, 2026
On a node with more than one Neuron runtime, roughly half of every per-core
Neuron metric is silently replaced by zero before it leaves the agent. A
NeuronCore pinned at 75% utilization reports 0.
neuron-monitor itself is correct -- it emits the full (neuroncore x runtime_tag)
cross-product with only the matching pairs non-zero:
neuroncore_utilization_ratio{neuroncore="0",runtime_tag="...core0"} 0.7523 <- real
neuroncore_utilization_ratio{neuroncore="1",runtime_tag="...core0"} 0.0
neuroncore_utilization_ratio{neuroncore="0",runtime_tag="...core1"} 0.0
neuroncore_utilization_ratio{neuroncore="1",runtime_tag="...core1"} 0.7517 <- real
A debug exporter on the live pipeline shows what the agent then produces: ONE
ResourceMetrics holding FOUR datapoints with only TWO distinct attribute sets --
aws.neuron.core=0 Value: 75.154815 <- real
aws.neuron.core=1 Value: 0.000000
aws.neuron.core=0 Value: 0.000000 <- same identity, shadows the real one
aws.neuron.core=1 Value: 75.363968
Cause, in transform/cw_k8s_ci_v0_neuron_promote:
1. It runs in `context: datapoint` but writes `resource.attributes`. Resource
attributes are per-ResourceMetrics, so with N runtimes in one RM the statement
executes N times and the last write wins -- which is why a single arbitrary
runtime tag ends up labelling the whole node.
2. It then deletes `runtime_tag` from the datapoint. That was the only attribute
distinguishing a core's real reading from another runtime's zero for that same
core, so two datapoints collapse onto one identity with different values.
Nothing downstream reconciles them (awsattributelimit early-returns below 150
attributes; batch only concatenates), so both go over the wire and one is
arbitrarily dropped at ingest.
It fails silently, which is why it went unnoticed: the collision happens
in-agent, so the exported surface shows too FEW series rather than duplicated
ones. Cardinality is the signal, not duplication. Single-runtime nodes are
unaffected -- it needs >=2 Neuron runtimes on one node to appear.
Blast radius is wider than utilization: neuron_runtime_memory_used_bytes has 4
series upstream (2 memory_location x 2 runtime_tag) and arrives as 2, so one
runtime's memory is dropped as well. Confirmed on neuroncore_utilization_ratio
and all five neuroncore_memory_usage_* families.
Fix: make `runtime_tag` a groupbyattrs grouping key, so each runtime gets its own
ResourceMetrics. groupbyattrs MOVES its grouping keys, so that also removes the
key from the datapoint, each datapoint identity stays unique, and the
last-write-wins clobber becomes a no-op. The promote then moves to
`context: resource`, which is where the attribute now lives -- and is how the
DCGM pipeline already does it (containerinsights/dcgm.yaml:66-83).
The six pod-identity statements are dropped as dead code, not as a behaviour
change: groupbyattrs has already deleted those keys from the datapoint by then,
so `attributes["k8s.pod.name"]` is always nil and they never fire.
Trade-off: this publishes the full cross-product, so series count grows
multiplicatively with runtime count -- 2 -> 4 on a 2-core/2-runtime node, and a
32-core trn1 with 4 runtimes goes from 32 to 128 series per per-core metric. This
is the cardinality neuron-monitor already emits; the previous behaviour reduced it
only by discarding data. The alternative is to aggregate the runtime dimension
away in-agent (max per core), which keeps the count flat but loses per-runtime
attribution; rejected because the OTLP/PromQL surface can carry the dimension and
a runtime legitimately spans multiple cores, so "which runtime is using this
core" is a question this surface should be able to answer. Not measured beyond a
2-core node.
Scope: the EMF / Container Insights path is unaffected. neuron.yaml's pipeline
exports to the forward/opentelemetry connector, consumed only by
metrics/opentelemetry -> otlphttp/metrics. The EMF Neuron path is a separate
pipeline (awscontainerinsightreceiver -> gpuattributes -> awsemf/containerinsights)
sharing no component instance with this one; its own max-per-core reduction in
gpuattributes/internal/awsneuron_metric_modifier.go is unchanged and remains
correct for that surface, which does not publish runtime_tag as a dimension.
Tests: neuron_multiruntime_test.go instantiates the real groupbyattrs and
transform processors from the //go:embed-ed neuron.yaml, chains them, and feeds
synthetic metrics in the multi-runtime shape (2 cores x 2 runtimes, non-owning
pairs at zero, all four sharing one pod). No cluster or Neuron hardware needed.
KeepsEveryCorePerRuntime all four (core, tag, value) triples survive
SeparatesRuntimesIntoResources one ResourceMetrics per runtime
PromotesPodIdentity pod/ns/container on the resource, off datapoints
KeepsUntaggedDatapoints device/hardware metrics with no runtime_tag land
in a resource with no aws.neuron.runtime.tag,
rather than inheriting an arbitrary runtime's
CollapsedConfigLosesData negative control: the pre-fix config collapses to
one tag and two datapoints per core
Config is read from the embedded template rather than restated, so a revert fails
the tests. Verified by mutation: removing `- runtime_tag` from the groupbyattrs
keys fails four of the five. The negative control keeps the others honest -- if
the collapse ever stops reproducing it starts passing, signalling the assertions
have gone vacuous.
Verified on a live 2-core inf2.xlarge running two Neuron runtimes: before, 2
series with core 0 reading 0 while pinned at ~75%; after, 4 correctly-labelled
series with core 0 -> 75.15 under its own runtime tag and core 1 -> 75.23 under
the other. Reverting the config on the live agent reproduced the loss and
re-applying restored it, so the fix is doing the work rather than a caching
artifact. 8 existing Neuron integration tests / 26 subtests still pass, including
TestNeuronNoPromotedDatapointKeys, which covers the six removed statements.
The four regenerated golden configs are the snapshots. Integration coverage for
the multi-runtime case is a companion change:
aws/amazon-cloudwatch-agent-test#749
The EKS add-on takes an independent copy of this pipeline from the
amazon-cloudwatch-observability Helm chart, which carries the identical defect;
that fix is aws-observability/helm-charts#365. Both are needed.
spanaik
force-pushed
the
neuron-multi-runtime-tests
branch
from
September 22, 2026 12:14
ba2699a to
17c44f9
Compare
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
Related: aws/amazon-cloudwatch-agent#2263
The Neuron per-core data-loss defect fixed in
aws/amazon-cloudwatch-agent# only appears on a node running two or more Neuron
runtimes, and no fixture in this repo produced that shape — so nothing here could have
caught it.
neuron-burn-coreisreplicas: 1holding one ofinf2.xlarge's two NeuronCores, so itsnode has exactly one runtime. It also carries a self-referential
requiredDuringSchedulingpodAntiAffinityonneuron-test=true, which activelyprevents a second Neuron pod from joining it. Raising
replicasto 2 does not help: theanti-affinity pushes the second replica onto the idle node, giving one runtime per node
again and destroying the idle-node fixture.
Worth noting the two existing tests that look like they should catch the defect and do
not:
TestNeuronRuntimeTagInResourceScopepasses on the broken agent — it asserts the tagexists, is non-empty, and is absent from the datapoint, and all three hold. The collapsed
resource still carries one (wrong) tag.
TestNeuronNoDuplicateSeriespasses because the collision happens in-agent: duplicateidentities are merged before export, so the surface shows too few series rather than
duplicated ones.
Cardinality is the signal, not duplication.
Description of changes
Fixture (
terraform/eks/daemon/otel-neuron/main.tf) — adds aneuron-burn-peerDeployment taking the second core, co-located with
neuron-burn-coreviapodAffinity, soone node runs two runtimes burning different cores.
Three constraints, each of which breaks an existing test if changed:
neuron-burn-core-*TestNeuronBurnWorkloadLabelsandTestNeuronBurnCorePodColormatchHasPrefix(pod, "neuron-burn-core")and assertworkload.name == "neuron-burn-core"/pod-color == orangeneuron-test: "true"labelneuron-burn-core'spodAntiAffinitytargets it, which would make the two mutually exclusive on a host and pushneuron-burn-coreonto the idle nodepodAffinity, not justnodeSelectorvar.instance_type, so anodeSelectoralone could land it on the idle node, breaking the uncorrelated-results fixturewait_neuron_monitornow waits for both burn deployments to beAvailableand fails ifthey did not co-locate — a silent split would make every multi-runtime test vacuously
skip.
Tests (
test/otel/neuron/multi_runtime_test.go) — 8 tests. The value-level one requires≥2 busy cores reported by ≥2 runtime tags; the structural one requires
cores × tagsseries, which catches a collapse even on an idle node where values cannot distinguish it.
Also included: a tripwire asserting no series carries
runtime_tag="DEFAULT", which is whatthe in-agent-aggregation alternative stamps — changing that trade-off should mean changing
the test, not a silent behaviour swap.
Deliberately no assertion on distinct pod count: two runtimes need not be two pods, and
one pod can host both, so requiring it would encode this fixture's shape rather than the
invariant. (That over-specification was caught by running against a real cluster.)
License
By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.
Tests
Validated against a live 2-runtime
inf2.xlargenode in both directions:FixtureIsBothBurnWorkloads, whichasserts the
neuron-burn-core/neuron-burn-peerpod names and therefore needs thisterraform cluster; the validation cluster used a differently-named workload.
reproduced first — 2 series instead of 4, one surviving runtime tag, core 0 reporting 0
while pinned at ~75% — then restored and re-ran to green.
That negative control also exposed a diagnostic problem, now fixed: all 8 failed via the
shared fixture guard rather than their own assertions, because the collapse destroys the
runtime tag itself, so
>= 2 distinct tagsnever holds. The guard's message previouslyblamed the fixture only. It now names both causes — the defect and a missing fixture —
with the
kubectlcommand to tell them apart, because a collapsed node is structurallyindistinguishable from a genuine single-runtime node on the metrics surface.
gofmtclean,go vet -tags integration ./test/otel/neuron/clean,terraform fmt -checkand
terraform validateclean.Not yet run:
terraform apply. The fixture is validated byterraform validateand bythe assertions passing against an existing 2-runtime node, but the specific
neuron-burn-peerplacement — that the device plugin grants one core to each pod andpodAffinityco-locates them — has not been exercised on a fresh cluster. Thewait_neuron_monitorco-location check is there to fail loudly if it doesn't.PR checklist
mainmakepasses locally —gofmtclean,go vet -tags integration ./test/otel/neuron/clean,terraform fmt -checkandterraform validateclean on the modified fixture.make lintandgo build ./...do fail, but only on pre-existing issues in files this PR does not touch —test/metric_value_benchmark/jmx_tomcat_jvm_test.go:62(staticcheck SA1006) andtest/metric_dimension/dimension_validation_helpers.go:82(undefined symbol) — both reproduce identically on an unmodifiedmain.inf2.xlargenode in both directions: 7 of 8 pass with the agent fix, and all 8 fail with it reverted, with the defect confirmed to reproduce first. The 8th,FixtureIsBothBurnWorkloads, asserts this terraform fixture's pod names and so needs this cluster.terraform applyhas not been run, so the specificneuron-burn-peerplacement — the device plugin granting one core to each pod andpodAffinityco-locating them — is unexercised on a fresh cluster. Thewait_neuron_monitorco-location gate fails loudly if it does not hold.translator/changes in this repootel-neuronfixture, raising that cluster's NeuronCore consumption.