Conversation
9 tasks
miconeilaws
reviewed
Aug 31, 2026
| // collapsed onto one identity and a core pinned at 75% reported 0. | ||
| // | ||
| // TestNeuronMultiRuntimePreFixConfigLosesData pins that failure mode, so the | ||
| // assertions below are known to discriminate rather than merely pass. |
Collaborator
There was a problem hiding this comment.
This reads like an artifact from the fix you've done rather than future-understandable preface for a set of tests... LEts refactor it to focus the intention of the tests rather than the historical bug
Collaborator
Author
There was a problem hiding this comment.
Fair — rewritten to lead with the invariant the tests protect rather than the history of how it broke. The context: datapoint/last-write-wins mechanism moved onto the negative control, which is the one test genuinely about that configuration shape, so it's local to where it's load-bearing.
Also renamed PreFixConfigLosesData → CollapsedConfigLosesData, since "pre-fix" ages the same way the comment did, and gave PromotesPodIdentity the same treatment — it described itself as guarding "the six promote statements removed by the fix" instead of stating that pod identity must land on the resource and not stay on the datapoint.
miconeilaws
reviewed
Aug 31, 2026
| // renderedNeuronProcessors returns the processors block of the shipped neuron.yaml. | ||
| // Read from the embedded template rather than restated, so reverting the config | ||
| // fails this test. | ||
| func renderedNeuronProcessors(t *testing.T) map[string]any { |
Collaborator
There was a problem hiding this comment.
Hmm I wonder if this is really a substitute for the integration tests? Do we have other tests like this? Or what are you trying to cover here that you can't cover with an integration test?
Collaborator
Author
There was a problem hiding this comment.
Not a substitute — complementary, and there is precedent for the pattern: translator/translate/otel/processor/metricsdecorator/translator_unix_test.go:52 (TestMetricDecoration) does the same thing with the same imports, and its own comment describes it as "using a test TransformProcessor yaml configuration, starting the processor and having it consume test metrics." One deliberate difference: it loads a testdata yaml, whereas this renders the shipped neuron.yaml off the //go:embeded template, so reverting the real config fails this test.
Three things the integration test structurally can't do:
-
The negative control can only live here. CollapsedConfigLosesData runs the broken config to prove the assertions discriminate rather than merely pass. You can't deploy a deliberately broken agent in CI.
-
The integration test can't distinguish a collapse from a single-runtime node. I hit this when I reverted the agent on a live cluster to check the integration tests actually fail: the collapse destroys runtime_tag itself, so exactly one tag survives per node — structurally identical to a node that genuinely runs one runtime. The integration test has to bail out with a "check both of these causes" message rather than assert. Here the input is fixed, so it asserts an exact four-series expectation.
-
It fails on a config revert, in this repo's build. The integration test only ever sees whatever config happens to be deployed, and only if the cluster fixture is right.
The integration side is in a companion PR against amazon-cloudwatch-agent-test — that's the layer that proves it on real inf2 silicon.
spanaik
added a commit
that referenced
this pull request
Aug 31, 2026
… the bug Review feedback on #2263: the file's preface read as a post-mortem of the fix rather than something a future reader could use. Lead with the invariant every (core, runtime) reading must satisfy; keep the collapsed-config mechanism on the negative control, which is the one test that is genuinely about that shape. Rename PreFixConfigLosesData -> CollapsedConfigLosesData so the name describes the configuration rather than when it existed. No assertions changed. Also drop an internal cluster name from a comment.
Contributor
Binary Size Reportlinux/amd64
Notable changes:
linux/arm64
windows/amd64
Investigating size changesUse go-size-analyzer to compare binaries: GOEXPERIMENT=jsonv2 go install github.com/Zxilly/go-size-analyzer/cmd/gsa@latest
gsa diff --old <baseline-binary> --new <new-binary> |
spanaik
added
the
ready for testing
9 tasks
Contributor
|
This PR was marked stale due to lack of activity. |
Contributor
|
This PR was marked stale due to lack of activity. |
spanaik
added a commit
to spanaik/helm-charts
that referenced
this pull request
Sep 22, 2026
Review feedback from the CloudWatchAgentSpecs code-reviewer: nothing in this repo regression-protects the fix. flag_matrix.sh covers only the 3-flag OTEL gating matrix and never sets neuronMonitor.enabled=true, so either half of this change could be reverted silently. The agent-side Go tests in aws/amazon-cloudwatch-agent#2263 cover the processor semantics, but not what this chart renders. Adds tests/neuron_pipeline_shape.sh, same developer-run style as the existing scripts. It unescapes the agent CR's otelConfig and asserts, scoped to each processor block: runtime_tag is a groupbyattrs grouping key; the promote renders in resource context and not datapoint context; it holds exactly the rename and its cleanup, with no pod/namespace/ container re-promotion. Plus a neuronMonitor.enabled=false control, so a processor rename cannot make the other assertions pass against empty input. Verified by mutation: against main's pre-fix template, 6 of 11 checks fail. helm lint (with region set -- it needs one regardless of this change), flag_matrix.sh and yaml_document_injection.sh all still pass.
spanaik
added a commit
to spanaik/helm-charts
that referenced
this pull request
Sep 22, 2026
Review feedback: the four sed expressions in render_otel_config are load-bearing but unexplained. Names the order and purpose of each, and points at this function as the one to update if the chart changes how it serializes otelConfig. Not applied: the reviewer flagged the promote block as having dropped the `where attributes["runtime_tag"] != nil` guard from delete_key. It is still there -- _otel-container-insights-config.tpl:635 -- so the guard is symmetric with the sibling set, matching the agent copy in aws/amazon-cloudwatch-agent#2263.
On a node with more than one Neuron runtime, roughly half of every per-core
Neuron metric is silently replaced by zero before it leaves the agent. A
NeuronCore pinned at 75% utilization reports 0.
neuron-monitor itself is correct -- it emits the full (neuroncore x runtime_tag)
cross-product with only the matching pairs non-zero:
neuroncore_utilization_ratio{neuroncore="0",runtime_tag="...core0"} 0.7523 <- real
neuroncore_utilization_ratio{neuroncore="1",runtime_tag="...core0"} 0.0
neuroncore_utilization_ratio{neuroncore="0",runtime_tag="...core1"} 0.0
neuroncore_utilization_ratio{neuroncore="1",runtime_tag="...core1"} 0.7517 <- real
A debug exporter on the live pipeline shows what the agent then produces: ONE
ResourceMetrics holding FOUR datapoints with only TWO distinct attribute sets --
aws.neuron.core=0 Value: 75.154815 <- real
aws.neuron.core=1 Value: 0.000000
aws.neuron.core=0 Value: 0.000000 <- same identity, shadows the real one
aws.neuron.core=1 Value: 75.363968
Cause, in transform/cw_k8s_ci_v0_neuron_promote:
1. It runs in `context: datapoint` but writes `resource.attributes`. Resource
attributes are per-ResourceMetrics, so with N runtimes in one RM the statement
executes N times and the last write wins -- which is why a single arbitrary
runtime tag ends up labelling the whole node.
2. It then deletes `runtime_tag` from the datapoint. That was the only attribute
distinguishing a core's real reading from another runtime's zero for that same
core, so two datapoints collapse onto one identity with different values.
Nothing downstream reconciles them (awsattributelimit early-returns below 150
attributes; batch only concatenates), so both go over the wire and one is
arbitrarily dropped at ingest.
It fails silently, which is why it went unnoticed: the collision happens
in-agent, so the exported surface shows too FEW series rather than duplicated
ones. Cardinality is the signal, not duplication. Single-runtime nodes are
unaffected -- it needs >=2 Neuron runtimes on one node to appear.
Blast radius is wider than utilization: neuron_runtime_memory_used_bytes has 4
series upstream (2 memory_location x 2 runtime_tag) and arrives as 2, so one
runtime's memory is dropped as well. Confirmed on neuroncore_utilization_ratio
and all five neuroncore_memory_usage_* families.
Fix: make `runtime_tag` a groupbyattrs grouping key, so each runtime gets its own
ResourceMetrics. groupbyattrs MOVES its grouping keys, so that also removes the
key from the datapoint, each datapoint identity stays unique, and the
last-write-wins clobber becomes a no-op. The promote then moves to
`context: resource`, which is where the attribute now lives -- and is how the
DCGM pipeline already does it (containerinsights/dcgm.yaml:66-83).
The six pod-identity statements are dropped as dead code, not as a behaviour
change: groupbyattrs has already deleted those keys from the datapoint by then,
so `attributes["k8s.pod.name"]` is always nil and they never fire.
Trade-off: this publishes the full cross-product, so series count grows
multiplicatively with runtime count -- 2 -> 4 on a 2-core/2-runtime node, and a
32-core trn1 with 4 runtimes goes from 32 to 128 series per per-core metric. This
is the cardinality neuron-monitor already emits; the previous behaviour reduced it
only by discarding data. The alternative is to aggregate the runtime dimension
away in-agent (max per core), which keeps the count flat but loses per-runtime
attribution; rejected because the OTLP/PromQL surface can carry the dimension and
a runtime legitimately spans multiple cores, so "which runtime is using this
core" is a question this surface should be able to answer. Not measured beyond a
2-core node.
Scope: the EMF / Container Insights path is unaffected. neuron.yaml's pipeline
exports to the forward/opentelemetry connector, consumed only by
metrics/opentelemetry -> otlphttp/metrics. The EMF Neuron path is a separate
pipeline (awscontainerinsightreceiver -> gpuattributes -> awsemf/containerinsights)
sharing no component instance with this one; its own max-per-core reduction in
gpuattributes/internal/awsneuron_metric_modifier.go is unchanged and remains
correct for that surface, which does not publish runtime_tag as a dimension.
Tests: neuron_multiruntime_test.go instantiates the real groupbyattrs and
transform processors from the //go:embed-ed neuron.yaml, chains them, and feeds
synthetic metrics in the multi-runtime shape (2 cores x 2 runtimes, non-owning
pairs at zero, all four sharing one pod). No cluster or Neuron hardware needed.
KeepsEveryCorePerRuntime all four (core, tag, value) triples survive
SeparatesRuntimesIntoResources one ResourceMetrics per runtime
PromotesPodIdentity pod/ns/container on the resource, off datapoints
KeepsUntaggedDatapoints device/hardware metrics with no runtime_tag land
in a resource with no aws.neuron.runtime.tag,
rather than inheriting an arbitrary runtime's
CollapsedConfigLosesData negative control: the pre-fix config collapses to
one tag and two datapoints per core
Config is read from the embedded template rather than restated, so a revert fails
the tests. Verified by mutation: removing `- runtime_tag` from the groupbyattrs
keys fails four of the five. The negative control keeps the others honest -- if
the collapse ever stops reproducing it starts passing, signalling the assertions
have gone vacuous.
Verified on a live 2-core inf2.xlarge running two Neuron runtimes: before, 2
series with core 0 reading 0 while pinned at ~75%; after, 4 correctly-labelled
series with core 0 -> 75.15 under its own runtime tag and core 1 -> 75.23 under
the other. Reverting the config on the live agent reproduced the loss and
re-applying restored it, so the fix is doing the work rather than a caching
artifact. 8 existing Neuron integration tests / 26 subtests still pass, including
TestNeuronNoPromotedDatapointKeys, which covers the six removed statements.
The four regenerated golden configs are the snapshots. Integration coverage for
the multi-runtime case is a companion change:
aws/amazon-cloudwatch-agent-test#749
The EKS add-on takes an independent copy of this pipeline from the
amazon-cloudwatch-observability Helm chart, which carries the identical defect;
that fix is aws-observability/helm-charts#365. Both are needed.
spanaik
force-pushed
the
fix-neuron-per-core-multi-runtime
branch
from
September 22, 2026 12:14
503cfa8 to
6f6a885
Compare
spanaik
added a commit
to spanaik/helm-charts
that referenced
this pull request
Sep 22, 2026
…untimes On a node with more than one Neuron runtime, roughly half of every per-core Neuron metric is replaced by 0 before export. A NeuronCore pinned at 75% reports idle. neuron-monitor reports every core from every runtime: the runtime that owns a core publishes the real value, the others publish 0 for that same core. runtime_tag is the only attribute separating those datapoints. Two things in this template destroy it: 1. groupbyattrs/cw_k8s_ci_v0_neuron does not group on runtime_tag, so every runtime on the node lands in a single ResourceMetrics. 2. transform/cw_k8s_ci_v0_neuron_promote runs in `context: datapoint` while writing resource.attributes. Resource attributes are per-ResourceMetrics, so the statement executes once per datapoint and the last write wins -- then it deletes runtime_tag from the datapoint. The two datapoints for a core now share one identity, whichever arrives last wins, and for one core that is a legitimate-looking 0. It fails silently: the collision happens in-agent, so the exported surface shows too FEW series rather than duplicated ones. Cardinality is the signal, not duplication. A 2-core node emits 2 series where it should emit 4. Fix: add runtime_tag to the groupbyattrs keys so each runtime gets its own ResourceMetrics, and move the promote to `context: resource`, which is the correct context for writing resource attributes. The six pod/namespace/container statements are removed as dead code, not as a behaviour change: groupbyattrs MOVES its grouping keys onto the resource and deletes the datapoint copies, so there was never anything left for them to promote. Net effect is 8 statements -> 2. Customer-visible effect: this publishes the full (core x runtime_tag) cross-product, so series count grows multiplicatively with runtime count -- 2 -> 4 on a 2-core/2-runtime node, and a 32-core trn1 with 4 runtimes goes from 32 to 128 series per per-core metric. This is the cardinality neuron-monitor already emits; the previous behaviour reduced it only by discarding data, half of it silently replaced with 0. The alternative is to aggregate the runtime dimension away in-agent (max per core), which keeps the count flat but loses per-runtime attribution; rejected because a runtime legitimately spans multiple cores, so "which runtime is using this core" is a question this surface should be able to answer. Not measured beyond a 2-core node. No chart value is added, removed or renamed, and the change is inert on single-runtime nodes. Adds tests/neuron_pipeline_shape.sh, wired into `make all` alongside the other two scripts in tests/, since nothing in this repo regression-protected the shape: flag_matrix.sh covers only the 3-flag OTEL gating matrix and never sets neuronMonitor.enabled=true, so either half of this change could have been reverted silently. It unescapes the agent CR's otelConfig and asserts, scoped to each processor block, that runtime_tag is a grouping key, that the promote renders in resource and not datapoint context, and that it holds exactly the rename and its cleanup with no pod/namespace/container re-promotion. A neuronMonitor.enabled=false control stops a processor rename making the other assertions pass against empty input. Verified: `make all` exits 0. By mutation, 6 of the 11 checks fail against the pre-fix template. helm lint clean (it needs --set region regardless of this change; the Makefile target passes one), and flag_matrix.sh and yaml_document_injection.sh both still pass. No existing chart test or golden file asserted the previous shape. The agent repo has an independent copy of this pipeline (translator/.../containerinsights/neuron.yaml) with the identical defect. aws/amazon-cloudwatch-agent#2263 fixes that copy and adds Go unit tests that run the real groupbyattrs + transform processors over a synthetic 2-core x 2-runtime payload, including a negative control that pins the collapse. The two copies serve different paths: the agent's serves the translator (config built from cwagentconfig.json), and this one serves the EKS add-on (the operator's AmazonCloudWatchAgent.spec.otelConfig). Add-on-managed clusters take this path only, so #2263 alone does not fix them. Ideally both land together.
spanaik
added a commit
to spanaik/amazon-cloudwatch-agent-test
that referenced
this pull request
Sep 22, 2026
…re data loss The Neuron per-core data-loss defect fixed in aws/amazon-cloudwatch-agent#2263 only appears on a node running two or more Neuron runtimes, and no fixture here produced that shape -- so nothing in this repo could have caught it. neuron-burn-core is replicas: 1 holding one of inf2.xlarge's two NeuronCores, so its node has exactly one runtime. It also carries a self-referential requiredDuringScheduling podAntiAffinity on neuron-test=true, which actively prevents a second Neuron pod from joining it. Raising replicas to 2 does not help: the anti-affinity pushes the second replica onto the idle node, giving one runtime per node again and destroying the idle-node fixture. Two existing tests look like they should catch the defect and do not: - TestNeuronRuntimeTagInResourceScope passes on the broken agent. It asserts the tag exists, is non-empty, and is absent from the datapoint; all three hold, because the collapsed resource still carries one (wrong) tag. - TestNeuronNoDuplicateSeries passes because the collision happens in-agent. Duplicate identities are merged before export, so the surface shows too FEW series rather than duplicated ones. Cardinality is the signal, not duplication. Fixture (terraform/eks/daemon/otel-neuron/main.tf): adds a neuron-burn-peer Deployment taking the second core, co-located with neuron-burn-core via podAffinity, so one node runs two runtimes burning different cores. Three constraints, each of which breaks an existing test if changed: not named neuron-burn-core-* TestNeuronBurnWorkloadLabels and TestNeuronBurnCorePodColor match HasPrefix(pod, "neuron-burn-core") and assert workload.name / pod-color on it no neuron-test: "true" label neuron-burn-core's podAntiAffinity targets it, which would make the two mutually exclusive on a host and push neuron-burn-core onto the idle node podAffinity, not nodeSelector both node groups use var.instance_type, so a nodeSelector alone could land it on the idle node, breaking the uncorrelated-results fixture wait_neuron_monitor waits for both burn deployments to be Available and fails if they did not co-locate -- a silent split would make every multi-runtime test vacuously skip. The validator's burn-loop readiness wait also gates on neuron-burn-peer, symmetric with neuron-burn-core: Available only means the container is Running, and the trace compile that starts it emitting runtime_tag takes about a minute more. peer carries no neuron-test label, so every `-l neuron-test=true` diagnostic dump gets an explicit `-l app=neuron-burn-peer` companion query. Tests (test/otel/neuron/multi_runtime_test.go): 8 tests. The value-level one requires >=2 busy cores reported by >=2 runtime tags; the structural one requires cores x tags series, which catches a collapse even on an idle node where values cannot distinguish it. Also a tripwire asserting no series carries runtime_tag="DEFAULT", which is what the in-agent-aggregation alternative stamps -- changing that trade-off should mean changing the test, not a silent behaviour swap. Deliberately no assertion on distinct pod count: two runtimes need not be two pods, and one pod can host both, so requiring it would encode this fixture's shape rather than the invariant. That over-specification was caught by running against a real cluster. Validated against a live 2-runtime inf2.xlarge node in both directions: - With the agent fix: 7 of 8 pass. The 8th, FixtureIsBothBurnWorkloads, asserts the neuron-burn-core / neuron-burn-peer pod names and therefore needs this terraform cluster; the validation cluster used a differently-named workload. - With the fix reverted on the live agent: all 8 fail. The defect was confirmed to reproduce first -- 2 series instead of 4, one surviving runtime tag, core 0 reporting 0 while pinned at ~75% -- then restored and re-run to green. That negative control also exposed a diagnostic problem, now fixed: all 8 failed via the shared fixture guard rather than their own assertions, because the collapse destroys the runtime tag itself, so ">= 2 distinct tags" never holds. The guard's message previously blamed the fixture only. It now names both causes -- the defect and a missing fixture -- with the kubectl command to tell them apart, because a collapsed node is structurally indistinguishable from a genuine single-runtime node on the metrics surface. gofmt clean, go vet -tags integration ./test/otel/neuron/ clean, terraform fmt -check and terraform validate clean.
9 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
On a node running more than one Neuron runtime, roughly half of every per-core Neuron
metric is silently replaced by zero. A NeuronCore pinned at 75% utilization reports 0.
transform/cw_k8s_ci_v0_neuron_promoteruns incontext: datapointbut writesresource.attributes. Resource attributes are per-ResourceMetrics, so with N runtimesin one RM the statement executes N times and the last write wins. It then deletes
runtime_tagfrom the datapoint — the only attribute distinguishing one runtime's realreading for a core from another runtime's zero for that same core — so two datapoints
collapse onto one identity with different values.
A debug exporter on the pipeline shows the agent emitting four datapoints with only
two distinct attribute sets:
Impact on a multi-runtime Neuron node: a saturated core renders as idle, a
low-utilization alarm fires falsely, and an idle-reclaim decision could reclaim a busy
core. Confirmed on
neuroncore_utilization_ratioand all fiveneuroncore_memory_usage_*families.Single-runtime nodes are unaffected, which is why this went unnoticed — it needs ≥2
Neuron runtimes on one node to appear.
neuron-monitoris not at fault. Scraping its own/metricsendpoint at the sameinstant shows it emitting the full
(core × runtime_tag)cross-product correctly, withonly the matching pairs non-zero.
Description of changes
Aggregation is not needed —
groupbyattrsalready does the right thing, it just wasn'tgiven the runtime dimension.
runtime_tagto thegroupbyattrs/cw_k8s_ci_v0_neuronkeys, so each runtimelands in its own
ResourceMetricsand the datapoints can no longer collide.transform/cw_k8s_ci_v0_neuron_promoteto acontext: resourcerename ofruntime_tag→aws.neuron.runtime.tag.The six
set/delete_keystatements fork8s.pod.name,k8s.namespace.nameandk8s.container.nameare removed as redundant: those keys are already in thegroupbyattrskeys, andgroupbyattrsmoves its grouping keys — seeprocessor/groupbyattrsprocessor/processor.go:so the promote was re-doing work already done one processor earlier, and its
datapoint-context write was the vehicle for the defect.
Net effect: 11 lines changed in
neuron.yamlplus the four regenerated golden configs.Trade-off
This publishes the full cross-product, so series count grows multiplicatively with
runtime count — 2 → 4 on a 2-core/2-runtime node; a 32-core
trn1with 4 runtimes goes32 → 128 series per per-core metric. This is the cardinality
neuron-monitoralreadyemits; the previous behaviour reduced it only by discarding data.
The alternative is to aggregate the runtime dimension away in-agent (
maxper core),which keeps the count flat but loses per-runtime attribution. Rejected here because the
PromQL/OTLP surface can carry the runtime dimension, and a runtime legitimately spans
multiple cores, so "which runtime is using this core" is a question this surface should
be able to answer.
Reviewers may want to weigh in on whether the cardinality growth is acceptable for large
trn1topologies. It has not been measured beyond a 2-core node.Scope
This does not affect the EMF / Container Insights path.
neuron.yaml's pipelineexports to the
forward/opentelemetryconnector, which is consumed only bymetrics/opentelemetry→otlphttp/metrics. In the combined v1+v2 config the solepipeline touching an EMF exporter is
logs/emf_logs(receiverudplog, one batchprocessor, no Neuron processors). The EMF Neuron path is a separate pipeline —
awscontainerinsightreceiver→gpuattributes→awsemf/containerinsights— and sharesno component instance with this one. Its own
max-per-core reduction ingpuattributes/internal/awsneuron_metric_modifier.gois unchanged and remains correctfor that surface, which does not publish
runtime_tagas a dimension.This alone does not fix EKS add-on clusters. The
amazon-cloudwatch-observabilityHelm chart carries an independent copy of this same pipeline in
templates/linux/_otel-container-insights-config.tplwith the identical defect, and thatis the copy the EKS add-on renders into
AmazonCloudWatchAgent.spec.otelConfig. This PRfixes the translator path (config built from
cwagentconfig.json); the chart fix isaws-observability/helm-charts#365. Both are needed.
License
By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.
Tests
New unit tests —
translator/translate/otel/pipeline/opentelemetry/containerinsights/neuron_multiruntime_test.go.Instantiates the real
groupbyattrsandtransformprocessors from the//go:embededneuron.yaml, chains them, and feeds synthetic metrics in the multi-runtime shape(2 cores × 2 runtimes, non-owning pairs at zero, all four sharing one pod). No cluster or
Neuron hardware required; runs in ~25ms.
KeepsEveryCorePerRuntime(core, tag, value)triples survive — both 75.x readingsSeparatesRuntimesIntoResourcesResourceMetricsper runtimePromotesPodIdentityKeepsUntaggedDatapointsruntime_tagland in a resource with noaws.neuron.runtime.tag, rather than inheriting an arbitrary runtime'sCollapsedConfigLosesDataConfig is read from the embedded template rather than restated, so a revert fails the
test. Verified by mutation: removing
- runtime_tagfrom thegroupbyattrskeys inneuron.yamlfails four of the five.The negative control matters because it keeps the other three honest — if the collapse
ever stops reproducing,
CollapsedConfigLosesDatastarts passing and tells us theassertions have gone vacuous.
Existing suites —
go test ./translator/tocwconfig/...and./translator/translate/otel/pipeline/opentelemetry/containerinsights/...pass. The fourgolden configs in this diff are the regenerated snapshots.
Live cluster — verified on a 2-core
inf2.xlargerunning two Neuron runtimes:75.23 under the other.
re-applying restored it. So the fix is doing the work, not a caching artifact.
including
TestNeuronPodName,TestNeuronNamespace,TestNeuronDeviceAttributesandTestNeuronNoPromotedDatapointKeys— the last of which specifically assertspod/namespace/container are not left on the datapoint, covering the six statements
removed here.
Integration test coverage for the multi-runtime case is a companion change to
amazon-cloudwatch-agent-test(adds a second co-located burn deployment so a node runstwo runtimes, plus 8 assertions on the resulting series shape). Worth noting that the
pre-existing integration tests could not catch this defect:
TestNeuronRuntimeTagInResourceScopepasses on the broken code because the collapsedresource still carries one tag, and
TestNeuronNoDuplicateSeriespasses because thecollision happens in-agent — the surface shows too few series rather than duplicated
ones. Cardinality is the signal, not duplication.
aws/amazon-cloudwatch-agent-test#749
Requirements
make fmtandmake fmt-sh— no changes producedmake lint— 0 issues (license and import-order checks pass)PR checklist
makepasses locally —make test: all packages pass, 0 failures;make lint: 0 issues (license + import-order clean);go build ./...clean. The-coverprofilestep ofmake test_and_buildaborts locally on a missingcovdatatool in the Go 1.25.8 install; reproduces identically on an unmodifiedmain, so it is environmental.Check changes,MakeBinaryand the unit-test jobs pass. 13 EC2 Linux jobs onsles-16anddebian-12fail atterraform applywithremote-exec provisioner error: timeout — dial tcp <ip>:22: connect: connection refused— the freshly-launched EC2 host is unreachable over SSH, a different IP each time. PRs Add Windows Azure VM integration test job #2296 and fix windows workload-discovery integ test #2297 show the same 14 failures in the same suites, so it is repo-wide. Those suites are EC2 host tests (ca_bundle,cloudwatchlogs,otlp,restart,ssm_document, …); this change only alters the EKS Neuron branch of the OTEL Container Insights translator and cannot affect SSH reachability of an EC2 instance.inf2, which that matrix does not cover. Evidence is live-cluster validation on a 2-coreinf2.xlargerunning two Neuron runtimes, in both directions: core 0 goes 0 → 75.15 with the fix; reverting the config on the live agent reproduces the loss and re-applying restores it. Detailed under Tests.neuron_multiruntime_test.go, includingCollapsedConfigLosesDataas a negative control pinning the defect. Config is read from the embedded template, so a revert fails the tests; verified by mutation (removing- runtime_tagfails 4 of 5).mainis exactly the 11-line neuron change.