fix(backend): Attempt at fixing critical deps - #3960
Conversation
Clears the last three critical advisories. All within-major.
…d, postcss, svgo, next Single-major highs, one override key each.
…ml, minimatch, rollup, tar-fs, vite, ws Each major line gets its own bounded key. Several of these packages have a clean newer major in the tree (minimatch 10, tar-fs 3, rollup 4, vite 7.3.6) that an unscoped override would have downgraded. immutable 3.8.3 and vite 4.5.14 have no fix inside their own major, so they stay. Listed in the PR description.
Both are declared deps, so the package.json files move alongside the override rather than drifting from it. astro stays on ~6.4.x. Its remaining advisories only patch in 7.x, which is a major and out of scope here.
Both riskier than the rest. sharp ships native binaries so CI linux is the real test, not local darwin. The propagator override is scoped to 2.x. The 1.25.1 copy comes from sdk-trace-node@1.25.1, which backend still pins on the otel v1 line; moving that is an otel migration, not a dep bump.
…2 migration GHSA-45rx-2jwx-cxfr affects @opentelemetry/propagator-jaeger 1.25.1, which comes in transitively via the backend's @opentelemetry/sdk-trace-node ^1.25.1. The only real fix is moving the backend's otel SDK deps (resources, sdk-metrics, sdk-trace-node) to the v2 line, which drops propagator-jaeger entirely. I attempted that, plus the exporter realignment it needs (the backend already has @opentelemetry/sdk-node@0.217.0 pulling in v2-era otlp-transformer alongside the v1-era exporter-trace-otlp-grpc/exporter-metrics-otlp-grpc, which crashes trace export outright due to clashing protobufjs globals). Bumping the two grpc exporters to ^0.221.0 to match sdk-node "fixed" that crash but traded it for a different one: v1's Span still has no instrumentationScope field, which the v2-era serializer requires, so export still fails, just with a different stack. Decided not to chase the SDK v2 migration for this pass. Ignoring the GHSA via pnpm.auditConfig.ignoreGhsas instead, so it's a stated, visible suppression rather than a silently reverted fix. `pnpm audit` still lists it, just tagged "(1 ignored)". Separately: telemetry tracing looks broken on this branch (and likely main) any time ENABLE_TELEMETRY_TRACES=true, independent of this GHSA - the sdk-node bump in an earlier merged Renovate PR didn't realign the grpc exporters, so trace export throws on the protobufjs global clash described above. Nothing catches this today since service.test.ts mocks the otel packages wholesale and CI never enables telemetry. Worth a separate look by whoever owns telemetry.
immutable 3.8.3 has two highs and no patched 3.x, so the only way out is to move the thing that depends on it. relay-compiler 13.0.2 wants immutable ^5.1.9, which the existing >=5.0.0 override already covers. relay-operation-optimizer 7.0.0 asks for relay-compiler ^12, but its own latest (7.1.8) asks for ^13.0.2 on the same optimizer major, so the API it uses didn't move. Codegen output is unchanged. Dropped the immutable@>=3.0.0 override since nothing wants immutable 3 now.
vite 4.5.14 comes in under @vanilla-extract/integration 6.2.1 via remix's classic compiler in mock-ASE. There's no vite this could upgrade to inside a range 6.x accepts, and vite 5 is out of support so GHSA-fx2h has no 5.x patch. integration v8 has no vite dependency at all, so vite just leaves. v8 renamed createCompiler to compile and remix 2.17.5 still calls createCompiler, but it's only reached from the .css.ts / .vanilla.css onLoad handlers, and the plugin bails out earlier anyway unless the app declares @vanilla-extract/css. mock-ASE does neither. This does become a landmine if mock-ASE ever adopts vanilla-extract.
external-editor is at its latest (3.1.0) and still pins tmp ^0.0.33, so the >=0.2.0 selector never reached that copy. No upstream fix to wait for. external-editor only calls tmp.tmpNameSync, which 0.2.7 still exports with the same signature, and the path is inquirer's editor prompt inside graphql-codegen which never runs interactively here. Forcing it across the 0.x major is the only option that clears GHSA-ph9p-34f9-6g65.
✅ Deploy Preview for brilliant-pasca-3e80ec canceled.
|
🚀 Performance Test ResultsTest Configuration:
Test Metrics:
📜 Logs |
Every Grype and Trivy scan was failing on the same 5 highs plus a critical, and none of them were ours. Trivy's paths gave it away: they all sit under /usr/local/lib/node_modules/npm in the base image. node:24-alpine3.23 ships npm 11.17.0, which bundles tar 7.5.16, brace-expansion 5.0.6, ip-address 10.2.0 and undici 6.26.0. I confirmed those versions by hand in the base image and they match the scanner output exactly. That's why the last few override bumps didn't move the needle. Our lockfile already resolves tar 7.5.22 / brace-expansion 5.0.9 / undici 6.28.0, but a pnpm override can't touch a copy of npm baked into the runner image. The runner stages only ever exec node (frontend goes through node_modules/.bin/remix-serve), corepack isn't enabled there, and nothing in localenv or helm shells out to npm inside these containers, so npm is just dead weight. Removing it takes out the whole class of finding rather than this month's batch of it. Built backend and frontend locally and scanned both tarballs: Trivy and Grype both exit 0 at the high cutoff now, and npm no longer shows up anywhere in the scanned path list. Grype passed even without .grype.yaml loaded, so it's clean on its own rather than by suppression. I left the existing ignore rules alone, though some of them may well be stale now.
The backend image had one high left that was actually ours: @opentelemetry/propagator-jaeger 1.25.1, CVE-2026-59892. The suppression I added earlier only went into pnpm.auditConfig.ignoreGhsas, which covers pnpm audit and nothing else, so Trivy kept failing on it. Turns out this doesn't need the otel v2 migration I backed away from last time. The backend already installs propagator-jaeger 2.10.0 via sdk-node 0.217.0, sitting right next to the 1.25.1 that sdk-trace-node 1.25.1 pulls in. Both were already in the tree. So I widened the override, which was scoped to @>=2.0.0 and therefore matched nothing, to just pin the package, and the two collapse onto 2.10.0. sdk-trace-node 1.x only ever constructs JaegerPropagator and registers it as a TextMapPropagator, so the v1/v2 core split underneath doesn't matter here. Checked it directly: NodeTracerProvider.register() with the v2 propagator works, and an uber-trace-id round-trips through extract/inject with the trace id, span id and flags intact. All 34 telemetry tests pass. Only fixes the one CVE. @opentelemetry/core 1.22.0 and 1.25.1 are still in there under GHSA-8988-4f7v-96qf, but that one's medium and below the scan cutoff, and clearing it does need the SDK v2 move.
Checked whether a newer tag would let us drop the npm removal. It won't. Every node:24 alpine tag (3.22, 3.23, 3.24 and the floating 24-alpine) is node 24.19.0 with npm 11.17.0 and the same bundled tar 7.5.16 / brace-expansion 5.0.6 / ip-address 10.2.0. Going further up doesn't fix it either. The newest published node image (node:alpine, 26.7.0) ships npm 11.19.0, which is still brace-expansion 5.0.7 and ip-address 10.2.0. Both remain vulnerable to the highs that fail us: GHSA-rgw5 wants 5.0.9, GHSA-mh99 wants 5.0.8, GHSA-mwp4 wants ip-address 10.3.1. So there's no base image at any node version that passes the high cutoff right now, and we're on node 24 per engines anyway. Just trimming the comment down and folding that finding into it, since it's repeated in five files and it's what the next person will want to know.
Three leftovers, none of which match anything anymore. pnpm.auditConfig.ignoreGhsas held GHSA-45rx-2jwx-cxfr for propagator-jaeger 1.25.1. That version is gone from the tree since the override collapsed it onto 2.10.0, so the entry can't match. pnpm audit reports the same 8 low and 23 moderate with it removed, and nothing tagged as ignored. .grype.yaml and .trivyignore both carried CVE-2026-12151 (plus GHSA-vxpw-j846-p89q on the grype side). Scanned all five prod images with no ignore config at all, across every severity, and neither ID shows up. Running grype with the config still loaded reports zero matches ignored, where CI was logging 6 before, so what those rules were catching is fixed now. Removed the files rather than leaving empty ones. Easy enough to re-add, and #3947 set the precedent of pruning these as they get fixed. All five images pass both scanners at CI's exact settings with the files gone.
|
|
||
| # npm isn't used at runtime and no node:24 tag ships a patched one, so its bundled | ||
| # deps (tar, brace-expansion, ip-address, undici) fail every image scan. | ||
| RUN rm -rf /usr/local/lib/node_modules/npm /usr/local/bin/npm /usr/local/bin/npx |
There was a problem hiding this comment.
There are at least 2 loose ends if we remove npm. interledger/helm-charts and packages/documentation/src/content/docs/admin/admin-user-guide.mdx reference npm in some commands. I think we could simply update both to use node --run instead.
Im unsure to what extent interledger/helm-charts are being used but we should update it either way I guess. Here is one of the places (the other being auth): https://github.com/interledger/helm-charts/blob/main/charts/rafiki-backend/templates/deployment.yaml#L45
There was a problem hiding this comment.
In terms of removing npm in principle, I think it's a good idea. We shouldn't need it at runtime and it only increases the vulnerability surface.
Alternatively we could maybe use a distroless runner but I think that would require some more changes to the docker files and in frontend we use sh which would need a workaround. gcr.io/distroless/nodejs24
There was a problem hiding this comment.
minor thing, but i updated this to fail if npm is still found (if npm at a different path rm silently returns OK).
There was a problem hiding this comment.
interledger/helm-charts directory is old (we actually should remove this, since we have everything in interledger/charts repository), and updating to use node for the user invitation script in the docs make sense as well, if that is possible.
| run: | | ||
| docker images | ||
| /tmp/trivy image --db-repository ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db --java-db-repository ghcr.io/aquasecurity/trivy-java-db,public.ecr.aws/aquasecurity/trivy-java-db --ignore-unfixed --format table --vuln-type os,library --exit-code 1 --severity HIGH --input /tmp/${{ github.sha }}-${{ matrix.package }}-${{ matrix.platform.name }}-${{ needs.version-generator.outputs.version }}.tar | ||
| /tmp/trivy image --db-repository ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db --java-db-repository ghcr.io/aquasecurity/trivy-java-db,public.ecr.aws/aquasecurity/trivy-java-db --ignore-unfixed --format table --vuln-type os,library --exit-code 1 --severity HIGH,CRITICAL --input /tmp/${{ github.sha }}-${{ matrix.package }}-${{ matrix.platform.name }}-${{ needs.version-generator.outputs.version }}.tar |
There was a problem hiding this comment.
--severity is a list, not a floor. we werent scanning for critical vulnerabilities with Trivy
| # update packages to patch known vulnerabilities that haven't made it into base image yet | ||
| RUN apk upgrade --no-cache |
There was a problem hiding this comment.
openssl had a vulnerability. It has been fixed but there is no alpine tag that includes it yet.
I think using distroless could have sidestepped this one as well.
This is a pattern I've seen elsewhere. Such as https://github.com/jupyterhub/configurable-http-proxy/blob/14568e1b426c4818317da567a8f8f4d19a7c28a2/Dockerfile#L11-L13
Trivy's org updates images like this as well: https://github.com/aquasecurity/kube-hunter/blob/bc47f08e88ea2a5fb059bf3b8a8edb1aefb4c6cc/Dockerfile#L10
| "lodash": ">=4.18.0", | ||
| "minimatch@>=3.0.0 <4.0.0": "^3.1.4", | ||
| "minimatch@>=9.0.0 <10.0.0": "^9.0.7", | ||
| "next": "15.5.24", |
There was a problem hiding this comment.
This one hurts.
Next is a dependency in our frontend. Our Remix frontend. We have this entirely seperate, large, frontend framework because its a dependency of @ory/integrations. Which we use for 2 utility functions.
This transitive next dependencies is a source of a few critical vulnerabilities and is 152.6MB of the final 455.2MB docker image. So this is kind of absurd.
Seems like we could probably just inline those utility function but Im not sure how likely that would come back to bite us (eg. ory changes integrations and client together, we dont get the integrations change, something doesn't work). I think it's related to CSRF.
And looking at @ory/integrations (https://github.com/ory/integrations) it looks purpose-built for next and also archived... so I'm really not sure we should be using it.
There was a problem hiding this comment.
could probably do this and drop @ory/integration (and next):
// packages/frontend/app/lib/kratos-ui.ts
import { UiNodeGroupEnum } from '@ory/client'
import type { UiNode, UiNodeAttributes, UiNodeInputAttributes } from '@ory/client'
export function formNodes(nodes: UiNode[], method: UiNodeGroupEnum): UiNode[] {
return nodes.filter(
(n) => n.group === method || n.group === UiNodeGroupEnum.Default
)
}
export function isInputNode(
attrs: UiNodeAttributes
): attrs is UiNodeInputAttributes & { node_type: 'input' } {
return attrs.node_type === 'input'
}There was a problem hiding this comment.
removed next.
i also noticed the checkAuthAndRedirect in all the loaders could now be done centrally in the new remix version (which is actually react router v7 -- remix was merged into that). That's a whole other can of worms though - I left a code comment.
219fcbd#diff-57b68df4807ecdfd319935e4b00ab56a223f5bae7559b62c65d42b717a6e05a1R30-R32
Changes proposed in this pull request
Clearing the critical and high advisories. Everything here is a lockfile or override change except the last few commits, which touch the prod Dockerfiles.
bad77efa5bump @babel/core, protobufjs, yaml, webpack, tmp, tar, undici. Covers the seven security PRs that still needed work: chore(deps): update dependency @babel/core to v7.29.6 [security] #3954, chore(deps): update dependency protobufjs to ^7.6.5 [security] #3952, chore(deps): update dependency yaml to v2.8.3 [security] #3902, chore(deps): update dependency webpack to v5.104.1 [security] #3826, chore(deps): update dependency tmp to v0.2.7 [security] #3953, chore(deps): update dependency tar to >=7.5.20 [security] #3867, chore(deps): update dependency undici@>=6.0.0 to ^6.25.0 [security] #3849. tmp is scoped to>=0.2.0here so the 0.0.33 copy under inquirer doesn't get dragged across a major.f5ccf8827handlebars, shell-quote, websocket-driver. Clears the last three criticals, all within-major.a22fb7fc4babel systemjs transform, devalue, fast-uri, flatted, postcss, svgo, next. Single-major highs, one override key each.2bed11910range-scoped bumps for brace-expansion, immutable, js-yaml, minimatch, rollup, tar-fs, vite, ws. Each major line gets its own bounded key rather than one unscoped override. Several of these packages already have a clean newer major in the tree (minimatch 10, tar-fs 3, rollup 4, vite 7.3.6) that an unscoped override would have downgraded.9a4595857astro 6.4.6, axios 1.18.0. Both are declared deps, so the package.json files move alongside the override instead of drifting from it. astro stays on ~6.4.x, since its remaining advisories only patch in 7.x.25cf61916sharp 0.35.0, plus a jaeger propagator override scoped to 2.x. sharp ships native binaries, so CI linux is the real test here, not local darwin.3f3081459suppress GHSA-45rx-2jwx-cxfr throughpnpm.auditConfig.ignoreGhsasinstead of chasing an otel v2 migration. I tried the migration and backed out; the reasoning is in the commit message. This one gets undone by930c7726candd44848a99, see Context.aeccbfaa9relay-compiler 13, to drop immutable 3. immutable 3.8.3 has two highs and no patched 3.x, so the only way out is moving the thing that depends on it. relay-operation-optimizer 7.0.0 asks for relay-compiler ^12, but its own latest (7.1.8) asks for ^13.0.2 on the same optimizer major, so the API it uses didn't move. Codegen output is unchanged.ada1ae50b@vanilla-extract/integration 8, to drop vite 4. vite 4.5.14 comes in under integration 6.2.1 via remix's classic compiler in mock-ASE. There's no vite it could move to inside a range 6.x accepts, and vite 5 is out of support so GHSA-fx2h has no 5.x patch. integration v8 has no vite dependency at all, so vite just leaves. Caveat: v8 renamedcreateCompilertocompileand remix 2.17.5 still callscreateCompiler, but that's only reached from the.css.ts/.vanilla.cssonLoad handlers, and the plugin bails out earlier anyway unless the app declares @vanilla-extract/css. mock-ASE does neither. It does become a landmine if mock-ASE ever adopts vanilla-extract.1dc9895bdunscope the tmp override so it catches 0.0.33. external-editor is at its latest (3.1.0) and still pins tmp ^0.0.33, so the>=0.2.0selector never reached that copy and there's no upstream fix to wait for. external-editor only callstmp.tmpNameSync, which 0.2.7 still exports with the same signature, and the path is inquirer's editor prompt inside graphql-codegen, which never runs interactively here.75a5d9c35drop npm from the prod runner images. This is the one that actually fixed the scan jobs, details below.930c7726ccollapse @opentelemetry/propagator-jaeger onto 2.10.0.3f7a50719record why bumping the base image isn't an alternative to the npm removal.d44848a99delete the scanner suppressions that are now dead.Context
The override bumps cleared
pnpm audit, but the Grype and Trivy jobs kept failing, and several rounds of bumping them further changed nothing. The reason is that the packages they were failing on weren't ours.Trivy's file paths gave it away: every finding sat under
/usr/local/lib/node_modules/npm.node:24-alpine3.23ships npm 11.17.0, which bundles tar 7.5.16, brace-expansion 5.0.6, ip-address 10.2.0 and undici 6.26.0. I checked those versions in the base image by hand and they match the scanner output exactly. That's 5 highs plus a critical in all five images. Our lockfile already resolves tar 7.5.22, brace-expansion 5.0.9 and undici 6.28.0, but a pnpm override can't reach a copy of npm baked into the runner image, so none of the bumps were ever going to help.The runner stages only exec
node(frontend goes throughnode_modules/.bin/remix-serve), corepack isn't enabled there, and nothing in localenv or helm shells out to npm inside these containers. So npm is dead weight and removing it takes out the whole class of finding rather than this month's batch of it.I did check whether a newer base tag would be a cleaner fix than deleting npm in five Dockerfiles, and it isn't an option today. Every node:24 alpine tag (3.22, 3.23, 3.24, and the floating
24-alpine) is node 24.19.0 with npm 11.17.0 and identical bundled versions. Going further up doesn't help either: the newest published node image,node:alpineat 26.7.0, ships npm 11.19.0, which is still brace-expansion 5.0.7 and ip-address 10.2.0. Both are still vulnerable to the highs that fail us, since GHSA-rgw5 wants 5.0.9, GHSA-mh99 wants 5.0.8 and GHSA-mwp4 wants ip-address 10.3.1. So no base image at any node version passes the high cutoff right now, and engines pins us to node 24 regardless.Backend had one more high that was genuinely ours: @opentelemetry/propagator-jaeger 1.25.1, CVE-2026-59892. The suppression in
3f3081459only went intopnpm.auditConfig.ignoreGhsas, which coverspnpm auditand nothing else, so Trivy kept failing on it. It turns out this didn't need the otel v2 migration I backed away from: the backend already installs propagator-jaeger 2.10.0 via sdk-node 0.217.0, right next to the 1.25.1 that sdk-trace-node 1.25.1 pulls in. Both were already in the tree. Widening the override (it was scoped to@>=2.0.0and so matched nothing) collapses them onto 2.10.0. sdk-trace-node 1.x only constructsJaegerPropagatorand registers it as aTextMapPropagator, so the v1/v2 core split underneath doesn't matter here.With that fixed, three suppressions were left pointing at nothing, so
d44848a99removes them. TheignoreGhsasentry can't match since propagator-jaeger 1.25.1 is gone from the tree..grype.yamland.trivyignoreboth carried CVE-2026-12151, plus GHSA-vxpw-j846-p89q on the grype side, and neither ID shows up in any of the five images at any severity. Running grype with the config still loaded now reports zero matches ignored, where CI was logging 6 before. I deleted the two files rather than leave them empty, following the pruning #3947 did.How this was verified
NodeTracerProvider.register()with the v2 propagator works, and anuber-trace-idround-trips through extract/inject with trace id, span id and flags intact. All 34 backend telemetry tests pass.pnpm auditis down to 8 low and 23 moderate, no highs or criticals, and unchanged by dropping theignoreGhsasentry.I only scanned amd64 locally. CI covers arm64 as well.
One thing worth knowing for future scan triage: grype reports
monorepo-symlink-testunder GHSA-2jcg-qqmg-46q6 as critical. It's a fixture with no fix available, so CI'sonly-fixedfilters it out, but it does show up if you run grype by hand without that flag.Still outstanding
Separately, and not caused by anything in this PR: telemetry tracing looks broken on main any time
ENABLE_TELEMETRY_TRACES=true. The backend has sdk-node 0.217.0 pulling v2-era otlp-transformer alongside the v1-era exporter-trace-otlp-grpc / exporter-metrics-otlp-grpc, and trace export throws on clashing protobufjs globals. Bumping the two grpc exporters to ^0.221.0 fixes that crash but trades it for another, since v1'sSpanhas noinstrumentationScopefield and the v2-era serializer requires it. Nothing catches this today becauseservice.test.tsmocks the otel packages wholesale and CI never enables telemetry. Worth a look by whoever owns telemetry.Checklist
fixes #numberuser-docslabel (if necessary)