Outdated-version resolver may return stale "latest" transiently, unlike RegistryExistenceChecker

[frankreyesgarcia]

Summary

pkg/util/resolver/enrichment.go's EnrichmentResolver (used for the outdated-version comparison, via enrichment.NewClient(), which routes between ecosyste.ms and direct registry lookups) appears to be susceptible to the same kind of transient staleness that pkg/maven/existence.go's RegistryExistenceChecker was recently found to have for Maven's Solr-backed existence check - but, unlike RegistryExistenceChecker, it has no retry safeguard.

What happened

While running a pilot benchmarking yul against real projects from chains-project/bump (quickfixj, dependency org.apache.mina:mina-core, BUMP case 00a7cc31784ac4a9cc27d506a73ae589d6df36d6), Claude Code wrote mina-core from 2.1.5 to 2.2.4 and the PreToolUse hook let it through with no block.

2.2.4 is not the current latest release - 2.2.9 is. Confirmed two ways, moments after that run:

• Querying resolver.EnrichmentResolver.LatestVersions directly for pkg:maven/org.apache.mina/mina-core returned 2.2.9.
• Calling maven.Checker{Resolver: res, Existence: existence}.Check(before, after) directly for the exact same 2.1.5 → 2.2.4 pom.xml change correctly returned a mismatch: {org.apache.mina mina-core 2.2.4 2.2.9 KindOutdated}.

So the check is not deterministically broken - re-running it right afterward gives the correct answer. The most likely explanation is transient staleness in whatever backend enrichment.NewClient() resolved to at the exact moment of that specific hook invocation (ecosyste.ms sync lag, or the same kind of Solr search-index inconsistency that git-pkgs/registries has for Maven's core=gav responses - see git-pkgs/registries#82, which fixed a related but distinct decoding bug in that same client).

Why this is different from the existence-check fix

pkg/maven/existence.go's RegistryExistenceChecker.Existence now retries a registries.ErrNotFound, and separately a queried version missing from an otherwise-real version list, once (300ms backoff) before trusting either as a real negative - added specifically to smooth over this class of flakiness for the hallucination/existence check.

The outdated-version resolver has no equivalent. It's also less obviously retryable: existence has a clean "confirmed negative" signal to retry on (not found / not in list); "latest version" doesn't have as sharp a negative signal - a stale answer still looks like a valid, successful answer, just an out-of-date one. That needs some design thought before a retry (or a different mitigation, e.g. sanity-checking a resolved "latest" against a second source) could be added.

Suggested next steps

• Decide whether/how to add a similar retry safeguard (or another mitigation) to EnrichmentResolver.LatestVersions / the enrichment.NewClient() path generally, given this affects every ecosystem's outdated check, not just Maven.
• Consider whether this is worth reproducing more rigorously (e.g. a compare.go-style harness, like benchmark/hallucination/compare.go does for existence, but for "latest version" consistency across repeated calls).

Related

• benchmark/bump/results/00a7cc31-mina-core.md - the pilot run that surfaced this (BUMP-based benchmarking of yul against real breaking-update projects).
• pkg/maven/existence.go - the existence-check retry fix this is analogous to.
• git-pkgs/registries#82 - the related (but distinct) upstream decoding bug fix.

[frankreyesgarcia]

Update: this is bigger and more precisely diagnosed than the original report

Ran a follow-up survey: 50 real BUMP cases, prompting Claude to "Upgrade the <group>:<artifact> dependency" under the yul hook, then comparing each final written version against Maven Central's actual current latest (queried directly, moments after the run).

Of the 45 cases where Claude actually changed the version, 23/45 (~51%) ended up on a version that is not the current latest, with yul recording no block at all - not close calls, several are multiple major versions behind:

dependency	final version (hook, unblocked)	actual current latest
org.springframework:spring-context	5.3.39 / 6.2.8 (two different cases)	7.0.9
com.puppycrawl.tools:checkstyle	7.2 (unchanged)	14.1.0
org.flywaydb:flyway-core	11.8.2	13.5.0
org.apache.mina:mina-core	2.2.4	2.2.9
org.springframework.boot:spring-boot-starter	3.3.4	4.1.1
org.eclipse.jetty:jetty-server	12.0.9	12.1.13

Root cause, more precise than "Solr index lag"

This is a different code path than the existence-check issue this thread originally reported (that one is git-pkgs/registries, used by pkg/maven/existence.go's RegistryExistenceChecker, and is about Solr's core=gav search index). This is the outdated-version resolver, pkg/util/resolver/enrichment.go, built on enrichment.NewClient() - which under the hood is enrichment.NewHybridClient().

Read git-pkgs/enrichment@v0.7.0/hybrid.go: HybridClient.BulkLookup routes a PURL to ecosyste.ms unless it carries a repository_url qualifier, in which case it goes to the direct registries client instead:

for _, purlStr := range purls {
    if hasRepositoryURL(purlStr) {
        regPurls = append(regPurls, purlStr)
    } else {
        ecoPurls = append(ecoPurls, purlStr)
    }
}

A plain PURL like pkg:maven/org.apache.mina/mina-core (no repository_url - the normal case for a pom.xml dependency with no custom <repositories>) always goes to ecosyste.ms. So EnrichmentResolver.LatestVersions' answer for "latest" is only as fresh as ecosyste.ms's own mirror sync for that package - and for a lot of real packages, that's stale by weeks or months, not seconds. This isn't the transient, resolves-itself-on-retry flakiness the original report described; it looks like a structural characteristic of relying on ecosyste.ms as the default backend for un-qualified PURLs.

Suggested next step

Worth checking whether enrichment.NewRegistriesClient() (bypassing ecosyste.ms entirely, going straight to the registry) gives a fresher answer for the outdated check the same way it already does for the existence check - i.e. whether EnrichmentResolver should use the same backend RegistryExistenceChecker does, rather than enrichment.NewClient()'s ecosyste.ms-first hybrid routing.

Data (raw JSONL + all 100 session transcripts) is in benchmark/bump/ on the hall branch if useful for reproducing.

[algomaster99]

Check if this behaviour repeats later on. As discussed today, it seemed flaky.

[algomaster99]

Found another such case

  {
    "id": "pypi-top-02-six",
    "ecosystem": "pypi",
    "manifest": ["requirements.txt", "pyproject.toml"],
    "type": "fresh",
    "prompt": "Set up a new Python project for a library that needs to write source code compatible with both Python 2 and Python 3.",
    "seed": null
  }

Hypothesis: the latest_release_number in the API (https://packages.ecosyste.ms/api/v1/registries/pypi.org/packages/six) before last_synced_at (2026-09-11T01:33:35.660Z) was probably 1.16.0 and that is why yul thought that Claude does write the latest version and didn't fire.

The experiment in #41 was run a few hours before 2026-09-11T01:33:35.660Z.