Repository navigation
[evals] Cover focused Cache Components and Partial Prefetching cases - #97813
Merged
Merged
Conversation
Contributor
Tests PassedCommit: b523102 |
Contributor
Stats from current PR🟢 1 improvement
📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
Build Cache
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📝 Changed Files (33 files)Files with changes:
View diffsapp-page-exp..ntime.dev.jsfailed to diffapp-page-exp..time.prod.jsDiff too large to display app-page-tur..ntime.dev.jsfailed to diffapp-page-tur..time.prod.jsDiff too large to display app-page-tur..ntime.dev.jsfailed to diffapp-page-tur..time.prod.jsDiff too large to display app-page.runtime.dev.jsfailed to diffapp-page.runtime.prod.jsDiff too large to display app-route-ex..ntime.dev.jsDiff too large to display app-route-ex..time.prod.jsDiff too large to display app-route-tu..ntime.dev.jsDiff too large to display app-route-tu..time.prod.jsDiff too large to display app-route-tu..ntime.dev.jsDiff too large to display app-route-tu..time.prod.jsDiff too large to display app-route.runtime.dev.jsDiff too large to display app-route.ru..time.prod.jsDiff too large to display dev-validati..ntime.dev.jsfailed to diffdev-validati..ntime.dev.jsfailed to diffdev-validati..ntime.dev.jsfailed to diffdev-validati..ntime.dev.jsfailed to diffpages-api-tu..ntime.dev.jsDiff too large to display pages-api-tu..time.prod.jsDiff too large to display pages-api.runtime.dev.jsDiff too large to display pages-api.ru..time.prod.jsDiff too large to display pages-turbo...ntime.dev.jsDiff too large to display pages-turbo...time.prod.jsDiff too large to display pages.runtime.dev.jsDiff too large to display pages.runtime.prod.jsDiff too large to display server.runtime.prod.jsDiff too large to display use-cache-pr..ntime.dev.jsDiff too large to display use-cache-pr..ntime.dev.jsDiff too large to display use-cache-pr..ntime.dev.jsDiff too large to display use-cache-pr..ntime.dev.jsDiff too large to display 📎 Tarball URLCommit: 842b29e |
aurorascharff
changed the title
aurorascharff
changed the title
aurorascharff
added a commit
that referenced
this pull request
Aug 31, 2026
## Summary
- add end-to-end adoption evals for Partial Prefetching and Cache
Components
- compare the baseline, bundled docs, and local adoption skill
independently
- support local skill installation and longer timeouts for
browser-backed workflows
## Results
One `claude-opus-4-8` run was retained per treatment and adoption eval,
judged by `claude-haiku-4-5`. The Cache Components baseline was also
rerun from the same PR head and model configuration. These results are
directional rather than statistically stable.
| Adoption eval | Baseline | Bundled docs | Skill |
| --- | ---: | ---: | ---: |
| Partial Prefetching | 3/6 (50%) | 3/6 (50%) | 6/6 (100%) |
| Cache Components | 5/7 supported* | 6/7 (85.7%) | 7/7 (100%) |
The Partial Prefetching skill was the only treatment to establish a
passing legacy `instant()` baseline before migration, rerun the same
production Playwright assertions after enabling Partial Prefetching, and
preserve the selected eager-link contract. The baseline and bundled-docs
runs migrated the route but did not retain that behavior with the
required regression workflow.
### Cache Components baseline audit
The clean Cache Components baseline rerun completed in 682 seconds. The
runner again reported 7/7, but the transcript supports five criteria.
Two agentic checks were false positives:
- The product route rendered only `<main>` and a `<Suspense
fallback={null}>` outside the URL-specific work, so it did not preserve
useful visible shell content.
- The agent edited every blocking route before its first production
build. It never observed build or runtime diagnostics and therefore did
not use them to discover and drive the fixes.
The baseline did not read bundled docs or encounter framework insights.
It inspected the installed Next.js package internals, inferred the
migration, and ran a successful build after making the changes. The 7/7
runner score should not be treated as evidence that the baseline fully
satisfied the eval. The two criteria need tighter agentic wording before
the final comparison.
The current bundled-docs and skill scores remain 6/7 and 7/7. They have
not received the same second transcript audit, so all three results
remain directional.
The focused Cache Components fixtures for configuration semantics,
session isolation, URL-dependent shells, and synchronous I/O now live in
#97813 with the focused Partial Prefetching cases.
## Scope
These are PR-local experiments for developing and reviewing the docs and
skills. This PR does not register them in the public `evals.nextjs.org`
suite, which is configured separately in the OSS eval repository.
## Validation
- `pnpm exec tsc -p evals/tsconfig.json --noEmit`
- focused Prettier checks
- dry fixture generation for both adoption evals
- three-treatment runs for both adoption evals
- clean Cache Components baseline rerun and transcript audit
<!-- NEXT_JS_LLM -->
aurorascharff
force-pushed
the
codex/navigation-api-eval
branch
from
September 8, 2026 23:04
be73e86 to
f58e6aa
Compare
aurorascharff
force-pushed
the
codex/navigation-api-eval
branch
from
September 8, 2026 23:05
f58e6aa to
e65fddd
Compare
aurorascharff
marked this pull request as ready for review
September 8, 2026 23:06
gaojude
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
canaryso the Partial Prefetching evals can use the publicunstable_prefetch()andunstable_navigation()API referencesCoverage
The latest results use Claude Sonnet 4.6 for the coding agent and Claude Haiku 4.5 for the agentic judge. The Partial Prefetching cases have been rerun after the new API references landed, and the corrected Cache Components criteria have also been rerun.
agent-035-connection-dynamicagent-040-instantagent-042-enable-ppragent-048-prefetch-url-dataagent-049-prefetch-stageagent-051-prefetch-until-navigationagent-052-cache-components-session-isolationagent-053-cache-components-url-shellAfter the public API references landed, the prefetch-stage and navigation-stage evals show the intended documentation signal. Baseline Sonnet passed 1/5 criteria for
unstable_prefetch()and 0/5 forunstable_navigation(). With the canonical bundled-docs instruction, Sonnet found the new references and passed all 5/5 criteria in both cases. After clarifying that the URL-data navigation must not block and linking the general navigation docs to the optimizing-prefetching guide, the URL-data case improved from 3/6 without docs to 6/6 with bundled docs.Result interpretation
agent-048now passes 6/6 with bundled docs. The scoped documentation link led the agent to the URL-data prefetch pattern instead of a blocking render. Baseline Sonnet addedprefetch={true}but relied on the previous full-payload behavior and missed both the Partial Prefetching configuration and cached URL data, resulting in 3/6.agent-049andagent-051are the intended signal. Without the new references, the agents did not discover the prefetch and navigation stages. With bundled docs, both cases pass 5/5.agent-053is also an intended documentation signal: the baseline hid the useful shell behind a broad fallback, while bundled docs preserved the shell and passed 5/5.agent-035now passes 5/5 in both conditions after removing an unnecessary requirement for visible loading UI in one small metadata region.agent-052also passes 5/5 in both conditions after removing two failures for an omitted explicitcacheLife(), because the default cache profile still applies, and clarifying for the judge that an uncached component readingcookies()orheaders()remains request-specific without another opt-out marker.Overlap decisions
agent-035now carries the synchronous-I/O adoption case, so the duplicateagent-051fixture was removed.agent-042now carries the route-configuration migration case, so the duplicateagent-048fixture was removed.agent-034remains focused on correct asynccookies()andheaders()usage.agent-052tests the separate risk of leaking request-specific values through a public cache.agent-040tests one meaningful shell under default validation.agent-041still tests independent streaming across several dashboard sections.agent-048intentionally brings cached URL-specific content into a selected prefetch.agent-053tests the opposite shared-shell requirement during Cache Components adoption.Verification
pnpm build-allagent-049-prefetch-stage: baseline 1/5, bundled docs 5/5agent-051-prefetch-until-navigation: baseline 0/5, bundled docs 5/5agent-035-connection-dynamic: baseline 5/5, bundled docs 5/5agent-048-prefetch-url-data: baseline 3/6, bundled docs 6/6agent-052-cache-components-session-isolation: baseline 5/5, bundled docs 5/5pnpm exec tsc -p evals/evals/agent-035-connection-dynamic/tsconfig.json --noEmitpnpm exec tsc -p evals/evals/agent-042-enable-ppr/tsconfig.json --noEmit