Repository navigation
test: add adoption skill evals - #97759
Merged
Merged
Conversation
Contributor
Stats from current PR🟢 1 improvement
📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
Build Cache
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URLCommit: ef35138 |
aurorascharff
marked this pull request as ready for review
August 26, 2026 07:42
…ion-guide' into codex/adoption-skill-evals
…ion-guide' into codex/adoption-skill-evals
This was referenced Aug 27, 2026
aurorascharff
added a commit
that referenced
this pull request
Aug 27, 2026
…ion (#97712) ## What - Documents how legacy `prefetch={true}` behavior changes with Partial Prefetching and how to choose the UI worth preserving. - Updates the adoption skill to capture a passing flag-off `instant()` baseline, use the same assertions to drive the migration, and retain them as regression coverage. - Adds a reusable production test rig, keeps additional per-link optimization separate from adoption, and surfaces `prefetch={false}` links that may no longer be needed. - Applies the shared test-rig fixes discovered during dogfooding to the Cache Components optimizer. ## Why Legacy `prefetch={true}` could include uncached dynamic content. With Partial Prefetching, it includes the shared App Shell and cached URL-specific content instead. A passing build or clean insight sweep does not prove that the intended prefetched UI survived the migration, so the skill now records that contract before adoption and verifies it unchanged afterward. The guide presents the same test workflow as an optional programmatic check for manual adopters. ## Testing - Prettier, Alex, and skill frontmatter validation - Dogfooded on v0 in vercel/v0#28842 - Partial Prefetching adoption eval: 6/6 in #97759
Base automatically changed from
codex/partial-prefetch-preservation-guide
to
canary
August 27, 2026 19:46
gaojude
approved these changes
Aug 31, 2026
icyJoseph
pushed a commit
that referenced
this pull request
Sep 7, 2026
…ion (#97712) ## What - Documents how legacy `prefetch={true}` behavior changes with Partial Prefetching and how to choose the UI worth preserving. - Updates the adoption skill to capture a passing flag-off `instant()` baseline, use the same assertions to drive the migration, and retain them as regression coverage. - Adds a reusable production test rig, keeps additional per-link optimization separate from adoption, and surfaces `prefetch={false}` links that may no longer be needed. - Applies the shared test-rig fixes discovered during dogfooding to the Cache Components optimizer. ## Why Legacy `prefetch={true}` could include uncached dynamic content. With Partial Prefetching, it includes the shared App Shell and cached URL-specific content instead. A passing build or clean insight sweep does not prove that the intended prefetched UI survived the migration, so the skill now records that contract before adoption and verifies it unchanged afterward. The guide presents the same test workflow as an optional programmatic check for manual adopters. ## Testing - Prettier, Alex, and skill frontmatter validation - Dogfooded on v0 in vercel/v0#28842 - Partial Prefetching adoption eval: 6/6 in #97759 (cherry picked from commit 24ed66a)
aurorascharff
added a commit
that referenced
this pull request
Sep 10, 2026
## Summary - add a Partial Prefetching optimizer Skill for selected source Links and destinations - keep prefetch stages and Link policy in the public [Optimizing prefetching guide](https://nextjs.org/docs/app/guides/optimizing-prefetching), and rely on the same static-shell documentation as the Cache Components optimizer for cache placement, Suspense, loading UI, and authorization - keep the Skill focused on the product decision, production `instant()` loop, differential verification, and mutation freshness when an optimization adds a cache around mutable data - reference the canonical [`unstable_prefetch()`](https://nextjs.org/docs/app/api-reference/functions/prefetch) and [`unstable_navigation()`](https://nextjs.org/docs/app/api-reference/functions/navigation) docs instead of explaining the APIs again in the Skill - link cache-writing optimizations to the canonical [Revalidating](https://nextjs.org/docs/app/getting-started/caching-and-revalidating#revalidating) documentation and verify a populated-cache mutation in the Skill and eval - bundle a self-contained production rig aligned with the related adoption and optimizer Skills, tuned to record each source-link and prefetch-stage contract - add an eval that compares baseline, the bundled guide, and the local Skill against the same prefetched-UI and cache-freshness goal - document the optimizer in the AI agents guide This follows the same guide/Skill split as #97737. Partial Prefetching adoption preserves existing behavior; this optimizer is the later decision about what is worth preparing before a click and what should wait for navigation. Related: #97712, #97737, #97759, #98122, #98175 ## Eval One run per treatment with Claude Opus 4.8 and Claude Sonnet 4.5, using a fixed Claude Haiku 4.5 judge. The bundled-guide and Skill treatments use a local Next.js tarball containing the new guide and API references. | Model | Treatment | Criteria | Score change | Overall | Duration | Tokens processed | Total run cost | Cost change | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Opus 4.8 | Baseline | 10/10 (100%) | — | Pass | 18m 42s | 3.39M | $3.46 | — | | Opus 4.8 | Bundled guide | 9/10 (90%) | -10 pp | Fail | 13m 37s | 3.02M | $2.84 | -18.0% | | Opus 4.8 | Local Skill | 10/10 (100%) | +10 pp | Pass | 17m 22s | 2.80M | $2.54 | -10.6% | | Sonnet 4.5 | Baseline | 2/10 (20%) | — | Fail | 15m 55s | 1.97M | $1.06 | — | | Sonnet 4.5 | Bundled guide | 5/10 (50%) | +30 pp | Fail | 15m 49s | 2.17M | $1.11 | +4.5% | | Sonnet 4.5 | Local Skill | 8/10 (80%) | +30 pp | Fail | 15m 31s | 1.41M | $0.77 | -30.4% | Token totals count each unique model message once and include uncached input, cache creation, cache reads, and output. Cache reads account for most of the total in every treatment. Total run cost applies the current AI Gateway base rates to each token category: Opus 4.8 costs $5/M input, $6.25/M cache writes, $0.50/M cache reads, and $25/M output; Sonnet 4.5 costs $3/M input, $3.75/M cache writes, $0.30/M cache reads, and $15/M output. The six agent runs cost $11.78 in total. Compared with baseline, the Opus Skill run maintained a 100% criteria pass rate while reducing cost by 26.7%. The Sonnet Skill run improved the criteria pass rate from 20% to 80%, a 60 percentage-point increase, while reducing cost by 27.3%. The stronger model could solve the fixture without the Skill, although it independently searched the bundled Next.js documentation and took slightly longer than the Skill treatment. The Opus guide-only run implemented the correct behavior but missed the required RED-before-GREEN proof. The Sonnet treatments show a clearer progression. Baseline did not implement the required caching, navigation stage, `instant()` contract, or cache invalidation. The bundled guide added the cached summary and production test, but missed the navigation stage, deferred-state assertions, tag invalidation, and RED-before-GREEN proof. The Skill treatment completed the `instant()` RED-to-GREEN loop and final production behavior, but left `connection()` in the related-sessions read and used path revalidation instead of tagging the cached read. It therefore passed 8 of 10 criteria but failed the overall eval. All six measured runs completed through grading. Chromium and its system dependencies are provisioned before the agent starts, so missing browser libraries did not determine these results. ## Verification - `pnpm --filter=next build` - `pnpm prettier --check` for the changed guide, prompt, Skill, and eval fixture - `pnpm exec tsc -p evals/evals/agent-051-optimize-partial-prefetching/tsconfig.json --noEmit` - `node --check run-evals.js` - `git diff --check` - packed a local Next.js tarball and confirmed it contains the optimizer guide plus the `navigation()` and `prefetch()` references - verified the `navigation()` contract in the minimal reproduction and NextBeats on Next.js 16.4.0-canary.22 - merged the latest `canary`, preserving both generic fixture setup and Playwright provisioning <!-- NEXT_JS_LLM -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Results
One
claude-opus-4-8run was retained per treatment and adoption eval, judged byclaude-haiku-4-5. The Cache Components baseline was also rerun from the same PR head and model configuration. These results are directional rather than statistically stable.The Partial Prefetching skill was the only treatment to establish a passing legacy
instant()baseline before migration, rerun the same production Playwright assertions after enabling Partial Prefetching, and preserve the selected eager-link contract. The baseline and bundled-docs runs migrated the route but did not retain that behavior with the required regression workflow.Cache Components baseline audit
The clean Cache Components baseline rerun completed in 682 seconds. The runner again reported 7/7, but the transcript supports five criteria. Two agentic checks were false positives:
<main>and a<Suspense fallback={null}>outside the URL-specific work, so it did not preserve useful visible shell content.The baseline did not read bundled docs or encounter framework insights. It inspected the installed Next.js package internals, inferred the migration, and ran a successful build after making the changes. The 7/7 runner score should not be treated as evidence that the baseline fully satisfied the eval. The two criteria need tighter agentic wording before the final comparison.
The current bundled-docs and skill scores remain 6/7 and 7/7. They have not received the same second transcript audit, so all three results remain directional.
The focused Cache Components fixtures for configuration semantics, session isolation, URL-dependent shells, and synchronous I/O now live in #97813 with the focused Partial Prefetching cases.
Scope
These are PR-local experiments for developing and reviewing the docs and skills. This PR does not register them in the public
evals.nextjs.orgsuite, which is configured separately in the OSS eval repository.Validation
pnpm exec tsc -p evals/tsconfig.json --noEmit