Skip to content

Sync upstream drift: ports for #132617, #133013 and #7735 - #9

Closed
jamiewest wants to merge 1 commit into
mainfrom
drift/sync-2026-09-21
Closed

jamiewest wants to merge 1 commit into
mainfrom
drift/sync-2026-09-21

Conversation

@jamiewest

Copy link
Copy Markdown
Owner

Incremental /drift sync covering 12 new in-scope upstream commits (5 dotnet/runtime, 7 dotnet/extensions) since pins baeeb465 / cc597aa2. Three ported, nine skipped. Pins advanced to d13c2266 / d2364e51.

⚠️ Public API under lib/src/ai/ changed — affects jamiewest/agents

NumericMetric.interpretScore() changes semantics (its signature is unchanged, so this is a behavioral break, not a compile break):

  • The minimum passing score moves from 3.0 to 4.0. Scores in [3.0, 4.0) that previously reported failed: false now report failed: true.
  • The rating bands change from >= 5 / >= 4 / >= 3 / >= 2 / else to upstream's (4,5] exceptional, (3,4] good, (2,3] average, (1,2] poor, (0,1] unacceptable, inconclusive outside that range.
  • A metric with no score, and a score outside 1–5 (including NaN), are now failed: true with a reason, where both previously reported failed: false.
  • QualityEvaluator subclasses now set interpretation unconditionally, so a metric left unscored by a failed parse carries a failed interpretation instead of null.

Anything downstream that gates on EvaluationMetricInterpretation.failed, or asserts on specific ratings, will see different results. The downstream canary job checks compilation, which is unaffected; the behavioral change is not something it can catch. I could not run the jamiewest/agents checkout locally — this session's GitHub access is scoped to this repository — so the ripple is unverified beyond CI.

Ported

Upstream Commit What changed
dotnet/runtime #132617 910c55a Transient file-system errors during polling. PollingWildcardChangeToken treated a directory scan that threw (network share down, directory briefly inaccessible) as an empty scan, which compared against the previous state as "every file was removed" and fired a spurious change notification. _getCurrentState now returns null on a failed scan; a failed scan reports no change and retries on the next poll, and the first scan that completes establishes the baseline rather than counting pre-existing files as additions (upstream's _previousHash is not null guard). PollingFileChangeToken no longer treats an unreadable file as changed — it leaves its recorded state untouched and retries. 5 tests added.
dotnet/runtime #133013 39b12a6 Doc clarification. "isn't required to exist" added to the PhysicalFileProvider factory, PhysicalFileProvider.watch (which carried no dartdoc at all), the PhysicalFilesWatcher constructor, and createFileChangeToken.
dotnet/extensions #7735 47b6b06 Quality metrics fail closed. See the API note above. 23 tests mirrored from upstream EvaluationMetricExtensionsTests.

Why #7735 was ported wider than the upstream diff

Upstream's new _ when rating is EvaluationRating.Inconclusive branch is only reachable because its rating bands map out-of-range values (> 5.0, <= 0.0, NaN) to Inconclusive. The Dart bands had drifted so that no non-null value produced inconclusive, which would have made that branch dead code. Porting the fix correctly therefore required aligning the bands and the threshold. The drift itself predates this sync.

PhysicalFilesWatcher's share of #132617 (the try around token.HasChanged in the central timer loop) has no Dart counterpart: the Dart watcher has no central polling loop — each token polls on its own timer — so that guard lives inside the per-token code this PR fixes.

Skipped (all recorded in the ledger)

Upstream Commit Reason
dotnet/runtime #131931 6fa6b31 Linked cache entry thread safety — Interlocked/Volatile/Thread.MemoryBarrier plus a copy-on-write ExpirationTokensList. A Dart isolate is single-threaded; there is no race to fix. New N/A entry for CacheEntry.ExpirationTokensList.
dotnet/runtime #133151 4878f22 MemoryCache OTEL unit/tag rename — the Dart caching subsystem publishes no metrics at all (the meterFactory ctor hook is a known open gap), so there is nothing to rename.
dotnet/runtime #133494 d13c226 Configuration allocation tuning — List capacity hints and a collection expression, no behavior change. New N/A entry covering this recurring class of edit.
dotnet/extensions #7749 d2364e5 Microsoft.Extensions.Http.Diagnostics, which no scope row covers (the http row maps to dotnet/runtime's Microsoft.Extensions.Http).
dotnet/extensions #7738 c74cd51, #7732 c4e8764 TypeScript / npm assets of the HTML report frontend; no C# library surface, and the Dart port has no HtmlReportWriter.
dotnet/extensions #7724 d0caf64, #7726 b4b46cc, #7727 2a8df4c Microsoft.Extensions.AI.OpenAI only, N/A by library. Note: #7724 deletes OpenAIAssistantsChatClient and MicrosoftExtensionsAIAssistantsExtensions upstream — two of the 15 names spelled out in that N/A row. The row is left intact so the audit's literal matcher keeps suppressing them.

Recorded rather than guessed at

A structural divergence surfaced while porting #7735 and is filed as a new open priority instead of being fixed here: the Dart RelevanceTruthAndCompletenessEvaluator returns early on a null/inconclusive rating (relevance_truth_and_completeness_evaluator.dart:106), leaving relevance, truth and completeness with no value and no interpretation. Upstream falls through to UpdateResult() in every branch and interprets, so with #7735 upstream now fails closed there and the Dart port still does not. Closing it means restructuring the evaluator — a larger change than the commit under review.

The equivalent one-line fix was applied to quality_evaluator_base.dart, where upstream's shape (_ = TryParse…; metric.Interpretation = metric.InterpretScore();) is unambiguous.

Release

packages/extensions → 0.8.1 (patch: fixes and docs, no new public API) with a matching CHANGELOG.md entry. extensions_flutter is unchanged and not bumped. No tags created and nothing published from this session.

Verification

Run locally on Flutter 3.47.5 / Dart 3.13.4 (matching CI's channel: stable):

Check Result
dart analyze --fatal-infos . (extensions) No issues found
dart format --output=none --set-exit-if-changed . (extensions) 883 files, 0 changed
dart test (extensions) 1063 passing (28 new)
flutter analyze --fatal-infos (extensions_flutter) No issues found
dart format --output=none --set-exit-if-changed . (extensions_flutter) 22 files, 0 changed
flutter test (extensions_flutter) 47 passing

🤖 Generated with Claude Code

https://claude.ai/code/session_014CCMWHQhqgyzMkQwBduWBG


Generated by Claude Code

Incremental `/drift sync` covering 12 new in-scope upstream commits
(5 dotnet/runtime, 7 dotnet/extensions) since pins baeeb465 / cc597aa2.
Three ported, nine skipped. Pins advanced to d13c2266 / d2364e51.

Ported:

* dotnet/runtime #132617 (910c55a) — handle transient file system errors
  during active polling. PollingWildcardChangeToken treated a directory
  scan that threw as an empty scan, which compared against the previous
  state as "every file was removed" and fired a spurious change
  notification. _getCurrentState now returns null on a failed scan; a
  failed scan reports no change and retries on the next poll, and the
  first scan that completes establishes the baseline (upstream's
  `_previousHash is not null` guard). PollingFileChangeToken no longer
  treats an unreadable file as changed. 5 tests added.

* dotnet/runtime #133013 (39b12a6) — document that watched paths need not
  exist. "isn't required to exist" added to the PhysicalFileProvider
  factory, PhysicalFileProvider.watch (which carried no dartdoc at all),
  the PhysicalFilesWatcher constructor and createFileChangeToken.

* dotnet/extensions #7735 (47b6b06) — fail closed when a quality metric
  has no valid score. NumericMetric.interpretScore now reports a null
  score and an out-of-range score as failures with a reason, and
  quality_evaluator_base interprets unconditionally so a metric left
  unscored by a failed parse still carries a failed interpretation.
  The port is widened deliberately: upstream's new "rating is
  Inconclusive" branch is only reachable with upstream's rating bands, so
  the bands and the 4.0 minimum passing score were aligned too. The Dart
  port had drifted to >= 5/4/3/2 with a 3.0 threshold, so scores in
  [3.0, 4.0) that previously passed now fail. 23 tests mirrored from
  upstream EvaluationMetricExtensionsTests.

Skipped (recorded in the ledger):

* dotnet/runtime #131931 (6fa6b31) — linked cache entry thread safety.
  Interlocked/Volatile/MemoryBarrier machinery plus a copy-on-write
  ExpirationTokensList; a Dart isolate is single-threaded. New N/A entry
  for CacheEntry.ExpirationTokensList.
* dotnet/runtime #133151 (4878f22) — MemoryCache OTEL unit/tag rename.
  The Dart caching subsystem publishes no metrics; nothing to rename.
* dotnet/runtime #133494 (d13c226) — Configuration allocation tuning.
  List capacity hints and a collection expression, no behavior change.
  New N/A entry covering this class of edit.
* dotnet/extensions #7749 (d2364e5) — Microsoft.Extensions.Http.Diagnostics,
  which no scope row covers.
* dotnet/extensions #7738 (c74cd51) and #7732 (c4e8764) — TypeScript/npm
  assets of the HTML report frontend; the Dart port has no HtmlReportWriter.
* dotnet/extensions #7724 (d0caf64), #7726 (b4b46cc), #7727 (2a8df4c) —
  Microsoft.Extensions.AI.OpenAI only, N/A by library.

Also recorded as a new open priority rather than guessed at: the Dart
RelevanceTruthAndCompletenessEvaluator returns early on an inconclusive
rating, leaving its three metrics with no interpretation, where upstream
falls through and interprets (so #7735 fails closed there and the Dart
port still does not). Closing it is a larger restructure than the commit
under review.

packages/extensions bumped to 0.8.1 with a matching changelog entry.
extensions_flutter is unchanged and not bumped.

Verified: dart analyze --fatal-infos, dart format --set-exit-if-changed
and dart test (1063 passing) in packages/extensions; flutter analyze
--fatal-infos, dart format and flutter test (47 passing) in
packages/extensions_flutter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014CCMWHQhqgyzMkQwBduWBG

Copy link
Copy Markdown
Owner Author

Closing as superseded by #11 — but this PR found a real bug, and it's worth recording where it went.

Two /drift sync runs overlapped. The 2026-09-07 run stalled mid-session and landed as #8 at 16:02Z today; this one ran on schedule at 12:24Z, branched off the pre-merge main, and ported the same three upstream commits independently. Nothing here was wasted — it just arrived against a base that moved underneath it.

Rebasing wasn't the cheaper path: all nine files conflict against merged main (ledger, CHANGELOG, pubspec, both polling tokens, both doc files, the metric extension, and a test file that would land at a different path from #8's — test/ai/ vs test/ai/evaluation/). #11 instead carries over only what #8 lacked, so there's nothing to resolve.

Carried over to #11:

  • The quality_evaluator_base.dart fix — the important one. Sync upstream drift through runtime 6fa6b31 / extensions 47b6b06 #8 ported interpretScore's fail-closed behavior but left the interpretation assigned only in the parse-success branch, so an unparseable judge reply still produced interpretation == null. This PR caught that; Sync upstream drift through runtime 6fa6b31 / extensions 47b6b06 #8 shipped without it.
  • Pins → d13c2266 / d2364e51, with the #133494 / #7749 / #7738 decisions and the new N/A row for allocation-tuning edits.
  • The RelevanceTruthAndCompletenessEvaluator open priority. Verified against the Dart source before recording — relevance_truth_and_completeness_evaluator.dart:106 does return early exactly as described here.
  • The extra test cases: explicit NaN, the full band table, the sub-threshold sweep.
  • The _refreshWriteTime() extraction.

Not carried over: the removal of the per-file try/catch in polling_wildcard_change_token.dart. That change lets one unreadable file abort the entire scan and report "no change" — and a permanently unreadable file would make the token blind to every other file indefinitely. Upstream only looks similar because its GetLastWriteUtc swallows per-file errors, so it doesn't abort on one bad file either. main's per-file skip is kept.

Also worth flagging for whoever picks up the next run: two in-scope commits landed after this run and are recorded in the ledger as its first items — e77687c (#7766, tokenizer punctuation/number tokens) and 561bdf7 (#7768, GLEU against the best matching reference). Both are in AI.Evaluation.NLP, which is ported.


Generated by Claude Code

@jamiewest jamiewest closed this Sep 21, 2026
jamiewest added a commit that referenced this pull request Sep 21, 2026
Carry over the drift-sync residue PR #9 found
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants