docs: publish September 24 live results - #79
Merged
Merged
Conversation
Contributor
Author
How to use the Graphite Merge QueueAdd either label to this PR to merge it via the merge queue:
You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
anandpant
marked this pull request as ready for review
September 25, 2026 05:22
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
anandpant
force-pushed
the
docs/live-results-0924
branch
from
September 25, 2026 05:44
2027bdc to
15b687b
Compare
anandpant
force-pushed
the
fix/opus-55-structured-output
branch
from
September 25, 2026 05:44
d427b53 to
f8db945
Compare
graphite-app
Bot
force-pushed
the
fix/opus-55-structured-output
branch
2 times, most recently
from
September 25, 2026 05:51
e62cce5 to
c8d09e3
Compare
graphite-app
Bot
force-pushed
the
docs/live-results-0924
branch
from
September 25, 2026 05:51
15b687b to
083f351
Compare
graphite-app
Bot
changed the base branch from
fix/opus-55-structured-output
to
graphite-base/79
September 25, 2026 05:53
graphite-app
Bot
force-pushed
the
graphite-base/79
branch
from
September 25, 2026 05:55
c8d09e3 to
ffa3811
Compare
graphite-app
Bot
force-pushed
the
docs/live-results-0924
branch
from
September 25, 2026 05:55
083f351 to
7258dd7
Compare
graphite-app
Bot
force-pushed
the
docs/live-results-0924
branch
from
September 25, 2026 05:56
7258dd7 to
41d4c96
Compare
Merge activity
|
## Summary Refresh every results table in `docs/README.md` from the September 24-25 catalog sweep, so all tables share one date and one set of SDK versions instead of mixing the September 14 and 15 surveys. - Rebuild the OpenAI, Anthropic, Google, xAI, third-party, and embedding tables from reviewed results. Add rows for the eight new models, drop the Claude Opus 4.1 row, and refresh the header date, catalog counts, and SDK versions. - Rewrite Findings: the four candidates left out of the catalog and why, the account rate limiting and its isolated reruns, the Opus 5.5 adaptive-thinking and forced-tool-use constraints, and Gemini 3.8 Flash returning no output for combined JSON + tools this run. - Add a DeepSeek V4.1 Flash route comparison recording why the native SDK was not adopted: it surfaces reasoning text the OpenAI-compatible route does not, but its tool continuation sends assistant `reasoning_content`, which Foundry rejects. - Correct the stale six-Grok counts in `docs/README.md` and `docs/xai.md` now that the catalog has seven. Basic text passed on 62 of 63 language aliases; Llama Nemotron Super remains the one entry with no usable adapter route. All three embeddings and all three realtime models passed. ## Validation The catalog sweep ran `pnpm run test:live -- --catalog` at concurrency 1 over all 63 language aliases plus embeddings, followed by `pnpm run test:realtime`. Twenty-seven cases returned HTTP 429 after the SDK's own retries; all of them passed on isolated rerun, and the tables use those reruns. Raw run directories stay under `.memory/` and are not committed. `pnpm run verify` passed at the top of the stack. Docs-only change with no runtime edits.
graphite-app
Bot
force-pushed
the
docs/live-results-0924
branch
from
September 25, 2026 06:08
41d4c96 to
41304f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refresh every results table in
docs/README.mdfrom the September 24-25 catalog sweep, so all tables share one date and one set of SDK versions instead of mixing the September 14 and 15 surveys.reasoning_content, which Foundry rejects.docs/README.mdanddocs/xai.mdnow that the catalog has seven.Basic text passed on 62 of 63 language aliases; Llama Nemotron Super remains the one entry with no usable adapter route. All three embeddings and all three realtime models passed.
Validation
The catalog sweep ran
pnpm run test:live -- --catalogat concurrency 1 over all 63 language aliases plus embeddings, followed bypnpm run test:realtime. Twenty-seven cases returned HTTP 429 after the SDK's own retries; all of them passed on isolated rerun, and the tables use those reruns. Raw run directories stay under.memory/and are not committed.pnpm run verifypassed at the top of the stack. Docs-only change with no runtime edits.