Conversation
…ucturedOutputCache Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…peSafeDescriptionScorer Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…orer Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…h stub drift) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ilent 4xx, and weak cache keys Post-review fixes for the zero-shot-typesafe preset: - from_context downgrades an HPO-sampled question_type="choice" to "noul" on a multilabel dataset (with a warning) instead of raising, so a single bad trial no longer aborts Pipeline.fit(); the explicit constructor ValueError is unchanged. get_implicit_initialization_params() now records the effective question_type. - _ask_one_sync/_ask_one_async re-raise non-retryable 4xx errors (auth, bad model, malformed request) via a new _is_fatal() helper instead of degrading them to a uniform row; 408/429 still fall back like a transient failure. - _cache_key now hashes resolved model, resolved base URL, and the full question payload (question_type + descriptions + instructions), replacing the narrower question_type/descriptions inputs; TypeSafeAnswer/_unpack carry the response's model name through to the cache and the INFO log line. - Cheap minors: _init_event_loop installs a freshly created loop via asyncio.set_event_loop; StructuredOutputCache.get/set(_async) skip computing the cache key entirely when the cache is disabled; choice fit rejects more than MAX_CHOICE_OPTIONS (255) descriptions up front; docstrings added to from_context, load, _fit_implementation, _ask_one_sync, _ask_one_async. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…, and cheap-minor fixes - fake_response derives its "winning" option from the utterance (via expected_best_index) when best_index is omitted, so predict() result order is actually observable instead of every case landing on index 0; updated the tests that assumed index 0 and extended the async/sync parity test to assert per-utterance argmax. - test_from_context_defaults_question_type_by_task asserts the new implicit-params shape; test_from_context_coerces_hpo_sampled_choice_to_noul_on_multilabel covers the choice->noul downgrade and its warning. - test_fatal_status_propagates_but_transient_status_falls_back (parametrized over max_concurrent) covers _is_fatal: a 401-shaped error propagates instead of producing a uniform row, a 503-shaped one still falls back. - test_cache_key_changes_with_model covers the widened cache key. - test_fit_rejects_too_many_choice_options covers the new MAX_CHOICE_OPTIONS guard; test_failed_request_yields_uniform_row_and_warns is parametrized over question_type too, exercising both the softmax and sigmoid fallback paths; test_dump_and_load_roundtrip is parametrized over max_concurrent. - test_create_clients_builds_real_sdk_clients exercises the real typesafe_sdk seam (skipped unless the extra is installed, no network call). - test_zero_shot_typesafe_preset_on_multilabel runs the zero-shot-typesafe preset's full HPO loop against a multilabel dataset end to end. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add "typesafe" to the test-scorers dependency-group matrix so test_create_clients_builds_real_sdk_clients runs against the real typesafe_sdk types (not just the fakes) on every PR; the extra is light (httpx/pydantic). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Sep 19, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TypeSafeDescriptionScorer(description_typesafe): zero-shot intent scoring via TypeSafe's System One modeljev— oneChoiceover all intent descriptions (multiclass) or oneNoulper description (multilabel), probabilities mapped to log/logit similarities sotemperature=1reproduces the model's distribution; non-retryable 4xx errors propagate, transient failures fall back to a uniform row.typesafe(typesafe-sdk>=0.7,<1); the SDK is imported lazily inside_create_clients, so nothing else needs it (atypesafeleg in thetest-scorersCI matrix exercises the real import).PydanticDiskCacheextracted fromStructuredOutputCache(behaviour-preserving); the new scorer caches answers in its own subdir keyed on model, base URL, questions and utterance.zero-shot-typesafe(HPO overquestion_typeandtemperature; a sampledchoiceis coerced tonoulon multilabel data), advisor registration, docs.chore(typing): one cast in_dump_tools/unit_dumpers.pyfor torch-stub drift that had mademypyred on a fresh lock.Design:
docs/superpowers/specs/2026-09-18-typesafe-zero-shot-scorer-design.md(local, gitignored).Validation
Comparison experiment (full pipeline, hold-out, 3 seeds) in Darinochka/AutoIntent-experiments#43: jev matches
description_llm/GPT-6 within 95% CIs on banking77 and hwu64 (0.830 vs 0.840, 0.870 vs 0.885 accuracy) at ≈300× lower cost, trails supervisedclassic-lightby 2–9 points, and gives a usable OOS signal on clinc150 (0.924 accuracy, OOS recall 0.77 vs 0.77 for classic-light).Follow-ups (not in this PR)
Generatorshould catchopenai.APIStatusError(an HTTP 402 mid-run crashed the experiment's GPT-6 stage and lost ≈$17 of calls).OptimizationConfig.seedisPositiveInt— rejectsseed=0.predictinside an already-running event loop (server/notebook) fails;LLMDescriptionScorerleaks a loop per trial on 3.14; itsmax_per_secondlimiter also throttles cache hits; its prompt puts the utterance before the descriptions, defeating prefix caching.BaseAPIDescriptionScorerextraction once the duplication settles.Test plan
classic-lightanddescription_llm/gpt-6 (Zero-shot jev vs GPT-6 vs classic-light Darinochka/AutoIntent-experiments#43)🤖 Generated with Claude Code