You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The advisor classifies description_llm and description_typesafe as embedder consumers: both are in _CACHE_HONORING_MODULES, _EMBEDDER_FORWARD_TRANSFORMER_MODULES and _EMBEDDER_CONSUMING_MODULES (_resource.py:99-117, :601-619). Neither loads an embedder — they call an external API — and the advisor has no notion of an API-cost driver, so the report is wrong in both directions.
autointent-advisor inspect zero-shot-typesafe --n-classes 77 --n-samples 300 --json on fa4df2d2 (zero-shot-typesafe is description_typesafe + the decision node only):
The 0.9 GB is the default sentence-transformers/all-MiniLM-L6-v2 ("already cached" here; "to download" on a fresh machine) and the RAM includes _embedder_load_ram_gb for it — a model the preset never loads.
drivers: [], time_hours: 0.0: nothing about the actual cost — the number of API calls (≈ unique utterances of train_1 + validation + test, × question types for typesafe, thanks to the answer cache), their price, or the wall-clock at max_concurrent/max_per_second. Measured in Zero-shot jev vs GPT-6 vs classic-light Darinochka/AutoIntent-experiments#43: description_llm on gpt-6 via OpenRouter ≈ $2.5–3.1 and ~10 s per 100 utterances; description_typesafe ≈ $0.01–0.02 per 100.
No preflight finding that the arm needs an API key in the environment (OPENAI_API_KEY / TYPESAFE_API_KEY), which is the thing that actually fails on a fresh machine.
Proposed
A third module class next to "embedder consumer" / "trainer": API scorer — no embedder, VRAM or model disk; an api driver row with the estimated call count (from the split sizes and, for typesafe, the number of question_type values in the search space), the configured rate limits → wall-clock, and cost: not estimated (external API, provider pricing) rendered explicitly rather than as $0/"ample"; plus a preflight check for the key env var.
What happens
The advisor classifies
description_llmanddescription_typesafeas embedder consumers: both are in_CACHE_HONORING_MODULES,_EMBEDDER_FORWARD_TRANSFORMER_MODULESand_EMBEDDER_CONSUMING_MODULES(_resource.py:99-117,:601-619). Neither loads an embedder — they call an external API — and the advisor has no notion of an API-cost driver, so the report is wrong in both directions.autointent-advisor inspect zero-shot-typesafe --n-classes 77 --n-samples 300 --jsononfa4df2d2(zero-shot-typesafeisdescription_typesafe+ the decision node only):sentence-transformers/all-MiniLM-L6-v2("already cached" here; "to download" on a fresh machine) and the RAM includes_embedder_load_ram_gbfor it — a model the preset never loads.drivers: [],time_hours: 0.0: nothing about the actual cost — the number of API calls (≈ unique utterances oftrain_1+ validation + test, × question types for typesafe, thanks to the answer cache), their price, or the wall-clock atmax_concurrent/max_per_second. Measured in Zero-shot jev vs GPT-6 vs classic-light Darinochka/AutoIntent-experiments#43:description_llmon gpt-6 via OpenRouter ≈ $2.5–3.1 and ~10 s per 100 utterances;description_typesafe≈ $0.01–0.02 per 100.OPENAI_API_KEY/TYPESAFE_API_KEY), which is the thing that actually fails on a fresh machine.Proposed
A third module class next to "embedder consumer" / "trainer": API scorer — no embedder, VRAM or model disk; an
apidriver row with the estimated call count (from the split sizes and, for typesafe, the number ofquestion_typevalues in the search space), the configured rate limits → wall-clock, andcost: not estimated (external API, provider pricing)rendered explicitly rather than as $0/"ample"; plus a preflight check for the key env var.Follow-up from #350.