Skip to content

Advisor treats description_llm / description_typesafe as embedder consumers and estimates no API cost for them #356

Description

@voorhs

What happens

The advisor classifies description_llm and description_typesafe as embedder consumers: both are in _CACHE_HONORING_MODULES, _EMBEDDER_FORWARD_TRANSFORMER_MODULES and _EMBEDDER_CONSUMING_MODULES (_resource.py:99-117, :601-619). Neither loads an embedder — they call an external API — and the advisor has no notion of an API-cost driver, so the report is wrong in both directions.

autointent-advisor inspect zero-shot-typesafe --n-classes 77 --n-samples 300 --json on fa4df2d2 (zero-shot-typesafe is description_typesafe + the decision node only):

"resource": {"disk_download_gb": 0.0, "disk_cached_gb": 0.91, "ram_gb": 1.80, "vram_gb": 0.0,
             "time_hours": 0.0, "drivers": []}
Verdict: feasible (headroom: ample)
  • The 0.9 GB is the default sentence-transformers/all-MiniLM-L6-v2 ("already cached" here; "to download" on a fresh machine) and the RAM includes _embedder_load_ram_gb for it — a model the preset never loads.
  • drivers: [], time_hours: 0.0: nothing about the actual cost — the number of API calls (≈ unique utterances of train_1 + validation + test, × question types for typesafe, thanks to the answer cache), their price, or the wall-clock at max_concurrent/max_per_second. Measured in Zero-shot jev vs GPT-6 vs classic-light Darinochka/AutoIntent-experiments#43: description_llm on gpt-6 via OpenRouter ≈ $2.5–3.1 and ~10 s per 100 utterances; description_typesafe ≈ $0.01–0.02 per 100.
  • No preflight finding that the arm needs an API key in the environment (OPENAI_API_KEY / TYPESAFE_API_KEY), which is the thing that actually fails on a fresh machine.

Proposed

A third module class next to "embedder consumer" / "trainer": API scorer — no embedder, VRAM or model disk; an api driver row with the estimated call count (from the split sizes and, for typesafe, the number of question_type values in the search space), the configured rate limits → wall-clock, and cost: not estimated (external API, provider pricing) rendered explicitly rather than as $0/"ample"; plus a preflight check for the key env var.

Follow-up from #350.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions