What happens
Generator._get_structured_output_openai_async catches only (ValidationError, ValueError, LengthFinishReasonError) (_generator.py:239), and LLMDescriptionScorer._process_utterance_async catches only RetriesExceededError (llm_encoder.py:214-224). Any openai.APIStatusError (a 4xx the SDK does not retry, or a 408/429/5xx after its 2 built-in retries) and any openai.APIConnectionError therefore propagates out of aiometer.run_all → run_until_complete → Pipeline.fit: the first failed utterance cancels the whole batch and the whole HPO run.
Observed
Darinochka/AutoIntent-experiments#43 (description_llm on openai/gpt-6-astra via OpenRouter, clinc150): OpenRouter answered HTTP 402 in_flight_budget_exhausted on one call, 358/370 calls into the stage. The process crashed, ≈$17.49 of already-made calls produced no result, and the run could only restart from Pipeline.fit (the completed answers were in StructuredOutputCache, but the stage's budget gate then refused the retry). The crash also left two empty cache directories that made every later Generator(...) raise FileNotFoundError until removed by hand — that part is #335.
Proposed
Same policy TypeSafeDescriptionScorer uses in #350 (_is_fatal):
- transient (408, 429, 5xx, connection errors): retry with backoff honouring
Retry-After beyond the SDK's default, then degrade that utterance to a uniform row (as RetriesExceededError already does) and keep going;
- fatal (401, 403, 404, 422 — and 402 when it is a hard "no credits" stop; OpenRouter's
in_flight_budget_exhausted 402 is arguably transient): raise a typed AutoIntent error with a clear message, before more calls are spent;
- catch per utterance inside the coroutine handed to
aiometer.run_all, so one failure never cancels the others in flight.
Follow-up from #350.
What happens
Generator._get_structured_output_openai_asynccatches only(ValidationError, ValueError, LengthFinishReasonError)(_generator.py:239), andLLMDescriptionScorer._process_utterance_asynccatches onlyRetriesExceededError(llm_encoder.py:214-224). Anyopenai.APIStatusError(a 4xx the SDK does not retry, or a 408/429/5xx after its 2 built-in retries) and anyopenai.APIConnectionErrortherefore propagates out ofaiometer.run_all→run_until_complete→Pipeline.fit: the first failed utterance cancels the whole batch and the whole HPO run.Observed
Darinochka/AutoIntent-experiments#43 (
description_llmonopenai/gpt-6-astravia OpenRouter, clinc150): OpenRouter answered HTTP 402in_flight_budget_exhaustedon one call, 358/370 calls into the stage. The process crashed, ≈$17.49 of already-made calls produced no result, and the run could only restart fromPipeline.fit(the completed answers were inStructuredOutputCache, but the stage's budget gate then refused the retry). The crash also left two empty cache directories that made every laterGenerator(...)raiseFileNotFoundErroruntil removed by hand — that part is #335.Proposed
Same policy
TypeSafeDescriptionScoreruses in #350 (_is_fatal):Retry-Afterbeyond the SDK's default, then degrade that utterance to a uniform row (asRetriesExceededErroralready does) and keep going;in_flight_budget_exhausted402 is arguably transient): raise a typed AutoIntent error with a clear message, before more calls are spent;aiometer.run_all, so one failure never cancels the others in flight.Follow-up from #350.