Skip to content

fix: propagate asynchronous pipeline failures - #61

Open
ikeago wants to merge 2 commits into
AI-Riksarkivet:mainfrom
ikeago:fix/reliability-failure-propagation
Open

ikeago wants to merge 2 commits into
AI-Riksarkivet:mainfrom
ikeago:fix/reliability-failure-propagation

Conversation

@ikeago

@ikeago ikeago commented Oct 12, 2026

Copy link
Copy Markdown

Description

When one document worker raises, the CLI currently discards the submitted future, can report success, and can continue to the Processed ... completion log. Separately, an exception while running or consuming an inference batch terminates the daemon thread while leaving callers' futures unresolved, so they can wait indefinitely.

This change makes both failure paths observable:

  • the CLI retains document futures and calls result() as they complete, so worker failures reach Typer and produce a nonzero exit;
  • the inference worker materializes and validates a whole batch before publishing results, sets the exception on every pending future when a batch fails, and stays available for later work.

The tests cover the real CLI exit/log behavior, multi-item batch failure followed by recovery, too-few model outputs, and generator failure during output iteration.

Validation:

  • CUDA_VISIBLE_DEVICES='' uv run pytest tests/unit/test_cli.py tests/unit/test_steps.py tests/unit/test_queue.py — 11 passed
  • CUDA_VISIBLE_DEVICES='' uv run pytest tests/unit --ignore=tests/unit/test_wordsegmentation.py — 55 passed, 1 skipped
  • uv run ruff format --check src/htrflow/cli.py src/htrflow/pipeline/steps.py tests/unit/test_cli.py tests/unit/test_steps.py
  • uv run ruff check src/htrflow/cli.py src/htrflow/pipeline/steps.py tests/unit/test_cli.py tests/unit/test_steps.py

The full suite is not claimed here: on the current unlocked dependency resolution, the model-backed word-segmentation test failed independently of this patch. PR #57 addresses the lock file and current-stack compatibility work.

OpenAI Codex assisted with implementation and testing, and Claude Sonnet independently reviewed the patch. I reviewed the final changes and take responsibility for them.

Checklist

  • I have performed a self-review of my code locally.
  • I have run ruff format to format my code.
  • My code passes ruff check.
  • If it is a core feature, I have added thorough tests.
  • Documentation has been updated or the changes are too minor to be documented.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant