Skip to content

test(crewai): keep incidental requests out of the VCR cassettes - #270

Merged
ralf0131 merged 2 commits into
alibaba:mainfrom
RichardoMrMu:fix/crewai-vcr-ignore-incidental-hosts
Sep 20, 2026
Merged

ralf0131 merged 2 commits into
alibaba:mainfrom
RichardoMrMu:fix/crewai-vcr-ignore-incidental-hosts

Conversation

@RichardoMrMu

Copy link
Copy Markdown
Contributor

Why

Hardening, not a bug fix. The CI failure reported in #230 is already fixed by #233,
and the crewai job is green on main (run 34801198149). This closes the gap that
made that failure possible, so it cannot recur.

litellm fetches its model cost map from raw.githubusercontent.com the first time
it is used. That request is incidental to the code under test, but VCR still sees
it, and the crewai conftest.py has no ignore_hosts. Under record_mode=none a
request to a host with no recorded counterpart raises — which is exactly the first
line of #230's CI log:

WARNING LiteLLM: Failed to fetch remote model cost map from
https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json:
Can't overwrite existing cassette
(.../cassettes/test_agent_workflow/TestAgentWorkflow.test_hierarchical_workflow.yaml)
in your current record mode (<RecordMode.NONE: 'none'>).

The method/host/path matcher failures that followed are the same cause from
the other side: the recordings are api.openai.com calls, so a GET to
raw.githubusercontent.com matches none of them.

Mechanism, reproduced

vcrpy alone — no crewai, no credentials, no network. A cassette holding one
api.openai.com interaction, then a request to raw.githubusercontent.com inside
it:

record_mode=none, no ignore_hosts:
  vcr.errors.CannotOverwriteExistingCassetteException
  Can't overwrite existing cassette (...) in your current record mode
  (<RecordMode.NONE: 'none'>). No match for the request
  (<Request (GET) https://raw.githubusercontent.com/BerriAI/litellm/...>)

record_mode=none, with ignore_hosts:
  request goes through, no VCR error

Same exception class and message as the CI log.

The change

  1. ignore_hosts in vcr_config. loongsuite-instrumentation-litellm's own
    conftest already does this, with the comment "Ignore telemetry/analytics to
    avoid slow recordings or playback failures"
    ; the crewai conftest had none.
    Added raw.githubusercontent.com plus the analytics hosts crewai and litellm
    reach.

  2. LITELLM_LOCAL_MODEL_COST_MAP=True via os.environ.setdefault, so the
    fetch never starts. litellm's get_model_cost_map documents it: "If
    LITELLM_LOCAL_MODEL_COST_MAP is set, uses the local backup only."
    Measured on
    import:

    socket connects during import litellm
    default several, to raw.githubusercontent.com
    with the env var 0

    setdefault so a developer re-recording cassettes can override it.

Both, deliberately: the env var removes this request, and ignore_hosts keeps any
other incidental host from reaching the cassettes the same way. The recorded
cassettes are untouched and no test logic changes — the diff only adds keys to
vcr_config and one module-level setdefault.

What I could not verify here

I could not run the crewai suite: crewai pulls in numpy, which has no wheel for
the Python in this environment and fails to build from source. The VCR behaviour is
therefore verified directly, as above, rather than end to end.

litellm fetches its model cost map from raw.githubusercontent.com
on first use. VCR sees that request, and under record_mode=none it
has no recorded counterpart, so it raises
CannotOverwriteExistingCassetteException and its matchers fail
against the recorded api.openai.com interactions.
@ralf0131

Copy link
Copy Markdown
Collaborator

Please also fix the CI failures

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The changes are small, test-scoped, and align with existing VCR hardening patterns used elsewhere in the repo to prevent incidental HTTP traffic from breaking cassette playback.

Review effort: Lite
Findings: None

What changed in this PR

This PR hardens the loongsuite-instrumentation-crewai test suite’s VCR playback stability by preventing incidental LiteLLM/telemetry HTTP requests from interfering with cassette matching under record_mode=none, addressing the failure mode described in #230.

Changes:

  • Set LITELLM_LOCAL_MODEL_COST_MAP=True (via os.environ.setdefault) so LiteLLM uses its local model cost map instead of fetching from raw.githubusercontent.com.
  • Add a VCR ignore_hosts list to keep incidental analytics/telemetry and the LiteLLM cost-map host from being considered for cassette matching.
File Description
instrumentation-loongsuite/​loongsuite-instrumentation-crewai/​tests/​conftest.py Forces LiteLLM to avoid remote cost-map fetches and configures VCR to ignore incidental hosts to prevent cassette playback failures.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@RichardoMrMu

Copy link
Copy Markdown
Contributor Author

Thanks for the review. The failure you saw was the changelog check on d68c6b6 - I had not added a changelog entry. That is fixed in 286ca27, which adds the entry under instrumentation/loongsuite-instrumentation-crewai.

All 21 checks are green on the current head (286ca27), including changelog. Let me know if anything else needs changing.

@ralf0131
ralf0131 merged commit 2872e4f into alibaba:main Sep 20, 2026
22 checks passed
@RichardoMrMu
RichardoMrMu deleted the fix/crewai-vcr-ignore-incidental-hosts branch September 20, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants