test(crewai): keep incidental requests out of the VCR cassettes - #270
Conversation
litellm fetches its model cost map from raw.githubusercontent.com on first use. VCR sees that request, and under record_mode=none it has no recorded counterpart, so it raises CannotOverwriteExistingCassetteException and its matchers fail against the recorded api.openai.com interactions.
|
Please also fix the CI failures |
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are small, test-scoped, and align with existing VCR hardening patterns used elsewhere in the repo to prevent incidental HTTP traffic from breaking cassette playback.
Review effort: Lite
Findings: None
What changed in this PR
This PR hardens the loongsuite-instrumentation-crewai test suite’s VCR playback stability by preventing incidental LiteLLM/telemetry HTTP requests from interfering with cassette matching under record_mode=none, addressing the failure mode described in #230.
Changes:
- Set
LITELLM_LOCAL_MODEL_COST_MAP=True(viaos.environ.setdefault) so LiteLLM uses its local model cost map instead of fetching fromraw.githubusercontent.com. - Add a VCR
ignore_hostslist to keep incidental analytics/telemetry and the LiteLLM cost-map host from being considered for cassette matching.
| File | Description |
|---|---|
| instrumentation-loongsuite/loongsuite-instrumentation-crewai/tests/conftest.py | Forces LiteLLM to avoid remote cost-map fetches and configures VCR to ignore incidental hosts to prevent cassette playback failures. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Assisted-by: Doubao
|
Thanks for the review. The failure you saw was the All 21 checks are green on the current head (286ca27), including |
Why
Hardening, not a bug fix. The CI failure reported in #230 is already fixed by #233,
and the crewai job is green on
main(run34801198149). This closes the gap thatmade that failure possible, so it cannot recur.
litellm fetches its model cost map from
raw.githubusercontent.comthe first timeit is used. That request is incidental to the code under test, but VCR still sees
it, and the crewai
conftest.pyhas noignore_hosts. Underrecord_mode=nonearequest to a host with no recorded counterpart raises — which is exactly the first
line of #230's CI log:
The
method/host/pathmatcher failures that followed are the same cause fromthe other side: the recordings are
api.openai.comcalls, so aGETtoraw.githubusercontent.commatches none of them.Mechanism, reproduced
vcrpyalone — no crewai, no credentials, no network. A cassette holding oneapi.openai.cominteraction, then a request toraw.githubusercontent.cominsideit:
Same exception class and message as the CI log.
The change
ignore_hostsinvcr_config.loongsuite-instrumentation-litellm's ownconftest already does this, with the comment "Ignore telemetry/analytics to
avoid slow recordings or playback failures"; the crewai conftest had none.
Added
raw.githubusercontent.complus the analytics hosts crewai and litellmreach.
LITELLM_LOCAL_MODEL_COST_MAP=Trueviaos.environ.setdefault, so thefetch never starts. litellm's
get_model_cost_mapdocuments it: "IfLITELLM_LOCAL_MODEL_COST_MAPis set, uses the local backup only." Measured onimport:
import litellmraw.githubusercontent.comsetdefaultso a developer re-recording cassettes can override it.Both, deliberately: the env var removes this request, and
ignore_hostskeeps anyother incidental host from reaching the cassettes the same way. The recorded
cassettes are untouched and no test logic changes — the diff only adds keys to
vcr_configand one module-levelsetdefault.What I could not verify here
I could not run the crewai suite:
crewaipulls innumpy, which has no wheel forthe Python in this environment and fails to build from source. The VCR behaviour is
therefore verified directly, as above, rather than end to end.