From b771d0f1c74a92ef7d3b06dfa14d3ea00a0fb6d7 Mon Sep 17 00:00:00 2001 From: Ben Cherry Date: Wed, 23 Sep 2026 22:23:40 -0700 Subject: [PATCH] Fix the simulations CI job: lk agent simulate needs the text subcommand Since livekit-cli v2.18, the root `lk agent simulate` command only prints help and exits 0; the mode is a required subcommand (`text` or `audio`). The simulations job has been running the old flat form, so it has been passing without starting a single simulation. Switch the job and the four places that show the command to `lk agent simulate text --scenarios scenarios.yaml`. Text mode is the right mode for CI: it exercises the LLM, tools and conversation logic without the audio pipeline, and is scheduled so it doesn't compete with live sessions for inference. --- .github/workflows/simulations.yml | 4 ++-- AGENTS.md | 2 +- README.md | 2 +- scenarios.yaml | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/simulations.yml b/.github/workflows/simulations.yml index 03f03d6..bebb4f1 100644 --- a/.github/workflows/simulations.yml +++ b/.github/workflows/simulations.yml @@ -1,7 +1,7 @@ # Runs the scenarios in scenarios.yaml against your agent on LiveKit Cloud. # Simulations spend real inference, so this runs on merges to main and on demand # rather than on every pull request push. Run -# `lk agent simulate --scenarios scenarios.yaml` locally while iterating. +# `lk agent simulate text --scenarios scenarios.yaml` locally while iterating. name: Simulations on: @@ -43,4 +43,4 @@ jobs: run: curl -sSL https://get.livekit.io/cli | bash - name: Run simulations - run: lk agent simulate --scenarios scenarios.yaml + run: lk agent simulate text --scenarios scenarios.yaml diff --git a/AGENTS.md b/AGENTS.md index 8ddadd3..e93b752 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,7 +42,7 @@ Voice AI agents are highly sensitive to excessive latency. For this reason, it's ## Testing -When possible, add tests for agent behavior. Add a scenario to `scenarios.yaml` and run it with `lk agent simulate --scenarios scenarios.yaml`. The scenarios run in CI on every merge to main; read the [simulations documentation](https://docs.livekit.io/agents/start/testing/simulations/) before editing them. +When possible, add tests for agent behavior. Add a scenario to `scenarios.yaml` and run it with `lk agent simulate text --scenarios scenarios.yaml`. The scenarios run in CI on every merge to main; read the [simulations documentation](https://docs.livekit.io/agents/start/testing/simulations/) before editing them. For turn-level checks that don't need a live session, use the in-process [testing framework](https://docs.livekit.io/agents/start/testing/); `tests/test_agent.py` has a commented-out example. Run those with `uv run pytest`. diff --git a/README.md b/README.md index d55659a..e32a547 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,7 @@ For advanced customization, see the [complete frontend guide](https://docs.livek Simulations run full multi-turn conversations between a simulated user and your agent on LiveKit Cloud, then judge each transcript. The scenarios live in [`scenarios.yaml`](scenarios.yaml). Run them locally with the [LiveKit CLI](https://docs.livekit.io/intro/basics/cli/): ```console -lk agent simulate --scenarios scenarios.yaml +lk agent simulate text --scenarios scenarios.yaml ``` The `Simulations` workflow in `.github/workflows/simulations.yml` runs the same file on every merge to `main` and on demand from the Actions tab. It runs there rather than on every pull request push because each run spends real inference. See the [simulations guide](https://docs.livekit.io/agents/start/testing/simulations/) for how to write scenarios and read results. diff --git a/scenarios.yaml b/scenarios.yaml index 812f50f..805c681 100644 --- a/scenarios.yaml +++ b/scenarios.yaml @@ -1,6 +1,6 @@ # Simulation scenarios for the starter assistant. Each one is a conversation a # simulated user has with your agent, judged against agent_expectations. Run them -# with `lk agent simulate --scenarios scenarios.yaml`; the Simulations workflow +# with `lk agent simulate text --scenarios scenarios.yaml`; the Simulations workflow # runs them on every merge to main. # # Add a scenario whenever you change instructions or add a tool, and pin any dates