diff --git a/AGENTS.md b/AGENTS.md index 2f9d434..e97b5db 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,57 +1,66 @@ # AGENTS.md -This is a LiveKit Agents project. LiveKit Agents is a Node.js SDK for building voice AI agents. This project is intended to be used with LiveKit Cloud. See @README.md for more about the rest of the LiveKit ecosystem. +This is a LiveKit Agents project. LiveKit Agents is a Node.js SDK for building voice AI agents. This starter is designed to run in LiveKit Cloud. See @README.md for more about the rest of the LiveKit ecosystem. -The following is a guide for working with this project. +## Tooling -## Project structure +This Node.js project uses the `pnpm` package manager. -This Node.js project uses the `pnpm` package manager. You should always use `pnpm` to install dependencies, run the agent, and run tests. +Be sure to maintain code formatting, using `pnpm format` and `pnpm lint`. -All app-level code is in the `src/` directory. In general, simple agents can be constructed with a single `main.ts` file. Additional files can be added, but you must retain `main.ts` as the entrypoint (see the associated Dockerfile for how this is deployed). +## LiveKit documentation -Be sure to maintain code formatting. You can use the prettier formatter and eslint to format and lint the code. Scripts are available in `package.json`, including `pnpm format` and `pnpm lint`. - -## LiveKit Documentation - -LiveKit Agents is a fast-evolving project, and the documentation is updated frequently. You should always refer to the latest documentation when working with this project. For your convenience, LiveKit offers both a CLI and an MCP server that can be used to browse and search its documentation. If the developer has not yet installed the CLI, you should recommend that they install it. +Always refer to the latest documentation when working with this project. LiveKit offers both a CLI (`lk docs`) and an MCP server for browsing and searching its documentation. ### LiveKit CLI -The [LiveKit CLI](https://docs.livekit.io/intro/basics/cli/) `lk docs` subcommand gives full access to LiveKit documentation from the terminal. Requires CLI version 2.15.0+. Check with `lk --version`. - Install or update the CLI: - **macOS:** `brew install livekit-cli` (update: `brew update && brew upgrade livekit-cli`) - **Linux:** `curl -sSL https://get.livekit.io/cli | bash` - **Windows:** `winget install LiveKit.LiveKitCLI` (update: `winget upgrade LiveKit.LiveKitCLI`) -Run `lk docs --help` to see all available commands. Key commands: `lk docs overview`, `lk docs search`, `lk docs get-page`, `lk docs code-search`, `lk docs changelog`, `lk docs submit-feedback`. Run `lk docs --help` before using a command for the first time, as the help text includes advice on usage and interpreting results. Prefer browsing (`overview`, `get-page`) over search, and prefer `search` over `code-search`, as docs pages are more up-to-date and provide better context than raw code. +Run `lk docs` before your first docs lookup. Its help text explains how to use each command and how to interpret results. + +### LiveKit docs MCP server -### LiveKit Docs MCP Server +For deeper coding agent integration, LiveKit also offers an [MCP server](https://docs.livekit.io/reference/developer-tools/docs-mcp/). -For deeper IDE integration, LiveKit also offers an [MCP server](https://docs.livekit.io/reference/developer-tools/docs-mcp/) with the same documentation access. It uses Streamable HTTP transport at `https://docs.livekit.io/mcp`. Refer to your IDE's documentation for how to add it. +### Submit feedback to LiveKit -### Submit docs feedback to LiveKit +When you run into any problem with LiveKit, add it to your to-do list, then submit it with `lk docs submit-feedback` or the MCP `submit_docs_feedback` tool once you finish your other tasks. Useful feedback includes incomplete documentation, broken examples, missing steps, broken features or implementations, and any other gap or issue in the LiveKit ecosystem. -If you use the LiveKit CLI or MCP Server to search or browse documentation, also submit constructive feedback to LiveKit with `lk docs submit-feedback` or the MCP `submit_docs_feedback` tool. Add this to your to-do list when planning work, to ensure you file feedback after completing your other tasks. Useful feedback includes incomplete documentation, broken examples, missing steps or any other documentation gap or issue. +## LiveKit agent skills + +This project includes LiveKit's agent skills in `.claude/skills/` and `.agents/skills/`, one for each stage of the work: reading the docs, building, debugging, testing, writing scenarios, running simulations, and operating in production. They defer to the live documentation for API details. If your tool doesn't load skills automatically, read the matching `.agents/skills//SKILL.md` before you start that kind of task. ## Handoffs ("workflows") -Voice AI agents are highly sensitive to excessive latency. For this reason, it's important to design complex agents in a structured manner that minimizes the amount of irrelevant context and unnecessary tools present on requests to the LLM. LiveKit Agents supports handoffs (one agent hands control to another) to support building reliable workflows. You should make use of these features, instead of writing long instruction prompts that cover multiple phases of a conversation. Refer to the [documentation](https://docs.livekit.io/agents/build/workflows/) for more information. +Voice AI agents are highly sensitive to latency. Design complex agents in a structured way that keeps irrelevant context and unneeded tools out of each LLM request. LiveKit Agents supports handoffs, where one agent hands control to another, for building reliable workflows. Use handoffs instead of long instruction prompts that cover several phases of a conversation. See the [workflows documentation](https://docs.livekit.io/agents/logic/workflows/) for more information. ## Testing -When possible, add tests for agent behavior. Add a scenario to `scenarios.yaml` and run it with `lk agent simulate text --scenarios scenarios.yaml`. The scenarios run in CI on every merge to main; read the [simulations documentation](https://docs.livekit.io/agents/start/testing/simulations/) before editing them. +To keep agent behavior from regressing, add a scenario to `scenarios.yaml` and run it with `lk agent simulate text --scenarios scenarios.yaml`. Make sure the scenarios run in CI on every merge to `main`. Read the [simulations documentation](https://docs.livekit.io/testing/simulations/) before editing them. + +Important: when you modify core agent behavior such as instructions, tool descriptions, or tasks, workflows, and handoffs, never guess at what works. Start by writing a scenario for the desired behavior. For example, if you're adding a tool, write a scenario that exercises it, then iterate on the tool until the scenario passes. This is how you produce a working, reliable agent. + +After changing the agent, try it with the [agent debugger](https://docs.livekit.io/testing/debugger/) (CLI 2.18.8 or later) before calling the change done. Start the agent with `lk agent debugger start`, send user turns with `lk agent debugger say "..."`, and read the tool calls in each turn as well as the reply. Run `lk agent debugger restart` after every code edit, since a running session keeps the old code, and `lk agent debugger stop` when you're done. + +## Debugging -For turn-level checks that don't need a live session, use the in-process [testing framework](https://docs.livekit.io/agents/start/testing/); `src/agent.test.ts` has a commented-out example. Run those with `pnpm test`. +To investigate unexpected agent behavior: -Important: When modifying core agent behavior such as instructions, tool descriptions, and tasks/workflows/handoffs, never just guess what will work. Always use test-driven development (TDD) and begin by writing tests for the desired behavior. For instance, if you're planning to add a new tool, write one or more tests for the tool's behavior, then iterate on the tool until the tests pass correctly. This will ensure you are able to produce a working, reliable agent for the user. +- Reproduce it with `lk agent debugger`: send the turns that trigger the problem and read the tool calls and errors in each one. Add `--logs` to `say` to see log lines, including tracebacks, next to the turn that produced them. +- Add a simulation scenario once it's fixed, so a later change can't bring it back unnoticed. +- Run `lk agent dev --log-level DEBUG` for verbose logs from a local agent connected to LiveKit Cloud. +- Run `lk agent logs` to stream logs from a deployed agent. +- Ask the developer to open the [Agent Console](https://docs.livekit.io/testing/agent-console/) for speech problems such as turn-taking, interruptions, or transcription, which the text-only debugger can't show. It shows events, tool calls, and model timing for a live session. +- Check [Agent Observability](https://docs.livekit.io/testing/observability/) for transcripts, traces, logs, and recordings of sessions with real users. -## Feature parity with Python SDK +## Feature parity with the Python SDK -The Node.js SDK for LiveKit Agents has most, but not all, of the same features available in Python SDK for LiveKit Agents. You should always check the documentation for feature availability, and avoid using features that are not available in the Node.js SDK. +The Node.js SDK for LiveKit Agents has most, but not all, of the features in the Python SDK. Always check the documentation for feature availability, and avoid features the Node.js SDK doesn't support. -## LiveKit CLI +## Other CLI commands -Beyond documentation access, the LiveKit CLI (`lk`) supports other tasks such as managing SIP trunks for telephony-based agents. Run `lk --help` to explore available commands. +Beyond documentation access, the LiveKit CLI (`lk`) handles tasks such as managing SIP trunks for telephony agents. Run `lk --help` to explore available commands. diff --git a/README.md b/README.md index 89090c1..c8b7145 100644 --- a/README.md +++ b/README.md @@ -2,117 +2,109 @@ LiveKit logo -# LiveKit Agents Starter - Node.js +# LiveKit Agents starter for Node.js -A complete starter project for building voice AI apps with [LiveKit Agents for Node.js](https://github.com/livekit/agents-js) and [LiveKit Cloud](https://cloud.livekit.io/). +A starter project for building voice AI apps with [LiveKit Agents for Node.js](https://github.com/livekit/agents-js) and [LiveKit Cloud](https://cloud.livekit.io/). -The starter project includes: +The starter includes: -- A simple voice AI assistant, ready for extension and customization -- A voice AI pipeline built on [LiveKit Inference](https://docs.livekit.io/agents/models/inference), providing zero-configuration access to [models](https://docs.livekit.io/agents/models) from top labs - - Uses the fast, open-weight Gemma 4 31B model, [hosted by LiveKit](https://docs.livekit.io/agents/models/llm/livekit/) and tuned for optimal performance in voice AI, as the default LLM - - Uses Fish Audio S2.1 Pro for TTS, which renders the inline delivery markup that expressive mode relies on - - Supports more than 50 models from OpenAI, Cartesia, Deepgram, and other providers - - Access to a wide range of other models, including [Realtime models](https://docs.livekit.io/agents/models/realtime), through extensive plugin ecosystem -- Expressive mode, enabled by default: the framework injects the TTS provider's markup guide into the LLM prompt, so the model emits inline delivery tags (emotion, pacing, non-verbal sounds) that the TTS renders and the transcript never shows -- Eval suite based on the LiveKit Agents [testing & evaluation framework](https://docs.livekit.io/agents/start/testing) -- [LiveKit Turn Detector](https://docs.livekit.io/agents/logic/turns/turn-detector/), an end-of-turn model that listens to the user's audio directly, combining semantic understanding with acoustic cues for state-of-the-art accuracy across 14 languages -- [Background voice cancellation](https://docs.livekit.io/transport/media/noise-cancellation/) -- Deep session insights from LiveKit [Agent Observability](https://docs.livekit.io/deploy/observability/) -- A Dockerfile ready for [production deployment to LiveKit Cloud](https://docs.livekit.io/deploy/agents/) +- A simple [voice AI assistant](https://docs.livekit.io/agents/start/voice-ai/) to extend and customize. +- A voice pipeline built on [LiveKit Inference](https://docs.livekit.io/agents/models/inference/), which gives you access to [models](https://docs.livekit.io/agents/models/) from top labs with no extra configuration: + - The default LLM is Gemma 4 31B, an open-weight model [hosted by LiveKit](https://docs.livekit.io/agents/models/llm/livekit/) and tuned for voice AI. + - The default TTS is [Fish Audio S2.1 Pro](https://docs.livekit.io/agents/models/tts/fishaudio/), an expressive and cost-effective voice. + - More than 50 other models are available from OpenAI, Cartesia, Deepgram, and other providers. + - [Realtime models](https://docs.livekit.io/agents/models/realtime/) and many others are available through the [plugin ecosystem](https://docs.livekit.io/agents/models/#plugins). +- [Expressive mode](https://docs.livekit.io/agents/models/tts/expressive/), on by default, so your agent's voice carries emotion and pacing that fit the conversation. +- [LiveKit Turn Detector](https://docs.livekit.io/agents/logic/turns/turn-detector/), which knows when the user has finished speaking, in 14 languages. +- [Adaptive interruption handling](https://docs.livekit.io/agents/logic/turns/adaptive-interruption-handling/), which tells a real interruption from an "uh-huh" or background noise, so your agent doesn't stop talking when it shouldn't. +- [Background voice cancellation](https://docs.livekit.io/transport/media/noise-cancellation/). +- Session transcripts, traces, and recordings from LiveKit [Agent Observability](https://docs.livekit.io/testing/observability/). +- [Simulations](https://docs.livekit.io/testing/simulations/) that test full conversations with your agent, run in CI on every merge to `main`. +- A `Dockerfile` for [deploying to LiveKit Cloud](https://docs.livekit.io/deploy/agents/). -This starter app is compatible with any [custom web/mobile frontend](https://docs.livekit.io/frontends/) or [telephony](https://docs.livekit.io/telephony/). +The starter works with any [custom web or mobile frontend](https://docs.livekit.io/frontends/) or with [telephony](https://docs.livekit.io/telephony/). ## Using coding agents -This project is designed to work with coding agents like [Claude Code](https://claude.com/product/claude-code), [Cursor](https://www.cursor.com/), and [Codex](https://openai.com/codex/). +This project works with coding agents like [Claude Code](https://claude.com/product/claude-code), [Cursor](https://www.cursor.com/), and [Codex](https://openai.com/codex/). -For your convenience, LiveKit offers both a CLI and an [MCP server](https://docs.livekit.io/reference/developer-tools/docs-mcp/) that can be used to browse and search its documentation. The [LiveKit CLI](https://docs.livekit.io/intro/basics/cli/) (`lk docs`) works with any coding agent that can run shell commands. Install it for your platform: - -**macOS:** +LiveKit offers both a CLI and an [MCP server](https://docs.livekit.io/reference/developer-tools/docs-mcp/) for browsing and searching its documentation. Search returns short excerpts, so fetch the full page to read the details: ```console -brew install livekit-cli +lk docs search "testing my agent" +lk docs get-page /testing/unit-tests ``` -**Linux:** +The project also includes an [`AGENTS.md`](AGENTS.md) file and LiveKit's [agent skills](https://docs.livekit.io/intro/coding-agents/#agent-skills), so your coding agent follows LiveKit's best practices for workflows, handoffs, and testing, and tries its changes with the [agent debugger](https://docs.livekit.io/testing/debugger/). See the [coding agents guide](https://docs.livekit.io/intro/coding-agents/) for more details, including MCP server setup and how to update the skill. -```console -curl -sSL https://get.livekit.io/cli | bash -``` +## Dev setup -**Windows:** +Install the [LiveKit CLI](https://docs.livekit.io/intro/basics/cli/), version 2.18.8 or later: -```console -winget install LiveKit.LiveKitCLI -``` +- **macOS:** `brew install livekit-cli` +- **Linux:** `curl -sSL https://get.livekit.io/cli | bash` +- **Windows:** `winget install LiveKit.LiveKitCLI` -The `lk docs` subcommand requires version 2.15.0 or higher. Check your version with `lk --version` and update if needed. Once installed, your coding agent can search and browse LiveKit documentation directly from the terminal: +Check your version with `lk --version`. To update an existing install, see [Update the CLI](https://docs.livekit.io/reference/developer-tools/livekit-cli/#updates). -```console -lk docs search "voice agents" -lk docs get-page /agents/start/voice-ai-quickstart -``` +Then create a project from this template. The CLI clones the template and configures your environment: -See the [Coding agent support](https://docs.livekit.io/intro/coding-agents/) guide for more details, including MCP server setup. - -The project includes a complete [AGENTS.md](AGENTS.md) file for these assistants. You can modify this file to suit your needs. To learn more about this file, see [https://agents.md](https://agents.md). - -## Dev Setup - -Create a project from this template with the LiveKit CLI (recommended): - -```bash +```console lk cloud auth lk agent init my-agent --template agent-starter-node ``` -The CLI clones the template and configures your environment. Then follow the rest of this guide from [Run the agent](#run-the-agent). - -This project uses [pnpm](https://pnpm.io/) as the package manager. -
-Alternative: Manual setup without the CLI +Set up the project manually -Clone the repository and install dependencies: +Clone the repository and install dependencies with [pnpm](https://pnpm.io/): ```console +git clone https://github.com/livekit-examples/agent-starter-node.git cd agent-starter-node pnpm install ``` -Sign up for [LiveKit Cloud](https://cloud.livekit.io/) then set up the environment by copying `.env.example` to `.env.local` and filling in the required keys: - -- `LIVEKIT_URL` -- `LIVEKIT_API_KEY` -- `LIVEKIT_API_SECRET` - -You can load the LiveKit environment automatically using the [LiveKit CLI](https://docs.livekit.io/intro/basics/cli/): +Sign up for [LiveKit Cloud](https://cloud.livekit.io/), then copy `.env.example` to `.env.local` and fill it in. To have the CLI write your project's URL and API keys into the file instead, run: -```bash +```console lk cloud auth -lk app env -w -d .env.local +lk app env --write --destination .env.local ```
## Run the agent -To run the agent during development, use the `dev` command: +The `lk agent console`, `lk agent dev`, and `lk agent debugger` commands run your agent on your own machine. To talk to it in your terminal: + +```console +lk agent console +``` + +To connect it to LiveKit Cloud so a frontend, a phone call, or the [Agent Console](https://docs.livekit.io/testing/agent-console/) can reach it: + +```console +lk agent dev +``` + +To let a coding agent or a script test it one text turn at a time, use the [agent debugger](https://docs.livekit.io/testing/debugger/). Each turn prints the agent's reply along with the tool calls and handoffs behind it: ```console -pnpm run dev +lk agent debugger start +lk agent debugger say "Hi, what can you do?" +lk agent debugger stop ``` -In production, use the `start` command: +In production, run the agent directly: ```console -pnpm run start +node src/main.ts start ``` -## Frontend & Telephony +## Frontends and telephony -Get started quickly with our pre-built frontend starter apps, or add telephony support: +Pair the agent with a prebuilt frontend starter, or add telephony: | Platform | Link | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | @@ -124,36 +116,37 @@ Get started quickly with our pre-built frontend starter apps, or add telephony s | **Web Embed** | [`livekit-examples/agent-starter-embed`](https://github.com/livekit-examples/agent-starter-embed) | Voice AI widget for any website | | **Telephony** | [Documentation](https://docs.livekit.io/telephony/) | Add inbound or outbound calling to your agent | -For advanced customization, see the [complete frontend guide](https://docs.livekit.io/frontends/). +For more options, see the [frontend guide](https://docs.livekit.io/frontends/). -## Tests and evals +## Testing and debugging -Simulations run full multi-turn conversations between a simulated user and your agent on LiveKit Cloud, then judge each transcript. The scenarios live in [`scenarios.yaml`](scenarios.yaml). Run them locally with the [LiveKit CLI](https://docs.livekit.io/intro/basics/cli/): +Simulations run full multi-turn conversations between a simulated user and your agent on LiveKit Cloud, then judge each transcript. The scenarios live in [`scenarios.yaml`](scenarios.yaml). Run them locally with the CLI: ```console lk agent simulate text --scenarios scenarios.yaml ``` -The `Simulations` workflow in `.github/workflows/simulations.yml` runs the same file on every merge to `main` and on demand from the Actions tab. It runs there rather than on every pull request push because each run spends real inference. See the [simulations guide](https://docs.livekit.io/agents/start/testing/simulations/) for how to write scenarios and read results. +The `Simulations` workflow in [`.github/workflows/simulations.yml`](.github/workflows/simulations.yml) runs the same file on every merge to `main`, and on demand from the Actions tab. It doesn't run on every pull request push because each run uses real inference. See the [simulations guide](https://docs.livekit.io/testing/simulations/) for how to write scenarios and read results. -For turn-level checks that don't need a live session, the LiveKit Agents [testing & evaluation framework](https://docs.livekit.io/agents/start/testing/) runs your agent in-process under `vitest`. A commented-out example lives in [`src/agent.test.ts`](src/agent.test.ts). +To check a change turn by turn without a live session, use the [agent debugger](https://docs.livekit.io/testing/debugger/) shown in [Run the agent](#run-the-agent). -## Using this template repo for your own project +To debug a running agent, open it in the [Agent Console](https://docs.livekit.io/testing/agent-console/). It shows events, tool calls, and model timing as you talk to the agent. To stream logs from a deployed agent, run `lk agent logs`. -Once you've started your own project based on this repo, you should: +## Using this template for your own project -1. **Check in your `pnpm-lock.yaml`**: This file is currently untracked for the template, but you should commit it to your repository for reproducible builds and proper configuration management. (The same applies to `livekit.toml`, if you run your agents in LiveKit Cloud) +After you create your own project from this template: -2. **Add your own repository secrets**: You must [add secrets](https://docs.github.com/en/actions/how-tos/writing-workflows/choosing-what-your-workflow-does/using-secrets-in-github-actions) for `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET` so that the simulations can run in CI. +- **Commit `pnpm-lock.yaml`.** The template doesn't track it, but your project should, for reproducible builds. If you deploy to LiveKit Cloud, commit `livekit.toml` too. +- **Add repository secrets.** Add `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET` as [repository secrets](https://docs.github.com/en/actions/how-tos/writing-workflows/choosing-what-your-workflow-does/using-secrets-in-github-actions) so the simulations can run in CI. ## Deploying to production -This project is production-ready and includes a working `Dockerfile`. To deploy it to LiveKit Cloud or another environment, see the [deploying to production](https://docs.livekit.io/deploy/agents/) guide. +To deploy the agent to LiveKit Cloud or another environment with the included `Dockerfile`, see the [deployment guide](https://docs.livekit.io/deploy/agents/). ## Self-hosted LiveKit -You can also self-host LiveKit instead of using LiveKit Cloud. See the [self-hosting](https://docs.livekit.io/transport/self-hosting/local/) guide for more information. If you choose to self-host, you'll need to also use [model plugins](https://docs.livekit.io/agents/models/#plugins) instead of LiveKit Inference and will need to remove the [LiveKit Cloud noise cancellation](https://docs.livekit.io/transport/media/noise-cancellation/) plugin. +You can self-host LiveKit instead of using LiveKit Cloud. See the [self-hosting guide](https://docs.livekit.io/transport/self-hosting/local/). If you self-host, use [model plugins](https://docs.livekit.io/agents/models/#plugins) instead of LiveKit Inference, and remove the [LiveKit Cloud noise cancellation](https://docs.livekit.io/transport/media/noise-cancellation/) plugin. ## License -This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. +This project is licensed under the MIT License. See [LICENSE](LICENSE) for details. diff --git a/package.json b/package.json index 1f3f37e..cf13ffa 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "format:check": "prettier --check \"**/*.{ts,js,json,md}\"", "test": "vitest --run", "test:watch": "vitest", - "dev": "node src/main.ts dev", + "dev": "lk agent dev", "start": "node src/main.ts start" }, "engines": { diff --git a/taskfile.yaml b/taskfile.yaml index 9535de9..e308c77 100644 --- a/taskfile.yaml +++ b/taskfile.yaml @@ -15,7 +15,7 @@ tasks: - echo '' - echo '{{ indent .INDENT "cd" }} {{ .REL_PATH }}' - task: help_install_hint_if_needed - - echo '{{ indent .INDENT "pnpm dev" }}' + - echo '{{ indent .INDENT "lk agent dev" }}' - echo '' - task: help_open_web_console - echo 'To deploy your agent to LiveKit cloud:'