Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/simulations.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Runs the scenarios in scenarios.yaml against your agent on LiveKit Cloud.
# Simulations spend real inference, so this runs on merges to main and on demand
# rather than on every pull request push. Run
# `lk agent simulate --scenarios scenarios.yaml` locally while iterating.
# `lk agent simulate text --scenarios scenarios.yaml` locally while iterating.
name: Simulations

on:
Expand Down Expand Up @@ -43,4 +43,4 @@ jobs:
run: curl -sSL https://get.livekit.io/cli | bash

- name: Run simulations
run: lk agent simulate --scenarios scenarios.yaml
run: lk agent simulate text --scenarios scenarios.yaml
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Voice AI agents are highly sensitive to excessive latency. For this reason, it's

## Testing

When possible, add tests for agent behavior. Add a scenario to `scenarios.yaml` and run it with `lk agent simulate --scenarios scenarios.yaml`. The scenarios run in CI on every merge to main; read the [simulations documentation](https://docs.livekit.io/agents/start/testing/simulations/) before editing them.
When possible, add tests for agent behavior. Add a scenario to `scenarios.yaml` and run it with `lk agent simulate text --scenarios scenarios.yaml`. The scenarios run in CI on every merge to main; read the [simulations documentation](https://docs.livekit.io/agents/start/testing/simulations/) before editing them.

For turn-level checks that don't need a live session, use the in-process [testing framework](https://docs.livekit.io/agents/start/testing/); `tests/test_agent.py` has a commented-out example. Run those with `uv run pytest`.

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ For advanced customization, see the [complete frontend guide](https://docs.livek
Simulations run full multi-turn conversations between a simulated user and your agent on LiveKit Cloud, then judge each transcript. The scenarios live in [`scenarios.yaml`](scenarios.yaml). Run them locally with the [LiveKit CLI](https://docs.livekit.io/intro/basics/cli/):

```console
lk agent simulate --scenarios scenarios.yaml
lk agent simulate text --scenarios scenarios.yaml
```

The `Simulations` workflow in `.github/workflows/simulations.yml` runs the same file on every merge to `main` and on demand from the Actions tab. It runs there rather than on every pull request push because each run spends real inference. See the [simulations guide](https://docs.livekit.io/agents/start/testing/simulations/) for how to write scenarios and read results.
Expand Down
2 changes: 1 addition & 1 deletion scenarios.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Simulation scenarios for the starter assistant. Each one is a conversation a
# simulated user has with your agent, judged against agent_expectations. Run them
# with `lk agent simulate --scenarios scenarios.yaml`; the Simulations workflow
# with `lk agent simulate text --scenarios scenarios.yaml`; the Simulations workflow
# runs them on every merge to main.
#
# Add a scenario whenever you change instructions or add a tool, and pin any dates
Expand Down
Loading