Repository navigation
agent_simulation: add simulator flow nodes to scenarios - #1856
chenghao-mou wants to merge 2 commits into
Conversation
A scenario could only describe one persona. For outbound calls the simulator needs to play the far end: voicemail, IVR menus, screeners, holds, and people, and the transitions between them. ScenarioGroup.steps holds named SimulatorSteps shared by the group's scenarios, and Scenario.start names the step a call starts at. Scenario.max_duration caps such a call.
🦋 Changeset detectedLatest commit: 6cffd83 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| // One step of the far end of a simulated outbound call: a machine line, a | ||
| // person, a voicemail box, or silence. Steps name each other in then, routes, | ||
| // timeout, max_visits, and voicemail.next. | ||
| message SimulatorStep { |
There was a problem hiding this comment.
Should we call them Node instead of Step?
|
|
||
| // livekit/voicemail with these parameters: a beep, then the agent is recorded | ||
| // in silence. | ||
| message Voicemail { |
|
|
||
| // livekit/voicemail with these parameters: a beep, then the agent is recorded | ||
| // in silence. | ||
| message Voicemail { |
There was a problem hiding this comment.
I think a lot of those options may not be that relevant to users?
They seem like a lot of "options" or "randomness" that we should handle automatically.
Then, the failure rate across multiple runs (let's say 5) would determine how robust your agent is at handling voicemails. (because we just pick "randomly")
There was a problem hiding this comment.
This is mainly for the machine node, where you want some degree of rigidity. But we can fix it, or at least, most of them, for the user too.
…lt-in SimulatorStep becomes SimulatorNode and ScenarioGroup.steps becomes nodes. Nodes no longer configure voicemail: the built-in livekit/voicemail picks the box per run (beep, waits, and whether a menu follows the message), so repeated runs show how an agent handles different boxes.
chenghao-mou
changed the title
2 participants
Adds simulator flows to agent simulation scenarios.
A scenario today describes one persona. For an outbound call, the simulator needs to play the far end: a voicemail greeting and beep, an IVR menu that routes on DTMF, a screener, a hold, and the person who picks up after it. A flow is a set of named nodes; the simulator walks them as the call goes.
ScenarioGroup.nodes: namedSimulatorNodes, shared by every scenario in the group. Nodes can also name the built-inslivekit/hangup,livekit/voicemail, andlivekit/unavailable.Scenario.start: the node a call starts at. A scenario setsstartorinstructions, not both.Scenario.max_duration: a cap on such a call (unset is 300s).livekit/voicemailis the only voicemail box, and nodes cannot configure one. Each run picks the box: whether it beeps, how long it waits for silence, and whether a send-or-re-record menu follows the message. Repeated runs then show how an agent handles different boxes.In
scenarios.yaml:All fields are new, so the change is wire-compatible. The server compiles and validates each scenario's flow at
CreateSimulationRunand keeps the nodes out ofSimulationDispatch, so the agent under test never reads them.Companion PRs: livekit/livekit-cli#1031 reads
nodesandstartfromscenarios.yaml; the agent-service compile, validator, and runtime are in a private PR.Addresses AGT-3725