> ## Documentation Index > Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt > Use this file to discover all available pages before exploring further. # Scripted Scenarios > The scripted scenario format: user turns, expected events, eval, text_contains and llm_marker assertions, function calls, and latency budgets. A scripted scenario writes the user's turns out, each with the events you expect your agent to emit in response. It lives in a [scenario file](/pipecat/evals/scenario-configuration#scenario-files)'s `scenarios:` list. A scenario with a `turns:` key is scripted; one with a `persona:` key is a [simulated scenario](/pipecat/evals/simulated-scenarios), where an LLM plays the user instead. This page covers the full scripted format. If you haven't run a scenario yet, start with the [quickstart](/pipecat/evals/quickstart). Use a scripted scenario when you want exact control over the user's side. You know what the user says on every turn, so you can assert exactly what the agent must do: call this tool with these arguments, say this, answer within this budget, recover from this interruption. The input is the same every run, so a failure is easy to reproduce and fix. Use a simulation when you want to check a goal instead, and let the caller adapt to the agent. ## Anatomy of a scenario ```yaml theme={null} name: multi_turn # required: the file's name judge: # optional: judge modality and LLM (defaults shown below) eval: service: ollama model: gemma4:12b extra: { reasoning_effort: none } scenarios: # required: the scenarios the file holds - name: multi_turn # required: this scenario's name turns: # required: the conversation, in order - user: "My name is Alex, and I'm planning a trip to Italy." expect: - event: response eval: "acknowledges the user's message (the name Alex and/or the trip to Italy)" - user: "Remind me, what's my name and where am I going?" expect: - event: response eval: "recalls that the user's name is Alex and the destination is Italy" ``` Each turn optionally sends a user utterance (`user:`) and lists the events expected in response (`expect:`). Expected events must arrive in the order listed, but the agent may emit other events in between, so you don't have to enumerate everything it does. The scenario runs as `multi_turn/multi_turn`, its file's name and then its own. A file can hold several scenarios, and any scenario key at the top of the file, like the `judge:` above, is the default for all of them. See [Several scenarios in one file](#several-scenarios-in-one-file). The rest of this page is in four parts: The shared `user:` and `judge:` blocks, plus the scripted-only `context:` and `stop_on_failure:`. Drive each turn with an utterance, keypresses, an image, or timing. The semantic events the agent emits, and what each one means. Check an event's content or timing with `eval:`, `text_contains:`, and more. ## Configuration The file format, the `user:` and `judge:` blocks, `factory:`, `!include`, running scenarios back to back, and the disconnect path are shared with [simulated scenarios](/pipecat/evals/simulated-scenarios) and documented once, on [Scenario Configuration](/pipecat/evals/scenario-configuration). A scenario with none of those blocks runs entirely in [text mode](/pipecat/evals/scenario-configuration#text-and-audio-modes) with the default judge, which is the fastest way to start. Two fields are scripted-only, since a simulation has no scripted turns to seed or to score. Like any scenario key, both may sit on the scenario or at the top of the file as a default for [every scenario in it](#several-scenarios-in-one-file): ### Seeding the context with `context:` By default the harness leaves the bot's LLM context alone: whatever the bot sets up for itself (for example, a system prompt added in its connect handler) is what the scenario runs against. Provide `context:` to replace that with messages of your own, which lets a scenario start mid-conversation: ```yaml theme={null} context: - role: developer content: "The user has already introduced themselves as Alex." - role: assistant content: "Nice to meet you, Alex! How can I help?" ``` The harness sends these right after the bot-ready handshake as an `LLMMessagesUpdateFrame` that replaces the bot's context wholesale. Omit `context:` and the harness sends nothing, leaving the bot's own context in place. ### Scoring every turn By default the first turn with a failed assertion ends the scenario, since a conversation that has gone wrong rarely tells you much about the turns after it. Set `stop_on_failure: false` when the turns are independent and you want a score across all of them, for example when benchmarking intent classification over a list of utterances: ```yaml theme={null} name: intent_benchmark stop_on_failure: false scenarios: - name: intent_benchmark turns: - user: "Book me a flight to Tokyo." expect: - event: function_call within_ms: 15000 calls: [{ name: book_flight }] ``` Give those turns an explicit `within_ms`. With the 60-second default, a bot that has stopped answering costs a full budget on every remaining turn. This governs progression from one turn to the next. Within a turn, an expectation that times out still ends that turn's matching, because a turn's expectations share one deadline anchored at the moment its input was sent. ## User turns Each turn drives the agent by speaking (a `user:` utterance) or pressing keys (a `dtmf:` sequence); the two are mutually exclusive. A turn can also register an `image:`, or be observation-only with no input. `send_after:` controls when the input is sent. ### Utterances with `user:` Each turn's `user:` field is the user's utterance for that turn, a plain string. You write it the same way in both modes; whether it's delivered as text or synthesized into real speech is set once by the [`user:` block](/pipecat/evals/scenario-configuration#user-delivery-with-user), not per turn. A turn without a `user:` field is observation-only: the harness just waits for the expected events. This is how you test agent-first behavior like an on-connect greeting: ```yaml theme={null} turns: # No user input: just wait for the agent to speak first. - expect: - event: response eval: "the bot opens the conversation with a greeting or an offer to help" ``` ### Playing audio files with `audio:` In audio mode, a turn can play a recording instead of synthesizing its `user:` text. The `audio:` field (a path relative to the scenario file) names the audio file to stream to the agent: ```yaml theme={null} turns: - user: "What is the capital of Germany?" audio: ../assets/capital_question.wav expect: - event: user_transcription text_contains: "capital of Germany" - event: response eval: "the response says the capital of Germany is Berlin" ``` The file is sent at its own sample rate (it need not match the agent's input rate), and any format `soundfile` reads works: WAV, MP3, FLAC, OGG. Multi-channel audio is downmixed to mono. `user:` is required alongside `audio:` and gives what the recording says, since the judge and `text_contains` see it as the turn's input. A scenario whose spoken turns all name an `audio:` file needs no [`user.speech:` block](/pipecat/evals/scenario-configuration#user-delivery-with-user), since nothing is synthesized. ### DTMF keypresses with `dtmf:` Instead of a `user:` utterance, a turn can press phone keypad keys with `dtmf:`. The two are mutually exclusive: a turn either speaks or presses keys. This drives keypad menus (IVR) and any agent that reacts to telephony tones: ```yaml theme={null} turns: - dtmf: "123#" expect: - event: user_transcription text_contains: "DTMF: 123#" - event: response eval: "confirms the entered digits" ``` Each character is sent as one `InputDTMFFrame`, the same path a telephony transport's keypress takes, regardless of the scenario's `user:`/`judge:` modality. Valid characters are the keypad entries `0`-`9`, `*`, and `#`; any other character is a parse error. Quote the value in YAML (`dtmf: "123#"`). An unquoted `#` starts a YAML comment, so `dtmf: 123#` would silently drop the `#`. An unquoted all-digit sequence (`dtmf: 123`) is coerced to a string for you, but quoting is the safe habit. A bot running a [`DTMFAggregator`](/api-reference/server/utilities/dtmf-aggregator) accumulates the keys and flushes them into a `DTMF: ...` transcription, which (with the default transcription-based turn-start strategy) drives a full user turn: `user_started_speaking`, `user_transcription`, `user_stopped_speaking`, and the agent's response. So a `dtmf:` turn can assert on `user_transcription` and `response` just like a spoken turn. The aggregator flushes either on the `#` terminator or on its idle timeout. To exercise the idle-timeout path, omit the `#` and pace the keys with a time-based [`send_after:`](#scheduling-with-send_after): ```yaml theme={null} turns: # Flushes immediately on the '#' terminator. - dtmf: "1#" expect: - event: response eval: "states the business hours" # No terminator: the aggregator flushes on its idle timeout instead. - dtmf: "2" expect: - event: response eval: "gives the office location" ``` Like any input turn, `expect:` is optional on a `dtmf:` turn: omit it for a turn that only presses keys, with the assertion living on a later turn. ### Vision with `image:` A turn may register an image with `image:` (a path relative to the scenario file). When a vision agent requests a user image during the turn, the eval transport serves it: ```yaml theme={null} turns: - user: "What do you see in this image?" image: assets/cat.jpg expect: - event: response eval: "the response describes a cat" ``` ### Scheduling with `send_after:` By default, a turn is sent once the agent has finished speaking, like a caller who waits for the sentence to end. This ensures a reply the previous turn was satisfied with early is never talked over. A turn may override this with `send_after:`, which controls when the input (its `user:` utterance or [`dtmf:`](#dtmf-keypresses-with-dtmf) keypresses) is sent relative to a prior event or after a plain delay. Anchoring it to an event is how you script barge-in tests: ```yaml theme={null} turns: - user: "Tell me a long, detailed story about the history of Paris." expect: - event: llm_started # Interrupt 2 seconds after the agent starts its long answer. - user: "Actually, never mind that. What's the capital of Japan?" send_after: event: llm_started delay_ms: 2000 expect: - event: response eval: "the response says the capital of Japan is Tokyo, instead of continuing the Paris story" ``` The `event:` anchor is optional. A bare `send_after: { delay_ms: 500 }` is a pure time delay measured from the previous turn's send, with no event to wait on. This is handy for pacing turns by time rather than off a bot event (for example, spacing out [DTMF keypresses](#dtmf-keypresses-with-dtmf) to exercise an aggregator's idle-timeout flush): ```yaml theme={null} turns: - dtmf: "1" # Wait 1.5s after the first keypress before sending the next. - dtmf: "2" send_after: delay_ms: 1500 ``` A `send_after:` with no `event:` and a zero `delay_ms` is rejected as a no-op: give it an `event:`, a positive `delay_ms`, or both. ## Events Scenarios assert on a small set of semantic events, mapped from the RTVI messages the agent emits: | Event | Meaning | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `response` | The agent's reply. In audio mode this is a transcription of the agent's actual synthesized speech; in text mode it resolves to `llm_response`. Prefer this for content checks. | | `llm_response` | The LLM's text output for the turn. Available in both modes. | | `tts_response` | The text the TTS reports speaking, one segment at a time. Audio mode only. | | `llm_started` | The LLM began generating a response. | | `llm_marker` | The turn-completion marker the agent's LLM produced, reported when the response ends. Only sent when a scenario asks for it. See [Turn-completion markers](#turn-completion-markers-with-llm_marker). | | `bot_started_speaking` | The agent reported itself as speaking (emitted by the pipeline). | | `bot_stopped_speaking` | The agent reported itself as quiet (emitted by the pipeline). | | `bot_interrupted` | The agent's reply was cut off by the user, as an anchor for a barge-in test. | | `function_call` | The LLM called a function. | | `function_call_stopped` | A function call ended. Its `args` carry `tool_call_id` and `cancelled`, so a scenario can tell work that was stopped from work that finished on its own. | | `user_transcription` | The agent's STT finalized a transcription of the user. Audio mode only, except on [DTMF turns](#dtmf-keypresses-with-dtmf), where the aggregated keys become a transcription in either mode. | | `user_started_speaking` | The agent's turn detection reported the start of a user turn. Audio mode, or a [DTMF turn](#dtmf-keypresses-with-dtmf) in either mode. | | `user_stopped_speaking` | The agent's turn detection reported the end of a user turn. Audio mode, or a [DTMF turn](#dtmf-keypresses-with-dtmf) in either mode. | | `vad_user_started_speaking` | The raw VAD signal that user speech started, ungated by turn detection. Audio mode only. | | `vad_user_stopped_speaking` | The raw VAD signal that user speech stopped, ungated by turn detection. A timing anchor when a turn-detection strategy defers `user_stopped_speaking`, as [filtering incomplete turns](/api-reference/server/utilities/turn-management/filter-incomplete-turns) does. Audio mode only. | Use `response` for the agent's reply unless you have a reason not to. It's modality-agnostic: the same scenario judges LLM text in text mode and the transcription of real spoken audio in audio mode, so one file covers both. ## Assertions Each entry in `expect:` names an event and, optionally, asserts on its content or timing. ### Semantic judging with `eval:` The `eval:` field is a natural-language criterion that the event's text must satisfy, decided by the [judge LLM](/pipecat/evals/scenario-configuration#judging-with-judge): ```yaml theme={null} - user: "What's 2 plus 2?" expect: - event: response eval: "the response says the answer is four" ``` The judge sees the whole conversation so far, so it can resolve terse or context-dependent replies (like "That's four"). It also understands that audio-mode responses come from a speech-to-text pass and judges intended meaning rather than exact spelling, so "for" transcribed instead of "four" still passes. The judge handles interim replies gracefully: if the agent says "Let me check on that." before the real answer, the harness keeps accumulating response text and re-judges until the criterion is met or the time budget runs out. `eval:` only makes sense on the agent's text output (`response`, `llm_response`, `tts_response`). ### Substring checks with `text_contains:` For exact content, `text_contains:` does a substring check, ignoring whitespace differences: ```yaml theme={null} - user: "What is the capital of France?" expect: - event: response text_contains: "Paris" ``` On `response`, `llm_response`, and `tts_response`, the harness accumulates successive segments and re-checks on each new segment until the check passes. On `user_transcription`, it accumulates the STT's final transcription segments within the turn, so an STT that finalizes an utterance in pieces can still satisfy a phrase that spans them. `text_excludes:` is the mirror: the event's text must not hold the substring. It fails with the kind `text_present`. Its use is a marker the LLM let slip into its reply, which would reach the user: ```yaml theme={null} - user: "What is the capital of Germany?" expect: - event: llm_response text_contains: "Berlin" text_excludes: "●" ``` ### Asserting nothing arrives with `absent:` `absent: true` inverts an expectation: no event of that type may arrive before its `within_ms` budget expires. It matches on the event type alone, so it can't be combined with `text_contains:`, `eval:`, or `calls:`. Set `within_ms` explicitly, since the default budget makes the quiet window a full minute. A `response` that continues the reply an earlier expectation matched is not a new one; only a reply the agent began after that match counts. This is the check for a duplicate reply, or for an agent that should hold its turn: ```yaml theme={null} - user: "I'd go to Japan because" expect: - event: response absent: true # the agent waits for the user to finish within_ms: 3000 ``` ### Latency budgets with `within_ms:` `within_ms:` bounds how long after the turn's user send the event may arrive. All of a turn's expectations share that one anchor: ```yaml theme={null} - user: "What is the capital of France?" expect: - event: llm_started within_ms: 2000 # the LLM must start responding within 2s - event: response text_contains: "Paris" ``` When omitted, an expectation defaults to a generous 60 second budget (configurable with `--timeout`), so timing is only asserted when you ask for it. Because every deadline is measured from the send, time spent matching earlier expectations counts against later ones. In the example above, if `llm_started` arrives at 1.5 seconds, the `response` (with the default 60 second budget) has 58.5 seconds left, and a turn that stalls completely fails within a single budget rather than one per expectation. ### Function calls A `function_call` expectation asserts that the turn invoked one or more tools. List the expected calls under `calls:`; they're matched by name in any order, and the expectation passes once all are found: ```yaml theme={null} - user: "What's the weather in San Francisco? And recommend a restaurant." expect: - event: function_call calls: - name: get_current_weather args: { location: "San Francisco" } - name: get_restaurant_recommendation - event: response eval: "describes the weather and recommends a restaurant" ``` `args` is a subset check: every listed key/value must be present in the call's arguments, and extra arguments are ignored. A single expected call can use the `name:`/`args:` shorthand directly on the expectation, and a bare `function_call` with neither just asserts that some call happened. Arguments take part in the matching, so the turn is satisfied by any call matching both the name and the arguments. A call the model gets wrong and immediately repeats correctly still passes. When nothing matches, the failure names the arguments that did arrive. #### Judging a call with `eval:` `args:` matches verbatim, which is no use for an argument the model phrases in its own words. A `function_call` expectation can carry an `eval:` instead, or as well: each call the expectation matches is put to the [judge LLM](/pipecat/evals/scenario-configuration#judging-with-judge) by name and arguments, over the conversation so far, under a judge prompt of its own. A rejected call fails the turn with the kind `judge_no`: ```yaml theme={null} - user: "I'd like to propose a talk on OpenTelemetry tracing. I'm Jennifer Smith." expect: - event: function_call calls: - name: submit_session_suggestion eval: "a session about OpenTelemetry tracing, submitted for Jennifer Smith" ``` A `function_call_stopped` carries no arguments, only how the call ended, so it takes no `eval:`. `function_call_stopped` takes the same `calls:` shape and reports a call ending, which is how a scenario asserts that a [cancellable tool](/pipecat/learn/function-calling#async-function-call-cancellation) was actually stopped: ```yaml theme={null} - user: "Actually, cancel that report." expect: - event: function_call_stopped calls: - name: write_report args: { cancelled: true } ``` ### Turn-completion markers with `llm_marker` An agent that [filters incomplete user turns](/api-reference/server/utilities/turn-management/filter-incomplete-turns) has its LLM open every response with a marker saying whether the user's turn was complete. The `llm_marker` event reports the marker the LLM produced when the response ends, and `marker:` names its meaning: `complete` (the turn was finished and the agent answers), `short` (the user was cut off and the agent waits), `long` (the user asked for time), or `incomplete` for either of the last two. It checks the marker's meaning, not its text, so a scenario holds whatever marker characters the agent configured. A bare `llm_marker` asserts only that the agent read a marker: ```yaml theme={null} turns: - user: "Let me think about it, hmmm" expect: - event: llm_marker marker: incomplete # the agent held the turn open - user: "I think I'd go to Japan." expect: - event: llm_marker marker: complete # ... and answered this one - event: response eval: "engages with the user's answer about Japan" ``` Markers never reach clients by default. A scenario that asserts on one asks the agent to report them, so no other client ever sees them. An agent that doesn't use turn-completion markers never emits the event, and the expectation times out. The event also carries the response's raw text, as the LLM produced it before the agent held anything back, so a scenario can check how well the LLM follows the marker protocol. `marker_first:` asserts that nothing comes before the marker, `markers:` how many markers the text holds, and `text_after:` whether text follows the first marker, which a complete turn should have and an incomplete one should not: ```yaml theme={null} - user: "I'd go to Japan because" expect: - event: llm_marker marker: short marker_first: true markers: 1 text_after: false ``` A marker the LLM lets slip into its reply reaches the user, so pair this with a [`text_excludes:`](#substring-checks-with-text_contains) on the reply. In audio mode, anchor the follow-up turn on `vad_user_stopped_speaking` rather than `user_stopped_speaking`, since turn detection defers the latter while the turn is held open. A run's result records what each expectation matched, the marker an `llm_marker` saw included, so a passed run keeps the marker it read. See [`results.jsonl`](/pipecat/evals/suites#run-output). ## Several scenarios in one file A file's `scenarios:` list can hold several scripted scenarios, and any key at the top of the file is the default for all of them. That suits testing one behavior through many short conversations: the file sets the `judge:` and the `context:` once, and each scenario is a conversation of a few turns. Each runs on its own, against its own bot, as `turn_completion/short_answer` and `turn_completion/cutoff` here, and a suite's `-s turn_completion` runs them all: ```yaml turn_completion.yaml theme={null} name: turn_completion judge: !include ../judge_text.yaml context: - role: system content: "You are a travel assistant." scenarios: - name: short_answer turns: - user: "Japan." expect: - event: llm_marker marker: complete - name: cutoff turns: - user: "I'd go to Japan because" expect: - event: llm_marker marker: short ``` Every scripted key can sit at the top of the file this way: `turns:`, `context:`, and `stop_on_failure:`, and the shared `user:`, `judge:`, and `trigger_disconnect:`. A scenario that sets the same key replaces the whole value, so a scenario's own `context:` is written out in full, never added to the file's. Sharing `turns:` is for a file whose scenarios hold the same conversation and differ in one thing only. With `turns:` at the top and one scenario per judge or modality, every judge sees exactly the same conversation, which is how Pipecat's `interruption` scenario runs in text and in audio: ```yaml interruption.yaml theme={null} name: interruption turns: - user: "Tell me a long, detailed story about the history of Paris." expect: - event: llm_started - user: "Actually, never mind that. What's the capital of Japan?" send_after: { event: llm_started, delay_ms: 2000 } expect: - event: bot_interrupted - event: response eval: "says Tokyo instead of continuing the Paris story" scenarios: - name: text judge: !include ../judge_text.yaml - name: audio user: !include ../user_audio.yaml judge: !include ../judge_audio.yaml ``` The full rules, and a simulation's side of them, are on [Scenario files](/pipecat/evals/scenario-configuration#scenario-files). ## Next steps Hand the user's side to an LLM with a persona and a goal, and judge the whole conversation. Let a coding assistant write agent code, run evals, and iterate automatically until the agent is better.