Writing Scenarios

A scenario is one test: who the simulated customer is, what they want, and what counts as success. Scenarios live on the Scenarios side of an agent’s Evals tab. Add them with the Evals Conductor or by hand.

With the Evals Conductor

Click Conductor to open a chat that writes tests for you. Ask for coverage in plain language — “cover refunds, an angry caller, and someone asking for a human” — and it creates the scenarios, then say “run them” to launch a batch and see results in the same chat. It can also edit, archive, and run existing tests, and you can attach documents (PDF, Markdown, TXT, or CSV; up to 5 files, 10 MB each) such as FAQs or call scripts for it to work from. Past conversations are kept under Conversations, and each one is private to the person who started it.

Manually

Choose Conductor ▾ → Add manually, fill in the form, and click Save scenario. Every scenario is one of two modes:

Simulation — an AI plays the customer, improvising from a goal and persona. Best for realistic, varied conversations.
Scripted — the customer sends a fixed list of messages, in order. Best for reproducing a specific conversation exactly.

Field	Mode	What it does
Scenario Name	Both	Required.
User’s goal	Simulation	Required. What the simulated customer is trying to achieve.
Scripted turns	Scripted	Required. The messages the customer sends; turn 1 is their first message.
Success criteria	Both	What the judge grades the run against, in plain language.
Success Checks / Forbidden Checks	Both	Specific things the agent must or must not do — see below.
Caller persona	Simulation	Optional name, background (Who they are), Mood / temperament, What they want, and Other notes, so the simulated customer behaves realistically.
Opening message	Simulation, chat agents	Optional. The customer’s first message.

Success and forbidden checks

Checks are precise pass/fail conditions evaluated on top of the success criteria. A success check must happen for the scenario to pass; if a forbidden check happens, the scenario fails regardless of anything else.

Check	Passes when
AI judge	An AI grader agrees with a plain-language statement — e.g. “The agent confirmed the appointment time.”
Tool called	The agent called the chosen tool during the run.
Regex	The text or pattern appears in what the agent said.
State reached	The agent reached the chosen conversation state (multi-prompt agents).
Field value	The agent collected the chosen structured field, optionally with a specific value.

Advanced settings

Setting	Default	What it does
Turn cap	16	Simulation only. The most back-and-forth turns before the run is ended and graded as it stands.
Soft cap	10	Simulation only. After this many turns, the simulated customer starts wrapping up naturally.
Disfluent caller	Off	Simulation, voice agents only. Adds filler words, hesitations, and interruptions (Low, Medium, High) to test messy real-world speech.
Input fields	—	Values for the `{{variables}}` in the agent’s prompts, the same values an API call or campaign CSV would supply.

Editing and archiving

To change a scenario, open it and use Edit (which opens the Evals Conductor) or Manual edit. Archive scenario removes it from the list; archived scenarios can’t be restored. Once you have scenarios, run them in a batch.

Get Started

Going Deeper

API Reference

With the Evals Conductor

Manually

Success and forbidden checks

Advanced settings

Editing and archiving

​With the Evals Conductor

​Manually

​Success and forbidden checks

​Advanced settings

​Editing and archiving

With the Evals Conductor

Manually

Success and forbidden checks

Advanced settings

Editing and archiving