Experiments

An experiment runs one of your Branches alongside your Live version and splits real customer calls between them. You compare how the two perform, then keep the one that did better. Use one for any change where you want evidence before every customer gets it, such as a new prompt, a reworked flow, a different routing rule or a model swap.

​

How it works

An experiment always compares two versions. Variant here only means the Branch under test. It has nothing to do with variants, which set per-site behaviour. You choose what share of calls the variant gets, anywhere from 1% to 99%, and the control gets the rest. The default is an even split. Each call is assigned to one version when it starts and stays on that version until it ends. Every call during the experiment is tagged with the experiment and with the version that handled it. That is what lets you filter your dashboards by experiment and compare the two versions side by side. You can keep updating both versions while the experiment runs. When you have enough data, you end the experiment and pick a winner, which then takes every call.

​

Before you start

You need:

• Something published to Live. This becomes the control.
• A Branch with the change you want to test, synced with Live. See Making a change. A Sub-branch can’t be tested on its own, so merge it into its Branch first.
• No other experiment running on the project.
• The Deployment permission. See Access control.

Run simulation tests on the Branch first to check the change works. The experiment then shows how it performs with real customers.

​

Start an experiment

​

While an experiment is running

The experiment appears on the Experiments tab of the Deployments page, marked Running, with its traffic split. If you talk to your Live agent from the chat or call panel in Agent Studio, either version may answer. You can keep improving both versions without stopping the experiment. To rename the experiment or change the split, open its menu on the Experiments tab and choose Update. Changing the split part way through makes the two versions harder to compare, so only do it when you need to. A few things wait until the experiment ends:

• Publishing or archiving the Branch under test.
• Starting another experiment.
• Rolling back Live.

​

Track performance

Because every call is tagged with the experiment and its version, you can compare the two in your dashboards or by asking Wren.

​

Dashboards

Once experiments are turned on for your project, every dashboard on the Analytics page has an Experiment filter and an Experiment version filter. See Self-serve dashboards. Past experiments stay in the Experiment list after they end, so you can go back to their results at any time.

​

Wren

Wren knows about your experiments, running and finished. Ask it how the versions compare and it works out each metric for both, side by side.

• “How is the refund flow experiment doing on containment this week?”
• “Compare call length between the two versions of last month’s experiment.”

Wren gives you the figures for each version but leaves the decision to you, because a small difference can be down to chance. You can also ask Wren to build a dashboard for an experiment, which opens with that experiment already selected. See Analyze conversations. Wren can start, re-split or end an experiment for you too. It always shows you exactly what it will do and waits for your approval first.

​

End an experiment

The winner takes every call from then on. If the winning Branch can’t merge because of conflicts, the experiment keeps running. Sync the Branch with Live, resolve the conflicts, then end the experiment again.

​

Past experiments

The Experiments tab lists every experiment with its traffic split. Finished experiments also show whether Control won or Variant won. Their data stays in your dashboards, and Wren can still answer questions about them. A/B tests from before experiments were turned on are listed too, but without a result.

​

Limits

• One experiment per project at a time.
• Two versions per experiment, your Live version and one Branch.
• Sub-branches can’t be tested. Merge them into their Branch first.
• Agent Studio doesn’t tell you whether a difference between the versions is big enough to trust. Your dashboards and Wren give you the figures, and you decide when there is enough data to call a winner.
• Charts grouped by deployment, and tables with a row grouping, don’t show while you compare versions. For a grouped table, remove the grouping or pick one version under Experiment version.

​

Related pages