Create and Run an Experiment

This guide walks through a two-variant A/B test from hypothesis to manual shutdown.

Before you launch: Experiments route eligible new web chat, email, and SMS conversations. Start experiment takes effect immediately; there is no scheduled start. Applied does not automatically evaluate guardrails, stop a test, or deploy a winner.

Before you start

Prepare these items outside the setup flow:

• One hypothesis: “Changing X will improve Y because Z.”
• Two saved versions: one control and one treatment with a single deliberate difference.
• One primary success metric: a conversation event that is already being emitted and visible in analytics.
• At least one guardrail metric: an emitted event that represents a harmful outcome, such as escalation. Guardrails are required at launch even though monitoring and intervention remain manual.
• An observation plan: owner, planned duration, minimum sample, and reasons to stop early.
• Guardrail thresholds and an owner: define what regression should cause a manual stop and who will monitor it.

If you still need versions, follow Create Agent Versions.

Fastest path: use Applied Assistant

Applied Assistant can inspect the agent’s live version, discover available metric events, create the treatment version, validate the complete experiment, and prepare the Draft for review.

Use a request that names the change, outcome, harm metric, allocation, and audience:

Create a Draft A/B test for Support. Keep the current version as a 50% control. In the 50% treatment, ask for the order number in the first reply and preserve every other behavior. Optimize successful resolution, use escalation rate as the guardrail, and include all eligible contacts. Prepare it for review; do not start traffic.

The Assistant will ask a follow-up only when it cannot identify the treatment, outcome, target agent, or audience from the request and workspace. In review mode, the resulting change is staged for approval. Creating a Draft never grants permission to start it.

Before approving, check the single treatment difference, discovered event names, contact-based attribution, traffic rule, and rollout. If the requested outcome is not instrumented, add that event before creating the experiment; do not substitute a different metric silently.

CLI workflow

The same workflow is available through Applied CLI. The server owns the canonical versioned schema and performs authoritative validation:

# Inspect the current schema and agent versions.
applied experiments schema -f json
applied agents <agent_id> revisions list -f json
applied agents <agent_id> revisions get <live_revision_id> -f json
 
# Validate the exact configuration file, then create one atomic Draft.
applied experiments validate --body-file experiment.json -f json
applied experiments create --body-file experiment.json -f json
 
# Launch only after an explicit decision and a passing server preflight.
applied experiments <experiment_id> preflight -f json
applied experiments <experiment_id> start -f json
 
# Check assignment and exposure integrity before interpreting results.
applied experiments <experiment_id> diagnostics -f json

An experiment configuration can reference an existing version by revision_id or include a complete inline version data object. Inline data is saved as a new version during atomic creation. Always copy the full live version and make one surgical treatment change; a partial object can remove unrelated behavior.

1. Create a Draft experiment

1. Open the agent.
2. Go to Deploy -> Experiments.
3. Select New experiment.

The guided setup has four stages.

Hypothesis

Use a name that describes the decision, not the expected outcome. Write a falsifiable hypothesis with one change and one primary outcome.

Example:

Asking for an order ID in the first reply will increase automated resolution because the agent can retrieve order context earlier.

Variants

Add at least two variants. For each variant:

1. Give it a short Key such as control or treatment.
2. Select a saved agent version.
3. Mark exactly one variant as Control.
4. Set its allocation.

Use a different saved version for each variant. Allocations must add up to 100%. A 50/50 allocation is the clearest starting point for a two-variant test; use a different split only when you have a documented reason.

Success metric

Choose one executable primary metric. The metric definition must reference conversation events that are already emitted in production.

For a conversion metric:

• the numerator event represents success; and
• the denominator event represents eligibility for that success.

Do not create a placeholder metric and plan to define it later. Confirm the event names in analytics before selecting the metric.

Experiment metrics use Contact attribution. This keeps one contact on one variant across eligible interactions and prevents a returning contact from switching arms.

Review

Confirm the hypothesis, distinct versions, control, allocation, and primary metric. Then select Save draft.

A Draft stores the experiment definition. It does not route traffic.

Guardrail metric

After saving the Draft, open its metrics and add at least one metric with the Guardrail role. Like the primary metric, it must have an executable event definition and use Contact attribution. Record the threshold, minimum exposure, and intended breach action for the human operator.

Applied stores these settings and checks that the guardrail is executable before launch. It does not yet evaluate the threshold or stop traffic automatically.

2. Create the traffic rule

From the experiment’s next step, open Traffic rules, or go to Deploy -> Traffic rules and select New rule.

1. Give the rule a name that states its audience.
2. Set Target to Experiment and select the Draft experiment.
3. Add the eligibility conditions.
4. Set the rollout for each condition set.
5. Save the rule as Active.
6. Move it to the intended priority in the ordered rule list.

An active rule that targets a Draft experiment does not route traffic until the experiment starts.

How rule matching works

Traffic rules run from top to bottom. The first rule whose conditions match and whose rollout gate passes wins.

• Conditions inside one condition set use AND.
• Multiple condition sets use OR.
• If a rollout gate fails, evaluation continues to the next condition set or rule.
• If no rule wins, traffic uses the default live version.

Place narrow rules above broad rules. A broad rule near the top can prevent the experiment rule from ever being evaluated.

Rollout versus variant allocation

These percentages answer different questions:

With a 20% rollout and 50/50 allocation, each variant receives about 10% of the matching traffic. Contacts that miss the rollout continue through the remaining rules.

3. Review the launch checklist

Return to the experiment and confirm:

• the experiment is still Draft;
• it has at least two variants using distinct saved versions;
• exactly one variant is the control;
• variant allocations total 100%;
• exactly one executable primary metric is selected;
• at least one executable guardrail metric is selected;
• every experiment metric uses Contact attribution;
• an active traffic rule targets the experiment;
• the traffic rule is in the intended order; and
• the default live version is a safe fallback.

Also verify that the planned audience uses supported web chat, email, or SMS traffic. Voice and social-channel routing are outside the current experiment path.

4. Start immediately

Select Start experiment only when you are ready for eligible production traffic. Starting changes the experiment to Running and makes the active traffic rule effective immediately for supported new web chat, email, and SMS conversations.

There is no scheduled start. Do not select Start experiment as a way to queue a future launch.

5. Verify routing with new traffic

After launch:

1. Run a new, eligible interaction on a supported channel.
2. Confirm the traffic rule still appears active and in the intended order.
3. Run experiment diagnostics and confirm both variants begin receiving assignments in roughly the intended allocation.
4. Confirm exposures appear for contacts whose live agent provider started.
5. Confirm the primary event appears in the underlying analytics.

Applied persists one sticky assignment per canonical contact and records an exposure when that assigned revision starts a live agent provider. Diagnostics exclude contaminated assignments and flag unaccounted variants and strong sample-ratio mismatches. They verify routing integrity; they do not measure the primary outcome or declare a winner.

6. Monitor the planned window

Check diagnostics before using the Legacy estimate for directional monitoring. Investigate contamination, exposure loss, unaccounted assignments, or a sample-ratio mismatch before comparing outcomes. Do not stop because an early lift or interval looks favorable. Follow the observation and sample plan you recorded before launch.

Guardrail configuration is required and stored but not automatically evaluated. Monitor quality, escalation, latency, and other operational guardrails in their source analytics and stop manually if a predefined safety condition is met.

See Read Legacy Estimates for the current analysis limitations.

7. Stop and deploy manually

There is no automatic winner promotion or rollback. Use this shutdown sequence:

1. Record the observation window and final directional readout.
2. Decide which saved version should become the default using your reviewed analysis and operational guardrails.
3. Open Deploy -> Versions and deploy that version.
4. Open Deploy -> Traffic rules and archive or update the experiment rule.
5. Return to the experiment and select Stop experiment.

Stopping completes the experiment. It does not change the live version or traffic rules for you.

Troubleshooting

No contacts appear

• Confirm the experiment is Running.
• Confirm the rule is Active and targets this experiment.
• Check whether a higher-priority rule matches first.
• Check the condition sets and rollout percentage.
• Generate a new, eligible web chat, email, or SMS conversation.

Contacts appear, but the success metric is empty

• Confirm the numerator and denominator events are emitted in production.
• Confirm the metric was fully defined before it was attached.
• Reproduce a successful interaction and check the underlying analytics first.

The split differs from the allocation

Run applied experiments <experiment_id> diagnostics -f json. Small samples vary naturally, but unaccounted > 0, valid: false, or mismatch: true require investigation before interpreting outcomes. Also confirm rule order, targeting, rollout, active variants, and weights.