Read Legacy Estimates

The experiment results page is labeled Legacy estimate. It is useful for checking whether traffic and metrics are moving, but it is not a decision-grade statistical analysis.

Do not ship a variant because the page shows favorable lift or an interval that excludes zero. Check assignment and exposure diagnostics first, then confirm the metric definition, observation window, and guardrails in a separately reviewed analysis before making a high-impact deployment decision.

What the page shows

For each variant and metric, the page may show:

• attributed contacts or assignments;
• the observed metric value;
• lift relative to the control; and
• an approximate interval for supported conversion metrics.

These outcome values are computed from analytics grouped by experiment and variant identifiers. Applied separately persists the contact assignment and live-agent exposure ledger used by experiment diagnostics; the Legacy estimate is not yet a canonical treatment-effect calculation over that ledger.

Terms

Avoid calling attributed contacts “exposures.” Assignment, exposure, and outcome attribution are different stages. An exposure proves the assigned revision started a live agent provider; it does not prove a successful outcome.

Known limitations

A safe reading order

1. Confirm routing integrity

Before reading lift, confirm:

• the intended rule was active for the whole observation window;
• no higher-priority rule intercepted the audience;
• rule rollout and variant allocation match the launch plan; and
• only supported web chat, email, or SMS traffic entered the experiment.

Run:

applied experiments <experiment_id> diagnostics -f json

The assignment sample counts clean canonical contacts. The exposure sample counts clean assigned contacts with at least one live provider instantiation. Contaminated assignments are shown separately and excluded from both samples. Investigate unaccounted > 0, valid: false, or mismatch: true before reading outcomes. Sample-ratio mismatch uses an alpha of 0.001; no mismatch at a small sample size is not proof that routing is correct. Report exposure loss as the observed count and rate rather than inventing an acceptable threshold.

2. Confirm the metric definition

Check that the primary metric’s numerator and denominator events were emitted throughout the observation window. Compare the trend with the source analytics. An empty or discontinuous event stream is an instrumentation problem, not a zero outcome.

3. Check the observation plan

Use the duration and sample requirement chosen before launch. Do not repeatedly stop and restart your decision process when the displayed estimate crosses a convenient threshold.

4. Review operational guardrails manually

Check quality, escalation, customer satisfaction, latency, cost, and any other predefined safety metrics in their source analytics. The experiment page does not evaluate or enforce them.

5. Treat lift as directional

Ask whether the direction is stable, practically meaningful, and consistent with the underlying data. For a consequential decision, use a separately reviewed analysis based on verified assignment and outcome data.

What not to infer

The Legacy estimate does not prove that:

• every counted contact saw the assigned version;
• the control and treatment were comparable for the full window;
• a displayed interval has the coverage implied by a governed experiment analysis;
• a favorable variant is safe on unmonitored guardrails; or
• the system will deploy the apparent winner.

Finish the experiment

The workflow is manual:

1. Record the exact observation window and directional estimates.
2. Review routing integrity, source metrics, and guardrails.
3. Choose the version through your team’s normal approval process.
4. Deploy that version from Deploy -> Versions.
5. Archive or update the experiment rule under Deploy -> Traffic rules.
6. Select Stop experiment on the experiment.

For the complete launch and shutdown sequence, see Create and Run an Experiment.