"Promote the winner?".
It stops the test and copies the winning version into each page's draft. Visitors keep seeing the control until you approve a second card that publishes it, and one undo restores the pages and the split.
OPTIMIZE · EXPERIMENTATION
Experiments held to the standard of a data team.
Lock the plan before the first visitor arrives. Read the result four ways, on the same visitors, corrected when several versions or measures are judged together, with a guard that withholds the verdict when the split itself looks wrong. A person promotes the winner, and publishing it is a second approval. From a team that has spent decades building experimentation and personalization platforms.
Every figure on this page is as of October 2026.
This video is a members-only file — its access is set on the file itself, not on this page. Sign in or create an account to watch it. This video couldn’t be loaded. Try again.
What it does
Experiments sit in Optimization, beside Personalization and Audiences. A test moves through five steps: you set it up, plan it, lock it, read it and decide it. Everything below uses the screens' own words.
Set it up
Two versions or more, and the people who should see them.
- "New experiment". A name and a key; "Traffic variations" with relative weights, the first of them the control; a "Primary goal" that defines success.
- "Eligibility". Who may take part, as all of, any of or none of your audiences. A visitor outside it sees the live page and is not counted.
- "Arm override". Send one chosen audience to one chosen version.
- On the page itself. In the Composer, "Editing for" a variation renders that version of the real page, and you edit it in place. "+ New experiment" starts a draft test from the page you are on, with "Edit name & split" and "Promote" beside it.
- From a campaign. "+ New experiment for this campaign" aims a test at the people that campaign brought in.
- The list. "Search name, key, description, or goal"; a status filter; each test's "Arms", "Primary goal", "Exposed" and "Results"; and "Edit", "Clone", "Stop" and "Archive" on every row.
Plan it
How many visitors, and how long, before you start.
- "Power & duration planner". "Precommit a defensible fixed horizon from observed entry traffic and governed metric evidence." Its tabs are "Setup", "Estimate", "Provenance" and "Saved plans".
- Your terms. 80% or 90% power, with alpha fixed at 0.05; for each measure, the change worth detecting, its direction and its windows; guardrails that must not get worse; and "Run for", 14 days unless you choose otherwise.
- An answer in one sentence. "To detect {target} change in {metric} with 80% power, plan for {N} eligible visitors", with the expected duration at your own site's pace. The split it plans for is the one your weights really give, "not an equal-allocation shortcut".
- "Runtime tradeoff". The smallest effect you could detect if you ran for 7, 14, 28, 42, 56 or 84 days. "This sensitivity table does not create additional decision looks."
- One snapshot, never edited. Each estimate is "one immutable fixed-horizon planning snapshot". Recalculating makes a new one, and a plan can be saved, shared by link and exported.
- Day one. On a new site the planner says "an estimate appears after two days of traffic", and a plan made before then uses the fixed-horizon read at the end date you chose. An experiment does not need history; it builds its own.
Lock it
The rules of judgment, fixed before anyone is counted.
- "Review & lock plan", then "Lock this metric plan permanently?". What counts as success, what must not get worse, who takes part and how long it runs are sealed before any traffic is split.
- "Activate" opens the plan before it is locked. "Put this experiment LIVE?" is the last card, and your approval starts the test, which reaches visitors as the platform's servers and cached pages refresh.
Read it
Every number with its uncertainty, and a verdict in words.
- "Results explorer". "Look at part of the visitors, switch the method, and inspect each measure without changing the recorded decision."
- The verdict. "These results are final. You can promote the winner.", or that it "cannot name a winner yet". Before the planned length is over: "The experiment has not run its planned length yet, so these results can still change. Keep it running."
- The numbers. Each version's rate with its interval, the effect with its interval, and the adjusted p.
- "Compare methods on one exact cohort". "Shared facts appear once. The cards below contain inference only." The same visitors, read side by side.
- "Change how this is read". "Method, visitor window and date range. These controls never change the recorded decision." Every measure and version shows its comparison with the control, the adjustment for multiple comparisons and its outcome.
- "Metric explorer" and date windows, to look closer without deciding anything.
- "CUPED adjustment". A lookback of 7 to 90 days, "find covariates", the adjusted result beside the unadjusted one, and "Balance and coverage".
- "Compare results by attribute". Slices, for exploration only.
- "Timeline". "Days since exposure", to see whether an early effect fades.
- "Configuration & QA". The locked plan and any drift from it; "Rendered bindings", which bind a version to a page; and "QA preview", whose forced views are labelled and kept out of the results.
Decide it
"Stop this experiment?".
Ends a test at any time, behind its own card.
In a sentence
- The copilot works these screens through the same governed actions, 26 of them for experiments: "Set up an A/B test of the homepage headline", "How long must it run?", "Lock the plan", "Start it", "Promote the winner". Locking, starting, stopping and promoting each wait at an approval card.
How it works
A test, start to finish
-
Each visitor gets one version, and keeps it.
A fixed hash of the experiment and the visitor's id places each visitor against your weights. Nothing is stored, so a returning visitor lands in the same version every time, and tests pin the function so nobody is ever moved.
-
Eligibility comes first,
then the arm override, then the split.
-
The plan is sealed.
Locking hashes the plan and fixes the unit, which is the visitor; the split, the targeting and the page bindings; the method, a fixed horizon at alpha 0.05, two-sided; every measure with its direction, delay and window; the correction families; and the one moment the decision is read.
-
A conversion counts inside its window,
from first exposure plus the delay to the end of the window: by default no delay and seven days. A visitor counts only once their window has closed, and no decision is made while anyone is still maturing.
-
Results are taken nightly,
at 03:40 UTC, and each snapshot carries its own fingerprint on screen. Before the planned length a snapshot is descriptive only; the first mature one at or after it is the decision.
-
A person decides.
If exactly one version improved, it wins; if every version regressed, the control wins; otherwise there is no winner. Promotion and publication are two approvals.
The methods, named
The methods, named.
| What is judged | How | Values |
|---|---|---|
| The primary measure | A two-sided pooled z-test for conversion goals; a delta-method z-test for averages and ratios; Holm's step-down correction across every version against the control | alpha 0.05 |
| Secondary measures | One Benjamini-Hochberg false-discovery family over every secondary measure and every version | q 0.10 |
| Guardrails | New plans check each limit on its own, one-sided, and pass a version only when harm beyond the limit is ruled out. Plans locked earlier keep a one-sided test of "worse than your limit", then Benjamini-Hochberg | alpha 0.05 (earlier plans q 0.10) |
| Intervals | Wilson for each version; Newcombe for the difference between two rates | 95% |
| Data floors | No comparison below 100 mature visitors per version, or 25 converting visitors per version on a conversion goal | 100 and 25 |
| The split | A chi-squared test of exposed visitors against your weights | alarm at p below 0.001, once every expected count is at least 5 |
Four lenses on the same visitors
One test, read four ways.
Switching the lens changes how you read the result, never the result. The screen says so: "View only · traffic unchanged".
"Fixed-horizon frequentist".
The default, and the recorded decision: one look, at the length you planned.
"Bayesian beta-binomial".
The chance each version is best and the expected loss of choosing it, from a uniform prior: exact for two versions, from 524,288 seeded draws for three or more. To be recorded, a winner needs at least a 95% chance to be best, an expected loss under the "Threshold of caring" you set, 1,500 mature visitors and 25 conversions per version, seven days, and the planned length.
"Group sequential · O'Brien-Fleming".
Four planned looks, at 25, 50, 75 and 100% of the planned visitors, each against a boundary that is hardest to cross early: z of 4.048, then 2.862, 2.337 and 2.024. For two-version tests planned with an estimate.
"CUPED-adjusted frequentist".
Each visitor's own behaviour in the 30 days before they first saw the test, used to take out noise the test did not cause. The adjusted result sits beside the unadjusted one, with its "Precision change".
Guards on every result
Built so a test cannot flatter you.
A guard on the split.
If visitors stop arriving in the proportions you set, the verdict is withheld, promotion is refused, and the screen says "Results are not trustworthy while this persists."
No verdict on thin data.
The floors above apply to every comparison, on every lens.
No early decision by accident.
You may look whenever you like. Only a planned look can authorize a promotion: the one decision look at the planned length, or, for a two-version test on the sequential lens, one of its four planned looks.
Outliers capped blind.
A measure's cap is computed over all versions pooled, without looking at which visitor saw which.
No quiet changes.
If the settings, the split, the targeting or the page bindings drift from the locked plan, promotion is refused.
Guardrails fail closed.
A guardrail that cannot be computed blocks the decision. A secondary measure that cannot be computed is left out of its correction and named on screen.
Explore without fooling yourself
- Slices by attribute are for discovery. They are kept out of every correction family, and the product says why: "Attribute slices help discover possibilities; they are not shipping decisions."
- Date windows describe. They never decide.
- "Days since exposure" groups results in 24-hour steps from each visitor's first view, so a novelty effect that fades is plain to see before you act on it.
Promote, then publish
Two approvals, on purpose.
- Promotion checks, inside a single write, that the plan and the snapshot match their fingerprints, that the split is clean, that nothing has drifted, that no visitor is still maturing, and that the chosen method's own conditions hold. Then it stops the test and places the winner in each page's draft. Visitors see the winner only after you approve the publish, and one undo puts the pages and the split back.
Aim it
Test on the people who matter.
- An eligibility rule narrows a test to the audiences you choose; "+ New experiment for this campaign" narrows it to a campaign's arrivals.
- The campaign's tag is recorded on arrival, and every page a visitor opens after it can carry the campaign's test.
Holdouts, measured the same way
The same arithmetic proves personalization.
- Each piece of targeted content can hold back a share of its audience on the default page, 10% unless you change it, and its lift is read with Wilson intervals, the Newcombe difference, a z-test, a split check and the same floors of 100 and 25.
- Across the whole site, a standing platform holdout keeps 5% of identified, consented visitors on the site's own defaults, so the value of personalization as a whole can be measured. Experiments run beside it, and it never suppresses them.
What keeps it honest
Consent first.
Only visitors who allowed personalization, which includes A/B tests, and who carry a visitor id take part. A browser that sends Global Privacy Control, a visitor in an opt-in country who has not chosen, and a visit whose location is unknown are never placed in a test.
A person at every step that matters.
Locking the plan, going live, stopping, promoting and publishing each stop at their own approval card. An edit to a live experiment's version waits for your approval too.
The platform enforces the pause.
The approval is checked by the platform, not by an instruction to the AI, whether the act starts on a screen or in the copilot.
A sealed plan.
Once locked, the plan cannot be edited, and any drift from it blocks promotion.
Forced views never count.
QA previews are labelled and excluded, and author and migration previews are never counted at all.
One undo.
A promotion comes back in one step: the pages and the split together.
Common questions
Can I look at results before the test ends?
Yes, as often as you like. Looking is free; deciding is not. Until the planned length, every view is descriptive. For a two-version test planned with an estimate, the sequential lens adds four planned looks at which the test may stop early, and if the plan has guardrails, only the last of them can stop it.
Do I need traffic history to start?
No. An experiment records its own history from the first visitor, and the planner's estimate appears after two days of traffic. A test started without an estimate is decided by the fixed-horizon read at the end date you chose: 14 days unless you set another, up to 365.
Will Praxis pick the winner and ship it?
No. A person promotes it, the winner lands in your page drafts, and a second approval publishes it.
Which reading decides?
The plan names its one deciding reading when it is locked: the fixed-horizon read, unless you choose another. Only that reading can promote a winner; the others are for information. Changing the lens on screen never changes what is recorded.
Which confidence level does Praxis use?
Every interval is 95% and alpha is 0.05, so every test on Praxis is read on the same terms.
How many versions can a test have?
Two or more, each with a relative weight. The sequential lens is for two-version tests; the Bayesian lens handles three or more by simulation.
Who is counted?
Visitors who have allowed personalization, which on Praxis includes A/B tests, and who have a visitor id. Everyone else sees the live page and is not counted.
Does a test change only the page I bind it to?
Yes: a version is bound to its page, and only that page changes. A visitor is counted as exposed only on the page under test.
Can I test only on the people a campaign brought?
Yes. "+ New experiment for this campaign" aims a test at them: the campaign's tag is recorded on arrival, and every page they open after it can carry the test.
Do I have to take the whole platform?
No. Experimentation is a feature you turn on for a site: its line joins your bill from that moment, and turning it off removes it. Coming soon.