Looks, sealed in advance.
The default decision is read once, at the moment the locked plan names; the sequential lens looks only at its four planned points.
OPTIMIZE · EXPERIMENTATION · THE STATISTICS
The statistics, in full.
Every method, threshold and rule Praxis uses to read an experiment, as the product holds them in October 2026. Most of it is fixed by the platform for every test, so every test is read on the same terms. Where a choice is yours, this page says so.
The words on screen
Each label below appears on an experiment's Results tab. The meaning beside it is the definition the product itself carries.
| On screen | What it means |
|---|---|
| "Adjusted p" | The probability of evidence this extreme if there were no effect, corrected for the number of comparisons in its family |
| "Confidence interval" | An interval built by a method that captures the true value in 95% of repeated samples, for one version or for the effect |
| "Decision-grade" | The one retained, mature snapshot at the planned horizon |
| "Chance better" | The posterior probability that a version beats the control in the measure's declared direction |
| "Credible interval" | The interval that holds 95% of the posterior probability of an estimate |
| "Expected loss" | The average cost, under the posterior, of choosing a version instead of the best one |
| "Threshold of caring" | The largest expected loss you accept, adopted explicitly, in the measure's own unit |
| "Information progress" | Mature visitors on the primary measure, divided by the planned exposure |
| "Scheduled look" | One of four planned chances to test the O'Brien-Fleming boundary |
| "Direction-adjusted z" | The standardised effect, signed so that a positive value favours the measure's declared direction |
| "Pre-exposure window" | The time before each visitor's first exposure that CUPED draws on |
| "Pooled adjustment" | One set of coefficients estimated across all versions, without knowing which is which |
| "Precision change" | How much the adjustment changed the variance of the comparison; it can rise as well as fall |
| "No prior history" | A separate column that tells a visitor with no history apart from one whose history is zero |
How it works
Assignment
- The hash. MurmurHash3 (x86, 32-bit) over the text "{experiment key}:{visitor id}", seed 1, divided by 2^32 and laid against the cumulative normalised weights. Nothing is stored, and tests pin the function, so no visitor is ever re-bucketed.
- The order. Eligibility first, where an ineligible visitor sees the live page and is not counted; then the "Arm override"; then the hash.
- Who takes part. Visitors with personalization consent and a visitor id. Global Privacy Control, an opt-in region before the visitor chooses, and an unknown location are never bucketed.
- Exposure. A visitor is counted as exposed only on the page under test.
- Timing. A newly activated experiment reaches visitors as the platform's containers and edge cache refresh. Results are snapshotted nightly at 03:40 UTC.
The locked plan
- Locking is permanent, behind "Lock this metric plan permanently?", and the plan is hashed with SHA-256.
- It fixes: the unit, which is the visitor; the split, the targeting and the page bindings; the engine, a fixed horizon at alpha 0.05, two-tailed; the primary, secondary and guardrail measures, each with its direction, delay and attribution window; the correction families; the stopping rule, a planned number of visitors or a planned end date, with exactly one decision look; the planning evidence; and, when the planner produced an estimate, an O'Brien-Fleming schedule of four looks.
When a conversion counts
- A conversion counts between first exposure plus the delay, and first exposure plus the delay plus the window. The planner's defaults are a delay of 0 and a window of 7 days.
- A visitor counts only once their window has closed, and every decision requires zero visitors still maturing.
- A measure may carry an outlier cap. It is computed over all versions pooled, without looking at which version is which.
The default read: a fixed horizon, corrected
| Part | Method | Values |
|---|---|---|
| Primary measure | Two-sided pooled two-proportion z-test for conversion goals; delta-method z-test for averages and ratios; Holm step-down across every version against the control | alpha 0.05 |
| Secondary measures | One Benjamini-Hochberg false-discovery family over every secondary measure and every version | q 0.10 |
| Guardrails | New plans check each limit on its own, one-sided, and pass a version only when harm beyond the limit is ruled out. Plans locked earlier keep a one-sided test of "worse than your limit", then Benjamini-Hochberg | alpha 0.05 (earlier plans q 0.10) |
| Data floors | No contrast below 100 mature visitors per version, or 25 converting visitors per version on a conversion goal, or while any visitor is still maturing | 100 and 25 |
| Intervals | Wilson score interval per version; Newcombe interval for the difference in rates; a normal interval for delta-method measures | 95% |
| Peeking | Before the planned length a snapshot is descriptive only ("Interim look: the precommitted fixed horizon has not been reached."); the first mature snapshot at or after it is the single decision look | one look |
| The winner | Exactly one version "improved": it wins. Every version regressed: the control wins. Otherwise, no winner |
- When a measure cannot be computed. A secondary measure is left out of its family's count and named on screen. A guardrail fails closed.
Averages and ratios
- Averages use SE = √(s² / n).
- Ratios use the delta method: Var(R) = (Var N + R² × Var D - 2R × Cov(N, D)) / (n × mean(D)²). A ratio is refused when n is below 2 or its denominator is too close to zero.
- These are normal approximations, guarded by the data floors.
The split guard
- A chi-squared goodness-of-fit test of exposed visitors per version against the configured weights, with one degree of freedom fewer than the number of versions.
- It alarms at p below 0.001, and only when every expected count is at least 5.
- While it alarms, the verdict is suppressed, the promotion gate refuses, and the screen says "Results are not trustworthy while this persists."
Power and duration
- Inputs. Complete days of the site's own history; 80% or 90% power; alpha fixed at 0.05; for each measure, a target effect, absolute or relative, with its direction and windows; and the draft's real weights.
- The formula. Total visitors N = (z for alpha + z for power)² × (σ² of the control / the control's share + σ² of the treatment / the treatment's share) / δ², taking the largest result over the treatments and over the measures that drive the length. The z values come from Acklam's inverse-normal approximation with one Halley refinement.
- Planning alpha. Split Bonferroni-style within each family: the primary's 0.05 over the treatments, two-sided; the secondary measures' 0.10 over their comparisons, two-sided. That is more cautious than the Holm and Benjamini-Hochberg corrections used to read the result. Each guardrail is planned on its own at 0.05, one-sided, and drives the length.
- Duration. The visitors needed, divided by the site's daily mean, with a shorter estimate from its busier days (the 75th percentile) and a longer one from its quieter days (the 25th), plus settling days for the attribution window. The smallest detectable effect is also shown at 7, 14, 28, 42, 56 and 84 days.
- A brand-new site. The experiment builds its own history, and an estimate appears after two days of traffic. The test then runs for the days you choose, 14 by default and at most 365, and the fixed-horizon read at that end date decides it.
- One snapshot per estimate. Each estimate is "one immutable fixed-horizon planning snapshot"; recalculating creates another and never edits it.
- An illustration, our own arithmetic on the formula and not a product output: a 5% baseline, a 10% relative lift to 5.5%, 80% power and a 50/50 split need about 62,000 visitors in total, about 31,000 per version.
The four lenses
The "Analysis lens" switches how the same visitors are read. It never changes the recorded decision, and the screen says so: "View only · traffic unchanged".
| Lens | Method | Conditions to be recorded and promoted |
|---|---|---|
| "Fixed-horizon frequentist" | The default read above, at one precommitted look. The product recommends it "for the experiment's retained terminal decision and for mean or ratio metrics that beta-binomial cannot model." | The default decision |
| "Group sequential · O'Brien-Fleming" | Four fixed looks at 25, 50, 75 and 100% of the planned visitors; boundary z = 2.024 / √t, which gives 4.048, 2.862, 2.337 and 2.024; an unpooled z for conversions; the first eligible crossing decides | A two-version test whose planner produced an estimate. Each look needs 100 mature visitors and 25 conversions per version and a clean split. With guardrails, looks 1 to 3 cannot stop the test |
| "Bayesian beta-binomial" | A Beta(1, 1) prior per version; posterior Beta(conversions + 1, visitors − conversions + 1); for two versions the exact chance to win (Evan Miller's closed form) and the expected loss; for three or more, 524,288 seeded Monte Carlo draws | At least a 95% chance to be best; expected loss under the "Threshold of caring" you adopt, which has no default; at least 1,500 mature visitors and 25 conversions per version; at least 7 days; the planned length reached; a clean split; and every guardrail at most a 5% chance of being worse than its limit |
| "CUPED-adjusted frequentist" | Each visitor's behaviour in the 30 days before first exposure (7 to 90 days, the exposure day excluded); up to 10 covariates from the plan's own measures, standardised; θ = (XᵀX + λI)⁻¹ Xᵀ(y − ȳ), with λ = 10⁻⁶ × n; fitted across all versions, blind to which is which; the adjusted result beside the unadjusted, with "Precision change" | Only the governed profile can be recorded and promoted: 30 days, the first ten eligible covariates, at the planned length. Any other setting reads "Analysis only" |
Exploration that never decides
- Attribute slices are excluded from every correction family: "Attribute slices help discover possibilities; they are not shipping decisions."
- Date windows are descriptive.
- "Days since exposure" shows results in 24-hour buckets from each visitor's first exposure, to inspect novelty.
What promotion requires
-
The checks, inside one write.
A plan and a snapshot that match their hashes; the canonical analysis; no sample-ratio alert; no drift in the settings, the split, the targeting or the bindings; no visitor still maturing; the chosen method's own check; and a decision reference recomputed inside the same write.
-
Promotion.
The test stops and the winner is copied into each bound page's draft.
-
Publication.
A separate approval. One undo restores the pages and the split.
Holdbacks and the holdout
- A holdback belongs to one piece of targeted content, 10% of its audience by default. Its lift, targeted against held back on a goal, is read with Wilson intervals, the Newcombe difference, a z-test, a sample-ratio check, and floors of 100 exposed and 25 converting.
- The platform holdout is a salted MurmurHash3 of the visitor id, with nothing stored. 5% of identified, consented visitors fall in the "default" arm and see the site's own defaults, which measures what personalization adds as a whole. The rest are personalized. An unidentified visitor cannot be held out, and experiments are never suppressed by the holdout.
What keeps it honest
Blind where it matters.
Outlier caps and CUPED coefficients are computed across all versions without knowing which is which.
Fingerprints, checked.
The plan and the snapshot are hashed, and promotion recomputes its decision reference inside the same write.
Failing closed.
A guardrail that cannot be computed blocks; a split that looks wrong withholds the verdict.
Exploration, labelled.
Slices, date windows and days since exposure never enter a decision.
No decision without a person.
Nothing is promoted, shifted or published automatically.
Common questions
Why Holm for the primary measure and Benjamini-Hochberg for the secondary measures?
Holm controls the family-wise error rate across the versions on the measure you will ship on; the screen labels it "holm-fwer". Benjamini-Hochberg bounds the false discovery rate within the secondary measures, and in plans locked earlier separately within the guardrails, labelled "bh-fdr". New plans check each guardrail on its own, to rule out harm beyond its limit.
Why plan with Bonferroni and read with Holm?
Bonferroni is the more cautious of the two, so the plan errs toward enough visitors, and the read is never weaker than the plan assumed.
Why a pooled z-test, but a Newcombe interval?
The test of "no difference" pools the versions, as that hypothesis assumes. The interval for the size of the difference cannot assume it, so it uses Newcombe's interval for a difference of proportions, built on the Wilson intervals of the two rates.
How are averages and ratios tested?
With normal approximations: a z-test for averages, the delta method for ratios, and no comparison below 100 mature visitors per version.
What prior does the Bayesian lens use?
Beta(1, 1), uniform over every conversion rate. The "Threshold of caring" is yours to adopt, and it has no default.
Why fixed O'Brien-Fleming values?
The four looks are planned at fixed shares of the planned visitors, each with a fixed boundary, so looks happen only at those four points.
Can CUPED change a recorded decision?
Only in its governed profile: 30 days, the first ten eligible covariates, at the planned length. Every other setting is "Analysis only".
What happens when a guardrail cannot be computed?
The decision fails closed.
What is the unit of analysis?
The visitor: one visitor id, which is one browser. A person who uses two devices can be counted as two visitors.
Where can I see the plan I locked?
On the test's "Configuration & QA" tab, with any drift from it; the planner keeps its "Provenance" and "Saved plans".