| Every event value counted in a revenue, duration or custom metric |
A value that is not a number, is infinite, or is larger than 10^12 in size, in the unit it was sent in, adds nothing to its user's total; the event still counts its user in conversions. A count metric counts events and does not read their value. |
| An event sent twice counted once |
Only an event with an event_id is recognised when it is sent again. Without one it is stored and counted twice: a count metric counts it again, and a revenue, duration or custom metric adds its value again. A request that timed out may have been stored, so sending it again without an event_id can double it. A conversion rate counts each user once and is not affected. See How events count. |
| A revenue, count, duration or custom metric's sequential analysis, Bayesian analysis, sample-size plan, daily series and Live tab by its values |
Each is of the share of users who sent the metric's event, as for a conversion. CUPED does not analyse these metrics. |
| A breakdown by segment of a revenue, count, duration or custom metric by its values |
The Breakdowns view of such a metric shows, in its column headed Mean, the share of users who sent the metric's event in each segment. The metric's headline row is its mean per user. |
| An experiment's results for the members of a rules segment |
Not offered: GET /api/v1/results/{experiment_id}/segments/{segment_id} answers 422 for a rules segment, because the attributes its rules match are not stored for every assigned user. An ID-list segment's members can be read. |
| A saved segment applied to the other results views or the export |
Only GET /api/v1/results/{experiment_id}/segments/{segment_id} filters by a saved segment. The other results routes, GET /api/v1/experiments/{experiment_id}/results and the export answer for all users, and ignore a segment_id parameter. |
| A segment's results counted by membership when each user was assigned, and corrected for the segments you look at |
A segment's results count the list as it is when they are computed: users added since are counted, users removed are not. The correction covers the comparisons inside the segment, not how many segments you look at, so the answer is exploratory. |
| The average time one event takes, such as page load time |
A duration metric is each user's total of its events' values, 0 for a user with none, averaged over every assigned user: a variant whose users send the event less often reads as less time. No metric averages over events rather than users. |
| Outlier control on a revenue, duration or custom metric (winsorization, trimming) |
None is applied. A revenue, duration or custom metric is each user's total, averaged; one very large user moves the mean and widens the interval. variance_reduction_config.method = winsorization is stored and clips nothing. Only a value that is not a number or exceeds 10^12 in size is dropped, as the first row of this section says. |
| A ratio metric (revenue per order, clicks per session) |
There is no ratio metric type. Metric types are conversion, revenue, count, duration and custom, each averaged over assigned users: see the row on the average time one event takes. Revenue per order can only be computed outside the platform. |
| A sample-ratio mismatch that stops the experiment or removes its winner |
Under a mismatch, recommendation is INCONCLUSIVE in GET /api/v1/results/{experiment_id}, in the data export and in the report. winner_variant, has_winner, the Bayesian decision and the sequential panel are unchanged. |
| A variant that is no better recommended to ship no more often than α says, whatever the split |
On zero-inflated, right-skewed revenue (most users spend nothing, a few spend a lot), measured in 2,000 experiments at each split, at α = 0.05, with 5% of users buying, order values lognormal and repeat purchases. When both variants have the same distribution (an A/A test) it holds: with 600 users in the control and 5,400 in the treatment, the treatment was recommended to ship in 48 and the control kept in 52, about 50 expected in each direction (before engine 1.13.0, which compared means with Welch's t-test, 150 and 12). A variant that changes the shape of the values without changing the mean is not covered: with 5,400 users in the control and 600 in the treatment, 12.5% of the treatment's users buying and each of their orders 0.4 times the size, 71 of 2,000 such experiments recommended shipping it, where about 50 are expected. The interval on the difference is Welch's and is unchanged: in the same A/A experiments at 600/5,400 it excluded 0 in 162 of 2,000, where about 100 are expected. |
| Intervals on a treatment's difference adjusted for the experiment's correction |
The intervals on a treatment's difference from the control, on its relative change and on its impact are not adjusted for comparing several treatments with the control, while is_significant is (Benjamini-Hochberg by default). With two or more treatments an interval can exclude 0 on a treatment that is not significant. For a metric analysed as a share of users, the interval and Fisher's exact test can also disagree near the threshold, either way. For a metric analysed as a mean, the interval is Welch's while is_significant follows the permutation test, so the two can disagree either way, most often at a lopsided split: in 2,000 A/A experiments with 600 users in the control and 5,400 in the treatment they disagreed in 142 (7.1%). |
| A mean metric tested by the permutation test everywhere |
A warehouse analysis has only each variant's sums, so a mean metric there keeps Welch's t-test (welch_t_test), which on zero-inflated, right-skewed values at a lopsided split names a variant that is no better more often than α says: 150 of 2,000 A/A experiments with 600 users in the control and 5,400 in the treatment recommended shipping it. An LLM experiment compares its variants with its own Welch's t-test. |
| An interval on the relative change for every treatment |
For a proportion, relative_improvement_pct_ci is null only when an arm has no users or the control has no conversions; at low counts it is wide: a treatment with 6 converting users out of 200 against a control with 2 out of 200 gives -34.2% to +1,206.3% at the 0.95 level. For a mean it is null when the control's mean plus or minus its margin reaches 0, where Fieller's set of ratios is unbounded. The interval on the difference is still reported. |
| A relative-change interval bound above +1,000,000% |
The dashboard shows a bound beyond 1,000,000% in either direction as 'above +1,000,000%' or 'below −1,000,000%'; the API returns the exact number. |
| The impact projected to all traffic, to a later period, or in a currency |
incremental_impact is the difference from the control times the treatment's own assigned users so far, as a plain number: nothing projects it further, and no currency or unit is read or added. |
| The intervals and the impact in the data export |
The export has no interval or impact column; GET /api/v1/results/{experiment_id} reports them. |
| An interval on the difference, or an impact, in a warehouse analysis of a mean |
A warehouse analysis gives a proportion metric's treatments their difference from the control with its interval, the relative change's interval and the impact, as /results does. A mean metric's treatments get none of them; their p-value and relative change are reported as before. |
| A traffic allocation for the experiment as a whole |
There is none. Variant weights must add up to exactly 100%, or the request is refused with 422. Users are left out only by targeting rules, a mutual exclusion group or the global holdout. |
| One user known by two ids, such as a visitor who signs in |
No identify or alias call: a user who signs in under a new id mid-funnel gets a second assignment. Assign with the id you will track purchases with. |
| An LLM experiment that picks the variant with the lowest value |
An LLM experiment's winner is the variant with the highest mean of the metric it is compared on: the business metric when every variant has one, otherwise the judge score. There is no lower-is-better setting for it. |
An experiment balanced within a nested or dotted attribute (user.tier) |
A balanced experiment reads its attribute from the top level of the assign context only: a user sent {"user": {"tier": "gold"}} or {"user.tier": "gold"} is balanced in unknown, and the setting refuses a key with a dot. See Balance the split within groups. |
| A balanced experiment with more than 20 groups |
At most 20 values are listed; every other value is balanced together as other, and users without the attribute as unknown. |
| A balanced experiment with more than 10 saved segments |
At most 10 segments are listed, as in one set of targeting rules. |
| A user balanced in each of the saved segments they belong to |
A user is balanced in the first listed segment they belong to, so the order of segments decides; a user in none of them is balanced in other. |
| Archiving a saved segment that an experiment balances within |
Refused with 409 while a draft, active or paused experiment balances within it, as for a segment its targeting rules use. |
| A balanced 34/33/33 split |
Every share of a balanced experiment must be a multiple of 5, such as 50/50, 20/80, 35/35/30 or 40/30/30; any other split is refused with 422. |
| A balanced bandit or split URL experiment |
Both are refused with 422: a bandit moves traffic between variants as results come in, and the edge router sends no user attributes. |
| Changing what a running experiment balances within |
balance_within cannot be changed once the experiment leaves draft (400). A clone copies it and can be changed while it is a draft. |
| A user who changes tier moving to their new group |
A user keeps the variant and the group of their first assignment whatever a later call sends. |
| Every user of a balanced experiment balanced within their group |
A user assigned while the server is busy balancing other users is assigned by the hash instead, keeps their group and is counted in experiment_balance_fallback_total. An older server version, during a rolling deploy or a rollback, assigns every new user by the hash with no group. Either way the user's chance of each variant is still its share, but they are outside the balance. |