Measurement Planning
Part of Ecommerce experimentation
Choosing a guardrail metric for a store test
Choose store-test guardrails from plausible harms, with the right denominator, timing, data source and response rule.
Choose a guardrail by naming a plausible harm that a store change could cause even if its primary outcome improves. Then choose a measure that can reveal that harm in a useful population and within a useful time. A guardrail constrains the decision; it is not another metric to search for a favourable result.
Choosing a guardrail: four moves
- Name a plausible harmAsk what the proposed change could make worse, even if the primary outcome improves.
- Choose a revealing measureFinish the sentence: "If the treatment causes this problem, this measure should change in this direction."
- Fix denominator and timingMatch the denominator to the harm, label attempts and shoppers separately, and state the adjustment cut-off.
- Set the response in advanceSeparate an urgent fault response from the final commercial trade-off, with direction and material loss thresholds.
Start with the failure path
Ask what the proposed change could make worse. A checkout change might disrupt payment. Heavier product-page content might slow loading, and a promotional prompt might reduce value retained after discounts and adjustments. These are possible failure paths, not findings about a store.
For each candidate, finish this sentence: “If the treatment causes this problem, this measure should change in this direction.” If the team cannot explain the path or act on the result, keep the measure as a diagnostic. Do not treat it as a decision guardrail.
| Proposed change | Possible guardrail | Measurement condition |
|---|---|---|
| New checkout step | Recorded payment failures per assigned eligible shopper | Failures and assignment can be joined for both groups. |
| Heavier product-page content | Loading time for eligible page requests | Timing is captured comparably across variants and devices. |
| Stronger promotional message | Retained product value per assigned eligible shopper | Discounts and later adjustments use a stated cut-off. |
Failure paths and the guardrail that would reveal them
- New checkout stepGuardrail: recorded payment failures per assigned eligible shopper. Condition: failures and assignment can be joined for both groups.
- Heavier product-page contentGuardrail: loading time for eligible page requests. Condition: timing is captured comparably across variants and devices.
- Stronger promotional messageGuardrail: retained product value per assigned eligible shopper. Condition: discounts and later adjustments use a stated cut-off.
Fix the denominator and timing
Choose a denominator that matches the harm. Failed payments per payment attempt describes attempts, but the treatment may change who reaches payment. If the decision covers the whole checkout experience, show an assigned-shopper measure. Label attempts, events and distinct shoppers separately.
Some signals arrive quickly, such as a recorded checkout error. Others mature after the test.
Shopify sales reversals, for example, are recorded when processed and can reflect refunds, cancellations or edits; they are not limited to physical returns. A retained-value guardrail needs an adjustment cut-off and may remain provisional. It cannot serve as an immediate safety alert.
Check whether the treatment changes measurement. A new checkout route could stop sending an analytics event while orders still appear in the commerce record. Record each guardrail's source, event or order rule, missing-data coverage and assignment join before trusting a difference.
Before trusting a guardrail difference
- Denominator labelled as attempts, events or distinct shoppers
- Assigned-shopper view used where the decision covers the whole checkout experience
- Source recordedanalytics event rule or commerce order rule
- Missing-data coverage checked across both variants
- Assignment join verified before reading any difference
- Adjustment cut-off stated for any retained-value measure
Set the response in advance
Separate an urgent fault response from a final commercial trade-off. A payment failure may warrant prompt investigation or rollback. A small difference in return-adjusted value may need a later review. Set the direction and material loss threshold for the store's decision, and use an analysis method appropriate to any repeated interim checks.
A non-significant guardrail result does not prove safety when its uncertainty interval still allows an unacceptable loss. Consider whether the planned sample can assess that loss with useful precision. If it cannot, report the guardrail as unresolved.
Keep the set short and tied to plausible harm. In the final readout, show group values, differences, uncertainty, observation cut-offs and data coverage beside the primary outcome.
