Measurement Planning
Part of Ecommerce experimentation
Defining a conversion experiment before changing the store
Write a store-test brief that fixes eligibility, assignment, the conversion formula, decision threshold and end rule before launch.
Define a conversion experiment in a written brief before changing the store. The brief should fix who qualifies, what each group sees, which conversion counts and when the team will decide. Without those rules, a later movement in conversion rate is hard to interpret.
Steps to define a reliable conversion experiment
- Write the question and treatmentDefine a clear hypothesis without assuming the outcome; isolate one change.
- Fix eligibility and counting rulesApply rules before observing behaviour; use consistent units and definitions.
- Decide what answer would be usefulSet minimum detectable effect, assess traffic, choose primary outcome and guardrails.
- Hand over the briefEnsure reproducibility by documenting all rules, assignment logic, and end conditions.
Write the question and treatment
A hypothetical brief might ask whether clearer delivery information on selected product pages increases the share of eligible shoppers who place a qualifying order within a chosen window. That is a question, not a predicted result. Name the decision owner and what a credible result would change.
Describe the control and treatment precisely. If the treatment also changes price, stock messaging and the purchase button, the comparison cannot isolate the delivery-information change. Preserve the page versions and launch time.
Fix eligibility and counting
State an eligibility rule that can be applied to both variants before treatment behaviour is observed. For a product-page change, that may be the first eligible request for an in-stock product in the selected group.
A shopper who never becomes eligible is outside that page-level comparison. Record every assigned eligible unit, including those that take no further action.
Choose whether assignment follows a shopper, account or device, and explain how returning visitors retain their variant. An account and a device do not identify the same population. If the available key can change, record that limit.
Define conversion as distinct qualifying purchasers divided by assigned eligible units, with a purchase window and order-status rule. Use the same unit in numerator and denominator.
If the question concerns orders rather than buyers, define a different metric. Keep an order-system outcome distinct from a Google Analytics purchase event, whose coverage depends on implementation.
Decide what answer would be useful
Choose the smallest improvement that could change the decision, then assess expected eligible traffic and variability before setting a target sample or duration. The minimum detectable effect describes design sensitivity under assumptions; it is not the uplift to expect. With insufficient traffic, a small commercially important effect may remain unresolved.
Name the primary outcome, relevant guardrails, data-quality checks and ordinary end condition. For the delivery-information example, checkout errors might be an immediate safety signal; later support contacts need a comparable collection rule and enough observation time. A delayed refund measure cannot serve as an immediate stop signal.
Specify the comparison method and how interim looks will be handled. Repeatedly stopping at a favourable conventional fixed-horizon p-value increases false-positive risk. Record concurrent campaigns and stock changes for the final review.
Hand over the brief
The handover should identify the question, owner, variants, eligibility, assignment unit, allocation, conversion formula, observation window, guardrails, data checks and end rule. Another team member should be able to reproduce who belongs in each group and which order counts.
If a rule changes after launch, retain the original brief and label the new analysis as a deviation.
