office, paperwork, workplace, accountant, advise, business, calculator, coffee, laptop, mobile phone, notebook, pen, phone, top view, flat lay, accountant, accountant, accountant, accountant, accountant
Photo by hapisadi12 on Pixabay

Attribution

Part of Basket and merchandising analytics

Measuring the effect of a product recommendation

Separate recommendation exposure and clicks from sales effect, then compare eligible groups on a defined commercial outcome.

This is an editorial pen name used by the publication, not a claim of an independently verified personal identity. Original editorial credit: Desk Customer Analytics.

To assess a recommendation's effect, compare a commercial outcome for eligible shoppers assigned to see it against the same outcome for a comparable group assigned a different experience. Record what was actually displayed. Clicks and tracked purchases describe engagement, but alone cannot show what would have happened without the recommendation.

Define what changes

Name the product-page placement, eligible audience, recommended products and comparison experience. State whether the change adds a block, substitutes an item or moves a placement. Record cases where no recommendation displays.

Shopify Search & Discovery recommendations need a suitable product-page theme section. Its display rules can suppress a product that is already in the visitor's basket. Complementary products must have stock above zero; related products may display with zero stock when continued selling is enabled. Saving a recommendation configuration therefore does not establish that a shopper saw it.

Key Considerations for Measuring Recommendation Effectiveness

  • Required theme sectionShopify Search & Discovery recommendations need a suitable product-page theme section
  • Stock requirementsComplementary products must have stock above zero; related products may display with zero stock when continued selling is enabled
  • Display confirmationSaving a configuration does not guarantee a shopper saw the recommendation

Separate exposure, engagement and outcome

MeasureCounting rulePurpose
ExposureEligible sessions in which the block was displayedEstablish opportunity to interact.
EngagementExposed sessions with a recommendation clickDiagnose selection of the displayed item.
Commercial outcomeQualifying orders, retained sales or defined contribution per assigned eligible sessionCompare the overall result.

Choose a primary commercial outcome before launch. If the aim is add-on sales, inspect purchases of the recommended item and total basket results; a recommendation could shift a purchase from another item. State order-status and adjustment rules for monetary outcomes, and mark recent orders provisional if later changes may affect them.

Shopify's behaviour reports include “Product recommendation conversions over time” and “Product recommendations with low engagement”. Check the definitions and coverage of any report before using it to assess a recommendation.

A tracked purchase rate is not an incremental sales rate. It describes observed activity, not what would have happened without the recommendation.

Make a fair comparison

Where feasible, assign eligible shoppers or sessions before they encounter the placement. Analyse the assigned groups, including people who never click, using the same outcome rule and observation period. Keep assignment consistent for returning shoppers. Record concurrent changes in price, promotion or product availability.

Report group sizes, the outcome in each group, their difference and missing-data limits. If a suitable comparison is unavailable, report displays, clicks and later orders as descriptive evidence and leave the incremental effect unestimated. Base the keep, revise or stop decision on the predefined commercial outcome rather than a click-rate change alone.

More from Attribution

Funnel Analysis

Comparing category navigation paths

Compare category routes from a defined entry task to relevant products, with clear counting rules and path-analysis limits.