Semantic search statistics from a randomized retail experiment

Updated 38 source citationsSources checked 12 November 2026

On this page
  1. Quick answer
  2. Key Takeaways
  3. What must a search rewrite preserve about the requested product?
  4. How large was the purchase effect in absolute terms?
  5. How did the search route change?
  6. Which buying questions remained on the product page?
  7. How should meaning and transaction details be evaluated?
  8. Which audience and purchase window did the test cover?
  9. Which reported measures support these findings?
  10. How much search cost could the observed sales estimate cover?
  11. Methodology and Sources
Download study text

Quick answer

A randomized retail experiment involving 1,849,382 consumers found that semantic query refinement increased sales by 2.93% and purchase probability by 0.00101. The latter equals 1.01 additional purchasers per 1,000 assigned consumers. Click-through improved, while the click-to-order and basket-value estimates remained statistically inconclusive. The intervention changed which products search returned (Generative AI and Sales Productivity researchers, Jun 2026).

A useful search rewrite must recover the intended product while preserving the constraint that matters to the shopper. This analysis examines the randomized purchase result, its absolute size and the search journey around it. The product-page evidence remains a separate part of the decision, which matters when a search assistant helps someone resume an interrupted purchase.

Key Takeaways

  • Semantic query refinement added 1.01 purchasers per 1,000 The published purchase-probability effect was 0.00101 across 1,849,382 consumers, equivalent to 1.01 additional purchasers per 1,000 assigned consumers (Generative AI and Sales Productivity researchers, Jun 2026).
  • The purchase interval remained above zero Using the published standard error of 0.000411, the approximate normal 95% interval for the purchase effect is 0.204 to 1.816 additional purchasers per 1,000 assigned consumers (Generative AI and Sales Productivity researchers, Jun 2026).
  • Search sales rose by an estimated $64.80 per 1,000 assignments The semantic query-refinement experiment reports a $0.0648 sales increase per assigned consumer, equivalent to $64.80 per 1,000 assignments and a reported 2.93% relative effect (Generative AI and Sales Productivity researchers, Jun 2026).
  • Click-to-order remained uncertain The source reports a click-to-order effect of −0.000088 with a standard error of 0.000182. Its uncertainty spans both improvement and decline, while the treatment changed search retrieval rather than product-page content (Generative AI and Sales Productivity researchers, Jun 2026).

What must a search rewrite preserve about the requested product?

A shopper can describe a product differently from the store’s catalog. They may use another language, an everyday term or a description of the job the product needs to do. A useful search system connects that request with suitable items while preserving the constraints that matter, such as size, quantity and compatibility.

Semantic search statistics: sales effect per 1,000 consumers is $64.80 with an approximate 95% interval $3.26 to $126.34. At an assumed 50% margin and $30 cost, the modeled net is $2.40 with interval negative $28.37 to positive $33.17.
Figure 1: Positive sales can leave a narrow cost allowance. Source: Generative AI and Sales Productivity, Table C3, Jun 2026. Original analysis of the cited source. Source periods and calculation limits appear in the text.

The source experiment tested one approach to that problem. Consumers in the treatment received search-query translation with semantic understanding. The control used basic translation without the same semantic refinement. Prices and the broader workflow stayed consistent across the randomized comparison.

The question was therefore whether a better interpretation of the query changed subsequent shopping outcomes. It offers stronger evidence for a business decision than a benchmark showing that a system can return plausible products, although the result remains specific to the intervention and platform tested.

How large was the purchase effect in absolute terms?

Across 1,849,382 consumers, the published purchase-probability effect was 0.00101. That is 0.101 percentage points, or 1.01 additional purchasers per 1,000 assigned consumers. The authors reported it as a 1.15% relative increase (Generative AI and Sales Productivity researchers, Jun 2026).

Using the published standard error, the approximate 95% interval is 0.204 to 1.816 additional purchasers per 1,000. Both ends are above zero, while the width still matters for an economic forecast. A store should keep that range beside the point estimate when deciding what to spend on search improvements (Generative AI and Sales Productivity researchers, Jun 2026).

The sales effect was $0.0648 per assigned consumer, equivalent to $64.80 per 1,000 assignments. Its reported relative increase was 2.93%. These figures describe purchases during the experiment window. An annual value estimate would require evidence about how long the effect persists (Generative AI and Sales Productivity researchers, Jun 2026).

How did the search route change?

The experiment increased product clicks by 0.0901 per assigned consumer, or 90.1 clicks per 1,000. Its additional analysis also found a higher click-through rate, measured as product clicks relative to product views (Generative AI and Sales Productivity researchers, Jun 2026).

The authors observed fewer product views before a click or purchase in their corresponding analyses. Those findings are consistent with shoppers reaching suitable products with less search effort. Which semantic changes made each result more useful remains a mechanism question.

The distinction matters when evaluating an assistant. A system that produces a longer list of product links may increase activity without making the decision easier. Inspect whether shoppers reach an appropriate item, understand the relevant difference and can proceed with their chosen variant.

Which buying questions remained on the product page?

The click-to-order estimate was −0.000088, with a standard error of 0.000182. Its uncertainty spans both directions. The experiment therefore gives clearer evidence of improved search engagement and overall purchase incidence than of a change in what happens after a shopper clicks an item (Generative AI and Sales Productivity researchers, Jun 2026).

That is consistent with the treatment’s scope. It changed the products retrieved immediately after a query. It did not change the information displayed on every product-detail page. A relevant search result can still lead to missing measurements, unclear delivery conditions or an unanswered compatibility question.

For a merchant, this is a reason to inspect the sequence. Findability, product understanding and checkout completion can each constrain the outcome. Each remaining stage still needs its own inspection.

How should meaning and transaction details be evaluated?

A semantic rewrite should retain details the shopper supplied. If someone asks for a two-pack, a replacement that fits a named device or a product below a stated price, the system must carry those conditions into retrieval and the eventual recommendation.

Review a sample of original queries alongside their interpreted meaning, returned products and selected variants. Separate a different wording for the same requirement from a substitution that changes the requirement. This makes the quality review more useful than checking whether a returned item looks broadly related.

For conversational selling, the same principle applies when a recovery message offers to help locate an alternative. A clearer interpretation can be valuable, but the reply still needs verified product details and an accurate route back to the relevant item or checkout.

Which audience and purchase window did the test cover?

The study consisted of three nine-day language subexperiments during May and June 2024. The authors accounted for entry-day and language cohorts in their estimates. That setting is important because the value of query refinement can depend on the quality of the translation and search system it replaces (Generative AI and Sales Productivity researchers, Jun 2026).

A store with a strong existing semantic search may have less room to improve. A store with sparse catalog data may face a different problem altogether. A local test is needed to estimate the lift for either store.

Use the result to define a measurable intervention. Specify which queries are eligible, what interpretation changes, the comparison experience and the purchase window. Then report purchase probability and sales per assigned shopper, with search activity used to explain the result. That preserves the connection between a technical change and the merchant outcome it is intended to improve.

Which reported measures support these findings?

The figures below preserve the source’s outcome labels and units. Read comparisons within their stated experiment, population and period.

Reported measureValue and unitSource
Views-0.549 views per consumer(Generative AI and Sales Productivity researchers, Jun 2026)
Views relative effect-0.18 %(Generative AI and Sales Productivity researchers, Jun 2026)
Views standard error0.887 views per consumer(Generative AI and Sales Productivity researchers, Jun 2026)
Product clicks0.0901 clicks per consumer(Generative AI and Sales Productivity researchers, Jun 2026)
Product clicks relative effect1.1 %(Generative AI and Sales Productivity researchers, Jun 2026)
Product clicks standard error0.0245 clicks per consumer(Generative AI and Sales Productivity researchers, Jun 2026)
Orders0.00154 orders per consumer(Generative AI and Sales Productivity researchers, Jun 2026)
Orders relative effect0.94 %(Generative AI and Sales Productivity researchers, Jun 2026)
Orders standard error0.00107 orders per consumer(Generative AI and Sales Productivity researchers, Jun 2026)
Sales0.0648 USD per consumer(Generative AI and Sales Productivity researchers, Jun 2026)
Sales relative effect2.93 %(Generative AI and Sales Productivity researchers, Jun 2026)
Sales standard error0.0314 USD per consumer(Generative AI and Sales Productivity researchers, Jun 2026)
Purchase probability0.00101 probability difference(Generative AI and Sales Productivity researchers, Jun 2026)
Purchase probability relative effect1.15 %(Generative AI and Sales Productivity researchers, Jun 2026)
Purchase probability standard error0.000411 probability difference(Generative AI and Sales Productivity researchers, Jun 2026)
Cart value0.37 USD per purchaser(Generative AI and Sales Productivity researchers, Jun 2026)
Cart value relative effect1.47 %(Generative AI and Sales Productivity researchers, Jun 2026)
Cart value standard error0.334 USD per purchaser(Generative AI and Sales Productivity researchers, Jun 2026)
Click-through rate0.000767 ratio difference(Generative AI and Sales Productivity researchers, Jun 2026)
Click-through rate relative effect2.02 %(Generative AI and Sales Productivity researchers, Jun 2026)
Click-through rate standard error7.9e-05 ratio difference(Generative AI and Sales Productivity researchers, Jun 2026)
Click-to-order rate-8.8e-05 ratio difference(Generative AI and Sales Productivity researchers, Jun 2026)
Click-to-order rate relative effect-0.35 %(Generative AI and Sales Productivity researchers, Jun 2026)
Click-to-order rate standard error0.000182 ratio difference(Generative AI and Sales Productivity researchers, Jun 2026)

How are the additional calculations reproduced?

These are arithmetic comparisons of the published inputs. They retain the same sample and limitations as the inputs and do not create an additional experiment.

CalculationExpressionResult and unit
Additional purchasers per 1,000.00101 * 10001.01 purchasers per 1,000
Additional sales per 1,000.0648 * 100064.8 USD per 1,000 consumers
Additional clicks per 1,000.0901 * 100090.1 clicks per 1,000 consumers
Purchase lift lower 95% limit per 1,000(.00101 - 1.96 * .000411) * 10000.20444 purchasers per 1,000
Purchase lift upper 95% limit per 1,000(.00101 + 1.96 * .000411) * 10001.81556 purchasers per 1,000
Click-to-order lower 95% limit-.000088 - 1.96 * .000182-0.000445 ratio points
Click-to-order upper 95% limit-.000088 + 1.96 * .0001820.000269 ratio points

How much search cost could the observed sales estimate cover?

The $64.80 sales increase per 1,000 assigned consumers is revenue, not a technology budget. Multiplying it by an assumed 50% incremental contribution margin leaves $32.40 before the cost of search, integration, support and other changes. At assumed margins of 30% and 70%, the corresponding amounts are $19.44 and $45.36. These are worked economic sensitivities, not margins reported by the researchers. (Generative AI and Sales Productivity researchers, Table C 3 sales input, Jun 2026).

The published sales standard error is $0.0314 per assigned consumer. A normal 95% interval for the sales effect is approximately $3.26 to $126.34 per 1,000 assignments. At the same fixed 50% margin assumption, the implied contribution interval is approximately $1.63 to $63.17 before costs. This linear transformation preserves the source estimate's uncertainty, conditional on that assumed margin; it does not add a measured profit outcome.

What if the supplier charges per query rather than per consumer?

Keep the unit conversion visible. For a worked cost of $0.01 per refinement and an average of three refinements per assigned consumer, variable cost would be $30 per 1,000 assignments. Subtracting that fixed modeled cost from the 50% margin calculation gives a point difference of $2.40, with the transformed interval spanning approximately negative $28.37 to positive $33.17, before implementation and other costs. The price and query count are assumptions, not a vendor quotation or observed usage.

That example changes the decision. A positive sales estimate can leave little room for cost when many calls occur per assigned shopper. Counting only shoppers who clicked a result or only successful queries would shrink the cost denominator selectively. Include all eligible assignments, including those with no purchase, when reconciling per-assignment value with usage charges.

Which outcomes remain unresolved despite more clicks?

The order-count effect is 0.00154 per consumer with a standard error of 0.00107. Its approximate 95% interval is negative 0.557 to positive 3.637 orders per 1,000. The purchase-incidence interval is positive, but the order-count interval crosses zero; repeated orders and different outcome variation mean those statements can coexist. A purchase is a binary consumer outcome, while orders can count more than once. (Generative AI and Sales Productivity researchers, Table C 3, Jun 2026).

A local evaluation should preserve query eligibility and assignment, then connect refinement usage, retrieval quality, paid orders and contribution. Track cases where a rewrite drops quantity, compatibility or a price limit as failures even if the returned product receives a click. A more capable existing search baseline may leave less room for improvement than the basic-translation comparison in the paper.

The economic sensitivity and intervals and calculation expose the assumptions. The nine-day subexperiments do not establish annual persistence, and no annual value or store-specific payback is inferred.

Methodology and Sources

We transcribed Tables C3 and C4 and the experiment description from the June 2026 paper. We calculated effects per 1,000 assigned consumers and approximate normal 95% intervals from the published coefficients and standard errors. Overall purchase probability, click-through and click-to-order use different denominators. We preserve the three language-specific nine-day subexperiments and do not combine their effects with other AI workflows (Generative AI and Sales Productivity researchers, Jun 2026).

Generative AI and Sales Productivity researchers: Version 4, 1 June 2026. Citation months identify the cited publication or version. Where a citation explicitly says “source checked”, November 2026 is the verification month, and the original study period remains stated in the text. Source observations, calculations and the proposed merchant applications have distinct meanings.

The data do not establish a click-to-order improvement or higher basket spending. Attribution is to the platform query-refinement intervention, not a generic LLM brand or checkout-recovery system.

Glossary

  • Query refinement: A change to the search wording before retrieval.
  • Purchase probability: The fraction of assigned consumers meeting the purchase outcome.
  • Semantic meaning: The product request conveyed by the shopper’s words.
  • Absolute effect: The arithmetic difference between two outcome probabilities.
  • Randomized comparison: A comparison based on assignment by chance.

Sources

Cite this page

Use the citation below when referring to this analysis. The original source and calculation scope accompany each finding.

Checkout Reply. "Semantic search statistics from a randomized retail experiment." Checkout Reply, November 2026. https://checkoutreply.com/guides/semantic-search-statistics-from-a-randomized-retail-experiment/
<a href="https://checkoutreply.com/guides/semantic-search-statistics-from-a-randomized-retail-experiment/">Semantic search statistics from a randomized retail experiment</a> (Checkout Reply, November 2026)