Quick answer
In Amazon ESCI’s US test slice, 72.42% of candidate pairs on one- or two-term searches were exact, versus 58.09% on searches with five or more terms. Substitutes appeared at least once on 72.15% of the longer queries, a query-level measure.
Key findings and sources
Long-tail candidate lists had fewer exact pairs
In Checkout Reply’s Amazon ESCI US test reanalysis, 72.42% of one- or two-term candidate pairs were exact versus 58.09% for five or more terms.
Amazon Science, Shopping Queries Dataset (ESCI)
22,458 US query IDs in the Amazon ESCI large-version test split Group by whitespace term count and divide exact pairs by all pairs in each length band.
Over seven in ten five-plus-term queries had a substitute
In the Amazon ESCI US test slice, 3,810 of 5,281 queries with at least five terms had a judged substitute (72.15%), compared with 3,122 of 5,878 shorter one- or two-term queries (53.11%).
Amazon Science, Shopping Queries Dataset (ESCI)
22,458 US query IDs in the Amazon ESCI large-version test split Collapse judged pairs to query ID and count any substitute per query.
Exact pair share changed more than ten points
In the Amazon ESCI US benchmark, exact share was 14.32 percentage points higher among one- or two-term candidate pairs than among pairs on queries with at least five terms.
Amazon Science, Shopping Queries Dataset (ESCI)
22,458 US query IDs in the Amazon ESCI large-version test split Subtract the unrounded five-plus-term exact share from the one- or two-term exact share.
Almost every query in the longest band still had an exact candidate
In the Amazon ESCI US test slice, only one of 5,281 queries with at least five terms lacked a judged exact candidate. This curated benchmark does not establish a live no-results rate.
Amazon Science, Shopping Queries Dataset (ESCI)
22,458 US query IDs in the Amazon ESCI large-version test split Collapse candidate pairs to query ID and count queries with zero E labels.
Compare the results
Longer searches had fewer exact candidate pairs
Longer requests need more evidence checks before suggesting a fallback.
Longer searches carry more requirements
A one-word search may leave the category or pack size up to the shopper. A longer request can combine an item, color, use case and compatibility detail. The extra words can be useful retrieval signals while also making each candidate answer more conditions.
Of 109,760 judged pairs on one- or two-term searches, 79,484 were exact. Of 105,714 pairs on searches with five or more terms, 61,413 were exact. Those are shares of the benchmark candidate pairs. They do not say that a shopper had fewer good choices, because the candidate set is curated and a query can still contain an exact match.
Count the queries where a fallback exists
Pair share answers how many of the judged entries belong to a label. Query presence answers whether a shopper request has at least one judged alternative. An individual query with many substitute candidates can raise pair share without expanding the set of requests where a fallback is available.
In the longest group, 3,810 of 5,281 queries had at least one substitute. The shortest group had 3,122 of 5,878. Only one query in the longest group lacked any exact candidate, so this benchmark cannot establish that long requests commonly return no useful item.
Test what the substitute keeps and changes
Build a small set of realistic multi-constraint queries and log which constraints a proposed result preserves. For each test, retain the original request, item type, requested variant and the explicit difference the fallback would introduce. Judge the recommendation at the query level and record whether a verified alternative actually exists.
For customer replies, preserve a hard constraint such as compatibility or pack size and make the changed preference explicit. A report that looks only at total matches can hide failure on rare but expensive constraints. Compare your own result set and final cart variant before applying this benchmark to a production ranking decision.
Study data
| Measure | Value | Context |
|---|---|---|
| 1–2 terms unique query IDs | 5,878 queries | Original grouped count |
| 1–2 terms candidate pairs | 109,760 query-product pairs | Original grouped count |
| 1–2 terms exact candidate pairs | 79,484 query-product pairs | Original grouped count |
| 1–2 terms substitute candidate pairs | 17,330 query-product pairs | Original grouped count |
| 1–2 terms irrelevant candidate pairs | 9,916 query-product pairs | Original grouped count |
| 1–2 terms queries without an exact candidate | 0 queries | Original grouped count |
| 1–2 terms queries with a substitute | 3,122 queries | Original grouped count |
| 1–2 terms queries with an irrelevant candidate | 2,007 queries | Original grouped count |
| 1–2 terms exact pair share | 72.42% | Original recomputed share or rate |
| 1–2 terms substitute pair share | 15.79% | Original recomputed share or rate |
| 1–2 terms query with substitute share | 53.11% | Original recomputed share or rate |
| 3–4 terms unique query IDs | 11,299 queries | Original grouped count |
| 3–4 terms candidate pairs | 210,288 query-product pairs | Original grouped count |
| 3–4 terms exact candidate pairs | 136,451 query-product pairs | Original grouped count |
| 3–4 terms substitute candidate pairs | 48,869 query-product pairs | Original grouped count |
| 3–4 terms irrelevant candidate pairs | 19,424 query-product pairs | Original grouped count |
| 3–4 terms queries without an exact candidate | 0 queries | Original grouped count |
| 3–4 terms queries with a substitute | 7,326 queries | Original grouped count |
| 3–4 terms queries with an irrelevant candidate | 3,986 queries | Original grouped count |
| 3–4 terms exact pair share | 64.89% | Original recomputed share or rate |
| 3–4 terms substitute pair share | 23.24% | Original recomputed share or rate |
| 3–4 terms query with substitute share | 64.84% | Original recomputed share or rate |
| 5+ terms unique query IDs | 5,281 queries | Original grouped count |
| 5+ terms candidate pairs | 105,714 query-product pairs | Original grouped count |
| 5+ terms exact candidate pairs | 61,413 query-product pairs | Original grouped count |
| 5+ terms substitute candidate pairs | 28,855 query-product pairs | Original grouped count |
| 5+ terms irrelevant candidate pairs | 13,098 query-product pairs | Original grouped count |
| 5+ terms queries without an exact candidate | 1 queries | Original grouped count |
| 5+ terms queries with a substitute | 3,810 queries | Original grouped count |
| 5+ terms queries with an irrelevant candidate | 2,284 queries | Original grouped count |
| 5+ terms exact pair share | 58.09% | Original recomputed share or rate |
| 5+ terms substitute pair share | 27.3% | Original recomputed share or rate |
| 5+ terms query with substitute share | 72.15% | Original recomputed share or rate |
What changes when you compare the figures
| Comparison | Result | Calculation |
|---|---|---|
| 1–2 terms exact pair share | 72.42% | = 79484 / 109760 * 100 |
| 1–2 terms substitute pair share | 15.79% | = 17330 / 109760 * 100 |
| 1–2 terms query with substitute share | 53.11% | = 3122 / 5878 * 100 |
| 3–4 terms exact pair share | 64.89% | = 136451 / 210288 * 100 |
| 3–4 terms substitute pair share | 23.24% | = 48869 / 210288 * 100 |
| 3–4 terms query with substitute share | 64.84% | = 7326 / 11299 * 100 |
| 5+ terms exact pair share | 58.09% | = 61413 / 105714 * 100 |
| 5+ terms substitute pair share | 27.3% | = 28855 / 105714 * 100 |
| 5+ terms query with substitute share | 72.15% | = 3810 / 5281 * 100 |
| Long minus short query substitute coverage | 19.03 percentage points | = (3810 / 5281 - 3122 / 5878) * 100 |
| Short minus long exact pair share | 14.32 percentage points | = (79484 / 109760 - 61413 / 105714) * 100 |
| Long minus short queries with irrelevant candidate | 9.11 percentage points | = (2284 / 5281 - 2007 / 5878) * 100 |
Methodology and sources
Filter public Amazon ESCI examples to US large-version test rows. Count whitespace-separated terms in the query after trimming outer whitespace. Group one or two, three or four, and at least five terms. First count candidate-pair labels. Independently collapse by query ID to count presence of at least one substitute or irrelevant candidate. Compare unrounded rates and round displayed numbers to two decimals. The benchmark provides judged candidates without observed impressions or rank.
Download the source data (CSV)
Download the calculations (CSV)
Download the citation notes (CSV)
Download the evidence behind the new findings (CSV)
Related reading
Ecommerce model number search statistics across exact and substitute products
AI shopping recommendation accuracy statistics
Ecommerce product review statistics and what shoppers use to decide
Sources and definitions
Amazon Science, Shopping Queries Dataset (ESCI)
The source data includes the population, period, denominator and limitations for every figure.