Product compatibility search statistics measure how often plausible options can be confused with the specific item a shopper needs. In Amazon’s U.S. held-out benchmark, 33 of 47 queries naming a selected device family had an exact match and at least one other relevance label. Baymard’s separate 2026 benchmark found that 56% of ecommerce sites and apps did not adequately support search (Reddy et al., Jun 2022) (Baymard, Apr 2026).
We examine device phrases, search wording, query length and candidate-set size. For a merchant, the work is deciding which near matches deserve verification before a product link goes to a shopper. The candidates represent judgments in a benchmark, with scope and denominators retained in each comparison.
Key Takeaways
- 33 of 47 held-out named-device queries had exact and non-exact candidates. (Reddy et al., Jun 2022).
- 44.7% of the same named-device queries included a substitute. (Reddy et al., Jun 2022).
- 76.3% of other U.S. held-out queries had mixed exact and other labels. (Reddy et al., Jun 2022).
- 10.0% of judged pairs in that other-query group were irrelevant. (Reddy et al., Jun 2022).
- 20 of 27 held-out Apple phone and tablet searches had mixed results. (Reddy et al., Jun 2022).
- 8 of 11 held-out console searches had mixed results. (Reddy et al., Jun 2022).
- 69.6% of held-out explicit sets with 11 to 20 results had mixed relevance. (Reddy et al., Jun 2022).
- Irrelevant pairs represented 8.3% within the same defined sets. (Reddy et al., Jun 2022).
- 65.5% of held-out 4-to-5-word explicit search sets were mixed. (Reddy et al., Jun 2022).
- Among held-out queries with six to eight words, 72.7% were mixed. (Reddy et al., Jun 2022).
- 58.0% of the mobile sites in Baymard’s benchmark had mediocre-or-worse overall search UX. (Baymard, Apr 2026).
- 94.7% of the other explicit fit-worded benchmark sets had mixed relevance. (Reddy et al., Jun 2022).
How often does a named device search contain competing options?
Explicit device names help locate a catalog result, but the judged set can include several plausible product relationships. Counting at query level and candidate level answers different questions.

What does a single query reveal?
An exact result can be available while other results are also plausible. A compatibility reply needs to identify the individual item and what relationship the verified record actually supports.
- Queries with at least one exact candidate. 100.0% (47 of 47) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 70.2% (33 of 47) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with at least one substitute. 44.7% (21 of 47) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with at least one complement. 19.1% (9 of 47) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 34.0% (16 of 47) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with only exact candidates. 29.8% (14 of 47) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 67.6% (513 of 759) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit wording (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 20.8% (158 of 759) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit wording (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 3.6% (27 of 759) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit wording (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 8.0% (61 of 759) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit wording (Reddy et al., Jun 2022).
- Mean judged candidates per query. 16.1 candidates in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
Does the wider training split show a similar pattern?
The training sample gives another view of how the benchmark was constructed. It is not additional held-out model evaluation, and it can include a different mix of search phrases.
- Queries with at least one exact candidate. 100.0% (131 of 131) in Amazon ESCI large U.S. train split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 59.5% (78 of 131) in Amazon ESCI large U.S. train split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with at least one substitute. 42.0% (55 of 131) in Amazon ESCI large U.S. train split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with at least one complement. 9.2% (12 of 131) in Amazon ESCI large U.S. train split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 32.1% (42 of 131) in Amazon ESCI large U.S. train split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Queries with only exact candidates. 40.5% (53 of 131) in Amazon ESCI large U.S. train split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 77.3% (1,654 of 2,140) in Amazon ESCI large U.S. train split, judged query–product pairs, explicit wording (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 13.8% (296 of 2,140) in Amazon ESCI large U.S. train split, judged query–product pairs, explicit wording (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 1.4% (29 of 2,140) in Amazon ESCI large U.S. train split, judged query–product pairs, explicit wording (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 7.5% (161 of 2,140) in Amazon ESCI large U.S. train split, judged query–product pairs, explicit wording (Reddy et al., Jun 2022).
- Mean judged candidates per query. 16.3 candidates in Amazon ESCI large U.S. train split, distinct benchmark queries, explicit wording (Reddy et al., Jun 2022).
Include an exact accessory, a related alternative and an incompatible candidate in a retailer’s QA case. Test a follow-up after the shopper specifies a model suffix or year. A valid answer should preserve the fit condition and link the correct product. If compatibility cannot be verified, ask the one missing question that changes the choice.
How different are searches without the selected explicit wording?
Other benchmark queries give context for the frequency of mixed results. That comparison pool is broader and can include relevant device searches that use different wording.

What does the held-out comparison group contain?
A query can express compatibility without saying for and a selected device name. The other-query percentage is a benchmark reference, not a clean control that differs only by the phrase.
- Queries with at least one exact candidate. 100.0% (22,410 of 22,411) in Amazon ESCI large U.S. test split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 76.3% (17,098 of 22,411) in Amazon ESCI large U.S. test split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with at least one substitute. 63.5% (14,237 of 22,411) in Amazon ESCI large U.S. test split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with at least one complement. 15.7% (3,510 of 22,411) in Amazon ESCI large U.S. test split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 36.9% (8,261 of 22,411) in Amazon ESCI large U.S. test split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with only exact candidates. 23.7% (5,312 of 22,411) in Amazon ESCI large U.S. test split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 65.1% (276,835 of 425,003) in Amazon ESCI large U.S. test split, judged query–product pairs, all other wording (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 22.3% (94,896 of 425,003) in Amazon ESCI large U.S. test split, judged query–product pairs, all other wording (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 2.6% (10,895 of 425,003) in Amazon ESCI large U.S. test split, judged query–product pairs, all other wording (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 10.0% (42,377 of 425,003) in Amazon ESCI large U.S. test split, judged query–product pairs, all other wording (Reddy et al., Jun 2022).
- Mean judged candidates per query. 19.0 candidates in Amazon ESCI large U.S. test split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
How do the training comparison groups behave?
The larger training pool can stabilize a summary of the dataset. The difference between it and the test split can reflect selection, sampling and distinct products or categories.
- Queries with at least one exact candidate. 100.0% (74,757 of 74,757) in Amazon ESCI large U.S. train split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 71.7% (53,571 of 74,757) in Amazon ESCI large U.S. train split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with at least one substitute. 58.9% (44,007 of 74,757) in Amazon ESCI large U.S. train split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with at least one complement. 13.3% (9,944 of 74,757) in Amazon ESCI large U.S. train split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 33.5% (25,069 of 74,757) in Amazon ESCI large U.S. train split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Queries with only exact candidates. 28.3% (21,186 of 74,757) in Amazon ESCI large U.S. train split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 69.6% (968,556 of 1,390,923) in Amazon ESCI large U.S. train split, judged query–product pairs, all other wording (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 19.7% (273,963 of 1,390,923) in Amazon ESCI large U.S. train split, judged query–product pairs, all other wording (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 2.1% (29,098 of 1,390,923) in Amazon ESCI large U.S. train split, judged query–product pairs, all other wording (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 8.6% (119,306 of 1,390,923) in Amazon ESCI large U.S. train split, judged query–product pairs, all other wording (Reddy et al., Jun 2022).
- Mean judged candidates per query. 18.6 candidates in Amazon ESCI large U.S. train split, distinct benchmark queries, all other wording (Reddy et al., Jun 2022).
For a meaningful merchant-side test, hold the product intent fixed and write paired search phrases for the same device, one explicit and one ambiguous. Validate whether the shortlist changes and whether the reply retains the agreed constraint. The public benchmark measures its selected candidate sets without estimating that paired effect.
Do specific device families show different kinds of near matches?
A compatibility engine may need a generation, connector, case size or console version. Device families organize the benchmark examples by the literal phrase that qualified them.

What is visible for Apple phones and tablets?
This slice identifies queries naming a phone or tablet in the chosen wording. A common family can contain different devices and generations, each with a different required fitting detail.
- Queries with at least one exact candidate. 100.0% (27 of 27) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 74.1% (20 of 27) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Queries with at least one substitute. 55.6% (15 of 27) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Queries with at least one complement. 14.8% (4 of 27) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 33.3% (9 of 27) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Queries with only exact candidates. 25.9% (7 of 27) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 69.1% (302 of 437) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 24.5% (107 of 437) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 1.1% (5 of 437) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 5.3% (23 of 437) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit Apple phones and tablets (Reddy et al., Jun 2022).
- Mean judged candidates per query. 16.2 candidates in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Apple phones and tablets (Reddy et al., Jun 2022).
How do the smaller families differ in observed labels?
The smaller groups can change substantially with one query. Use them to design representative examples. A percentage alone cannot rank the reliability of every device family.
- Queries with exact and differently judged candidates. 50.0% (3 of 6) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Android devices (Reddy et al., Jun 2022).
- Queries with at least one substitute. 33.3% (2 of 6) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Android devices (Reddy et al., Jun 2022).
- Queries with at least one complement. 0.0% (0 of 6) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Android devices (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 10.4% (10 of 96) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit Android devices (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 72.7% (8 of 11) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Consoles (Reddy et al., Jun 2022).
- Queries with at least one substitute. 27.3% (3 of 11) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Consoles (Reddy et al., Jun 2022).
- Queries with at least one complement. 45.5% (5 of 11) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit Consoles (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 8.4% (15 of 178) in Amazon ESCI large U.S. test split, judged query–product pairs, explicit Consoles (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 66.7% (2 of 3) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit E-readers (Reddy et al., Jun 2022).
- Queries with at least one substitute. 33.3% (1 of 3) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit E-readers (Reddy et al., Jun 2022).
- Queries with at least one complement. 0.0% (0 of 3) in Amazon ESCI large U.S. test split, distinct benchmark queries, explicit E-readers (Reddy et al., Jun 2022).
A useful catalog record keeps the manufacturer’s model code and compatible generation separate from the host product family. A reply should state what part fits, what condition was checked, and what still needs confirmation. A substitute that solves the same job belongs in a separate branch from the exact listing.
Related research on Product substitute statistics.
Does a larger candidate set change the interpretation?
If a judged result set contains more candidates, it has more opportunities to expose a non-exact option. The query-level mix and the share of individual candidate labels should be read together.

What do typical held-out candidate sets show?
Most named-device held-out sets have 11 to 20 judged results. The larger group is exceptionally small. Retain each query and pair denominator to avoid reading one group as a stable trend.
- Queries with at least one exact candidate. 100.0% (46 of 46) in Amazon ESCI large U.S. test split, distinct benchmark queries, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 69.6% (32 of 46) in Amazon ESCI large U.S. test split, distinct benchmark queries, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with at least one substitute. 43.5% (20 of 46) in Amazon ESCI large U.S. test split, distinct benchmark queries, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with at least one complement. 17.4% (8 of 46) in Amazon ESCI large U.S. test split, distinct benchmark queries, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 34.8% (16 of 46) in Amazon ESCI large U.S. test split, distinct benchmark queries, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with only exact candidates. 30.4% (14 of 46) in Amazon ESCI large U.S. test split, distinct benchmark queries, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 67.9% (500 of 736) in Amazon ESCI large U.S. test split, judged query–product pairs, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 20.2% (149 of 736) in Amazon ESCI large U.S. test split, judged query–product pairs, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 3.5% (26 of 736) in Amazon ESCI large U.S. test split, judged query–product pairs, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 8.3% (61 of 736) in Amazon ESCI large U.S. test split, judged query–product pairs, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
- Mean judged candidates per query. 16 candidates in Amazon ESCI large U.S. test split, distinct benchmark queries, 11–20 labeled candidates per query (Reddy et al., Jun 2022).
What does more training evidence make visible?
The training candidates let the reader compare relevant and related item mixes when the cases differ in depth. Result-set breadth is established by benchmark sampling, not the number of candidates a live storefront would rank.
- Queries with at least one exact candidate. 100.0% (1 of 1) in Amazon ESCI large U.S. train split, distinct benchmark queries, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 100.0% (1 of 1) in Amazon ESCI large U.S. train split, distinct benchmark queries, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with at least one substitute. 100.0% (1 of 1) in Amazon ESCI large U.S. train split, distinct benchmark queries, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with at least one complement. 0.0% (0 of 1) in Amazon ESCI large U.S. train split, distinct benchmark queries, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 0.0% (0 of 1) in Amazon ESCI large U.S. train split, distinct benchmark queries, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Queries with only exact candidates. 0.0% (0 of 1) in Amazon ESCI large U.S. train split, distinct benchmark queries, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 50.0% (4 of 8) in Amazon ESCI large U.S. train split, judged query–product pairs, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 50.0% (4 of 8) in Amazon ESCI large U.S. train split, judged query–product pairs, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 0.0% (0 of 8) in Amazon ESCI large U.S. train split, judged query–product pairs, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 0.0% (0 of 8) in Amazon ESCI large U.S. train split, judged query–product pairs, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
- Mean judged candidates per query. 8 candidates in Amazon ESCI large U.S. train split, distinct benchmark queries, 6–10 labeled candidates per query (Reddy et al., Jun 2022).
When validating a reply, move the compatible item lower in a result list and include irrelevant but similarly named products ahead of it. A merchant should measure whether the assistant returns the right SKU when the most attractive lexical match has the wrong connector or generation.
Do longer compatibility searches settle the question?
An extra word can supply a missing specification or carry irrelevant context. Grouping by query length shows whether longer examples in this dataset still produced a mixed candidate set.

What happens across held-out length bands?
Length does not say which extra terms specify the conditions. Compare the number of eligible phrases and the mix of products across bands before interpreting a difference as evidence of better query wording.
- Queries with at least one exact candidate. 100.0% (5 of 5) in Amazon ESCI large U.S. test split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 80.0% (4 of 5) in Amazon ESCI large U.S. test split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with at least one substitute. 20.0% (1 of 5) in Amazon ESCI large U.S. test split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with at least one complement. 40.0% (2 of 5) in Amazon ESCI large U.S. test split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 40.0% (2 of 5) in Amazon ESCI large U.S. test split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with only exact candidates. 20.0% (1 of 5) in Amazon ESCI large U.S. test split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 77.4% (65 of 84) in Amazon ESCI large U.S. test split, judged query–product pairs, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 4.8% (4 of 84) in Amazon ESCI large U.S. test split, judged query–product pairs, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 3.6% (3 of 84) in Amazon ESCI large U.S. test split, judged query–product pairs, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 14.3% (12 of 84) in Amazon ESCI large U.S. test split, judged query–product pairs, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Mean judged candidates per query. 16.8 candidates in Amazon ESCI large U.S. test split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
Does the training evidence support broad generalization?
A larger sample can add breadth but still reflects different language and products. Resolve the decisive model property rather than asking a shopper to restate a long query that already contains everything needed.
- Queries with at least one exact candidate. 100.0% (7 of 7) in Amazon ESCI large U.S. train split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 28.6% (2 of 7) in Amazon ESCI large U.S. train split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with at least one substitute. 28.6% (2 of 7) in Amazon ESCI large U.S. train split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with at least one complement. 0.0% (0 of 7) in Amazon ESCI large U.S. train split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 0.0% (0 of 7) in Amazon ESCI large U.S. train split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Queries with only exact candidates. 71.4% (5 of 7) in Amazon ESCI large U.S. train split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 96.0% (119 of 124) in Amazon ESCI large U.S. train split, judged query–product pairs, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 4.0% (5 of 124) in Amazon ESCI large U.S. train split, judged query–product pairs, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 0.0% (0 of 124) in Amazon ESCI large U.S. train split, judged query–product pairs, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 0.0% (0 of 124) in Amazon ESCI large U.S. train split, judged query–product pairs, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
- Mean judged candidates per query. 17.7 candidates in Amazon ESCI large U.S. train split, distinct benchmark queries, query length 1–3 whitespace-separated tokens (Reddy et al., Jun 2022).
Pair generic device-name prompts with concise full model strings. Ask for clarification only when the unresolved attribute could change compatibility or availability. Measure correct link, correct fit explanation and unnecessary clarification separately.
Which compatibility questions does the first phrase rule miss?
Explicit fit or compatible-with wording can surface evidence outside the original device list. A separate search usability benchmark shows why merchants also need to test their own routes.

How does an alternate phrase capture additional cases?
Fit language can refer to products outside electronics, and the alternative set is disjoint from the primary rule. It shows what a literal query definition can miss, without claiming that all selected phrases represent device compatibility.
- Queries with at least one exact candidate. 100.0% (38 of 38) in Amazon ESCI large U.S. test split, distinct benchmark queries, other explicit fit expressions (Reddy et al., Jun 2022).
- Queries with exact and differently judged candidates. 94.7% (36 of 38) in Amazon ESCI large U.S. test split, distinct benchmark queries, other explicit fit expressions (Reddy et al., Jun 2022).
- Queries with at least one substitute. 84.2% (32 of 38) in Amazon ESCI large U.S. test split, distinct benchmark queries, other explicit fit expressions (Reddy et al., Jun 2022).
- Queries with at least one complement. 18.4% (7 of 38) in Amazon ESCI large U.S. test split, distinct benchmark queries, other explicit fit expressions (Reddy et al., Jun 2022).
- Queries with at least one irrelevant candidate. 42.1% (16 of 38) in Amazon ESCI large U.S. test split, distinct benchmark queries, other explicit fit expressions (Reddy et al., Jun 2022).
- Queries with only exact candidates. 5.3% (2 of 38) in Amazon ESCI large U.S. test split, distinct benchmark queries, other explicit fit expressions (Reddy et al., Jun 2022).
- Candidate pairs judged exact. 65.2% (542 of 831) in Amazon ESCI large U.S. test split, judged query–product pairs, other explicit fit expressions (Reddy et al., Jun 2022).
- Candidate pairs judged substitute. 27.9% (232 of 831) in Amazon ESCI large U.S. test split, judged query–product pairs, other explicit fit expressions (Reddy et al., Jun 2022).
- Candidate pairs judged complement. 1.4% (12 of 831) in Amazon ESCI large U.S. test split, judged query–product pairs, other explicit fit expressions (Reddy et al., Jun 2022).
- Candidate pairs judged irrelevant. 5.4% (45 of 831) in Amazon ESCI large U.S. test split, judged query–product pairs, other explicit fit expressions (Reddy et al., Jun 2022).
- Mean judged candidates per query. 21.9 candidates in Amazon ESCI large U.S. test split, distinct benchmark queries, other explicit fit expressions (Reddy et al., Jun 2022).
What does independent site research add?
The usability study rates whole sites and apps. It is not another product-judgment dataset. It helps prioritize real storefront evaluation by route and interface.
- Sites and apps with inadequate ecommerce search UX. 56.0% in Baymard 2026 overall Search UX benchmark, device-specific rows use their own base (Baymard, Apr 2026).
- Desktop retail sites with mediocre-or-worse search UX. 46.0% in Baymard 2026 overall Search UX benchmark, device-specific rows use their own base (Baymard, Apr 2026).
- Mobile retail sites with mediocre-or-worse search UX. 58.0% in Baymard 2026 overall Search UX benchmark, device-specific rows use their own base (Baymard, Apr 2026).
- Retail apps with mediocre-or-worse search UX. 64.0% in Baymard 2026 overall Search UX benchmark, device-specific rows use their own base (Baymard, Apr 2026).
Run an offline query check and a live customer experience check. Confirm the buyer can navigate from the corrected result to the exact available product, including selecting the right catalog variation. A correct product answer that only links a generic category can still leave the shopper deciding what fits.
Related research on Model-number search statistics.
Methodology and Sources
We grouped Amazon ESCI large-version U.S. query–product rows by query ID within train and held-out splits. An explicit query contains for followed by supported Apple devices, Android device names, selected consoles or Kindle. Candidate judgments are Exact, Substitute, Complement, or Irrelevant. Token length uses spaces and candidate count uses the observed group size. The alternative explicit wording is selected by a separate fit or compatible-with expression. Benchmarks do not represent all merchant searches or current inventory. The separate Baymard comparison describes overall site and app search usability. The public CSV retains query-level and candidate-pair denominators.
Glossary
- Exact: A candidate marked as an exact match under the benchmark’s relevance instructions.
- Substitute: A candidate labeled as an alternative.
- Complement: An item labeled as useful alongside the target.
- Irrelevant: A product pair with a negative relevance label.
- Query set: All judged candidates associated with one benchmark query.
- Candidate pair: One query matched to one judged product.
- Held-out split: Dataset examples reserved for evaluation by the producer.
- Compatibility evidence: A verified field or manufacturer entry binding the product to the device.