Quick answer
In the US test slice of Amazon ESCI, 13.80% of exact candidates and 17.72% of substitutes had blank product color fields for searches containing a basic color term. Confirm the shopper’s requested variant against the store’s catalog before promising that the color is available.
Key findings and sources
Exact matches had blank color on 13.80% of pairs
Checkout Reply’s Amazon ESCI US test reanalysis found blank structured color values for 2,037 of 14,756 exact candidate pairs on searches containing a basic color term, or 13.80%.
Amazon Science, Shopping Queries Dataset (ESCI)
25,466 query-product pairs for US test searches containing a basic color term, ESCI large dataset Join on product ID and locale, then divide blank color pairs by all exact pairs in the lexical slice.
Substitute matches had blank color on 17.72% of pairs
The same ESCI reanalysis found blank color values for 1,422 of 8,024 substitute pairs, or 17.72%. This was 3.92 percentage points above the exact pair rate.
Amazon Science, Shopping Queries Dataset (ESCI)
25,466 query-product pairs for US test searches containing a basic color term, ESCI large dataset Divide blank substitute pairs by all substitute pairs, then subtract the unrounded exact share.
All 25,466 candidate pairs could be joined to a unique record
Every color-term query-product pair in the US ESCI test slice matched one product record. The 4,040 blank color fields were measured among those matched pairs, or 15.86% of all pairs.
Amazon Science, Shopping Queries Dataset (ESCI)
25,466 query-product pairs for US test searches containing a basic color term, ESCI large dataset Count unique ID-locale matches and null or trimmed empty product_color fields across all labels.
Exact and substitute results together had 3,459 blank fields
Across the exact and substitute labels in the US ESCI color-term slice, 3,459 of 22,780 candidate pairs had blank color fields. The combined pair-weighted share was 15.18%.
Amazon Science, Shopping Queries Dataset (ESCI)
25,466 query-product pairs for US test searches containing a basic color term, ESCI large dataset Pool the two class numerators and denominators before dividing.
Compare the results
A search match can carry an incomplete variant record
Across 25,466 benchmark pairs for color-term queries, 4,040 catalog rows had a blank color field. An exact judgment by an evaluator can coexist with a blank field because the benchmark relevance label and the structured attribute come from different parts of the dataset. For a person asking “do you have the blue one?”, product relevance gets you to the item. The variant record is where you check the promise.
Blank fields appeared on 2,037 of 14,756 exact pairs and 1,422 of 8,024 substitute pairs. The latter share was 3.92 percentage points higher. These are candidate-pair rates, so products that occur for several queries count several times. The rates describe this benchmark, with its own labeling and candidate selection.
Distinguish a known substitute from an unresolved attribute
If the requested color is unavailable, a store can offer another color only once it knows which variant is available and has checked the shopper’s other requirements. A product-level title such as “linen shirt” can hide several colors, while an exact search result can be available in only one size. Move from the query to a variant identifier and then check inventory for that variant.
If the source field is blank, look for an authoritative variant option or explicitly label the choice as unconfirmed. A filled field can still be a broad color family, an inconsistent merchant value or a stale catalog entry. The benchmark does not measure those error modes. Separate field coverage, correct mapping and current stock in the store’s own audit.
Audit the answers that create an immediate checkout decision
Sample real catalog combinations with high-intent questions about color and size. Start with products that are frequently viewed together, recently remerchandised, or offered in shade names that differ from the product feed. Record the requested variant, which authoritative source could confirm it, and whether the link opens with the same option selected.
Report the share of questions that can be answered with a current sellable variant and the share that need clarification. Keep both denominators visible. Review the questions that led to substitutions separately so the team can see whether an additional choice helped the shopper or sent them to an item they could not buy.
Study data
| Measure | Value | Context |
|---|---|---|
| Exact candidate pairs | 14,756 query-product pairs | Original join and aggregation |
| Exact pairs with a blank color field | 2,037 query-product pairs | Original join and aggregation |
| Exact pairs with a filled color field | 12,719 query-product pairs | Original join and aggregation |
| Exact distinct products | 14,343 products | Original join and aggregation |
| Exact pairs with blank color | 13.8% | Original count divided by class total |
| Substitute candidate pairs | 8,024 query-product pairs | Original join and aggregation |
| Substitute pairs with a blank color field | 1,422 query-product pairs | Original join and aggregation |
| Substitute pairs with a filled color field | 6,602 query-product pairs | Original join and aggregation |
| Substitute distinct products | 7,924 products | Original join and aggregation |
| Substitute pairs with blank color | 17.72% | Original count divided by class total |
| Complement candidate pairs | 437 query-product pairs | Original join and aggregation |
| Complement pairs with a blank color field | 95 query-product pairs | Original join and aggregation |
| Complement pairs with a filled color field | 342 query-product pairs | Original join and aggregation |
| Complement distinct products | 437 products | Original join and aggregation |
| Complement pairs with blank color | 21.74% | Original count divided by class total |
| Irrelevant candidate pairs | 2,249 query-product pairs | Original join and aggregation |
| Irrelevant pairs with a blank color field | 486 query-product pairs | Original join and aggregation |
| Irrelevant pairs with a filled color field | 1,763 query-product pairs | Original join and aggregation |
| Irrelevant distinct products | 2,225 products | Original join and aggregation |
| Irrelevant pairs with blank color | 21.61% | Original count divided by class total |
What changes when you compare the figures
| Comparison | Result | Calculation |
|---|---|---|
| Exact pairs with blank color | 13.8% | = 2037 / 14756 * 100 |
| Substitute pairs with blank color | 17.72% | = 1422 / 8024 * 100 |
| Complement pairs with blank color | 21.74% | = 95 / 437 * 100 |
| Irrelevant pairs with blank color | 21.61% | = 486 / 2249 * 100 |
| Substitute minus exact blank-field rate | 3.92 percentage points | = (1422 / 8024 - 2037 / 14756) * 100 |
| All color-term pairs with a blank field | 15.86% | = 4040 / 25466 * 100 |
| Exact and substitute pairs with a blank field | 15.18% | = (2037 + 1422) / (14756 + 8024) * 100 |
Methodology and sources
Select the US test rows in the large ESCI examples dataset whose query contains one of twelve basic English color terms as a whole word. Join examples to the product file using product ID and locale. Count null or trimmed empty product_color values, separately by the benchmark relevance label. All 25,466 pairs in the slice had one matching catalog record. A phrase can contain a color word for another reason. We count populated fields without deciding whether their values are accurate or match the query.
Download the source data (CSV)
Download the calculations (CSV)
Download the citation notes (CSV)
Related reading
AI shopping recommendation accuracy statistics
Ecommerce product review statistics and what shoppers use to decide
AI shopping price and availability error statistics
Sources and definitions
Amazon Science, Shopping Queries Dataset (ESCI)
The source data includes the population, period, denominator and limitations for every figure.