Checkout Reply

AI shopping recommendation accuracy statistics

AgenticShop measures how well shopping agents meet buyer requirements. See what its results mean for compatibility, price and other checkout replies.

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Full article, downloads and prompt

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Download articleDownload AI source digest

Copy the complete article or download it for later.

About 17 min read
Updated:
Statistics
Learn about our Editorial policyLearn about our Research methodology

Sources checked 11 November 2026.

Contents

Quick answer

Finding the product and answering the buyer’s question are separate tests. In the 2026 EComAgentBench study of 662 shopping tasks, GPT-5.4 finished 99.4% and met the task accuracy criterion in 47.0%. In ShoppingBench, GPT-4.1 met strict criteria in 59.6% of 250 product finder tasks and 30.4% of 250 coupon and budget tasks. Verify the exact variant and the condition the shopper needs before sending a link.

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent · ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents

Key findings and sources

Task completion versus accuracy

In the 2026 EComAgentBench v3 study of 662 valid shopping tasks, GPT-5.4 finished 99.4% but scored 47.0% on task accuracy, defined as exact target match or satisfaction of every rubric. The measures share the benchmark task set but describe different outcomes.

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

Constraints from the opening and later turns

On EComAgentBench v3, GPT-5.4 satisfied 88.1% of rubric checks sourced from the opening query, 69.8% sourced from the shopper profile and 70.9% from clarification, using the 2026 benchmark’s scripted shopping scenarios.

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

Finding a product versus satisfying an offer

ShoppingBench v1 (2025) reports 59.6% strict success for GPT-4.1 on 250 product finder tests and 30.4% on 250 coupon and budget tests. The sandbox used real product records and different success conditions for the two intents.

ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents

Average relevance and complete success

For GPT-4.1 on ShoppingBench v1’s 250 product finder tests, the reported average product relevance was 83.6% and strict success was 59.6%. The former credits matching attributes while the latter requires a full relevance score.

ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents

AgenticShop’s 2026 product-attribution analysis reports failure rates of 24.76% for ChatGPT Search and 30.00% for Claude Sonnet 4 Search. The paper distinguishes invalid pages from irrelevant destinations in its benchmark.

AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping

An evaluator needs product evidence as well as a URL

In AgenticShop’s 50-instance human comparison, Pearson correlation with the human ratings was 0.7320 for the full evidence approach and 0.3792 for URL-only evaluation. Human-to-human correlation was 0.7736. Correlation measures agreement in scores.

AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping

Screenshots alone did not recover the same agreement

In AgenticShop Table 3, the grading approach with a URL and screenshots but no extracted product information correlated 0.4395 with human scores, compared with 0.7320 for the full approach over the 50-instance study.

AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping

How much the ranking depended on what the grader could see

In AgenticShop’s 50-instance evaluator study, the full evidence approach had Spearman correlation 0.7425 with human grades, versus 0.3885 for the URL-only approach. This is rank agreement on benchmark examples.

AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping

Shopping scores changed when the title was removed

In the displayed ShoppingBench cells, explicit-title scores exceeded attribute-specification scores by 2.46 points for ChatGPT Search and 1.49 for Claude Sonnet 4 Search. These differences compare benchmark task conditions.

AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping

GPT-5.4 and Opus traded leads across sources of shopper intent

In EComAgentBench Table 4, GPT-5.4 led Opus 4.6 by 7.5 points on query-visible constraints, while Opus led by 3.0 points on persona constraints. GPT-5.4 led by 2.3 points on clarification constraints.

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

Published rubric-satisfaction results for the 662-sample 2026 benchmark, separated by constraint source. Subtract the two displayed model scores within each source.

The visible-to-persona gap differed by 10.5 points

The query-visible minus persona rubric gap was 18.3 percentage points for GPT-5.4 and 7.8 for Opus 4.6 in EComAgentBench. The difference between the two gaps was 10.5 points.

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

By-source rubric satisfaction in EComAgentBench version 3. Compute each within-model difference, then compare the differences.

Compare the results

Strict success varies with the shopping task. Exact values are also in the study data.
Strict success varies with the shopping task.

A finished shopping task can still be wrong

EComAgentBench logged GPT-5.4 as finishing 99.4% of its 662 tasks, while its accuracy was 47.0%. Claude Opus 4.6 finished fewer tasks, at 84.6%, and scored higher on task accuracy, at 57.1%. GPT-5.4 scored higher on individual checks, 81.0% to 76.6%. The choice between those numbers depends on what the store actually wants to know.

A model can satisfy most of a shopper’s request and still fail the one requirement that determines whether the buyer can use the product. Consider a shopper who has the right style of water filter in their cart and asks whether it fits a particular housing. The finish event is that a reply was delivered. Product relevance is that the model named a matching type of filter. The useful outcome is a verified compatibility answer for the exact item or a question that resolves the remaining ambiguity.

Keep these events as separate columns in a checkout test. Record whether the agent replied, whether it identified the exact variant, whether it supported every material condition, and whether the shopper could use the path it offered. Counting only resolved-looking replies can hide false assurances. Counting only full completion can penalize a deliberate and useful request for a missing detail.

The hardest constraint may arrive after the first question

In EComAgentBench, GPT-5.4 satisfied 88.1% of checks derived from the opening query, 69.8% from the shopper profile and 70.9% from clarifications. The opening-to-profile gap was 18.3 percentage points. Claude Opus 4.6 showed the same broad ordering but a different pattern, at 80.6%, 72.8% and 68.6%. This was a controlled exercise with scripted information and an available tool for retrieving it.

A shopper may start with “does this fit?” and clarify the appliance model only after a question. The answer still has to use the SKU in the cart and the earlier size or market constraint. For a useful test, prepare two near-matches. Give one the exact required fitting but the wrong dimensions. Give the other the right dimensions but a fitting that only looks similar. Record whether the reply asks for the one detail that distinguishes them and uses that answer before offering a replacement.

For a hard requirement, such as a fitting, date or total price ceiling, require an explicit check. For a soft preference, such as color, explain which alternatives are available. A clarification can improve decision quality without turning every checkout reply into an interrogation.

A product match gets less useful when the offer has conditions

In ShoppingBench, GPT-4.1 reached 59.6% strict success on 250 product finder instructions. Its average relevance there was 83.6%. For 250 coupon and budget instructions, strict success fell to 30.4%, while average relevance was 72.8%. The benchmark used different pass conditions by intent. The comparison shows where a relevant result still fell short of the complete task in that sandbox.

A shopper asking whether a coupon will keep the order under a ceiling needs the payable amount for their actual cart. The eligible variant, minimum spend, region, shipping and discount combination can change the answer. Check the final route and total in the state the shopper is in. If taxes or a region-dependent fee are still unknown, explain what remains to be calculated rather than giving a guarantee.

The counterexample is a lookup with a verified identifier and one static property. It may be safe to resolve in one reply. The deeper checks earn their time when an exact item can satisfy part of the question while the selected offer fails another part.

Verify where the link leads before treating the recommendation as grounded

AgenticShop’s attribution analysis reports a 24.76% failure rate for ChatGPT Search and 30.00% for Claude Sonnet 4 Search in the evaluated product curation runs. The paper describes both invalid destinations and irrelevant destinations such as search listings. A URL can resolve successfully and still be useless for the shopper’s decision.

On a merchant’s own recovery path, open the destination with the variant selected and check whether the relevant property and availability match what you told the shopper. Keep a separate control for a live link that lands on the wrong product, and for a correct product link that drops the selected size or price. One passing network response cannot certify a usable recommendation.

These studies make a practical scorecard possible. Track exact item identification, verified conditions, usable destination and finished checkout separately. When the evidence is incomplete, measure the quality of the clarification or handoff as well. That protects the shopper’s ability to act without treating a benchmark model ranking as a store-level revenue forecast.

The evidence shown to the grader changes what it can verify

AgenticShop tested its evaluator on 50 instances that two people independently annotated. Pearson correlation with the human scores was 0.7320 when the evaluator had extracted product information and screenshots, 0.4395 without extracted product information and 0.3792 when it had only the URL. The two people correlated at 0.7736. Those are score correlations, not shares of correctly answered checkout questions.

A merchant scoring its own assistant should preserve the item, the version of the product page or offer the answer relied on, and the question itself. A grader that only sees the link may reward a live URL even though the page displays the wrong size, or miss a correct response because the data was behind a selected variant. Keep a human-reviewed sample where a wrong answer costs a shopper time or money.

If the product evidence cannot be checked, give that case a separate unresolved outcome. That keeps the accuracy denominator explicit and gives the team a reason to improve the catalog, the tool or the escalation path instead of treating grader agreement as a product capability.

Rank agreement is useful when deciding which failures to fix first

The AgenticShop evaluator paper reports a second way to read the same 50 examples. Spearman rank correlation with the human graders was 0.7425 for the full approach, 0.4405 without extracted product information and 0.3885 for the URL-only setup. The full approach led by 0.354 correlation points over URL-only.

Pearson correlation measures how closely scores vary together. Spearman measures whether their ordering agrees. A merchant using automated evaluation to prioritize the most severe failures cares whether the same cases rise to the top. A high overall correlation still does not establish agreement on individual payment, fit or shipping mistakes.

In your test, agree on the critical cases with a human before tuning a grader. Keep a short review list of severe disagreements, such as the grader accepting a valid-looking link while the exact product is wrong. Compare rank agreement and the recall of critical defects. Let the grader handle broad triage where it works and keep a human decision for the cases where the cost of a false pass is high. Retain the same product snapshot for adjudication so a later price or inventory update cannot rewrite the evidence.

Ask whether the product name is doing the work

An assistant can look good when a prompt contains the exact title from a catalog. A customer often knows the feature they need and only later learns which product has it. Compare the two question forms on the same product set before using a title-led evaluation as evidence that an assistant can sell from attributes.

In the ShoppingBench rows shown here, moving from an explicit title to attribute specifications changes the reported score by 2.46 points for ChatGPT Search and 1.49 for Claude Sonnet 4 Search. That is a 0.97-point difference between the wording gaps. These are score differences in published cells. They provide a reason to test phrasing in your own catalog, with no estimate of checkout conversion or a reliable difference between the two systems.

Write each shopper task in two forms. Start one with the exact title and the other with the size, material or use case. Use the same exact eligible product set, store data and scoring rule. If attribute-led requests lose accuracy, inspect product titles, attribute coverage and retrieval together. If both versions fail on the same item, check that the store exposes the relevant field before changing the assistant prompt.

The answer can look complete while losing the shopper’s hidden constraint

The ranking depends on what you ask a reviewer to check. In EComAgentBench, GPT-5.4 scored 7.5 points above Opus 4.6 on query-visible criteria and 2.3 points above it on criteria learned through clarification. Opus scored 3.0 points higher on persona criteria. The same published task set can therefore send a reviewer to different models, depending on whether the hard part of the job is recognizing the visible wording, retrieving a stored preference or using an answer learned during the exchange.

Compare the deficits within each model before shopping for another model. GPT-5.4’s query-to-persona gap was 18.3 points, compared with 7.8 points for Opus. Changing the mix of tasks changes what that difference means. If most requests are inventory lookups for a named product, the visible-constraint score may carry more weight. If the shopper expects a previously established allergy, fit preference or delivery limit to carry over, the persona result is closer to the real risk.

Build a check around the first place the recommendation could go wrong

Take one product family from your own catalog and write three parallel tasks. In the first, the shopper states the relevant requirement immediately. In the second, the requirement is already known from the conversation and appears outside the final question. In the third, the shopper only supplies it after the assistant asks. Keep the available variants and required answer fixed so the comparison tests where the requirement came from.

Grade the final recommendation for each source separately. Retain the resolved product and variant, the decisive wording and the store record behind the answer. If an answer can only be verified by opening the variant link, record that inspection before counting the recommendation as correct. When every listed variant is wrong, a useful response may instead explain the constraint and offer to check a genuinely different option. Let that case have its own outcome so it does not distort the exact-match score.

The shopper already knows what they want to buy

A shopper leaves a pair of shoes in checkout and replies, “Will these fit my orthotics?” They may have found the right style and color hours ago. What matters now is the fit question. Knowing the title gets the assistant to the right product family. The answer depends on the selected variant, the space in the shoe, whether the insole comes out and what the store can verify.

You can see a version of that problem in AgenticShop. Its researchers gave agents shopping queries and profiles, then scored how many checklist criteria the suggested products met. The criteria included things like technical requirements, budget and purchase preferences. In an abandoned checkout conversation, a well-matched product name can make the reply sound grounded before any of the conditions that matter to the buyer have been checked.

Naming the product only moved the checklist score a little

In the benchmark's target-finding tasks, ChatGPT Search scored 37.81% when it received an explicit title, 35.35% with an attribute description and 34.79% with a brand and category. Claude Sonnet 4 Search scored 37.93%, 36.44% and 34.96% in the same three columns. The explicit title improved the checklist score over attributes by 2.46 percentage points for ChatGPT Search and 1.49 for Claude Sonnet 4 Search.

Read those numbers as criteria scores. The benchmark averages judgments for individual product and criterion pairs, so a product can satisfy several criteria and fail one that would rule it out for that shopper. The scores are also from open-web agents on curated grocery tasks. An assistant with access to a store's own product data faces a different setup. The comparison tells you which distinction to keep in a merchant test, even when the underlying systems differ.

Give one answer when the evidence supports it

Take a shopper who asks, “Is it under $80 delivered?” The item in their checkout costs $72. The useful reply depends on their delivery destination, the shipping charge and whether the figure they mean includes tax. If the store can verify a complete total for their checkout, give that total and explain what is included. If the location is missing, ask for the detail that determines shipping or direct the shopper to the verified total in checkout.

Use the same rule for compatibility. If the catalog confirms that the mount fits the exact model and year the customer gave, cite the relevant specification in plain language. When the fit depends on a year they haven't supplied, ask for it. When the source never specifies their model, say what the store can confirm and give them a way to check. This makes the answer useful even if the shopper needs one more step.

Give a required condition its own score

Put three things in your evaluation record. Record the product and variant in checkout, write the shopper's exact condition, and attach the source that would let an answer pass. Add a case where the expected result is to ask a question. Add another where the expected result is to say that the requested condition cannot be confirmed. Read each response for the factual claim it asks the shopper to trust.

Track item identity, requirement satisfaction and the quality of the next step separately. A product title match can pass while an all-in price answer fails. A response can also handle a missing detail correctly and give the shopper a short path back to checkout. This split tells you whether you need better product data, a way to read current checkout conditions or a better way to handle a question the store cannot answer yet.

Study data

AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping
Measure Value Context
ChatGPT Search, explicit title 37.81% of checklist criteria Displayed checklist curation score is 37.81.
ChatGPT Search, attribute specifications 35.35% of checklist criteria Displayed checklist curation score is 35.35.
ChatGPT Search, brand and category 34.79% of checklist criteria Displayed checklist curation score is 34.79.
Claude Sonnet 4 Search, explicit title 37.93% of checklist criteria Displayed checklist curation score is 37.93.
Claude Sonnet 4 Search, attribute specifications 36.44% of checklist criteria Displayed checklist curation score is 36.44.
Claude Sonnet 4 Search, brand and category 34.96% of checklist criteria Displayed checklist curation score is 34.96.
ChatGPT Search, product-attribution failures 24.76% Failure rate as reported in the paper’s product-attribution analysis
Claude Sonnet 4 Search, product-attribution failures 30.0% Failure rate as reported in the paper’s product-attribution analysis
Full grading approach, Pearson correlation with humans 0.732 correlation Grader receives extracted product information plus screenshots
Grading without product information, Pearson correlation 0.4395 correlation URL and screenshots without extracted product information
URL-only grading, Pearson correlation 0.3792 correlation URL without screenshots or extracted product information
Human-to-human Pearson correlation 0.7736 correlation Reference agreement between the two human evaluators
Full grading approach, Spearman correlation with humans 0.7425 correlation Rank correlation with human grades
Grading without product information, Spearman correlation 0.4405 correlation Rank correlation with human grades
URL-only grading, Spearman correlation 0.3885 correlation Rank correlation with human grades
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Measure Value Context
Claude Opus 4.6, task accuracy 57.1% Exact product match or full rubric satisfaction
Claude Opus 4.6, rubric satisfaction 76.6% Share of individual checks met
Claude Opus 4.6, finished tasks 84.6% Share recorded as finished
GPT-5.4, task accuracy 47.0% Exact product match or full rubric satisfaction
GPT-5.4, rubric satisfaction 81.0% Share of individual checks met
GPT-5.4, finished tasks 99.4% Share recorded as finished
Kimi K2.6, task accuracy 46.4% Exact product match or full rubric satisfaction
Kimi K2.6, rubric satisfaction 73.4% Share of individual checks met
Kimi K2.6, finished tasks 88.7% Share recorded as finished
GPT-5.4, requirements from opening query 88.1% Individual rubric satisfaction; results use a different base from task accuracy
GPT-5.4, requirements from shopper profile 69.8% Individual rubric satisfaction; results use a different base from task accuracy
GPT-5.4, requirements from answer to a clarification 70.9% Individual rubric satisfaction; results use a different base from task accuracy
Claude Opus 4.6, requirements from opening query 80.6% Individual rubric satisfaction; results use a different base from task accuracy
Claude Opus 4.6, requirements from shopper profile 72.8% Individual rubric satisfaction; results use a different base from task accuracy
Claude Opus 4.6, requirements from answer to a clarification 68.6% Individual rubric satisfaction; results use a different base from task accuracy
GPT-5.4, query-visible constraints met 88.1% Published by-source rubric satisfaction
GPT-5.4, persona constraints met 69.8% Published by-source rubric satisfaction
GPT-5.4, clarification constraints met 70.9% Published by-source rubric satisfaction
Opus 4.6, query-visible constraints met 80.6% Published by-source rubric satisfaction
Opus 4.6, persona constraints met 72.8% Published by-source rubric satisfaction
Opus 4.6, clarification constraints met 68.6% Published by-source rubric satisfaction
ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents
Measure Value Context
GPT-4.1, product finder absolute success 59.6% Full attribute relevance for product finder tasks
GPT-4.1, product finder average relevance 83.6% Average fraction of product relevance conditions met
GPT-4.1, shopping knowledge absolute success 62.0% Strict success under the knowledge task rules
GPT-4.1, same-seller multi-product absolute success 46.4% Strict success under the same-seller multi-product task rules
GPT-4.1, coupon and budget absolute success 30.4% Strict success under the coupon and budget task rules
GPT-4.1, coupon and budget average relevance 72.8% Average product relevance in coupon and budget tasks
Claude Sonnet 4, coupon and budget absolute success 24.0% Strict success under the same coupon and budget task rules

What changes when you compare the figures

Calculations from the published values above.
Comparison Result Calculation
ChatGPT Search checklist difference, explicit title versus attributes 2.46 percentage points = Explicit title − attribute specifications
Claude Sonnet 4 Search checklist difference, explicit title versus attributes 1.49 percentage points = Explicit title − attribute specifications
GPT-5.4 difference between finished tasks and task accuracy 52.4 percentage points = 99.4 − 47.0
GPT-5.4 opening-query versus profile rubric satisfaction 18.3 percentage points = 88.1 − 69.8
GPT-5.4 opening-query versus clarification rubric satisfaction 17.2 percentage points = 88.1 − 70.9
GPT-4.1 product finder relevance versus strict success 24 percentage points = 83.6 − 59.6
GPT-4.1 product finder versus coupon and budget strict success 29.2 percentage points = 59.6 − 30.4
AgenticShop Claude versus ChatGPT attribution failure gap 5.24 percentage points = 30.00 − 24.76
Full grader minus URL-only Pearson correlation 0.3528 correlation points = .7320 − .3792
Full grader minus screenshot approach without product information 0.2925 correlation points = .7320 − .4395
Full grader minus URL-only Spearman correlation 0.354 correlation points = .7425 - .3885
ChatGPT Search, explicit title to attribute-only score change 2.46 score points = 37.81 - 35.35
Claude Sonnet 4 Search, explicit title to attribute-only score change 1.49 score points = 37.93 - 36.44
Difference between the two wording gaps 0.97 score points = 2.46 - 1.49
Visible intent advantage for GPT-5.4 7.5 percentage points = 88.1 - 80.6
Persona advantage for Opus 4.6 3 percentage points = 72.8 - 69.8
Clarification advantage for GPT-5.4 2.3 percentage points = 70.9 - 68.6
Difference between query-to-persona gaps 10.5 percentage points = (88.1 - 69.8) - (80.6 - 72.8)

Methodology and sources

Figures are from AgenticShop Table 4, Target Finding, and compare its three query forms for two selected configurations. The benchmark builds shopping profiles and checks curated products against a checklist using evidence extracted from online product pages and an LLM judge. Point differences were calculated from displayed scores. The unit is a benchmark checklist average. A store would need its own cases to measure checkout compatibility answers or conversion.

We transcribed the named model rows in AgenticShop Table 6, EComAgentBench Tables 3 and 4, and ShoppingBench Table 1 and its held-out sample description. Each percentage retains the original unit: finished tasks, exact-or-full task accuracy, fraction of requirements met, product relevance, or attribution failure. Differences subtract published rounded percentages only. The ShoppingBench intent comparison describes tests with different strict criteria. It does not isolate how hard a single extra requirement is. The three environments and model versions differ, so we do not pool their scores or rank models across studies.

Download the source data (CSV)

Download the calculations (CSV)

Download the citation notes (CSV)

Download the evidence behind the new findings (CSV)

Ecommerce product review statistics and what shoppers use to decide

AI shopping price and availability error statistics

Product search statistics on exact matches and substitutes

Browse checkout recovery research

Sources and definitions

AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping

EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents

AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping

The source data includes the population, period, denominator and limitations for every figure.

About the publisher

Checkout Reply

Research and practical guides for DTC merchants.