Can AI agents check out on their own? First pilot: 35 stores, 3 models
This is a pilot. On 2 November 2026, over two runs on the same 35 non-Shopify stores, Claude Sonnet 5.5 reached checkout in 32 of 68 attempts, Gemini 3.8 Flash reached checkout in 28 of 68 attempts and Qwen3-VL 235B reached checkout in 20 of 68 attempts (one store was not reachable and is left out). The 95% intervals overlap for every pair of models, so this pilot does not rank them.
Visa's Agentic Commerce Benchmark (5 November 2026) scores how well AI models decide what to buy; it stops before checkout. This pilot measures whether an agent reaches checkout on real stores. It stops before payment too.
- Claude Sonnet 5.5
- 32 of 68 reached checkout
- 47% · 95% CI 36%–59%
- Gemini 3.8 Flash
- 28 of 68 reached checkout
- 41% · 95% CI 30%–53%
- Qwen3-VL 235B
- 20 of 68 reached checkout
- 29% · 95% CI 20%–41%
2 runs on 2026-11-02 · method task-v1 · harness 9f3d75e, e0ba668 · 35 stores · 3 models · data CC-BY-4.0
What this is not
- Not a ranking. This is a pilot, not a ranking. Two runs; the pooled intervals still overlap for every model pair, so the pilot does not separate the models.
- A pilot: 35 non-Shopify stores, two runs, one reviewer. From one run to the next, 6 of 102 cells changed between checkout and no checkout.
- Shopify stores aren't included yet. 64 of the ARC-100 stores are on Shopify and were not run, so this is not an ARC-100 result.
- Models via the OpenRouter API, not the consumer apps. These are the models called through an API in ARC's browser harness, not any consumer chat or shopping app.
Reached checkout, per model
Hand-verified checkouts out of n, pooled over run 3 and run 4. The bar is the Wilson 95% interval of the pooled count; the dark line is the observed rate. Scale 0–100%.
| Model | Pooled | Pooled 95% CI | run 3 checkout | run 3 95% CI | run 4 checkout | run 4 95% CI |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 anthropic/claude-sonnet-5.5 | 32/68 | 36%–59% | 16/34 | 32%–63% | 16/34 | 32%–63% |
| Gemini 3.8 Flash google/gemini-3.8-flash | 28/68 | 30%–53% | 12/34 | 22%–52% | 16/34 | 32%–63% |
| Qwen3-VL 235B qwen/qwen3-vl-235b-a22b-instruct | 20/68 | 20%–41% | 9/34 | 15%–43% | 11/34 | 19%–49% |
n leaves out the store that was not reachable and counts real store blocks.
run 3 detail
| Model | Checkout | Cart or better | Blocked | Median steps | Median time | Cost | Cost per checkout |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 16/34 | 22/34 | 3 | 5 | 49 s | $2.774 | $0.173 |
| Gemini 3.8 Flash | 12/34 | 21/34 | 3 | 5 | 63.9 s | $1.052 | $0.088 |
| Qwen3-VL 235B | 9/34 | 19/34 | 3 | 6 | 55.6 s | $0.281 | $0.031 |
run 4 detail
| Model | Checkout | Cart or better | Blocked | Median steps | Median time | Cost | Cost per checkout |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 16/34 | 22/34 | 3 | 5 | 41.2 s | $2.516 | $0.157 |
| Gemini 3.8 Flash | 16/34 | 22/34 | 3 | 5 | 71.6 s | $1.010 | $0.063 |
| Qwen3-VL 235B | 11/34 | 21/34 | 3 | 7 | 71 s | $0.320 | $0.029 |
Medians leave out blocked and not-run cells. Cost is the model token cost through OpenRouter.
Every store, every model
Each cell shows run 3, then run 4.
| Store | Category | Claude Sonnet 5.5 | Gemini 3.8 Flash | Qwen3-VL 235B |
|---|---|---|---|---|
| vuoriclothing.com | apparel | ✅✅ | ❌✅ | ✅✅ |
| www.crewclothing.co.uk | apparel | ❌❌ | ❌❌ | ❌❌ |
| www.patagonia.com | apparel | ➖➖ | ➖➖ | ➖➖ |
| www.phase-eight.com | apparel | ❌❌ | ❌❌ | ❌❌ |
| theordinary.com | beauty | ✅✅ | ✅✅ | 🛒🛒 |
| www.drunkelephant.com | beauty | ✅✅ | ✅✅ | 🛒🛒 |
| north-america.beyerdynamic.com | electronics | ✅✅ | 🛒✅ | ✅✅ |
| powercity.ie | electronics | 🛒🛒 | 🛒🛒 | 🛒🛒 |
| www.alternate.de | electronics | ✅✅ | ✅✅ | ✅✅ |
| www.marshall.com | electronics | ✅✅ | ✅✅ | ✅✅ |
| www.dovesfarm.co.uk | food-drink | 🛒🛒 | 🛒🛒 | 🛒🛒 |
| www.palaisdesthes.com | food-drink | 🛒🛒 | 🛒🛒 | 🛒🛒 |
| www.cheaney.co.uk | footwear | ❌❌ | ✅✅ | ❌✅ |
| www.office.co.uk | footwear | ⛔⛔ | ⛔⛔ | ⛔⛔ |
| www.rmwilliams.com | footwear | ❌❌ | ❌❌ | ❌❌ |
| www.dunelm.com | home | ✅✅ | 🛒✅ | ❌❌ |
| www.angara.com | jewelry | ❌❌ | ❌❌ | ❌❌ |
| www.georgjensen.com | jewelry | 🛒🛒 | ✅✅ | ❌✅ |
| www.jared.com | jewelry | ❌❌ | ❌❌ | ❌❌ |
| www.queensmith.co.uk | jewelry | ✅✅ | ❌❌ | ✅✅ |
| www.bugaboo.com | kids | ✅✅ | 🛒🛒 | ✅✅ |
| www.childrensplace.com | kids | ⛔⛔ | ⛔⛔ | ⛔⛔ |
| www.cybex-online.com | kids | ✅✅ | ✅✅ | ✅✅ |
| www.nameit.com | kids | ❌❌ | ❌❌ | ❌❌ |
| paramo-clothing.com | outdoor | ❌❌ | ❌❌ | ❌❌ |
| www.bergans.com | outdoor | ❌❌ | ❌❌ | ❌❌ |
| www.cotswoldoutdoor.com | outdoor | 🛒🛒 | 🛒🛒 | 🛒🛒 |
| www.hardloop.co.uk | outdoor | 🛒🛒 | 🛒🛒 | 🛒🛒 |
| www.fressnapf.de | pets | ✅✅ | ✅✅ | 🛒🛒 |
| www.maxizoo.ie | pets | ✅✅ | ✅✅ | 🛒🛒 |
| www.poochandmutt.co.uk | pets | ✅✅ | ✅✅ | ✅✅ |
| drinklmnt.com | supplements | ✅✅ | ✅✅ | 🛒🛒 |
| huel.com | supplements | ✅✅ | 🛒✅ | ❌❌ |
| seed.com | supplements | ⛔⛔ | ⛔⛔ | ⛔⛔ |
| www.bulknutrients.com.au | supplements | ✅✅ | ✅✅ | ✅✅ |
- ✅ Checkout: reached a checkout step with the right product
- 🛒 Cart only: product in cart, no checkout step
- ❌ Failed: did not get the product into the cart
- ⛔ Blocked: store blocked the browser at the door
- ➖ Not run: store not reachable (maintenance page)
- Hand verdict where reviewed; other cells use the harness verdict. Which cells were reviewed is in each run file.
Method
- Task (task-v1): task-v1 (product page): open one fixed product URL, add that product at qty 1, reach the first checkout step.
- No-assist: no-assist: the agent gets the goal text only, no store-specific hints.
- Stop before payment: stop before payment: no payment control is pressed, no order is placed.
- No accounts: no accounts: no sign-in, no account created, no personal data entered.
- Limits: 16 steps, 180 s, $0.50 per attempt.
- What counts as checkout: A checkout step loaded (contact, address or e-mail step) with the right product at qty 1 visible in the summary or in the cart just before it. A sign-in or guest-choice page with no form does not count.
- Models: OpenRouter API, benchmark personas, not the consumer apps.
- Stores: ARC-100 v1 (frozen 2026-10-07). Shopify stores are behind a switch and were not run. This pilot covers the 35 non-Shopify stores of ARC-100 only.
- Hand review: Every harness checkout and every refused row (38), plus a seeded random sample of 30 other executed rows (random.Random(2026).sample over a pool of 57). One reviewer. Strict pass rule as in method.checkout_rule. Agreement with the harness on the required set (mandatory + random): run 3 67/68, 0 false passes; run 4 72/73, 0 false passes.
- Harness, run 3: commit 9f3d75e, run 2026-11-02 08:49-09:24 UTC. Total spend $3.97.
- Harness, run 4: commit e0ba668, run 2026-11-02 11:47-12:20 UTC. Total spend $3.69.
Known issues
- run 3: ARC safety filter refused two harmless guest-checkout buttons for Gemini (Dunelm, Bugaboo); at least one checkout lost to ARC, not the model.
- run 4: Only seed changed from run 3 (101, was 100). The guest-checkout guard fix (#465) let Gemini reach checkout on Dunelm; on Bugaboo Gemini stopped at the sign-in choice page on its own, so the guard was not exercised there.
- run 4: Maxi Zoo / Qwen: the harness scored product_only with the item in the basket (cart-level under-score, same row and cause as run 3).
Dataset
Download run 3 (results.json) · Download run 4 (results-run-4.json) (CC-BY-4.0). One immutable file per run: every figure on this page is read from them, and the pooled view is computed from both. They hold no personal data and no screenshots.
Cite this (CC-BY 4.0)
Free to reuse with attribution: ARC, arcreport.ai.
ARC (2026). ARC Agent Shopping Index pilot: Can AI agents check out on their own? First pilot: 35 stores, 3 models. Data as of 2 Nov 2026. https://www.arcreport.ai/benchmarks/agent-shopping/pilot-2026-11
@misc{arc_pilot_2026_11_20261102,
author = {{ARC}},
title = {ARC Agent Shopping Index pilot: Can AI agents check out on their own? First pilot: 35 stores, 3 models},
year = {2026},
note = {Data as of 2 Nov 2026. Licence CC-BY-4.0},
url = {https://www.arcreport.ai/benchmarks/agent-shopping/pilot-2026-11}
}