Versioning and reproducibility

The rules every ARC benchmark follows, then the method for each one.

Rules for every benchmark

Every figure has a date, a denominator and a version. A percentage is never published without its n and the snapshot it came from.
Not measured means no number. Cells are labelled Measured, Inferred or Not measured. A planned benchmark shows no figure.
Rates carry a 95% Wilson interval. Results with fewer than 10 attempts are listed but not ranked.
Stop before payment. No agent ever enters payment details, places an order or creates an account.
Polite collection. The honest ARC user agent, robots.txt and opt-outs respected, one session per store at a time, no CAPTCHA solving. A challenge counts as blocked.
Open data. Every headline can be recomputed from files published under CC-BY-4.0. If it can't, it isn't published.

Versions and editions

Each benchmark has an id and a version: asi-v1.0. A major version changes the test set or the pass condition, so results are not comparable and a new trend line starts. A minor version fixes a bug or wording and keeps comparability.
Editions are dated by the snapshot they read, for example 2026-10-30. A correction to a published edition gets a new id (2026-10-30.r2). Old editions stay downloadable under /data/<version>/.
Test sets are frozen. A store that closes or opts out stays in the set as not run and leaves the denominator.
Every method change is logged in the changelog before the edition that uses it.

Why there is no single ARC score

The benchmarks count different units: attempts, stores, protocol stages, channel signals and claims. Averaging them would hide the gaps people need to see, such as stores that publish a valid protocol profile but don't return a checkout link. ARC reports one figure per benchmark and quotes them side by side, never blended.

ARC Agent Shopping Index (beta, asi-v0.1)

Which shopping agents can get a product into the cart on real stores, and at what cost?

Unit: Attempt: one agent, one store, one fixed product URL. Reach-checkout rate with a 95% interval, plus cost and steps per success.
Method: ARC's browser shopper is given one task per store: add the fixed product to the cart. Success is the product in the cart, or a checkout URL or session reached. The agent stops before payment: it never types card data, never places an order and never creates an account. Protocol lanes (UCP create_cart, storefront MCP cart) are scored on the same board with no model cost.
Sample: ARC-100 v1 (100 stores, frozen 7 Oct 2026) is the headline set. Today's figure counts every store in the open snapshot with a browser cart test.
Cadence: Weekly (planned for v1)
Reproduce or submit: Download the ARC-100 CSV, run your agent, and send the run to /api/leaderboard/submit (or MCP submit_leaderboard_run). Runs show as unverified until ARC re-runs a sample.

ARC Store Readiness Index (beta, sri-v0.1)

How ready is each store, and each platform, category and market, for AI shopping agents?

Unit: Store. Pass, fail or not measured on each readiness measure, grouped into find, understand and reach checkout.
Method: Public HTTP reads only: the store's catalog or agent endpoint, robots.txt, product-page structured data (price, stock, variants), merchant policies, guest checkout. Each measure is pass, fail or not measured. No opaque 0–100 blend in v1.
Sample: Every store in the Index with a completed check. Opted-out stores are excluded. Segment figures need n ≥ 30 and always show the platform split.
Cadence: About weekly, with each open snapshot
Reproduce or submit: Each row of store_answers.csv links to its evidence page. Store owners can re-check at /scan and dispute a result at /corrections.

ARC Protocol Conformance Index (live, pci-v1.0)

For UCP, ACP, MCP, WebMCP and x402: how many stores declare it, how many are valid, and at how many does an agent get a checkout link back?

Unit: Store × protocol × stage (declared, valid, usable, transacts), each with an evidence label.
Method: Four stages per protocol: declared (the store publishes it), valid (it passes the spec checks), usable (a tool or catalog answered), returns checkout link (a cart or checkout link came back, stopped before payment; the matrix column id is transacts). Each cell is labelled Measured, Inferred or Not measured.
Sample: Same stores as Store Readiness. The denominator is completed checks, not attempted ones.
Cadence: With each open snapshot
Reproduce or submit: protocol_matrix.csv carries the definition of every cell. Anyone can GET /.well-known/ucp and call create_cart on the tested product.

ARC Channel Reach Index (planned, cri-v0)

Through which agent channels (Google AI Mode, Copilot, Perplexity, Amazon, ChatGPT) can merchants be reached, and does in-chat checkout work?

Unit: A public channel signal per store (census), plus a monthly sampled panel of shopping queries per assistant (directional).
Method: Two lanes, never mixed: a census of public channel signals (today only /.well-known/ucp), and a monthly panel of fixed shopping queries per assistant, labelled sampled and directional. Closed feeds are reported as vendor claims with source and date, marked unverified.
Sample: Census: every tracked domain. Panel: a frozen, published query set.
Cadence: Census weekly, panel monthly (planned)
Reproduce or submit: The query set and raw panel answers (assistant, date, region) are published with each run.

ARC Product Truth Index (beta, pti-v0.1)

When an AI agent tells a shopper the price, stock or variants, does it match the store's own page?

Unit: Claim: one price, stock, variant or policy statement, scored match, mismatch, partial or couldn't verify.
Method: A model is asked about a product. Each price, stock, variant or policy claim is checked against the store's own page: the quote must come from the product (not navigation or related items), and price and stock prefer JSON-LD, product JSON or the buy box. Ambiguous cases are 'couldn't verify' and are reported separately, never counted as mismatches.
Sample: Today: stores scanned with the fact-check lane (one model, via API, not a consumer app). v1: the ARC-100 product URLs × the Agent Shopping model panel.
Cadence: Weekly with the Agent Shopping Index (planned)
Reproduce or submit: Each published claim carries the model's quote, the page excerpt, the source field and the time, so a reader can check the verdict by hand.