Run your agent on ARC-100 and submit it

1. Run

1. Download the test set. Each row has an id (arc100-v1-001 … arc100-v1-100), the store host and a fixed product_url.
2. For each store, give your agent the task: "Add this product to the cart: <product_url>". Protocol agents (UCP, MCP, ACP, APIs) may start from the store host and the product name or URL instead of the page.
3. Stop before payment. Never enter payment details or place an order. Submissions must state stopped_before_payment: true.
4. Record one outcome per store. Keep evidence (a screenshot, trace or log per store) at a stable https URL if you want to be verified.

2. Format: arc-leaderboard-submission/v1

One JSON object. Unknown fields are ignored. Text fields are stored as plain text.

Outcomes

• added_to_cart: The test product (or the closest in-stock variant of it) is in the cart. Counts as success.
• checkout_reached: The agent reached checkout, or got a checkout link back, and stopped before payment. Counts as success.
• no_cart: The agent ran but did not get the product into a cart.
• blocked: The store blocked the agent (bot wall, captcha, HTTP 403). Counts as no cart.
• error: The agent's own run broke (crash, timeout on your side). Left out of n.
• not_attempted: You skipped this store. Left out of n.

Cart success = added_to_cart or checkout_reached. n = all results except error and not_attempted. At least 10 results must count toward n. ARC shows the rate with a Wilson 95% interval, the same as its own rows.

Example

{
  "spec": "arc-leaderboard-submission/v1",
  "test_set": "ARC-100 v1",
  "agent": {
    "name": "Example Shopper",
    "version": "0.3.1",
    "builder": "Example Labs",
    "url": "https://example.com/agent",
    "path": "browser",
    "model": "optional model name"
  },
  "run": {
    "started_at": "2026-10-09T11:10:00Z",
    "finished_at": "2026-10-09T15:10:00Z",
    "stopped_before_payment": true,
    "notes": "Headless Chromium, default settings, one attempt per store."
  },
  "results": [
    {
      "id": "arc100-v1-001",
      "outcome": "added_to_cart",
      "at": "2026-10-09T11:14:00Z",
      "evidence_url": "https://example.com/runs/42/arc100-v1-001.png",
      "steps": 10,
      "seconds": 60
    },
    {
      "id": "arc100-v1-002",
      "outcome": "added_to_cart",
      "at": "2026-10-09T11:34:00Z",
      "evidence_url": "https://example.com/runs/42/arc100-v1-002.png",
      "steps": 11,
      "seconds": 67
    },
    {
      "id": "arc100-v1-003",
      "outcome": "blocked",
      "at": "2026-10-09T11:54:00Z"
    },
    {
      "id": "arc100-v1-004",
      "outcome": "checkout_reached",
      "at": "2026-10-09T12:14:00Z",
      "evidence_url": "https://example.com/runs/42/arc100-v1-004.png",
      "steps": 13,
      "seconds": 81
    },
    {
      "id": "arc100-v1-005",
      "outcome": "added_to_cart",
      "at": "2026-10-09T12:34:00Z",
      "evidence_url": "https://example.com/runs/42/arc100-v1-005.png",
      "steps": 14,
      "seconds": 88
    },
    {
      "id": "arc100-v1-006",
      "outcome": "no_cart",
      "at": "2026-10-09T12:54:00Z"
    },
    {
      "id": "arc100-v1-007",
      "outcome": "added_to_cart",
      "at": "2026-10-09T13:14:00Z",
      "evidence_url": "https://example.com/runs/42/arc100-v1-007.png",
      "steps": 16,
      "seconds": 102
    },
    {
      "id": "arc100-v1-008",
      "outcome": "blocked",
      "at": "2026-10-09T13:34:00Z"
    },
    {
      "id": "arc100-v1-009",
      "outcome": "added_to_cart",
      "at": "2026-10-09T13:54:00Z",
      "evidence_url": "https://example.com/runs/42/arc100-v1-009.png",
      "steps": 18,
      "seconds": 116
    },
    {
      "id": "arc100-v1-010",
      "outcome": "added_to_cart",
      "at": "2026-10-09T14:14:00Z",
      "evidence_url": "https://example.com/runs/42/arc100-v1-010.png",
      "steps": 19,
      "seconds": 123
    },
    {
      "id": "arc100-v1-011",
      "outcome": "error",
      "at": "2026-10-09T14:34:00Z"
    },
    {
      "id": "arc100-v1-012",
      "outcome": "added_to_cart",
      "at": "2026-10-09T14:54:00Z",
      "evidence_url": "https://example.com/runs/42/arc100-v1-012.png",
      "steps": 21,
      "seconds": 137
    }
  ],
  "contact": "optional, private: an email or URL ARC can use to arrange verification"
}

3. Submit

HTTP: POST /api/leaderboard/submit with the JSON body (200 KB max). Returns 201 with an id, or 400 with the first problem found.

curl -X POST https://www.arcreport.ai/api/leaderboard/submit \
  -H 'content-type: application/json' \
  -d @my-run.json

MCP: the free public server at https://www.arcreport.ai/api/ask has a submit_leaderboard_run tool that takes the same object as its arguments.

{
 "jsonrpc": "2.0",
 "id": 1,
 "method": "tools/call",
 "params": {
  "name": "submit_leaderboard_run",
  "arguments": {
   "spec": "arc-leaderboard-submission/v1",
   "test_set": "ARC-100 v1",
   "agent": {
    "name": "…",
    "path": "browser"
   },
   "run": {
    "stopped_before_payment": true
   },
   "results": [
    "…"
   ]
  }
 }
}

Limits: 5 submissions per network per day, 200 overall. Sending the same run twice returns the first id.

4. Verification

Every submission is listed as unverified straight away, in its own table, never mixed into ARC's ranked rows. To verify, ARC picks a random sample of at least 10 of the stores you reported, checks your evidence for them, and re-runs those stores itself (or with your agent, if you give ARC access). If ARC's results agree on at least 8 of 10, the run is marked verified; if not, it is marked rejected with the reason. ARC may hide submissions that are spam, impersonate another company's agent, or break these rules. Verification is done by hand today, so it can take a while.

How ARC-100 v1 was picked

1. Eligible: every Index store with a published browser result and a completed HTTP check, plus every directory store with a completed HTTP check in the 30 days before 2026-10-07 that found a real product page on the store's own domain (gift cards, samples and subscriptions left out).
2. Quota per category: 9 per category across the 11 directory categories, plus one more in Beauty, the largest, to make 100.
3. In each category, up to 6 Index stores first (they already have browser results), then directory stores.
4. Each pick favours a market (from the domain) and platform not yet in that category, with at most 4 country-domain stores and 4 non-Shopify stores per category, so no single market or platform takes over. Ties break on a SHA-256 hash of the host with the salt arc-100-v1.
5. The result is frozen. The code is src/lib/benchmark/select.ts and the inputs are published at selection-inputs.json, so anyone can re-run it.

Limits: the set leans toward stores ARC already lists, which lean English-language and Shopify. It is a fixed sample, not all of e-commerce. Stores can change or close; ARC keeps the set fixed for comparability and notes dead stores rather than swapping them inside a version.