AI agents could read the pricing and reach a salesperson. Signing up is where they were stopped.
We gave one AI browsing agent three commercial jobs on 30 B2B SaaS websites, and ran every job twice. It completed 85.6% of valid journeys, and 93.1% of the journeys where the site did not refuse it access. Of its 25 failures, 14 — 56% — were the site refusing the agent access, concentrated at the point of signing up.
30 SaaS companies · 180 planned journeys · 3 tasks · 2 runs per task
Where the funnel inverted
| Company | Read the pricing | Start self-service signup | Reach sales |
|---|---|---|---|
| Airtable | Completed both runs | Failed both runs | Completed both runs |
| Cloudflare | Completed both runs | Failed both runs | Completed both runs |
| DigitalOcean | Completed both runs | Failed both runs | Completed both runs |
| Webflow | One of two runs | Failed both runs | Completed both runs |
- Completed both runs
- Failed both runs
- One of two runs
- Not measurable
Signup-journey evidence
What we found
Three findings carry the study. Each is stated with the exact numerator and denominator it comes from, and every one of them is reproducible from the published evidence.
- 01
Most commercial journeys completed, and the failures were not spread evenly
The agent reached its goal in 149 of 174 valid runs — 85.6%, with a 95% interval of 79.7% to 90.1%. Reading pricing succeeded in 49 of 58 valid runs and reaching sales in 53 of 58. Self-service signup was the weakest journey at 47 of 58, and the failures there were overwhelmingly one thing rather than many.
- 02
The largest single cause of failure was the site refusing access, not the agent getting lost
14 of 25 failures — 56% — were the audited site refusing the agent access to its own commercial route, through a bot challenge or verification page. Our agent's own behaviour accounted for 10 of 25: 8 losing the route or exhausting the 20-turn budget, and 2 reaching the right page and reporting the wrong answer. The one remaining failure was a site's own primary button reporting a successful click without navigating anywhere.
- 03
On four sites the agent could reach a salesperson but not the product
On Airtable, Cloudflare, DigitalOcean and Webflow the route to contact sales completed in both runs while the self-service signup route failed in both runs. The slower path, the one that needs a human on the other end, worked. The automatic one did not.
Why this matters
An AI agent that can read your website is not the same as an AI agent that can buy from it. Reading is a content problem: is the pricing published, is it parseable, is the page reachable. Converting is a path problem — a sequence of real clicks through real controls, ending at a form, a signup or a checkout.
In this benchmark the reading was rarely the hard part. The agent found official pricing and identified a paid entry price in 49 of 58 valid runs, and found the route to contact sales in 53 of 58. Failures concentrated at the moment a journey stops being reading and becomes an action, and more than half of them were not the agent getting lost — they were the site refusing it access.
That asymmetry has a commercial shape. A buyer who cannot find your pricing can ask a salesperson instead. An agent stopped at signup is stopped at the one step that converts without a human being involved at all.
The failures were reproducible, not random
Every company-task pair was run twice, independently, so repeatability is measured rather than assumed. 72 of 87 resolvable pairs succeeded both times. 10 failed both times. Only 5 — 5.8% — produced one success and one failure.
That matters for how much weight a single failure can carry. Every one of the seven pairs where the site refused access was refused in both runs: those are reproducible site behaviours on the measured date, not one unlucky attempt.
One hypothesis we expected to confirm did not survive. Repeatable completion, 82.8%, is only about three points below first-run completion, 86.2%, and the two intervals overlap heavily. On this evidence we cannot claim agents are markedly less reliable on a second attempt than on a first.
Three completion rates, three different questions
- Single-run completion149 / 17485.6%
If an agent attempts this journey once, how often does it finish? This is the headline figure and the one most comparable to a real visit.
95% CI 79.7% – 90.1%
- Repeatable completion72 / 8782.8%
If an agent attempts the same journey twice, how often does it finish both times? A stricter test, and the one that tells you a result was not luck.
95% CI 73.5% – 89.3%
- Completion where access was not refused149 / 16093.1%
When the site did not refuse the agent access, how often did the journey finish? This isolates how much of the shortfall is access policy rather than navigation.
95% CI 88.1% – 96.1%
The third figure is not the 'real' success rate with an inconvenience removed. A site refusing an agent at signup is a commercial outcome, not measurement noise: the journey did not complete. The figure exists to separate two causes, not to replace the headline.
The three journeys
Each company was given the same three jobs, phrased as a customer would phrase them and frozen before any run executed. Each job stops one action short of the irreversible step.
- Read the pricing49 / 58 · 84.5%
- Find the official pricing and identify the lowest clearly non-free paid entry price, with its billing unit and cadence. Judged on plan, unit and cadence rather than on a currency amount.
- Start self-service signup47 / 58 · 81.0%
- Find the official self-service route and follow it until the next action would create an account, submit personal information or initiate payment. Stop there.
- Reach sales53 / 58 · 91.4%
- Find the official route to contact sales or request a demo and follow it until the next action would submit a form or send customer information. Stop there.
Pair outcomes by journey
- Read the pricing23 / 29 succeeded twice
- 23Completed both runs
- 3One of two runs
- 3Failed both runs
- 1Not measurable
- Start self-service signup23 / 29 succeeded twice
- 23Completed both runs
- 1One of two runs
- 5Failed both runs
- 1Not measurable
- Reach sales26 / 29 succeeded twice
- 26Completed both runs
- 1One of two runs
- 2Failed both runs
- 1Not measurable
Each bar is 30 company-task pairs: 30 companies, one pair each, two runs per pair.
What actually went wrong
All 25 valid failures, grouped by the cause assigned during adjudication. Each failure was classified by reading its full step trace against the frozen task definition, not by its score.
All 25 failures by adjudicated cause
- ACCESS_BLOCKED14 / 25 · 56%
The site returned a bot challenge or verification page on its own commercial route, and the agent could go no further. Observed on 5 of the 29 measurable companies. No intent is implied: a managed bot challenge is not necessarily a deliberate decision about AI agents.
- STEP_TIMEOUT6 / 25 · 24%
The agent exhausted its 20-turn budget without reporting an answer. In at least 3 of these the answer was already on screen when the budget ran out.
- CONTENT_FAILURE2 / 25 · 8%
The agent reached the right page and reported the wrong thing — in both cases a correct price attached to the wrong plan name.
- NAVIGATION_FAILURE2 / 25 · 8%
The agent never reached the required destination, having guessed at URLs instead of using a link the page offered.
- CONTROL_FAILURE1 / 25 · 4%
The site's own primary call to action reported a successful click without navigating anywhere, and the agent did not recover.
Adjudication assigns each failure to the side it originated on. Site-facing means the site's own behaviour stopped a journey it publishes as self-service; agent behaviour means our agent failed on a site that offered it a working path. The split is roughly three to two, and we report both rather than only the half that flatters the instrument.
Named evidence
The examples below are named because the evidence is public and reproducible. Each describes the measured journeys and nothing beyond them: a specific route, on a specific date, from one cloud network, twice. None of it is a claim about how these sites treat all AI agents, or about how they behave today.
- AirtableSignupFailed both runs
Following Airtable's own signup route to airtable.com/signup returned a verification page in both measured runs. The agent stopped there and did not reach a signup form.
- Route
- airtable.com/signup
- Page observed
- “Verify it’s you — Airtable”
- CloudflareSignupFailed both runs
Both runs followed Cloudflare's primary call to action to dash.cloudflare.com/sign-up and both observed an interstitial challenge page rather than a signup form.
- Route
- dash.cloudflare.com/sign-up
- Page observed
- “Just a moment...”
- DigitalOceanSignupFailed both runs
Both runs reached cloud.digitalocean.com/registrations/new and both observed an access-denied page from a bot protection service rather than a registration form.
- Route
- cloud.digitalocean.com/registrations/new
- Page observed
- “Attention Required! | Cloudflare”
- WebflowSignupFailed both runs
Both runs reached webflow.com/signup and both observed a page stating that access had been denied. Webflow's pricing page carried no such challenge in either run, and the route to contact sales completed in both.
- Route
- webflow.com/signup
- Page observed
- “Access to this page has been denied”
- MakePrice · Signup · SalesFailed both runs
All six runs on make.com — two per journey — observed an interstitial challenge at the site's entry page. No site content was observable, so nothing beyond the entry behaviour was measured.
- Route
- www.make.com/en
- Page observed
- “Just a moment...”
- PostHogSignupOne of two runs
In both runs the agent clicked PostHog's primary 'Get started' calls to action, and in both runs the click reported success without the page changing. One run recovered by typing the signup URL directly; the other finished on the homepage.
- Route
- posthog.com
A score is a summary, not a verdict
Our own product gives each run a 0–100 score. Across the benchmark that score agreed with independent adjudication on 166 of 174 valid runs, which is a good agreement rate. The disagreements are the interesting part, because they all run the same way.
Seven of the 25 failed journeys scored 90 or above. Two scored 100. Not one completed journey was scored below 70 — so every disagreement in this dataset is a score that was too kind, never one that was too harsh.
The sharpest case is a single pair. Two runs followed the same call to action to the same signup URL and both were met by the same challenge page. The journey outcome is identical and reproducible. The scores were 50 and 98.
What follows from that
Not that scores are useless — this one agreed with adjudication 95% of the time and is a genuinely useful summary of how a run went. What follows is narrower and more practical: a score is not sufficient as the verdict on whether an agent converted. Only the outcome of the journey answers that, and in the seven cases where the two disagreed above 90 the score was wrong about the thing a buyer would care about most.
Don't score whether your website looks agent-ready. Test whether an agent can actually convert.
This is the product implication we draw from the discrepancy above, not a proven general law. It rests on 8 disagreements in 174 runs of one agent on 29 sites.
One company was excluded, and the reason is ours
One of the 30 selected companies produced no valid measurement. Our own page-observation step exceeded its internal deadline on that site in all 12 attempts, consuming each run's entire time budget before the agent could act. No other company in the benchmark produced a single such failure. This is a defect in our measuring instrument, not a result about that company: nothing was measured about its website, and no conclusion about it is drawn anywhere in D-001. The cohort is therefore reported as 30 selected, 29 yielding comparable measurements — and the figures with that company counted as a failure are published in the analysis as well, so the choice can be checked rather than taken on trust.
All 30 selected companies
| Company | Read the pricing | Start self-service signup | Reach sales |
|---|---|---|---|
| Airtable | Completed both runs | Failed both runs | Completed both runs |
| Asana | Completed both runs | Completed both runs | Completed both runs |
| Calendly | Completed both runs | Completed both runs | Completed both runs |
| ClickUp | Completed both runs | Completed both runs | Completed both runs |
| Cloudflare | Completed both runs | Failed both runs | Completed both runs |
| Datadog | Completed both runs | Completed both runs | Completed both runs |
| DigitalOcean | Completed both runs | Failed both runs | Completed both runs |
| Figma | Completed both runs | Completed both runs | Completed both runs |
| Framer | Completed both runs | Completed both runs | Completed both runs |
| GitHub | Completed both runs | Completed both runs | Completed both runs |
| GitLab* | Not measurable | Not measurable | Not measurable |
| HubSpot | Failed both runs | Completed both runs | Completed both runs |
| Intercom | Failed both runs | Completed both runs | Completed both runs |
| Klaviyo | Completed both runs | Completed both runs | Completed both runs |
| Linear | Completed both runs | Completed both runs | Completed both runs |
| Loom | Completed both runs | Completed both runs | Completed both runs |
| Mailchimp | Completed both runs | Completed both runs | Completed both runs |
| Make | Failed both runs | Failed both runs | Failed both runs |
| Miro | Completed both runs | Completed both runs | Failed both runs |
| monday.com | Completed both runs | Completed both runs | Completed both runs |
| Notion | Completed both runs | Completed both runs | Completed both runs |
| PostHog | One of two runs | One of two runs | Completed both runs |
| Render | Completed both runs | Completed both runs | Completed both runs |
| Sentry | One of two runs | Completed both runs | Completed both runs |
| Slack | Completed both runs | Completed both runs | Completed both runs |
| Supabase | Completed both runs | Completed both runs | Completed both runs |
| Typeform | Completed both runs | Completed both runs | One of two runs |
| Vercel | Completed both runs | Completed both runs | Completed both runs |
| Webflow | One of two runs | Failed both runs | Completed both runs |
| Zapier | Completed both runs | Completed both runs | Completed both runs |
- Completed both runs
- Failed both runs
- One of two runs
- Not measurable
Not measurable — not a failure. Our own page-observation step exceeded its deadline on this site in all 12 attempts, so nothing about the website was measured and no result about it is reported. See the exclusion note above.
What would change if we had judged it differently
Three adjudication decisions in D-001 could each defensibly have gone the other way. All three are shown here, with the figures they would have produced, because a reader cannot check a judgement call they are not shown. None of them changes the primary finding.
| Judgement | Runs completed | Completed twice |
|---|---|---|
| As published | 85.6%149/174 | 82.8%72/87 |
| Excluded company counted as a failure | 82.8%149/180 | 80.0%72/90 |
| Pricing rule read literally | 86.2%150/174 | 83.9%73/87 |
| Safety-boundary runs counted invalid | 85.5%147/172 | 82.6%71/86 |
The excluded company — and why the exclusion flatters us
The exclusion described above raises our headline, which is exactly why both figures belong on this page. Counted as six site failures rather than an invalid measurement, completion would be 149 of 180 (82.8%) instead of 149 of 174 (85.6%). All 12 of our observation attempts on that site exceeded their deadline and no other company in the benchmark produced a single such failure, so the company did not fail: nothing about its website was measured.
One pricing verdict that depends on how strictly the rule is read
The pricing task requires the right plan, billing unit and cadence, and the frozen rule says naming the wrong plan fails. One run reported the correct price with the correct unit and cadence but attached it to a tier that is actually free. Read literally — the prompt asks for a price, a unit and a cadence — that run would be a success. We kept the stricter verdict, because the rule was frozen before the runs and we are not changing it after seeing a result. Reading it the other way moves completion from 149 of 174 (85.6%) to 150 of 174 (86.2%) and takes the unstable pairs from 5 to 4. No finding, figure or named example on this page depends on which reading is used.
Two runs that stopped at a safety boundary and were counted as successes
Our agent will not cross from the audited site to a third-party domain. Two runs reached their goal and then halted at that boundary, which the system recorded as a safety stop rather than a completion. We adjudicated them as successes because the evidence shows the journey had already reached the required point. Counting them invalid instead would give 147 of 172 (85.5%). The safety policy itself was not relaxed to reach this conclusion.
Neither of the two runs is carrying the result
Read as first runs only, 75 of 87 pairs completed (86.2%). Read as second runs only, 74 of 87 (85.1%). The order in which the two runs happened is not doing any work.
Where our own evaluator disagreed
The production evaluator's score disagreed with independent adjudication on 8 of 174 valid runs, and every disagreement was in the same direction: the score was too generous. This is reported as a limitation of our scoring, not as a finding about any website.
Methodology
Every number in D-001 comes from this design, which was frozen before any run was executed.
Cohort
- 30 companies selected against pre-registered eligibility criteria.
- 29 yielded comparable measurements.
- Ground truth for every company was established from public pages and frozen before the benchmark ran.
Design
- 180 declared runs: 30 companies x 3 tasks x 2 independent runs.
- 174 valid runs after excluding measurements our own instrument failed to collect.
- Two independent runs per company-task pair, so repeatability is measured rather than assumed.
- Every run was adjudicated against the frozen task definition and the pre-registered ground truth by reading its full step trace. The evaluator's score is evidence, not the verdict.
- One retry was authorised under the frozen retry policy, for a single run that failed inside our own infrastructure. Its superseded attempt is excluded and is not one of the 180.
The three tested tasks
- PRICE_TRUTH
- find the official pricing for a new business customer and identify the lowest clearly non-free paid entry price, with its billing unit and cadence.
- SELF_SERVE_START
- find the official self-service route and follow it until the next action would create an account, submit personal information, authorize an external identity provider or initiate payment. Stop before that action.
- SALES_START
- find the official route to contact sales or request a demo and follow it until the next action would submit a form, schedule an appointment or send customer information. Stop before submission.
Execution limits
- A hard ceiling of 20 model turns per run (MAX_STEPS = 20), frozen as a product limit rather than a benchmark setting.
- A 90-second wall-clock ceiling on the agent loop.
- The agent operates only within the audited registrable domain and its subdomains. Reaching a third-party boundary is recorded as evidence, never crossed.
- No run completed a conversion. Every task stops one action before the irreversible step: nothing was purchased, no account was created and no form was submitted.
What these measurements do and do not cover
- One browsing agent driven by one model. These results are not a statement about AI agents in general.
- One network vantage point - an ephemeral cloud sandbox - on one date. Bot detection depends on IP reputation, headers and timing, so an access result observed here is not a claim that every agent from every network is treated the same way.
- A single observation window. No variation across time of day, deployments or A/B tests was sampled.
- Two runs per company-task pair detect gross instability only. The measured instability rate is a floor, not an estimate.
- 29 measurable companies chosen for eligibility are not a random sample, so no population-level inference is drawn.
- Published prices move. Ground truth carries its verification date, and pricing answers are judged on plan, billing unit and cadence rather than on a currency amount, because the agent and our verification ran from different geographies.
One company was excluded, and the reason is ours
One of the 30 selected companies produced no valid measurement. Our own page-observation step exceeded its internal deadline on that site in all 12 attempts, which consumed each run's entire time budget before the agent could act. No other company in the benchmark produced a single such failure. This is a defect in our measuring instrument, not a result about that company: nothing was measured about its website, and no conclusion about it is drawn anywhere in D-001. The cohort is therefore reported as 30 companies selected, 29 yielding comparable measurements.
Evidence
Each run persisted its full step trace, the pages it observed, the final URL it reached and the evaluation of the result. Those reports are publicly addressable and each one carries a benchmark badge linking back to this page. They are the evidence every finding above cites.
Limitations
These bound every number on this page. They are not caveats added at the end; they are what the design can and cannot support.
- One browsing agent driven by one model. This is not a statement about AI agents in general, and a different agent would produce different numbers.
- One network vantage point — an ephemeral cloud sandbox — on one date. Bot detection depends on IP reputation, headers and timing, so an access result here is not a claim that every agent from every network is treated the same way.
- A single observation window. No variation across time of day, deployments or A/B tests was sampled. Access behaviour can change without a site changing anything intentionally.
- Two runs per company-task pair detect gross instability only. The 5.8% instability rate is a floor, not an estimate.
- 30 companies selected against eligibility criteria are not a random sample of anything. No population-level inference is drawn, and 56% describes 14 of 25 observed failures — not 56% of websites.
- A hard ceiling of 20 model turns and 90 seconds per run. 8 runs were bound by the turn budget, and in at least 3 of those the answer was already on screen.
- Named results describe the measured journeys on the measured date. They are not permanent properties of any company's website.
Test your own website
Agent Conversion runs this same measurement on one URL and one task you choose, and shows you the agent's full path — including where it stopped.