Every team that ships a pricing page faces the same wait. You launch a new design, then sit on your hands for two or three weeks while real traffic trickles in, hoping you’ve gathered enough sessions to tell whether the change helped or hurt. By the time the data is conclusive, the quarter is half gone.

So before I asked for a live test slot, I tried something different: I built a panel of AI agents that role-play as real buyers and turned them loose on two versions of the page. Each agent lands on one variant blind, reads the page, picks a plan or walks away, and explains why. I ran 160 of these simulated visits across ten distinct buyer types — and I had a defensible read in the time it takes to drink a coffee, for under a dollar of API credit.

Here’s how it works, what it found, the number I’m proudest of (the one that wasn’t statistically significant), and how you can run the same pre-screen yourself with Claude.

The redesign being tested

The page started as a six-tab layout: Standard, VPS, High Performance, Agency, and so on. The visitor self-selects by guessing which tab applies to them.

The redesign replaces all of that with a two-question router: “What are you building?” (“A website,” “An online store,” “App or AI agent,” “Client sites”) and “What stage are you at?” (“Just starting,” “Growing,” “High traffic”). Answer two questions, see exactly three relevant plans.

Crucially, the prices are identical in both variants. Only the architecture — how a visitor navigates to a decision — changes. So any difference in behavior is caused by the layout, not by a discount.

The idea: synthetic shoppers, not synthetic data

This isn’t about generating fake numbers. It’s about generating fake people and watching what they actually do.

Each agent is handed a persona: a short brief describing who it is, its goal, its budget, and its technical sophistication. Ten personas span the full range, from a price-sensitive blogger to an enterprise IT manager:

• blogger — one personal site, price-sensitive
• smb_owner — a small business, wants to feel secure
• marketer — moderate traffic, wants headroom
• store_new — launching a first online store
• store_scale — an established store doing real volume
• dev_agent — a developer running automation/AI workloads
• devops — production infrastructure, plans for scale
• agency — manages many client sites
• freelancer — a few client sites, growing
• enterprise — high-traffic corporate, needs a conversation

Each agent is a persona-conditioned Claude call. It lands on one variant (blind, randomized, between-subjects — it never sees both), reads the rendered page the way a person would (plan names, prices, “recommended” badges, feature bullets, tabs and toggles), weighs them against its goals, and decides: select a plan, ask to talk to sales, or abandon. Every session logs the action, the plan and price chosen, the number of steps, a confusion score, and a one-line rationale.

That rationale is the gold. It’s not a star rating — it’s the agent thinking out loud, the way you wish every real visitor would. Here’s a developer agent talking itself into a bigger plan:

And here’s an enterprise buyer giving up on the tabbed page entirely:

Multiply that by 160 visits and you have a usability study with a transcript for every participant.

The headline result

Across 160 visits (80 per variant), the share of agents choosing a high-value plan — defined as $9.99/mo or higher, or converting to a sales lead — moved sharply:

That’s a 26-point gap. In statistical terms it’s z = 3.40, p = 0.0007 — meaning if the two pages were secretly identical, you’d see a gap this large less than once in roughly 1,400 repeats of the whole experiment. And it replicated across two independent runs. Luck is a bad explanation. The router genuinely shifts choices toward higher tiers — at least for these AI agents.

The mechanism: who moved, and who was left alone

The aggregate number is only convincing because the per-persona log shows exactly where it came from. Four personas moved up exactly one tier. Zero moved down.

• marketer: Business ($6.99) → Pro ($9.99) — won over by the visits-per-month headroom and phone support the router surfaced.
• dev_agent: NVMe 4 ($4.18) → NVMe 8 ($8.36) — reasoning that automation plus an AI agent would eat 4 GB alive.
• devops: NVMe 8 ($8.36) → NVMe 16 ($17.67) — citing production load and the visible upgrade path to a dedicated server.
• enterprise: scattered choices and outright abandonment on the tabs → a clean sales lead on the router.

The router also reframed store_scale (an established, high-volume store) away from grabbing the $9.99 self-serve plan and toward a managed-eCommerce sales conversation — “I’m doing $40k a month and traffic spikes are killing me.”

The part I find most important is what didn’t move. The budget personas stayed put. The blogger still picked the cheapest Starter ($3.99). The small-business owner stayed on Business ($6.99). The new store owner stayed on eCommerce Essentials ($14.99). The router guided people who wanted more toward plans that fit — it didn’t shove price-sensitive visitors upward. That’s the line between guidance and a dark pattern, and the data sits on the right side of it.

The most useful number was the one that wasn’t significant

Here’s the result I’m proudest of, precisely because it’s the inconvenient one.

The router clearly shifts people toward higher tiers. But does it make more money per sale? Best single guess: +$24 of first-term revenue per selection. But dollar averages are jumpy — one agent picking a $140/mo Cloud plan swings the mean more than five agents picking the $3.99 Starter. So I bootstrapped it: re-sampled the results 10,000 times and recomputed the difference each time. The 95% confidence interval came out at [−$27, +$73].

That range includes zero. So “no revenue difference per sale at all” is still on the table. Honest conclusion: I proved the router shifts people toward higher tiers; I have not yet proved it earns more per sale.

And one more piece of self-incrimination. The significance test assumes 80 independent visitors per page. But the trial log shows each persona made the same choice 8 times out of 8. Within a persona, the agents are near-clones — like asking the same friend a question eight times instead of asking eight different friends. So the honest size of the evidence is closer to 10 independent voters than 160 visitors, which means even that beautiful p-value is optimistic.

The honest caveat (and why it’s still worth doing)

These are synthetic shoppers, not customers. An AI agent role-playing a blogger is a model of a blogger: fast, nearly free, directionally useful — but more uniform and more rational than the messy, distracted, trust-driven humans who actually convert. Treat this as a pre-screen, not a verdict. Its job isn’t to decide what ships. Its job is to decide what earns a live test slot — to catch the obvious wins, dead ends, and friction points cheaply, so the expensive real-world A/B test starts from a much smarter design.

There was also an unexpected payoff. The agents’ one-line rationales surfaced things I never wrote into their briefs — the enterprise personas independently complained about missing SLAs and procurement requirements. That’s the kind of qualitative signal you usually pay for moderated user interviews to get.

My own next step before fully trusting this pipeline: calibration. Replay a few past A/B tests with known real outcomes through the same panel and score its hit rate. Calibration first, conclusions second.

How you can run this yourself with Claude

You don’t need a research team or a testing budget. Here’s the whole recipe.

1. Write your personas. Five to ten is plenty. Each is a few sentences: who they are, their goal, their budget, their technical level, and the context they carry. Be specific — “a developer running n8n plus an AI agent who knows 4 GB will be tight” produces far richer behavior than “a technical user.”
2. Get both variants in front of the agent. Either point Claude at each live URL with its browser tools so it reads the page exactly as a visitor would, or paste/upload the HTML for each variant (faster and fully repeatable). Keep prices identical so you’re testing architecture, not discounts.
3. Fix the output schema. Have Claude return the same fields every time: action (selected / sales_lead / abandoned), plan, price, steps, a confusion score (1–5), and a free-text rationale. Consistent columns are what make the run analyzable.
4. Run it blind and randomized. Each agent sees one variant only. Run every persona through each variant several times so you see a range of behavior, not a single roll of the dice.
5. Log to a CSV and analyze by persona. Compare variants on conversion, plan mix, abandonment, and confusion — broken out by persona, because the aggregate hides the real story. In this run the averages looked similar; the segment-level movements were where everything interesting lived.
6. Be your own skeptic. Bootstrap the revenue difference. Check whether your agents are really independent or just one model re-deciding in slightly different moods. Report the weak numbers next to the strong ones. And before you trust the pipeline for real decisions, calibrate it against past tests with known outcomes.

What you need to start: access to Claude (the browser tools make live-page testing trivial), the two variants, a short persona list, a fixed output schema, and somewhere to drop the results. The whole loop — design, simulate, analyze — fits in an afternoon and costs about a dollar. It turns “let’s launch it and see” into “let’s find out what deserves a test first.”

Have a page you’re about to ship? Run a few synthetic shoppers through it before you do. The cheapest usability insight is the one you get before launch — as long as you stay honest about what it can and can’t tell you.