The Mirage of Silicon Users: Why LLMs Can’t Replace Human A/B Testing

In the race to accelerate product development, a seductive promise has emerged: replace slow, expensive, and traffic-heavy human A/B tests with the instant, cost-effective predictions of Large Language Models (LLMs). The pitch is compelling—why wait weeks for user feedback when an AI can simulate thousands of outcomes in a matter of hours?

However, a new paper authored by researchers investigating the efficacy of these models warns that this technological shortcut comes with a dangerous trade-off. While LLMs can indeed predict user behavior, they do so not by design, but by a series of precarious assumptions. When those assumptions fail—as they often do for truly innovative, "first-of-its-kind" features—the result is not just noise, but systematic, misleading bias.

The Statistical Gold Standard vs. The Proxy Shortcut

For decades, randomized controlled trials (RCTs) have served as the bedrock of digital product innovation. By randomly assigning users to treatment or control groups, researchers create a causal "gold standard." This design ensures that any observed difference in outcome is attributable to the intervention, not to pre-existing user differences.

The emerging trend of "LLM-based A/B testing" attempts to bypass this by treating the model as a stand-in for the human. The researchers behind the new study argue that this practice fundamentally misinterprets the role of a surrogate. Using the framework of surrogate endpoint theory—a concept borrowed from clinical trials—they illustrate that for an LLM to be a valid substitute, it must meet two rigorous, often unverified, criteria.

Decoding the Failure: The Upworthy Experiment

To test the viability of these models, the research team analyzed the Upworthy Research Archive, the most comprehensive open-access repository of digital A/B tests. The dataset contains thousands of real-world headlines tested against actual user click-through rates (CTR).

The Findings

When the researchers prompted gpt-4o-mini to predict the CTR for these headlines, the results were sobering. Raw LLM predictions failed to capture the nuances of human behavior, recovering only 39% of the observed treatment effect.

The bias was not merely random "hallucination" or noise; it was systematic. The models consistently attenuated treatment effects toward zero, effectively "dampening" the perceived impact of a change. For a product team relying on these metrics, this creates a dangerous blind spot: promising innovations might be shelved simply because the LLM failed to register their true potency, leading to a massive underestimation of the value delivered to users.

Two Pillars of Validity: Surrogacy and Comparability

The study formalizes the conditions required for an LLM to serve as a legitimate proxy. Without these, any "experiment" conducted on an LLM is effectively untethered from reality.

1. The Surrogacy Assumption

This requires that the LLM output fully mediates the treatment effect. In simpler terms, the LLM must capture every piece of information about the headline that influences a human to click. If there is a hidden factor—a user’s mood, a specific cultural context, or a visual aesthetic—that the LLM ignores but the human acts upon, the surrogacy assumption collapses.

2. The Comparability Assumption

This demands that the relationship between the LLM’s prediction and the human’s response (the "calibration function") remains constant. If a company shifts from testing text headlines to testing interactive UI layouts, the way the LLM processes information may change. If the calibration function drifts, the model’s output becomes an unreliable artifact of the training data rather than a reflection of current user behavior.

The Calibration Paradox: Why Simple Methods Fail

The research team found that "plug-and-play" calibration is insufficient. When they applied linear calibration—using Ordinary Least Squares (OLS)—the model failed significantly, landing nearly four standard errors away from the human benchmark.

Success was only achieved through more sophisticated machine learning techniques, such as gradient-boosted trees, which were flexible enough to map the non-linear relationship between model outputs and human reality. Furthermore, the team highlighted the "sampling temperature" problem. Because LLMs are inherently probabilistic, a single prediction is noisy. By averaging multiple outputs per unit, they were able to dampen the noise, proving that accurate simulation requires rigorous statistical handling rather than raw model generation.

The Innovation Trap: Where LLMs Fail Most

The most striking conclusion of the study is the "Innovation Paradox." The researchers note that the conditions required for LLM-based testing can only be validated through historical data.

For routine, iterative changes—such as minor copy adjustments in a familiar context—LLMs may be useful because the "surrogacy" is well-understood. However, for truly transformative innovations—new pricing models, experimental interfaces, or features that break the mold—there is no historical data to calibrate against.

"The promise of LLM-based A/B testing is least justified precisely when it offers the most benefit," the authors state. When a company is building something truly new, they are in uncharted territory where the assumptions required for AI validation simply cannot be proven.

Industry Implications and Future Outlook

What does this mean for the future of Product Management and Data Science? The consensus is that while LLMs cannot replace human experiments, they can fundamentally augment them.

Recommendations for Product Teams:

• Filter, Don’t Replace: Use LLMs as a "first-pass" filter to eliminate weak ideas before they consume valuable traffic in a real-world experiment.
• Variance Reduction: Utilize model outputs as covariates to make human-based A/B tests more efficient, rather than using them to replace the user outcome entirely.
• Calibration Maintenance: Treat calibration functions as living assets. Because LLM providers (like OpenAI or Anthropic) frequently update their models, a calibration function that worked six months ago may be obsolete today.
• Respect the Design: Human experiments are valid by design; they account for the "unknown unknowns" of human behavior. LLM experiments are valid only by assumption.

Conclusion: The Persistence of Human Insight

The allure of the "all-knowing" LLM is powerful, but the data confirms that it is no substitute for the unpredictable, messy, and authentic nature of human reaction.

As organizations scale, the pressure to cut costs and accelerate cycles will only increase. However, the researchers emphasize that the investment in actual user data is not a waste—it is the essential foundation that makes any subsequent AI prediction trustworthy. By substituting real user outcomes for model predictions, firms are not optimizing their processes; they are trading objective truth for an assumption that, when the stakes are highest, is likely to be wrong.

In the final analysis, innovation requires the courage to test the unknown. A model can only predict the past; human users, through the medium of the A/B test, remain the only authority on the future.