The Illusion of the Synthetic User: Why LLMs Cannot Yet Replace Human A/B Testing

In the race to optimize digital products, a seductive proposition has taken hold of the tech industry: what if we could eliminate the slow, expensive, and traffic-heavy process of A/B testing by simply asking a Large Language Model (LLM) how a user would react? The promise is intoxicating—near-instantaneous insights, zero opportunity cost, and the ability to iterate at the speed of thought.

However, a groundbreaking study recently published by researchers suggests that while this "synthetic A/B testing" holds promise, it is built on a foundation of precarious assumptions rather than the rigorous causal design that defines the gold standard of scientific experimentation. By analyzing thousands of real-world A/B tests from the Upworthy Research Archive, researchers have demonstrated that LLMs are not merely "noisy" simulators; they are systematically biased, and relying on them without rigorous calibration could lead companies to fundamentally misjudge the impact of their product decisions.

The Core Fact: Design vs. Assumption

At the heart of the debate is the distinction between identification by design and identification by assumption. Traditional randomized controlled trials (RCTs) are the industry gold standard because they causally identify treatment effects through the random assignment of users. When you flip a coin to decide who sees a new feature, you ensure that any observed difference in outcome is attributable to that feature.

When an organization swaps human users for an LLM, that causal guarantee evaporates. The researchers formalize this shift using surrogate endpoint theory—a framework borrowed from biostatistics, where biomarkers (like blood pressure) are used as proxies for clinical outcomes (like heart attack risk). In this new digital context, an LLM’s prediction becomes the "biomarker." The study argues that for this to be valid, two rigorous conditions must be met: Surrogacy and Comparability.

Surrogacy: The "Everything That Matters" Clause

The surrogacy assumption dictates that the LLM’s output must fully mediate the treatment effect. In plain terms, if you know the LLM’s prediction and the baseline characteristics of the user, the fact that a user was assigned to the "treatment" group should provide zero additional information about how they will behave. If the LLM misses any aspect of the treatment that influences human decision-making, the experiment fails to capture the true effect.

Comparability: The Stability of Calibration

The second pillar, comparability, assumes that the relationship between the LLM’s "prediction" and actual human behavior is stable. If the calibration function—the way the model’s digital output maps to real-world clicks—shifts because the nature of the experiment has changed, the entire framework collapses. These conditions are rarely verified in industry practice, yet they are the only things preventing an experiment from producing essentially random or misleading data.

A Chronology of the "Synthetic User" Experiment

The researchers’ investigation into this phenomenon proceeded in three distinct phases, utilizing the Upworthy Research Archive, which contains thousands of A/B tests on news headline engagement.

1. The Raw Prediction Phase: The team first prompted gpt-4o-mini to predict click-through rates (CTR) for thousands of headlines. When these raw predictions were analyzed as if they were human data, the results were sobering: the model recovered only 39% of the actual observed human treatment effect. The LLM was consistently "dampening" the impact of the headlines, making effective changes look mediocre.
2. The Calibration Phase: Recognizing the systematic bias, the team attempted to "calibrate" the LLM outputs using both simple linear regression (OLS) and more sophisticated machine learning models, such as gradient-boosted trees.
3. The Falsification Phase: Finally, the researchers put their models to the test using "hold-out" experiments—tests the models hadn’t seen during training. This served as a stress test to see if their calibration methods held up under the pressure of new data.

Supporting Data: The Cost of Bias

The data revealed a stark reality: simple calibration methods, like ordinary least squares (OLS), were insufficient. They failed the falsification test by a wide margin, landing 3.8 standard errors away from the human benchmark.

More advanced machine learning models fared better, successfully bringing the calibrated estimates within the range of statistical significance. However, the researchers also identified a "noise" problem. LLMs possess an inherent randomness—often controlled by "temperature" settings—which introduces variance that further biases results toward zero. By drawing multiple outputs from the LLM and averaging them, the team successfully mitigated this noise, effectively "smoothing out" the model’s uncertainty.

Yet, even with these advanced techniques, the underlying issue remained: the model was not seeing the user; it was predicting based on training data that mirrors the past.

Industry Implications and Official Perspectives

The implications for product teams are profound. If a company relies on synthetic A/B testing across multiple features, they risk "systematic attenuation"—the phenomenon where every good idea looks only half as good as it really is. This could lead to a catastrophic cycle where high-performing innovations are shelved because the synthetic data suggests they aren’t worth the effort.

Industry experts and the authors of the study alike point to a paradox: The promise of LLM-based A/B testing is least justified precisely when it offers the most benefit.

When a company is testing a radical new UI, a disruptive pricing model, or a feature unlike anything they have shipped before, the assumptions of surrogacy and comparability are at their weakest. Because there is no historical data for these novel interventions, the LLM cannot be accurately calibrated. Thus, the very experiments where companies most want to avoid the "cost" of real user traffic are the ones where synthetic data is most likely to lead them astray.

Moving Forward: A Hybrid Approach

The researchers conclude that we must stop viewing LLMs as a replacement for human experiments and start viewing them as a complement.

1. Filtering, Not Replacing

LLMs are exceptionally good at "pre-screening." By using them to prune low-performing variants before they reach a live experiment, companies can increase the efficiency of their testing slots. This saves human traffic for the variants that have the highest probability of success.

2. Variance Reduction

Beyond prediction, LLMs can act as sophisticated covariates. By using LLM-generated predictions as control variables in a standard user-based A/B test, companies can significantly reduce the amount of traffic needed to reach statistical significance. This makes human-based testing faster and cheaper without sacrificing the "design-based" rigor that makes results trustworthy.

3. The Need for Continuous Human Validation

The most critical takeaway is the "expiration date" of any calibration model. Because LLMs are constantly updated by their providers, a calibration function that works today may be obsolete in six months. Furthermore, the "drift" in user behavior means that even a perfectly calibrated model will eventually lose its edge.

"The upfront investment in collecting actual user responses is not optional," the researchers note. "It is what makes the entire testing apparatus trustworthy."

Conclusion: Don’t Give Up the Gold Standard

The allure of synthetic intelligence is powerful, but in the realm of causal inference, there is no shortcut to the truth. By trading identification by design for identification by assumption, companies risk trading their competitive edge for a comforting but potentially false narrative of efficiency.

LLMs can make our experiments more efficient, more surgical, and more intelligent—but they cannot replace the fundamental, messy, and essential act of observing how a human being actually responds to a product. As the digital landscape continues to evolve, the teams that win will not be those who rely on the synthetic, but those who use the synthetic to enhance the human.