The Measurement Paradox: Why Your Marketing Mix Model Is Failing You (And How to Fix It)

Measuring the return on marketing investment (ROMI) has long been the "holy grail" of growth marketing. In recent years, Marketing Mix Modelling (MMM) has seen a massive resurgence, propelled by a wave of sophisticated open-source tools. From Meta’s Robyn and Google’s Meridian to the robust PyMC-Marketing library, the technical barrier to running an MMM has effectively collapsed. However, a dangerous misconception persists: that because it is easy to run a model, it is easy to trust the results.

The truth is more sobering. As growth teams push for data-driven decisions, they are often running models on data that lacks the fundamental variation required to separate channel effects. This article explores why your model is likely misinterpreting your data, how "budget phasing" can restore its integrity, and why the solution isn’t a better algorithm—it’s better experimental design.

Main Facts: The Triple Threat to MMM Accuracy

Google’s 2017 seminal paper, Challenges and Opportunities in Media Mix Modeling, remains the definitive diagnostic for why these models fail. The core issue is that while we have sophisticated statistical engines, we are feeding them "feature-poor" fuel. Specifically, three systemic problems plague the majority of modern MMM implementations:

1. Variance: Because marketing budgets are typically planned in lockstep, channels often move together. When TV, Meta, Search, and TikTok all scale up and down simultaneously, the model struggles to assign credit. Any single fit of the model could produce widely different results, meaning your "optimal" strategy is essentially a coin flip.
2. Bias (Omitted Variable Bias): Marketing spend is rarely random; it follows demand forecasts. When a seasonal peak approaches, brands increase spend. The model sees a rise in sales and a rise in spend and concludes the marketing caused the sales—even if the underlying demand was the true driver. Because this bias is baked into the relationship between spend and demand, more data doesn’t fix it; it only gives you a more precise estimate of the wrong number.
3. Identifiability: To understand saturation (the point of diminishing returns) and adstock (how long an ad effect lasts), a model needs to see a channel operate at different intensity levels for sustained periods. If your spend is always perfectly proportional to your total budget, the model cannot distinguish between a linear response and a curved, saturated one.

Chronology: A Decade of Evolving Measurement

The evolution of marketing measurement has followed a distinct path. In the early 2010s, MMM was a black-box service offered by boutique consultancies. By 2017, the publication of the aforementioned Google research identified the inherent flaws in the methodology, sparking a shift toward "incrementality testing."

For much of the last decade, the industry’s primary response has been to rely on geo-testing—running campaigns in specific regions to measure "lift." While this is a step forward, it is a slow, expensive, and often unreliable process. A 2026 study by Recast revealed a startling reality: many open-source geo-testing tools report a "false lift" between 14% and 30% of the time.

Recognizing that testing every channel individually is neither scalable nor sufficiently accurate, the focus has recently shifted back to the MMM—not by changing the model, but by changing the input. The current state-of-the-art involves "budget phasing": the deliberate, algorithmic injection of variation into marketing plans to ensure that the data fed into the MMM is actually capable of being modeled.

Supporting Data: Simulating the Ground Truth

To understand the impact of budget phasing, we must look at a simulated data environment. By creating a synthetic world where we know the "ground truth" of channel performance, we can measure exactly how much the model deviates from reality.

The Simulation Framework

In a simulation featuring TV, Meta, Search Generic, and TikTok, we assume a high correlation (0.7) between channels, mirroring a standard corporate planning cycle. We also simulate a "demand proxy" that correlates with spend at 0.65.

The findings are stark:

• Variance: With correlated spend, the model’s estimate for incremental revenue is highly unstable. Across 50 simulations, the range for which channel performs best is so wide that no single estimate can be trusted.
• Bias: Because spend follows demand, every channel is consistently over-credited. TV was found to be over-credited by 44%, Meta by 33%, TikTok by 20%, and Search by 17%.
• Identifiability: The model failed to identify the saturation curves of the channels entirely. The range of possible saturation exponents covered almost the entire spectrum (0.20 to 1.00), rendering the model useless for budget optimization.

The Power of Phasing

By introducing a "Combined" phasing strategy—which incorporates "dark weeks" (where a channel is turned off), peak months, and subtle weekly nudges—the results shift dramatically:

• Variance dropped from 0.23 to 0.07.
• Bias was nearly halved, dropping from 28.8% to 15.3%.
• Identifiability improved significantly; saturation and adstock ranges narrowed, providing the model with a much tighter, more reliable picture of reality.

Official Responses and Expert Consensus

Industry experts—including those behind the latest open-source libraries—increasingly agree that the "model-first" approach is reaching its limit. Ryan O’Sullivan, a lead data scientist specializing in causal inference, notes: "Priors can’t invent information that was never in the data. If TV and Meta always move together, no prior, no matter how sophisticated, can tell you which one actually drove the sales."

The consensus is shifting toward the idea that data design is as important as model architecture. While hierarchical Bayesian models can help by pooling data across regions, they cannot overcome the fundamental issue of national-level spend correlation. If you buy your media in a way that prevents the model from seeing independent variance, you are essentially asking your data scientists to perform statistical alchemy.

Implications for Growth Strategy

The implications for CMOs and growth teams are profound.

1. The Cost of Clarity

Accuracy comes at a price. In our simulation, the "Combined" phasing strategy cost roughly 1.5% of total sales revenue. However, this is not "wasted" spend; it is the cost of buying information. By accepting a minor, temporary dip in efficiency, you gain the ability to optimize your entire annual budget with high confidence.

2. Operationalizing the Fix

The path forward is clear:

• Phase your plans: Don’t let your media agency set a flat, consistent weekly spend. Introduce intentional variation.
• Utilize open-source diagnostic tools: Packages like how_wrong_is_your_mmm allow teams to run their own history through a diagnostic engine to see just how much their models are being misled.
• Prioritize "Dark Periods": For channels like brand search or affiliates, where spend is inherently tied to demand, "dark periods" are the only way to break the endogeneity loop.

3. The End of the "Model-Only" Era

The era of simply throwing data into an MMM and trusting the output is over. The future of measurement belongs to those who view their marketing calendar as an experimental design. If you want to know what your marketing is worth, you must first ensure that your marketing spend is not just a reflection of your demand, but a clear, distinct signal that your model can actually learn from.

Conclusion: It Was Never the Model’s Fault

If your MMM has been providing erratic, unbelievable, or "too good to be true" results, stop blaming the software. The model is simply a mirror; if the data you feed it is inherently correlated and biased, the model will faithfully reflect that chaos.

Budget phasing offers a way out. It is a rigorous, scientific approach to marketing planning that prioritizes the health of the data. By sacrificing a small amount of short-term efficiency for long-term clarity, brands can finally move beyond the guessing game of attribution and into the realm of true, causal marketing measurement. The question isn’t whether your model is "smart" enough—it’s whether you have given it a fair chance to be right.