The End of the Data Bottleneck: How Spotify Scaled Expertise with AI

At Spotify, the path to a data-driven insight used to be a fragmented, manual ordeal. For years, the workflow followed a predictable and frustrating pattern: a product manager or engineer would search the internal dashboard for a specific metric, only to find it missing. The next step was a "ping" to a data scientist on Slack, followed by a waiting game that could last hours or days.

As Spotify expanded into a global powerhouse—managing music, podcasts, audiobooks, and advertising—the sheer volume of requests outpaced the capacity of its human experts. With over 70,000 datasets and petabytes of information flowing through its systems daily, the traditional model of one-on-one data consulting had reached a breaking point.

To bridge this gap, Spotify developed an AI-powered data assistant, now known internally as "Vedder." However, the solution was not as simple as feeding the entire data warehouse into a Large Language Model (LLM). As the company discovered, the real challenge wasn’t just building an AI; it was ensuring that AI could be trusted with the truth.

The Limits of Scale: Why Raw LLMs Failed

When Spotify engineers first experimented with LLMs, they hit a wall immediately. Even with a context window of a million tokens, the vast, labyrinthine complexity of a massive data warehouse simply cannot fit into a single prompt.

Moreover, data schemas are inherently deceptive. A database column labeled INT64 provides only technical structure; it says nothing about the underlying business logic. For example, a model might see numbers below 100 in a specific column, but it wouldn’t know those values represent legacy test data, nor would it understand the nuanced, evolving definition of an "active user." When fed too much raw, uncurated data, LLMs become dangerously confident—and frequently wrong.

Spotify concluded that they needed a middle layer: a "knowledge architecture" that captures the semantic meaning of a data slice, owned and curated by the very experts who understand the domain.

Chronology of the AI Transformation

The journey toward Vedder was marked by several critical phases of development:

• Pre-2024: The era of the "Data Bottleneck." Analytics teams were overwhelmed by ad-hoc requests, and self-service dashboards were insufficient for deep-dive questions.
• Early 2024: Exploratory phase. Engineers attempted to use automated query history to train the model, only to realize that 87.5% of historical queries were "noise"—debugging sessions, one-off tests, or flawed SQL.
• August 2025: The official rollout of the Vedder assistant. The system moved into production, integrated directly into Slack, IDEs, and a dedicated web UI.
• Late 2025 – Present: Refinement and expansion. The focus shifted from simple query generation to "cluster health" monitoring, where the AI proactively notifies curators when data schemas or business logic drift.

Supporting Data: The Scale of the Operation

The metrics behind Vedder reveal the sheer intensity of Spotify’s data landscape. The platform processes 1.4 trillion data points daily, a massive infrastructure that supports over 2,100 active users within the company.

Since its launch in August 2025, the agent has facilitated over 13,000 conversations and processed more than 60,000 messages. Perhaps most tellingly, more than 25% of these users have no prior SQL experience, demonstrating that the tool has successfully democratized data access. The assistant covers 177 distinct data clusters, ranging from advertising and financial reporting to creator tools and audio consumption patterns.

The "Cluster Model": Curation as a Core Competency

The heart of Spotify’s approach is the "Cluster Model." Instead of one monolithic AI, the system is broken down into domain-specific clusters. Each cluster is managed by a named team of domain experts—data scientists and analytics engineers—who act as curators.

This model is built on three pillars:

1. Ownership: Every cluster has a human steward responsible for defining what is "canonical" data.
2. Curation: Curators decide how to segment their domain, which tables are relevant, and which query examples are the gold standard.
3. Trust through Verification: The model does not learn from the entire history of a database; it learns only from the specific, expert-approved examples curated by the domain owners.

The data supports this conservative approach. During the curation phase, when experts were asked to review automated SQL-to-question pairings from history, they accepted only 12.5% of the suggestions. The remaining 87.5% was deemed too messy or misleading to serve as a training baseline. By forcing human oversight, Spotify ensured that the AI wouldn’t just hallucinate a "technically correct" but "contextually false" answer.

Keeping the System Healthy: The "Health Score"

An AI system is only as good as the data it sits upon. Because schemas evolve, columns get renamed, and tables are deprecated, Spotify implemented a "Cluster Health Score."

This dynamic metric monitors:

• Validity: Are the curated examples still compatible with the current schema?
• Coverage: Does the cluster provide answers to the questions users are actually asking?
• Reproducibility: Is the SQL generated by the model consistent and error-free?

If a cluster’s health score dips, it triggers a notification to the domain owners, allowing them to perform targeted maintenance. This prevents the "knowledge rot" that plagues most AI-driven data tools.

Implications: The Strategic Shift of the Data Expert

The introduction of Vedder has fundamentally shifted the role of the data scientist at Spotify. Rather than being "human calculators" answering one-off queries, data experts are becoming "knowledge architects."

By spending less time on repetitive tasks, they have gained the bandwidth to shape the semantic layer of the organization. This allows the company to scale its institutional knowledge. Every time a curator refines a definition or approves a query pair, the entire organization benefits.

Official Perspective: The Human-in-the-Loop

"The model reasons over context," the Spotify engineering team notes. "It doesn’t decide what’s true about the data; the experts do."

This philosophy underscores a broader trend in enterprise AI: the move away from "black-box" models toward "Human-in-the-Loop" systems. By placing the expert in the driver’s seat, Spotify has avoided the pitfalls of ungrounded LLMs while simultaneously removing the operational friction that once hampered their internal workflows.

The Future: Beyond the Schema

As Spotify looks to the future, the team is exploring how to move beyond the schema. The next frontier involves integrating unstructured documentation, internal process definitions, and organizational wikis into the Vedder agent.

The success of the data agent proves a vital lesson for the industry: the most effective AI implementations are not those that attempt to replace human understanding with raw compute, but those that treat human expertise as the most valuable, and most scarce, resource.

For other enterprises looking to replicate this success, the blueprint is clear: build for trust, enforce ownership, and remember that even in an era of petabytes, context is king. The "Data Bottleneck" at Spotify hasn’t just been widened—it has been dismantled by the realization that AI is at its best when it is, quite literally, guided by the experts.