The Infrastructure Paradox: Why Traditional CI/CD Fails Foundational Platforms

In the modern software development landscape, "Continuous Delivery" (CD) is often treated as the gold standard for engineering excellence. From the industry-shaping "Accelerate" book to the widespread adoption of DORA metrics, the consensus is clear: automate everything, deploy early, and deploy often. However, for a specific subset of engineers—those building "foundational platforms" like compute engines, storage layers, and core APIs—this dogma often leads to instability, outages, and organizational friction.

Ian Nowland, co-founder of Junction Labs and former AWS/Datadog executive, argues that the application-centric model of CI/CD is fundamentally ill-suited for the infrastructure that supports the rest of the business. For foundational platforms, "moving fast" can result in catastrophic, system-wide failure.

The Core Conflict: Applications vs. Foundations

The tension in most engineering organizations stems from a misalignment of definitions. A "foundational platform" is distinguished by one harsh reality: if it breaks, the business ceases to function. When a frontend team experiences a bug, the impact is often localized; when a compute platform team pushes a breaking change, the entire organization—from data ingestion to financial trading—can grind to a halt.

Nowland notes that platform teams are frequently criticized for using "outdated" technology, having slower release cycles, and causing the company’s most severe outages. These criticisms, while often framed as a lack of engineering talent or rigor, are usually symptoms of a mismatch in deployment strategy.

"We are doing a different type of software engineering," Nowland explains. Unlike application teams that can leverage statelessness and rapid rollbacks, platform teams must manage stateful systems and deep integrations with physical hardware. Applying a "move fast" mentality to these layers is not just risky—it is a recipe for disaster.

Chronology: A Decade of Lessons from the Trenches

The EC2 Era (2010–2016): The Birth of Progressive Safety

When Nowland joined AWS in 2010, the cloud was a "wild west." The platform was notoriously flaky, with 1 in 1,000 instance launches failing due to obscure DHCP issues. The culture at the time was defined by "you build it, you own it," with no dedicated test or SRE teams to shield developers from the consequences of their code.

The solution was not more automation, but more discipline. AWS engineers adopted a rigorous deployment planning process. Every release involved a detailed, peer-reviewed plan outlining the changes, the phasing of the rollout, and, most importantly, the verified rollback path. This process—often derided by engineers as "bureaucratic paperwork"—was the only thing preventing massive, region-wide outages.

The Datadog Transition (2019–2023): Scaling in a High-Velocity Culture

Transitioning to Datadog, Nowland encountered a different problem: how to introduce "slow", safe deployment practices into a company defined by its speed and product-centric culture. Datadog’s success was built on feature flags and rapid iteration. However, as the company grew to serve Fortune 100 clients, the "acceptably flaky" nature of their early days became a liability.

Nowland learned that imposing top-down process from the executive suite was ineffective. Instead, he fostered a culture of "champions"—influential staff engineers who understood that while application teams could push code multiple times a day, the platform layer required a more measured, progressive approach.

Supporting Data: Why "Standard" Practices Fall Short

The Fallacy of Staging

One of the most persistent myths in software engineering is the "perfect staging environment." Nowland asserts that for foundational platforms, staging is a "tragedy of the commons." At small scales, staging works; at large scales, it becomes a graveyard of configuration drift and perpetual instability. Because platforms are the foundation upon which everyone else builds, they cannot effectively simulate the complexity of production.

The Rise of Synthetic Monitoring

If staging is doomed, how does one test? The answer lies in synthetic monitoring—running automated workflows in production that mimic actual customer behavior. Unlike standard integration tests, which are often "flaky" and prone to false negatives, synthetic tests provide a high-fidelity signal. They force the platform team to experience the same pain as the customer. If the synthetic test fails, the team knows exactly where the production issue lies, because the test is running against the same dependencies as the live product.

The "Time Finds Problems" Principle

Nowland points to the "Marc’s Law" (named after Distinguished Engineer Marc Brooker): Time finds problems; people fix them. Complex systems, particularly those involving garbage collection, network latency, and TCP parameters, often exhibit failures that only manifest after days of operation.

In a 2015 EC2 incident, a seemingly minor SSL certificate change caused a subtle increase in garbage collection overhead. This, in turn, triggered a chain reaction that resulted in a massive DDoS attack from the platform’s own host fleet against its mapping service. No amount of pre-deployment testing could have caught this; only a slow, phased rollout—allowing "time to find the problem"—prevented a total, permanent collapse of the service.

Official Perspectives and Operational Reality

The divide between executive expectations and engineering reality remains a major point of contention. Executives often view "testing in production" as a lack of rigor, whereas experienced platform engineers view it as the only way to handle "unknown unknowns."

Nowland argues that the industry needs to rebrand its approach. Instead of talking about "testing in production"—which triggers alarm bells for non-technical leadership—teams should focus on "progressive deployments." This strategy includes:

• Small-batch deployments: Limiting the blast radius.
• Phased rollout: Deploying to the least sensitive regions first.
• Observability-driven releases: Watching graphs manually or via automated triggers during every stage of the rollout.

Implications for the Future of Platform Engineering

The primary implication of Nowland’s findings is that "Continuous Delivery" is not a one-size-fits-all metric. For a frontend team, DORA metrics are a valid North Star. For a foundational platform team, the North Star must be reliability and stability.

Engineering organizations should stop forcing platform teams to conform to the same deployment cadence as application teams. Instead, they should:

1. Prioritize Ownership: Ensure the team building the platform is the one on-call, reinforcing the incentive to write stable code.
2. Invest in Synthetic Monitoring: Build high-fidelity tests that run continuously in production.
3. Accept the "Manual" Necessity: Acknowledge that for critical infrastructure, a human-in-the-loop review process is not a sign of failure, but a necessary safeguard against the inherent complexity of modern distributed systems.

Ultimately, the goal of a platform team is not to match the velocity of product teams, but to provide the stable, reliable surface that allows those product teams to thrive. By shifting the focus from "how fast can we deploy" to "how safely can we evolve," organizations can prevent the outages that define a company’s worst days and instead focus on the innovation that drives their long-term success.