The Ethernet Evolution: How MetaRoCE is Redefining AI Infrastructure at Scale

In the rapidly evolving landscape of artificial intelligence, the bottleneck is no longer just the compute power of individual GPUs—it is the fabric that connects them. As AI models scale to incorporate hundreds of thousands of accelerators, the network has become the critical path. Today, Meta is announcing a transformative leap in data center networking: MetaRoCE, a protocol engineered from the ground up to handle the demands of million-GPU clusters while utilizing the ubiquitous, cost-effective foundation of Ethernet.

The Case for Ethernet in the AI Era

For years, the industry has debated the ideal network fabric for AI. While proprietary solutions have historically held an edge, a growing consensus has emerged that Ethernet should be the standard for AI infrastructure. Meta has been a primary advocate for this shift, having already demonstrated that RDMA over Converged Ethernet (RoCE) can power distributed AI training at massive scales.

However, as clusters expand across multiple data centers and regions, traditional RoCE faces inherent limitations. It relies on the network to deliver every frame in order, utilizing Priority Flow Control (PFC) to maintain "losslessness." This approach often discourages the "packet spraying" techniques necessary for high-performance, multi-plane networks. MetaRoCE removes these constraints, providing high throughput, ultra-low tail latency, and operational simplicity designed for the next decade of AI growth.

Chronology: From RoCEv2 to MetaRoCE

The development of MetaRoCE represents the culmination of years of infrastructure research at Meta.

• Early Scaling: Initially, Meta focused on optimizing existing RoCEv2 implementations to handle large-scale distributed training.
• Identifying Friction: Through extensive testing, engineering teams discovered that standard network architectures often "strand" compute capacity. Small amounts of network friction—whether through reordering delays or congestion-induced pauses—resulted in significant drops in GPU efficiency.
• Architectural Pivot: Meta researchers realized that while the fabric sees only packets, the Network Interface Card (NIC) understands the intent of the data. This insight led to the decision to move network intelligence from the switches to the endpoints.
• Collaboration and Validation: Over the past year, Meta partnered with AMD to implement the MetaRoCE protocol on Pensando programmable NICs. This testing phase proved the protocol’s viability in real-world scenarios, leading to the decision to open-source the specification via the Open Compute Project (OCP).

How MetaRoCE Works: Intelligence at the Edge

The core philosophy of MetaRoCE is the decentralization of network management. In traditional architectures, switches are burdened with enforcing losslessness and maintaining packet order. MetaRoCE shifts this responsibility to the NIC, decomposing the network into fine-grained logical paths.

Native Out-of-Order Delivery

Unlike standard protocols that require reorder buffers—which often lead to "head-of-line blocking"—MetaRoCE is designed to handle out-of-order packets natively. Because every packet contains its destination information, data is written directly to its final memory location upon arrival. This eliminates the need for a secondary assembly step, significantly reducing latency.

Native Multipathing and Loss Tolerance

MetaRoCE treats the Ethernet fabric as inherently lossy. By ignoring the need for PFC or pause frames, the protocol avoids the catastrophic "all-stop" congestion events common in legacy networks. If a packet is lost, the protocol identifies the gap in the sequence immediately and triggers a retransmission on the specific path that lost it, rather than stalling the entire connection.

Congestion Control

MetaRoCE employs a hybrid approach: traditional ECN-based, sender-driven AIMD (Additive Increase/Multiplicative Decrease) congestion control combined with receiver-driven "fair-share" rate hints. By returning bandwidth allocation data to the sender with every acknowledgment, the system reaches optimal throughput in one or two round trips, drastically outperforming standard congestion recovery mechanisms.

Supporting Data: Performance at Scale

Meta’s internal testing on a 64-node AMD GPU cluster provides compelling evidence for the protocol’s efficiency. When comparing MetaRoCE to standard RoCEv2, the results were unequivocal:

1. Throughput Gains: MetaRoCE consistently delivered higher total throughput and reduced flow completion times for collective operations like all-reduce and all-to-all.
2. Resilience to Packet Loss: Under simulated network stress, RoCEv2 performance typically degrades rapidly. MetaRoCE maintained ~86% throughput even at a 1% packet loss rate and continued to deliver stable, useful bandwidth at loss rates as high as 10%.
3. Autonomous Recovery: In multi-plane topologies, the protocol demonstrated the ability to redistribute traffic automatically during simulated plane failures, requiring zero intervention from operators or application developers.

Implications for the Industry

The introduction of MetaRoCE is not just a technical update; it is a strategic move to commoditize high-performance networking. By keeping the fabric "simple" and moving complexity to the endpoints, Meta enables infrastructure providers to use standard, off-the-shelf Ethernet switches.

A New Standard for Open Computing

Meta is contributing the full MetaRoCE specification to the Open Compute Project (OCP), aligning it with the Ethernet Scalable Unified Network (ESUN) initiative. This ensures that the protocol is not a proprietary silo but an interoperable standard.

• Vendor Agnostic: The protocol is designed to run on both programmable and fixed-function NICs.
• Compliance Framework: Meta is providing a production-grade compliance suite, allowing hardware vendors to verify that their implementations adhere to the protocol’s requirements.
• Software Reference: The libsoftmetaroce library, which allows the protocol to run on commodity Linux over standard UDP sockets, serves as a behavioral model for future silicon development.

The Road Ahead: Scale-Up and Scale-Across

Meta acknowledges that while MetaRoCE is a major step forward, the AI infrastructure landscape is still evolving. The next phase of development focuses on three distinct regimes:

1. Scale-Up: Within a single rack, where every nanosecond is critical, Meta is optimizing the fast signaling path for short memory operations, further stripping away latency.
2. Scale-Across: As models span buildings separated by thousands of kilometers, the round-trip times stretch into milliseconds. Meta is working to refine how paths are prioritized to maintain fairness across contended long-haul links.
3. Storage and KV-Cache: To address the "incast" problem—where a single request triggers a flood of responses from distributed storage servers—Meta is refining receiver-driven rate hints to prevent network overwhelm during high-volume data retrieval.

Conclusion

The release of the MetaRoCE specification at the 2026 OCP Global Summit marks a pivotal moment for the AI ecosystem. By removing the constraints of traditional, switch-centric network designs, Meta has provided a blueprint for a more resilient, scalable, and cost-effective future.

For engineers building the next generation of AI infrastructure, the message is clear: the future of high-performance networking is not found in proprietary, opaque fabrics, but in the open, intelligent, and highly efficient evolution of Ethernet. Through broad industry collaboration, MetaRoCE aims to become the backbone upon which the next generation of frontier AI models will be trained and served to the world.