AI Traffic & 800G: The Ethernet Switch Buffer Battle

In the era of Generative AI, the data center network has ceased to be just plumbing. It has become the computer. As Large Language Models (LLMs) scale to trillions of parameters, the bottleneck has shifted from the GPU compute capability to the network’s ability to feed those GPUs. At the heart of this new bottleneck lies a component that was often overlooked in the cloud era: the switch buffer.

The “Buffer War” is heating up. On one side, we have the philosophy of Deep Buffers (championed by Broadcom’s Jericho3-AI), designed to absorb the massive traffic bursts characteristic of AI training. On the other, we have the Lossless/Shared Buffer approach (led by NVIDIA’s Spectrum-X and Broadcom’s Tomahawk 5), which relies on end-to-end congestion control and telemetry to prevent buffers from filling up in the first place.

This article delves deep into this technical battlefield, exploring how 800G and 1.6T speeds, Co-Packaged Optics (CPO), and PAM4 modulation are rewriting the rules of Ethernet switching for AI.

---

1. The Anatomy of an AI Traffic Spike: Why Standard Ethernet Fails

To understand the buffer war, we must first understand the enemy: AI Traffic Patterns.

Traditional data center traffic (north-south) or even standard cloud east-west traffic is relatively stochastic (random) and comprised of many small flows. AI training workloads, however, are distinct:

• Synchronized Bursts: During the “All-Reduce” phase of training, thousands of GPUs pause computation to exchange gradients. They all transmit data simultaneously, creating massive “Incast” events.
• Elephant Flows: These are not small requests; they are massive, long-lived data streams that can saturate links instantly.
• Low Entropy: Unlike web traffic which uses thousands of varying ports (high entropy) allowing for effective ECMP (Equal-Cost Multi-Path) load balancing, AI traffic often looks like a few massive streams, causing “collisions” on specific links while others sit idle.

When an Incast event occurs, thousands of packets arrive at a switch port destined for the same output. No amount of bandwidth—not even 800G or 1.6T—can physically move multiple packets onto a single wire at the same time. They must queue. This is where the buffer comes in.

The “Buffer Wall”

If the buffer is too shallow, packets drop. In traditional TCP, a dropped packet means a retransmission. But in AI clusters using RDMA (RoCEv2), packet loss is catastrophic. It forces the Go-Back-N mechanism, halting the entire training cluster (waiting for the straggler) and destroying Job Completion Time (JCT).

2. The Two Schools of Thought: Deep vs. Shallow

The industry has split into two camps to solve this problem.

Camp A: The “Deep Buffer” Fortress (Broadcom Jericho3-AI)

Philosophy: “Traffic is unpredictable and bursts are inevitable. We need a massive tank to hold the water when the flood comes.”

The Broadcom Jericho3-AI represents this approach. It is not just a switch; it is a router with deep buffers, often backed by external memory (like HBM or huge on-die SRAM).

• Architecture: It uses a “Scheduled Fabric” (DNX architecture). When a packet arrives, it is stored in the ingress leaf, and a request is sent to the egress. The data is only moved across the fabric when the destination is ready.
• Buffer Size: Gigabytes of buffer (vs. Megabytes in standard switches).
• Advantage: It can absorb massive Incast bursts without dropping a single packet. It offers perfect load balancing because it sprays packets across all fabric links (packet spraying) and reorders them at the destination.
• Disadvantage: Higher latency (store-and-forward behavior), higher power consumption, and higher cost due to the complex silicon and memory.

Camp B: The “Smart Lossless” Racers (NVIDIA Spectrum-X, Broadcom Tomahawk 5)

Philosophy: “Buffering is a crutch. If you are buffering, you are adding latency. The solution is to control the traffic source.”

NVIDIA’s Spectrum-X platform (built on the Spectrum-4 ASIC) and Broadcom’s Tomahawk 5 argue that deep buffers effectively pause the network.

• Architecture: Shared-Buffer XGS architecture. High-speed, low-latency, single-chip switching.
• Innovation: Instead of deeper buffers, they use advanced telemetry and congestion control.
  • NVIDIA: Uses “Direct Data Placement” and precise RoCEv2 extensions (adaptive routing) to signal GPUs to slow down before the buffer fills. They claim their “Lossless Ethernet” acts like InfiniBand.
  • Broadcom Tomahawk 5: Features “Cognitive Routing” and dynamic flowlet steering to move traffic away from congested paths in nanoseconds.
• Advantage: Ultra-low latency, lower power, better cost/performance ratio for standard Ethernet deployments.
• Disadvantage: Requires tight integration between the switch and the NIC (e.g., NVIDIA BlueField SuperNICs) to work effectively. If the congestion control loop is too slow, packets will drop.

3. The Velocity Factor: 800G, 1.6T, and PAM4

As we move from 400G to 800G and look toward 1.6T, the physics of the buffer changes.

The SerDes Challenge

To achieve 800G, we rely on 112G SerDes (Serial-Deserializer) lanes using PAM4 (Pulse Amplitude Modulation 4-level).

• PAM4 Sensitivity: PAM4 encodes 2 bits per symbol, doubling throughput compared to NRZ, but it is more susceptible to noise. This requires Forward Error Correction (FEC).
• Latency Penalty: FEC adds latency. A deep buffer switch adding milliseconds of queuing delay on top of FEC latency can stall the GPUs, leaving expensive H100s/H200s idle.

1.6T and the “Radix” Explosion

1.6T Ethernet (224G SerDes) is the next frontier. A 51.2T switch (like Tomahawk 5 or Marvell Teralynx 10) provides 64 ports of 800G. The next generation (102.4T) will enable high-radix 1.6T fabrics.

• Impact on Buffers: At 1.6T speeds, a buffer fills up twice as fast. A 100MB buffer at 400G provides X microseconds of burst absorption. At 1.6T, it provides X/4. The “time-to-live” of a buffer is shrinking, favoring the “Congestion Control” camp because you simply cannot build on-chip buffers big enough to hold 1.6T floods for long.

4. The Physical Limit: CPO and the Heat Barrier

Why not just add more memory for buffers? Heat.

Moving data from the switch ASIC to external memory (for deep buffering) consumes power. Moving data through copper cables (DAC) at 1.6T consumes even more power per bit.

• CPO (Co-Packaged Optics): To solve this, the industry is moving toward CPO, where the optical engine is placed on the same substrate as the switch ASIC.
• The Buffer Implication: CPO aims to reduce power. A deep-buffer architecture (like Jericho) is inherently more power-hungry. There is a conflict between the drive for “Green AI” (power efficiency) and “Deep Buffer” reliability.
• Optical Interconnects (SR/DR/FR):
  • SR (Short Reach): Multimode fiber, getting harder at 800G/1.6T.
  • DR (Datacenter Reach): Single-mode, parallel (DR4/DR8). The standard for AI clusters.
  • FR (Far Reach): For longer campus links.
  • As we move to DR/FR for larger clusters (10k+ GPUs), the latency of light in fiber becomes a factor. The congestion control loop (Round Trip Time) takes longer, potentially making deep buffers more necessary for long-distance links, even if shallow buffers win inside the rack.

5. Market Analysis & Competitor Landscape

Feature	Broadcom Jericho3-AI	NVIDIA Spectrum-X (Spectrum-4)	Marvell Teralynx 10	Broadcom Tomahawk 5
:—	:—	:—	:—	:—
Architecture	Deep Buffer (DNX)	Shared Buffer (XGS-like)	Programmable Shared Buffer	Shared Buffer (XGS)
Primary Use Case	AI Back-end Fabric (Massive Scale)	AI Cloud / Hyperscale	AI / Cloud / Edge	Hyperscale / AI
Congestion Mgmt	Packet Spraying, Reordering	End-to-End Telemetry, Adaptive Routing	Teralynx Flashlight Telemetry	Cognitive Routing
Key Strength	Zero Packet Loss (Incast immunity)	End-to-End Optimization (with NICs)	Low Latency & Programmability	Bandwidth Density & Cost

Identifying the Market Blank Spot:

Most content focuses on “Throughput” (51.2T). The “Blank Spot” is the cost-of-idle-time. A Jericho3-AI network is expensive. But if a cheaper Spectrum-X network causes 10% GPU idle time due to congestion, the cost of the wasted GPU cycles dwarfs the network savings. The market needs a TCO calculator that correlates Buffer Depth to GPU Utilization.

6. FAQ: Understanding the Buffer War

Q: Why can’t we just use InfiniBand?

InfiniBand is excellent (credit-based flow control means no drops), but it is a proprietary, single-vendor ecosystem (NVIDIA). Hyperscalers (Meta, Google, Microsoft) want open Ethernet to avoid vendor lock-in and leverage the massive Ethernet supply chain.

Q: Does 800G solve congestion?

No. Bandwidth is the width of the pipe; congestion is the jam at the intersection. 800G allows you to crash faster if you don’t manage the traffic.

Q: What is the role of the NIC in this war?

Crucial. In the “Shallow Buffer” camp, the NIC (SmartNIC/DPU) acts as the brake pedal. It must react instantly to switch signals (ECN) to throttle traffic. Without smart NICs, shallow buffer switches fail in AI workloads.

7. Conclusion: The Hybrid Future?

The “Buffer War” is not a binary choice. We are seeing a convergence.

• Small Clusters: Lossless Ethernet (Spectrum-X/Tomahawk) is sufficient and more cost-effective.
• Massive “Superclusters” (32k+ GPUs): The reliability of Deep Buffers (Jericho3-AI) provides an insurance policy against the chaos of scale.

As AI models grow, the “Buffer” is no longer just memory on a chip; it is the shock absorber for the world’s most valuable intelligence.