Hot Chips 2026: HBM Architectures for AI Performance

For years, we thought AI speed was mainly limited by how fast chips could do math. Today, the bigger limit is memory.

Scaling rules show models improve when parameters, data, and compute grow together. But the balance fails when processors can’t fetch data fast enough.

Compute on modern AI accelerators has risen about 3× every two years. HBM bandwidth has grown at less than half that rate. The gap creates an “AI memory wall.” Expensive chips sit idle, utilization drops, and energy is spent moving data instead of computing.

What is High Bandwidth Memory, and How Does It Fix the Gap?

HBM closes this gap through architecture and packaging.

DDR5 modules sit away from the GPU/accelerator on the motherboard. HBM stacks DRAM dies vertically and places them next to the accelerator in a single package.

Through-silicon vias connect the stacked layers to a base die. A short, dense interposer then links the base die to the accelerator.

This design uses thousands of low-frequency connections. That delivers very high total bandwidth without relying on power-hungry high-speed serial links.

HBM Generational Performance: From HBM1 to HBM4

Each HBM generation increases speed and capacity while keeping the same core approach.

HBM1 provided 1,024 data I/Os and about 128 GB/s per stack. HBM3E keeps 1,024 I/Os but raises signaling rate, channelization, bank count, and stack capacity to reach about 1 TB/s per stack.

HBM4 will double the interface to 2,048 I/Os. It expands to 32 main channels and 64 pseudo-channels. It targets roughly 2.8 TB/s per stack.

Across generations, HBM improves transfer speed, access granularity, and density. That boosts sustained bandwidth and total capacity for parameters, KV caches, activations, and training state.

Performance Impact Explained With the Roofline Model

The roofline model helps explain why HBM speeds up AI.

Arithmetic intensity is the number of operations per byte moved. If a workload sits under the “bandwidth ceiling,” it is memory-bound. Adding compute won’t help because it is waiting on data.

More HBM bandwidth raises that ceiling. Memory-heavy kernels—attention, embeddings, sparse ops, and data movement—can then run more compute before hitting the limit.

At system scale, eight HBM3 stacks can deliver ~5.3 TB/s in theory. An 8-channel DDR5 server is around ~307 GB/s. DDR5 is cheaper and offers more capacity. But HBM provides much higher bandwidth right where the accelerator needs it.

Tradeoffs: The Cost and Engineering Challenges of HBM

HBM performance comes with cost and complexity.

HBM3E uses many more banks and interface circuits than DDR5. Its packaging and manufacturing are also harder. Overall, it can consume about 3× more silicon per unit of capacity than DDR5.

Higher throughput and taller stacks increase heat flux. Different materials also expand differently with temperature. That can cause chip-package reliability issues over time.

Taller stacks make cooling, power delivery, mechanical strength, and testing harder. So features like liquid cooling, better thermal paths, hybrid bonding, glass substrates, and larger interposers often become required—not optional.

Reliability: Scaling Error Protection With Higher Bandwidth

Reliability must scale with bandwidth in large AI clusters.

From HBM3 onward, protection works at two levels. System-visible metadata enables CRC or ECC checks on each access. On-die Reed–Solomon correction protects stored data and inter-layer transfers.

These schemes must integrate with fault isolation, repair, telemetry, and service tools. The goal is to prevent a small defect from taking down a large accelerator system.

Balancing Bandwidth and Capacity for Real-World Workloads

Capacity still matters as models, context windows, and caches grow.

Designers must weigh usable bandwidth, locality, contention, refresh overhead, and fault tolerance together.

Nominal interface speed helps only if controllers, banks, and software can expose enough parallelism. Otherwise, real workloads won’t sustain the advertised rate.

Why This Matters for the Future of AI

Future memory systems will require cross-layer co-optimization.

That spans DRAM process, base-die logic, die-to-die PHYs, interposers, substrates, cooling, power delivery, firmware, and workload scheduling.

Base dies may add custom control logic, data-movement acceleration, or even in-memory functions to cut traffic.

Co-packaged optics can extend high-bandwidth links beyond the package. Fusion and hybrid bonding can increase vertical interconnect density and reduce energy per bit.

The goal is no longer peak compute alone. It is balanced bandwidth, capacity, reliability, and energy efficiency.

HBM is the bridge between fast accelerators and data-heavy AI workloads. Memory architecture increasingly determines achievable performance per watt, per package, and per dollar.