Micron says AI "memory wall" crisis worsening, warns HBM lagging behind processors
Micron highlights worsening HBM memory bottleneck in AI systems as compute speeds outpace bandwidth by 3x.
At the Hot Chips 2026 conference, Raghu Sreeramaneni, HBM Architecture Fellow at Micron, highlighted a major challenge facing the artificial intelligence industry, stating that the "Memory Wall" problem is steadily intensifying due to imbalanced hardware growth.
According to the presentation, AI accelerator compute performance (Compute TFLOPS) has grown by an average of about 3x every two years, whereas high-bandwidth memory (HBM) bandwidth has increased by less than 2x over the same period. This widening gap forces processors to waste time waiting for data transfers, turning memory into a primary bottleneck for large-scale AI systems.
Additionally, Micron cited compelling statistics showing that HBM-related failures accounted for up to 17% of all downtime incidents during the training of Meta's Llama 3 models.
Regarding physical constraints, modern System-in-Package GPU modules featuring 12-layer HBM stacks mean that memory silicon accounts for roughly 90% of the total silicon area—about 8 times larger than the GPU chip itself—directly impacting thermal dissipation and thermal resistance.
To address this, Micron stated that the industry must rely on advanced packaging innovations such as Hybrid Bonding or Fusion Bonding, die-to-die interfaces, and new architectures like Processing-in-Memory (PIM DRAM) to help reduce power loss and manage future thermal challenges.
This reflects a global hardware limitation that makes developing Large Language Models (LLMs) more costly and time-consuming, which will ultimately impact the speed at which end-users gain access to AI services.