The Memory Wall
Why AI's Next Bottleneck Is Bigger Than HBM
11 July 2026 · YK Research
Contents
Executive Summary
Reasoning models, persistent agents, long context, multimodal workloads and reinforcement learning keep more state alive for longer. HBM holds the hottest state. Server DRAM, LPDDR and flash absorb everything that does not fit. Networking moves it between tiers. The addressable market is therefore broader than HBM alone.
The near-term trade is physical scarcity. DRAM wafers, HBM stacking, packaging and yield learning cannot expand as quickly as AI workloads. The longer-term option is custom HBM, or cHBM, which can turn part of memory from a standard component into customer-specific silicon.
This remains a semiconductor cycle. High prices fund new capacity and force customers to compress, recompute, pool and offload. The investment problem is not whether AI needs more memory. It is whether workload growth outruns efficiency and supply for long enough to protect earnings revisions.
Theory of Edge
The market pays memory suppliers to solve a physical mismatch: demand can compound in software time while fabs, packaging lines and qualified yields move in industrial time. Hyperscalers will pay a high price for scarce memory when those bits unlock expensive accelerators and revenue-generating tokens.
Own the scarce bits while agents and inference expand faster than wafer, stacking, packaging and yield capacity. Harvest before high margins create their own supply response.
The variant view is that memory per task can fall while total memory demand rises. Better architectures lower bytes per token, but cheaper inference encourages more users, more agents, longer contexts and more deployed accelerators. Efficiency can expand the market it was supposed to shrink.
What Is Driving Demand
Weights
Trillion-parameter and mixture-of-experts models still need the full model stored somewhere, even when only a fraction of parameters computes each token.
Reasoning and agents
Longer traces, tool outputs, parallel sub-agents and reusable prefixes keep sessions active and make concurrency as important as model size.
Multimodal and physical AI
Video, audio, robotics and world models create far more state than a short text prompt. The data must be stored, moved and reused.
RL and synthetic data
Many simultaneous rollouts require model replicas, environment state, checkpoints and data pipelines. The bottleneck becomes the whole system.
The Memory Hierarchy
Treating memory as one market hides where value moves. Each tier trades speed for capacity and cost. AI inference increasingly uses all of them.
HBM
Holds hot weights, active KV cache and training state next to the accelerator. It has the highest bandwidth, highest price and hardest qualification process.
Server DRAM and LPDDR
Holds CPU tools, orchestration state, cold model shards and offloaded KV cache. It expands effective working memory at lower cost than HBM.
Enterprise flash
Stores checkpoints, embeddings, reusable prefixes and cold KV cache. Latency matters because agent workloads revisit context instead of reading once.
Networking and cache software
Links GPU, CPU memory and flash. Bad movement policy leaves expensive GPUs idle. Good tiering makes cheaper memory behave like a larger pool.
What the Evidence Says
1. Hyperscaler demand is moving beyond GPUs
Reuters reported from an internal Meta memo that the company planned to deploy 7GW of compute infrastructure in 2026 and reach 14GW in 2027, with up to $145B of AI-infrastructure spending in 2026. The same report described multi-year agreements with Samsung for memory and Sandisk for flash. Contract volumes, prices and take-or-pay terms were not disclosed.
2. Consumer buyers are losing bargaining power
Apple raised prices across Macs, iPads and several home devices after saying it could no longer absorb memory and storage inflation. It then reportedly lobbied Washington and negotiated to source DRAM from China's CXMT and NAND from YMTC for devices sold in China. CXMT is not yet a confirmed Apple supplier. The signal is the sequence: higher prices followed by a politically difficult search for supply.
3. Public model configurations show why concurrency matters
Kimi K2.5 discloses 1T total parameters, 32B active parameters and a 256K context window. GLM-5.2 discloses 744B total, 40B active and a 1M context window. Config-derived BF16 compressed-KV estimates are about 68.6 KiB per token for Kimi and 87.8 KiB per token for GLM. One full GLM 1M context is roughly 87.8 GiB before allocator, runtime and workspace overhead. Eight full sessions approach 0.69 TiB.
4. The fastest GPU does not always deliver the best economics
Dylan Patel said SemiAnalysis benchmarks more than $80M of accelerators against over $5M of Claude Code traces. His reported result is workload-specific: memory offload and storage can matter more than peak chip speed, and higher-memory AMD GPUs beat faster NVIDIA GPUs in some cases. The underlying agent benchmark and winner ranking were not published.
Official specifications support the direction, not a universal AMD lead. MI300X and MI325X carried more memory than H100 and B200. Current MI355X and B300 each have 288GB. AMD's MI400 roadmap raises the bar to as much as 432GB of HBM4, but NVIDIA retains the stronger software and rack-scale ecosystem.
cHBM: The Structural Option
Standard HBM uses a largely standardized interface. Custom HBM can place customer-specific controllers, compression, preprocessing, security or interface logic in the base die. That turns part of the memory stack into co-designed silicon.
Potential system gains
- Less interface power and data movement.
- More compute-die area available for useful logic.
- More memory stacks or capacity per accelerator.
- Workload-specific control of compression and cache movement.
Potential business gains
- Longer co-design and qualification cycles.
- NRE, design-service and base-die revenue.
- Higher switching costs and better demand visibility.
- Possibly better trough margins than commodity DRAM.
Marvell's architecture work claims up to 70% lower interface power, 25% compute-die area savings and 33% more HBM stacks. These are vendor architecture claims, not disclosed NVIDIA Feynman specifications.
NVIDIA has shown custom HBM at roadmap level for Feynman. Full memory-on-logic is not confirmed. Customers will also demand second sources and may keep most design ownership. cHBM can improve memory economics without ending the semiconductor cycle.
Who Wins
The exact winner in SemiAnalysis' private agent benchmark is unknown. The public evidence supports a stack-level ranking instead.
SK hynix
NVIDIA
Micron
Sandisk
AMD
Broadcom and Arista
Other important exposures
Samsung has the strongest theoretical vertical-integration option across memory, foundry, logic and packaging, but weaker HBM execution makes the equity expression less clean. Private storage platforms such as VAST Data and WEKA may capture software value from shared KV-cache infrastructure, but public benchmarks are often vendor-sponsored and the companies are not directly investable.
The Cycle Still Matters
Bull case
- Reasoning, agents, multimodal workloads and accelerator shipments outgrow efficiency gains.
- HBM remains captive for the hottest data and highest-value compute.
- cHBM embeds suppliers deeper into customer architecture.
- Deposits and long agreements extend scarcity and improve visibility.
Bear case
- New wafer, packaging and yield capacity catches demand in 2027-28.
- Quantization, sparse attention, recomputation and offload reduce HBM per task.
- High prices destroy consumer demand and accelerate substitution.
- Foundries, ASIC designers and customers capture the value from cHBM.
Shortages lengthen lead times, induce double ordering and hide true consumption. Prices rise, customers secure too much capacity, suppliers build, and inventory arrives after demand has slowed. Stocks usually peak before the income statement.
Portfolio Framework
Expression
- Core: a basket across SK hynix, Micron and selective flash exposure. Winner selection remains uncertain.
- Platform hedge: NVIDIA benefits if tiered memory raises useful GPU utilization.
- Challenger option: AMD if high-capacity accelerators convert specifications into shipped rack-scale systems.
- Sizing: assume 20% up days, 50% drawdowns and cycle reversals before earnings peak.
What to monitor
Accelerator shipments, memory per accelerator, KV bytes per token, agent concurrency, hyperscaler capex and useful-token revenue.
Wafer output, HBM stack and base-die yields, packaging capacity, lead times, inventories, customer deposits and supplier capex.
Named design wins, second sources, ownership of base-die IP, NRE revenue, custom mix, thermals and production yield.
Harvest when contract prices or sold-out horizons roll over, inventories rise, capex catches demand, efficiency outruns workload growth, or memory stocks stop responding to positive earnings revisions.
Sources and Caveats
For research use only. It does not constitute a recommendation. Product roadmaps, customer agreements and model configurations can change. Scenario calculations are sensitivities rather than forecasts.