TL;DR

·AI's memory requirements go beyond HBM. Training necessitates a complete memory hierarchy, spanning from HBM, system DRAM, to SSDs and shared storage.

·The real bottleneck in inference occurs during the decoding phase. The KV Cache grows with context length and concurrent users, and memory capacity may limit commercial scaling earlier than model weights.

·Agents further amplify storage pressure. Multi-step calls repeatedly generate context, invoke external tools, and increase CPU, DRAM, and KV Cache demands.

·Several new hierarchical levels are emerging between HBM and traditional SSDs, including CXL, "Storage Next," and CMX, all aiming to cost-effectively handle the expanding inference data.

·Not all new technological pathways can be implemented. HBF, zHBM, NVHBM, ZAM, and PIM each face challenges such as cooling, yield rates, ecosystem compatibility, or realignment of supply chain interests.

·Bernstein continues to favor Samsung Electronics, SK Hynix, Micron, SanDisk, Seagate, and Western Digital, maintaining an "underperform" rating on Intel.

Over the past two years, the AI storage market narrative has largely revolved around HBM. However, as large models transition from training to large-scale inference, simply increasing HBM capacity has become insufficient to address all issues.

In its latest Global Storage report, Bernstein pointed out that different AI workloads have distinct memory requirements: training emphasizes computing power and bandwidth, the decoding stage in inference relies more on capacity, RAG necessitates extensive databases, and Agent workflows simultaneously add burden to traditional servers and AI servers.

This suggests that the changes brought about by AI are percolating down from HBM to system DRAM, SSDs, HDDs, and even tape storage. Around the "memory wall," the industry chain is beginning to insert new products between existing tiers, hoping to find a new balance among performance, capacity, and cost.

The Limit of Inference Scale Could Depend on KV Cache

During the large model training phase, GPUs and HBM remain central.

Training requires frequent reads of model parameters and intermediate data, demanding high computational power and memory bandwidth. However, training a large model does not solely rely on HBM: the raw dataset needs to be stored in lower-cost storage media; data usually undergoes caching and preprocessing by system DRAM and local SSDs before entering the GPU; and weeks to months of continuous training also require periodic checkpoint saving to prevent tasks from restarting due to hardware or software failures.

Therefore, a large-scale training session actually involves accessing the full spectrum of memory from HBM and system DRAM to local SSDs and network storage.

As we move into the inference stage, the memory requirements further differentiate.

Inference is usually divided into two stages: Prefill and Decode. Prefill is responsible for processing user input to generate the first Token, mainly performing large-scale matrix operations, leaning more towards "compute-bound" scenarios. During this stage, GPU utilization and HBM bandwidth become more crucial, with the key metric being the latency of the first Token.

Decode, on the other hand, is different. The model needs to generate Tokens one by one and call upon previously generated information when creating a new Token. To avoid redundant calculations, the system often stores this data in a KV Cache.

The KV Cache has two important characteristics: its capacity scales linearly with the context length, and each user needs an independent cache. Therefore, as the context grows longer and the number of concurrent users increases, these two factors together drive up memory usage.

Bernstein believes that in large-scale AI deployments, the memory occupied by the KV Cache may exceed that of the model weights, becoming a key factor limiting the number of concurrent users and the context window. The model's ability to serve users and maintain a long context will ultimately directly impact the revenue scale.

From this perspective, the competitive focus of the inference era is not only on how much computation the chip can perform but also on how the system can affordably store and retrieve the ever-expanding context.

RAG and Agent Push Demand Toward Conventional Memory

The proliferation of RAG and Agent further expands AI storage requirements beyond HBM.

RAG mainly consists of two stages: database construction and database retrieval. During the construction of the database, the system needs to process a large amount of unstructured data such as PDFs, web pages, and code, and then transform it into searchable vectors and indexes. This process relies more on high-capacity SSDs and system DRAM, with HBM playing a relatively limited role.

Once the database is set up, user queries are first converted into vectors, which are then matched against the contents of the database. While vector generation can be quickly done in GPUs and HBM, actual searching typically depends more on system DRAM. The retrieval results are then merged with the user query, entering the standard Prefill and Decode process.

The burden of the Agent workflow is heavier.

Traditional dialogue is usually a single-call "input-model-output" process, but an Agent needs to break down the goal into multiple steps, call other models or external tools, store intermediate results, and replan based on feedback. The result of each call may then become the input for the next model invocation.

This will simultaneously increase two types of requirements: on one hand, tool invocation and non-AI tasks will require more CPU and system memory; on the other hand, continuously passing context between multiple models will rapidly escalate the burden on prefilling, decoding, and KV Cache.

Therefore, the development of Agent applications is not only beneficial for GPUs and HBMs but may also drive the demand for server DRAM, enterprise-grade SSDs, and lower-cost storage media.

A New Memory Tier is Emerging Between HBM and SSD

The memory hierarchy of traditional servers can roughly be divided into processor internal cache, system DRAM, local SSD, and shared storage. AI servers have introduced HBM into this structure, but the limited capacity and high cost of HBM make it difficult to bear the entire dataset.

The current industry's solution is to introduce new products at different levels.

CXL aims to consolidate physically dispersed memory into a shared resource pool, allowing CPUs, GPUs, and expansion devices to flexibly access DRAM. Some products also use DRAM or SRAM as a cache, combined with NAND, to reduce costs while shortening access latency.

Driven by NVIDIA, "Storage Next" seeks to shift some storage management from the CPU to the GPU, enabling NAND to achieve latency, IOPS, and data access granularity closer to DRAM. The GP series SSD introduced by Kioxia based on XL-FLASH represents this direction.

CMX is primarily aimed at KV Cache. It deploys SSDs in separate data nodes, connecting them to compute nodes via DPU, Ethernet, and switching chips. Its goal is to share inference context among different GPUs, reduce redundant storage, and overcome the memory capacity limitation of a single server.

These solutions collectively point to the same trend: AI systems cannot keep all active data in HBM for an extended period. Based on data access frequency and latency requirements, the data needs to be distributed across different tiers.

Hot data remains in HBM, some context is moved to system DRAM or high-performance SSDs, and colder data continues to sink to regular SSDs, HDDs, or even tape storage. The finer the memory hierarchy, the more likely the system is to strike a balance between performance and cost.

New Technologies Abound, but Commercialization Still Faces Uncertainties

Around the "Memory Wall," the industry has proposed multiple new approaches.

Samsung's zHBM plan involves stacking HBM on top of the processor to further shorten the data transfer distance. However, this design requires managing the heat generated by the GPU, while also placing higher demands on wafer-level hybrid bonding yield and cost.

NVIDIA's driven NVHBM will entrust the base wafer to NVIDIA's design and may be manufactured by TSMC. This solution is expected to reduce power consumption, increase bandwidth, but it may also diminish the design and manufacturing value of storage manufacturers on the HBM base wafer. With the standardization of products, some added value may be shifted from storage manufacturers to NVIDIA and wafer foundries.

SanDisk and SK Hynix's promoted HBF aims to utilize NAND to provide HBM-like bandwidth while achieving larger capacity and lower unit cost. However, there is still a significant gap between NAND and DRAM in terms of latency and performance. HBF needs to span multiple technology levels, making implementation challenging.

Intel's ZAM attempts to rotate the DRAM wafer by 90 degrees to improve heat dissipation, aiming for practicality in the 2029 fiscal year; Qualcomm's HBC uses LPDDR and traditional packaging to bypass CoWoS costs at the expense of some performance.

In addition, PIM is trying to directly integrate computing capability into the memory chip to reduce data movement between the processor and memory. However, this will alter the existing computing architecture, requiring collaboration among processors, software, and networks, and will also impact the well-established division of labor between logic chips and memory chips. Bernstein believes that its industry adoption is still limited.

From this perspective, the rapid increase in the number of new solutions does not mean that all paths will lead to a scalable market. Whether they are compatible with existing software and hardware ecosystems, have cost advantages, and whether all parties in the supply chain can achieve a balance of interests will determine the ultimate commercialization outcome.

AI Storage Beneficiaries, Not Just HBM Manufacturers

From an investment perspective, Bernstein's assessment is clear: AI's pull on the storage industry is moving from a few high-end products to more tiers.

HBM remains at the core of training and high-performance inference, and Samsung Electronics, SK Hynix, and Micron will continue to benefit from high-bandwidth storage demand. However, as inference scales up, the importance of system DRAM and NAND will increase. KV Cache overflow, RAG databases, and the large amount of intermediate data generated by agents will also increase the demand for SSDs and shared storage.

Colder data will continue to sink. Bernstein noted that the data growth driven by AI has already started benefiting HDDs; in some scenarios, due to the insufficient capacity of NAND and HDDs, the demand for tape storage, traditionally used mainly for archiving, is also increasing.

The report continues to give Samsung Electronics, SK Hynix, Micron, SanDisk, Seagate, and Western Digital an "Outperform" rating. Among them, Samsung Electronics, SK Hynix, and Micron correspond to DRAM and HBM, SanDisk benefits from NAND and HBF, while Seagate and Western Digital relate to lower-cost high-capacity storage. Kioxia was rated as "Underperform".

However, the core value of this report is not to list a bunch of new technology abbreviations, but to redefine the boundaries of the AI storage market.

The bottleneck in the Training era is mainly focused on GPUs and HBM; in the Inference and Agent era, the bottleneck is beginning to spread across the entire memory hierarchy. The future competition in AI infrastructure depends not only on how fast chips can compute but also on whether data can flow efficiently at a low enough cost between HBM, DRAM, NAND, and shared storage.

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia