5 hours Ago By Chris Mellor
: Gen AI is washing through the IT world like water flooding across a landscape when a dam breaches. It is being viewed by many suppliers as an epoch-changing event, similar to the first arrival of the Internet and, feared by some as a dotcom bubble-like event in the making. Be that as it may, IT storage is being strongly altered by AI, from the memory-storage interface upwards, as the rising tide of Gen AI lifts all the storage boats.
A look at the storage world reveals six ways Gen AI is transforming the world of block, file and object storage. At its lowest level, where storage systems talk to a server’s memory, the arrival of Nvidia GPUs with their high-bandwidth memory (HBM) has put a premium on file storage system and array controller processors and their DRAM just getting out of the way, and letting NVME flash drives connect directly to a GPUs memory to transfer data at high speed using remote direct memory access and the GPUDirect protocol. DapuStor, DDN, Dell, Huawei, IBM (Storage Scale), NetApp, Pure Storage, VAST Data, WEKA,YanRong plus others such as PEAK:AIO and drive slinger Western Digital (OpenFlex) are active here and even Nutanix aims to add GPUDirect support.
This style of unstructured data transfer is being extended to object storage so that the petabytes of data fenced off in that storage reservation can be freed up for use in AI training and, subsequently, inferencing as well, when that is done by GPUs. See the news from , and in recent months. Storage media manufacturers are reacting to Gen AI too.
The SSD manufacturers and NAND fabricators are recognizing that Gen AI needs fast read access to lots of data, meaning they better produce high-capacity drives, 62TB and, latterly, 123TB SSDs, using affordable QLC (4bits.cell) 3D NAND. Gen AI training also needs fast job checkpointing to enable quicker training job restarts.
Solidigm recognized this need early on with its 61.44TB D5-P5336 drive in July 2023 and has been followed by Micron, parent SK hynix and Samsung, and Phison has entered this market, matching Solidigm’s latest drive with its own Pascari D205V 122.8TB.
We will probably see news of double that capacity late this year or early in 2026. The use of hard disk drives (HDDs) for AI training is not happening. They are too slow and the drives too limited in capacity.
Where GPUs are used for AI inferencing SSDs will certainly be the storage choice as well, for the same speed and capacity reasons, and that will likely be true for x86 servers too. It’s likely that AI PCs, if they take off, will all be using SSDs and not HDDs for identical reasons. What this means is that HDDs will only be used for Gen AI secondary storage and, so far, that has not happened to any significant degree.
Seagate,Western Digital and, no doubt, Toshiba, are pinning their hopes of HDD market expansion on Gen AI data storage needs, and seem confident it will happen. The tape market has not been directly affected by Gen AI data storage needs at all, and likely will not be. Above the drive media level in the storage stack, we have block array, filer and object storage systems.
The filer suppliers and, as we saw above, object storage suppliers nearly all been affected by enabling GPUDirect access to their drives. Several have built AI-specific systems, such as Pure Storage’s offering. VAST Data, DDN, WEKA and others have shown sales increases by having NVIDIA SuperPOD certification.
With GenAI chatbots being trained on unstructured data, transformed into vector embeddings, there has been no GPUDirect-like access provided for block storage, for the great mass of transactional data bases and ERP systems. There is activity in the Knowledge Graph area to enable such data to be made available for AI training, witness and . Storage array suppliers, as well as data platform suppliers are all transforming their software to support the addition of proprietary and up-to-date unstructured data to help with, to augment AI Inference by GenAI’s Large Language Models (LLMs) trained on older and more general data.
Such data has to be vectorized and the resulting vectors stored in a database for use by the LLM in Retrieval-Augmented Generation (RAG). Existing non-RDBMs database, data warehouse, and data lake suppliers are adding vector storage to their products, for example . Database startups like Pinecone and Zilliz have developed specialized vector databases, promising better performance and supportive facilities for LLMs.
The data warehouse and lakehouse vendors are in a hyped up frenzy of Gen AI-focused development to be the data source for AI training and inference data. The highpoint of this was Databricks getting a $10 billion VC investment late last year to continue its GenAI business building evolution. A fifth storage area affected greatly by GenAI is data protection, where vendors have realised that their backup stores hold great swathes of data usable by GenAI agents.
Vendors like Cohesity, Commvault and Rubrik are offering their own AI agents, like Cohesity’s Gaia and also developing RAG support facilities. In general no datastore vendor can afford to ignore RAG, as it’s presumed all data stores will have to supply data for RAG. Supplying such data is not as simple as giving API access to an LLM and stepping aside, letting the model extract whatever data it wants.
An organization will generally have many different datastores and enabling their contents to be appropriately filtered, excluding totally private information or data below an accessing LLM’s access privileges, will need the GenAI equivalent of an extract, transform (into vectors) and load (ETL) pipeline setup. Data management and orchestration suppliers like Arcitecta, Datadobi, DataDynamics, Hammerspace and Komprise are all playing their part in mapping data sources, providing a single virtual silo, building data pipelines and so feeding the data they manage to LLMs. Data storage suppliers are also starting to use GenAI agents inside their own offerings, to help with support for example, or to simplify and improve storage product administration and security.
This will affect all suppliers of storage systems and will be transformed by the use of GenAI agents; think Agentic AIOPs. The cyber-resilience area is going to have to withstand GenAI agent-assisted malware attacks and will certainly use GenAI agents in building responses to such attacks. We are going to see the ongoing transformation of the storage world by GenAI throughout 2025.
It seems unstoppable and should be, apart from malware agents, beneficial..
