The Cranky Admin

The Looming Storage Crisis

The "perfect storm" of events that led to the current shortage.

• By Trevor Pott
• 01/31/2017

The new year sees us smack in the middle of a storage crisis. This has been building for some time, and several things have conspired to create a perfect storm for storage. Naturally, this will affect everything else in the datacenter.

First up, flash has been doing a great job for almost a decade now. Piece by piece it's been whittling away high end hard drive sales. This has had the domino effect of reducing both R&D spend by HDD manufacturers as well as investment in production facilities. It is generally accepted that 2017 will be the last year 15K RPM drives are produced, and that 2018 will probably see the last of the 10K RPM drives.

Tipping Point
Over the past 18 months, flash got so good that everyone started buying it, and everyone started selling it. The storage industry reached an inflection point, and there has been no going back. By the middle of 2016, the sweet spot was the Samsung 3.84TB SSD. Everyone seemed to want these to form the backbone of their storage offering.

Several large datacenters went up in 2016 and are going up in 2017. They bought up all the 3.84TB SSDs. This created a crisis for companies like Dell, who are having problems sourcing the drives, let alone selling them in any quantity to customers. The trickle-down affect rippled down the supply chain such that supply problems started hitting even smaller drive sizes.

In an attempt to keep large customers happy (and not lose market share), each of the major NAND suppliers decided to tuck into the reserve supply of NAND chips they maintain to smooth out fluctuations in market demand. Demand didn't end up dropping, and all of the manufacturers exhausted their reserves at roughly the same time (mid 2016).

Surging Demand
Inability to source SSDs drove rising demand for mechanical hard drives, but the HDD manufacturers had allowed much of their production capacity to lapse. Even if they wanted to rebuild that capacity (and they don't, because this is likely just a temporary bump in demand,) they couldn’t do so quickly enough. By the time they could tool up enough to spit out the machines that spit out hard drives, the surge in demand would be dealt with by adding flash production capacity.

Somewhere in here, demand for RAM flew through the roof, creating pressure on precious fab capacity that otherwise could be making NAND flash. This has affected not only current production, but caused some reconsideration in the RAM/flash split of future facilities.

Disk Death
The result of all of the above is that SSD prices have already gone up, and will go up rather a lot more in 2017. So will disk drive prices. This is good news if you make SSDs (in the short term). It's terrible news if you make hard drives (in the medium term). This is largely because the additional flash capacity being brought on-line in response to all of the above is the nail in the coffin of disks.

While we may never fully have the planetary fab capacity to meet global storage demand, there are rather a lot of expansions and outright new fabs being thrown up. Combined with the "high capacity, low write life" of 3D NAND, it looks like semiconductor-based storage could kill hard drives outright within five years.

In the short term, there will be a huge demand for NAND. All parties are responding by building fab capacity. This transformation is, however, coinciding with the transition to 3D NAND. While some non-3D NAND fabs will be required in order to provide high write life devices, nobody really knows what this is all going to look like in 2019, after the majority of the 3D plants are online.

Global Uncertainty
How many new datacenters will be in demand? If U.S. President Trump tries hard enough, he could destabilize the entire global economy, causing a collapse in demand for IT. Alternately, the rise of nationalism could cause the balkanization of the Internet, driving a frenzied creation of nation-local datacenters.

President Trump, U.K. Prime Minister Theresa May, Prime Minister Matteo Renzi in Italy and whichever party is running Australia this week are inherently unpredictable. There are also major elections in France, Germany and The Netherlands in 2017 which threaten to add yet more populist players to the table, all of which have an interest in destabilizing existing global power structures.

The net result of this is that just as mechanical disk evaporates as top revenue source for companies like Toshiba, Western Digital and Seagate, the basket into which they have so carefully put their eggs has become unstable and unpredictable. Making raw storage is no longer a sure-fire route to continued revenue growth, let alone vast riches.

To deal with supply uncertainty, as we move from an industry based on mechanical hard drives (which has dedicated production facilities) to one based on commodity NAND, vertically integrated solutions will be optimal. Organizations that control everything from NAND supply to controllers to the software will be in a much better position to deliver consistently than those that don't.

An Array of Crises
This may cause an existential crisis for the external storage array. Creating, validating and successfully marketing a new external storage array in a saturated market is difficult. It is unlikely today's storage vendors will be trying to move up the value chain by reinventing the array.

Hyper-converged solutions, on the other hand, are relatively easy: there are a whole lot of smaller hyper-converged players that can be bought up cheaply and turned into the basis for a storage vendor's vertically integrated play.

For the small virtual administrator, none of this may be relevant. Our needs are simple and we should be able to find storage even if supplies become a little tight. If, however, you measure your datacenter in acres, by the end of next year you may well find yourself negotiating for your virtual infrastructure from a company that last year you would have thought of as just a disk peddler.