In a recent post, I mentioned that VCs have been pumping more than a billion USD a year into the storage market for the past few years. That makes perfect sense to me. Almost nothing has changed for storage architectures in the past three decades. The SCSI standard was adopted in 1982 and still forms the basis of storage architectures.
1982 was a long time ago! In 1982, there were less than 10 million PCs in the world and the world’s first ‘mobile’ phone came out in the form of a 22-pound car phone. Today we are trending towards TWO BILLION PCs and there are more mobile phones than human beings.
Yet here we are 33 years later, and SCSI (or SCSI variants, such as SAS and SATA) is still the dominant interconnect for storage. Little has changed.
What has changed dramatically…especially lately…are underlying use cases for storage. Look at these trends shaping our technology world today:
- Internet of Things
- DNA sequencing
- Adanced credit card fraud detection
- The use of digital animation in feature-length films
Who would have imagined any of these things in 1982? And what are these trends driving in terms of our storage needs?
- The amount of information we need to store (and access) has grown astronomically
- We see a clear stratification between speed and volume. Some applications are all about really fast access to information (financial, for example). Other applications require massive volumes (Big Data).
- Location matters. In 1982, location was simple – in your data center. Today choose between the data and the cloud, basing your decision on the needs of your application.
In short, a LOT has changed since 1982.
Let me give you one more example of how things have changed. In 1982, a disk drive failure was a disaster; a memorable event that required ‘all hands on deck’ attention. Today within a hyper-scale infrastructure, such as those at Google and Facebook, somewhere within their vast data centers full of disks, servers and switches, there are multiple failures each hour and yet, their service hums along. It is no longer a black swan experience: It is something to be taken in stride.
How Storage Needs to Change
So, I could continue to rant about how antiquated storage architectures are, but let me be more constructive and tell you how I believe storage architectures need to change to keep up with changing use cases.
First, we need to tackle storage tiering once and for all. No, not the complex, tower-of-Babel HSM monstrosities that never worked for anyone. The web-scale guys have solved tiering for themselves with ultra-simple “fast stuff goes here, everything else goes there” architectures and easy (even self-service) tools to move data back and forth. I believe these solutions will filter down to the rest of us over time.
Second, we have to start adding meta-intelligence to our data files. This is a tricky one: Microsoft tried with WinFS and failed. The key is a solution that won’t get in the way for heads-down, performance-driven use cases but provides the granularity and context necessary to manage massive stores of dissimilar data.
And, finally, we need to move to a fault-tolerant model for storage infrastructure. Failures need to be absorbed and healed automatically with zero human intervention and zero performance degradation.
Anyway, that’s my view. What do you think? Let me know how you think storage architectures need to change to work better for you.