The State of Storage Management

“Are we there yet?” my son of five, Jerome, kept asking me as we were driving to one of his favor places, Monterey Bay Aquarium, this past Sunday from our Silicon Valley home.  “We’re almost there… just a few more minutes,” I answer.

As the wheels were racing towards our destination, my mind drifted onto a topic I have been thinking a lot about lately: the current state of digital file storage management.  Are we there yet?  No, not really. Unfortunately, not even close.

Disk drive prices continue falling by the month, and for the past two decades or so capacity increases by more than double every year, on average. Combine this with the sky-rocketing demands for digital file storage, and you will see that the state of storage management is spinning out of control. What makes it worse is that no one seems to even realize it, or if they do, no one is really adequately address it—certainly not the major players in the IT industry.  No big deal. Disk drives are cheap, let’s just buy more storage.  Of course, this is exactly what the storage industries like to hear: buy more and buy again soon.  I almost think the lack of storage management features with all the storage products, applications, and operating systems are by design. The hardware cost is indeed very inexpensive, relatively speaking. But is that the only cost?

We have come a long way from the early days of computer storage.  In the early 1980’s  when I first worked on a Wang computer in Singapore, 10MB of disk drive storage took up the space of a small refrigerator and was as noisy as a small airplane taking off.  My first Apple computer did not even come with a disk drive or even a floppy drive. I had to save my files on a cassette tape using the same cassette player I used to listen to music — a lost art of computing this days.  Nowadays, we have more storage in an average PDA and some cell phones than we had 20 years ago in a Wang mini-computer!  We have more storage in an average digital camera than I had in my first Apple computer.  We have more storage in our PC than we had in our server just a few years ago.  And our servers today have more storage than a mainframe computer did just less than a decade ago.

This is all well and good until we need to find something—that picture we took last month on our vacation—or until the day that we wake up to find our disk destroyed by a virus or crashed.  No, that has never happened before to anyone we know.  Now, what if this is a business and not our home computer?  Think about it. How much storage does an average business of 50 users have today? How about 100 users, 1000 users, 10,000 or more users?  And where are those files—are they on my local disk drive or are they on network servers?  And what is the information these workers are storing? Likely data that is more important to their business than pictures of their last vacation.

As the computing hardware advancements race forward, software application at all levels have not been able to keep up. By my own estimate, software is at least 10 years behind hardware advancements and the gap is increasing each year.  One of the biggest issues is the lack of standards and discipline in software development as compared to hardware development, relatively speaking.  But that is a topic for another day.  The point here is how to manage storage well? The software layer is where the solution should come from, and I argue, operating systems are where the implementation should begin.

When it comes to operating systems and application development, the importance of storage management has traditionally be ignored and treated as a low priority in the PC world.  There are many competing file systems within all popular operating systems today, and none of them have good storage management built-in.  A few other software developers have seen the opportunity and developed add-ins, but none of these works with one another. Moreover, the features are too weak to be able to keep up with sky-rocketing storage demands.  Storage management largely exists within applications, not operating systems where it should be.  The issue here is if there are not storage management standards enforced by operating systems, applications that are then developed have no way of easily inter-operating together.

So, given this situation, what can one do to cope with the pending headaches and try to manage the unmanageable situation a little less painfully?

Let us first understand where the main issues are when it comes to storage management.  There are two contributing factors and three issues we are facing and then the real cost factors:

Contributing Factors to Increased Demand

	We are saving more digital data than ever and a good percentage of it is mission critical
	The low cost of the storage encourages waste and inefficiency

	There are no storage management standards, nor a disciplined approach
	We are facing mostly unstructured data types No priority No relationship No grouping No version control No central interface to find all data (including emails)
	File / Data are controlled by applications, not operating systems.  Operating systems today simply provide a place to store the files and link them together in a given isolated space, but storage management is not provided.  Individual applications are required to read or write the same file, and this is why we now have hundreds if not thousands of file types.  And there are other issues even deeper than this.

	Always separate application from data, and refuse to buy any application that cannot do so
	Convert unstructured data into structured data by using SQL technologies
	Group file structures so that the file structure makes sense to the users, but keep it simple at the same time
	Employ existing technologies to reduce the number of interfaces needed to access files
	Centralize data and use replication for mobile users only when necessary
	Use security to control the flow of data to help enhance the above practices and disciplines

	Time needed to create the file
	Time needed to find the file each time
	Time and resources needed to backup and store the file (if we are not already doing it, the cost will be even higher)
	Time and resources needed to keep the file secure (safe from man-made and natural disasters, including viruses, etc.)
	Time and resources needed to keep up with the changes in and maintain compliance with policies and regulations