Data Deluge needs Smarter Storage

During the early days of my professional life, I was fortunate to witness the advent of Personal Computers (PCs). I was excited about the impact it could have on our lives. When PCs began to get a wide customer acceptance, the question was more on how much of hard drive is required. Discussions veered around 20 or 40 GB as perhaps the right requirement. Today, we are talking in the order of 800 GB, a 1 TB or perhaps more in some cases. In short, our ability to anticipate and accommodate data deluge shapes most of our buying decisions for any consumer device.

Digital data is being generated in mind-boggling quantity. This growth shows no signs of stopping. Data volumes in general are doubling every 18 to 24 months. Some analysts expect the annual worldwide information growth rates of 50 to 100 percent to be the norm as businesses and economies recover. And in just three years, the world’s annual IP traffic is expected to total more than half a zetabyte.  (That’s a trillion gigabytes — or a one followed by 21 zeroes.)

So, where is all this massive amounts of data coming from? While analyzing it and making sense of it comes much later, storing comes before all else. Each of us generates tons and tons of data, both in our personal and professional lives. While tools and applications at work help us to manage our data (which is in the form of mails, documents spread sheets, presentations), personal data has some options to store. With attractive offers, mail service providers and Internet companies, today, offer a route to store all your data at a premium. The easiest thing to do is to perhaps buy devices like hard drives and flash drives to carry all your personal data with you.

An IBM Study to understand the massive amounts of data being generated revealed the following:

• Number of Emails sent per second – 2.9 million
• Data Consumed by households each day – 375 MB
• Video uploaded to YouTube every minute – 20 Hours
• Data processed by Google per day – 23 Petabytes
• Tweets per day – 50 million
• Total minutes spent on Facebook each month – 700 billion
• Data sent and received by Mobile Internet Users – 1.3 Exabytes
• Products ordered on Amazon per second – 72.9 items

Dealing with huge volumes data would be challenging enough; what makes it even more daunting is that information isn’t just growing, it’s changing! Unstructured data comes from sources like emails, documents, spreadsheets, blogs, web pages, images, audio files and video files—and it now constitutes upto 80 percent of data. Because it doesn’t conform to traditional formats, it requires someone or something to interpret it in order for it to be transformed into actionable information that organizations can use.

Organizations are not only dealing with growth in structured data that is neatly stored in fields in databases, but also with unstructured data. In order to make sense of all the data at any given point in time, it is important that we adopt smarter ways to store it.

Smarter Storage can be defined as the adoption of best practices that ensure all your data is not only useful at any given point, but is also safe and secure for future use. From a design standpoint, we must always look for efficient devices that are self optimizing and cloud ready in nature. Today, hard drives are shrinking in size and hence, easy to carry. So, it is important to tier your data as hot and cold. For instance, while replicating data to different devices all those key pictures or mails that are frequently accessed can be part of your computer’s hard drive. Those that are rarely accessed could be stored externally. While enterprise storage has varied options for tiering data basis the priority, we need to manually do this in the case of consumer data. It is important to undo the habit of dumping everything into one common hard drive.

From a best practice standpoint, there are several things that come to mind. It is important to not share personal information on various web-based on social applications (like Facebook, Twitter etc) for you will never know where it can be used. Although changing password frequently can be cumbersome, it is perhaps the simplest and an effective way to keep your data secure.

Further, encrypting sensitive data on your laptop is a must, especially if you are someone who travels often. Various firms offer encrypting options to help here. Off late, there are several advertisements that hit radio and TV spots about security software available for safeguarding data. Investing in good software and more importantly, renewing security software on time for a nominal amount is critical. While purchasing online, one must look for those secure tags (those that have an image of a lock) next to the websites, especially when entering credit card details.  Since most of us are connected through wireless networks at work, it is equally important to look for secure networks while on the move. Remote wiping of data (an option available for most smart phones today) is also the best way to ensure that data never falls into the wrong hands. Lastly, wipe that recycle bin clean, periodically.

With cloud being another safe option to store your data, the key here is to clearly differentiate what you store on an external hard drive versus electronic storage. Looking for the right vendor is important. Of course, storing options of this nature comes at a price!

It is expected that Consumer and Enterprise spending is likely to remain flat for some time because of uncertainties about world economic conditions. Continued explosive data growth as well as the number of producers of data (including sensor based) as well as increased number of consumers (those who use, i.e. consume, the newly-minted information) leads to the famous do-more-with-less dictum. But that is not all. The possibilities of taking advantage of the multitude of benefits that a digital device can deliver remain grossly underdeveloped. Smarter matters, if the information infrastructure is to be exploited to the needed degree. Smarter Storage is a must!

By Ajay Mittal, Director, Systems & Technology Group, IBM India/South Asia

Disclaimer: "The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions."

The thoughts and opinions shared here are of the author.