Big Data is a hot topic. I went to a session on Big Data Analytics for Social Media at the recent Enterprise 2.0 conference. Computerworld reported on an IDC study that predicts we will see a 50 times increase in the world’s data in the next ten years. In 2011 alone they report that 1.8 zettabytes (or 1.8 trillion gigabytes) of data will be created. This is the equivalent to every U.S. citizen writing 3 tweets per minute for 26,976 years. Then over the next decade, the number of servers managing the world’s data stores will grow by ten times to match the 50 times increase in data. The report adds that IT execs will likely have trouble finding enough people with the skills and experience to manage this increase This is all covered in the fifth annual IDC Digital Universe study.

This data growth is fueled, in part, by the spread of smart devices such as sensors in clothing, medical devices, and structures like buildings and bridges. In addition, unstructured information – such as files, email and video – will account for 90% of all data created over the next decade. Some of this growth is through the rise of high bandwidth data such as videos.

There is some good news as new hardware and software has driven the cost of creating, capturing, managing and storing information down to one-sixth of what it was in 2005. This is likely why servers will only grow ten times while the data they store will grow fifty times. Relative costs have also dropped as since 2005 the annual investments by enterprises in hardware, software and cloud services technologies, along with the staff to manage information, has only increased 50% to $4 trillion.

The cloud accounts for some of the cost reduction and will account for more going forward. Today, cloud computing accounts for only 2% of all IT spending. However, by 2015, though, close to 20% of all information will be attached to cloud services some way, and as much as 10% will reside in a cloud infrastructure, IDC stated.

According to David Reinsel, IDC’s vice president of storage and semiconductor research, the next step is to enable companies to better extract value out of their mountains of data, via big data analytics. “This is where real opportunities lie, and where some folks may miss the boat. As soon as big data success stories are advertised and people see that there is gold in their data … then you will find more companies desiring to put more data online.”

Gartner also weighs in on this issue in a recent report. While the volume within big data is a significant issue, Gartner analysts “said the real issue is making sense of big data and finding patterns in it that help organizations make better business decisions.” I could not agree more.

Yvonne Genovese, vice president and distinguished analyst at Gartner: “The ability to manage extreme data will be a core competency of enterprises that are increasingly using new forms of information—such as text, social and context—to look for patterns that support business decisions in what we call Pattern-Based Strategy. Pattern-Based Strategy, as an engine of change, utilizes all the dimensions in its pattern-seeking process. It then provides the basis of the modeling for new business solutions, which allows the business to adapt…”

The ability to handle this explosion of data and make sense of it should be a priority of enterprises or they will be swamped in the next few years by both the data and their competitors.