How much data is enough?

In the past, the only way of storing data was in your head. So, in less developed societies, older people were revered because their heads contained information about how to deal with problems. They remembered how the tribe or village survived last time everywhere flooded or the last major drought. They knew high spots or springs that don’t dry up.

Many societies have tried to make this information available to others by creating songs or poems that recite the lore – so that future generations will not forget the deeds of great heroes, but, more importantly, the list of tips and tricks for surviving in difficult situations.

Then came writing. There was now the ability to list the kings, so it was obvious who should be the next ruler. Or to list the goods that were to be bartered for some other set of goods. Or to record just how many sheep and goats someone owned at the end of each year. And writing in stone, which could last for many years, was later replaced by writing on papyrus, and eventually on paper. With the advent of printing, it became possible to create definitive lists of laws or stories that everyone had access to (well, provided they could read!).

And in our headlong rush through time, we get to the early computers. People could store information on them – like the milk yields of their cows and how much food the herd consumed, or how many coffees were sold in each coffee shop in a chain, or what you wanted for Christmas. And not only could you store data, you could calculate answers to questions. But it was all pretty much driven by humans for humans. And generally you knew where it was stored – even if it was on a mainframe in a different country, you could find the address of site.

But now two of those factors are changing, and changing in a very big way. Now, a lot of the data isn’t created by humans, it’s coming from devices. Let me expand on that. Every time you use your credit card, information about your purchases and where you purchased them is very likely stored somewhere, so the retailer can target adverts at you, and so they have a better idea of what types of product sell well in different parts of the country. Requiring even less human intervention is all the footage that gets stored from security cameras. No humans are involved in the recording and storage. And there are many other examples of devices that just store information. As time goes on, and more-and-more information is stored, there’s going to be a requirement for more-and-more storage. I’m sure someone somewhere has worked out the numbers, but if 500GB of storage occupies half a square foot, and the amount of data stored doubles every year, then in 20 years time, the whole planet will be covered in storage devices to the depth of 18 inches!!

But where is your data? Originally it was on your hard drive or on DASD in your organization’s buildings. But nowadays, I have most of my customer-facing files on Google Drive, and, so long as I’m in a wifi area, I use the CloudOn app on my tablet to show customers PowerPoint presentations as well as Word documents and Excel spreadsheets. I also use Dropbox for sharing files with friends. The point I’m making is that I have no idea where those files are physically stored. And, for business files, that brings a huge security issue. For example, European organizations are pretty much prevented from using US-based, or indeterminate location, because they’re outside the EU.

It seems at this rate that there will always be more data to store (somewhere) – and hopefully there will always be enough storage, without covering the whole planet. But will there be any more knowledge in the world?