Big Data versus One Person

It’s not about pop culture, and it’s not about fooling people, and it’s not about convincing people that they want something they don’t. We figure out what we want. And I think we’re pretty good at having the right discipline to think through whether a lot of other people are going to want it, too. That’s what we get paid to do.

- Steve Jobs

Big Data, as far as I can tell, means collecting acres of information about something and then using it to help predict behavior. Like watching how millions of people use your website in order to decide how to change your website to sell more sardines. 

There are lots of people in love with this idea. I can understand why. It's alluring in its big black boxness, its promise that the algorithm will tell me something I couldn't think of myself. In some areas it has worked, like science and genomes for example. Google too, they use massive amounts of data all the time to make search better. In some areas it has failed. The financial crisis of 2008 was, to some extent, a result of big data going through a black box no one understood and then bombing globally, spectacularly.

I think the most important thing to realize about big data is that it's a strategy. Like all strategies, you use the right one to fit the crime. Wal-Mart and Hermés both have great selling strategies but it would be suicide to use one's approach with the other. Same for big data. There are times to employ it. Then there are times to run away.

Here's a good article on big data comparing it to intuition. And then, in contrast, an excellent piece on what you can do as one person to make decisions when you actually love to use the thing you make — or, in our case, love to eat the food you sell. (Note you can embrace or ignore the business mumbo jumbo in the article, there are other juicy bits littered throughout.)