The paradox of big data: It's only useful if you break it into small data

Credit: BooostedAWD
Credit: Allen Bonde
Allen Bonde, Digital Clarity Group

Big Data is all the rage right now. It is seen by many as the key to business decision nirvana, but many companies struggle to understand how to use that data to make better decisions.

Allen Bonde is principal analyst at Digital Clarity Group and an adviser to Art+Science Partners. He has also worked as chief marketing officer for three companies and was an early proponent of data-driven marketing. 

Bonde has said part of the problem with big data in business today is that it has for the most part remained the province of data science geeks. For real business people to take advantage of this data, we need to break it down into the last little bit that actually helps make a decision -- what Bonde calls "small data."

I interviewed Bonde and asked him about his views on big data and how it can help organizations. Here are some highlights:

  • Companies actually need to concentrate on small data. "The idea of big data has been around for a couple decades (see timeline here), yet it’s still generally the domain of data scientists and technologists -- and hard to apply to everyday tasks, by everyday people. In a lot of ways, big data is all about volume and machines and processing power, while small data is about end-users, context, and individual requirements."
  • Marketers and business users need to drive big data initiatives -- not data scientists. "It needs a purpose. If it’s primarily the domain of a bunch of PhDs in a lab and there’s no near-term payoff for others in the organization, it’s not going to click.... Marketers need to step up and play a more active role in defining and driving big data initiatives. And of course they must demand personalized analytics tools that are tailored to business roles versus technical users, and integrated platforms that make it easy to access, test, apply, and share insights in the stream of their daily tasks."
  • How to avoid the bias trap that can arise from too much data. "If we start with a purpose, assess the core question we are looking to answer, think about the right data that is needed, and then work our way up the funnel to the models and sources needed, there’s a good chance we can limit our search space and prune off the sources that are not relevant to the problem at hand."
  • It's wrong to write intuition completely out of the equation. "If we are talking about areas like process automation, computer trading, or even determining which prisoners should get parole...I agree it’s generally all about the data -- and of course the algorithms or rules that have been set up by humans! But in more complex, less outcome-driven scenarios I think we need more of a balance of art and science involving creativity/judgment and insights. Particularly in cases, say in marketing or executive decision-making, where our decisions are influencing other human decisions. 

The full interview is below:

Ron Miller: You've written about the concept of small data versus big data. How do you define the difference between the two.

Allen Bonde: The idea of big data has been around for a couple decades (see timeline here), yet it’s still generally the domain of data scientists and technologists -- and hard to apply to everyday tasks, by everyday people. In a lot of ways big data is all about “volume” and machines and processing power, while small data is about end-users, context, and individual requirements.

For enterprises that already have big data initiatives, small data may become the last mile of their efforts, that turns large data sets and discovered insights into actionable alerts, apps, or dashboards -- for the rest of us. More specifically, in a recent DCG study we offered this definition: “small data connects people with timely, meaningful insights (derived from big data and/or local sources), organized and packaged -- often visually -- to be accessible, understandable, and actionable for everyday tasks.” This last part is key: to be valuable, your data (whether big or small) has to be actionable!

RM: We've been hearing about Big Data for a number of years now, but it has yet to click in a big way with organizations. What in your view is holding it back?

AB: I think there are two primary reasons, both related to the ideas above. First, big data is a big idea -- really a vision -- that needs to be attached to specific problems, front-line users, and everyday tasks in order to demonstrate value. It needs a purpose. If it’s primarily the domain of a bunch of PhDs in a lab and there’s no near-term payoff for others in the organization, it’s not going to click. Second, and this perspective comes from my own experience as a data scientist who became a CMO, is that marketers need to step up and play a more active role in defining and driving big data initiatives. And of course demand personalized analytics tools that are tailored to business roles versus technical users, and integrated platforms that make it easy to access, test, apply, and share insights in the stream of their daily tasks.

RM: Last week, there was a story in the Wall Street Journal in which a former NSA official was quoted as saying the agency was drowning in data. Is there such a thing as too much data and how do you prevent that from happening?

You can definitely have too much data, which is a trap in traditional big data thinking. Nate Silver actually talks about the problems with really big data, in terms of introducing too much noise and losing track of the signal. He highlights the risk when there is so much information to choose from, and we pick the parts we like and ignore the rest, which introduces real bias -- unintentionally or not -- into our analysis. In a way, taking a bottom-up approach, informed by a small data perspective, is one hedge against this challenge. If we start with a purpose, assess the core question we are looking to answer, think about the right data that is needed, and then work our way up the funnel to the models and sources needed, there’s a good chance we can limit our search space and prune off the sources that are not relevant to the problem at hand.

RM: MIT's Andrew McAfee has been preaching about the value of data-driven decision making over using your gut. Do you agree that it should be all about the data or is there a role for human intuition in interpreting the data?

It depends on the application. If we are talking about areas like process automation, computer trading, or even determining which prisoners should get parole (an example he uses) I agree it’s generally all about the data -- and of course the algorithms or rules that have been set up by humans! But in more complex, less outcome-driven scenarios I think we need more of a balance of art and science involving creativity/judgment and insights. Particularly in cases, say in marketing or executive decision-making, where our decisions are influencing other human decisions. 

RM: Can you provide an example or two of your favorite big data applications and how they help users make sense of a lot of data?

In the business sector, I see a lot of potential in medicine, especially on the payer side. There are some good examples in this slideshow from CIO.com. On the consumer side, I really like travel-site Kayak’s “when to book” tool since it boils down data from a billion search queries along with fare history data and generates a simple “book now” or “wait” recommendation. A brilliant example of harnessing big data to create small data, applied in a helpful way for an everyday task.

Free Insider Download: CITE presentations now available
Join the discussion
Be the first to comment on this article. Our Commenting Policies