The problem with data driven decisions

The mathematician searches around the lamppost on his hands and knees. “What are you looking for?” a bystander asks.

“My keys, I dropped them as I was leaving the bar” comes the reply. The bystander looks over his shoulder, “But the bar is back that way”, he says, pointing into the distance. “Yeah, I know, but the light is much better here” the mathematician replies.

We’ve all heard the joke. The point being that precision is only useful when you’re applying the right methods. This is what bugs me about designers and developers who are slaves to data but ignorant of their customers.

Data is Just One Perspective

There are numerous quotes about how important data is, and how decisions should always be backed by data. Data is one perspective. What your users are saying is another perspective. What you internally want to do is another. What makes financial sense is another.

For example if Apple were driven by data points, they would release a €400 netbook or shut down their Genius Bar after years of no activity. If Ryanair were customer driven they’d remove all their sneaky fees & charges. If Zappos were driven by margins, they’d abandon their generous returns policy. They’re all just perspectives. Just because data is objective, it doesn’t mean that it guides you to the right decision. Just because it’s precise, it doesn’t follow that it’s valuable.

To make a decision you gather the perspectives that matter to you, weight them according to your judgement and then make your call. When we elect presidents, choose companies to work for, or even CEOs to admire, it is because we’re impressed with how they make decisions.

Rich in Data and Poor in Insight

This is what frustrates me about those who put data above all else. Data is a false God. You can tag every link, generate every metric, and run split tests for every decision, but no matter how deep you go, no matter how many hours you invest, you’re only looking at one piece of the puzzle. Besides, has anyone ever A/B tested the practice of running A/B tests? There could be better, albeit less measurable things, for a development team to do.

Recall the mathematician and the lamppost. You can be super efficient and very comprehensive but still be looking in the wrong place. It’s common to remark “the plural of anecdote is not data”, but similarily you can’t mistake precision with value. Not everything that can be counted counts, not everything that counts can be counted. This is especially true for start-ups.

Data Driven Start-Ups

If you’re entering a well defined market, looking to disrupt, then you should know that the success of your company won’t be measured by the same metrics used by the big players in the market. Thus it’s hard to know what data to look at.

Alternatively if you’re establishing a new space, or new type of app it’s near impossible to know what metrics you should be optimising for. Sure you study your signup funnel, optimise your landing page and simplify the onboarding, that’s the easy bit. The real challenge is working out how to turn sign-ups into serious long term customres. The relevant metrics for this only emerge over time. Until then you can stare aimlessly under streetlights, or start talking to customers.

The highest rewards come from the biggest risks. Sometimes you need to be willing to say “We believe in this, Fuck what the data says”. That’s when things gets exciting.

You can innovate or you can predict and measure performance, but not both. Which will it be, Mr Businessman?

— Alan Cooper

Photo credit Sam Hames