You are excited, you turned your dreams into a product. Now you sit and watch as new users trickle in. One by one, they start using your product. You see more users in your database; maybe you see orders being placed. Great! Now what’s next? Is there anything you can do to make the user experience better? Maybe that button should really be blue? Does there really need to be an intro video? What if the loading screen is taking too long for the users in India? Answers to these questions and many more can be derived from data. The problem is, you need to know what data to collect and how to collect it. More importantly, you need to know how to look at the data and how to reason with it. In doing so, it is vital to keep one thing in mind: The only thing worse than not having data is drawing bad conclusions from bad data.
Seven years ago, I was sitting at a research lab in Tokyo working on ways to improve the allocation of security guards all over Japan. From that moment until today, I have had the good fortune of working on various problems and building data-science teams across different companies. I have seen and learned from various data pitfalls and want to share some of the most interesting and impactful ones here.
Related: 3 Ways Scrappy Entrepreneurs Can Keep Data Scientists on Board and Motivated
Be careful of biases.
Humans are creatures of habit — and biases. For example, we routinely pick data that fits/confirms our preconceptions (confirmation bias), choose data that may not be representative of the user base (selection bias) or focus on the most obvious attributes of the users/products ignoring anything else which be the actual cause of changes (salience Bias). Some of these biases can have tremendous effects on evaluating performance.
The quickest remedy here is to take a step back and see if the population you are considering for your analysis is representative of the overall population you are targeting. If any subpopulation of the users are not represented or their ratio in the overall population is different than what is observed in testing, it is time to reconsider how the users are sampled for the test.
Learn to look at the ‘right-size’ picture.
Having a clear view of the big picture is always important but depending on the problem in hand, it can also be misleading.
Consider the case where we are trying to optimize for new-user conversions, and we have an app that runs both on iOS and Android. Overall, we may see that it has 20,000 users, out of which 10,000 converted and 10,000 were lost. Then we dig a bit deeper and look at each individual platform. We may see that on Android, we are doing really well, with 400 conversions to 100 users lost, while on iOS, it is basically even for conversions and for users lost. But if we had only looked at the overall picture, we would have lost the fact that whatever we are doing on Android is working. This effect of having a seemingly obvious overall trend disappear or reverse itself when looking at individual groups vs. the combination of all those groups is called “Simpson’s Paradox,” and it shows up more often than one would expect.
Related: 4 Tips to Landing Your Dream Job in Big Data
Make sure you have enough data to make informed decisions
It is common to look at changes in user behavior and assume that the recent change you have made is responsible. But how do you know it was not based on sheer luck? This concept of statistical significance can easily lead to bad decisions that get made, as the observed changes are likely to be purely based on chance. Therefore, they may not only be ineffective but also have the very opposite effect of what was originally desired.
After all, you wouldn’t make a decision based on a single user, but how about 10, hundred, or even a thousand users? Is there a magic number? As it turns out, there isn’t, and this really is a bit of a trick question. It really depends on what you are trying to track and compare. If you are going to be tracking users for a long period of time, you can work with a smaller set of users. Or if this is an event that happens very often, then you can also work with a small set of users.
Data can be your biggest ally and your most versatile tool when finding the right direction for your product. However, just as good data can be of great value, bad data can be hugely detrimental. When you are faced with making a data-driven decision, ask yourself just one question: Am I looking at the right type of data, representing the right population of users, and do I have the right amount of it in order to see significance? Using this as your guidepost will hopefully lead you to the right outcomes.
Related: 4 Ways to Construct a ‘Data-Innovation’ Map for Your Business