Macadamian Blog

Product Management Research Pitfalls 3

Mistake #3: We’ve Gathered Tons of Data

WITH THE RISE OF ANALYTICS, this research trap has been growing. Tools like Google Analytics, Flurry, and Preemptive Mobile give you more data than you know what to do with, but that’s not inherently a good thing.

If you ask enough questions, and gather enough data, meaningless patterns will emerge.

Simple example: if you ask enough people what month they were born in, and a hundred other research questions, you will “discover” things like people born in November drink a disproportionate amount of Budweiser compared to everyone else, or that 86% of people with July birthdays like driving foreign cars over North American cars.

You haven’t actually discovered anything except a statistical phenomenon known as clustering. In a large set of random data, you get clusters of the same type of information. Roll a pair of dice 100,000 times and you’ll get strings of snake eyes or 12’s and other unlikely sequences. It doesn’t reveal a strange meaningful pattern. There are just a lot more clustered patterns than non-clustered possibilities out there.

To find out if the trend has meaning, you would have to run a separate follow-up experiment. Automatically assuming that a cluster of data has meaning — and making design decisions based on that — can lead you astray.

For example, a client once asked us why their website was so popular on Tues¬days, as shown by their demographic data. It didn’t make sense, so we asked them to redo the test. In the second test, the pattern disappeared.

When gathering web or mobile analytics, survey data, or any kind of quantita¬tive measure, your biggest asset is knowing the right questions to ask. Start with a theory that you want to prove or disprove, or use analytics and surveys to steer more detailed user research. For example, you could use survey data to get an overall preference for a feature, then use more qualitative methods like user interviews to understand that preference in context and detail.