Macadamian Blog

Product Management Research Pitfalls 7

In case you missed the introduction to this series, I’ll be identifying one product management user research pitfall per post. Missed the previous mistakes? Catch up!
Mistake #1 Mistake #2 Mistake #3 Mistake #4 Mistake #5 Mistake #6

Mistake #7: Misuse of A/B Testing

SOMETIMES DESIGN QUESTIONS END UP in an internal debate between proposed solutions. The lead architect is convinced the features should look or function one way, and the product manager has a different theory.

A situation like this can be a good candidate for A/B testing, a fast form of user research where you launch a few designs to different user groups, perform some testing, and compare the results.

A/B testing measures which of several designs produces the most conversions, fewest clicks, fastest time, most intense emotional response, or whatever met¬ric you decide to measure. Online testing tools like verifyapp.com take advan¬tage of crowdsourcing to make this process even more convenient.

But you need to be aware of the drawbacks of A/B testing. First, you have to rely on your best guess as to the real reason why Design A performed better than Design B. You don’t get any feedback on whether or not the user “gets” the system. (This makes it hard to stay in the problem space.)

Also, you can inadvertently commit yourself to a non-optimal solution. Incre¬mental A/B testing finds the solution that, relative to other presented solu¬tions, produced the best results. However, because you always test one solu¬tion against the others, you run the risk of getting stuck with the best you’ve got, not the best possible solution.

A/B testing should be used as a quick cheat, a complement to other forms of research like the contextual interviews, concept walkthroughs, and usability testing that we described earlier.