Macadamian Blog

Product Management Research Pitfalls 4

In case you missed the introduction to this series, I’ll be identifying one product management user research pitfall per post. Missed the previous mistakes? Catch up!

Mistake #1 Mistake #2 Mistake #3

Mistake #4: Misunderstanding Statistical Significance

CLIENTS SOMETIMES ASK FOR MORE DATA to prove a data point is “sta¬tistically significant,” or believe they have a meaningful result because some¬thing is “statistically significant.”

Research methods such as surveys demand a certain amount of rigor to prove statistical significance. However, when it comes to testing how users actually interact with the system, statistical significance is far from a sufficient criterion for meaningful or proper research on its own.

On the other hand, you can’t draw conclusions from interviewing only two or three users. So how many users do you need to speak with or observe? It depends on the range of user groups your product is targeted to, the scope of product interactions you want to observe, how the results will be used, and how many rounds of research you conduct.

If you want to test a software application for usability and opportunities to improve the design, focus on one or two primary user groups, and five to six users each time will detect most usability issues.

Jakob Nielsen has shown that approximately five to six users will likely detect 80% of usability problems for a specific use of a product. Keep in mind this “formula only holds for comparable users who will be using the product in fairly similar ways.” (Jakob Neilsen, Useit.com. 2000, 03, 19. Why You Only Need to Test with 5 Users)

Laura Faulkner tested Nielsen’s theory on a web-based employee timesheet (Faulkner, L. Beyond the Five User Assumption: Benefits of Increased Sample Sizes in Usability Testing. Behavior Research Methods, Instruments and Com¬puters (Volume 35(3) 379-383)). She ran tests with a group of 65 users, then selected random groups of five participants and compared the percentage of issues each group of five detected compared to issues detected by all par¬ticipants. Nielsen’s 80% rule held, but variability meant some groups of five identified as few as 55% of the issues. By increasing the number of users to ten, groups found an average of 95% of issues.

Ultimately, the number of users you need to draw on depends on what type of information you want to get from your users. For a product benchmark study, you’ll need to run with a much greater number of users than a usability test, and you have to make sure you test across all user groups. For customer inter¬views, the number can vary significantly.

Many important research goals can be satisfied with a limited amount of research from a limited number of participants. We encourage clients to seek advice from a User Experience professional with a background in experimental practices to help determine the extent you need for your product goals.