Macadamian Blog

A little knowledge is a dangerous thing: Poor research and its cost to EMR redesign.

We've talked in the past about the importance of research in improving a product's user experience and usability. In the healthcare field this is especially critical. Thankfully we see more and more companies engaging in getting user feedback. Unfortunately, since research is a highly specialized skill set, we are also seeing a proliferation of mistakes and misinterpretations in some of the more technical aspects of research, being made by people who don't have specific training in usability research or a general foundation in experimental practices. Here are 3 of the most common.

Misunderstanding Statistical Significance
We often come across clients who either ask for more data points, so its "statistically significant" or believe they have a meaningful result because it is "statistically significant". While research methods like surveys demand a certain amount of rigor with regards to statistical significance, when it comes to usability testing or observation of specific behaviours, it is not at all a sufficient criteria (although can be related) for meaningful or "proper" research. So how many research participants are enough? Well, that depends...on the range of target user groups, scope of activities or tasks you want to observe, how the results will be used and how many rounds/iterations of research you will be conducting. If you are iteratively testing an application or web application and only have to worry about one or two user groups, several iterations with 5-6 users each time will detect most of the usability issues. However, if you are doing a benchmark study, you'll want to run a greater number of users, across all user groups, to ensure the results are comparable.

Past studies indicate (Nielsen and Dumas, 1987) that approximately 5 to 6 users are likely to detect 80% of usability problems for a specific use of a product. But keep in mind this "formula only holds for comparable users who will be using the site/product in fairly similar ways" (Nielsen 3/19/00). More recently, this 5 user assumption was tested by Laura Faulkner (2003 "Beyond the Five User Assumption: Benefits of Increased Sample Sizes in Usability Testing, Behaviour Research Methods, Instruments and Computers Volume 35(3) 379-383) who tested with 65 experienced and and inexperienced users on a web-based employee timesheet application. She then selected random groups of five participants and looked at the number of issues each group of 5 detected [compared to issues detected by all participants]. While the 80% rule identified by Nielsen held, the variability was such that some groups of 5 identified as few as 55% of the issues. However, by increasing the number of users to 10, on average 95% of issues were identified and a minimum of 80% were identified by any one group.

Not Triangulating Research
No single type of research gets you everything. Each research method has its benefits and drawbacks. In order to come to proper design decisions you need to draw from various methods and perspectives. For example, if a client comes to us and indicates (or we discover) they have workflow issues with their application, we'll often recommend that we first conduct user observations and contextual interviews to fully understand the context in which the user is working (i.e. what they do and why) and how and when they are interacting with and using the product for regular workday tasks. Later, once the research data has been synthesized and used to inform design changes in the product, we'll recommend conducting "usability walkthroughs" on the new design concepts to validate the changes, ensure the new design matches the user's workflow and expectations, and identify any usability issues that may be associated with the interaction design.

Using Data that Generates a Hypothesis to "Prove" It
If you gather enough data while not trying to test any particular hypothesis, then patterns will emerge that are often meaningless. This is a fundamental tenet of randomness that rigorous research attempts to minimize.

For instance, if you ask enough people their birth month, and then 100 other questions, you are bound to "discover" things like people born in November drink a disproportionate amount of Budweiser compared to everyone else, or that almost all the July born people like driving foreign cars over North American cars.

You haven't actually likely discovered anything; except a statistical phenomenon known as "clustering". In a large set of data that is random you will get clusters of the same type of information. Roll a dice 100,000 times and you'll get strings of snake eyes or 12's or something. It has nothing to do with some strange meaningful pattern. There's just a lot more "clustered" patterns than non clustered possibilities out there.

It's not until you run a separate independent follow up experiment that you'd actually know if there is a meaningful trend. Where we've seen this lead people to the incorrect conclusion is when they take 2 pieces of information and make assumptions based on the clusters they see. For example:

1. Doctors are very mobile in a hospital setting as they do rounds.
2. Doctors are buying iPads in disproportionate numbers because they are easy to use.

These 2 data points can lead to the wrong assumption that doctors will use iPads to assess patient x-rays if one doesn't understand the context and workflow that is involved with this process.

Contributors
Lorraine Chapman is a management and usability research professional at Macadamian. In addition to her role as Director of Research, Ms. Chapman has provided a broad range of clients with strategic direction on business, product and customer issues.

Scott Plewes is Vice President of User Experience Design at Macadamian and an expert in user experience design, user research, and incorporating the voice of the customer into product design.