Usability testing: Common mistakes UX research teams make

Usability tests are an incredibly powerful tool to help you evaluate any design at any fidelity, from a napkin sketch to a fully functional piece of software. 

Usability testing can help you identify issues before they are live, save your design and development teams time when deciding direction, and find opportunities to deliver just the right experience to your users.

But you only get these benefits from a well-run usability study. There are many ways a usability test can go sideways and yield skewed results. 

This article will review a handful of common mistakes many UX researchers, product, and design teams make around planning, recruiting and compensating the participants, and effectively moderating the sessions when they’re first starting out. 

Using usability tests for the wrong research question

Usability testing has become so well-known that it sometimes gets used interchangeably with “ux research,” but usability tests are just one of dozens of methods. They’re great tools for assessing the ease of use of a design, but insufficient for learning much else. 

For example, they’re not suitable vehicles for understanding more about a user’s motivations or patterns of use. 

To make sure you’ve picked the right kind of method to use, start by clearly defining your research goal. Think about the questions you want answered, and what you want to do with that information. 

If your core goal is to assess the ease of use of something, your questions may include: 

• “Can our users successfully buy tickets for the correct train using this new interaction?”

• “Where are users struggling with the current sign up process?” 

…and other similar queries. But if your questions are well outside the scope of whether users can easily interact with your product, you may need to use a different testing method. 

Recruiting inappropriate participants 

The benefits of usability testing are ultimately centered around truly understanding how well a design works for a specific user type. 

Theoretically your design should be easy enough to use by just about anyone, but you could miss important context if you don’t recruit real or representative users of your product.

Let’s say you’re working on a tool for medical providers and patients to communicate about ongoing treatment plans for specific diagnoses. 

Medical professionals are going to have specific knowledge and context, may use more precise clinical language, and might only be checking in once in a while. 

Patients, on the other hand, may be scared or stressed, physically uncomfortable, and interacting with your product every time they experience a side effect or symptom. These are vastly different experiences with the same tool. You need to make sure it’s easy for both cohorts to interact with your product. 

A random person could give you feedback on the interactions and you’d likely identify some core issues. But you may also miss nuances in feedback that both the medical professionals and the patients would be able to provide. 

The closer your usability test participants are to the real users you’ll have, the more authentic and valuable their feedback. If you have multiple different sets of user types (like the patients and providers here), make sure that you recruit participants from each group to participate. 

For tips on finding the right participants, check out our recruiting guide. 

And make sure that as a part of your recruiting process, you think through a proper incentive strategy that will attract your target audience.

Leading the participant 

Facilitating usability test sessions is one of the toughest things for new researchers to learn. There is a fine balance between leading the conversation and avoiding leading the participants to answers.

As a facilitator, it’s your job to elicit feedback on the core target areas, keep participants focused on their tasks, and probe for further understanding without introducing bias. 

While I absolutely recommend recording yourself while you facilitate and asking for others to provide feedback to improve, here are a few tips to moderate effectively:

Intervene appropriately

It can be so tempting to jump in while a participant is quiet, either to ask for more feedback or to offer help, but it’s usually best to err on the quiet side and let the participant speak up. Sometimes they’re reading, processing, or thinking through their response. Your interruption may throw them off. On the flip side, if someone is truly struggling and it’s obvious that they won’t be able to complete a task, you’ll want to acknowledge their struggle but move on to make the most of the time. 

Ask open-ended probes

It’s important to ask follow up questions to better understand the context behind an action. At the same time, it’s also an easy to accidentally introduce bias. Let’s say you noticed a hesitation before completing a task.  If you have a hypothesis that the button text is unclear, it might be tempting to ask about that in the follow up. But maybe the user was just re-reading the form before they submitted or got distracted by a message. If you bring up the text clarity, they may unconsciously anchor themselves to that idea and agree, giving you a false confirmation. Better to leave follow-up questions short and open, like “It seemed like you hesitated, can you tell me what your experience was like? 

Keep body language & questions

While you want to make participants comfortable and engaged, being too friendly or encouraging may lead their behavior. For instance, if they are on a specific track and you smile and nod, even if they’re going to the wrong location, they may take it as a cue to keep going. While we should be reminding participants  that we want their most authentic effort and to act as though we aren’t there, participants can’t help but respond to the fact that they are sitting near a human and have been asked to provide feedback.

Conclusion

Usability testing is one of the most frequently used and powerful UX research tools. It’s fairly easy to get up and running. But it’s also important to be aware of common challenges to get the most out of your sessions. 

If you’re ready to get started running usability tests and want to explore sending incentives with Tremendous, Sign up now, or chat with our sales team.