What is Usability Testing?
You’ll hear the phrase usability testing (also referred to as user testing) a lot these days. Essentially, usability testing is a technique to evaluate a product by testing it on users and is an essential part in producing quality user experiences.

Here at e3, lab-based user testing is just one of the methods we use to review the user experience of our projects, involving a facilitator and observer sitting with the user, using the Think Aloud protocol, whilst completing the required tasks/scenarios.

Working in a commercial environment where time and budget are limited, we are often under pressure to make best use of the available resources. There has been much debate about the ideal number of Users to test to catch the majority of usability issues. What research informs our decision? Often it goes back to research performed by Jakob Nielsen in the early 90s, but both the web and users’ behaviour have changed substantially since the early 90s.

In this post, I wanted to take a look at what the research says about the numbers of users required to identify the majority of usability issues and discuss some of the changes that have affected the web and its users; finally thinking about ways in which we can optimise our usability testing methods.

Current Research
In the 90s, Nielsen and Molich reported that 5 users would find 80% of usability issues, and testing with more users gives diminishing returns as more and more duplicate issues are reported. Jump forward to 2010, where Hwang/Salvendy report that 10±2 was the number of users required to identify 80% of usability issues. They went on to recommend that if sample size needed to be reduced we should investigate changing other conditions such as: evaluator expertise, duration, task type and report type.

In 2012, Schmettow added to the debate, stating that the 10±2 rule was still underestimating the number of users required to identify 80% of usability issues. He went on to say that “Magic Numbers are strictly hocus-pocus, so usability studies must test many more subjects than is usually assumed.” 

But today we're still quoting this research from over 20 years ago, even though other research is at odds with it. Is it still valid?Well, the web and behaviour of users has changed a huge amount in that time. Some elements that could have an effect on the number of users required to identify the majority of usability issues include:

The complexity of the tasks that we perform online
The web has become much more interactive. With advances in design and technology we now perform tasks online that we never did in the past.

The types of interactions available
Think of the new opportunities and issues touch devices bring with them.

The environments and devices the web is accessed from
The web is now available on tablets, phones, TV’s, games consoles, even cars. These different scenarios provide various challenges such as limited screen size, environmental distractions etc.

The experience of users
Users are familiar with a vast array of interactions, new UI standards and patterns are constantly emerging to take advantage of new tasks and technologies.

Conclusions and recommendations
So what does all this mean? How do we make best use of the time and budget available? Some of the elements that can impact on usability testing include:

• How complex is the task? Is there a heavy load on cognition?
• How novel is the task? Is it familiar or are you introducing new concepts?
• How important is the task (is completion key to the service? E.g. the checkout process on an ecommerce site)?

Being smarter about how you test will help save time, along with providing better return on your testing. Another important concept to consider is testing throughout the project life-cycle, with different fidelities, identifying different issues. For example you could test:

• Sketches
• Clickable wireframes
• Limited functionality prototype
• Functioning prototype

But whichever view we take, there are ways we can improve the effectiveness of our testing. These include:

• Testing different fidelity deliverables throughout the project life-cylce
• Focusing our efforts on complex tasks
• Key tasks
• Novel interactions

And remember people, any form of usability testing is better than none!


If you'd like to chat to our UX team here at e3, drop us a line at info@e3.co.uk or call us on 0117 9021333

(1) M.Schmettow - Sample Size in Usability Studies - Communications of the ACM, Vol 55 No 4, Pages 64-70
(2) Wonil Hwang , Gavriel Salvendy - Number of people required for usability evaluation: the 10±2 rule - Communications of the ACM, v.53 n.5, May 2010
(3) Nielsen - J. Why You Only Need to Test with 5 Users - Jakob Nielsen's Alertbox (Mar. 19, 2000)