The usability super star Jakob Nielsen points out that seemingly small number user group were almost as effective as a large group when it comes to identify usability problems in an interface. In my opinion there are other factors to consider when presenting usage tests that among other things requires a slightly larger group of users.
Jakob Nielsen is by many regarded as the worlds foremost usability expert, almost like a usability god father and founder of many of the practices used in todays usability. Through his site Useit.com and his Bi-weekly column the Alert box Jakob are spreading the world to an ever growing international audience.As a usability consultant i pay good attention to what goes on at the Nielsen headquarters in Fremont California. As a designer however my feelings are a bit more mixed. The actual products produced are not particularly aesthetically appealing to my Scandinavian minimalistic taste. There is no question about how important Jakob’s work is for the development of easy to use web sites on the Internet today, witch brings us right back to the article in question:
In early 2000 Jakob Nielsen together with Tom Landauer sought out to find exactly how many users a usage test needed in order to find all errors in an interface. The result was presented in an The result was presented in an article that would influence the usability community for ever. The result pointed out that a group of 15 respondents was needed to find all errors in the tested interface witch is interesting in itself. The interesting part however was that the percentage of new errors found by each user added started to decline drastically after testing the interface with approximately five users.
The main conclusion was that to get the most out of a usage test a test of five users was recommended. It turns out that with this smaller group 85% of all errors were found and for a significantly lower cost than a full test would show. Each user gives rise to a vast amount of research material that deeds to be analyzed thoroughly in order to extract the findings and it start to look like a waste of energy when the users start finding the same problems over and over.
Nielsen conclusion: Do smaller tests update the design and test again!
After creating the new design, you need to test again. Even though I said that the redesign should “fix” the problems found in the first study, the truth is that you think that the new design overcomes the problems. But since nobody can design the perfect user interface, there is no guarantee that the new design does in fact fix the problems. A second test will discover whether the fixes worked or whether they didn’t. Also, in introducing a new design, there is always the risk of introducing a new usability problem, even if the old one did get fixed.
Also, the second test with 5 users will discover most of the remaining 15% of the original usability problems that were not found in the first test. (There will still be 2% of the original problems left — they will have to wait until the third test to be identified.)
For me this article has been very important for me when it comes to setting up usage tests for my clients. In my opinion however there are more to it than just identifying the errors.
Presenting the results from a usage test is most often imply bringing bad news to the clients attention and in this context the methods used for gathering the data is always questioned. Is the problem identified not only a coincidence created by the special circumstances in the test session? In my experience there are important to be able to repeat the most critical errors a few times in order to create a persuading result for my clients. And for this purpose five users are not sufficient. My recommendation is there for to use 6 to 8 users to boot the liability of the test as a whole.
Further more, when conducting usage tests with an eye tracking device five users are not enough to create nice eye movement visualizations like a heat map.
For me is makes perfect sense to publish the video material from each session and give access to the client. This has proved to be very appreciated by the clients that then can watch the whole material in order to draw there own conclusions.
Usage testing is in fact a brilliant design tool. but it is also the most effective way of communicating the need of improvements in an interface. No wone can know if a design works as expected if it is not tested on real users and testing should there for be a standard procedure when designing an Interface.
I conduct all kinds of usage tests on all kinds of interactive solutions. I even do competitors analysis with this technique. So even if you don’t have an interface yet usage testing on the existing competitors are often a good idea!
/ Mårten