My place or yours: Use and abuse of research facilities

Photo courtesy of the science Museum, London.

You fought hard for funding, the contractors have finished and the paint is almost dry. Naturally you want to make sure that your new usability lab earns it keep. But don't get too enthusiastic about having a full schedule if it means that your development process will suffer as a consequence.

Here's the thing: while audio and video recording of test sessions, focus groups and interviews in a controlled setting can be very useful, so can ethnographic research and candid feedback. Unfortunately, these last two activities (and several others) cannot be done in a usability lab. Ethnographic research in particular must be done in the field. Hauling participants into your test facility, no matter how charmingly decorated, will not produce the same results at all. We need to see how users really work, in their own environments, with their own equipment, materials and other resources. This is especially true of third-party interactions, where aside from the computer and its immediate user, there is a customer, passenger, patient or other "ultimate" user. These complex interactions simply will not be the same in an artificial environment as they will in the real world.

Furthermore, because you have much easier access to your organisation's own staff (then to, say, your customers) there is a very real concern that the web-based system you are developing will target the wrong users. Let's take an example. Suppose that you are in the travel business, developing a new customer-facing system for use by your staff. While it is very important that this system be highly usable by staff, it would be to no avail if it did not at the same time meet the needs of customers. And you really don't have much choice here. Either you go out into the field and observe real customers with real needs, or you become a playwright, drafting scripts to be enacted on this stage that is your lab environment.

Of course, the problem is not limited to three-party interactions. In ethnographic observation we are able to win the confidence of users: by granting them anonymity and by trying to see the problem through their eyes. This means spending time and being a careful listener more than an interviewer. Back in the lab, though, we have little choice but to become an interrogator. Users are not going to be free to carry out their normal routine, partly through a lack of time, but also because they are in a totally artificial environment. Also, anonymity is rarely available - certainly not if sessions are being video recorded. In fact, there is probably no better way to get recitals of official policy than to invite your staff to sit in front of a video camera and tell you how they do their job. But there are much quicker and less expensive ways of getting that information.

What about usability testing itself? Surely you can do as much of that as you'd like in your new lab? Well, not necessarily. Most usability testing is based on prearranged tasks. If these tasks are not realistic, assessing users' performance of them is an exercise in self-deception. We need to have done enough fieldwork ahead of time to know what the real tasks are and how important each is in the scheme of things. What this comes down to is a usability testing “sandwich”:

The early fieldwork is to find out who the users are, their goals, tasks, working methods, resources, artifacts and other contexts of use. Usability testing is done in the lab with paper prototypes , working prototypes and eventually early releases of the system being designed. Then, to be sure that we have done all of the earlier work correctly, we check that the system really does meet users' needs by observing it in practical use.

If this approach leaves embarrassing holes in your lab schedules, there are two things you can do. You can either live contentedly with the knowledge that you will be able to handle a larger number of projects than originally anticipated or, if you would like to keep your facilities as fully booked as possible, sell your design teams on the idea of paper prototyping (see Carolyn Snyder's book, Paper Prototyping.) In my view this is always an excellent investment of effort and a good use of a usability lab into the bargain.

The Author

William Hudson is principal consultant for Syntagm Ltd, based near Oxford in the UK. His experience ranges from firmware to desktop applications, but he started by writing interactive software in the early 1970's. For the past ten years his focus has been user interface design, object-oriented design and HCI.

Other free articles on user-centred design: www.syntagm.co.uk/design/articles.htm

© 2001-2005 ACM. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in interactions, {Volume 11, Issue 3, May + June 2004} http://doi.acm.org/10.1145/986253.986270