Last week, UPA-DC hosted a talk by Dr. Patricia Chalmers, a usability researcher for the US Air Force. The subject, ostensibly, was usability metrics — the kinds of data collected during testing. Dr. Chalmers provided an overview of testing and discussed at a high level different kinds of metrics to collect. For the second hour, she opened the floor to discussion.

The usual themes emerged: usability professionals are unappreciated, developers are nothing but obstacles, and there’s never enough money to go around. For me, what emerged was a picture of “old school” usability — the notion that usability professionals conduct experiments within the confines of a lab and are judged based on the quality of their statistical analysis.

One attendee, for example, reminded us to distinguish between formative and summative testing. These are big words to describe the purpose of a test — as a tool during the design process, or as a means for predicting the product’s success. I asked the group whether this distinction still holds meaning within the context of the web. The assembled crowd wanted to believe that there was still a place for summative testing, that clients would pay for a big test at the end of the process to confirm the feasibilty of the design, but they didn’t convince me, much less themselves.

Though confined to the largest of the commercial sites these days, there is technology that allows us to monitor what works and what doesn’t. Amazon gradually introduces new screen designs to a small percentage of users and tracks their performance. The scale may scare you away from such an approach, but the cost of performing these “real-time studies” will diminish quickly. The contribution of a usability professional changes when data pours in constantly. Instead of designing and running tests, perhaps organizations can squeeze more value out of usability engineers by asking them what data they should be tracking and how to interpret the results.

What surprised me most about the talk was the lack of emphasis on goals. Lisa Battle, UPA-DC’s current president, indicated that many people track the easy metrics for no reason other than they’re easy to capture. Working for a hard-core business analyst in my government days, I learned the importance of traceability, of tying metrics to goals to ensure that the data actually tells us something about our objectives.

Dr. Chalmers did mention goals, but I don’t think she gave them the attention they deserve, especially in a talk about metrics. In doing so, she mentioned Jakob’s five quality components of usability. As a reminder, they are:
Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?

Efficiency: Once users have learned the design, how quickly can they perform tasks?

Memorability: When users return to the design after a period of not using it, how easily can they reestablish proficiency?

Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?

Satisfaction: How pleasant is it to use the design?

It occurred to me that this is a useful framework for establishing goals. We can ask our clients questions in each of these areas that surface very specific objectives for the test. Here are some initial ideas:

Learnability

How long do you want users to retain information about your site?
Is your site the kind that users visit every once in a while and will need to re-learn during each visit?
Are there some tasks that are more important to learn than others?
Efficiency

How do users perceive the need for speed?
What factors contribute to the user’s perception of speed?
What will users use as a benchmark to compare speed of completion?
Memorability

What do you want users to take away from their experience with the site?
What do you expect users to retain after they’ve left the site?
What do you expect users to recognize upon returning to the site?
Errors

Is it more important to avoid errors altogether or successfully recover from errors?
How are your users likely to handle or interpret errors?
For which interactions is error-handling most important?
Satisfaction

What factors contribute to the user’s perception of satisfaction?
Do users expect more than successful completion of tasks?

I don’t think there’s anything earth-shattering here, but I do think usability tests could benefit from a good dose of critical thinking, especially about objectives. A test’s goals serves as the foundation for the entire effort, and proceeding without them — or with poorly formulated goals — leads to generating lots of meaningless data. It’s easy enough for clients to say that they want the site learnable and memorable, and that users should complete tasks efficiently and feel very satisfied at the end, but in creating tests such goals are no more useful than “to determine if the site is usable.”

As the web changes software development methodologies, usability professionals will also need to evolve (without fuss), bringing new methods to the table that provide increasingly meaningful and employable results while still meeting established goals.