As more and more organizations understand the importance of user studies while building their products, in-house usability sessions are becoming increasingly common. Although conducting a usability session is not a very demanding task, if not done right, it may not help you get the right kind of insights for the product that you are building.


The Problem

Many beginners make the mistake of treating a user session as a way of getting feedback about a product. This works great if you are in the final stages of the product and are mainly looking for ways to make minor improvements but it falls flat if your user sessions are meant to provide you with meaningful insights that need to translate into product and design requirements. The problem is that ‘feedback’ approach of conducting a usability session is that you end up with a bunch of opinions. Its a well known saying - ‘Opinions are like assholes, everyone has one’. This causes opinions to be biased and personal and is more than likely to differ from person to person.


Preparing for the study



Setting the goals for your study

When you have a clear vision of what you want to achieve, you are more than likely to achieve it. This holds true not just for life goals but also when you need to plan a user study.

Write down the goals on a blank sheet of paper. Go through them and make sure that the language is plain and simple enough that anyone above and below the hierarchy can clearly interpret it. This is important since all stakeholders might have different expectations from the study. You need to bring them all on the same page. This clarity of direction will also pave the way for the next important step - task design.

Task Design

We already talked about how a user study is not worthwhile if you are going to merely ask about opinions. The only true way to derive value from a study is to define very specific tasks that you would like to see the user perform on your product.

The reason this works is because even if people can have different opinions, most people can’t deny their instincts. And there is a very close relationship between intuitiveness and instincts. The more your interface maps with a user’s instinctive behaviour, the more usable your product will be.

A good user task would excel at the following

  • Create a situation in mind of the user: For this to happen, the task designer must be an excellent story teller. You need to make the user believe that he/she is the person performing the task.
  • Be realistic: There is no point of indulging in your own imagination if your tasks don’t match a real world use case. You’re just fooling yourself because your end users will never use the feature you are so aggressively testing.

Prototype Design

Prototyping is a very complex activity because you need to find the right balance between a number of things
- What your users need - What the management wants you to build if you are working on the redesign of an existing product - what the previous product already did.
- ‘Wishlist’ interactions that you observed in other products that you and stakeholders feel would benefit the product being designed.

I won’t go into the details of how to do prototyping in this post, but before you bring in users to test your prototype, you need to ensure a few things.

  • Completeness of the prototype: What this means is that you need to finish the prototype to the point where it is possible to perform each task in the prototype, as determined in the previous step, from start to finish without any hiccups. This is important because if you need to assist your users to complete a task because the prototype is not up to the mark, you would be artificially increasing the difficulty of the prototype. This will skew the results of your study if you measure stuff like time-on-task during your study.

  • Multiple versions of the prototype: This is helpful if you are debating the effectiveness of one interaction over another. In such a case, you probably want to prototype multiple versions of your product and test it out preferably with different users if possible. During such tests, any metrics that you collect will help you settle on a conclusion that should end the debate.


During the study



Breaking the ice

No matter what you told your users in the study invitation email/call, its safe to assume that the best way to start the conversation is by reiterating the purpose of the study. One key point to remember when breaking the ice is that you never want to use the term ‘user-testing’. The term a misnomer because you are not actually testing the user, instead you are evaluating your product.
You can always phrase the purpose of your study by saying that you are ‘evaluating the interface of something that we are still building and is a big work in progress’ and then emphasize on the fact that the reason for the study is to understand how you can build it better and make progress in the right direction.
This is critical because it allows the user to know that (s)he can express their thoughts freely since the product is not yet built. Moreover, it builds camaraderie and an openness during the session since at no point should the user feel that his/her skills at using the product are being tested.

Pre-Task Warm Up

When you are testing out an interactive prototype, you probably haven’t had the time to handle all the edge cases. Or maybe you haven’t gotten that far into the design yet. For example, take the case of an application that presents information differently in the signed-in and the signed-out states and the first page that you will show the user is that of the signed out state. In such a situation it is important to inform the user that the interface that (s)he will see is a signed out interface.

Assigning Tasks

This should be simple enough given that you have the objectives of each task listed down on paper - one objective per sheet. You probably don’t want to present the user with all the tasks at one time and overwhelm them.
Let the user read the task and then allow them to ask questions if they are not clear on what they are expected to do.
This is also a very good opportunity to find out if the task objective is realistic so that in the hopefully rare case that it is not, you can make amends, if possible, before the next user study.

Observing

This is the meat of your user study. Its great to have an assistant that takes notes but if not, make sure that you yourself take notes in some form or the other. Although there are many things that you can observe during a user study, there are a few that are so fundamental that they should never be ignored.
- Facial expressions - The speed at which they move their mouse when performing an action. - The approximate time that it takes them to perform the next step in the sequence of actions for completing the task. - The places on the interface that the user first clicks on, even when they are incorrect. - The number of incorrect attempts it takes the user to take the next step in the sequence of actions. - How long does the user pause on each screen.

Although there are many more things that can be observed, these are just a few ‘instinctive’ characteristics that you can quite easily observe without a lot of hassle.

The reason why the ones mentioned above are important is because the outcome of a user study should always be to understand and make ‘inferences’ from user ‘behaviour’. When people interact with your product, their voluntary and involuntary actions convey a lot more meaning than the words they speak. Simple things like their facial expressions, the time they take to look at the interface, the direction in which they instinctively move their mouse cursor and even the rate at which they move their mouse cursor can tell you a lot about the perceived usability of your product.

This post is meant to outline a few important things to keep in mind when conducting a usability session if you fall into the latter category - i.e. You are in the initial - mid stages of designing your product you are looking to


Finishing the study

Now is the time to ask the user for his/her opinion. There are a couple of things that you MUST find out in this phase

  • Were any of the tasks inherently wrong. This directly maps to the question - Is the feature that you are prototype-testing appropriate.
  • Was there a mismatch between the stated task objective and the task that they actually performed on the prototype. i.e. Did they feel that the interface supported the functions and interactions that they were expecting in order to perform the task.
  • What was missing that they feel could have made things more clear. This is a good one especially for tasks that the participant had a difficulty performing. Ask the user to rate the interface/product using one of the popular System Usability Scale if applicable to your product. If not, its always good to have some form of rating system that you can yourself construct to help the user rate your product. Just make sure that the you use an odd scale i.e. 1-5, 1-7, 1-9. This is done so that you always have a proper ‘middle’ point in your scale.

Not all user studies are created equal but the items listed above are pretty much all the basic ingredients you need for conducting a good user study. If you are'nt doing these already, try making these changes and watch how your user studies will yield a lot more information than they currently do.