Engaging usability testing

“Imagine that you want to…” – these are the words we often use to start a task in usability testing. It is essential to properly introduce the participant to the context so that they feel that they actually want to use the tested application for a specific purpose. However, how much can we rely on their imagination if the purpose or method of completing the task differs from their actual needs and experiences?

These thoughts were prompted by the study of a business registration form I recently conducted at Łukasiewicz – PIT. Participants were given a list of test data to enter so that they did not have to provide their own personal details. We also wanted everyone to register the same business. This allowed us to control the course of the study and then compare its results. Despite this, at some point, some participants entered their own data or data related to the business they planned to set up, and using the application began to come naturally to them.

Only after being asked to stick to the test data did they realize that they had filled out the form with real data. The visible engagement of the participants, who used the application as if they were fulfilling their actual needs, caught my attention. I began to wonder whether, in such cases, it would be worth changing the approach to the research method. I had already read about a type of research scenario that uses interview-based tasks in a book by Iga Mościchowska and Barbara Rogoś-Turek. I decided to test this method in my next study of a finished product.

The opportunity arose when I was asked to test the production version of a company search engine. The application was already connected to a real database, so it could be used without restrictions. I think everyone has needed to find data on a specific company at some point. However, what data is involved and what it is needed for is individual, depending on the user’s needs. So I decided not to impose the same task and context on all testers. After all, what kind of engagement can you expect when you ask someone to find information that they will never need?

I prepared a list of questions for the interview, which I wanted to use to start the study. Their main purpose was to get to know the study participant as well as possible, their experiences with companies, and their habits in using search engines. Thanks to this, most participants practically came up with tasks for themselves. In the next part of the study, I only referred to our conversation and asked them to search for the information we had discussed. After conducting this research, I gathered a few observations that I would like to share with you.

What should you keep in mind when preparing such a study?

1. This method works best when testing a finished application containing real information and data. You will decide on the content of the task during the study. You can predict some of them, but some of the participants’ needs may surprise you. Therefore, you cannot prepare data only for a specific scenario. If you want to replicate the real experience of using the application, the user should be able to use it without restrictions.
2. At the beginning, explain that the study will start with a conversation aimed at getting to know the participant better and understanding in what situations they might use the application being tested. In order to get to know the interviewee well, you will probably also ask more personal questions, which may surprise a person prepared to test the application. Therefore, it is important to explain why such questions are asked.
3. Be sure to inform the participant that they have the right to refuse to answer. This is particularly important in this type of research due to the possible personal questions I mentioned above. Participants may not feel comfortable answering such questions to a stranger.
4. During the interview, try to avoid asking questions and reading questions from a list. Remember the key questions you prepared in advance, but focus on a relaxed conversation. If the participant feels comfortable talking to you, you are more likely to get them to open up and get to know them better. This, in turn, will allow you to tailor the task to interest them.
5. Ask follow-up questions if the participant says something that catches your attention and could be an idea for a task. If the answer is not very detailed, it is worth exploring the topic further, paraphrasing the question, and asking it again. People often do not remember all their experiences. It may happen that they will not give you a comprehensive answer to the first question, but with the next one, they will remember something that can be used.
6. Prepare some sample emergency tasks. You may encounter someone who does not want to open up to you. In that case, you will not be able to find out enough about them to find inspiration for a task. In such a situation, it is worth having tasks prepared that are as universal as possible and whose context can apply to most people. During the interview, you can ask questions related to your backup tasks (“Have you ever done X?”, “Have you ever needed X?”). This may help you select the backup task that best suits the person.

Examples

To better illustrate how such a study might proceed and how to apply the above tips, I will describe two examples from a study of a company search engine.

Example 1

When asked about his job, the participant talks about his profession and full-time employment. When asked if he has ever searched for information about companies, he says that he works with other companies and sometimes looks for their contact details.

In the second part of the study, the moderator asks the participant if they can remember the name of one of the companies they work with. After receiving an affirmative answer, the moderator refers back to the conversation and asks the participant to imagine that they are at work and need to find the email address for that company.

Example 2

The participant answers the questions briefly. After asking all the questions on the list, the moderator still has no idea what to do. As a backup, a task has been prepared which involves verifying a company that provides home repair services. The moderator asks the participant if he ever calls a professional to repair a fault. The participant answers in the affirmative.

In the second part of the study, the moderator gives the participant the name of a sample plumbing company. He asks him to imagine that he has a problem in his bathroom and has found a professional he would like to hire. However, he first wants to verify that the company is registered.

 

Pros

1. This method allows you to test various scenarios similar to real-life use, as you draw inspiration from the users themselves. This gives you a broader picture of how the application can be used, as well as its strengths and weaknesses.
2. Such tasks engage users more. You can therefore observe them in a more natural situation in which they would use the application themselves. If you manage to tailor the task so that the participant feels like they are doing their own thing, their engagement may cause them to go beyond the scope of the specific task and explore different options out of curiosity. In such a situation, the study may even reveal bugs in the application that would not have been found if all participants had been given the same task.
3. Participants see the value of the application and it arouses their greater interest. This gives them the feeling that they have spent their time on the study in a meaningful way and learned something interesting.
4. You also have fun! There is no routine of conducting several identical studies in a row. Each one is completely different and allows you to look at the application from a new perspective. Each session is also a challenge to tailor the task to the interviewee.

 

Cons

1. Each participant performs a different task, so the results obtained are not repeatable and can only be analyzed qualitatively. Due to the lack of repeatability, such analysis is extremely demanding.
2. If you are interested in examining specific elements of the application, you must ensure that the participant uses them. The method gives the participant a lot of freedom in performing the task, so it may happen that they complete it without paying any attention to the element you are interested in. In such a situation, after completing the task, it is worth asking the participant to perform an additional, short task requiring the use of this element or asking whether they noticed it and why they did not use it.
3. Conducting such a study is more demanding for the moderator due to the inability to prepare for a specific scenario and the need to improvise during the study. The moderator must be very well prepared for the interview and know in which direction to lead the conversation in order to come up with an idea for a task.

 

Bibliography

• Research as the basis for User Experience design, I. Mościchowska, B Rogoś-Turek,  Wydawnictwo Naukowe PWN, Warsaw, 2015
• Center Centre – UIE,  https://articles.uie.com/interview_based_tasks/
• Perfetti Media, https://perfettimedia.com/articles/usability-tests-in-a-nutshell-part-3-creating-tasks/