The Ultimate Guide to Moderated User Research

As an optimizer, you know that gathering qualitative data is an essential part of shaping your conversion strategy. User research is a broad term often conflated with usability testing (or ‘user testing’) or market research style user interviews. 

However, this post is talking about a hybrid of the two which we refer to as user research, in particular, moderated user research– where a person (known as a moderator) is present during the research session to talk, prompt, and ask questions as the session progresses. Moderated user research can either be done in person, or moderated remotely. But this shouldn’t be confused with “unmoderated remote user research” where questions/tasks are set and the user completes them by themselves while recording the session, to be reviewed later by the researcher. 

Here's an overview of what we'll cover in this post:

Why use moderated user research?

For customer experience optimization work, one-on-one sessions with participants are a great source of hypotheses for experimentation. The sessions tend to include some questions about the user's thoughts, feelings, desires, as well as performing tasks within the experience itself while verbalizing what they feel and any issues they are having. It’s essentially a combination of a survey and more traditional task based user testing. 

This type of research is one of the most powerful methods allowing you to dive into your audience's perspective, understand their motivations and apprehensions. It’s a necessary exercise because when you've spent endless hours creating a website or crafting copy, you’re too close to the project. You’ll likely miss or ignore important details, as our client Sky Boulton, Owner & Principal of BlastZone explains:

“Having spent the last decade developing a brand and product line, we found we were unable to disengage emotionally and to truly understand our market and our customer’s mindset.

We needed to remove the emotional and sophistication bias we have with our market, and allow the data and testing to paint the real picture, presenting our audience with what THEY need, rather than what WE need, to ultimately accomplish our goals.”

But in order to uncover insights that are valuable and actionable, you'll need to perfect the art of moderating. It goes beyond simply chatting and hoping for the best. You'll need to know exactly what to say, how to say it, and when to say nothing.

The first step in the process is to determine what the purpose is of your research:

1. Generative: you're searching for new ideas.
2. Evaluative: you're trying to validate existing solutions or ideas. 
   

Once you've figured out the intent behind your user research, you'll need to hone the process. Many people follow a process (or lack thereof) similar to this:

But to carry out meaningful sessions, you'll need to follow a process that looks more like this:

Done properly, user research usually takes a few weeks of meticulous planning and preparation. Here’s my process for conducting moderated user research. 

Your research plan should include:

• Goals: what you want to discover by conducting the research sessions.
• Background information: any important discoveries, trends, previous research, or statistics that have led you to carry out user research.
• Schedule: map out when you intend to carry out the research, recruitment, and analyzing the results.
• Methodology: who and how you intend to carry out the research. E.g. how many participants will you speak to, what tasks are involved, what devices or material do you need to conduct the research? Think about any consent forms you might need participants to complete for the session recordings too. 
• Questions: what you plan to ask and any discussion topics will you cover.
• Participants: the defining characteristics of the users you want to research. E.g. age, where they sit in your sales funnel, how they typically browse online.
• Budget: factor in the costs for participant recruitment, incentives, tools to conduct the research or research facilities you need.
  

Your research won’t be useful if you don’t have the right participants. For example, getting your grandparents to review an 18-30’s holiday website won’t give you insights into the people who’ll really be using the website.

To create participant criteria, try to recruit similar profiles to those who will go on to use the website/product or feature. 

• Do you want to speak with existing customers or non-customers? E.g think about frequent purchasers, biggest spenders, individuals who help the main decision marker, internal business users, lapsed customers, failed to purchase users? 

• Is there any demographic criteria? E.g. location, gender, age, job role, education,  income level.
• What are the main motivational characteristics you’re looking for? E.g. they’re looking to buy a new car or they use the internet to shop online at least once a month, etc. 

• Is there anyone you don’t want to include in your user research? E.g. must not be a UX designer, must not be a brand/product/service rejector, etc.
  

While selecting the right participants is important, being too specific with your criteria can prevent you from attracting enough participants. You might need to whittle down the criteria to must-haves or broader criteria. Most recruitment services charge more, the more criteria you set for finding participants. So it’s better to not select criteria just for the sake of it, and instead, consider which elements are important for the research at hand. 

You can also use screening questions to help select / eliminate certain participants. 

• Think of your screener as a funnel by filtering out unsuitable users quickly. You can do this by using a staggered approach that starts by asking broad questions and gets more specific, e.g. ‘which of the following best describes you’ followed by ‘how often do you do x or y’? 

• Consider both inclusion and exclusion questions. E.g. If you were recruiting for a gun seller, you wouldn’t want to recruit someone who was against gun ownership. It may also be the case that someone meets all of the screener requirements but wouldn’t be the key decision-maker - so try to ascertain this too if it’s important to the research goals. 

• Ask open questions so users don’t know what the ‘right’ answer is, e.g. ‘What household furniture items are you planning to purchase in the next 3 months.” is better than asking “Are you planning to purchase a new sofa within the next 3 months?’

• Try to limit screener questions to 2-3.

Depending on the type of participants you need, (e.g non-customers or users, people who aren’t aware of your business, etc.) you might need to use a recruitment agency to help you.

But vet the participants any recruitment agency sends you carefully - there are “professional user research participants” who continually put themselves forward for research studies. This can introduce biases into your data as these people might not be motivated to answer your questions truthfully. They are also expert interviewees meaning their answers might not be what you’d get from ‘normal’ users who don’t regularly participate in research. 

Other methods of recruiting participants include

• Online user recruitment companies such as Ethnio 
• Remote user research tools such as Userzoom, Sharewell or User Testing which tend to have their own participant panels built into the platform. 
• Live recruiting using an intercept pop-up tool such as Hotjar to recruit users as they are performing a certain action. Although there are pros and cons to all recruitment methods, this method ensures the participants have a genuine need / interest in your product / service as they accessed the website naturally and without prompt. 
• Emails to your existing user base with screener questions. 
  

When it comes to how many participants there’s no hard and fast rule. I generally aim for 5-6 sessions per audience profile (ensure you consider devices here too, so ideally 5-7 sessions per profile per device). The rationale for this is that I generally find you get diminishing returns on insights over this number.

As Nielsen points out, it would be better to run multiple smaller studies than one huge study per year. Note: running 7 sessions per profile / device accounts for no shows or sessions with limited insights (more on that later). 

Regarding sample size, it’s important to remember why you’re running user research in the first place. In website optimization, user research is one (important) source of data and insight but it is not considered the single source of truth. It helps us uncover potential issues and areas of opportunity that can be validated (or not) by other sources (analytics data, large-scale surveys, heatmaps, session recordings).

Keep this in mind if anyone questions your qualitative research sample size, it’s only an issue if you try to validate a huge change based on only three participants. 

Where you choose to conduct your research can impact what questions you ask, or tasks you set. There are pros and cons of each location. Here’s what you need to consider for each before deciding. 

Research studios or labs

Pros 

• You can easily see participants’ physical behavior.
• You can set up and use additional technology to track biometric data such as Galvanic Skin Response (GSR) if needed.
• If you are using a research lab with a viewing room you can invite stakeholders from across your business to come and watch the research live (a great way to get buy-in for research!)
• A controlled (albeit artificial) environment where you can easily record and run the sessions. 
• Lab-based setups really come into their own if protecting intellectual property, or sensitive information is a concern. If you are a bank or hospital, for example, lab-based research means you can control who can see what, and prevent anyone from saving or screen-capturing things you are showing them. You might also encounter users who have concerns about sharing what’s on their screens such as government, or health workers–so a lab with independent equipment is favored. 
  

Cons

• Participants need to take time out of their day to travel to a new location and some might get lost (this happens surprisingly often). And if participants are no show’s it’s hard to replace them at short notice during the time you’ve hired the lab for.
• There are higher costs involved in hiring research lab space vs. other options. 
• It’s an artificial environment that can make the participant feel uncomfortable or anxious. I’m not a fan of one-way mirrors as it creates a bit of a 'goldfish bowl' effect where users are more aware they are being observed. A better option would be to stream the research live into another room, ideally not next door so there's no chance of users hearing your own team reacting to their feedback. 
• If the user journey/product/website you’re researching requires the participant to have their own “technological ecosystem” it can be a hassle for them to bring in their own equipment. E.g. if you are researching a photo printing service you’ll likely want the participant to be using their own laptop to see how they go about uploading their own photos. 
• Your participants might not be familiar with the devices you have for the research (think Mac vs. Windows), adding usability issues they don’t usually encounter on their own equipment. 

At a participant’s home, workplace, or public space

Pros

• Real-life everyday settings help participants be primed in a similar frame of mind as they would usually be when using your product or service. 
• For home locations you can test out post-purchase parts of the journey e.g have the purchase fulfilled and sent to the participant allowing your research to include the delivery/packaging part of the journey. 

Cons

• Can be hard to manage the setting and record the session. 
• More logistics planning needed along with travel time and costs for the researcher. 

Remote using a laptop or phone

Pros

• The last year has basically been one long video conferencing lesson. We can utilize screen-sharing functionality to set tasks and observe behavior. 
• Cost-effective.
• Easier to recruit participants from remote or non-central locations.
• Allows you to test experiences in a real user environment with their own technology. 
  

Cons 

• Remote webcam-based research makes it hard to observe body language cues.
• Poor internet connections or users who struggle with research software can hamper the session. To avoid this, make sure your participants have access, software guides, and you’ve asked them to test their internet speed ahead of the session. 

At your retail site

Pros

• Great location if your research objective relates to the physical environment. Being at the retail site engages all of their senses and leads to more insightful feedback. 

Cons 

• Can be hard to manage the environment e.g too loud to talk/hear your participant.
• It can be hard to record and observe participant’s responses.
• Both researcher and participant need to travel to a location. 

Which location you choose will depend on your research aims, budget, and participant criteria. 

However, if you are conducting research at a physical location other than the participants’ own home/work, make sure you send detailed information on how to get there. I usually provide;

• Photos of landmarks participants will pass when arriving from public transport
• The addresses of local car parks and their costs
• Information for accessible entrances and facilities. 
• My mobile number so that participants can call if they need help with directions. 

To incentives or not? 

It’s commonplace to offer incentives to participants for taking part in research but this can introduce biases such as;

• People trying to game the selection process by giving false information about themselves in order to be selected.
• Self-selection bias–attracting people who are more motivated by the incentive vs. those who aren’t, which means you could only attract participants who are price-sensitive as one example. 
• Reciprocal bias - because you are giving participants something for taking part they may feel like they want to return the favor by answering questions in a way they think the researcher wants them to respond, rather than their true feelings. 
  

Research has shown that the monetary value of incentives doesn’t impact the level of deception (or bias), but rather the fact that there is an incentive at all. So it’s best to consider if you could attract participants without incentives or think about other types of incentives e.g only reimbursing out-of-pocket costs, gift vouchers rather than cash, or something without monetary value e.g reasons why your research is important and the impact it will have on others.   

If you do want/need to offer a monetary incentive Ethnio has a handy research incentive calculator based on;

• Your research type (moderated/unmoderated, remote, diary study, survey, etc.) 
• The country of your business and the participant. They calculate the cost of living from the World Population Review. This is useful if you need to recruit participants from around the world. 
• The amount of time the research session will take.
• The availability of the types of research participants you need.
• Your level of urgency to find participants.
  

When it comes to conducting the session itself you’ll ideally want two people; one person to conduct the interview itself, called a ‘moderator’ and another as the “observer” who makes notes on any behavior the participant is displaying, key insights, questions you might want to add or change, etc. It also means that if you are running many interviews over a few days you can swap roles to avoid burning out during the process. 

Instead of going off-piste or winging it, it’s best to have a guide on how you want your interviews to go. E.g.

1. Introduction: welcoming the participant, re-confirming consent to record the session, putting them at ease with small talk, and explaining what’s going to happen. (5 min)
2. Warm-up questions: a few simple questions to get the participant talking and to build rapport. E.g ‘Tell me about your job?’ These questions can also be relevant to the topic at hand, if you’re doing research on a travel site, ask them about their holiday plans / travel habits (5 mins)
3. Main questions or discussion topics/tasks: Asking the questions you have, to uncover insights (30 min)
4. Conclusion: Time to cover anything you missed, and wrap up the interview. (15 min)
   

Following a plan doesn't mean you can't improvise or amend your technique during the research itself. In fact, you should deliberately prepare yourself for improvisation, as you get to know your participants and encounter new information, you might need to adapt your questions or approach. 

Each part of your outline should have a rough estimate of how long you’ll spend. The conclusion shouldn't be rushed, as interviewees may be loosening up at this stage or they might have remembered something they wanted to add. Having a discussion guide will help you stay on track as well as ensuring you cover the same topics across all sessions so you have comparable information. 

It’s advisable to leave a 15-20 minute gap between your interview sessions to allow for any overrun, bathroom breaks, or late arrivals. 

If you can include longer gaps between sessions you can use the time to brainstorm the key points from the last session with the other researchers. This can be done with post-it notes on the wall of an observation room and can help you summarize things as you go. It also helps identify any areas you might want to delve into further during the next interview or drop certain questions if you aren’t getting much insight. 

‍

A good place to start is with the ‘5 Ws and an H’ which consists of six familiar areas: who, what, when, where, why, and how. 

• Who: these questions uncover more about your users or their perceptions of who might use a product or service. 
• What: these questions examine what users are doing on your site or with your product.
• When: these questions can uncover trends, routines, or behavioral habits of your users.
• Where: these questions uncover the locations of your audience, browsers or devices they use, and where they have difficulty within your experience itself. 
• Why: these questions seek to uncover the psychological or root causes behind the actions your users take. 
  

You should find it easy to create a list of questions you'd like to ask. It's then a case of prioritizing the most important questions that need addressing. Some questions might be 'nice to know', while others could significantly alter your business strategy.

To avoid leading questions and confirmation bias it’s best to ask open-ended neutral questions.

Here are some examples of good question starters: 

• How do you …?
• What are the different ways you …?
• What do you think about …?
• Show me how you would….
• Tell me why you think….
  

When you do ask questions, pay close attention to how you are doing the asking. Body language and tone will come into play, and you'll need to avoid acting like the expert. It's essential your participants feel like they aren't a novice, as this could cause them to hold back for fear of saying the wrong thing. Instead, make it clear that you're the novice in the situation and you respect their opinion.

You don't need to follow a standard question-and-answer format. Beyond answering simple questions, there are plenty of ways your participants could share valuable insights. For example, you could:

• Introduce a roleplay exercise, e.g. "if you're the customer support, tell me how..."
• Set tasks, e.g "Find a pair of shoes you’d like to buy."
  

By setting tasks, you can uncover issues you might not have thought to address. While (unintentionally) leading questions can sometimes encourage a participant to think or answer in a certain way, a manual task or roleplay could shine a light on more naturally-occurring problems. 

Once you do have your questions and tasks, it’s good practice to test them out by conducting a practice session (or pilot) with a colleague to see if they understand what you are asking of them. 

If your research objective is to look at a website experience you should begin with an exploratory / research-based task. Most people don’t just land directly on a website so I try to emulate a more realistic user experience by asking users to start a session by finding a website to do/buy X (X being your product/service category.)This has multiple benefits including; 

• Gathering insight that can be used to branch into discussions around acquisition strategy.
• Helps us understand users' frame of mind / the wider user journey.
• Eases users into the research session before they arrive on the desired website. 
  
  

While users conduct tasks, I ask them to “talk aloud.” That is, to tell me what they are doing, seeing, and thinking as they complete a task. This is a great way to get the most out of research sessions by encouraging the verbalization of thoughts, alongside being able to note down any physical expressions or reactions users make. 

Getting comfortable with silence

As a moderator, you need to get comfortable with silences (to an extent) as this provides participants with important thinking time and can indicate when a user is struggling. If a user really doesn’t know what to do or where to go next to complete a task, rather than giving them the answer, ask them what they’d do, or where they’d go to figure it out. 

The less you talk as a moderator the better. Your questions, tasks, and prompts help guide the session and ensure you meet your research objectives but you don't need to ask them much at all if they are naturally giving you feedback.

What to do when participants are holding back

Participants can sometimes hold back from providing the answers they really want to share. This could be due to a poor rapport between the moderator and participant or a participant who is naturally shy or apprehensive. You'll need to be patient and allow the participant to get more comfortable over the session. 

Keep an eye out for any visual signs of discomfort, like avoiding eye contact or shuffling. You can give positive reinforcement or try different question styles to see if they open up. This is also a reason why I plan 1-2 extra sessions than the number needed, as sometimes you will get participants who just don’t open up or provide much insight. 

What to do when participants won’t stop talking

Sometimes, the opposite can occur and your participant is too open with you. This can derail the conversation and push you away from your research objectives.

While you should take a somewhat passive role in terms of listening to your participant, you still need to be in control of how the session progresses. When this happens, you'll need to be persistent and repeat any questions/prompts you feel haven't been answered. Don't interrupt, though - allow them to finish their story first.
 

During the research sessions, you’ll want to make sure to record as much as you can by:

• Recording the conversation with your phone or microphone. It's best to capture exactly what was said, and how. Apps like Otter.ai automatically record, identify different people speaking, and transcribe audio which can help speed things up in the analysis stage. Alternatively have the ‘observer’ transcribe the sessions live. 
• Recording the screen so you can look back at what users clicked on, what pages they viewed etc. 
• Record the participant. As well as helping bring the sessions to life by showing the faces of the participants within your analysis (if they consent to this), this can also provide another source of insight via facial impressions. 
  

Tools such as Lookback.oi and Morae provide recording solutions that bring all three of these together into one recording, which can really speed up the analysis process. The output also looks more polished for use in presentations. If you’re conducting research remotely, you could simply record a Zoom meeting and ask the participant to share their screen during the task part of the session. 

If possible, start to analyze the findings during the research sessions themselves because this can improve your research by:

• Highlighting things you need to delve into deeper in your next session.
• Record a greater level of detail as it’s fresh in your mind.
• Discuss any discrepancies felt between the observer and moderator.
• Highlight areas where you have enough insight and can ease up-on in the following sessions. 
• Save time after the research sessions are finished by getting a head start on the analysis. It’s great to use the time between sessions to create a post-it note wall of the key insights/findings alongside the other researchers. 
  

‍

Write up your notes/recordings

You are going to review each recording one at a time. Using a spreadsheet, type out relevant quotes from each participant. Use quotation marks so you can distinguish them from any moderation notes you may want to add. You will need to pause and rewind recordings several times, so plan at least two times the length of the interview itself for writing up the notes. 

You are watching for any pain-points, struggles, or issues that are verbalized or expressed through behavior. You should also record any positives. Don't worry about making sense of everything initially, just work on getting the recording into a compiled written format. 

Don’t forget to pay attention to both what users say and what they do. Sometimes the unspoken behaviour is actually more insightful as it’s more likely to be natural behaviour. 

Identify themes

1. Conduct a simple grouping exercise: Use a colour coding approach to highlight; issues (red), opportunities (yellow) and positives (green). This helps focus you on the insights in order of priority (starting with issues and finishing with positives).

2. Create buckets: Go through your colour coded your transcripts and start putting the written quotes/notes into buckets based on themes. For example, someone reviewing a clothing website might have said “I really don't see why this is so expensive. Seems like I can get it elsewhere for less.” This theme could be ‘price’, ‘brand’, ‘value’, or ‘competition’ (it doesn’t really matter what you call it at this stage as long as you understand what it means and continue to use it consistently). 

As you begin to find patterns start tagging the quotes/notes with these themes. You could use comments or notes to tag insights or simply use a specific word, e.g. ‘pricing’ and add this to the end of each insight, once you come to create your report you can simply search for these ‘tags’ and find all of the relevant insights. You are looking for patterns of similarity. Remember at this point, you shouldn’t be thinking about the potential solutions.

3. Do it again. No really! Chances are your ‘issues’ category is broken down into several smaller themes, however now you need to get more specific. What are all of the different insights within pricing? There might be some that relate to perception of value and others that relate a bug with how the price is displayed for example. These are different insights with different solutions. 

What to use

Using a Miro board for this process can really help. You can use sticky notes to represent individual issues, colour coded by participant and then you can group sticky notes with the same or similar insights together. You can also label these groups of sticky notes with the theme title. If you’re getting overwhelmed by the spreadsheet method, this is a more visual way of conducting the analysis. 

This process will ultimately help you determine what the ‘big hitter’ insights are from your research project as they will be the insights that come up again and again across different session transcripts. 

By the end of this process, you should have uncovered plenty of opportunities for optimization.

Once you have a solid set of insight themes, you can start thinking about solutions. The best way to do this is to categorize your individual insights into solution buckets. This is how we bucket issues: 

It's then important to create an action plan/recommendation for each issue. You can use our PXL framework to help you prioritize your recommendations. 

Each recommendation should be ranked in order of potential for your business, and then broken down into the next steps. For example, will you need to carry out an A/B test to monitor the success of any new design features? 

Depending on the project, you may or may not want to write up a formal report of insights, however doing so is the best way to ensure your research gets good visibility and ultimately sparks impact across the business. 

There are varying degrees of detail you could go into with your research which should be determined by the intended audience. I would recommend focusing on your ‘key themes’ in detail in your report and then providing access to your coded transcripts for anyone interested in the specific details. 

Each key theme section should include information on:

1. Theme
2. What happened?
3. To how many people?
4. On what device/s?
5. What was the impact of this? (you could use quotes to illustrate this)
6. Recommendation/s / next steps (in the case of UX issues, how do we stop this happening?) 
   

For some insights, using video footage of the participant can help to illustrate the insight and can have a greater impact than written commentary. I would recommend using this method either in the case where the insight is technical / complex or when the impact on the participant was significant, an issue that would cause a user to leave the website and take their money elsewhere for example. 

Conclusion

To wrap up, here's how you can gain unbeatable insights from your research:

1. Create a research plan and give yourself plenty of time to organize the research.
2. Recruit the right participants to answer your questions and use screening questions to avoid the wrong ones.
3. Decide where you should conduct your research based on your research goals and budget. 
4. Create an interview outline or discussion guide that will address your core research objectives.
5. Write questions, tasks, or discussion topics. 
6. Moderate your user research sessions. Have two researchers: one to ask questions, and another to carefully document the responses.
7. Document your findings. 
8. Analyze your user research data using a spreadsheet or Miro board (or sticky notes on a wall). 
9. Create an action plan. Invest time in analyzing, proritinings and classifying the best next steps. 
10. Report your findings and share them with your wider team/business. 

‍