Moderated Vs. Unmoderated Usability Testing: Which Is The Right Method?

When running usability tests, one dilemma that UX researchers, product managers and designers encounter is choosing between moderated and unmoderated tests. Choosing between moderated vs unmoderated usability testing isn't about finding a right or wrong answer.

Each approach offers distinct advantages in gathering customer insights. Nonetheless, both moderated and unmoderated usability testing approaches have key differences. By the conclusion of this discussion, you'll gain clarity in determining which method aligns best with your product team's dynamics, workflow, and resource considerations.

What is moderated usability testing?

Moderated usability testing involves real-time, human-to-human interaction, with a designated moderator or facilitator guiding participants throughout the test session. The moderator plays a pivotal role in facilitating the overall session, addressing inquiries and follow-up questions, and delving into unplanned topics that may arise spontaneously during the test, fostering organic conversations that unveil unexpected insights.

Unlike unmoderated testing, which restricts testers to pre-defined questions, moderated testing offers the flexibility to deep dive into emergent topics. Whether conducted remotely or in person, moderators rely on cues such as body language and facial expressions to gauge test subjects' responses effectively.

Traditionally, in-person testing face several limitations due to all the various logistics involved from inviting participants, welcoming to the lab, and so forth. It also entails limited diversity due to geographic constraints. However, advancements in technology have expanded its reach and streamlined analysis processes. Remote moderated usability testing is now a regular practice within many teams.

Video conferencing softwares like Microsoft Teams and Zoom facilitate global recruitment panel, while AI-powered tools such as Hubble's remote moderated testing automate analysis tasks, enabling researchers to efficiently extract and summarize qualitative feedback. These innovations enhance the effectiveness and efficiency of moderated usability testing process, empowering product teams to glean valuable insights and iterate on their designs more swiftly than ever before.

What are the pros of moderated usability testing?

1. Real-Time Interaction: User testing highlights the detailed user actions as moderator and observers are present during the session, making it easy to scrutinize details and ask follow-up questions in real time.
2. In-Depth Exploration: Moderators can delve deeper insights, probing for underlying reasons and exploring unexpected insights by asking detailed questions.
3. Flexibility: Moderators have the flexibility to adapt the test flow based on a person's reactions and responses, allowing for a more personalized and tailored testing experience.
4. Participant Engagement: Interactions with a moderator often lead to increased study engagement and establish trust, as users feel more comfortable expressing their thoughts and opinions in a conversational setting.
5. Facilitated Collaboration: Moderated sessions facilitate collaboration among team members, as stakeholders can observe the session in real-time, ask follow-up questions, and discuss observations together.
6. Non-Verbal Cues: Moderators can observe participants' non-verbal cues such as body language, facial expressions and user actions in detail, providing additional context to their verbal responses. This is especially effective during an in-person session.
7. Effective Problem Identification: Moderators can identify usability issues as they occur, enabling immediate intervention and resolution, which is crucial for optimizing user experience.
8. Iterative Improvement: By uncovering actionable insights during moderated testing, teams can iteratively improve their designs, leading to more user-centric products and ultimately higher user satisfaction.

What are the cons of moderated usability testing?

1. Time-Intensive: Moderated testing sessions typically require significant time commitments from both participants and moderators, making them less scalable compared to unmoderated testing.
2. Costly: The need for a skilled moderator and the time investment involved can make moderated testing more expensive than unmoderated methods, particularly if conducted in person.
3. Geographical Constraints: Conducting moderated testing in person may limit the pool of participants to those within close proximity, potentially excluding diverse perspectives from remote or international users.
4. Subjectivity: Moderators' interpretations of participants' responses may introduce bias into the analysis, especially if moderators have preconceived notions about the product or user behavior.
5. Participant Influence: The presence of a moderator may influence participants' behavior and responses, potentially leading to altered or less authentic feedback compared to unmoderated settings.
6. Limited Observations: Moderators may miss subtle cues or overlook important details during the session, particularly if they are managing multiple participants simultaneously.
7. Resource-Intensive Analysis: Post-test analysis in moderated testing often involves manually reviewing recordings or transcripts, which can be time-consuming and labor-intensive, especially for large datasets.
8. Difficulty in Scaling: Scaling moderated testing to accommodate a larger number of participants or iterations can be challenging due to logistical constraints and resource requirements.

Despite these drawbacks, moderated usability testing still remains a valuable method for gaining qualitative insights and fostering user-centric design decisions when used judiciously and in conjunction with other research approaches.

How can I run more effective moderated usability tests?

Hubble's UX Researcher, Jin Jeon, draws from experiences working as a UX researcher at Microsoft and shares 9 tips for product and research teams so that they can run smoother moderated user tests:

1. Plan and prepare thoroughly

First and foremost, outline clear research goals, develop a structured test plan, and ensure all materials are ready before the session begins. Familiarize yourself with the testing protocol or any software or tools you'll be using. Making sure that you have all the basics covered and are well prepared before kicking off the sessions significantly increases the quality of the research as well as the efficiency of the research processes.

2. Recruit diverse participants

Decide on your main recruiting strategy and aim for a diverse pool of participants that reflects your target user demographics. Consider factors such as age, gender, experience level, and cultural background to gather comprehensive feedback. Use external participant pools such as Hubble's participant pool to find personas that match your criteria.

If you have the option, also use internal users to run research. Having a diverse source of users will ensure to minimize the impacts of cognitive bias and provide you with more diverse sample data that will help you validate your assumptions.

3. Establish rapport with study participants

Build rapport with participants at the start of the session to create a comfortable and open atmosphere. Explain the purpose of the test, set expectations, and encourage participants to voice their thoughts and opinions freely. Always remember that the most valuable insights from moderated studies come from free opinions and thoughts so make sure to warm up your participants so that they can be honest and not be afraid of thinking out loud.

4. Practice active listening and don't be afraid of silences

During the test, it's crucial to maintain a neutral and unobtrusive stance. Allowing the user to navigate tasks independently, including moments of search and struggle, yields invaluable insights. Jin says "Make sure to refraining from reacting to participants' actions, whether they encounter ease or difficulty. Instead, remain curious and attentive to their behavior."

Jin also advises against providing hints, likening it to taking control of the task yourself. If a person appears uncomfortable, reassure them that there are no right or wrong answers, no judgment, and no pressure involved in the process.

5. Encourage Thinking Aloud

Thinking out loud is one of the most desired behaviors during moderated usability tests. Jin recommends: "When conducting moderated usability tests, I would always encourage participants to verbalize their thought processes as they are interacting with the product. I would be amazed because of the insights and the hidden intentions that our participants would have."

6. Stay neutral and guard against cognitive bias

One of the most important factors in ensuring high quality results from moderated user testing is minimizing the impact of bias and subjectivity because moderated studies can be heavily subjective when noting behavioral trends around user's body language and facial expressions. For example, what one researcher considers worth noting down, another may not even notice.

Furthermore, the expression of a particular emotion can vary significantly between individuals, influenced by factors such as cultural backgrounds, neurodiversity, and age. This variance not only affects how users convey their opinions but also influences how moderators perceive and interpret them.

Jin says: "Moderated studies always have room for cognitive bias because by nature, the data is subjective. Set up a testing guide and make sure to use the testing script as a reference to keep consistency across different sessions."

If you need help analyzing and summarizing qualitative insights from moderated tests, please read this blog post by Jin on using thematic analysis and affinity mapping.

7. Be open minded and adapt to participant needs

Be flexible in your approach, adjusting the test flow and level of guidance based on participants' comfort levels and preferences. Tailor your interactions to accommodate diverse personalities and communication styles. This is particularly important if you are targeting specific user personas for your moderated usability tests.

8. Manage your time effectively

Time management is of utmost importance for moderated sessions. Always keep the session on track by managing time efficiently. Allocate sufficient time for each task or scenario while ensuring you have ample time for debriefing and addressing additional questions.‍

9. Document your observations

Closely document observations, insights, and notable quotes during the session to maintain an accurate record of participant feedback. Use a structured note-taking method or recording software to capture key moments. Make sure to share these insights from your team (especially if they participated in the same sessions) so that you can compare notes and cross check your insights.‍

10. Conduct post-session debriefs

Schedule debrief sessions with your team immediately after each test to discuss observations, share insights, and brainstorm potential improvements with the rest of your product team. Collaborate on interpreting findings and prioritizing actionable next steps.

When should I use moderated usability tests?

Moderated usability tests are well-suited for stages in the product development process where additional explanation and guidance are necessary. This applies to both the researcher, who may need to clarify aspects of the prototype, and the test subject, who may require assistance understanding and articulating their answers. For instance, during prototype testing, participants might benefit from live support to navigate tasks, or they may need further clarification on the prototype's limited functionality.

Given these characteristics, moderated testing naturally fits into the early phases of the design process. Here, qualitative insights regarding opinions and overall product experience can inform subsequent functionalities and specific UX features. Conducting moderated testing early on allows UX teams to engage with participants throughout the development cycle, probing deeper into their responses and presenting revised iterations based on feedback.

This approach fosters a culture of continuous product discovery, enabling design teams to gather feedback from ideal customer profiles and iterate on designs accordingly. It also provides opportunities to revisit participants to assess if their feedback has been effectively incorporated into each product rendition.

Moderated usability tests offer several advantages that can greatly benefit product and UX teams:‍

• Early Identification of Issues: By conducting moderated usability tests in the early stages of product development, product and UX teams can identify usability issues, pain points, and areas for improvement before investing significant resources in development.
• Validation of Design Decisions: Moderated testing provides an opportunity to validate design decisions and hypotheses. By observing how users interact with prototypes or early versions of the product, teams can assess whether the design effectively addresses user needs and goals.
• Alignment Across Stakeholders: Moderated testing sessions often involve multiple team members and stakeholders observing the sessions. This promotes alignment and shared understanding of user needs and priorities across the team, helping to ensure that everyone is working towards the same goals.
• Iterative Design Process: Moderated testing supports an iterative design process, allowing teams to make incremental improvements based on user feedback. By testing multiple iterations of the product with participants, teams can refine the design and optimize the user experience over time.

‍

What is unmoderated usability testing?

Unmoderated usability testing does not involve a moderator. In this method, users independently complete tasks using usability testing tools that record their decisions and actions.

In this self-paced remote testing participants are solely responsible for completing tasks and answering questions to the best of their ability, without the option to seek assistance. Typically, testers conduct the test in their own natural environment or space.

Unmoderated, remote usability testing is frequently employed for high-fidelity prototypes that require minimal explanations. This approach is particularly suitable for the final stages of design, such as when you have a fully interactive app that is nearly ready for launch and only requires final tweaks.

Unmoderated usability testing is also used when product teams:

• Require rapid access to abundant responses
• Seek broader demographic engagement
• Desire quantitative data for hypothesis validation
• Operate within budget and time constraints
  

Elevate your continuous UX Research

Validate your concepts and collect post-launch feedback through Hubble

Schedule a demo

What are the pros of unmoderated usability testing?

1. Scalability: Unmoderated tests can accommodate a larger number of participants and can be conducted simultaneously, making them highly scalable.
2. Convenience: Participants can complete tests at their own time, reducing scheduling conflicts and enabling broader participation from diverse demographics.
3. Cost-Effectiveness: With no need for a moderator's time or presence, unmoderated tests are often more cost-effective, making them suitable for teams with limited budgets.
4. Efficiency: Unmoderated tests can be set up quickly and run autonomously, allowing researchers to focus on other tasks while data is collected.
5. Reduced Bias: The absence of a moderator can mitigate bias in responses, as there is no influence or interpretation from a third party.
6. Quantitative Insights: Unmoderated tests are well-suited for gathering quantitative data, such as completion rates, task times, and success metrics, providing valuable usability metrics for analysis.
7. Flexibility: Participants have the freedom to navigate the test at their own pace and provide feedback in their own words, offering insights that may not emerge in a structured, moderated setting.

What are the cons of unmoderated usability testing?

1. Limited Insight Depth: Without a moderator to probe and clarify responses, unmoderated tests may provide less nuanced insights into user behaviors and motivations.
2. Inability to Address Complex Scenarios: Certain tasks or scenarios may require guidance or clarification, which unmoderated tests may struggle to provide effectively.
3. Difficulty in Handling Unexpected Issues: Technical glitches or participant confusion may arise during unmoderated tests, requiring researchers to troubleshoot remotely without direct interaction.
4. Lack of Immediate Feedback: Researchers cannot address issues or clarify instructions in real-time during unmoderated tests, potentially delaying insights or necessitating follow-up studies.

Because of the lack of immediate feedback and interaction, unmoderated tests must be tested thoroughly to ensure that there are no unclear instructions and lack of clarity for every tester.

When should I use unmoderated usability tests?

Unmoderated usability tests offer immense value at any stage of the design process, particularly when extensive data collection or feedback from diverse participants is needed.

Unmoderated testing is especially beneficial for teams with budget constraints or time limitations, as its remote testing process allows test participants to complete the study at their own convenience. It is a cost effective solution to easily scale relative to moderated tests.

Unmoderated testing is well-suited for straightforward UX processes that require minimal guidance, making it a popular choice for evaluating high-fidelity prototypes in later stages of the design process.

Unmoderated tests is a diverse set of tools that can include other usability testing methods:

• IA Tests: Use card sorting to see if users can properly categorize information based on the applications structure
• UX writing tests: Supporting the launch of your design and helping your team to decide on which copy to use in the product and marketing
• A/B test prototypes: Split test everything from graphics to button colors and in-app messaging with a design preference test for usability
• Test design hypotheses: Set up an unmoderated usability test to see if what you think will work actually does, then move forward based on data

Even though unmoderated usability testing is commonly employed towards the conclusion of a design process, when assessing final designs or testing prototypes, many organizations choose to utilize it throughout the entire design process due to its cost-effectiveness and potential for significant benefits, provided participants are adequately briefed and the testing platform is intuitive.

How do I choose between unmoderated vs. moderated tests?

There's no definitive answer on whether moderated vs unmoderated usability tests are superior. Both methods hold value in any design process.

Please ask the following questions before choosing between the two methods:

• Which usability metrics do you intend to track?
• What specific knowledge gap needs to be addressed to progress your design?
• What user feedback are you looking for? How do you intend on gathering user feedback?
• What is your research timeline?
• What type of artifact are you testing?
• What is your budget allocation?
• Where is your target audience geographically located?

Once you've addressed these questions, you'll be better equipped to determine the most suitable type of usability testing for your needs. If you have the time and resources, utilizing both moderated and unmoderated testing may prove beneficial. Jin says: "I've always preferred to utilize both moderated and unmoderated studies for my projects working at both large enterprises and startups. They complement each other to uncover so much user insights and have very distinct pros and cons that are important for early stage of the product development process."

Hubble offers both moderated and unmoderated study support with access to more than 8 million external test participants from more than 150 different countries partnered with Respondent and User Interviews.

Using Hubble, you can conduct moderated usability testing sessions and unmoderated research for various types of usability testing methods such as card sorting, prototype tasks, IA tests, design preference tests and other types of surveys.

Both unmoderated and moderated studies empower product managers, user researchers and designers to eliminate errors, test their product features and ensure success through powerful continuous discovery and customer centric product development.

Elevate your continuous UX Research

Validate your concepts and collect post-launch feedback through Hubble

Schedule a demo

‍