Moderated vs. Unmoderated Testing

How do I choose between moderated vs. unmoderated usability tests?

When running usability tests, one dilemma that UX researchers, product managers and designers have face is choosing between moderated and unmoderated tests. Choosing between moderated and unmoderated usability testing isn't about finding a right or wrong answer. Each method offers distinct advantages in gathering essential user experience insights, aiding in the creation of products tailored to user needs. Nonetheless, both moderated and unmoderated usability testing approaches present their own set of advantages and drawbacks. By the conclusion of this discussion, you'll gain clarity in determining which method aligns best with your product team's dynamics, workflow, and financial considerations.

What is moderated usability testing?

Moderated usability testing involves real-time, human-to-human interaction, with a designated moderator or facilitator guiding participants throughout the test session. The moderator plays a pivotal role in recording observations, addressing inquiries, and delving into unplanned topics that may arise spontaneously during the test, fostering organic conversations that unveil unexpected insights. Unlike unmoderated testing, which restricts participants to pre-defined questions, moderated testing offers the flexibility to explore emergent topics in depth. Whether conducted remotely or in person, moderators rely on cues such as body language and facial expressions to gauge participants' responses effectively.

Traditionally, moderated testing faced limitations in terms of participant diversity due to geographic constraints and the labor-intensive nature of post-test analysis. However, advancements in technology have expanded its reach and streamlined analysis processes. Video conferencing platforms like Google and Zoom facilitate global participant recruitment, while AI-powered tools such as Hubble's moderated studies automate analysis tasks, enabling researchers to efficiently extract and summarize qualitative feedback. These innovations enhance the effectiveness and efficiency of moderated usability testing, empowering product teams to glean valuable insights and iterate on their designs more swiftly than ever before.

What are the pros of moderated usability testing?

1. Real-Time Interaction: With a moderator guiding the session, participants receive immediate feedback and can clarify doubts or concerns in real-time, leading to a more comprehensive understanding of user experiences.
2. In-Depth Exploration: Moderators can delve deeper into participants' responses, probing for underlying reasons and exploring unexpected insights, which may not emerge in a structured, unmoderated setting.
3. Flexibility: Moderators have the flexibility to adapt the test flow based on participants' reactions and responses, allowing for a more personalized and tailored testing experience.
4. Participant Engagement: Interaction with a moderator often leads to higher participant engagement, as users feel more comfortable expressing their thoughts and opinions in a conversational setting.
5. Facilitated Collaboration: Moderated sessions facilitate collaboration among team members, as stakeholders can observe the session in real-time, ask questions, and discuss observations together.
6. Non-Verbal Cues: Moderators can observe participants' non-verbal cues such as body language and facial expressions, providing additional context to their verbal responses.
7. Effective Problem Identification: Moderators can identify usability issues as they occur, enabling immediate intervention and resolution, which is crucial for optimizing user experience.
8. Iterative Improvement: By uncovering actionable insights during moderated testing, teams can iteratively improve their designs, leading to more user-centric products and ultimately higher user satisfaction.

What are the cons of moderated usability testing?

1. Time-Intensive: Moderated testing sessions typically require significant time commitments from both participants and moderators, making them less scalable compared to unmoderated testing.
2. Costly: The need for a skilled moderator and the time investment involved can make moderated testing more expensive than unmoderated methods, particularly if conducted in person.
3. Geographical Constraints: Conducting moderated testing in person may limit the pool of participants to those within close proximity, potentially excluding diverse perspectives from remote or international users.
4. Subjectivity: Moderators' interpretations of participants' responses may introduce bias into the analysis, especially if moderators have preconceived notions about the product or user behavior.
5. Participant Influence: The presence of a moderator may influence participants' behavior and responses, potentially leading to altered or less authentic feedback compared to unmoderated settings.
6. Limited Observations: Moderators may miss subtle cues or overlook important details during the session, particularly if they are managing multiple participants simultaneously.
7. Resource-Intensive Analysis: Post-test analysis in moderated testing often involves manually reviewing recordings or transcripts, which can be time-consuming and labor-intensive, especially for large datasets.
8. Difficulty in Scaling: Scaling moderated testing to accommodate a larger number of participants or iterations can be challenging due to logistical constraints and resource requirements.

Despite these drawbacks, moderated usability testing remains a valuable method for gaining qualitative insights and fostering user-centric design decisions when used judiciously and in conjunction with other research approaches.

How can I run more effective moderated usability tests?

Hubble’s UX Researcher, Jin Jeon, draws from experiences working as a UX researcher at Microsoft and shares 9 tips for product and research teams so that they can run smoother moderated user tests:

1. Plan and prepare thoroughly

First and foremost, outline clear objectives, develop a structured test plan, and ensure all materials are ready before the session begins. Familiarize yourself with the testing protocol and any software or tools you'll be using. Making sure that you have all the basics covered and are well prepared before kicking off the sessions significantly increases the quality of the research as well as the efficiency of the research processes.

2. Recruit diverse participants

Decide on your main participant recruitment strategy and aim for a diverse pool of participants that reflects your target user demographics. Consider factors such as age, gender, experience level, and cultural background to gather comprehensive feedback. Use external participant pools such as Hubble's participant pool to find personas and participants that match your criteria.

If you have the option, also use internal users and participants to run your research. Having a diverse source of participants will ensure to minimize the impacts of cognitive bias and provide you with more diverse sample data that will help you validate your assumptions.

3. Establish rapport with study participants

Build rapport with participants at the start of the session to create a comfortable and open atmosphere. Explain the purpose of the test, set expectations, and encourage participants to voice their thoughts and opinions freely. Always remember that the most valuable insights from moderated studies come from free opinions and thoughts so make sure to warm up your participants so that they can be honest and not be afraid of thinking out loud.

4. Practice active listening and don't be afraid of silences

During the test, it's crucial to maintain a neutral and unobtrusive stance. Allowing the participant to navigate tasks independently, including moments of search and struggle, yields invaluable insights. Jin says "Make sure to refraining from reacting to participants' actions, whether they encounter ease or difficulty. Instead, remain curious and attentive to their behavior."

Jin also advises against providing hints, likening it to taking control of the task yourself. If a participant appears uncomfortable, reassure them that there are no right or wrong answers, no judgment, and no pressure involved in the process.

5. Encourage Thinking Aloud

Thinking out loud is one of the most desired behaviors during moderated usability tests. Jin recommends: "When conducting moderated usability tests, I would always encourage participants to verbalize their thought processes as they are interacting with the product. I would be amazed because of the insights and the hidden intentions that our participants would have."

6. Stay neutral and guard against cognitive bias

One of the most important factors in ensuring high quality results from moderated usability tests is minimizing the impact of bias and subjectivity because moderated studies can be heavily subjective when noting behavioral trends around body language and facial expressions. For example, what one researcher considers worth noting down, another may not even notice.

Furthermore, the expression of a particular emotion can vary significantly between individuals, influenced by factors such as cultural backgrounds, neurodiversity, and age. This variance not only affects how participants convey their opinions but also influences how moderators perceive and interpret them.

Jin says: "Moderated studies always have room for cognitive bias because by nature, the data is subjective. Set up a testing guide and make sure always to stick to the testing script, asking the same questions instead of asking different questions to different participants."

If you need help analyzing and summarizing qualitative insights from moderated tests, please read this blog post by Jin on using thematic analysis and affinity mapping.

7. Be open minded and adapt to participant needs

Be flexible in your approach, adjusting the test flow and level of guidance based on participants' comfort levels and preferences. Tailor your interactions to accommodate diverse personalities and communication styles. This is particularly important if you are targeting specific user personas for your moderated usability tests.

8. Manage your time effectively

Time management is of utmost importance for moderated sessions. Always keep the session on track by managing time efficiently. Allocate sufficient time for each task or scenario while ensuring you have ample time for debriefing and addressing additional questions.‍

9. Capture your observations

Document observations, insights, and notable quotes during the session to maintain an accurate record of participant feedback. Use a structured note-taking method or recording software to capture key moments. Make sure to share these insights from your team (especially if they participated in the same sessions) so that you can compare notes and cross check your insights.‍

10. Conduct post-session debriefs

Schedule debrief sessions with your team immediately after each test to discuss observations, share insights, and brainstorm potential improvements with the rest of your product team. Collaborate on interpreting findings and prioritizing actionable next steps.

When should I use moderated usability tests?

Moderated usability tests are well-suited for stages in the product development process where additional explanation and guidance are necessary. This applies to both the researcher, who may need to clarify aspects of the prototype, and the participant, who may require assistance understanding and articulating their answers. For instance, during prototype testing, participants might benefit from live support to navigate tasks, or they may need further clarification on the prototype's limited functionality.

Given these characteristics, moderated testing naturally fits into the early phases of the design process. Here, qualitative insights regarding opinions and overall product experience can inform subsequent functionalities and specific UX features. Conducting moderated testing early on allows UX teams to engage with participants throughout the development cycle, probing deeper into their responses and presenting revised iterations based on feedback.

This approach fosters a culture of continuous product discovery, enabling design teams to gather feedback from ideal customer profiles and iterate on designs accordingly. It also provides opportunities to revisit participants to assess if their feedback has been effectively incorporated into each product rendition.

Moderated usability tests offer several advantages that can greatly benefit product and UX teams:‍

• Early Identification of Issues: By conducting moderated usability tests in the early stages of product development, product and UX teams can identify usability issues, pain points, and areas for improvement before investing significant resources in development.
• Validation of Design Decisions: Moderated testing provides an opportunity to validate design decisions and hypotheses. By observing how users interact with prototypes or early versions of the product, teams can assess whether the design effectively addresses user needs and goals.
• Alignment Across Stakeholders: Moderated testing sessions often involve multiple team members and stakeholders observing the sessions. This promotes alignment and shared understanding of user needs and priorities across the team, helping to ensure that everyone is working towards the same goals.
• Iterative Design Process: Moderated testing supports an iterative design process, allowing teams to make incremental improvements based on user feedback. By testing multiple iterations of the product with participants, teams can refine the design and optimize the user experience over time.

What is unmoderated usability testing?

Unmoderated usability testing, as the name implies, does not involve a moderator. In this method, users independently complete tasks using usability testing tools that record their decisions and actions.

In this self-paced test process, participants are solely responsible for completing tasks and answering questions to the best of their ability, without the option to seek assistance. Typically, testers conduct the test in their own environment or space.

Unmoderated, remote usability testing is frequently employed for high-fidelity prototypes that require minimal explanations. This approach is particularly suitable for the final stages of design, such as when you have a fully interactive app that is nearly ready for launch and only requires final tweaks.

Unmoderated usability testing is also used when product teams:

• Require rapid access to abundant responses
• Seek broader demographic engagement
• Desire quantitative data for hypothesis validation
• Operate within budget constraints
  

Elevate your continuous UX Research

Validate your concepts and collect post-launch feedback through Hubble

Schedule a demo

What are the pros of unmoderated usability testing?

1. Scalability: Unmoderated tests can accommodate a larger number of participants and can be conducted simultaneously, making them highly scalable.
2. Convenience: Participants can complete tests at their own convenience, reducing scheduling conflicts and enabling broader participation from diverse demographics.
3. Cost-Effectiveness: With no need for a moderator's time or presence, unmoderated tests are often more cost-effective, making them suitable for teams with limited budgets.
4. Efficiency: Unmoderated tests can be set up quickly and run autonomously, allowing researchers to focus on other tasks while data is collected in the background.
5. Reduced Bias: The absence of a moderator can mitigate bias in participant responses, as there is no influence or interpretation from a third party.
6. Quantitative Insights: Unmoderated tests are well-suited for gathering quantitative data, such as completion rates, task times, and success metrics, providing valuable metrics for analysis.
7. Flexibility: Participants have the freedom to navigate the test at their own pace and provide feedback in their own words, offering insights that may not emerge in a structured, moderated setting.

What are the cons of unmoderated usability testing?

1. Limited Insight Depth: Without a moderator to probe and clarify responses, unmoderated tests may provide less nuanced insights into user behaviors and motivations.
2. Inability to Address Complex Scenarios: Certain tasks or scenarios may require guidance or clarification, which unmoderated tests may struggle to provide effectively.
3. Difficulty in Handling Unexpected Issues: Technical glitches or participant confusion may arise during unmoderated tests, requiring researchers to troubleshoot remotely without direct interaction.
4. Lack of Immediate Feedback: Researchers cannot address issues or clarify instructions in real-time during unmoderated tests, potentially delaying insights or necessitating follow-up studies.

Because of the lack of immediate feedback and interaction, unmoderated tests must be tested thoroughly to ensure that there are no unclear instructions and lack of clarity for every tester.

When should I use unmoderated usability tests?

Unmoderated usability tests offer immense value at any stage of the design and development process, particularly when extensive data collection or feedback from diverse participants is needed.

Unmoderated testing is especially beneficial for teams with budgetary constraints or time limitations, as it allows for the setup of tests to autonomously gather responses while team members can focus on other pressing tasks.

Unmoderated testing is well-suited for straightforward UX processes that require minimal guidance, making it a popular choice for evaluating high-fidelity prototypes in later stages of the design process.

Unmoderated usability tests can also be used to help UX design teams:

• IA Tests: Use card sorting to see if users can properly categorize information based on the applications structure
• UX writing tests: Supporting the launch of your design and helping your team to decide on which copy to use in the product and marketing
• A/B test prototypes: Split test everything from graphics to button colors and in-app messaging with a design preference test for usability
• Test design hypotheses: Set up an unmoderated usability test to see if what you think will work actually does, then move forward based on data

Even though unmoderated usability testing is commonly employed towards the conclusion of a design process, when assessing final designs or testing prototypes, many organizations choose to utilize it throughout the entire design process due to its cost-effectiveness and potential for significant benefits, provided participants are adequately briefed and the testing platform is intuitive.

How do I choose between moderated and unmoderated usability tests?

There's no definitive answer on whether moderated or unmoderated usability tests are superior. Both methods hold value in any design process.

Please ask the following questions before choosing between the two methods:

• Which usability metrics do you intend to track?
• What specific knowledge gap needs to be addressed to progress your design?
• What is your timeline for obtaining results?At what stage is your design: high-fidelity or low-fidelity?
• What is your budget allocation?
• Where is your target audience geographically located?

Once you've addressed these questions, you'll be better equipped to determine the most suitable type of usability testing for your needs. In certain situations, utilizing both methods may prove beneficial. Jin says: "I've always used both moderated and unmoderated studies for my projects working at both large enterprises and startups. They both uncover so much and have very distinct pros and cons that are important for eery stage of the product development process"

Hubble offers both moderated and unmoderated study support with access to more than 3 million external participants from more than 150 different countries. Using Hubble's Zoom integration, you can run moderated user research sessions while also conducting unmoderated studies to run different types of tasks such as prototype tasks, IA tests, design preference tests and other types of surveys. Both unmoderated and moderated studies empower product managers, user researchers and designers to eliminate errors, test their product features and ensure success through powerful continuous discovery of user research insights.

Elevate your continuous UX Research

Validate your concepts and collect post-launch feedback through Hubble

Schedule a demo

‍

‍