Presentation
Making Sense of the Gaps: Bridging FDA Guidance and Real-World Complexity in OTC Self-Selection Testing
DescriptionTopic & Background
This presentation builds on the authors’ HFES 2023 talk, “Validating Medical Device Self-Selection: The Journey Toward Understanding the Latest Medical Device OTC Guidelines.” In 2025, more over-the-counter (OTC) Software as a Medical Device (SaMD) products are reaching consumers through complex, multi-channel ecosystems. This talk addresses new complexities and unresolved issues in self-selection testing for OTC devices, especially SaMDs. The FDA guidance for self-selection – defined here as including CDRH’s 2016 and 2022 human factors (HF) guidance documents, CDER’s 2013 Self-Selection Studies for Nonprescription Drug Products guidance, relevant CDRH online resources about OTC medical devices, and CDRH’s presentations on self-selection testing methodologies -- offers a helpful framework. However, gray areas remain that affect study design, data quality, and safety.
This presentation will examine the self-selection process for OTC SaMDs, highlighting three challenges our HF team faced when applying the FDA guidelines outlined above. The goal is not to prescribe solutions, but rather to spark discussion on practical validation approaches. Because each device is unique, FDA pre-submissions remain essential to ensure methods are tailored appropriately.
Application
As more devices, particularly SaMDs, are available directly to consumers, ensuring accurate self-selection is a regulatory and ethical imperative. The FDA has provided helpful methodologies and parameters describing how to test self-selection. However, there are many specific methods yet to be fully defined, particularly when recruiting diverse user groups, such as minors or older adults with limited technology experience, or when determining where to initiate the self-selection workflow when testing in complex digital ecosystems. Effective self-selection testing requires a nuanced approach reflecting real-world behaviors, motivations, and access to technology.
Overview of Three Challenges and Practical Validation Ideas
1. Testing Critical Tasks for Non-Intended Users
Guidance Gap
FDA guidance generally does not consider non-intended users a primary testing group. However, for high-risk non-intended user subgroups with contraindicated conditions (e.g., pregnant women using an OTC Class II abdominal electrical muscle stimulator for body toning), self-selection is critical. Use of electrical stimulation over or near the uterus during pregnancy may cause uterine contractions or fetal harm, posing a risk of miscarriage, preterm labor, or other serious complications. The FDA recommends recruiting enough participants to obtain 15 intended users, but does not specify a minimum for non-intended users or even a requirement for their inclusion. Random recruitment from the general population may miss high-risk non-intended users, leaving them untested and vulnerable.
Validation Ideas
* Risk-based sampling minimums: Aim to recruit 15 participants per high-risk non-intended subgroup. If 15 is not feasible, set a minimum of 5. Research by usability expert Jakob Nielsen [https://www.nngroup.com/articles/how-many-test-users/] indicates that a single test participant can identify approximately 33% of usability issues, while five participants can uncover roughly 85%, with diminishing returns from additional participants. Recruiting at least 5 participants helps ensure reliable safety and effectiveness data.
* Structured proxy scenarios: When direct recruitment is impractical or challenging (e.g., rare patient populations or minors), consider incorporating scenarios involving natural proxies (such as parents or caregivers of minors) into self-selection performance testing to validate safe use.
* FDA pre-submission: If formative sessions reveal gaps in high-risk recruitment, deliberate recruitment plans with the FDA in a pre-submission.
2. Identifying and Testing Self-Selection Entry Points for SaMDs
Guidance Gap
The FDA suggests starting the self-selection workflow at the user’s first point of contact. For SaMD, this can include app stores, manufacturer websites, social media platforms, QR codes, packaging, press articles, and in-app notifications. Questions arise: With multiple entry points, which should be tested with the 15 intended users? How is official ‘device labeling’ distinguished from marketing content? Should testing include marketing that influences self-selection, and if so, how can we ensure that this marketing UI (e.g., a company product webpage) meets medical device standards?
Validation Ideas
* Most likely user entry path testing: Collaborate with the manufacturer to identify the most likely user entry points for their product—such as a website, app store, news headline, or word of mouth—and test how users respond at these real-world starting points to ensure appropriate self-selection.
* Multi-path assignment: Randomize or rotate entry points across participants to reflect real-world discovery paths, and analyze both pooled and path-specific outcomes. Collect formative data or use a neutral screener question to estimate how participants would access the SaMD and assign them accordingly.
* Targeted labeling evaluation: Test the effectiveness of key labeling regardless of entry point, and encourage sponsors to maintain consistent labeling across all critical entry points.
3. Self-Selection for Users with Technology Limitations
Guidance Gap
The FDA appears to assume intended users’ independent performance, but real-world behavior often includes informal support (children of seniors, caregivers, tech-savvy friends, parents of minors). Prohibiting all assistance during testing may exclude realistic intended users or create an inaccurate use scenario.
Validation Ideas
* Assistance-aware scoring: Ensure that users make self-selection decisions independently, without direct influence from the moderator, but allow test participants to receive assistance from simulated or real-world helpers when they would realistically be getting this form of help during actual use. Record assistance levels and analyze outcomes with and without help.
* Screen for realistic motivation: Include screener items that capture interest in and likelihood of technology adoption. This approach helps ensure participants genuinely represent potential users and avoids those who would never consider using the product. At the same time, it's essential to balance these recruiting criteria to avoid being overly restrictive, which could reduce sample diversity and limit the study’s applicability.
* Exit criteria for unrealistic users: Define when an enrolled intended user should be classified as ineligible (i.e., refuses or cannot engage with the technology), and when the session should be terminated. Document the decision, reason, and stage of exit to separate protocol deviations from usability findings.
Importance of the Message
Strict adherence to FDA guidance alone may not ensure the comprehensive safety or usability of OTC devices, as the current guidance does not cover all situations that may arise during testing. Real-world complexity, including high-risk non-intended users, multiple entry points, and technological barriers for some users, can create gaps in HF data. The goal of HF testing is to align with FDA intent by applying the spirit of FDA guidance to ensure user safety and effective study design.
Key Takeaways
1. High-risk non-intended users may be missed: FDA guidance does not require quotas for non-intended users, even when incorrect self-selection could cause serious harm. Random recruitment may fail to capture these participants, creating safety and ethical blind spots. Solutions include risk-based minimum sample sizes, structured proxy strategies, and FDA pre-submissions.
2. Identify and test SaMD self-selection entry points: Users may discover SaMDs through app stores, websites, marketing, QR codes, or digital media. Selecting a single starting point may overlook critical paths; allowing users to choose freely can skew the results. Strategies include testing the most likely user entry path, multi-path testing, and targeted labeling evaluation.
3. Technological barriers exclude some intended users: Strictly prohibiting naturally occurring assistance during testing can lead to invalid data and may discourage continued use among tech-wary intended users. Solutions include assistance-aware scoring, targeted screening, and well-defined exit criteria to maintain real-world relevance while preserving accurate results.
This presentation builds on the authors’ HFES 2023 talk, “Validating Medical Device Self-Selection: The Journey Toward Understanding the Latest Medical Device OTC Guidelines.” In 2025, more over-the-counter (OTC) Software as a Medical Device (SaMD) products are reaching consumers through complex, multi-channel ecosystems. This talk addresses new complexities and unresolved issues in self-selection testing for OTC devices, especially SaMDs. The FDA guidance for self-selection – defined here as including CDRH’s 2016 and 2022 human factors (HF) guidance documents, CDER’s 2013 Self-Selection Studies for Nonprescription Drug Products guidance, relevant CDRH online resources about OTC medical devices, and CDRH’s presentations on self-selection testing methodologies -- offers a helpful framework. However, gray areas remain that affect study design, data quality, and safety.
This presentation will examine the self-selection process for OTC SaMDs, highlighting three challenges our HF team faced when applying the FDA guidelines outlined above. The goal is not to prescribe solutions, but rather to spark discussion on practical validation approaches. Because each device is unique, FDA pre-submissions remain essential to ensure methods are tailored appropriately.
Application
As more devices, particularly SaMDs, are available directly to consumers, ensuring accurate self-selection is a regulatory and ethical imperative. The FDA has provided helpful methodologies and parameters describing how to test self-selection. However, there are many specific methods yet to be fully defined, particularly when recruiting diverse user groups, such as minors or older adults with limited technology experience, or when determining where to initiate the self-selection workflow when testing in complex digital ecosystems. Effective self-selection testing requires a nuanced approach reflecting real-world behaviors, motivations, and access to technology.
Overview of Three Challenges and Practical Validation Ideas
1. Testing Critical Tasks for Non-Intended Users
Guidance Gap
FDA guidance generally does not consider non-intended users a primary testing group. However, for high-risk non-intended user subgroups with contraindicated conditions (e.g., pregnant women using an OTC Class II abdominal electrical muscle stimulator for body toning), self-selection is critical. Use of electrical stimulation over or near the uterus during pregnancy may cause uterine contractions or fetal harm, posing a risk of miscarriage, preterm labor, or other serious complications. The FDA recommends recruiting enough participants to obtain 15 intended users, but does not specify a minimum for non-intended users or even a requirement for their inclusion. Random recruitment from the general population may miss high-risk non-intended users, leaving them untested and vulnerable.
Validation Ideas
* Risk-based sampling minimums: Aim to recruit 15 participants per high-risk non-intended subgroup. If 15 is not feasible, set a minimum of 5. Research by usability expert Jakob Nielsen [https://www.nngroup.com/articles/how-many-test-users/] indicates that a single test participant can identify approximately 33% of usability issues, while five participants can uncover roughly 85%, with diminishing returns from additional participants. Recruiting at least 5 participants helps ensure reliable safety and effectiveness data.
* Structured proxy scenarios: When direct recruitment is impractical or challenging (e.g., rare patient populations or minors), consider incorporating scenarios involving natural proxies (such as parents or caregivers of minors) into self-selection performance testing to validate safe use.
* FDA pre-submission: If formative sessions reveal gaps in high-risk recruitment, deliberate recruitment plans with the FDA in a pre-submission.
2. Identifying and Testing Self-Selection Entry Points for SaMDs
Guidance Gap
The FDA suggests starting the self-selection workflow at the user’s first point of contact. For SaMD, this can include app stores, manufacturer websites, social media platforms, QR codes, packaging, press articles, and in-app notifications. Questions arise: With multiple entry points, which should be tested with the 15 intended users? How is official ‘device labeling’ distinguished from marketing content? Should testing include marketing that influences self-selection, and if so, how can we ensure that this marketing UI (e.g., a company product webpage) meets medical device standards?
Validation Ideas
* Most likely user entry path testing: Collaborate with the manufacturer to identify the most likely user entry points for their product—such as a website, app store, news headline, or word of mouth—and test how users respond at these real-world starting points to ensure appropriate self-selection.
* Multi-path assignment: Randomize or rotate entry points across participants to reflect real-world discovery paths, and analyze both pooled and path-specific outcomes. Collect formative data or use a neutral screener question to estimate how participants would access the SaMD and assign them accordingly.
* Targeted labeling evaluation: Test the effectiveness of key labeling regardless of entry point, and encourage sponsors to maintain consistent labeling across all critical entry points.
3. Self-Selection for Users with Technology Limitations
Guidance Gap
The FDA appears to assume intended users’ independent performance, but real-world behavior often includes informal support (children of seniors, caregivers, tech-savvy friends, parents of minors). Prohibiting all assistance during testing may exclude realistic intended users or create an inaccurate use scenario.
Validation Ideas
* Assistance-aware scoring: Ensure that users make self-selection decisions independently, without direct influence from the moderator, but allow test participants to receive assistance from simulated or real-world helpers when they would realistically be getting this form of help during actual use. Record assistance levels and analyze outcomes with and without help.
* Screen for realistic motivation: Include screener items that capture interest in and likelihood of technology adoption. This approach helps ensure participants genuinely represent potential users and avoids those who would never consider using the product. At the same time, it's essential to balance these recruiting criteria to avoid being overly restrictive, which could reduce sample diversity and limit the study’s applicability.
* Exit criteria for unrealistic users: Define when an enrolled intended user should be classified as ineligible (i.e., refuses or cannot engage with the technology), and when the session should be terminated. Document the decision, reason, and stage of exit to separate protocol deviations from usability findings.
Importance of the Message
Strict adherence to FDA guidance alone may not ensure the comprehensive safety or usability of OTC devices, as the current guidance does not cover all situations that may arise during testing. Real-world complexity, including high-risk non-intended users, multiple entry points, and technological barriers for some users, can create gaps in HF data. The goal of HF testing is to align with FDA intent by applying the spirit of FDA guidance to ensure user safety and effective study design.
Key Takeaways
1. High-risk non-intended users may be missed: FDA guidance does not require quotas for non-intended users, even when incorrect self-selection could cause serious harm. Random recruitment may fail to capture these participants, creating safety and ethical blind spots. Solutions include risk-based minimum sample sizes, structured proxy strategies, and FDA pre-submissions.
2. Identify and test SaMD self-selection entry points: Users may discover SaMDs through app stores, websites, marketing, QR codes, or digital media. Selecting a single starting point may overlook critical paths; allowing users to choose freely can skew the results. Strategies include testing the most likely user entry path, multi-path testing, and targeted labeling evaluation.
3. Technological barriers exclude some intended users: Strictly prohibiting naturally occurring assistance during testing can lead to invalid data and may discourage continued use among tech-wary intended users. Solutions include assistance-aware scoring, targeted screening, and well-defined exit criteria to maintain real-world relevance while preserving accurate results.
Event Type
Oral Presentations
TimeWednesday, March 258:52am - 9:15am EDT
LocationGramercy
Medical and Drug Delivery Devices