Presentation
Bridging the Standards: A Comprehensive Roadmap for SaMD Usability Testing
DescriptionTopic:
This topic explores the evolving landscape for usability testing for software as a medical device (SaMD), focusing on unique challenges and real-world insights from the lens of existing standards and best practices.
Application:
As SaMD solutions become increasingly central to healthcare delivery—ranging from mobile diagnostic tools and patient-facing apps to complex clinical decision support systems—a framework for planning and executing usability testing tailored to the unique demands of SaMD is necessary. In this presentation, we will use existing standards and best practices to explore how to design usability testing for SaMD, bridge regulatory gaps, and compare SaMD to hardware medical devices and combination products.
Background:
The emergence of SaMD is transforming healthcare delivery, enabling innovative diagnostics, decision support, and patient engagement. Unlike traditional medical devices, SaMD relies on complex algorithms, dynamic user interfaces, artificial intelligence (AI), and integration with electronic health records (EHR) and other health IT systems. These features offer significant potential for improving patient care but also introduce unique usability and regulatory challenges.
A fundamental distinction lies in the variability of user interactions and use environments. SaMD may be used across different digital platforms—such as mobile phones, tablets, and desktop computers—and in diverse settings, ranging from hospitals and clinics to patients’ homes. This range of contexts, combined with varied levels of user digital literacy, makes it challenging to capture realistic use behaviors and anticipate all potential use scenarios. As a result, potential use problems may go undetected in conventional usability testing frameworks that were originally designed with static, hardware-based devices in mind.
Regulatory bodies such as the FDA and international organizations like the International Medical Device Regulators Forum (IMDRF) have established foundational expectations for SaMD usability. Among others, relevant documents include the FDA’s guidance documents on Applying Human Factors and Usability Engineering to Medical Devices and Software as a Medical Device (SAMD): Clinical Evaluation, IEC 62366-1:2015, and IMDRF’s guidance on SaMD: Clinical Evaluation. While these resources offer important baseline requirements, they often fall short of addressing the specific demands and evolving nature of software-centric products. In particular, some of the guidelines lack detailed and harmonized methodologies that reflect the variability of modern digital interfaces and the iterative nature of software development. As a result, human factors professionals must not only interpret existing standards but also extend and adapt them to ensure meaningful and comprehensive evaluation of SaMD safety and efficacy.
In practice, resource constraints often further complicate usability testing. For startups and small-to-mid-sized companies developing SaMD, allocating sufficient time, budget, and expertise for robust usability engineering can be a challenge—especially with competing alongside clinical validation, cybersecurity, and product development priorities. In these cases, usability efforts may be fragmented, deprioritized, or conducted under suboptimal conditions, increasing the likelihood of design flaws, unexpected user behaviors, or user dissatisfaction following launch.
Compounding these challenges is the rapid pace of technological change. Modern SaMD products commonly follow agile development cycles, with frequent bug fixes, feature releases, and major updates incorporating emerging technologies like machine learning and AI. This reality makes it clear that usability testing cannot be treated as a one-time event, but rather as a continuous, embedded process throughout the product lifecycle. Yet, conducting ongoing, rigorous usability testing while maintaining regulatory compliance across different software iterations introduces both logistical and procedural hurdles, particularly when changes affect user-facing features, workflows, and critical tasks. Traditional medical device validation often lacks the flexibility and responsiveness required to keep pace.
In summary, the usability evaluation of SaMD requires an adjusted mindset than that used for hardware medical devices or combination products. It demands agile, user-centered methodologies that account for various use environments, changing technology, and the complexity of software-based interactions, all while aligning with evolving regulatory expectations.
Overview of Presentation:
• Introduction to SaMD and regulatory landscape: We will begin by defining SaMD and reviewing the international regulatory standards that shape usability requirements, emphasizing the difference from hardware medical devices and combination products.
• Key challenges in SaMD usability testing: We will discuss four primary challenges: software variability, regulatory hurdles, resource constraints, and the rapid pace of technological change.
• Comparative analysis: Usability testing approaches for SaMD versus hardware medical devices, hybrid hardware and software medical devices, and combination products will be contrasted, underlining the need for scenario-based testing, built-in instructional mitigations, and continuous validation.
• Best practices for SaMD usability testing: Drawing on our experience, we will outline best practices, including:
o Implementing a code freeze prior to validation to ensure testing integrity
Before validation, place the SaMD under a final code freeze, where no code changes, feature updates, or fixes are allowed. This ensures study results reflect the actual software version under review, avoiding new usability issues or changes that could invalidate results or require retesting.
o Scenario-based testing with realistic data and multiple user pathways
Allow sufficient time for testing session dry runs, including both ideal and error-prone paths. Since software allows multiple routes to the same outcome, success criteria should focus on the goal and not the path, unless the path carries risk.
Use consistent, realistic test data (e.g. predefined patient values) to keep scenarios standardized yet authentic.
o Recruitment strategies to ensure diversity in tech literacy and platform/operational system familiarity
o Leveraging technical support during testing
Have technical support available during sessions to confirm whether issues are true software usability problems or test artifacts, and to resolve technical issues as needed without disrupting the study flow.
o Focusing on workflow-embedded mitigations rather than traditional labeling
For SaMD, prioritize evaluating workflow-integrated mitigations (e.g., prompts, text pop-ups, etc.) rather than relying on instructional labeling, as labeling is considered secondary unless users specifically reference it.
• Case study: We will present an example of our company’s approach to SaMD validation, detailing the process, challenges, and lessons learned. Key elements included enforcing a code freeze, designing consistent user prompts, ensuring robust technical support during testing, and identifying critical/in-scope tasks relative to routine platform usage.
• Knowledge gaps and future directions: We will discuss the need for clearer guidance on SaMD usability validation, as well as formative usability testing timing, including testing timing, methods, and limitations of traditional root cause analyses in evaluating software usability.
Importance of Message:
A nuanced understanding of SaMD usability testing is critical for patient safety, regulatory success, and user adoption. By adapting best practices and learning from real-world examples, human factors professionals can bridge the gap between innovation and safe, effective healthcare solutions.
This topic explores the evolving landscape for usability testing for software as a medical device (SaMD), focusing on unique challenges and real-world insights from the lens of existing standards and best practices.
Application:
As SaMD solutions become increasingly central to healthcare delivery—ranging from mobile diagnostic tools and patient-facing apps to complex clinical decision support systems—a framework for planning and executing usability testing tailored to the unique demands of SaMD is necessary. In this presentation, we will use existing standards and best practices to explore how to design usability testing for SaMD, bridge regulatory gaps, and compare SaMD to hardware medical devices and combination products.
Background:
The emergence of SaMD is transforming healthcare delivery, enabling innovative diagnostics, decision support, and patient engagement. Unlike traditional medical devices, SaMD relies on complex algorithms, dynamic user interfaces, artificial intelligence (AI), and integration with electronic health records (EHR) and other health IT systems. These features offer significant potential for improving patient care but also introduce unique usability and regulatory challenges.
A fundamental distinction lies in the variability of user interactions and use environments. SaMD may be used across different digital platforms—such as mobile phones, tablets, and desktop computers—and in diverse settings, ranging from hospitals and clinics to patients’ homes. This range of contexts, combined with varied levels of user digital literacy, makes it challenging to capture realistic use behaviors and anticipate all potential use scenarios. As a result, potential use problems may go undetected in conventional usability testing frameworks that were originally designed with static, hardware-based devices in mind.
Regulatory bodies such as the FDA and international organizations like the International Medical Device Regulators Forum (IMDRF) have established foundational expectations for SaMD usability. Among others, relevant documents include the FDA’s guidance documents on Applying Human Factors and Usability Engineering to Medical Devices and Software as a Medical Device (SAMD): Clinical Evaluation, IEC 62366-1:2015, and IMDRF’s guidance on SaMD: Clinical Evaluation. While these resources offer important baseline requirements, they often fall short of addressing the specific demands and evolving nature of software-centric products. In particular, some of the guidelines lack detailed and harmonized methodologies that reflect the variability of modern digital interfaces and the iterative nature of software development. As a result, human factors professionals must not only interpret existing standards but also extend and adapt them to ensure meaningful and comprehensive evaluation of SaMD safety and efficacy.
In practice, resource constraints often further complicate usability testing. For startups and small-to-mid-sized companies developing SaMD, allocating sufficient time, budget, and expertise for robust usability engineering can be a challenge—especially with competing alongside clinical validation, cybersecurity, and product development priorities. In these cases, usability efforts may be fragmented, deprioritized, or conducted under suboptimal conditions, increasing the likelihood of design flaws, unexpected user behaviors, or user dissatisfaction following launch.
Compounding these challenges is the rapid pace of technological change. Modern SaMD products commonly follow agile development cycles, with frequent bug fixes, feature releases, and major updates incorporating emerging technologies like machine learning and AI. This reality makes it clear that usability testing cannot be treated as a one-time event, but rather as a continuous, embedded process throughout the product lifecycle. Yet, conducting ongoing, rigorous usability testing while maintaining regulatory compliance across different software iterations introduces both logistical and procedural hurdles, particularly when changes affect user-facing features, workflows, and critical tasks. Traditional medical device validation often lacks the flexibility and responsiveness required to keep pace.
In summary, the usability evaluation of SaMD requires an adjusted mindset than that used for hardware medical devices or combination products. It demands agile, user-centered methodologies that account for various use environments, changing technology, and the complexity of software-based interactions, all while aligning with evolving regulatory expectations.
Overview of Presentation:
• Introduction to SaMD and regulatory landscape: We will begin by defining SaMD and reviewing the international regulatory standards that shape usability requirements, emphasizing the difference from hardware medical devices and combination products.
• Key challenges in SaMD usability testing: We will discuss four primary challenges: software variability, regulatory hurdles, resource constraints, and the rapid pace of technological change.
• Comparative analysis: Usability testing approaches for SaMD versus hardware medical devices, hybrid hardware and software medical devices, and combination products will be contrasted, underlining the need for scenario-based testing, built-in instructional mitigations, and continuous validation.
• Best practices for SaMD usability testing: Drawing on our experience, we will outline best practices, including:
o Implementing a code freeze prior to validation to ensure testing integrity
Before validation, place the SaMD under a final code freeze, where no code changes, feature updates, or fixes are allowed. This ensures study results reflect the actual software version under review, avoiding new usability issues or changes that could invalidate results or require retesting.
o Scenario-based testing with realistic data and multiple user pathways
Allow sufficient time for testing session dry runs, including both ideal and error-prone paths. Since software allows multiple routes to the same outcome, success criteria should focus on the goal and not the path, unless the path carries risk.
Use consistent, realistic test data (e.g. predefined patient values) to keep scenarios standardized yet authentic.
o Recruitment strategies to ensure diversity in tech literacy and platform/operational system familiarity
o Leveraging technical support during testing
Have technical support available during sessions to confirm whether issues are true software usability problems or test artifacts, and to resolve technical issues as needed without disrupting the study flow.
o Focusing on workflow-embedded mitigations rather than traditional labeling
For SaMD, prioritize evaluating workflow-integrated mitigations (e.g., prompts, text pop-ups, etc.) rather than relying on instructional labeling, as labeling is considered secondary unless users specifically reference it.
• Case study: We will present an example of our company’s approach to SaMD validation, detailing the process, challenges, and lessons learned. Key elements included enforcing a code freeze, designing consistent user prompts, ensuring robust technical support during testing, and identifying critical/in-scope tasks relative to routine platform usage.
• Knowledge gaps and future directions: We will discuss the need for clearer guidance on SaMD usability validation, as well as formative usability testing timing, including testing timing, methods, and limitations of traditional root cause analyses in evaluating software usability.
Importance of Message:
A nuanced understanding of SaMD usability testing is critical for patient safety, regulatory success, and user adoption. By adapting best practices and learning from real-world examples, human factors professionals can bridge the gap between innovation and safe, effective healthcare solutions.
Authors
Event Type
Oral Presentations
TimeMonday, March 231:30pm - 1:52pm EDT
LocationNassau
Digital Health