Presentation
Human Factors and Usability Practices for AI-Enabled Medical Devices: A Systematic Review of Transparency, Explainability, and Regulatory Alignment
SessionDH7: Beyond the Algorithm: Building Trust, Transparency, and Safety into Smart Medical Device Design
DescriptionThe increasing reliance on Artificial Intelligence (AI) in clinical practice, particularly through AI-Enabled Medical Devices (AIaMD), presents a foundational challenge to traditional patient safety and usability engineering paradigms. The probabilistic and often opaque nature of AI models, particularly as they incorporate advanced techniques like Large Language Models (LLMs), introduces unique risks. These include the potential for automation bias, difficulty in diagnosing model failure, and the critical issue of user trust calibration (i.e., over-reliance or unwarranted rejection of AI advice). Effective mitigation of these risks hinges on ensuring the AIaMD is not only clinically accurate but also safe and effective for the end-user - a core goal of Human Factors (HF) and Usability Engineering (UE).
Application and Background: Current regulatory frameworks, including the US FDA's Human Factors guidance, the EU Medical Device Regulation (MDR), and industry standards like ISO 9241-11 and IEC 62366-1 predate the widespread complexity of AIaMD. Consequently, a significant regulatory and practical gap exists regarding how to rigorously evaluate the usability of AI systems, especially those requiring high degrees of Transparency and Explainability for safe use. This systematic review was designed to bridge this gap by synthesizing current global practices in AIaMD usability assessment.
Overview of Presentation: This presentation will deliver the synthesized findings of a rigorous systematic review, conducted following PRISMA guidelines, which analyzed 58 studies across five major scientific and engineering databases. The review sought to answer: 1) How are usability and HF being evaluated in AIaMD development? 2) Is there consensus on best practices regarding the integration of Transparency and Explainability into the usability lifecycle? 3) Do current practices meet or fall short of the expectations set by major regulatory bodies? The thematic analysis revealed a fragmented landscape where Transparency and Explainability components are frequently discussed but rarely rigorously validated through comprehensive usability testing with intended users. We will present the identified common practices, methodological deficiencies, and the implications of these gaps for real-world patient safety and regulatory compliance.
Importance of Message and Take Away Points: The central take-away for the audience is that the current approach to AIaMD usability assessment is often insufficient, creating a latent risk environment. We will provide concrete, evidence-based recommendations for developers and regulators, emphasizing that Transparency and Explainability must be treated as critical usability requirements, not just technical features. The presentation concludes with a call to action: to inform the rapid development of harmonized, AI-specific usability frameworks that integrate Transparency and Explainability into the entire product lifecycle, ensuring regulatory alignment and clinical safety across jurisdictions like the US (FDA) and EU.
Application and Background: Current regulatory frameworks, including the US FDA's Human Factors guidance, the EU Medical Device Regulation (MDR), and industry standards like ISO 9241-11 and IEC 62366-1 predate the widespread complexity of AIaMD. Consequently, a significant regulatory and practical gap exists regarding how to rigorously evaluate the usability of AI systems, especially those requiring high degrees of Transparency and Explainability for safe use. This systematic review was designed to bridge this gap by synthesizing current global practices in AIaMD usability assessment.
Overview of Presentation: This presentation will deliver the synthesized findings of a rigorous systematic review, conducted following PRISMA guidelines, which analyzed 58 studies across five major scientific and engineering databases. The review sought to answer: 1) How are usability and HF being evaluated in AIaMD development? 2) Is there consensus on best practices regarding the integration of Transparency and Explainability into the usability lifecycle? 3) Do current practices meet or fall short of the expectations set by major regulatory bodies? The thematic analysis revealed a fragmented landscape where Transparency and Explainability components are frequently discussed but rarely rigorously validated through comprehensive usability testing with intended users. We will present the identified common practices, methodological deficiencies, and the implications of these gaps for real-world patient safety and regulatory compliance.
Importance of Message and Take Away Points: The central take-away for the audience is that the current approach to AIaMD usability assessment is often insufficient, creating a latent risk environment. We will provide concrete, evidence-based recommendations for developers and regulators, emphasizing that Transparency and Explainability must be treated as critical usability requirements, not just technical features. The presentation concludes with a call to action: to inform the rapid development of harmonized, AI-specific usability frameworks that integrate Transparency and Explainability into the entire product lifecycle, ensuring regulatory alignment and clinical safety across jurisdictions like the US (FDA) and EU.
Event Type
Oral Presentations
TimeTuesday, March 243:30pm - 3:50pm EDT
LocationNassau
Digital Health