Presentation
Transparency and Explainability in Human Factors: A Systematic Review of Usability Assessment Practices for AI Medical Devices
SessionPoster Session 2
DescriptionThe integration of Artificial Intelligence (AI) into medical devices (AI as a Medical Device, AIaMD) represents a transformative, yet challenging, leap for healthcare. While AI promises enhanced diagnostic accuracy and patient outcomes, its "black-box" nature, particularly in complex models like Large Language Models (LLMs), introduces novel risks for end-users - clinicians, patients, and caregivers. The effective and safe use of any medical device is intrinsically linked to its usability, and with AIaMD, this extends beyond mere interface design to crucial concepts like Transparency, Explainability, and Trustworthiness.
Application and Background: Current regulatory and industry standards, including the US FDA's Human Factors and Usability Engineering Guidance, the EU Medical Device Regulation (MDR), and technical standards like IEC 62366-1 and ISO 9241-11 - were not originally designed to fully address the unique challenges of dynamic, probabilistic AI systems. This misalignment has created a critical, cross-jurisdictional regulatory and practical gap concerning the best practices for evaluating and assuring Human Factors and Usability in AIaMD. Unaddressed usability issues in this context can lead to serious risks, such as automation bias, miscalibration of user trust (over- or under-reliance), and ultimately, incorrect clinical decision-making.
Topic and Overview: To address this critical gap, a systematic review was carried out to comprehensively evaluate the current landscape of usability assessment practices for AI-enabled Medical Devices. The review aimed to establish: (1) how usability is being assessed in AIaMD research and development, (2) if there is an emerging consensus on best practices, and (3) the extent to which current efforts align with established industry guidelines.
The methodology adhered strictly to the PRISMA guidelines. A rigorous search was conducted across five major databases: PubMed, Science Direct, IEEE Xplore, ACM Digital Library, and Web Of Science, yielding an initial 2,462 records. After a thorough title and abstract screening, 90 records were sought for retrieval, leading to 85 studies being eligible for full-text screening. Ultimately, 53 studies were included in the final review. Inter-rater reliability was confirmed by two independent reviewers assessing 20% of the records, which resulted in a high Cohen's Kappa of k=0.91, signifying excellent agreement. Data extraction employed a custom-made questionnaire, followed by a thematic analysis to synthesize the qualitative findings.
Importance of Message and Take Away Points: The poster presentation will detail the systematic review process, present the key findings from the thematic analysis, and highlight the most prominent and emerging practices in AI Usability. The central take-away for the audience will be a clear understanding of the current state-of-the-art - identifying where researchers and developers are succeeding, where critical gaps in practice remain, and how the essential components of Transparency and Explainability are (or are not) being integrated into usability assessments. This work provides an evidence-based foundation to inform the immediate development of pragmatic, robust usability frameworks for AIaMD, bridging the gap between technological advancement and clinical safety.
Application and Background: Current regulatory and industry standards, including the US FDA's Human Factors and Usability Engineering Guidance, the EU Medical Device Regulation (MDR), and technical standards like IEC 62366-1 and ISO 9241-11 - were not originally designed to fully address the unique challenges of dynamic, probabilistic AI systems. This misalignment has created a critical, cross-jurisdictional regulatory and practical gap concerning the best practices for evaluating and assuring Human Factors and Usability in AIaMD. Unaddressed usability issues in this context can lead to serious risks, such as automation bias, miscalibration of user trust (over- or under-reliance), and ultimately, incorrect clinical decision-making.
Topic and Overview: To address this critical gap, a systematic review was carried out to comprehensively evaluate the current landscape of usability assessment practices for AI-enabled Medical Devices. The review aimed to establish: (1) how usability is being assessed in AIaMD research and development, (2) if there is an emerging consensus on best practices, and (3) the extent to which current efforts align with established industry guidelines.
The methodology adhered strictly to the PRISMA guidelines. A rigorous search was conducted across five major databases: PubMed, Science Direct, IEEE Xplore, ACM Digital Library, and Web Of Science, yielding an initial 2,462 records. After a thorough title and abstract screening, 90 records were sought for retrieval, leading to 85 studies being eligible for full-text screening. Ultimately, 53 studies were included in the final review. Inter-rater reliability was confirmed by two independent reviewers assessing 20% of the records, which resulted in a high Cohen's Kappa of k=0.91, signifying excellent agreement. Data extraction employed a custom-made questionnaire, followed by a thematic analysis to synthesize the qualitative findings.
Importance of Message and Take Away Points: The poster presentation will detail the systematic review process, present the key findings from the thematic analysis, and highlight the most prominent and emerging practices in AI Usability. The central take-away for the audience will be a clear understanding of the current state-of-the-art - identifying where researchers and developers are succeeding, where critical gaps in practice remain, and how the essential components of Transparency and Explainability are (or are not) being integrated into usability assessments. This work provides an evidence-based foundation to inform the immediate development of pragmatic, robust usability frameworks for AIaMD, bridging the gap between technological advancement and clinical safety.
Event Type
Poster Presentation
TimeTuesday, March 244:45pm - 6:15pm EDT
LocationRhinelander Gallery
Digital Health