Presentation
Beyond Algorithms: Human-Centered Explainability for Clinicians, Patients, and Caregivers
SessionPoster Session 1
DescriptionArtificial intelligence (AI) is increasingly positioned as a transformative force in healthcare, where applications range from diagnostic imaging to personalized treatment recommendations. Yet despite its promise, adoption remains slow. Patients and caregivers often experience unresolved mistrust and safety concerns regarding AI use, while also expressing a desire for connections to trusted local supports and reputable community resources to guide its use (1).
Traditional explainable AI (XAI) approaches have centered on algorithmic techniques such as feature attribution, saliency maps, or decision trees (2,3). While these may provide valuable insights for developers, they often fail to provide benefits to the end users (2). Patients and caregivers are left dependent on clinicians to understand system-derived algorithmic explanations (4), who themselves may lack sufficient clarity and justification to communicate AI reasoning effectively (5). This cycle limits the understanding of AI in patient-centered care. Because system-generated explanations often fall short in real-world care, there is a need for more human-centered approaches that emphasize interaction and learning rather than technical detail alone.
To address these challenges, researchers have introduced non-algorithmic approaches to explainability (6). These methods do not rely solely on computational outputs but instead provide human-centered strategies for helping users make sense of AI. Two approaches could be particularly relevant for healthcare: Collaborative Explainable AI (CXAI) (7) and Cognitive Tutorials (8). These have been applied in other safety-critical fields, such as the Intelligent Transportation domain (9). SEIPS 2.0/3.0 (10,11) provides a strong conceptual foundation for adapting them to healthcare, as it emphasizes three connected processes: the work clinicians do to explain and act on results (professional work process), the work patients and caregivers do to understand and manage their care (patient work process), and the shared work they do together to reach safe decisions (collaborative work process). SEIPS 3.0 adds journey context and links mistrust points. In this report, we use short vignettes to show how these two non-algorithmic approaches can be applied in healthcare. We put forward that these approaches can (a) give patients and caregivers user-centric explanations, along with a clearer sense of how AI works, and (b) provide clinicians the confidence to interpret AI outputs and explain them more effectively to patients and caregivers if required. We will also provide a simple design guide for Human Factors (HF) researchers that tackles the barriers of explainability mentioned earlier.
CXAI acts as a form of community-building explainability. Within the framework of SEIPS 2.0, it functions as a collaborative process that moves beyond one-way communication, which allows clinicians, patients, and caregivers to interpret AI results together, ask questions, and connect explanations to trusted local supports or other explanations in the domain. By doing so, CXAI not only strengthens collaboration and builds trust but also helps create a community of support that extends beyond the clinical encounter.
CXAI vignette: When an AI tool flags a possible problem on a chest X-ray, the radiologist in a traditional workflow reviews it and gives the patient a brief explanation, with little chance for questions. In a CXAI workflow, the radiologist, treating clinician, patient, and caregiver review the result together. Junior clinicians can use the discussion as a learning moment, while senior clinicians can compare their own judgment with the AI. To make the explanation clearer, the clinician may also show de-identified results from other patients, giving families a chance to see how the AI is conducting its analysis. Patients and caregivers are encouraged to ask questions, and clinicians tailor their explanations to the asker’s level of understanding. This makes the process genuinely user-centric rather than “one-for-all.” The conversation ends with a shared plan that addresses next steps, strategies for managing uncertainty, and links to trusted resources. Beyond the visit, families can share how they understood or questioned AI outputs in similar situations. For instance, one family might describe how a tutorial/conversation helped them ask better questions at discharge, which another family could then learn from (12).
However, collaboration alone is not enough. Both clinicians and patients/caregivers need structured learning to build accurate mental models of AI, which is where Cognitive Tutorials play a role. While CXAI emphasizes dialogue, Cognitive Tutorials focus on structured learning. They explain how systems work in plain language and through real-life examples. That shows not only when the AI performs well but also when it falls short. These tutorials help clear up common misunderstandings and set realistic expectations about what the technology can and cannot do. From the perspective of SEIPS 2.0, Cognitive Tutorials support care processes of both clinicians and families: they give clinicians greater confidence in interpreting results and provide patients and caregivers with the knowledge and resources needed to make safe, informed choices.
Cognitive tutorials Vignette: A diabetes monitoring app could include a Cognitive Tutorial for clinicians, with simple case examples that show both correct predictions and common mistakes. This helps clinicians know when they can rely on the system and when to double-check. Patients could see a version of the tutorial in clear language with pictures, such as: “This tool can help track patterns but cannot replace an emergency check.” CXAI then adds the next step by bringing clinicians, patients, and caregivers together to review the results, ask questions, and decide on a shared plan.
Design Guide: To adapt non-algorithmic explainability methods for healthcare, we propose that HF researchers begin by mapping the SEIPS 2.0 work system elements (people, tasks, tools/technologies, organization, environment) to the points in care where mistrust and safety concerns can emerge. This mapping could clarify necessary elements in the workflow, such as who the key actors are (clinicians at different experience levels, patients, and caregivers), what tasks they must perform (interpreting AI outputs, making care decisions, seeking support), and what tools are available (AI dashboards, patient portals, tutorial modules). By identifying these elements, it becomes possible to pinpoint where breakdowns in explanation and communication are most likely to occur and guide the design for targeted interventions.
Building on this mapping, the next step is to develop interventions that fit naturally into the workflow. For CXAI, this means creating structured prompts and interfaces that foster dialogue (7). For example, shared dashboards or portal add-ons where patients, caregivers, and clinicians can view AI outputs together and ask questions. For Cognitive Tutorials, the goal is to design layered learning modules that illustrate both the strengths and the limitations of AI systems. These tutorials should be offered in multiple formats, such as case-based examples, infographics, and videos, so that both clinicians and families can engage with the material in ways that match their learning preferences (11).
To apply SEIPS 2.0 further, align each intervention with its core processes. Place CXAI where teamwork is needed. Use Cognitive Tutorials to strengthen what clinicians do in their own practice and provide tutorials that help patients and families manage their part of care. Matching the interventions to these processes makes them easier to use and more effective in real care settings.
Evaluation should be tied to the three SEIPS 2.0 processes. For the professional process, assess whether clinicians understand the explanation, gain confidence in interpreting AI outputs, and communicate them clearly. For the patient process, measure patient and caregiver comprehension/satisfaction, their sense of safety, and whether they feel more confident in decisions supported by AI. For the collaborative process, evaluate levels of trust and the quality of shared discussions. Feedback from all groups should be collected and used in iterative cycles to refine both the tutorials and the CXAI workflows. This ensures that the interventions remain practical, relevant, and sustainable in everyday care.
Traditional explainable AI (XAI) approaches have centered on algorithmic techniques such as feature attribution, saliency maps, or decision trees (2,3). While these may provide valuable insights for developers, they often fail to provide benefits to the end users (2). Patients and caregivers are left dependent on clinicians to understand system-derived algorithmic explanations (4), who themselves may lack sufficient clarity and justification to communicate AI reasoning effectively (5). This cycle limits the understanding of AI in patient-centered care. Because system-generated explanations often fall short in real-world care, there is a need for more human-centered approaches that emphasize interaction and learning rather than technical detail alone.
To address these challenges, researchers have introduced non-algorithmic approaches to explainability (6). These methods do not rely solely on computational outputs but instead provide human-centered strategies for helping users make sense of AI. Two approaches could be particularly relevant for healthcare: Collaborative Explainable AI (CXAI) (7) and Cognitive Tutorials (8). These have been applied in other safety-critical fields, such as the Intelligent Transportation domain (9). SEIPS 2.0/3.0 (10,11) provides a strong conceptual foundation for adapting them to healthcare, as it emphasizes three connected processes: the work clinicians do to explain and act on results (professional work process), the work patients and caregivers do to understand and manage their care (patient work process), and the shared work they do together to reach safe decisions (collaborative work process). SEIPS 3.0 adds journey context and links mistrust points. In this report, we use short vignettes to show how these two non-algorithmic approaches can be applied in healthcare. We put forward that these approaches can (a) give patients and caregivers user-centric explanations, along with a clearer sense of how AI works, and (b) provide clinicians the confidence to interpret AI outputs and explain them more effectively to patients and caregivers if required. We will also provide a simple design guide for Human Factors (HF) researchers that tackles the barriers of explainability mentioned earlier.
CXAI acts as a form of community-building explainability. Within the framework of SEIPS 2.0, it functions as a collaborative process that moves beyond one-way communication, which allows clinicians, patients, and caregivers to interpret AI results together, ask questions, and connect explanations to trusted local supports or other explanations in the domain. By doing so, CXAI not only strengthens collaboration and builds trust but also helps create a community of support that extends beyond the clinical encounter.
CXAI vignette: When an AI tool flags a possible problem on a chest X-ray, the radiologist in a traditional workflow reviews it and gives the patient a brief explanation, with little chance for questions. In a CXAI workflow, the radiologist, treating clinician, patient, and caregiver review the result together. Junior clinicians can use the discussion as a learning moment, while senior clinicians can compare their own judgment with the AI. To make the explanation clearer, the clinician may also show de-identified results from other patients, giving families a chance to see how the AI is conducting its analysis. Patients and caregivers are encouraged to ask questions, and clinicians tailor their explanations to the asker’s level of understanding. This makes the process genuinely user-centric rather than “one-for-all.” The conversation ends with a shared plan that addresses next steps, strategies for managing uncertainty, and links to trusted resources. Beyond the visit, families can share how they understood or questioned AI outputs in similar situations. For instance, one family might describe how a tutorial/conversation helped them ask better questions at discharge, which another family could then learn from (12).
However, collaboration alone is not enough. Both clinicians and patients/caregivers need structured learning to build accurate mental models of AI, which is where Cognitive Tutorials play a role. While CXAI emphasizes dialogue, Cognitive Tutorials focus on structured learning. They explain how systems work in plain language and through real-life examples. That shows not only when the AI performs well but also when it falls short. These tutorials help clear up common misunderstandings and set realistic expectations about what the technology can and cannot do. From the perspective of SEIPS 2.0, Cognitive Tutorials support care processes of both clinicians and families: they give clinicians greater confidence in interpreting results and provide patients and caregivers with the knowledge and resources needed to make safe, informed choices.
Cognitive tutorials Vignette: A diabetes monitoring app could include a Cognitive Tutorial for clinicians, with simple case examples that show both correct predictions and common mistakes. This helps clinicians know when they can rely on the system and when to double-check. Patients could see a version of the tutorial in clear language with pictures, such as: “This tool can help track patterns but cannot replace an emergency check.” CXAI then adds the next step by bringing clinicians, patients, and caregivers together to review the results, ask questions, and decide on a shared plan.
Design Guide: To adapt non-algorithmic explainability methods for healthcare, we propose that HF researchers begin by mapping the SEIPS 2.0 work system elements (people, tasks, tools/technologies, organization, environment) to the points in care where mistrust and safety concerns can emerge. This mapping could clarify necessary elements in the workflow, such as who the key actors are (clinicians at different experience levels, patients, and caregivers), what tasks they must perform (interpreting AI outputs, making care decisions, seeking support), and what tools are available (AI dashboards, patient portals, tutorial modules). By identifying these elements, it becomes possible to pinpoint where breakdowns in explanation and communication are most likely to occur and guide the design for targeted interventions.
Building on this mapping, the next step is to develop interventions that fit naturally into the workflow. For CXAI, this means creating structured prompts and interfaces that foster dialogue (7). For example, shared dashboards or portal add-ons where patients, caregivers, and clinicians can view AI outputs together and ask questions. For Cognitive Tutorials, the goal is to design layered learning modules that illustrate both the strengths and the limitations of AI systems. These tutorials should be offered in multiple formats, such as case-based examples, infographics, and videos, so that both clinicians and families can engage with the material in ways that match their learning preferences (11).
To apply SEIPS 2.0 further, align each intervention with its core processes. Place CXAI where teamwork is needed. Use Cognitive Tutorials to strengthen what clinicians do in their own practice and provide tutorials that help patients and families manage their part of care. Matching the interventions to these processes makes them easier to use and more effective in real care settings.
Evaluation should be tied to the three SEIPS 2.0 processes. For the professional process, assess whether clinicians understand the explanation, gain confidence in interpreting AI outputs, and communicate them clearly. For the patient process, measure patient and caregiver comprehension/satisfaction, their sense of safety, and whether they feel more confident in decisions supported by AI. For the collaborative process, evaluate levels of trust and the quality of shared discussions. Feedback from all groups should be collected and used in iterative cycles to refine both the tutorials and the CXAI workflows. This ensures that the interventions remain practical, relevant, and sustainable in everyday care.
Event Type
Poster Presentation
TimeMonday, March 234:45pm - 6:15pm EDT
LocationRhinelander Gallery
Patient Safety Research and Initiatives