Presentation
AI as Panacea or Pitfall: The Cognitive Costs of Clinical Decision Support
DescriptionTopic Summary:
Artificial Intelligence (AI) is becoming increasingly touted as a panacea for addressing system inefficiencies, cost burdens, and staffing shortages in healthcare. From diagnostic support to administrative streamlining and even patient monitoring, AI is being marketed as a grand solution to the systemic pressures faced in healthcare, but this stance overlooks an equally pressing reality: large-scale integration of AI, specifically AI intended to provide decision-making support, introduces cognitive and behavioral risks in users.
Working with AI introduces vulnerabilities into healthcare systems, such as hallucinations (false or misleading outputs) (Kim et al., 2025; Geroimenko, 2025), automation bias (Abdelwanis et al., 2024; Khera, Simon, & Ross, 2023), improper trust calibration in which users are either undertrusting or overtrusting the AI (Rosenbacke et al., 2024), and problematic cognitive outsourcing (Ahlstrom‐Vij, 2016; Tao et al., 2024). Based on these vulnerabilities, the overconfidence in the technology’s abilities could expand users' risk perception, encouraging clinicians to take actions they might normally avoid under the assumption that the AI will compensate for unfavorable conditions (Parasuraman & Riley, 1997).
As such, AI integration carries a dual nature: it has the potential to be a powerful tool, but also a source of skill degradation and overreliance if uncritically adopted (Jabbour et al., 2023). This gradual skill loss creates a potentially risky environment, leaving clinicians less prepared to perform effectively when AI is unavailable or fails in high-stakes scenarios (Budzyń et al., 2025). One notable example of this skill degradation is the study done by Budzyń and colleagues (2025), in which AI was introduced to improve adenoma detection rates during colonoscopies. They found that continuous use and exposure to AI reduced detection rates of endoscopists when they did not have access to the AI tool by approximately 6%.
Overall, healthcare cannot afford to adopt AI as a panacea without recognizing and addressing its pitfalls. Through the critical exploration of how users interact with AI, this submission aims to reframe the conversation of AI adoption. AI’s value will not be determined by its raw capabilities, but by how successfully it is integrated with an understanding of Human Factors (HF). Simulation-based training provides a vital pathway for this integration, giving clinicians safe opportunities to interact with AI, recognize errors, and prepare for AI-related failures in real-world scenarios. Simulated practice with AI tools, such as training users on when to accept or reject AI recommendations, can facilitate the preservation of situation awareness, ensure proper trust calibration, and help to build resilience against AI errors.
Background:
AI is rapidly expanding into healthcare, with applications spanning diagnosis, screening purposes, decision-making, and patient education (Morone et al., 2025). These AI systems are designed to improve efficiency and accuracy in care delivery, and have been able to outperform traditional methods regarding clinical decision support and detection tasks (Li et al., 2019; Schwartz et al., 2021). While these applications highlight AI’s potential to enhance accuracy and efficiency across domains, they also expose a critical trade-off: the benefits of these AI tools will introduce risks. The overall goal of this submission is to demonstrate how HF principles and simulation-based training can be used to help balance these outcomes, safeguarding against misuse while preserving the advantages of AI.
The potential vulnerabilities and repercussions that could result from uncritical acceptance of AI are a serious risk to patient safety and clinician performance. Users of AI are often subject to automation bias, in this case, the tendency to accept AI-generated outputs without critical evaluation of the suggestion(s) made (Abdelwanis et al., 2024). This risk, while problematic in and of itself, is magnified by two things: AI hallucinations and AI bias. The phenomenon of AI hallucinations, in which AI outputs involve fabricated or misleading information due to errors in information processing, data retrieval, or system overload, is often hard to detect and can influence user decision-making (Rosenbacke et al., 2024). Systemic biases embedded in AI training data may also influence user decision-making capabilities by unintentionally propagating disparities in care by suggesting biased information regarding those of different racial, socioeconomic, and gendered demographics (Cross et al., 2024; Jumreornvong et al., 2025).
The overall perceived accuracy and efficiency of AI, and the innate automation bias users experience, create a cycle of overreliance and overconfidence in the technology, resulting in improper trust calibration, where users assume the system is more accurate or capable than it truly is (Dratsch et al., 2023). Overtrust often leads to complacency, where users stop double-checking or questioning recommendations, allowing errors to go unnoticed, which can directly affect patient outcomes. Cognitive outsourcing, the delegation of mental tasks to an external system, compounds these challenges (Ahlstrom‐Vij, 2016; Tao et al., 2024). As clinicians increasingly rely on AI, they may offload cognitive effort to the system and disengage from active information processing (Jabbour et al., 2023).
These vulnerabilities may not seem like pressing issues. Users can ultimately save time and mental effort on tasks they can offload to an AI system, and have more time to dedicate to other facets of their job. However, when users are unaware of how to identify AI hallucinations and biased recommendations, and are ignorant of the ramifications of overly engaging in cognitive outsourcing and improper trust calibration, direct decrements to their performance will be seen. Overall, these dynamics illustrate that AI, while offering potential to revolutionize the healthcare domain, introduces serious vulnerabilities that can erode decision quality and safety in healthcare. Addressing these requires proactive strategies grounded in HF principles and implemented through regular simulation-based training. The following application section details how these approaches can be used to strengthen resilience while ensuring that the benefits of AI are preserved.
Application:
HF principles provide the framework to structure training and create operational guidelines based on human capabilities and limitations. Healthcare can draw from HF concepts to ensure that clinicians are interacting with AI in ways that increase accuracy and efficiency, while preserving their skills and decision-making (Ruskin et al., 2020). Simulation offers a safe, controlled environment where clinicians can practice responding to AI errors without jeopardizing patient safety (Komasawa & Yokohira, 2023). Simulation emphasizes experimental training, rather than abstract instruction, allowing users to experience how factors such as automation bias, hallucinations, and skill degradation can emerge in realistic scenarios when working with new AI tools (Elendu et al., 2024).
Automation bias can be addressed through scenarios where an AI system provides advice that contradicts best practices, requiring clinicians to decide whether to accept or reject the recommendation. Similarly, hallucinations and biased outputs can be introduced through simulated imaging interpretations or treatment suggestions, training clinicians to recognize inconsistencies and determine when outputs are unreliable. Overtrust and complacency can be mitigated by exposing trainees to incorrect but highly confident outputs, reinforcing the need for trust calibration and independent verification. To counter cognitive outsourcing, clinicians can be required to complete tasks without AI support, while alternating between AI-assisted and manual conditions helps prevent skill degradation and maintain situation awareness.
Together, these training suggestions demonstrate how simulation-based training can utilize HF principles to preserve the benefits of AI tools while actively preventing cognitive and behavioral consequences. Directly confronting the challenges of AI integration equips clinicians to recognize errors, preserve critical skills, and maintain responsibility in decision-making. More broadly, simulation provides a pathway for researchers, trainers, and healthcare leaders to integrate AI responsibly, ensuring that efficiency and accuracy gains do not come at the expense of patient safety.
Reference List Available Upon Request.
Artificial Intelligence (AI) is becoming increasingly touted as a panacea for addressing system inefficiencies, cost burdens, and staffing shortages in healthcare. From diagnostic support to administrative streamlining and even patient monitoring, AI is being marketed as a grand solution to the systemic pressures faced in healthcare, but this stance overlooks an equally pressing reality: large-scale integration of AI, specifically AI intended to provide decision-making support, introduces cognitive and behavioral risks in users.
Working with AI introduces vulnerabilities into healthcare systems, such as hallucinations (false or misleading outputs) (Kim et al., 2025; Geroimenko, 2025), automation bias (Abdelwanis et al., 2024; Khera, Simon, & Ross, 2023), improper trust calibration in which users are either undertrusting or overtrusting the AI (Rosenbacke et al., 2024), and problematic cognitive outsourcing (Ahlstrom‐Vij, 2016; Tao et al., 2024). Based on these vulnerabilities, the overconfidence in the technology’s abilities could expand users' risk perception, encouraging clinicians to take actions they might normally avoid under the assumption that the AI will compensate for unfavorable conditions (Parasuraman & Riley, 1997).
As such, AI integration carries a dual nature: it has the potential to be a powerful tool, but also a source of skill degradation and overreliance if uncritically adopted (Jabbour et al., 2023). This gradual skill loss creates a potentially risky environment, leaving clinicians less prepared to perform effectively when AI is unavailable or fails in high-stakes scenarios (Budzyń et al., 2025). One notable example of this skill degradation is the study done by Budzyń and colleagues (2025), in which AI was introduced to improve adenoma detection rates during colonoscopies. They found that continuous use and exposure to AI reduced detection rates of endoscopists when they did not have access to the AI tool by approximately 6%.
Overall, healthcare cannot afford to adopt AI as a panacea without recognizing and addressing its pitfalls. Through the critical exploration of how users interact with AI, this submission aims to reframe the conversation of AI adoption. AI’s value will not be determined by its raw capabilities, but by how successfully it is integrated with an understanding of Human Factors (HF). Simulation-based training provides a vital pathway for this integration, giving clinicians safe opportunities to interact with AI, recognize errors, and prepare for AI-related failures in real-world scenarios. Simulated practice with AI tools, such as training users on when to accept or reject AI recommendations, can facilitate the preservation of situation awareness, ensure proper trust calibration, and help to build resilience against AI errors.
Background:
AI is rapidly expanding into healthcare, with applications spanning diagnosis, screening purposes, decision-making, and patient education (Morone et al., 2025). These AI systems are designed to improve efficiency and accuracy in care delivery, and have been able to outperform traditional methods regarding clinical decision support and detection tasks (Li et al., 2019; Schwartz et al., 2021). While these applications highlight AI’s potential to enhance accuracy and efficiency across domains, they also expose a critical trade-off: the benefits of these AI tools will introduce risks. The overall goal of this submission is to demonstrate how HF principles and simulation-based training can be used to help balance these outcomes, safeguarding against misuse while preserving the advantages of AI.
The potential vulnerabilities and repercussions that could result from uncritical acceptance of AI are a serious risk to patient safety and clinician performance. Users of AI are often subject to automation bias, in this case, the tendency to accept AI-generated outputs without critical evaluation of the suggestion(s) made (Abdelwanis et al., 2024). This risk, while problematic in and of itself, is magnified by two things: AI hallucinations and AI bias. The phenomenon of AI hallucinations, in which AI outputs involve fabricated or misleading information due to errors in information processing, data retrieval, or system overload, is often hard to detect and can influence user decision-making (Rosenbacke et al., 2024). Systemic biases embedded in AI training data may also influence user decision-making capabilities by unintentionally propagating disparities in care by suggesting biased information regarding those of different racial, socioeconomic, and gendered demographics (Cross et al., 2024; Jumreornvong et al., 2025).
The overall perceived accuracy and efficiency of AI, and the innate automation bias users experience, create a cycle of overreliance and overconfidence in the technology, resulting in improper trust calibration, where users assume the system is more accurate or capable than it truly is (Dratsch et al., 2023). Overtrust often leads to complacency, where users stop double-checking or questioning recommendations, allowing errors to go unnoticed, which can directly affect patient outcomes. Cognitive outsourcing, the delegation of mental tasks to an external system, compounds these challenges (Ahlstrom‐Vij, 2016; Tao et al., 2024). As clinicians increasingly rely on AI, they may offload cognitive effort to the system and disengage from active information processing (Jabbour et al., 2023).
These vulnerabilities may not seem like pressing issues. Users can ultimately save time and mental effort on tasks they can offload to an AI system, and have more time to dedicate to other facets of their job. However, when users are unaware of how to identify AI hallucinations and biased recommendations, and are ignorant of the ramifications of overly engaging in cognitive outsourcing and improper trust calibration, direct decrements to their performance will be seen. Overall, these dynamics illustrate that AI, while offering potential to revolutionize the healthcare domain, introduces serious vulnerabilities that can erode decision quality and safety in healthcare. Addressing these requires proactive strategies grounded in HF principles and implemented through regular simulation-based training. The following application section details how these approaches can be used to strengthen resilience while ensuring that the benefits of AI are preserved.
Application:
HF principles provide the framework to structure training and create operational guidelines based on human capabilities and limitations. Healthcare can draw from HF concepts to ensure that clinicians are interacting with AI in ways that increase accuracy and efficiency, while preserving their skills and decision-making (Ruskin et al., 2020). Simulation offers a safe, controlled environment where clinicians can practice responding to AI errors without jeopardizing patient safety (Komasawa & Yokohira, 2023). Simulation emphasizes experimental training, rather than abstract instruction, allowing users to experience how factors such as automation bias, hallucinations, and skill degradation can emerge in realistic scenarios when working with new AI tools (Elendu et al., 2024).
Automation bias can be addressed through scenarios where an AI system provides advice that contradicts best practices, requiring clinicians to decide whether to accept or reject the recommendation. Similarly, hallucinations and biased outputs can be introduced through simulated imaging interpretations or treatment suggestions, training clinicians to recognize inconsistencies and determine when outputs are unreliable. Overtrust and complacency can be mitigated by exposing trainees to incorrect but highly confident outputs, reinforcing the need for trust calibration and independent verification. To counter cognitive outsourcing, clinicians can be required to complete tasks without AI support, while alternating between AI-assisted and manual conditions helps prevent skill degradation and maintain situation awareness.
Together, these training suggestions demonstrate how simulation-based training can utilize HF principles to preserve the benefits of AI tools while actively preventing cognitive and behavioral consequences. Directly confronting the challenges of AI integration equips clinicians to recognize errors, preserve critical skills, and maintain responsibility in decision-making. More broadly, simulation provides a pathway for researchers, trainers, and healthcare leaders to integrate AI responsibly, ensuring that efficiency and accuracy gains do not come at the expense of patient safety.
Reference List Available Upon Request.
Authors
Event Type
Oral Presentations
TimeMonday, March 231:30pm - 2:00pm EDT
LocationMorgan
Simulation and Education