Presentation
Measuring and Enhancing Shared Mental Models in Nursing Teams Using Multimodal Data
SessionPoster Session 1
DescriptionEffective teamwork is essential to ensuring patient safety and care quality in today’s dynamic healthcare environments. Among the many factors influencing teamwork, the development and maintenance of Shared Mental Models (SMMs) stand out as a critical determinant of success. An SMM represents the common cognitive framework that allows team members to remain aligned in their understanding of tasks, roles, tools, and goals. Well-aligned SMMs enable team members to anticipate each other’s needs, coordinate seamlessly, and adapt rapidly to evolving patient conditions. Conversely, poorly aligned SMMs contribute to communication breakdowns, duplication or omission of critical care steps, and, in the most severe cases, preventable harm. In fact, over 70% of sentinel events in healthcare have been associated with breakdowns in teamwork and SMMs.
Despite their importance, SMMs remain difficult to measure and support in practice. Current strategies rely heavily on subjective and retrospective approaches such as self-report surveys, expert observation, and post-event debriefing. While these methods provide some value, they are resource-intensive, prone to bias, and lack the temporal resolution required to capture the dynamic emergence and breakdown of SMMs during real-time interactions. Consequently, they offer limited potential to guide adaptive interventions that could strengthen team alignment in the moment. This limitation underscores a pressing need for innovative, scalable methods that can continuously monitor and support SMMs in clinical practice.
Our project addresses this need by proposing a machine learning–based framework for the dynamic measurement and real-time intervention of SMMs in nursing teams. Using multimodal data streams—including eye-tracking, EEG, and verbal communication—collected during immersive virtual reality (VR) nursing simulations, we aim to identify the behavioral and physiological markers that distinguish well-aligned from poorly aligned SMMs. UbiSim, an advanced VR simulation platform, provides an ideal environment for this study. It supports realistic, multi-user clinical scenarios, such as asthma exacerbations or heart failure, while allowing the controlled introduction of disruptive events (e.g., uncooperative family behavior or sudden patient deterioration). These conditions create rich opportunities to observe how SMMs are established, fail, and recover under stress without compromising patient safety.
Thirty nursing students will be organized into 15 two-person teams. Each team will complete three simulation scenarios in counterbalanced order, with each scenario embedding unexpected events designed to stress test team cognition. During these sessions, participants will wear Tobii Pro eye-tracking glasses and G.Nautilus EEG sensors while interacting within VR. Communication exchanges, gaze patterns, and neural activity will be synchronized and recorded alongside video observations. Immediately after each scenario, participants will complete structured self-reports to provide subjective measures of their shared understanding. Expert raters will annotate video data using established criteria such as information coherence, physical coherence, authority-responsibility alignment, and cognitive coherence, forming a robust ground truth for training models.
The resulting dataset will serve as the foundation for developing machine learning models capable of detecting SMM quality in real time. These models will explore convergent behavioral patterns such as synchronized gaze, balanced communication, and shared task execution. Once validated, the models will be used to implement an AI-driven intervention system that provides adaptive support when poor SMMs are detected. For example, if inefficient communication patterns emerge, the system could issue context-sensitive prompts to encourage clarification or closed-loop communication. This framework, conceptualized as a “Human Digital Cognitive Twin,” computationally mirrors the evolving cognitive state of the team and delivers targeted, timely interventions.
Beyond methodological innovation, this work has significant implications for healthcare training and patient safety. It extends SMM measurement beyond static, post-event evaluations to continuous, real-time monitoring, advancing the state of the art in team cognition research. Furthermore, by demonstrating the feasibility of AI-driven interventions, it provides a model for how intelligent systems can augment human teamwork in high-risk domains. Nursing students stand to benefit from immediate, adaptive feedback that strengthens their coordination skills, while educators gain a scalable and objective tool for training evaluation. More broadly, the proposed framework could be applied in other high-stakes environments such as emergency medicine, aviation, and nuclear operations, where effective teamwork is equally vital.
In conclusion, this project contributes three key advances. First, it introduces a multimodal, machine learning–based approach for objectively and dynamically measuring shared mental models in nursing teams. Second, it pioneers the integration of real-time AI-driven interventions to strengthen teamwork when misalignment is detected. Third, it lays the groundwork for next-generation training systems that enhance safety, efficiency, and resilience in healthcare and beyond. By bridging cognitive theory, sensor-based analytics, and intelligent support, this work charts a path toward more reliable and adaptive teamwork in complex, high-pressure environments.
Despite their importance, SMMs remain difficult to measure and support in practice. Current strategies rely heavily on subjective and retrospective approaches such as self-report surveys, expert observation, and post-event debriefing. While these methods provide some value, they are resource-intensive, prone to bias, and lack the temporal resolution required to capture the dynamic emergence and breakdown of SMMs during real-time interactions. Consequently, they offer limited potential to guide adaptive interventions that could strengthen team alignment in the moment. This limitation underscores a pressing need for innovative, scalable methods that can continuously monitor and support SMMs in clinical practice.
Our project addresses this need by proposing a machine learning–based framework for the dynamic measurement and real-time intervention of SMMs in nursing teams. Using multimodal data streams—including eye-tracking, EEG, and verbal communication—collected during immersive virtual reality (VR) nursing simulations, we aim to identify the behavioral and physiological markers that distinguish well-aligned from poorly aligned SMMs. UbiSim, an advanced VR simulation platform, provides an ideal environment for this study. It supports realistic, multi-user clinical scenarios, such as asthma exacerbations or heart failure, while allowing the controlled introduction of disruptive events (e.g., uncooperative family behavior or sudden patient deterioration). These conditions create rich opportunities to observe how SMMs are established, fail, and recover under stress without compromising patient safety.
Thirty nursing students will be organized into 15 two-person teams. Each team will complete three simulation scenarios in counterbalanced order, with each scenario embedding unexpected events designed to stress test team cognition. During these sessions, participants will wear Tobii Pro eye-tracking glasses and G.Nautilus EEG sensors while interacting within VR. Communication exchanges, gaze patterns, and neural activity will be synchronized and recorded alongside video observations. Immediately after each scenario, participants will complete structured self-reports to provide subjective measures of their shared understanding. Expert raters will annotate video data using established criteria such as information coherence, physical coherence, authority-responsibility alignment, and cognitive coherence, forming a robust ground truth for training models.
The resulting dataset will serve as the foundation for developing machine learning models capable of detecting SMM quality in real time. These models will explore convergent behavioral patterns such as synchronized gaze, balanced communication, and shared task execution. Once validated, the models will be used to implement an AI-driven intervention system that provides adaptive support when poor SMMs are detected. For example, if inefficient communication patterns emerge, the system could issue context-sensitive prompts to encourage clarification or closed-loop communication. This framework, conceptualized as a “Human Digital Cognitive Twin,” computationally mirrors the evolving cognitive state of the team and delivers targeted, timely interventions.
Beyond methodological innovation, this work has significant implications for healthcare training and patient safety. It extends SMM measurement beyond static, post-event evaluations to continuous, real-time monitoring, advancing the state of the art in team cognition research. Furthermore, by demonstrating the feasibility of AI-driven interventions, it provides a model for how intelligent systems can augment human teamwork in high-risk domains. Nursing students stand to benefit from immediate, adaptive feedback that strengthens their coordination skills, while educators gain a scalable and objective tool for training evaluation. More broadly, the proposed framework could be applied in other high-stakes environments such as emergency medicine, aviation, and nuclear operations, where effective teamwork is equally vital.
In conclusion, this project contributes three key advances. First, it introduces a multimodal, machine learning–based approach for objectively and dynamically measuring shared mental models in nursing teams. Second, it pioneers the integration of real-time AI-driven interventions to strengthen teamwork when misalignment is detected. Third, it lays the groundwork for next-generation training systems that enhance safety, efficiency, and resilience in healthcare and beyond. By bridging cognitive theory, sensor-based analytics, and intelligent support, this work charts a path toward more reliable and adaptive teamwork in complex, high-pressure environments.
Event Type
Poster Presentation
TimeMonday, March 234:45pm - 6:15pm EDT
LocationRhinelander Gallery
Simulation and Education