Presentation
(Ambient) Listening to Nurses: Evaluating a New AI Tool Through In-Hospital Observations
DescriptionBackground
The growing adoption of artificial intelligence (AI) in healthcare documentation has accelerated in response to increasing regulatory demands and patient acuity. There is hope that AI tools can be used to ease the documentation burden for clinical staff, notably nurses and patient care techs (PCTs). While researchers point to the transformative potential of such tools, integration of AI documentation tools is complex (Alotaibi et al., 2025). In our organization, multidisciplinary teams oversee AI implementation, led by a Chief Digital and Artificial Intelligence Officer. The pilot for a new AI tool involved a private preview model among three partners: the supplier, the electronic health record (EHR) vendor, and our healthcare system. This partnership allowed for continuous feedback and iterative development.
Objective
Our goal was to evaluate the benefits and challenges faced by nurses and PCTs using a new AI documentation tool and to find strategies to support its adoption. Insights would be shared internally and with the supplier to improve the tool and implementation.
Approach
Methods
To observe “work as done” (Hollnagel and Clay-Williams, 2022), we used ethnography and contextual inquiry with teammates completing their daily tasks with patients (Bunger et al., 2021).
Participants
Twenty-two teammates from a Cardiac Telemetry department participated in observations and interviews over two days:
• 15 nurses, 7 Patient Care Techs
• 15 day shift, 7 night shift
• Experience ranged from 2 weeks to 40 years
Setup and Procedure
A Human Factors Engineer and two Clinical Informaticists observed and conducted feedback sessions with staff in the pilot hospital department. At the start, observers explained their role and emphasized confidentiality. Observations occurred in patient rooms and hallways. Staff used hospital-issued mobile devices and desktop EHR software. When appropriate, observers also suggested possible actions for participants to try and also gathered feedback on those experiences.
Results
Several themes emerged around teammate experience. Many staff initially abandoned the tool due to reduced efficiency and accuracy compared to established methods. One nurse said, “When you have 6 patients, you have to be fast!” Teammates’ documentation activities were slower at first due to navigational delays, initiating the recording, and reviewing generated entries. Early language models of health scoring scales, such as Glasgow Coma, Braden, and Hester Davis, were found to be inaccurate—sometimes recording unaddressed responses or misinterpreting spoken ones (e.g., “mmhmm” interpreted as “No”). Users sharedstories of interesting errors which negatively affected others’ perceptions of the tool. Some staff hesitated to send in-app feedback, not wanting to be “rude,” even though supplier improvements rely on it.
Another theme was workflow mismatch: narrating care was not usual practice, and initial training did not emphasize the “care out loud” approach. Nurses have multiple documentation demands and requirements which can be difficult to complete on-time. Users expected the tool to work more like familiar dictation or voice assistant systems. Experience level affected confidence—seasoned nurses were uncertain with new tech, while newer staff were unsure about nursing processes overall. Preceptors, managers, and educators strongly influenced staff attitudes about the tool.
An unexpected finding was that AI tool workflows sometimes displaced nurses’ personal communications with patients. Narrating technical terms limited casual conversations, valued for patient rapport. Some worried about inadvertent recording of non-clinical exchanges and inclusion of these in the medical record. Staff also expressed discomfort holding a cell phone around patients and visitors and doubted the capability of the microphones to capture audio in the room if the nurse did not hold the mobile device close to his/her mouth while speaking
Despite challenges, we observed several positive experiences. When observers encouraged users to retry tasks, successful results led to “Aha!” moments and suggestions for new uses. Realizing the tool learned from their feedback surprised and motivated staff. One nurse gave an observer a high five after a previously inaccurate task, and said, “’All systems within defined limits’ didn’t capture before…Oh wait! It worked! That’s awesome.”
Training emerged as a crucial opportunity. It is difficult to add dedicated training time into nurse and PCT schedules. Nurses often learn from different users and preceptors with varying degrees of success in disseminating consistent and accurate information. In response, the team recommended developing streamlined, immersive training sessions enabling staff to practice patient scenarios together in mini simulations. Team huddles can also be leveraged to share positive experiences and then integrate these real-world examples into training materials.
Conclusions
AI tools generate excitement, but concerns persist, and positive outcomes may be limited, especially at first regarding time savings or workload improvement. Suppliers need to continually improve the design of their tools based on actual use and user needs. Hands-on, realistic training is critical for comprehension, confidence and adoption. Competing priorities—such as timely documentation—make tool implementation challenging, and nurses and PCTs need to experience clear advantages to support their ongoing use.
Lessons Learned
Psychological safety was crucial for open feedback. Staff were more willing to share with internal observers than supplier observers. Including a behavioral scientist facilitated unbiased data collection and including clinical informaticists ensured accurate interpretations of technical performance and feedback. A diverse participant pool enabled a broad range of perspectives on AI documentation tools.
The growing adoption of artificial intelligence (AI) in healthcare documentation has accelerated in response to increasing regulatory demands and patient acuity. There is hope that AI tools can be used to ease the documentation burden for clinical staff, notably nurses and patient care techs (PCTs). While researchers point to the transformative potential of such tools, integration of AI documentation tools is complex (Alotaibi et al., 2025). In our organization, multidisciplinary teams oversee AI implementation, led by a Chief Digital and Artificial Intelligence Officer. The pilot for a new AI tool involved a private preview model among three partners: the supplier, the electronic health record (EHR) vendor, and our healthcare system. This partnership allowed for continuous feedback and iterative development.
Objective
Our goal was to evaluate the benefits and challenges faced by nurses and PCTs using a new AI documentation tool and to find strategies to support its adoption. Insights would be shared internally and with the supplier to improve the tool and implementation.
Approach
Methods
To observe “work as done” (Hollnagel and Clay-Williams, 2022), we used ethnography and contextual inquiry with teammates completing their daily tasks with patients (Bunger et al., 2021).
Participants
Twenty-two teammates from a Cardiac Telemetry department participated in observations and interviews over two days:
• 15 nurses, 7 Patient Care Techs
• 15 day shift, 7 night shift
• Experience ranged from 2 weeks to 40 years
Setup and Procedure
A Human Factors Engineer and two Clinical Informaticists observed and conducted feedback sessions with staff in the pilot hospital department. At the start, observers explained their role and emphasized confidentiality. Observations occurred in patient rooms and hallways. Staff used hospital-issued mobile devices and desktop EHR software. When appropriate, observers also suggested possible actions for participants to try and also gathered feedback on those experiences.
Results
Several themes emerged around teammate experience. Many staff initially abandoned the tool due to reduced efficiency and accuracy compared to established methods. One nurse said, “When you have 6 patients, you have to be fast!” Teammates’ documentation activities were slower at first due to navigational delays, initiating the recording, and reviewing generated entries. Early language models of health scoring scales, such as Glasgow Coma, Braden, and Hester Davis, were found to be inaccurate—sometimes recording unaddressed responses or misinterpreting spoken ones (e.g., “mmhmm” interpreted as “No”). Users sharedstories of interesting errors which negatively affected others’ perceptions of the tool. Some staff hesitated to send in-app feedback, not wanting to be “rude,” even though supplier improvements rely on it.
Another theme was workflow mismatch: narrating care was not usual practice, and initial training did not emphasize the “care out loud” approach. Nurses have multiple documentation demands and requirements which can be difficult to complete on-time. Users expected the tool to work more like familiar dictation or voice assistant systems. Experience level affected confidence—seasoned nurses were uncertain with new tech, while newer staff were unsure about nursing processes overall. Preceptors, managers, and educators strongly influenced staff attitudes about the tool.
An unexpected finding was that AI tool workflows sometimes displaced nurses’ personal communications with patients. Narrating technical terms limited casual conversations, valued for patient rapport. Some worried about inadvertent recording of non-clinical exchanges and inclusion of these in the medical record. Staff also expressed discomfort holding a cell phone around patients and visitors and doubted the capability of the microphones to capture audio in the room if the nurse did not hold the mobile device close to his/her mouth while speaking
Despite challenges, we observed several positive experiences. When observers encouraged users to retry tasks, successful results led to “Aha!” moments and suggestions for new uses. Realizing the tool learned from their feedback surprised and motivated staff. One nurse gave an observer a high five after a previously inaccurate task, and said, “’All systems within defined limits’ didn’t capture before…Oh wait! It worked! That’s awesome.”
Training emerged as a crucial opportunity. It is difficult to add dedicated training time into nurse and PCT schedules. Nurses often learn from different users and preceptors with varying degrees of success in disseminating consistent and accurate information. In response, the team recommended developing streamlined, immersive training sessions enabling staff to practice patient scenarios together in mini simulations. Team huddles can also be leveraged to share positive experiences and then integrate these real-world examples into training materials.
Conclusions
AI tools generate excitement, but concerns persist, and positive outcomes may be limited, especially at first regarding time savings or workload improvement. Suppliers need to continually improve the design of their tools based on actual use and user needs. Hands-on, realistic training is critical for comprehension, confidence and adoption. Competing priorities—such as timely documentation—make tool implementation challenging, and nurses and PCTs need to experience clear advantages to support their ongoing use.
Lessons Learned
Psychological safety was crucial for open feedback. Staff were more willing to share with internal observers than supplier observers. Including a behavioral scientist facilitated unbiased data collection and including clinical informaticists ensured accurate interpretations of technical performance and feedback. A diverse participant pool enabled a broad range of perspectives on AI documentation tools.
Authors
Event Type
Oral Presentations
TimeMonday, March 2311:30am - 12:00pm EDT
LocationNassau
Digital Health