Presentation
AI Doesn’t Care: The Dilemma of AI Anthropomorphism in Healthcare and Best Practices for Safer Design
SessionDH7: Beyond the Algorithm: Building Trust, Transparency, and Safety into Smart Medical Device Design
DescriptionThis presentation examines how anthropomorphic design elements in AI interfaces (e.g., first-person language, names, personalities, and simulated emotions) can lead to unintended safety risks due to users’ overestimation of system capabilities, inappropriate trust, and other known social and cognitive biases that arise when ascribing humanlike features to nonhuman systems. Human-to-human communication is a highly complex phenomenon that relies on a diverse range of social cues (e.g., body language, facial expressions, voice tonality) to inform critical assumptions about our conversation partners (e.g., intention, theory of mind, social status). Inanimate tools, on the other hand, are not the types of things we evolved to treat as conversation partners. Nonetheless, by the 1950s, popular culture had normalized the idea of talking with a computer program as if it were human. Now, over 70 years later, most of us are on a first-name basis with Siri, Alexa, Cortana, and other AI-powered “virtual assistants”. With high levels of anthropomorphism now being intentionally designed into systems of all kinds, it is important to critically consider the risks associated with personifying these tools. These risks are particularly relevant in the healthcare space where safety and effectiveness are priorities, and where AI tools are being deployed to support a wide range of spaces such as outpatient clinics, mental health apps, and emergency rooms. In these high-stakes contexts, interface design is not just an aesthetic concern but a direct influence on clinical reasoning and patient safety.
Drawing on existing literature in cognitive and social psychology, documented safety issues associated with anthropomorphized AI systems, and our firsthand industry experience, we analyze the ways in which anthropomorphic design elements can drive users to falsely attribute agency, understanding, reasoning, and other human abilities to systems - and explore how this personification amplifies downstream safety risks in the healthcare space and beyond. Our analysis is grounded in human factors frameworks and human-computer interaction (HCI), including trust calibration, distributed cognition, and Norman’s gulfs of execution and evaluation. These lenses help clarify how interface-level design decisions shape user expectations, decision-making, and error detection in clinical workflows.
Though the topic is relatively new, its consequences are very real. In the news, multiple reports of AI’s alleged goading of violence and self-harm have occurred in the past year. In empirical research, Moore et al. (2025) found that AI models trained as therapists acted against accepted medical standards, including stigmatizing mental health conditions and encouraging delusional thinking in simulated patients. It should not surprise us that the more AI acts like a human, the more likely patients will be to confuse emotional responsiveness, faux-familiarity, and other anthropomorphized elements in a chatbot with clinical competence, especially in contexts where access to licensed care is limited. With mounting social and financial pressures further funneling patients towards cheaper, faster alternatives to healthcare, the potential for harm is further emphasized. Accordingly, when an AI purports to replace the responsibilities of a healthcare provider, it is critical to consider how its design can mitigate both the well-known issues of AI systems (hallucinations, lack of transparency, etc.) and foreseeable novel scenarios, such as overreliance by particularly vulnerable patients, populations, and health conditions.
Our analysis includes a catalog of common anthropomorphic design elements (e.g., simulated backstories, memories, personality traits) paired with potential alternative non-anthropomorphic approaches (e.g., choosing mechanical terms over cognitive ones, third-person self-references) and mitigation strategies (e.g., capability reminders, visual differentiation between retrieved and generated content) to reduce anthropomorphism and its associated risks. This analysis serves as a practical design reference to support interface designers, clinical informaticists, and safety engineers in critically evaluating how anthropomorphic choices affect healthcare-specific outcomes such as diagnostic accuracy, situation awareness, and user trust calibration. We then raise and address common justifications for personification and conclude by stressing the need for human-centered design practices to determine whether anthropomorphism is appropriate for a particular application (e.g., adding “eyes” and “emotions” to grocery store robots to deter vandalism) – or if alternative approaches could still maintain the benefits without requiring the misleading theatrics of anthropomorphism. In sum, our presentation challenges prevailing industry practices and establishes a foundation for critically assessing when and how to avoid anthropomorphic user interfaces, particularly in healthcare and other safety-critical industries. We propose that anthropomorphism, while often well-intentioned, may violate the principles of transparent, user-centered healthcare design, introducing new forms of user error and system-level risk that HCI and human factors professionals are uniquely positioned to address.
Drawing on existing literature in cognitive and social psychology, documented safety issues associated with anthropomorphized AI systems, and our firsthand industry experience, we analyze the ways in which anthropomorphic design elements can drive users to falsely attribute agency, understanding, reasoning, and other human abilities to systems - and explore how this personification amplifies downstream safety risks in the healthcare space and beyond. Our analysis is grounded in human factors frameworks and human-computer interaction (HCI), including trust calibration, distributed cognition, and Norman’s gulfs of execution and evaluation. These lenses help clarify how interface-level design decisions shape user expectations, decision-making, and error detection in clinical workflows.
Though the topic is relatively new, its consequences are very real. In the news, multiple reports of AI’s alleged goading of violence and self-harm have occurred in the past year. In empirical research, Moore et al. (2025) found that AI models trained as therapists acted against accepted medical standards, including stigmatizing mental health conditions and encouraging delusional thinking in simulated patients. It should not surprise us that the more AI acts like a human, the more likely patients will be to confuse emotional responsiveness, faux-familiarity, and other anthropomorphized elements in a chatbot with clinical competence, especially in contexts where access to licensed care is limited. With mounting social and financial pressures further funneling patients towards cheaper, faster alternatives to healthcare, the potential for harm is further emphasized. Accordingly, when an AI purports to replace the responsibilities of a healthcare provider, it is critical to consider how its design can mitigate both the well-known issues of AI systems (hallucinations, lack of transparency, etc.) and foreseeable novel scenarios, such as overreliance by particularly vulnerable patients, populations, and health conditions.
Our analysis includes a catalog of common anthropomorphic design elements (e.g., simulated backstories, memories, personality traits) paired with potential alternative non-anthropomorphic approaches (e.g., choosing mechanical terms over cognitive ones, third-person self-references) and mitigation strategies (e.g., capability reminders, visual differentiation between retrieved and generated content) to reduce anthropomorphism and its associated risks. This analysis serves as a practical design reference to support interface designers, clinical informaticists, and safety engineers in critically evaluating how anthropomorphic choices affect healthcare-specific outcomes such as diagnostic accuracy, situation awareness, and user trust calibration. We then raise and address common justifications for personification and conclude by stressing the need for human-centered design practices to determine whether anthropomorphism is appropriate for a particular application (e.g., adding “eyes” and “emotions” to grocery store robots to deter vandalism) – or if alternative approaches could still maintain the benefits without requiring the misleading theatrics of anthropomorphism. In sum, our presentation challenges prevailing industry practices and establishes a foundation for critically assessing when and how to avoid anthropomorphic user interfaces, particularly in healthcare and other safety-critical industries. We propose that anthropomorphism, while often well-intentioned, may violate the principles of transparent, user-centered healthcare design, introducing new forms of user error and system-level risk that HCI and human factors professionals are uniquely positioned to address.
Event Type
Oral Presentations
TimeTuesday, March 244:10pm - 4:30pm EDT
LocationNassau
Digital Health