Presentation
CANCELED - Formatives Are Not Summative Rehearsals: Why “Playing It Safe” Early Can Make Devices Less Safe Later.
SessionPoster Session 2
DescriptionIn the field of human factors, formative studies play a critical role in shaping safe and effective medical devices and other complex systems. Their purpose is to explore how people interact with a device, identify risks, and refine designs while improvements are still possible. However, an emerging challenge is that formative studies are increasingly being conducted as “pre-summative” assessments. This shift results in studies that mimic summative evaluations: rigid scripts, narrow focus on critical tasks, rehearsed mitigations, and pressure for participants to succeed. While such an approach may ease the path toward regulatory acceptance, it undermines the exploratory value of formative research and, ultimately, limits the device potential, increase the risk of finding critical issues later on in the process and does not take the full potential of having the user in the loop.
Formative studies thrive on errors and unexpected behaviours, as these are the moments that highlight design weaknesses, usability barriers, and unexplored safety risks. By constraining studies to summative-like designs, manufacturers may miss the opportunity to uncover unforeseen risks, understand real-world patterns of interaction, and explore broader user experiences such as trust, likeability, and intuitive use. To fully realize the value of formative evaluation, different approaches can be applied. These include diversifying methods beyond simulated use (e.g., cognitive walkthroughs, heuristic evaluations), designing flexible study setups, adjusting moderation styles to encourage naturalistic use and failure, and documenting findings in a way that drives meaningful design iteration rather than simply preparing for summative evidence.
This presentation will highlight how HF engineers can reclaim the true purpose of formative evaluations. Attendees will gain practical strategies to balance regulatory expectations with the deeper insights needed to improve user safety, device usability, and overall product success. Examples and case studies will be presented, together with hints on how to have valid results.
2. Key Takeaways
• Understand the risks of treating formative studies as pre-summative evaluations.
• Recognize the unique value of errors, and naturalistic behaviours in formative work.
• Learn practical strategies for structuring formative studies to maximize insight, including alternative methods, flexible setups, and adaptive moderation styles.
• Identify approaches for documenting formative findings in a way that supports iterative design improvements while still aligning with regulatory pathways.
• Gain tools to advocate within organizations for the proper use and design of formative evaluations.
Formative studies thrive on errors and unexpected behaviours, as these are the moments that highlight design weaknesses, usability barriers, and unexplored safety risks. By constraining studies to summative-like designs, manufacturers may miss the opportunity to uncover unforeseen risks, understand real-world patterns of interaction, and explore broader user experiences such as trust, likeability, and intuitive use. To fully realize the value of formative evaluation, different approaches can be applied. These include diversifying methods beyond simulated use (e.g., cognitive walkthroughs, heuristic evaluations), designing flexible study setups, adjusting moderation styles to encourage naturalistic use and failure, and documenting findings in a way that drives meaningful design iteration rather than simply preparing for summative evidence.
This presentation will highlight how HF engineers can reclaim the true purpose of formative evaluations. Attendees will gain practical strategies to balance regulatory expectations with the deeper insights needed to improve user safety, device usability, and overall product success. Examples and case studies will be presented, together with hints on how to have valid results.
2. Key Takeaways
• Understand the risks of treating formative studies as pre-summative evaluations.
• Recognize the unique value of errors, and naturalistic behaviours in formative work.
• Learn practical strategies for structuring formative studies to maximize insight, including alternative methods, flexible setups, and adaptive moderation styles.
• Identify approaches for documenting formative findings in a way that supports iterative design improvements while still aligning with regulatory pathways.
• Gain tools to advocate within organizations for the proper use and design of formative evaluations.
Event Type
Poster Presentation
TimeTuesday, March 244:45pm - 6:15pm EDT
LocationRhinelander Gallery
Medical and Drug Delivery Devices