Presentation
One Metric to Rule Them All? Piloting Global Usability Measures in Healthcare
DescriptionThis presentation builds upon our poster at the 2025 HFES Healthcare Symposium, focusing on our metrics journey. The team continues to evolve the approach of usability metrics within the healthcare domain. In our previous session, we outlined the process of selecting a global usability metric framework suitable for enterprise-wide adoption. During 2026 HFES, we are excited to share the next phase of the journey: piloting and evaluating two selected metrics – the Efficiency Equation and the Single Usability Metric (SUM).
The Efficiency Equation is a straightforward yet powerful metric that combines Time on Task (ToT) with Task Success Rate, offering a quantifiable measure of user performance. In contrast, the SUM provides a more comprehensive view by integrating four key usability parameters:
• Time on Task (ToT)
• Task Success Rate
• Error Rate
• User Satisfaction, measured via a brief three-question survey
These metrics were chosen for their potential to provide both high-level insights for leadership and actionable data for product teams.
The team ran several pilot usability studies to evaluate the feasibility of applying these metrics to real-life clinical workflows. These pilots revealed a set of unique challenges that differ from those encountered in traditional formative or summative usability testing. The nature of metric-driven evaluation introduces complexities that require thoughtful consideration and cross-functional collaboration.
This presentation delves into each of these challenges from piloting these two metrics and provides solutions to each of these challenges based on our lessons learned. Some of the challenges we have encountered are:
• Virtual versus in-person technology issues
• Calculating ToT using free software
• Defining error opportunities with cross functional teams
• How/what tasks to select to include in the metrics
• Proficiency of participants on the system
• SUM analysis observations
• Comparing metrics across products
• When to apply a global metric
A critical component of our initiative is ensuring that usability data is consumable and actionable for different audiences. We will present an example data visualization and findings to be consumed by the business leadership and product development teams. It is imperative to curate the findings and observations for the different audiences, because leadership may want a single number to quickly ascertain if the product (or set of features) is “good” or “better” than a predecessor, while the product development team may be looking for actionable recommendations to iterate the product.
Lastly, we will discuss rolling out a global metric to a broader audience and the challenges with such an initiative.
The Efficiency Equation is a straightforward yet powerful metric that combines Time on Task (ToT) with Task Success Rate, offering a quantifiable measure of user performance. In contrast, the SUM provides a more comprehensive view by integrating four key usability parameters:
• Time on Task (ToT)
• Task Success Rate
• Error Rate
• User Satisfaction, measured via a brief three-question survey
These metrics were chosen for their potential to provide both high-level insights for leadership and actionable data for product teams.
The team ran several pilot usability studies to evaluate the feasibility of applying these metrics to real-life clinical workflows. These pilots revealed a set of unique challenges that differ from those encountered in traditional formative or summative usability testing. The nature of metric-driven evaluation introduces complexities that require thoughtful consideration and cross-functional collaboration.
This presentation delves into each of these challenges from piloting these two metrics and provides solutions to each of these challenges based on our lessons learned. Some of the challenges we have encountered are:
• Virtual versus in-person technology issues
• Calculating ToT using free software
• Defining error opportunities with cross functional teams
• How/what tasks to select to include in the metrics
• Proficiency of participants on the system
• SUM analysis observations
• Comparing metrics across products
• When to apply a global metric
A critical component of our initiative is ensuring that usability data is consumable and actionable for different audiences. We will present an example data visualization and findings to be consumed by the business leadership and product development teams. It is imperative to curate the findings and observations for the different audiences, because leadership may want a single number to quickly ascertain if the product (or set of features) is “good” or “better” than a predecessor, while the product development team may be looking for actionable recommendations to iterate the product.
Lastly, we will discuss rolling out a global metric to a broader audience and the challenges with such an initiative.
Event Type
Oral Presentations
TimeWednesday, March 2511:15am - 11:37am EDT
LocationNassau
Digital Health