Presentation
Verbal vs. Written Administration of Usability Study Knowledge Tasks: Implications for Data Quality, Participant Experience, and Regulatory Alignment
DescriptionHuman factors validation studies are high-stakes activities that play a critical role in demonstrating the safety and usability of medical devices. A key component of these studies involves knowledge tasks, where participants may recall or articulate safe use information. The method of presenting these tasks, verbally versus in writing, may introduce significant artifacts that affect study outcomes.
Knowledge tasks tend to be administered verbally in most human factors validation studies. Task cards are sometimes used to supplement verbal task prompts, but they are rarely implemented on their own. Despite this, verbally administered knowledge tasks may place participants under stress, encourage rushed or socially-desirable responses, and inadvertently test personal experience rather than task-relevant comprehension. Written administration, by contrast, may yield more accurate and less biased data, all while decreasing stress on the respondent. Written administration can give participants time for reflection and reduce moderator influence, but could also introduce its own challenges, such as differences in literacy and the artificiality of written test conditions.
The current regulatory climate is relevant. The FDA has recently scrutinized knowledge task methodology with increasing conservatism, emphasizing the need for methods that minimize participant bias and maximize data integrity. However, the guidance leaves open questions about how best to administer these tasks. Recently announced AAMI HE75 updates (which were approved by FDA) somewhat address this issue, but fail to address concerns related to participant stress and data quality.
This research study empirically evaluates differences in the following between verbal and written administration of knowledge tasks:
• Knowledge task performance: number of use errors, close calls, and difficulties observed
• Data integrity: frequency of study artifacts and/or participants inventing answers based on prior knowledge rather than device labeling
• Efficiency metrics: time required to administer knowledge tasks verbally vs. in writing
• Moderator influence: intervention rate and degree of clarification needed
• Stress indicators: perceived anxiety levels and self-reported stress
In addition to primary outcomes, we explore methodological considerations: how root cause debriefing and follow-up questioning differ when tasks are written rather than verbal. For example, written responses may require a structured probe framework to clarify ambiguous answers, while verbal questioning naturally allows real-time probing but risks moderator influence. The study examines best practices for mitigating these tradeoffs.
Background literature from cognitive psychology and assessment science indicates that written formats often lead to greater accuracy, reduced performance anxiety, and less susceptibility to moderator bias compared to oral exams. Conversely, oral questioning may better approximate real-world device use scenarios where individuals must recall information under pressure. By systematically comparing the two modalities, our study aims to clarify whether written administration represents a more scientifically rigorous and regulatorily defensible approach, or whether verbal questioning, despite its pitfalls, remains an appropriate and realistic method.
Knowledge tasks tend to be administered verbally in most human factors validation studies. Task cards are sometimes used to supplement verbal task prompts, but they are rarely implemented on their own. Despite this, verbally administered knowledge tasks may place participants under stress, encourage rushed or socially-desirable responses, and inadvertently test personal experience rather than task-relevant comprehension. Written administration, by contrast, may yield more accurate and less biased data, all while decreasing stress on the respondent. Written administration can give participants time for reflection and reduce moderator influence, but could also introduce its own challenges, such as differences in literacy and the artificiality of written test conditions.
The current regulatory climate is relevant. The FDA has recently scrutinized knowledge task methodology with increasing conservatism, emphasizing the need for methods that minimize participant bias and maximize data integrity. However, the guidance leaves open questions about how best to administer these tasks. Recently announced AAMI HE75 updates (which were approved by FDA) somewhat address this issue, but fail to address concerns related to participant stress and data quality.
This research study empirically evaluates differences in the following between verbal and written administration of knowledge tasks:
• Knowledge task performance: number of use errors, close calls, and difficulties observed
• Data integrity: frequency of study artifacts and/or participants inventing answers based on prior knowledge rather than device labeling
• Efficiency metrics: time required to administer knowledge tasks verbally vs. in writing
• Moderator influence: intervention rate and degree of clarification needed
• Stress indicators: perceived anxiety levels and self-reported stress
In addition to primary outcomes, we explore methodological considerations: how root cause debriefing and follow-up questioning differ when tasks are written rather than verbal. For example, written responses may require a structured probe framework to clarify ambiguous answers, while verbal questioning naturally allows real-time probing but risks moderator influence. The study examines best practices for mitigating these tradeoffs.
Background literature from cognitive psychology and assessment science indicates that written formats often lead to greater accuracy, reduced performance anxiety, and less susceptibility to moderator bias compared to oral exams. Conversely, oral questioning may better approximate real-world device use scenarios where individuals must recall information under pressure. By systematically comparing the two modalities, our study aims to clarify whether written administration represents a more scientifically rigorous and regulatorily defensible approach, or whether verbal questioning, despite its pitfalls, remains an appropriate and realistic method.
Event Type
Oral Presentations
TimeMonday, March 2310:52am - 11:15am EDT
LocationGramercy
Medical and Drug Delivery Devices