Think Aloud


Participants are instructed to follow “think-aloud” protocols which call for them to verbalize all thoughts as they interact with the system. This approach is particularly well suited for studies exploring adoption and implementation issues.


Usability Testing will be conducted using 3 different types of testing sessions: think aloud, near live, and live. The overall goal of usability testing of the CPR tool is to locally adapt, integrate and prepare for implementation of the CPR in a clinical setting. Through usability testing, the tool is analyzed on how to be embedded into a clinic workflow and perform as a valuable aid to clinical management and patient care.

Design and Methods
The exact number of participants who perform each type of session will depend on data saturation. Participants will do one or two of these sessions with each session lasting approximately 30 minutes.The testing will be conducted at each site where the tool is implemented to ensure the tool is conducive to the clinical workflow. No actual patient data will be utilized during any phase of the usability testing. Using adaptive design, changes are made to the tool based on usability testing results to optimize its clinical utility. As is the usual nature of usability testing studies, testing sessions are scheduled until the point of data saturation. The number and exact nature of the tasks asked during each session will be changed based on the results of previous sessions.

Upon the participant’s arrival to the testing session, staff welcomes the participant and reviews the general procedures for conducting the usability testing (refer to template of welcome script). Prior to starting the testing session, participants are asked to complete a pre-session questionnaire to assess their experience, and perceived usefulness of clinical decision support tools. The questionnaire also includes basic demographic questions.

Task-Directed Think Aloud Sessions
Participants will be set up in front of a computer workstation running the electronic medical records (EMR) system with the tool embedded. They will be provided instruction on how to perform “think aloud” and run through an exercise to practice the new skill. “Think aloud” involves talking through what you are thinking and doing as well as expectations while they perform tasks. Participants will be given instructions by a staff member on specific tasks to perform using the tool. Participants will use different functions and operations of the tool including risk calculator scoring, smartset ordering and locating the tool. Participants will describe what they are doing and what they want to achieve and also to verbalize their experience using the tool.

Case-Based Think Aloud Sessions
Participants will be set up in front of a computer workstation running the EMR system with the tool embedded. They will be provided instruction on how to perform “think aloud” and run through an exercise to practice the new skill. The team will construct clinical scenarios for (insert disease or diagnosis) tool and for each clinical risk level. Participants will be given up to two of these clinical scenarios to test how different aspects of the tool perform. Participants will perform a “think aloud” while they are using the tool for the clinical scenarios.

Data Recording
During all usability testing sessions, data is collected electronically using software on a computer workstation. The user testing software will record both the verbal and electronic interactions of each tester while performing usability testing tasks on the computer workstation. The recordings will include computer screens, what was entered into the tool as well as all participant’s verbalizations. Staff will take notes of the participant’s behaviors, comments, errors and completion (success or failure) on each task. All participants will complete a post-session questionnaire that asks for their perceptions of the tool.

Data Analysis
Qualitative data from the recorded “think aloud”, screen recordings and post-session interviews will be analyzed using content analysis. They will be coded for barriers and facilitators to using the tool. Two raters will review the audio and visual data for each participant. The raters will go through the same training for coding the videos and complete the evaluation matrix listing facilitators and barriers. Descriptive analysis of quantitative pre and post-session questionnaires will be performed to determine means and standard deviations.

Material List

  • Passwords
  • 1 computer workstation with screen capture software loaded
  • Display port adapter
  • Wireless mouse and keyboard adapter
  • 1 monitor
  • 1 keyboard
  • 1 mouse
  • 1 microphone
  • 1 DVI cable
  • 1 HDMI cable
  • 1 VGA cable
  • USB extension cord
  • 1 power strip
  • 1 extension cord
  • 1 consent form w/ID#
  • 1 pre-session questionnaire w/ID#
  • 1 post-test questionnaire w/ID#
  • Welcome script
  • Subject scripts (Scenarios) – separate for each scenario
  • Moderator script
  • Patient information
  • Video


Evaluation Tools and Methods