Live Usability

Purpose

Live usability testing captures usability data during a real patient-physician encounter with real time pressure and patient-case complexity. This data may uncover obstacles to efficient workflow that are unmeasured in simulated clinical scenarios.

Protocol

Objectives
Usability Testing will be conducted using 3 different types of testing sessions: think aloud, near live, and live. The overall goal of usability testing of the CPR tool is to locally adapt, integrate and prepare for implementation of the CPR in a clinical setting. Through usability testing, the tool is analyzed on how to be embedded into a clinic workflow and perform as a valuable aid to clinical management and patient care.

Design and Methods
The exact number of participants who perform each type of session will depend on data saturation. Participants will do one or two of these sessions with each session lasting approximately 30 minutes.The testing will be conducted at each site where the tool is implemented to ensure the tool is conducive to the clinical workflow. No actual patient data will be utilized during any phase of the usability testing. Using adaptive design, changes are made to the tool based on usability testing results to optimize its clinical utility. As is the usual nature of usability testing studies, testing sessions are scheduled until the point of data saturation. The number and exact nature of the tasks asked during each session will be changed based on the results of previous sessions.

Upon the participant’s arrival to the testing session, staff welcomes the participant and reviews the general procedures for conducting the usability testing (refer to template of welcome script). Prior to starting the testing session, participants are asked to complete a pre-session questionnaire to assess their experience, and perceived usefulness of clinical decision support tools. The questionnaire also includes basic demographic questions.

Live Sessions: Using data from the “think alouds” and near live testing the tool will be refined, as needed. Over the development period prototypes will be tested in increasingly realistic scenarios with final versions being tested during live clinical care sessions that reproduce and simulate the issues of time pressures and patient-case complexity. This phase in the usability testing may uncover obstacles to efficient workflow that are unmeasured in simulated clinical scenarios. By understanding these obstacles, the tool can be better designed and implemented to fit the clinical workflow.

Data Recording: During all usability testing sessions, data is collected electronically using software on a computer workstation. The user testing software will record both the verbal and electronic interactions of each tester while performing usability testing tasks on the computer workstation. The recordings will include computer screens, what was entered into the tool as well as all participant’s verbalizations. Patients’ identifiers visible during screen capture will be digitally blocked out and deleted from audio recordings. Staff will take notes of the participant’s behaviors, comments, errors and completion (success or failure) on each task. All participants will complete a post-session questionnaire that asks for their perceptions of the tool.

Data Analysis: Qualitative data from the recorded “live”, screen recordings and post-session interviews will be analyzed using content analysis. They will be coded for barriers and facilitators to using the tool. Two raters will review the audio and visual data for each participant. The raters will go through the same training for coding the videos and complete the evaluation matrix listing facilitators and barriers. Descriptive analysis of quantitative pre and post-session questionnaires will be performed to determine means and standard deviations.

Material List

  • Passwords
  • 1 computer workstation with screen capture software loaded
  • Display port adapter
  • Wireless mouse and keyboard adapter
  • 1 monitor
  • 1 keyboard
  • 1 mouse
  • 1 microphone
  • 1 DVI cable
  • 1 HDMI cable
  • 1 VGA cable
  • USB extension cord
  • 1 power strip
  • 1 extension cord
  • 1 consent form w/ID#
  • 1 pre-session questionnaire w/ID#
  • 1 post-test questionnaire w/ID#
  • Welcome script
  • Patient information
  • Video

Templates

Evaluation Tools