Near Live

Purpose

The “near live” usability testing is where participants conduct encounters with mock patients as if they were working in a live environment. If required, additional cycles are conducted to model the impact of proposed changes to the tool before implementation.

Protocol

Objectives
Usability Testing will be conducted using 3 different types of testing sessions: think aloud, near live, and live. The overall goal of usability testing of the CPR tool is to locally adapt, integrate and prepare for implementation of the CPR in a clinical setting. Through usability testing, the tool is analyzed on how to be embedded into a clinic workflow and perform as a valuable aid to clinical management and patient care.

Design and Methods
The exact number of participants who perform each type of session will depend on data saturation. Participants will do one or two of these sessions with each session lasting approximately 30 minutes.The testing will be conducted at each site where the tool is implemented to ensure the tool is conducive to the clinical workflow. No actual patient data will be utilized during any phase of the usability testing. Using adaptive design, changes are made to the tool based on usability testing results to optimize its clinical utility. As is the usual nature of usability testing studies, testing sessions are scheduled until the point of data saturation. The number and exact nature of the tasks asked during each session will be changed based on the results of previous sessions.

Upon the participant’s arrival to the testing session, staff welcomes the participant and reviews the general procedures for conducting the usability testing (refer to template of welcome script). Prior to starting the testing session, participants are asked to complete a pre-session questionnaire to assess their experience, and perceived usefulness of clinical decision support tools. The questionnaire also includes basic demographic questions.

Near Live Sessions: Using data from the “think alouds,” the tool will be refined, as needed. Over the development period prototypes will be tested in increasingly realistic scenarios with final versions being tested during clinical care sessions that reproduce and simulate the issues of time pressures and patient-case complexity. This process will allow the developers to ascertain characteristics of the tool experience that are functional, need improvement, fit user expectations, miss expectations, fail to function, or are opportunities for further development. Participants will be set up in front of a computer workstation running the electronic medical records (EMR) system with the tool embedded. Participants will interact with standardized patients (actors) or on another computer they will view videos of standardized patients (actors) who are playing the role of a patient sick with possible (insert disease or diagnosis). Participants will use the tool as they would during an actual patient encounter.

Data Recording: During all usability testing sessions, data is collected electronically using software on a computer workstation. The user testing software will record both the verbal and electronic interactions of each tester while performing usability testing tasks on the computer workstation. The recordings will include computer screens, what was entered into the tool as well as all participant’s verbalizations. Staff will take notes of the participant’s behaviors, comments, errors and completion (success or failure) on each task. All participants will complete a post-session questionnaire that asks for their perceptions of the tool.

Data Analysis: Qualitative data from the recorded “near live”, screen recordings and post-session interviews will be analyzed using content analysis. They will be coded for barriers and facilitators to using the tool. Two raters will review the audio and visual data for each participant. The raters will go through the same training for coding the videos and complete the evaluation matrix listing facilitators and barriers. Descriptive analysis of quantitative pre and post-session questionnaires will be performed to determine means and standard deviations.

Material List

  • Passwords
  • 1 computer workstation with screen capture software loaded
  • Display port adapter
  • Wireless mouse and keyboard adapter
  • 1 monitor
  • 1 keyboard
  • 1 mouse
  • 1 microphone
  • 1 DVI cable
  • 1 HDMI cable
  • 1 VGA cable
  • USB extension cord
  • 1 power strip
  • 1 extension cord
  • 1 consent form w/ID#
  • 1 pre-session questionnaire w/ID#
  • 1 post-test questionnaire w/ID#
  • Welcome script
  • Subject scripts (Scenarios) – separate for each scenario
  • Patient information
  • Video

Templates

Evaluation Tools

Case Studies