Medical Call Center Nurse Inter-Rater Agreement in Determining “Who should the Telephone Triage Nurse Speak to Next”; CPI v2
- David A. Thompson MD FACEP 1
- Lorie Whittemore RN, BC, BSN, CCM 2
1 Department of Emergency Medicine, Northwestern Memorial Hospital, Chicago, IL
2 Intellicare, South Portland, ME
Objectives: In many larger medical call centers, a non-clinical service representative answers all inbound calls, records patient demographics, and documents the reason for call using the caller’s own words. The call is entered into a call queue and then may be re-ordered by a supervising nurse based upon the call center’s prioritization policy. The purpose of this study is to examine medical call center nurses’ inter-rater agreement in prioritizing telephone triage calls using a revised call prioritization index (CPI v2).
Methods: A random sample of 200 calls was selected from all inbound telephone triage calls (N = 5,520) for a month period. Four nurses were asked to rank calls using a CPI of 1 (life-threatening; e.g., choking, cardiac arrest, coma) to 5 (health information request, no symptoms) to each of the 200 calls using the provided information for each call of age, gender, and reason for call (chief complaint). The nurses were instructed that the CPI answers the question “who should the telephone triage nurse speak to next?”; the nurses were given the CPI as a table containing the ranking (1-5), description, and examples.
Results: Statistical analysis included computation of overall inter-rater reliability for the four nurses using a Fleiss’ Kappa statistic. The resulting Kappa value for four nurses was 0.534 (“moderate agreement”). The computations of the Kappa statistic will change as the number of raters change; for two nurses (e.g., nurse 1, nurse 3) the Kappa statistics was 0.689 (“substantial agreement”). Spearman’s rank order correlations were performed to determine if nurses were uniform in their prioritization directionally, even if their rankings were not exactly the same. All bivariate correlations between nurses (paired comparisons of two nurses) were significant (p < .01) and showed strong, direct associations (correlation coefficient = 0.679 – 0.774).
Conclusion: Using a call prioritization index (CPI), four nurses achieved moderate agreement ranking 200 telephone triage calls when presented with only the patient age, gender, and a brief reason for call. Individually each nurse prioritizes calls in a manner that is directionally uniform with the other nurses.
Citation: Thompson DA, Whittemore L. Medical Call Center Nurse Inter-Rater Agreement in Determining “Who should the Telephone Triage Nurse Speak to Next”. Call Prioritization Index (CPI Version 2). 2009. Health Navigator, LLC.
- Cartaya S. Improving Patient Care through Technology and Best-in-Industry Process Improvements. Poster Presentation. American Academy of Ambulatory Care Nursing (AAACN) 41st Annual Meeting. 2016. Poster presentation.
- Thompson DA, Bolster A. Medical Call Center Nurse Inter-Rater Agreement in Determining “Who should the Telephone Triage Nurse Speak to Next”. Call Prioritization Index (Version 1). 2008. Abstract and Chart.
- Health Navigator Natural Language Processing engine
View Research and References.