Medical Call Center Nurse Inter-Rater Agreement in Determining “Who should the Telephone Triage Nurse Speak to Next”; CPI v1



  • David A. Thompson MD FACEP 1
  • Anita Bolster RN, BS, CCM 2

1 Department of Emergency Medicine, MacNeal Hospital, Berwyn, IL

2 Intellicare, South Portland, ME 

Objectives:  In many larger medical call centers, a non-clinical service representative answers all inbound calls, records patient demographics, and documents the reason for call using the caller’s own words. The call is entered into a call queue and then may be re-ordered by a supervising nurse based upon the call center’s prioritization policy. The purpose of this study is to examine medical call center nurses’ inter-rater agreement in prioritizing telephone triage calls using a call prioritization index (CPI).

Methods: A random sample of 200 calls was selected from all inbound telephone triage calls (N = 5,520) for a month period. Five nurses were asked to rank calls using a CPI of 1 (life-threatening; e.g., choking, cardiac arrest, coma) to 5 (non-urgent; e.g., runny nose, health information request) to each of the 200 calls using the provided information for each call of age, gender, and reason for call. The nurses were instructed that the CPI answers the question “who should the telephone triage nurse speak to next?”; the nurses were given the CPI as a table containing the ranking (1-5), description, and examples.

Results: Statistical analysis included computation of overall inter-rater reliability for the five nurses using a Fleiss’ Kappa statistic. The resulting Kappa value for five nurses was 0.134 (“slight agreement”); the Kappa score for two nurses was 0.402 (“fair to moderate agreement”). Spearman’s rank order correlations were performed to determine if nurses were uniform in their prioritization directionally, even if their rankings were not exactly the same. All bivariate correlations between nurses (paired comparisons of two nurses) were significant (p < .01) and showed strong, direct associations (correlation coefficient = 0.615 – 0.724).

Conclusion: Using a call prioritization index (CPI), five nurses achieved better than chance agreement ranking 200 telephone triage calls. Individually each nurse prioritizes calls in a manner that is directionally uniform with the other nurses.


Citation:  Thompson DA, Bolster A. Medical Call Center Nurse Inter-Rater Agreement in Determining “Who should the Telephone Triage Nurse Speak to Next”. Call Prioritization Index (CPI Version 1). 2008. Health Navigator, LLC.

Related References

  • Cartaya S.  Improving Patient Care through Technology and Best-in-Industry Process Improvements. Poster Presentation. American Academy of Ambulatory Care Nursing (AAACN) 41st Annual Meeting. 2016. Poster presentation.
  • Thompson DA, Whittemore L. Medical Call Center Nurse Inter-Rater Agreement in Determining “Who should the Telephone Triage Nurse Speak to Next”. Call Prioritization Index (Version 2). 2009. Abstract and Chart.

Related Links

View Research and References.