Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties

S Boet; S Larrigan; L Martin; H Liu; K J Sullivan; Cole Etherington

doi:10.1016/j.bja.2018.07.028

Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties

Br J Anaesth. 2018 Dec;121(6):1218-1226. doi: 10.1016/j.bja.2018.07.028. Epub 2018 Sep 6.

Authors

S Boet¹, S Larrigan², L Martin³, H Liu³, K J Sullivan⁴, Cole Etherington⁴

Affiliations

¹ Department of Anaesthesiology and Pain Medicine, Ottawa Hospital, Ottawa, ON, Canada; Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada; Department of Innovation in Medical Education, University of Ottawa, ON, Canada. Electronic address: sboet@toh.ca.
² Translational and Molecular Medicine Program, ON.
³ Faculty of Medicine, ON.
⁴ Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada.

Abstract

Background: Non-technical skills, such as communication or leadership, are integral to clinical competence in anaesthesia. There is a need for valid and reliable tools to measure anaesthetists' non-technical performance for both initial and continuing professional development. This systematic review aims to summarise the measurement properties of existing assessment tools to determine which tool is most robust.

Methods: Embase (via OVID), Medline and Medline in Process (via OVID), and reference lists of included studies and previously published relevant systematic reviews were searched (through August 2017). Quantitative studies investigating the measurement properties of tools used to assess anaesthetists' intraoperative non-technical skills, either in a clinical or simulated environment, were included. Pairs of independent reviewers determined eligibility and extracted data. Risk of bias was assessed using the COSMIN checklist.

Results: The search yielded 978 studies, of which 14 studies describing seven tools met the inclusion criteria. Of these, 12 involved simulated crisis settings only. The measurement properties of the Anaesthetists' Non-Technical Skills (ANTS) tool were most commonly assessed (n=9 studies), with studies of two types of validity (content, concurrent) and two types of reliability (internal consistency, interrater). Most of these studies, however, were at serious risk of bias.

Conclusions: Though there are seven tools for assessing the non-technical skills of anaesthetists, only ANTS has been extensively investigated with regard to its measurement properties. ANTS appears to have acceptable validity and reliability for assessing non-technical skills of anaesthetists in both simulated and clinical settings. Future research should consider additional clinical contexts and types of measurement properties.

Keywords: anaesthesiology; communication; professional competence; professionalism; psychometrics; situational awareness.

Publication types

Systematic Review

MeSH terms

Anesthesiologists*
Bias
Checklist
Clinical Competence*
Educational Measurement
Humans
Operating Rooms*
Reproducibility of Results