The effect of assessor expertise on reliability of analysis of video signs associated with concussion in Australian football

James R M McLaren; Michael Makdissi; Stephen J C Hearps; Gavin A Davis

doi:10.1016/j.jsams.2024.08.200

The effect of assessor expertise on reliability of analysis of video signs associated with concussion in Australian football

J Sci Med Sport. 2024 Aug 15:S1440-2440(24)00485-7. doi: 10.1016/j.jsams.2024.08.200. Online ahead of print.

Authors

James R M McLaren¹, Michael Makdissi², Stephen J C Hearps³, Gavin A Davis⁴

Affiliations

¹ Geelong Sports Physicians at 10 South, Australia. Electronic address: jmclaren@geelongcats.com.au.
² Australian Football League, Australia; Florey Institute of Neuroscience and Mental Health, Austin Campus, Melbourne Brain Centre, Australia; La Trobe Sport and Exercise Medicine Research Centre, La Trobe University, Australia.
³ Brain & Mind, Murdoch Children's Research Institute, Australia.
⁴ Brain & Mind, Murdoch Children's Research Institute, Australia; Department of Neurosurgery, Austin Hospital, Australia; Department of Neurosurgery, Cabrini Health, Australia.

PMID: 39217069
DOI: 10.1016/j.jsams.2024.08.200

Abstract

Objectives: To determine whether spotters with medical training and experience in managing concussion have higher inter-rater reliability and accuracy than non-medical personnel when identifying video signs associated with concussion in Australian football.

Design: Retrospective cohort study.

Methods: Video clips were collected of all impacts potentially resulting in concussion during 2012 and 2013 Australian Football League (AFL) seasons. Raters were divided into medical doctors and a non-medical group comprising allied health practitioners (physiotherapists) and non-medical/non-allied health personnel (performance analysts). Raters assessed 102 randomly selected videos for signs of concussion. The inter-rater reliability was calculated. Sensitivity, specificity, positive and negative predictive values were calculated by comparing the rater responses to the consensus opinion from two highly experienced clinicians with expertise in concussion.

Results: No statistically significant difference in inter-rater reliability was observed between the medical doctors and the non-medical group. Both groups demonstrated good to excellent agreement for slow to get up, clutching at head/face and facial injury. Both groups displayed intra-class coefficient >0.55 for no protective action-floppy, loss of responsiveness, and motor incoordination, and displayed lowest agreement for no protective action-tonic posturing, impact seizure and blank/vacant look. No statistically significant difference was found between the groups for sensitivity, specificity, positive and negative predictive values for correctly classifying video signs compared to the expert consensus opinion.

Conclusions: After completing sufficient standardised training and testing, medical and non-medical personnel demonstrate comparable reliability in identifying video signs of concussion in professional Australian football and may be suitable for the role of video spotter.

Keywords: Brain concussion; Inter-observer variability; Spotter; Team sports; Video analysis; Video assisted techniques.