Objective dairy cow mobility analysis and scoring system using computer vision-based keypoint detection technique from top-view 2D videos

Abstract

The objective of this study was to assess the applicability of a computer vision-based keypoint detection technique to extract mobility variables associated with mobility scores from top-view 2-dimensional (2D) videos of dairy cows. In addition, the study determined the potential of a machine learning classification model to predict mobility scores based on the newly extracted mobility variables. A data set of 256 video clips of individual cows was collected, with each clip recorded from a top-view perspective while the cows were walking. The cows were visually assessed using a 4-level mobility scoring system, comprising Score 0 (good mobility: 78 cows), Score 1 (imperfect mobility: 71 cows), Score 2 (impaired mobility: 87 cows), and Score 3 (severely impaired mobility: 20 cows). The video clips were analyzed using a keypoint detection model capable of detecting 10 keypoints (i.e., head, neck, withers, back, hip ridge, tail head, left and right hooks, and left and right pins). From the time-series XY-coordinate data of the keypoints, 25 mobility variables were extracted, including lateral movements of keypoints (10 variables), coefficients of variation of keypoint speeds (10 variables), walking speed (1 variable), and standard deviations of keypoint angles (4 variables: neck angle, withers angle, back angle, and hip angle). Due to the limited number of cows classified as Score 3, they were combined with Score 2 cows into a single class. Subsequently, a 3-level mobility score classification model (Score 0, 1, and 2 + 3) was developed using the random forest algorithm, based on the extracted mobility variables. The model's performance was evaluated using the repeated holdout method, where the data set was randomly divided into 80% for training and 20% for testing, repeated 10 times. The model's overall 3-class classification performance achieved a weighted kappa coefficient of 0.72 and an area under the curve of the receiver operating characteristic curve of 0.89. Based on the variable importance analysis conducted over the cross-validation, back lateral movement, withers lateral movement, walking speed, and tail head lateral movement were identified as crucial for predicting mobility scores. These findings indicate that the computer vision-based keypoint detection technique is effective for extracting mobility variables relevant to mobility scores from top-view 2D videos, and the machine learning classification model based on the newly extracted variables has the potential for objective mobility scoring in dairy cows.

Keywords: cattle; lameness; overhead; two-dimension.