Automatic Posture and Movement Tracking of Infants with Wearable Movement Sensors

Abstract

Infants' spontaneous and voluntary movements mirror developmental integrity of brain networks since they require coordinated activation of multiple sites in the central nervous system. Accordingly, early detection of infants with atypical motor development holds promise for recognizing those infants who are at risk for a wide range of neurodevelopmental disorders (e.g., cerebral palsy, autism spectrum disorders). Previously, novel wearable technology has shown promise for offering efficient, scalable and automated methods for movement assessment in adults. Here, we describe the development of an infant wearable, a multi-sensor smart jumpsuit that allows mobile accelerometer and gyroscope data collection during movements. Using this suit, we first recorded play sessions of 22 typically developing infants of approximately 7 months of age. These data were manually annotated for infant posture and movement based on video recordings of the sessions, and using a novel annotation scheme specifically designed to assess the overall movement pattern of infants in the given age group. A machine learning algorithm, based on deep convolutional neural networks (CNNs) was then trained for automatic detection of posture and movement classes using the data and annotations. Our experiments show that the setup can be used for quantitative tracking of infant movement activities with a human equivalent accuracy, i.e., it meets the human inter-rater agreement levels in infant posture and movement classification. We also quantify the ambiguity of human observers in analyzing infant movements, and propose a method for utilizing this uncertainty for performance improvements in training of the automated classifier. Comparison of different sensor configurations also shows that four-limb recording leads to the best performance in posture and movement classification.

Conflict of interest statement

The authors declare no competing interests.

Figures

Figure 1

Experimental design. (a) Photograph of the smart jumpsuit with four proximally placed movement sensors. (b) The annotation setup displaying annotations with synchronized video and movement data.

Figure 2

Design of the automatic classification pipeline. (a) The convolutional neural network (CNN) architecture used in the study as the main classifier. The role of the sensor module is to perform sensor-specific feature extraction, the sensor fusion module fuses sensor-level features into frame-level features, and the time series modeling module captures the temporal dependencies across frame-level features. (b) Block diagram for the iterative annotation refinement (IAR) procedure used to improve CNN performance through classifier-assisted resolution of inter-annotator inconsistencies on the training data.

Figure 3

(a) Total cumulative confusion matrices across all possible annotator pairs for the Posture (top) and Movement (bottom) track. The percentages of each column sum up to one and the absolute values denote the number of frames corresponding each cell. (b) t-SNE visualization of the entire dataset based on SVM input features. Color coding is based on the annotations of Posture (top left) and Movement categories (the rest). The visualization of the Movement track has been broken down into 3/3 (top right), 2/3 (bottom right), and 1/3 (bottom left) annotator agreement levels. The ambiguity differences in the annotation accuracies between the tracks can be clearly seen.

Figure 4

Performance of classifiers. Class-specific F-score box plots for individual recordings for the Posture and Movement tracks using the CNN (blue) and SVM (red) classifiers. Statistically significant recording-level differences (p 

Figure 5

The effect of sensor setup on classification performance (UAR) using the SVM classifier. Any individual sensor configuration is inferior to the four-sensor setup. However, classification of data from a combination of one arm and one leg leads to almost comparable results.

Figure 6

Differentiation of high and low motor performance infants with the smart jumpsuit. The plots show individual category distributions (as log-probability) of the given posture and movement category, presented for the entire dataset. Results from both human annotation (x) and the classifier output (triangle) are shown for comparison, and the hairlines connect individuals to assess the individual level reliability. The highlighted recordings correspond to a sample of high performing (red; High perf.) and low performing infants (blue; Low perf.). Rest of the infant cohort is plotted with light gray lines. No statistically significant differences were found between the movement distributions from the human annotations and from classifier outputs (Mann-Whitney U-test, N = 22).

Figure 7

Total CNN classifier confusion matrices of the posture (a,b) and movement (c,d) tracks obtained from LOSO cross-validation of the (a,c) full annotation agreement subset and (b,d) complete data set. The percentage values inside the cells indicate the class-specific recall values, and the absolute values denote the number of frames. Average metrics are presented in Table 1.

Figure 8

The effect of IAR-based label refinement on the agreement (κ) between CNN classifier outputs and the three original human annotation tracks (gray lines) as a function of IAR iterations on held-out test data. The mean pair-wise agreement between classifier and the three human annotations is shown with the blue line and Fleiss’ κ agreement across all human annotators is shown with the dotted black line. A clear improvement in classifier performance is observed due to label refinement on the training data, reaching the human-to-human agreement rate.