Machine Learning for the Diagnosis of Orthodontic Extractions: A Computational Analysis Using Ensemble Learning

Abstract

Extraction of teeth is an important treatment decision in orthodontic practice. An expert system that is able to arrive at suitable treatment decisions can be valuable to clinicians for verifying treatment plans, minimizing human error, training orthodontists, and improving reliability. In this work, we train a number of machine learning models for this prediction task using data for 287 patients, evaluated independently by five different orthodontists. We demonstrate why ensemble methods are particularly suited for this task. We evaluate the performance of the machine learning models and interpret the training behavior. We show that the results for our model are close to the level of agreement between different orthodontists.

Keywords: ensemble methods; machine learning; neural network; orthodontics; random forests.

Conflict of interest statement

The authors and the University of Connecticut have filed U.S. Provisional Patent Application No. 62/915,725 based on this work on October 16, 2019.

Figures

Figure 1

Schematic of the procedure followed in this work, from data collection to the machine learning diagnosis.

Figure 2

Index of the different extraction options. The diagram on the left shows the locations of the upper and lower premolars. The 14 options on the right list the specific extraction procedures in terms of the locations of the teeth. NE refers to no extraction.

Figure 3

Demographic background of patients. (A) Age distribution, and (B) gender distribution.

Figure 4

Performance of the single classifiers, when considering (A) only the primary diagnosis, and (B) both the primary and alternative.

Figure 5

Training time for the single classifiers. Logistic regression was also trained with product terms (i.e., two-way interactions), dramatically increasing the training time. Due to the large dynamic range needed on the y-axis, it is drawn in the pseudolog transform.

Figure 6

Effect of the weight regularization in the neural network/multinomial regression model on the training set error. The x-axis shows the regularization weight. The top row is for the neural network utilizing the raw input features while the bottom row is for using two-way interactions. The 3 columns show the results for the weight regularization penalty term formulated as an elastic net, L1 norm, and L2 norm.

Figure 7

Effect of the weight regularization in the neural network/multinomial regression model on the test set error. The x-axis shows the regularization weight, with rows and columns corresponding to the interactions and weight regularization method as in Figure 6.

Figure 8

Effect of various training parameters on the random forest model for the prediction of the specific extraction. The minimum node size, features tried at every level of split, and the number of trees are varied and the error rates for the training and test split are plotted. In (A), a prediction is considered as an error if it does not agree with the expert’s primary diagnosis, and in (B), it is considered an error if the prediction does not agree with the primary or alternative diagnosis.

Figure 9

Effect of various training parameters on the random forest model for the binary prediction problem. The minimum node size, features tried at every level of split, and the number of trees are varied and the error rates for the training and test split are plotted. In (A), a prediction is considered as an error if it does not agree with the expert’s primary diagnosis, and in (B), it is considered an error if the prediction does not agree with the primary or alternative diagnosis.

Figure 10

Saturating effect of increasing the number of classifiers in the random forest. The out-of-bag accuracy (an estimate of the test accuracy) plotted against the number of trees for the random forest model predicting the specific type of interaction. This is for a minimal node size of 1 and trying all possible features at every split.

Figure 11

Performance of all the classifiers for predicting (A) the primary diagnosis, and (B) where agreement with either the primary or the alternative diagnoses is considered to be accurate. Here, both the single and ensemble (random forest) classifiers are included.

Figure 12

Effect of individual features. The test error for the predicting of the specific treatment plan using the neural network after independently deleting single features from the dataset.