Machine learning algorithm validation with a limited sample size

Abstract

Advances in neuroimaging, genomic, motion tracking, eye-tracking and many other technology-based data collection methods have led to a torrent of high dimensional datasets, which commonly have a small number of samples because of the intrinsic high cost of data collection involving human participants. High dimensional data with a small number of samples is of critical importance for identifying biomarkers and conducting feasibility and pilot work, however it can lead to biased machine learning (ML) performance estimates. Our review of studies which have applied ML to predict autistic from non-autistic individuals showed that small sample size is associated with higher reported classification accuracy. Thus, we have investigated whether this bias could be caused by the use of validation methods which do not sufficiently control overfitting. Our simulations show that K-fold Cross-Validation (CV) produces strongly biased performance estimates with small sample sizes, and the bias is still evident with sample size of 1000. Nested CV and train/test split approaches produce robust and unbiased performance estimates regardless of sample size. We also show that feature selection if performed on pooled training and testing data is contributing to bias considerably more than parameter tuning. In addition, the contribution to bias by data dimensionality, hyper-parameter space and number of CV folds was explored, and validation methods were compared with discriminable data. The results suggest how to design robust testing methodologies when working with small datasets and how to interpret the results of other studies based on what validation method was used.

Conflict of interest statement

The authors have declared that no competing interests exist.

Figures

Fig 1. Relationship between log10 transformed sample size and reported accuracy.

For 55 studies in the survey which applied ML methods in autism research. A: Relationship between log10 transformed sample size and accuracy, with a dark blue regression line and light blue area showing 95% confidence intervals. B: Classifiers used in the studies. C: Relationship between reported accuracy and log10 transformed sample size by year, bottom scatter-plots are for the studies published in that year. Year 2019 includes studies up to 18/04/2019 when the literature search was performed. N—sample size. D: Relationship between reported accuracy and log10 transformed sample size by modality of data used in the study.

Fig 2. Validation methods.

A: Train/Test Split. B: K-Fold CV. C: Nested CV. D: Partially nested CV. ACC—overall accuracy of the model, ACCi.—accuracy in a single CV fold.

Fig 3. Gaussian noise classification accuracy distributions with different validation approaches.

K-Fold, Nested, Train/Test Split and two types of partially nested validation methods used. Thick lines show mean validation accuracy and dash-dot lines show 95% confidence intervals for 50 runs. A: SVM-RFE feature selection and SVM classification. B: t-test feature selection and logistic regression classification.

Fig 4. Other factors apart from sample size influencing overfitting when K-Fold CV is used.

SVM-RFE and t-test feature selection, SVM classification, and sample size fixed at N = 100. A: Feature number manipulated from 20 to 200. B: Parameter tuning grid size manipulated from 2 × 2 to 20 × 20 with C = 2j, where j varied from 2 to 20 and γ = 2i, where i varied from −2 to −20. C: Number of CV folds varied from two-fold to leave-one out. Thick dashed lines show fitted 5th order polynomial trend.

Fig 5. Other factors apart from sample size influencing overfitting when K-Fold CV is used.

SVM-RFE and t-test feature selection, logistic regression classification, and sample size fixed at N = 100. A: Feature number manipulated from 20 to 200. B: Parameter tuning grid size manipulated from 2 × 2 to 200 × 2 with penalty set to L1, L2 and C = ei, where i varied from −4 to 4. Thick lines show fitted 5th order polynomial trend. C: Number of CV folds varied from two-fold to leave-one-out. Thick dashed lines show fitted 5th order polynomial trend.

Fig 6. K-Fold CV with different feature-to-sample ratios.

Sample size ranged from 14 to 446 and feature number was set accordingly to keep feature-to-sample ratios at 20, 10, 3, 2, 1, 1/2 and 1/3. A: SVM-RFE feature selection and SVM classifier. B: t-test feature selection and logistic regression classifier.

Fig 7. Classification with discriminable data using K-Fold CV, Nested CV and Train/Test Split validation methods.

A: Comparison of different validation methods. Dash-dot lines show 95% confidence intervals for 50 runs. B: Size of 95% confidence intervals. Inset plot shows more refined view of confidence intervals for K-fold CV in a sample size range of 20 to 200 (in the inset plot sample sizes were N = 20, 22, … 198, 200, in the main plot N = 20, 40, … 980, 1000).

Fig 8. Illustrative examples of why models overfit.

A: SVM-RBF decision boundary. Red and blue circles/crosses show data points from two classes, red and blue areas show learned decision boundary by SVM-RBF. Left: Classifier trained on both train data points (circles) and validation data points (crosses). Right: Classifier trained only on train data points (circles). B: Two-sample t-test feature selection performed both, on pooled and on independent train and validation data. Y axis shows mean t-statistic for selected 10 features from the pool of features ranging from 20 to 100.