Confidence intervals and sample size planning for optimal cutpoints

Abstract

Various methods are available to determine optimal cutpoints for diagnostic measures. Unfortunately, many authors fail to report the precision at which these optimal cutpoints are being estimated and use sample sizes that are not suitable to achieve an adequate precision. The aim of the present study is to evaluate methods to estimate the variance of cutpoint estimations based on published descriptive statistics ('post-hoc') and to discuss sample size planning for estimating cutpoints. We performed a simulation study using widely-used methods to optimize the Youden index (empirical, normal, and transformed normal method) and three methods to determine confidence intervals (the delta method, the parametric bootstrap, and the nonparametric bootstrap). We found that both the delta method and the parametric bootstrap are suitable for post-hoc calculation of confidence intervals, depending on the sample size, the distribution of marker values, and the correctness of model assumptions. On average, the parametric bootstrap in combination with normal-theory-based cutpoint estimation has the best coverage. The delta method performs very well for normally distributed data, except in small samples, and is computationally more efficient. Obviously, not every combination of distributions, cutpoint optimization methods, and optimized metrics can be simulated and a lot of the literature is concerned specifically with cutpoints and confidence intervals for the Youden index. This complicates sample size planning for studies that estimate optimal cutpoints. As a practical tool, we introduce a web-application that allows for running simulations of width and coverage of confidence intervals using the percentile bootstrap with various distributions and cutpoint optimization methods.

Conflict of interest statement

The authors have declared that no competing interests exist.

Copyright: © 2023 Thiele, Hirschfeld. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Figures

Fig 1. Coverage probabilities and widths of 95% confidence intervals for the optimal cutpoints on normally distributed data.

Fig 2. Coverage probabilities and widths of 95% confidence intervals for the optimal cutpoints on lognormally distributed data.

Fig 3. Jitter plot of the width of 95% confidence intervals in terms of the pooled standard deviation in different scenarios with normally distributed data depending on the total sample size, the ratio of the sizes of the two groups and for four ranges of Cohen’s d.