Bayesian Hodges-Lehmann tests for statistical equivalence in the two-sample setting: Power analysis, type I error rates and equivalence boundary selection in biomedical research

Abstract

Background: Null hypothesis significance testing (NHST) is among the most frequently employed methods in the biomedical sciences. However, the problems of NHST and p-values have been discussed widely and various Bayesian alternatives have been proposed. Some proposals focus on equivalence testing, which aims at testing an interval hypothesis instead of a precise hypothesis. An interval hypothesis includes a small range of parameter values instead of a single null value and the idea goes back to Hodges and Lehmann. As researchers can always expect to observe some (although often negligibly small) effect size, interval hypotheses are more realistic for biomedical research. However, the selection of an equivalence region (the interval boundaries) often seems arbitrary and several Bayesian approaches to equivalence testing coexist.

Methods: A new proposal is made how to determine the equivalence region for Bayesian equivalence tests based on objective criteria like type I error rate and power. Existing approaches to Bayesian equivalence testing in the two-sample setting are discussed with a focus on the Bayes factor and the region of practical equivalence (ROPE). A simulation study derives the necessary results to make use of the new method in the two-sample setting, which is among the most frequently carried out procedures in biomedical research.

Results: Bayesian Hodges-Lehmann tests for statistical equivalence differ in their sensitivity to the prior modeling, power, and the associated type I error rates. The relationship between type I error rates, power and sample sizes for existing Bayesian equivalence tests is identified in the two-sample setting. Results allow to determine the equivalence region based on the new method by incorporating such objective criteria. Importantly, results show that not only can prior selection influence the type I error rate and power, but the relationship is even reverse for the Bayes factor and ROPE based equivalence tests.

Conclusion: Based on the results, researchers can select between the existing Bayesian Hodges-Lehmann tests for statistical equivalence and determine the equivalence region based on objective criteria, thus improving the reproducibility of biomedical research.

Keywords: Bayes factor; Bayesian Biostatistics; Bayesian equivalence testing; Bayesian testing; Region of practical equivalence (ROPE); Student’s t-test.

Conflict of interest statement

The author declares that he has no competing interests.

© 2021. The Author(s).

Figures

Fig. 1

Influence of sample size n on the type I error rate attained by Bayesian equivalence approaches based on the Bayes factor (left) and the ROPE (right); the default equivalence region R=[−0.1,0.1] is used in all settings

Fig. 2

Power analysis for the Bayesian equivalence testing approaches based on the Bayes factor for small, medium and large effect sizes

Fig. 3

Power analysis for the Bayesian equivalence testing approaches based on the ROPE for an underlying small effect size

Fig. 4

Power analysis for the Bayesian equivalence testing approaches based on the ROPE for an underlying medium effect size

Fig. 5

Power analysis for the Bayesian equivalence testing approaches based on the ROPE for an underlying large effect size

Fig. 6

Influence of the equivalence region on the type I error rates for the Bayesian equivalence testing approaches based on the Bayes factor

Fig. 7

Influence of the equivalence region on the type I error rates for the Bayesian equivalence testing approaches based on the ROPE

Fig. 8

Total error rates for the Bayesian equivalence testing approaches based on the Bayes factor for small, medium and large effect size

Fig. 9

Total error rates for the Bayesian equivalence testing approaches based on the ROPE for an underlying large effect size

Fig. 10

Total error rates for the Bayesian equivalence testing approaches based on the ROPE for an underlying medium effect size

Fig. 11

Total error rates for the Bayesian equivalence testing approaches based on the ROPE for an underlying small effect size

Fig. 12

Differences between precise frequentist hypothesis testing and equivalence testing: Standard NHST for a sharp point null hypothesis H0:θ=0 against its alternative H1:θ≠0 (top left) or, in general, of H0:θ=θ0 against H1:θ≠θ0 (top right); TOST procedure for testing H0:θ<δL orθ>δU (shown in red) against H1:δL≤θ≤δU (shown in blue) (bottom left) or H0:θ<θ0−δL orθ>θ0+δU (shown in red) against H1:θ0−δL≤θ≤θ0+δU (shown in blue) (bottom right)