SCORE Study Report 8: Closed Tests for All Pair-Wise Comparisons of Means

Abstract

We compare five closed tests for strong control of family-wide type I error (FWE) while making all pair-wise comparisons of means in clinical trials with multiple arms such as the SCORE Study. We simulated outcomes of the SCORE Study under its design hypotheses, and used p-values from chi-squared tests to compare performance of a "pairwise" closed test described below to Bonferroni and Hochberg adjusted p-values. "Pairwise" closed testing was more powerful than Hochberg's method by several definitions of multiple-test power. Simulations over a wider parameter space, and considering other closed methods, confirmed this superiority for p-values based on normal, logistic, and Poisson distributions. The power benefit of "pair-wise" closed testing begins to disappear with 5 or more arms, and with unbalanced designs. For trials with 4 or fewer arms and balanced designs, investigators should consider using "pair-wise" closed testing in preference to Shaffer's, Hommel's, and Hochberg's approaches when making all pairwise comparisons of means. If not all p-values from the closed family are available, Shaffer's method is a good choice.

Figures

Figure 1

Implication graph for “pair-wise” closed testing of all pair-wise differences between three means. Solid lines connect unadjusted p-values whose maximum gives the adjusted p-value for elementary hypothesis 1 2.

Figure 2

Implication graph for “pair-wise” closed testing of all pair-wise differences between four means. Solid lines connect unadjusted p-values whose maximum gives the adjusted p-value for elementary hypothesis 1 2.

Figure 3

FWE for Normal, Balanced, Halves

Figure 4

Power for Normal, Balanced, Halves