Choosing an imbalance metric for covariate-constrained randomization in multiple-arm cluster-randomized trials

Abstract

Background: In cluster-randomized controlled trials (C-RCTs), covariate-constrained randomization (CCR) methods efficiently control imbalance in multiple baseline cluster-level variables, but the choice of imbalance metric to define the subset of "adequately balanced" possible allocation schemes for C-RCTs involving more than two arms and continuous variables is unclear. In an ongoing three-armed C-RCT, we chose the min(three Kruskal-Wallis [KW] test P values) > 0.30 as our metric. We use simulation studies to explore the performance of this and other metrics of baseline variable imbalance in CCR.

Methods: We simulated three continuous variables across three arms under varying allocation ratios and assumptions. We compared the performance of min(analysis of variance [ANOVA] P value) > 0.30, min(KW P value) > 0.30, multivariate analysis of variance (MANOVA) P value > 0.30, min(nine possible t test P values) > 0.30, and min(Wilcoxon rank-sum [WRS] P values) > 0.30.

Results: Pairwise comparison metrics (t test and WRS) tended to be the most conservative, providing the smallest subset of allocation schemes (10%-13%) meeting criteria for acceptable balance. Sensitivity of the min(t test P values) > 0.30 for detecting non-trivial imbalance was 100% for both hypothetical and resampled simulation scenarios. The KW criterion maintained higher sensitivity than both the MANOVA and ANOVA criteria (89% to over 99%) but was not as sensitive as pairwise criteria.

Conclusions: Our criterion, the KW P value > 0.30, to signify "acceptable" balance was not the most conservative, but it appropriately identified imbalance in the majority of simulations. Since all are related, CCR algorithms involving any of these imbalance metrics for continuous baseline variables will ensure robust simultaneous control over multiple continuous baseline variables, but we recommend care in determining the threshold of "acceptable" levels of (im)balance.

Trial registration: This trial is registered on ClinicalTrials.gov (initial post: December 1, 2016; identifier: NCT02979444 ).

Keywords: Cluster randomization; Cluster-randomized controlled trial; Continuous covariate; Covariate-constrained randomization; Imbalance.

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1

Distribution of site-level randomization variables by arm for trial NCT02979444. We planned for a total of 42 randomized sites (6:18:18), but owing to dropout we have 38 active sites. Sample size and power considerations accounted for dropout that we observed. Medians (interquartile ranges) are displayed in each arm for each variable above

Fig. 2

Maximum pairwise imbalance observed for scenarios meeting adequacy threshold (P > 0.30) in simulated trials by criterion. Each panel represents the distribution of the max(mean difference) for simulated scenarios meeting the criterion for “adequate” overall variable balance across arms. All simulated schemes meet criteria for adequate under simple random allocation (panel a), but the remaining panels illustrate only those allocation schemes meeting the P > 0.30 criterion for each metric. The mean difference depicted is on the standard deviation unit scale. Those meeting this criterion would ideally have a small max(mean difference), and we deem a max(mean difference) > 1.0 (vertical line) unacceptable since previously a value of 0.8 would be deemed “large” [16]

Fig. 3

Pairwise scatterplots exploring associations between Kruskal–Wallis (KW) P value with other measures. The panels here present a selection of pairwise plots to illustrate the relationships between the imbalance metric we used in our randomization algorithm, the KW test P value, and additional candidate imbalance metrics explored. Each plot includes 5000 observations from the resampled scenarios using 1:3:3 allocation as in our present study. In each plot, there is often a non-linear relationship. For example, the first plot illustrating min(KW P value) in comparison with the multivariate analysis of variance (MANOVA) demonstrates a somewhat noisy relationship between the two whereby the min(KW P value) tends to be lower than the overall MANOVA P value, but the two are related. The comparison of the min(KW P value) versus the min(Wilcoxon rank-sum [WRS] P value) shows a more pronounced relationship and a clearer, non-linear pattern. All metrics of imbalance as determined are highly related; broadly, the more global tests (e.g., MANOVA) tended to be less conservative (i.e., have larger P value) than the corresponding more specific tests based on more than one comparison (e.g., WRS). The line y = x has been added for reference