The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method

Joanna IntHout, John P A Ioannidis, George F Borm, Joanna IntHout, John P A Ioannidis, George F Borm

Abstract

Background: The DerSimonian and Laird approach (DL) is widely used for random effects meta-analysis, but this often results in inappropriate type I error rates. The method described by Hartung, Knapp, Sidik and Jonkman (HKSJ) is known to perform better when trials of similar size are combined. However evidence in realistic situations, where one trial might be much larger than the other trials, is lacking. We aimed to evaluate the relative performance of the DL and HKSJ methods when studies of different sizes are combined and to develop a simple method to convert DL results to HKSJ results.

Methods: We evaluated the performance of the HKSJ versus DL approach in simulated meta-analyses of 2-20 trials with varying sample sizes and between-study heterogeneity, and allowing trials to have various sizes, e.g. 25% of the trials being 10-times larger than the smaller trials. We also compared the number of "positive" (statistically significant at p < 0.05) findings using empirical data of recent meta-analyses with > = 3 studies of interventions from the Cochrane Database of Systematic Reviews.

Results: The simulations showed that the HKSJ method consistently resulted in more adequate error rates than the DL method. When the significance level was 5%, the HKSJ error rates at most doubled, whereas for DL they could be over 30%. DL, and, far less so, HKSJ had more inflated error rates when the combined studies had unequal sizes and between-study heterogeneity. The empirical data from 689 meta-analyses showed that 25.1% of the significant findings for the DL method were non-significant with the HKSJ method. DL results can be easily converted into HKSJ results.

Conclusions: Our simulations showed that the HKSJ method consistently results in more adequate error rates than the DL method, especially when the number of studies is small, and can easily be applied routinely in meta-analyses. Even with the HKSJ method, extra caution is needed when there are = <5 studies of very unequal sizes.

Figures

Figure 1
Figure 1
DerSimonian-Laird and Hartung-Knapp-Sidik-Jonkman error rates for continuous outcomes, for various I2 and mixtures of trial sizes. Legend: A: Equally sized trials; B: One small trial, 1/10th of other trials; C: 50–50 small and large trials (ratio 1:10); D: one large trial (10 times larger than other trials). Vertical bars refer to the minimum and maximum error rates over the group sizes. The lines connect the means of these error rates. DL: DerSimonian-Laird meta-analysis method. HKSJ: Hartung-Knapp-Sidik-Jonkman meta-analysis method.

References

    1. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–188. doi: 10.1016/0197-2456(86)90046-2.
    1. The Cochrane Collaboration. Review Manager (RevMan) 5.1.4. Copenhagen: The Nordic Cochrane Centre; 2011.
    1. Borenstein M, Hedges L, Higgins J, Rothstein H. Comprehensive Meta-analysis Version 2. Englewood NJ: Biostat; 2005.
    1. Hartung J. An alternative method for meta-analysis. Biom J. 1999. pp. 901–916.
    1. Hartung J, Knapp G. A refined method for the meta analysis of controlled clinical trials with binary outcome. Stat Med. 2001;20(24):3875–3889. doi: 10.1002/sim.1009.
    1. Hartung J, Knapp G. On tests of the overall treatment effect in meta analysis with normally distributed responses. Stat Med. 2001;20(12):1771–1782. doi: 10.1002/sim.791.
    1. Follmann DA, Proschan MA. Valid inference in random effects meta-analysis. Biometrics. 1999;55(3):732–737. doi: 10.1111/j.0006-341X.1999.00732.x.
    1. Hartung J, Makambi KH. Reducing the number of unjustified significant results in meta-analysis. Commun Stat Simul Comput. 2003;32(4):1179–1190. doi: 10.1081/SAC-120023884.
    1. Makambi KH. The effect of the heterogeneity variance estimator on some tests of treatment efficacy. J Biopharm Stat. 2004;14(2):439–449. doi: 10.1081/BIP-120037191.
    1. Sidik K, Jonkman JN. Robust variance estimation for random effects meta-analysis. Comput Stat Data Anal. 2006;50(12):3681–3701. doi: 10.1016/j.csda.2005.07.019.
    1. Sidik K, Jonkman JN. A simple confidence interval for meta-analysis. Stat Med. 2002;21(21):3153–3159. doi: 10.1002/sim.1262.
    1. Sidik K, Jonkman JN. On constructing confidence intervals for a standardized mean difference in meta-analysis. Commun Stat Simul Comput. 2003;32(4):1191–1203. doi: 10.1081/SAC-120023885.
    1. Sánchez-Meca J, Marín-Martínez F. Confidence intervals for the overall effect size in random-effects meta-analysis. Psychol Meth. 2008;13(1):31.
    1. Sidik K, Jonkman JN. Simple heterogeneity variance estimation for meta analysis. J Roy Stat Soc. 2005;54(2):367–384. doi: 10.1111/j.1467-9876.2005.00489.x.
    1. Davey J, Turner R, Clarke M, Higgins J. Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis. BMC Med Res Methodol. 2011;11(1):160. doi: 10.1186/1471-2288-11-160.
    1. Higgins J, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. Bmj. 2003;327(7414):557. doi: 10.1136/bmj.327.7414.557.
    1. Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses. Bmj. 2007;335(7626):914–916. doi: 10.1136/bmj.39343.408449.80.
    1. IntHout J, Ioannidis JP, Borm GF. Obtaining evidence by a single well-powered trial or several modestly powered trials. Stat Methods Med Res. 2012. [Epub ahead of print]
    1. Borm GF, Lemmers O, Fransen J, Donders R. The evidence provided by a single trial is less reliable than its statistical analysis suggests. J Clin Epidemiol. 2009;62(7):711–715. doi: 10.1016/j.jclinepi.2008.09.013. e711.
    1. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3):1–48.
    1. Harbord RM, Higgins JP. Meta-regression in Stata. Meta. 2008;8(4):493–519.
    1. Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–1558. doi: 10.1002/sim.1186.
    1. The Cochrane Collaboration. .
    1. Singh M, Das RR. Zinc for the common cold. Cochrane Database Syst Rev. 2011;2:CD001364.
    1. Pidala J, Djulbegovic B, Anasetti C, Kharfan‒Dabaja M, Kumar A. Allogeneic hematopoietic cell transplantation for acute lymphoblastic leukemia (ALL) in first complete remission. Cochrane Library. 2011;10:CD008818.
    1. Thorlund K, Wetterslev J, Awad T, Thabane L, Gluud C. Comparison of statistical inferences from the DerSimonian–Laird and alternative random-effects model meta-analyses – an empirical assessment of 920 Cochrane primary outcome meta-analyses. Res Synth Meth. 2011;2(4):238–253. doi: 10.1002/jrsm.53.
    1. Kontopantelis E, Reeves D. Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: a simulation study. Stat Methods Med Res. 2012;21(4):409–426. doi: 10.1177/0962280210392008.
    1. Brockwell SE, Gordon IR. A simple method for inference on an overall effect in meta-analysis. Stat Med. 2007;26(25):4531–4543. doi: 10.1002/sim.2883.
    1. Hardy RJ, Thompson SG. A likelihood approach to meta-analysis with random effects. Stat Med. 1996;15(6):619–629. doi: 10.1002/(SICI)1097-0258(19960330)15:6<619::AID-SIM188>;2-A.
    1. Shuster JJ. Empirical vs natural weighting in random effects meta-analysis. Stat Med. 2010;29(12):1259–1265.
    1. Henmi M, Copas JB. Confidence intervals for random effects meta analysis and robustness to publication bias. Stat Med. 2010;29(29):2969–2983. doi: 10.1002/sim.4029.
    1. Guolo A. Higher-order likelihood inference in meta-analysis and meta-regression. Stat Med. 2012;31(4):313–327. doi: 10.1002/sim.4451.
    1. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. Chichester, UK: Wiley; 2009.
    1. Knapp G, Hartung J. Improved tests for a random effects meta-regression with a single covariate. Stat Med. 2003;22:2693–2710. doi: 10.1002/sim.1482.

Source: PubMed

Patrocinadores y Colaboradores

Condiciones médicas

Intervenciones de drogas

3
Suscribir