The trim-and-fill method for publication bias: practical guidelines and recommendations based on a large database of meta-analyses

Abstract

Publication bias is a type of systematic error when synthesizing evidence that cannot represent the underlying truth. Clinical studies with favorable results are more likely published and thus exaggerate the synthesized evidence in meta-analyses. The trim-and-fill method is a popular tool to detect and adjust for publication bias. Simulation studies have been performed to assess this method, but they may not fully represent realistic settings about publication bias. Based on real-world meta-analyses, this article provides practical guidelines and recommendations for using the trim-and-fill method. We used a worked illustrative example to demonstrate the idea of the trim-and-fill method, and we reviewed three estimators (R0, L0, and Q0) for imputing missing studies. A resampling method was proposed to calculate P values for all 3 estimators. We also summarized available meta-analysis software programs for implementing the trim-and-fill method. Moreover, we applied the method to 29,932 meta-analyses from the Cochrane Database of Systematic Reviews, and empirically evaluated its overall performance. We carefully explored potential issues occurred in our analysis. The estimators L0 and Q0 detected at least one missing study in more meta-analyses than R0, while Q0 often imputed more missing studies than L0. After adding imputed missing studies, the significance of heterogeneity and overall effect sizes changed in many meta-analyses. All estimators generally converged fast. However, L0 and Q0 failed to converge in a few meta-analyses that contained studies with identical effect sizes. Also, P values produced by different estimators could yield different conclusions of publication bias significance. Outliers and the pre-specified direction of missing studies could have influential impact on the trim-and-fill results. Meta-analysts are recommended to perform the trim-and-fill method with great caution when using meta-analysis software programs. Some default settings (e.g., the choice of estimators and the direction of missing studies) in the programs may not be optimal for a certain meta-analysis; they should be determined on a case-by-case basis. Sensitivity analyses are encouraged to examine effects of different estimators and outlying studies. Also, the trim-and-fill estimator should be routinely reported in meta-analyses, because the results depend highly on it.

Conflict of interest statement

The authors have no conflicts of interest to disclose.

Figures

Figure 1

Histogram of publications that used the trim-and-fill method from 2000 to 2018.

Figure 2

Funnel plots of the meta-analysis by Andersen et al[26] before (panel A) and after (panel B) applying the trim-and-fill method. The closed dots indicate the observed studies, and the open dots indicate the missing studies imputed by the trim-and-fill method (based on the estimator L0). The dashed lines that create a triangular area indicate the 95% confidence limits (under the fixed-effect setting), and the vertical solid line represents the overall effect size.

Figure 3

Histograms of the estimated number of missing studies (upper panels) and the number of iterations to achieve the convergence of the trim-and-fill algorithm (lower panels) based on the three estimators R0, L0 and Q0 among Cochrane meta-analyses with binary outcomes.

Figure 4

Histograms of the estimated number of missing studies (upper panels) and the number of iterations to achieve the convergence of the trim-and-fill algorithm (lower panels) based on the three estimators R0, L0 and Q0 among Cochrane meta-analyses with non-binary outcomes.