Impact of adding a limitations section to abstracts of systematic reviews on readers' interpretation: a randomized controlled trial

Amélie Yavchitz, Philippe Ravaud, Sally Hopewell, Gabriel Baron, Isabelle Boutron, Amélie Yavchitz, Philippe Ravaud, Sally Hopewell, Gabriel Baron, Isabelle Boutron

Abstract

Background: To allow an accurate evaluation of abstracts of systematic reviews, the PRISMA Statement recommends that the limitations of the evidence (e.g., risk of bias, publication bias, inconsistency, imprecision) should be described in the abstract. We aimed to evaluate the impact of adding such limitations sections on reader's interpretation.

Method: We performed a two-arm parallel group randomized controlled trial (RCT) using a sample of 30 abstracts of systematic reviews evaluating the effects of healthcare intervention with conclusions favoring the beneficial effect of the experimental treatments. Two formats of these abstracts were derived: one reported without and one with a standardized limitations section written according to the PRISMA statement for abstracts. The primary outcome was readers' confidence in the results of the systematic review as stated in the abstract assessed by a Likert scale from 0, not at all confident, to 10, very confident. In total, 300 participants (corresponding authors of RCT reports indexed in PubMed) were randomized by a web-based randomization procedure to interpret one abstract with a limitations section (n = 150) or without a limitations section (n = 150). Participants were blinded to the study hypothesis.

Results: Adding a limitations section did not modify readers' interpretation of findings in terms of confidence in the results (mean difference [95% confidence interval] 0.19 [-0.37-0.74], p = 0.50), confidence in the validity of the conclusions (0.07 [-0.49-0.62], p = 0.80), or benefit of the experimental intervention (0.12 [-0.42-0.44], p = 0.65).This study is limited because the participants were expert-readers and are not representative of all systematic review readers.

Conclusion: Adding a limitations section to abstracts of systematic reviews did not affect readers' interpretation of the abstract results. Other studies are needed to confirm the results and explore the impact of a limitations section on a less expert panel of participants.

Trial registration: ClinicalTrial.gov (NCT01848782).

Figures

Figure 1
Figure 1
Flow chart of participants in the trial.

References

    1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–72. doi: 10.1136/bmj.312.7023.71.
    1. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7:e1000326. doi: 10.1371/journal.pmed.1000326.
    1. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126:376–380. doi: 10.7326/0003-4819-126-5-199703010-00006.
    1. Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med. 1997;127:37–42. doi: 10.7326/0003-4819-127-1-199707010-00007.
    1. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. doi: 10.1186/1471-2288-7-10.
    1. Saint S, Christakis DA, Saha S, Elmore JG, Welsh DE, Baker P, Koepsell TD. Journal reading habits of internists. J Gen Intern Med. 2000;15:881–884. doi: 10.1046/j.1525-1497.2000.00202.x.
    1. Scherer RW, Langenberg P, von Elm E: Full publication of results initially presented in abstracts.Cochrane Database Syst Rev 2007, (2):MR000005.
    1. The PLoS Medicine Editors The impact of open access upon public health. PLoS Med. 2006;3:e252. doi: 10.1371/journal.pmed.0030252.
    1. Hopewell S, Clarke M, Askie L. Reporting of trials presented in conference abstracts needs to be improved. J Clin Epidemiol. 2006;59:681–684. doi: 10.1016/j.jclinepi.2005.09.016.
    1. Beller EM, Glasziou PP, Hopewell S, Altman DG. Reporting of effect direction and size in abstracts of systematic reviews. JAMA. 2011;306:1981–1982. doi: 10.1001/jama.2011.1620.
    1. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700. doi: 10.1136/bmj.b2700.
    1. Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, Gøtzsche PC, Lasserson T, Tovey D. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013;10:e1001419. doi: 10.1371/journal.pmed.1001419.
    1. Hopewell S, Boutron I, Altman DG, Ravaud P. Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. 2013;3:e003342.
    1. The Editors Addressing the limitations of structured abstracts. Ann Intern Med. 2004;140:480–481. doi: 10.7326/0003-4819-140-6-200403160-00015.
    1. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol. 2010;63:834–840. doi: 10.1016/j.jclinepi.2010.02.005.
    1. Guyatt GH. GrADe: what is “quality of evidence” and why is it important to clinicians? BMJ. 2008;336:995–998. doi: 10.1136/.
    1. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Sch\ünemann HJ. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924. doi: 10.1136/.
    1. Soffer A. Abstracts of clinical investigations. A new and standardized format. Chest. 1987;92:389–390. doi: 10.1378/chest.92.3.389.
    1. Huth EJ. Structured abstracts for papers reporting clinical trials. Ann Intern Med. 1987;106:626. doi: 10.7326/0003-4819-106-4-626.
    1. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, Schulz KF. CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLoS Med. 2008;5:e20. doi: 10.1371/journal.pmed.0050020.
    1. Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors’ implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178. doi: 10.1136/bmj.e4178.
    1. Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA. 2001;285:437–443. doi: 10.1001/jama.285.4.437.
    1. Yank V, Rennie D, Bero LA. Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ. 2007;335:1202–1205. doi: 10.1136/.
    1. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303:2058–2064. doi: 10.1001/jama.2010.651.
    1. Kesselheim AS, Robertson CT, Myers JA, Rose SL, Gillet V, Ross KM, Glynn RJ, Joffe S, Avorn J. A randomized study of How physicians interpret research funding disclosures. N Engl J Med. 2012;367:1119–1127. doi: 10.1056/NEJMsa1202397.
    1. Bucher HC, Weinbacher M, Gyr K. Influence of method of reporting study results on decision of physicians to prescribe drugs to lower cholesterol concentration. BMJ. 1994;309:761–764. doi: 10.1136/bmj.309.6957.761.
    1. Forrow L, Taylor WC, Arnold RM. Absolutely relative: how research results are summarized can affect treatment decisions. Am J Med. 1992;92:121–124. doi: 10.1016/0002-9343(92)90100-P.
Pre-publication history
    1. The pre-publication history for this paper can be accessed here:

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe