A protocol of a cross-sectional study evaluating an online tool for early career peer reviewers assessing reports of randomised controlled trials

Anthony Chauvin, David Moher, Doug Altman, David L Schriger, Sabina Alam, Sally Hopewell, Daniel R Shanahan, Alessandro Recchioni, Philippe Ravaud, Isabelle Boutron, Anthony Chauvin, David Moher, Doug Altman, David L Schriger, Sabina Alam, Sally Hopewell, Daniel R Shanahan, Alessandro Recchioni, Philippe Ravaud, Isabelle Boutron

Abstract

Introduction: Systematic reviews evaluating the impact of interventions to improve the quality of peer review for biomedical publications highlighted that interventions were limited and have little impact. This study aims to compare the accuracy of early career peer reviewers who use an innovative online tool to the usual peer reviewer process in evaluating the completeness of reporting and switched primary outcomes in completed reports.

Methods and analysis: This is a cross-sectional study of individual two-arm parallel-group randomised controlled trials (RCTs) published in the BioMed Central series medical journals, BMJ, BMJ Open and Annals of Emergency Medicine and indexed with the publication type 'Randomised Controlled Trial'. First, we will develop an online tool and training module based (a) on the Consolidated Standards of Reporting Trials (CONSORT) 2010 checklist and the Explanation and Elaboration document that would be dedicated to junior peer reviewers for assessing the completeness of reporting of key items and (b) the Centre for Evidence-Based Medicine Outcome Monitoring Project process used to identify switched outcomes in completed reports of the primary results of RCTs when initially submitted. Then, we will compare the performance of early career peer reviewers who use the online tool to the usual peer review process in identifying inadequate reporting and switched outcomes in completed reports of RCTs at initial journal submission. The primary outcome will be the mean number of items accurately classified per manuscript. The secondary outcomes will be the mean number of items accurately classified per manuscript for the CONSORT items and the sensitivity, specificity and likelihood ratio to detect the item as adequately reported and to identify a switch in outcomes. We aim to include 120 RCTs and 120 early career peer reviewers.

Ethics and dissemination: The research protocol was approved by the ethics committee of the INSERM Institutional Review Board (21 January 2016). The study is based on voluntary participation and informed written consent.

Trial registration number: NCT03119376.

Keywords: peer reviewers; randomized controlled trials; reporting.

Conflict of interest statement

Competing interests: None declared.

© Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

Figures

Figure 1
Figure 1
The study design.

References

    1. Kassirer JP, Campion EW. Peer review. Crude and understudied, but indispensable. JAMA 1994;272:96–7.
    1. Jefferson T, Rudin M, Brodney Folse S, et al. . Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev 2007:MR000016 10.1002/14651858.MR000016.pub3
    1. Rennie R. Editorial peer review: its development and rationale : Godlee F, Jefferson T, Peer review in health sciences. Second edition London: BMJ Books, 2003:1-–13.
    1. Public Library of Science. “Peer review—optimizing practices for online scholarly communication,” in Peer Review in Scientific Publications, Eighth Report of Session 2010–2012, Vol. I: Report, Together with Formal, Minutes, Oral and Written Evidence, eds House of Commons Science and Technology Committee, editor. London: The Stationery Office Limited, 2011:p21–p22.
    1. Public Library of Science. “Peer review—optimizing practices for online scholarly communication,” in Peer Review in Scientific Publications, Eighth Report of Session 2010–2012, Vol. I: Report, Together with Formal, Minutes, Oral and Written Evidence, eds House of Commons Science and Technology Committee, editor. London: The Stationery Office Limited, 2011:p174–p178.
    1. Association of American Publishers. Digital licenses replace print prices as accurate reflection of real journal costs, 2012. (Accessed 08 june 2016).
    1. Ghimire S, Kyung E, Kang W, et al. . Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals. Trials 2012;13:77 10.1186/1745-6215-13-77
    1. Boutron I, Dutton S, Ravaud P, et al. . Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 2010;303:2058–64. 10.1001/jama.2010.651
    1. Hopewell S, Collins GS, Boutron I, et al. . Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ 2014;349:g4145.
    1. Bohannon J. Who's afraid of peer review? Science 2013;342:60–5. 10.1126/science.342.6154.60
    1. Schroter S, Black N, Evans S, et al. . What errors do peer reviewers detect and does training improve their ability to detect them? . J R Soc Med 2008;101:507–14.
    1. Walbot V. Are we training pit bulls to review our manuscripts? J Biol 2009;8:24 10.1186/jbiol125
    1. Lazarus C, Haneef R, Ravaud P, et al. . Peer reviewers identified spin in manuscripts of nonrandomized studies assessing therapeutic interventions, but their impact on spin in abstract conclusions was limited. J Clin Epidemiol 2016;77:44–51. 10.1016/j.jclinepi.2016.04.012
    1. Stahel PF, Moore EE. Peer review for biomedical publications: we can improve the system. BMC Med 2014;12:179 10.1186/s12916-014-0179-1
    1. Bruce R, Chauvin A, Trinquart L, et al. . Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Med 2016;14:85 10.1186/s12916-016-0631-5
    1. Galipeau J, Moher D, Campbell C, et al. . A systematic review highlights a knowledge gap regarding the effectiveness of health-related training programs in journalology. J Clin Epidemiol 2015;68:257–65. 10.1016/j.jclinepi.2014.09.024
    1. Chauvin A, Ravaud P, Baron G, et al. . The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors. BMC Med 2015;13:158 10.1186/s12916-015-0395-3
    1. . Tracking switched outcomes in clinical trials. (accessed 2 May 2016).
    1. Munafò M. Metascience: reproducibility blues. Nature 2017;543:619–20.
    1. Chan AW, Hróbjartsson A, Haahr MT, et al. . Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457–65. 10.1001/jama.291.20.2457
    1. Moher D, Hopewell S, Schulz KF, et al. . CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869.
    1. Mathieu S, Chan AW, Ravaud P. Use of trial register information during the peer review process. PLoS One 2013;8:e59910 10.1371/journal.pone.0059910
    1. Barnes C, Boutron I, Giraudeau B, et al. . Impact of an online writing aid tool for writing a randomized trial report: the COBWEB (Consort-based WEB tool) randomized controlled trial. BMC Med 2015;13:221 10.1186/s12916-015-0460-y
    1. Ioannidis JP, Evans SJ, Gøtzsche PC, et al. . Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141:781–8.
    1. Chan AW, Tetzlaff JM, Altman DG, et al. . SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med 2013;158:200–7. 10.7326/0003-4819-158-3-201302050-00583
    1. Yordanov Y, Dechartres A, Porcher R, et al. . Avoidable waste of research related to inadequate methods in clinical trials. BMJ 2015;350:h809.
    1. Dwan K, Altman DG, Clarke M, et al. . Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Med 2014;11:e1001666 10.1371/journal.pmed.1001666
    1. Page MJ, Shamseer L, Altman DG, et al. . Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med 2016;13:e1002028 10.1371/journal.pmed.1002028

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe