Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study

Anthony Chauvin, Philippe Ravaud, David Moher, David Schriger, Sally Hopewell, Daniel Shanahan, Sabina Alam, Gabriel Baron, Jean-Philippe Regnaux, Perrine Crequit, Valeria Martinez, Carolina Riveros, Laurence Le Cleach, Alessandro Recchioni, Douglas G Altman, Isabelle Boutron, Anthony Chauvin, Philippe Ravaud, David Moher, David Schriger, Sally Hopewell, Daniel Shanahan, Sabina Alam, Gabriel Baron, Jean-Philippe Regnaux, Perrine Crequit, Valeria Martinez, Carolina Riveros, Laurence Le Cleach, Alessandro Recchioni, Douglas G Altman, Isabelle Boutron

Abstract

Background: The peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process.

Methods: We performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9).

Results: The mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal's usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88-1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82-89] versus 20% [16-24] and in identifying a switch in primary outcome 61% [44-77] versus 11% [3-26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57-65] versus 77% [74-81] and to identify a switch in primary outcome 77% [67-86] versus 98% [92-100].

Conclusions: Trained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting.

Trial registration: Clinical.Trials.gov NCT03119376 (Registered April, 18, 2017).

Keywords: CONSORT statement; Peer reviewers; Randomized controlled trials; Reporting.

Conflict of interest statement

The authors have completed the ICMJE uniform disclosure form. Author AR is the Senior Editor of BMC Medicine and thus recused himself from the handling of this article at this journal. All other authors declare that they have no competing interests.

Figures

Fig. 1
Fig. 1
Example of the CONSORT-based peer-review tool (COBPeer)
Fig. 2
Fig. 2
Example of the CONSORT-based peer-review tool (COBPeer)
Fig. 3
Fig. 3
Proportions of items evaluated by early career reviewers and usual peer review classified as true positive (TP), false negative (FN), true negative (TN), and false positive (FP)

References

    1. Kassirer JP, Campion EW. Peer review. Crude and understudied, but indispensable. JAMA. 1994;272(2):96–97. doi: 10.1001/jama.1994.03520020022005.
    1. Rennie D. Editorial peer review in biomedical publication: the first international congress. JAMA. 1990;263(10):1317. doi: 10.1001/jama.1990.03440100011001.
    1. Ghimire S, Kyung E, Kang W, et al. Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals. Trials. 2012;13(1):77. doi: 10.1186/1745-6215-13-77.
    1. Boutron I, Dutton S, Ravaud P, et al. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058–2064. doi: 10.1001/jama.2010.651.
    1. Hopewell S, Collins GS, Boutron I, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ. 2014;349:g4145. doi: 10.1136/bmj.g4145.
    1. Bohannon J. Who’s afraid of peer review? Science. 2013;342(6154):60–65. doi: 10.1126/science.342.6154.60.
    1. Schroter S, Black N, Evans S, et al. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med. 2008;101(10):507–514. doi: 10.1258/jrsm.2008.080062.
    1. Walbot V. Are we training pit bulls to review our manuscripts? J Biol. 2009;8(3):24. doi: 10.1186/jbiol125.
    1. Chauvin A, Ravaud P, Baron G, et al. The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors. BMC Med. 2015;13(1):158. doi: 10.1186/s12916-015-0395-3.
    1. Mathieu S, Chan A-W, Ravaud P. Use of trial register information during the peer review process. PLoS One. 2013;8(4):e59910. doi: 10.1371/journal.pone.0059910.
    1. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. doi: 10.1136/bmj.h5527.
    1. Chauvin A, Moher D, Altman D, et al. A protocol of a cross-sectional study evaluating an online tool for early career peer reviewers assessing reports of randomised controlled trials. BMJ Open. 2017;7(9):e017462. doi: 10.1136/bmjopen-2017-017462.
    1. Barnes C, Boutron I, Giraudeau B, et al. Impact of an online writing aid tool for writing a randomized trial report: the COBWEB (Consort-based WEB tool) randomized controlled trial. BMC Med. 2015;13(1):221. doi: 10.1186/s12916-015-0460-y.
    1. Moher D, Schulz KF, Altman DG, for the CONSORT Group* The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Ann Intern Med. 2001;134(8):657. doi: 10.7326/0003-4819-134-8-200104170-00011.
    1. Goldacre B, Drysdale H, Dale A, et al. COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials. 2019;20(1):118. doi: 10.1186/s13063-019-3173-2.
    1. COBPeer training module. . Accessed 24 Sept 2019.
    1. COBPeer module. . Accessed 24 Sept 2019.
    1. O'Brien, R.G., Muller, K.E. Applied analysis of variance in behavioral science Marcel Dekker, New York (1993) Chapter 8 pp. 297–344.
    1. Collett D. Modelling Binary Data, texts in statistical science 2nd ed. Boca Raton: Chapman & Hall/CRC; 1999. pp. 24.
    1. Simel DL, Samsa GP, Matchar DB. Likelihood ratios with confidence: sample size estimation for diagnostic test studies. J Clin Epidemiol. 1991;44(8):763–770. doi: 10.1016/0895-4356(91)90128-V.
    1. epiR: tools for the analysis of epidemiological data. 2018. Available from: . Accessed 1 Dec 2018.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe