Testing for the Presence of Authorship Bias in Peer Review

April 14, 2016 updated by: Seth Leopold, University of Washington

No consensus exists among biomedical journals on the subject of blinding during the peer review process. Some journals attempt to remove all identifiers of authorship from potential manuscripts before delivering to referees for peer review while others prefer to leave the authorship transparent. Although peer review remains the gold standard for manuscript evaluation, the lack of a standardized blinding process between editorial offices may be a source of publication bias and make comparisons of manuscripts published in different journals more difficult to interpret.

Proponents of a blinded peer review system believe that knowledge of authorship may leave referees vulnerable to biases about those authors' previous research, perceived expertise, institution, nationality, or gender. This shifts judgment away from the merits of the scientific work and introduces pre-conceived notions about the identity and background of the author into the review process. Conversely, supporters of transparent authorship argue that knowledge of author identity makes it easier for the referee to provide more appropriate critiques. Open author identification allows referees to compare the current manuscript to previously published work by the author, and to recognize or identify potentially important conflicts of interest.

The investigators therefore propose an experimental study to address the question of whether blinding affects the likelihood a reviewer will recommend acceptance of a study being peer reviewed. The investigators plan to work in partnership with a journal to have a fabricated manuscript peer reviewed by a large number of reviewers; the authors will send this "test" manuscript out either in blinded form (authors' names/institutions not shown to reviewers) or in open form, with the names of several well-known, well-regarded authors and their institutions visible to peer reviewers. The manuscripts will otherwise be identical.

The partnering journal, Clinical Orthopaedics and Related Research allows both open and blinded peer review, regularly employing both approaches (authors can choose what kind of review they prefer when they submit to CORR), and so the reviewers are unlikely to be troubled by seeing a manuscript in either format.

The primary goal of this study is to determine whether the identification of a manuscript with a prominent group of authors (open authorship) will affect the likelihood that reviewers will recommend the manuscript for publication.

Additionally, the investigators plan to purposely place several errors in the manuscript, and tally how often reviewers detect these mistakes, to determine whether reviewers read the work of prominent authors less critically.

The investigators will also compare the reviewers' grading of the (identical) methods section, to determine whether the identification of a manuscript with prominent authors results in higher reviewer grades for methodological rigor.

This study tests the following specific hypotheses:

  1. The visibility of prominent author names and institutions on test manuscripts will be associated with increased likelihood that the manuscript will be recommended for publication by peer reviewers. (Primary Study Endpoint)
  2. The visibility of prominent author names and institutions will be associated with a decreased likelihood that purposefully placed "errors" in the experimental manuscript will be detected by the reviewers.
  3. The visibility of prominent author names and institutions will be associated with increased scores given by reviewers for the methods section, despite the fact that the methods sections of the experimental manuscript will be identical.

NOTE:

All elements of the research protocol were approved by the University of Washington Human Subjects Review Committee, including the study's opt-out design for reviewer recruitment, since it was considered important that reviewers not know that the paper they evaluated was part of a study. To minimize the potential for harm, the authors chose as the topic of the fabricated test manuscript an intervention that could not be immediately put into practice at reviewers' own centers (a particular team-training process in the operating room). In addition, participating reviewers were informed which paper was the test manuscript at the completion of data collection, so that they did not pursue this sort of team training based on the results of the fabricated study.

To reduce the risk of an observer (Hawthorne) effect, it was important to minimize the likelihood that participant reviewers would learn of the study while it was in progress. To this end, the investigators elected not to pre-register the study in a trial registry.

Study Overview

Status

Completed

Conditions

Intervention / Treatment

Study Type

Interventional

Enrollment (Actual)

119

Phase

  • Not Applicable

Participation Criteria

Researchers look for people who fit a certain description, called eligibility criteria. Some examples of these criteria are a person's general health condition or prior treatments.

Eligibility Criteria

Ages Eligible for Study

30 years and older (Adult, Older Adult)

Accepts Healthy Volunteers

Yes

Genders Eligible for Study

All

Description

Inclusion Criteria:

  • Registered peer reviewers in relevant subspecialty sections of CORR's reviewer database.

Exclusion Criteria:

  • Reviewers who opt out (electronically, by phone, or by post).

Study Plan

This section provides details of the study plan, including how the study is designed and what the study is measuring.

How is the study designed?

Design Details

  • Primary Purpose: Health Services Research
  • Allocation: Randomized
  • Interventional Model: Parallel Assignment
  • Masking: Double

Arms and Interventions

Participant Group / Arm
Intervention / Treatment
Experimental: Blinded Reviewers
Subjects receiving randomized sham manuscript for review
Other: Blinded manuscript
Blinded version of sham manuscript
Experimental: Unblinded Reviewers
Subjects receiving randomized sham manuscript for review
Other: Unblinded manuscript
Unblinded version of sham manuscript

What is the study measuring?

Primary Outcome Measures

Outcome Measure
Time Frame
Manuscript acceptance rate
Time Frame: 13 months
13 months

Secondary Outcome Measures

Outcome Measure
Time Frame
Error detection rate
Time Frame: 13 months
13 months
Methods section score
Time Frame: 13 months
13 months

Collaborators and Investigators

This is where you will find people and organizations involved with this study.

Sponsor

University of Washington

Investigators

  • Principal Investigator: Seth L Leopold, MD, University of Washington

Publications and helpful links

The person responsible for entering information about the study voluntarily provides these publications. These may be about anything related to the study.

General Publications

Study record dates

These dates track the progress of study record and summary results submissions to ClinicalTrials.gov. Study records and reported results are reviewed by the National Library of Medicine (NLM) to make sure they meet specific quality control standards before being posted on the public website.

Study Major Dates

Study Start

December 1, 2013

Primary Completion (Actual)

August 1, 2015

Study Completion (Actual)

August 1, 2015

Study Registration Dates

First Submitted

April 4, 2016

First Submitted That Met QC Criteria

April 14, 2016

First Posted (Estimate)

April 15, 2016

Study Record Updates

Last Update Posted (Estimate)

April 15, 2016

Last Update Submitted That Met QC Criteria

April 14, 2016

Last Verified

April 1, 2016

More Information

Terms related to this study

Keywords

Other Study ID Numbers

  • 46129-EA

Plan for Individual participant data (IPD)

Plan to Share Individual Participant Data (IPD)?

NO

This information was retrieved directly from the website clinicaltrials.gov without any changes. If you have any requests to change, remove or update your study details, please contact register@clinicaltrials.gov. As soon as a change is implemented on clinicaltrials.gov, this will be updated automatically on our website as well.

Clinical Trials on Peer Review

Clinical Trials on Blinded manuscript

Search Similar Trials

Sponsors and Collaborators

Medical Conditions

Drug Interventions

CROs by country

CROs in Kosovo

Conditions

Rare Diseases

Drug Interventions

Dietary Supplements

Sponsor/Collaborators

Locations

Subscribe