Triple-masking v Double-masking: a Trial of Scientific Publication in Public Health (TripleMasking)

The trial is designed to determine whether knowledge of the identity of the authors, their institutions and of the reviewers of a given article submitted to American Journal of Public Health impacts the final editorial decision. The concept of triple masked editorial process (where editors are unaware of author identities and affiliations in addition to the authors' and reviewers' identities being masked from one another) compared to a double masked process (only authors' and reviewers' identities are hidden from one another; editors are aware of author and reviewer identities) has not been formally tested in a scientific journal, and particularly not in medicals social science, or public health journal. Triple-masking is expected to lead to greater acceptance of articles submitted overall because it will neutralize biases against some authors and reviewers because of who they are or the institutions they are related to.

The triple masked editorial process, in which the editors, the authors and the reviewers ignore their respective identities will be compared to a double masked, in which the editor knows the identity of the authors and reviewers, because double masked is currently the editorial process used by the American Journal of Public Health.

Even though only manuscripts are randomized, the trial will collect information about the identity of the authors (eg, genes, race/ethnicity, seniority) and the reviewers to be used for secondary analysis. In this sense it is dealing with human subjects and has obtained an exemption from the Institutional Review Board of Queens College.

Study Overview

• Status: Recruiting
• Conditions: Publication of Articles Submitted to the American Journal of Public Health
• Intervention / Treatment: Other: Triple-Masked; Other: Usual Procedures

Detailed Description

Aim:

To determine whether masking (i.e., triple-masked: editors, authors, reviewers) or not (i.e., double-masked: authors, reviewers) the identity, seniority, gender, race, ethnicity, and institutions of the authors and of the reviewers to the editors leads to a greater proportion of accepted manuscripts compared with double-masked review. Additionally, we will conduct exploratory analyses to determine whether the differences are related to the identity, seniority, gender, race, ethnicity, and institutions of the authors and of the reviewers.

Trial design: two group, parallel, superiority trial with 1:1 allocation

Study setting: The trial will solely take place at the American Journal of Public Health in a virtual capacity. All submissions will be electronic via the Editorial Manager (https://www.editorialmanager.com/ajph/default2.aspx). Eligible manuscripts in the blinded editorial process arm will be masked electronically before being entered in the trial by the editorial assistant, who will also place all manuscripts in Dropbox folders to be accessed by the editors. The questionnaire that will be sent to all authors, reviewers, and editors is housed in Survey Monkey and will be administered via email and/or text messaging by someone external to the trial.

Inclusions: All eligible, new submissions (research, notes from the field, non-commissioned editorials, analytic essays, systematic reviews) for the regular issue and supplements from September 5, 2023 to March 1, 2024.

Exclusions: Commissioned editorials, book reviews, Letters to the Editor, Public Health of Consequence editorials, Editor's Choice, and re-submissions

Randomization procedure: 1:1 allocation of eligible articles to either triple-masked or usual process in the order of submission, after exclusion of non-eligible submissions by the editorial assistant.

Planned generation: Randomization, stratified by three article types (ie, Research Articles, Analytic Essay, Editorial & Notes from the Field), by blocks of random sizes to balance articles by type over time, and to conceal allocation order.

Administration/implementation: Editorial assistant allocates the submissions following computer generated allocation orders for each of the three article types.

Masking securities: Editors will be restricted from accessing the Editorial Manager for the duration of the project. Additionally, since articles won't be assigned to editors, they won't be able to see anything from their immediate accounts. This should block them from the masked information and preserve the integrity of the project. Editors will be able to search for reviewers with specific expertise from an external, cleaned nominative version of the Editorial Manager database. They won't however know who, from the reviewers selected, accepted the invitation and wrote the review. In this sense, reviews' authorship will be masked to the editor.

Procedures:

1. Questionnaire: All authors of new, eligible submissions will receive an email from a research assistant external to the trial that will explain the on-going trial. The email will invite them to fill out an optional, short, Survey Monkey questionnaire asking for age, gender, race, ethnicity, career position, and institution location. The questionnaire data will be collected in a stand-alone database, handled by the research assistant. Missing value are expected to be equally distributed in both arms of the study. However, for secondary analysis they may be correlated with the outcome. The same questionnaire will be sent but only once to the editors.

Primary outcome: Difference in the proportion of manuscripts submitted within the six-month window that are accepted for publication (i.e., not rejected at any stage); risk difference, unadjusted

Secondary outcome: Difference in the proportion of manuscripts submitted within the six-month window that are sent for peer review (i.e., not desk rejected); risk difference, unadjusted

Proportion accepted in the double-masked group: ~10% Clinically meaningful difference: 5% Framework: Superiority N= 1500 (750 manuscripts in each group), 80% power to detect a difference of 4.8% (RR=1.48) in acceptance rate with 5% type I error for a two-sided test. This translates to an acceptance rate of 14.8% for the triple=masked. Additionally, the trial is projected to have 84% power to detect a difference of 5% (RR=1.5), and 90% power to detect a difference of 5.5% (RR=1.55) while remaining a 5% type I error for a two-sided test.

The statistical power to detect differences if they exist in subgroups defined by article type or authors identities will be very small because of the small numbers of articles. It is expected that the proportion of articles with a diverse authorship will not exceed one third of the submissions in each groups.

Ethics committee The protocol has been exempted from institutional review board by the IRB of Queens College CUNY, where the editor in chief and PI of the supporting grant has his primary appointment (Appendix B).

• Study Type: Interventional
• Enrollment (Estimated): 1500
• Phase: Not Applicable

Contacts and Locations

• Study Contact: Name: ALFREDO MORABIA; Phone Number: 9176648789; Email: amorabia@qc.cuny.edu
• Study Contact Backup: Name: ALFREDO MORABIA; Email: amorabia@qc.cuny.edu

Study Locations

• United States
  • New York
    • New York, New York, United States, 11365
      • Recruiting
      • QueensC
      • Contact: ALFREDO MORABIA; Email: amorabia@qc.cuny.edu

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: No

Description

Inclusion Criteria:

• New submissions
• Article formats: research articles, notes from the field, non-commissioned editorials, analytic essays, systematic reviews

Exclusion Criteria:

• Resubmissions
• Article formats: Commissioned editorials, book reviews, Letters to the Editor, Public Health of Consequence editorials, Editor's Choice.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Other
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Triple

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Triple-Masked; For articles allocated to the blinded (masked) editorial process, the editorial assistant will delete identifiable/unmasking content of cover page and letters. All articles will be flagged as being in triplemasked editorial process arm in the document title	Other: Triple-Masked; Eligible manuscripts in the triple-masked editorial process arm will be masked electronically before being entered in the trial by the editorial assistant
Active Comparator: Usual Procedures; All articles will be flagged as being in usual editorial process arm in the document title. For articles allocated to the usual (double-masked) editorial process, the editorial assistant will leave identifiable/unmasking content of cover page and letters for the editors.	Other: Usual Procedures; Articles will be processed by the editors with identifiable/unmasking content of cover page and letters.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Difference in the proportion of manuscripts submitted within the six-month window that are accepted for publication (i.e., not rejected at any stage); risk difference, unadjusted	Primary outcome	full duration of the study, 6 months, from September 5 2023 to March 1 2024

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Secondary outcome	Difference in the proportion of manuscripts submitted within the six-month window that are sent for peer review (i.e., not desk rejected); risk difference, unadjusted	full duration of the study, 6 months, from September 5 2023 to March 1 2024

Collaborators and Investigators

• Sponsor: Queens College, The City University of New York
• Investigators: Principal Investigator: ALFREDO MORABIA, Queens College, The City University of New York

Publications and helpful links

General Publications

• Brodie S, Frainer A, Pennino MG, Jiang S, Kaikkonen L, Lopez J, Ortega-Cisneros K, Peters CA, Selim SA, Vaidianu N. Equity in science: advocating for a triple-blind review system. Trends Ecol Evol. 2021 Nov;36(11):957-959. doi: 10.1016/j.tree.2021.07.011. Epub 2021 Aug 26.

Study record dates

Study Major Dates

• Study Start (Actual): September 6, 2023
• Primary Completion (Estimated): March 1, 2024
• Study Completion (Estimated): March 1, 2024

Study Registration Dates

• First Submitted: September 21, 2023
• First Submitted That Met QC Criteria: October 2, 2023
• First Posted (Actual): October 5, 2023

Study Record Updates

• Last Update Posted (Actual): October 5, 2023
• Last Update Submitted That Met QC Criteria: October 2, 2023
• Last Verified: October 1, 2023

More Information

Terms related to this study

• Other Study ID Numbers: 2023-0516-QC

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: Only anonymized and any re-identifiable information will be shared.

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No