Article Text

Download PDFPDF

The effect of masking manuscripts for the peer-review process of an ophthalmic journal
  1. S J Isenberg1,2,
  2. E Sanchez2,
  3. K C Zafran3
  1. 1
    Jules Stein Eye Institute at the David Geffen School of Medicine at UCLA, Los Angeles, California, USA
  2. 2
    Harbor-UCLA Medical Center, Torrance, California, USA
  3. 3
    Los Angeles Biomedical Research Institute at Harbor-UCLA Medical Center, Torrance, California, USA
  1. Dr S J Isenberg, Jules Stein Eye Institute, 100 Stein Plaza, UCLA, Box 957000, Los Angeles, CA 90095-7000, USA; isenberg{at}ucla.edu

Abstract

Background/aim: To investigate the effect of masking the author’s identity to peer reviewers on reviewer bias in an ophthalmic subspecialty journal.

Design: A retrospective study of 531 manuscripts, involving 1182 masked reviews, submitted to Journal of American Association for Pediatric Ophthalmology and Strabismus from 2000 to 2005.

Methods: Data were extracted from recommendation forms completed by each referee during review. Investigated variables consisted of reviewer’s knowledge of author identity, recommendation (accept, revise, or reject publishing), eventual manuscript status (published or not), review quality, gender, country, medical practice setting (academic or private) and editorial board status.

Results: This study involved the largest number of manuscripts ever used to evaluate the importance of author masking. Reviewer’s knowledge of the author’s identity had no effect on review quality. However, proportionally fewer manuscripts were published when there was no idea of the author’s identity, compared with when it was allegedly known or suspected (p<0.0001). Manuscripts had lower recommendation scores when there was no idea of the author’s identity compared with when allegedly known (p = 0.0001) or suspected (p = 0.004).

Conclusion: Reviewers were more favourable when they allegedly knew or suspected the author’s identity. Double-masking may improve the quality of biomedical publishing or at least reduce reviewer bias for effectively masked manuscripts.

Statistics from Altmetric.com

Biomedical publications rely upon peer reviewer’s objective evaluations to publish the latest, most compelling and relevant research. The peer review process, however, has given authors and editors concerns over possible publication bias and reviewer subjectivity.1 Each journal is left to develop its own system to extract the most informed and unbiased opinions from its reviewers.

In medical publications, masking (“blinding”) can be accomplished by three different methods: author masked (the author’s name(s) are unknown to the reviewers), reviewer masked (the reviewer’s names are unknown to the author/s) and double masked (the author’s and the reviewer’s names both are unknown to each other). Despite measures to mask authors, some reviewers may still know the author’s identity based on familiarity with past work or presentations. The Journal of the American Association for Pediatric Ophthalmology and Strabismus (Journal of AAPOS) has used the double-masked method of review from its inception in 1996 and is one of the few ophthalmic publications that regularly practices this method.

McNutt and colleagues were the first to seek scientific evidence to support masked peer review.2 Their study indicated that masking improved the quality (3.5 compared with 3.1 on a five-point scale) of reviews for manuscripts submitted to a general medicine journal. Since then, various non-ophthalmic medical journals have tested the effectiveness of masking the author’s identity to reduce reviewer bias. Most studies agree that masking manuscripts at least partially reduces reviewer bias.4 5 No such study, to our knowledge, has been conducted before for a surgical subspecialty, in this case ophthalmic, journal.

In this retrospective study, we wished to investigate the effect of masking the author’s identity to peer reviewers on reviewer bias in an ophthalmic subspecialty journal. Our results might validate the importance of masking for ophthalmic and possibly other surgical subspecialty journals.

METHODS

We reviewed all manuscripts submitted to Journal of AAPOS from 2000 to 2005. Evaluated manuscripts consisted of short reports and regular research articles. A total of 531 manuscripts and 1182 reviews were evaluated for this study.

The journal’s policy, as set by the editorial board and implemented by the then current Editor-in-chief (SJI), was to double-mask all manuscripts (both author’s and reviewer’s names were unknown to each other) and excluded all identifying names or initials of authors, hospitals and universities, regional information and nationalities (as appropriate).

The journal’s review procedure dictated that each manuscript be sent to two reviewers. One reviewer was an editorial-board member; the other a reputable expert in the field not then on the editorial board. Reviewers chose one of five possible decision categories (1 = accept, 2–4 = needs revisions to various degrees, 5 = reject). If both reviewers chose any decision but “reject,” the manuscript was ultimately published, often after modifications. If both reviewers chose “reject,” the manuscript was not published. An additional review step was required if only one referee recommended rejection; then the paper was sent to a third, non-editorial board, reviewer. The manuscript was published depending on the third reviewer’s decision. The Editor-in-Chief did not individually decide whether to publish.

All study data were extracted from evaluation and recommendation forms completed by reviewers for each manuscript at the time of review. These forms included information regarding the decision of the reviewer and the reviewer’s knowledge of the author’s identity (categorised into three choices of “I know,” “Strongly suspect,” or “I have no idea”). We also analysed the country from where the manuscript was submitted, the country where the reviewer resided, the reviewer’s gender, the reviewer’s type of medical practice setting (academic or private), the editorial-board status of the reviewer (member or non-member), the status of the manuscript (published or not published) and finally the Editor-in-Chief’s unmasked rating of review quality. The quality of the review was based on a five-point scale (5 being the best and 1 worst), and was done at the time of editorial office manuscript evaluation.

Final publishing decisions were determined before the study began, which eliminated any possible Hawthorne effect.2 IRB approval to use the manuscript information of the Journal of APPOS was applied for, and any consent issues were waived by the Human Subjects Committee of the Los Angeles Biomedical Research Institute at the Harbor-UCLA Medical Center, Torrance, California.

After all data were compiled, we contacted the 30 most recent reviewers who indicated they knew or highly suspected the author’s identity to find whether their documented response on the evaluation and recommendation sheet was indeed accurate. These reviewers were given the manuscript title and asked whom they believed was the actual author of the study. Reviewers could give the name of the person or persons they suspected authored the manuscript or indicate if they could no longer remember. Those who gave a specific name or names were not told if the suspicion was correct.

Data analyses used generalised estimating equations, to account for the fact that reviewers reviewed differing numbers of manuscripts, to test for shifts in the distributions of proportions of manuscripts with decisions to accept, revise or reject, according to other characteristics as knowledge of author identity, gender, country, medical practice setting (academic or private), and editorial board status (member or non-member) of the reviewer. p Values ⩽0.05 were considered significant.3

RESULTS

Of the 531 manuscripts, 71.9% were regular original research articles, while 28.1% were short reports. A total of 61.0% of the manuscripts came from North America, and 39.0% came from outside North America.

The proportion of manuscripts published did not differ between reviews when the author’s identity was known (79.6%), compared with when it was suspected (81.0%; p = 0.80). However, manuscripts were less likely to be published (61.1%) when there was no idea of the author’s identity, compared with when it was known or suspected (p<0.0001 for both) (fig 1).

Figure 1

Reviewer’s knowledge of author identity relative to publication.

Manuscripts had lower recommendation scores when there was no idea of the author’s identity compared with when it was allegedly known (overall shift in accept, revise, reject p = 0.0001, with p = 0.19 for comparing proportions accepted and p = 0.0001 for proportions rejected) or suspected (overall shift in accept, revise, reject p = 0.004, with p = 0.02 for comparing proportions accepted and p<0.0001 for proportions rejected). No statistical difference was found between decisions and the reviewer’s gender or location (p = 0.62, 0.36, respectively).

Manuscripts were less likely to receive more favourable recommendations for publishing from academic reviewers (p = 0.007) than from reviewers in a private practice setting, with p = 0.03 for comparing proportions accepted and p = 0.02 for proportions rejected. Manuscripts were also less likely to be recommended for publishing from board members (p = 0.005) than from non-members, with p = 0.21 for comparing proportions accepted and p = 0.009 for proportions rejected (table 1).

Table 1 Reviewer’s decision and knowledge of author’s identity, reviewer gender and reviewer location

There was no significant difference in review quality between reviewers who had no idea or reviewers who allegedly knew or highly suspected the author’s identity (fig 2). Results from the 30 enquiries regarding reviewer’s recall of author’s names were 53% correct, 40% could not remember, and 7% were incorrect (fig 3).

Figure 2

Quality of reviews based on a reviewer’s alleged knowledge of the author’s identity (5 = best, 1 = worst).

Figure 3

Results from enquiries to reviewers regarding their recall of authors’ names.

Papers presented at the annual meeting of AAPOS were no more likely to have been recommended for publication than other manuscripts (p = 0.092, χ2 test).

Manuscripts were successfully author masked at a rate of 77.7%, according to the reviewers’ documented response, which reported that they had no idea about the author’s identity.

DISCUSSION

Our study found that, at least for the Journal of AAPOS, reviewers were less likely to recommend manuscripts for publication when there was no idea of the author’s identity compared with when they knew or highly suspected the author’s identity. Additionally, manuscripts were more likely to be published when reviewers allegedly knew or highly suspected the author’s identity compared with when the author was not known. This suggests that reviewer bias was present in that reviewers were less critical when they knew or suspected the author’s identity, or alternatively that investigators well known for their good research continue their good research.

A recent study to support masking of author and institution names was conducted on 67 275 abstracts submitted to the American Heart Association’s Scientific Sessions research meeting.4 This study found that unmasked abstracts were subject to reviewer bias favouring authors from the USA, as well as studies from prestigious academic institutions. They concluded that masking manuscripts at least partially reduces reviewer bias, and our findings support this conclusion.

Two prior publications also concluded that masking manuscripts reduces reviewer bias in a smaller sample size. The first found that masking the reviewer’s identity to the author improved review quality (3.1 compared with 3.5 on a five-point scale). However, this study used only 127 manuscripts and 254 reviewers.2 The second study showed that masked reviewers provided more unbiased reviews but utilised only 57 manuscripts with the same four reviewers for each manuscript.5 In seeking a large sample, our study used data from multiple years and is the largest study to evaluate the importance of masking a manuscript’s authors’ identity to reviewers (531 manuscripts involving 1182 reviews).

Other studies found no statistically significant reviewer bias, and several factors could account for the different results.68 First, these investigations studied reviewer bias using reviewer masking alone, while we utilised double masking. Another reason for the different conclusions could have been the type of journal. One study, close in size to this study, investigated articles from a large, general medicine journal (BMJ), while our study utilised a small, ophthalmic subspecialty journal (600 pages a year).6 Subspecialty journals usually involve a smaller network of colleagues, creating a greater possibility of a reviewer recognising the work of a peer. Masking may, therefore, be more imperative for smaller journals. Another study had the unusual approach of sending the same manuscript to 420 different reviewers.8 A separate study utilised a sample size of only 118 manuscripts derived from five different journals (Annals of Emergency Medicine, Annals of Internal Medicine, JAMA, Obstetrics & Gynecology and Ophthalmology).7 Each journal’s masking policy, frequency of publishing, number of manuscripts received and number of reviewers varied greatly. Therefore, it might have been inappropriate to combine these five different journals in a randomised sample.

In addition to our primary findings, we found that academic reviewers rejected manuscripts more frequently than reviewers from private practice settings. Because academic reviewers are more experienced with the peer-review process due to the greater number of manuscripts that they review, and presumably have published a larger number of articles, they may be more critical of research published in their respective field. No previous study has considered this variable, to our knowledge.

Editorial board members tended to reject manuscripts more frequently than non-editorial board members (p = 0.005). Other factors in decision-making such as gender and geographic location of reviewers did not differ statistically. This agrees with a study of JAMA’s peer review process, which found that the gender of reviewers had no apparent effect on the final outcome of the peer review process.9 Another study found that American reviewers have a preference for manuscripts from American authors.10

In our study, manuscripts were successfully author masked 77.7% of the time. The reported masking success of previous studies ranges from 58% to 76%.2 6 7 11 12 When reviewers were surveyed in our study, the main reasons cited for masking failure were that they heard the manuscript at a meeting or presentation, or that the authors often referenced themselves. Reasons for masking failure given by other studies were usually editorial-office error, a small research field, authors widely known or acclaimed, reviewer’s knowledge and investigation in similar subspecialty interests.2 6 7 11 12

There were limitations in our study that could have affected our results. First, our study was retrospective. This may have helped reduce the possibility of a Hawthorne effect raising awareness in reviewers of a potential bias when reviewing known authors.2 The reviewer’s response regarding knowledge of the author’s identity was not a validated approach. In order to help validate the reviewer’s documented response, we contacted a sample of reviewers who indicated that they knew or highly suspected the author’s identity. We found that most reviewers were correct about their assumptions. A smaller group also reported that they could no longer remember, but at the time they reviewed the paper, the author’s identity was familiar. An alternative explanation for a higher acceptance rate when the author was known or suspected could be that authors who are better known may produce higher-quality papers. Our finding that reviewer’s recommendations for papers presented at the annual AAPOS meetings (which may represent a sample of better-known authors) were no different from papers not presented there would suggest that the alternative explanation may not be correct.

Some manuscripts that were unsuccessfully masked had been presented at a meeting, and so these manuscripts had been preselected by a meeting review board, while other papers were rejected from the meeting. Hence, it is possible that reviewers who knew or suspected the authors’ identity recommended the manuscripts for publication because these manuscripts were of a higher quality anyway. To evaluate this possible limitation, we analysed manuscripts that were presented at annual AAPOS meetings and found no significant relationship between those papers and the reviewer’s recommendation for publication.

Our results were based on manuscripts submitted to a subspecialty journal and may not apply to larger, more general, journals that receive a greater number of manuscript submissions. Additionally, larger journals may have a broader network of peer reviewers lowering the chance of reviewers recognising an author’s work. Finally, we did not explore the effect of masking the reviewer (to the author) on the quality of the reviews.

In conclusion, the reviewer’s recommendations for the manuscripts were affected by their knowledge of the author’s identity. Our findings indicate that masking of authors reduces the chance of bias among reviewers for manuscripts that are effectively masked. Therefore, our study supports double-masked manuscript peer review, at least in subspecialty medical journals. This may not apply to a more generalised medical journal. We recommend that subspecialty journals evaluate their own manuscript review process and consider double-masking manuscripts to reduce reviewer bias.

Acknowledgments

We are grateful to T Guiterrez, K Hoang, P Christenson, M Del Signore and D Hunter, for their assistance.

REFERENCES

Footnotes

  • Competing interests: None.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • At a glance
    Harminder S Dua Arun D Singh