Effectiveness of Anonymization in Double-Blind Review

[Indigophoton]

Peer review is a cornerstone of the academic publication process but can be subject to the flaws of the humans who perform it. Evidence suggests subconscious biases influence one's ability to objectively evaluate work: In a controlled experiment with two disjoint program committees, the ACM International Conference on Web Search and Data Mining (WSDM'17) found that reviewers with author information were 1.76x more likely to recommend acceptance of papers from famous authors, and 1.67x more likely to recommend acceptance of papers from top institutions.6

A study of three years of the Evolution of Languages conference (2012, 2014, and 2016) found that, when reviewers knew author identities, review scores for papers with male-first authors were 19% higher, and for papers with female-first authors 4% lower.4 In a medical discipline, U.S. reviewers were more likely to recommend acceptance of papers from U.S.-based institutions.2

These biases can affect anyone, regardless of the evaluator's race and gender.3 Luckily, double-blind review can mitigate these effects1,2,6 and reduce the perception of bias,5 making it a constructive step toward a review system that objectively evaluates papers based strictly on the quality of the work.

https://m.cacm.acm.org/magazines/20...anonymization-in-double-blind-review/fulltext

 

[Jonathan Edwards]

I prefer double unblinded review and publication of the review and reviewer attached to the paper. Then if the reviewer says anything stupid or irrelevant or vindictive it is apparent for all to see. PLOS One already has something a bit like this.

 

[BruceInOz]

I prefer double unblinded review and publication of the review and reviewer attached to the paper. Then if the reviewer says anything stupid or irrelevant or vindictive it is apparent for all to see. PLOS One already has something a bit like this.

Wouldn't this effectively preclude young researchers from reviewing papers by senior, influential authors in their field? The knowledge that those authors may be on a future job selection panel would undoubtedly impact what the young reviewer felt able to say. My memory of being asked to review during my first postdoc is that I was glad of the anonymity.

My feeling is that the review should stand on it's merits. If the review is unsupportable the authors will challenge it and an editorial review of the review should see it for what it is, perhaps eventually leading to that reviewers services not being required in the future.

I understand your concerns but that solution could put young reviewers in an untenable position. How would one then learn the art of reviewing?

 

[Jonathan Edwards]

I understand your concerns but that solution could put young reviewers in an untenable position. How would one then learn the art of reviewing?

I don't think there is an 'art of reviewing'. Reviewing is just giving an honest opinion. It is not unreasonable that senior people are asked to review if you have a system as at present. My idea is that you do not have formal prepublication review. You post your paper on your university website and then it is open to anyone to comment on. If junior people are worried about upsetting people they need not post comments.

The problem with the current system is that most editors are pretty dumb and do not see where reviewers are being unreasonable. The whole thing has been a shambles for decades.

 

[Sean]

Who is reviewing the reviewers?

 

[Jonathan Edwards]

Nobody

 

[Woolie]

Wouldn't this effectively preclude young researchers from reviewing papers by senior, influential authors in their field? The knowledge that those authors may be on a future job selection panel would undoubtedly impact what the young reviewer felt able to say. My memory of being asked to review during my first postdoc is that I was glad of the anonymity.

My feeling is that the review should stand on it's merits. If the review is unsupportable the authors will challenge it and an editorial review of the review should see it for what it is, perhaps eventually leading to that reviewers services not being required in the future.

I understand your concerns but that solution could put young reviewers in an untenable position. How would one then learn the art of reviewing?

I agree there are still unresolved issues. The main problem is that its hard to get anyone to review at all, there's a major reviewer recruitment crisis. And I certainly don't think making reviews unblinded will help with that. Quite the opposite.

 

[Sean]

It's a fiendish problem for science.

Are reviewers going to have to be paid? Do we in effect need a class of professional reviewers?

 

[Woolie]

It's a fiendish problem for science.

Are reviewers going to have to be paid? Do we in effect need a class of professional reviewers?

Well, that's not ideal either. Having payment associated with it will create its own problems. Reviewers might be motivated to be tougher or gentler if they think it will get them more future gigs. That sort of thing.