Peer Review Lacks Transparency

Who peer reviewed an academic paper? That’s secret.

More importantly, what criticisms did the peers reviewers come up with? That’s secret.

What changes were made to the paper due to peer review? That’s secret.

When your error correction process is secret, the public can’t see how good it is. They can’t see how complete or robust the criticisms were, nor what counter-arguments or concessions were made to those criticisms.

When a member of the public thinks of a criticism of a paper, they can’t tell if a peer reviewer already brought up that issue, or if the reviewers just missed it. They don’t know if the issue was analyzed or neglected. All a reader can tell is that the paper doesn’t answer the issue in a way that’s clear to him. So maybe the author has never heard of that issue, or maybe he has a great answer but prioritized including other material in the paper. (Academic publishing often has severe length limits, more so than internet publishing, which leads to leaving out useful information. And, despite all the citations, they tend not to cite many counter-arguments to preemptively address potential criticism. They mostly cite things to back up positive claims.)

Peer reviewers sometimes have access to raw data that readers of the final article don’t have access to. Many articles can’t be evaluated properly by the public due to lack of access to raw data. I think peer reviewers sometimes don’t get to see the raw data either, which is extra bad.

Peer review is, in essence, private debate. It’s like if I posted drafts of all my articles on a private forum, debated with the people I thought were smartest and had the most relevant expertise, edited my articles, and then hid all of that discussion from the public. Then I told the public that very vigorous, intelligent debate happened using lots of expertise they couldn’t understand, so just trust me on this conclusion. But no you can’t see any of that debate.

I think peer review debates often don’t actually reach conclusions, just compromises. Authors and reviewers still disagree, so some wording is changed or some citation added, which no one is fully satisfied with. And there’s no way for readers of the final article to know which points the authors or reviewers weren’t fully satisfied with – to know what is an open controversy instead of a point of full agreement.

Private debate is mostly just worse than just having that discussion history visible on a public forum which is linked in a note at the end of the paper. Even if the forum is only publicly readable, but the public can’t actually participate, that’s a big help compared to it being private.

Peer review is worse than a private forum though because even people who are frequently peer reviewers themselves still never get to see the discussion around the articles they don’t review. And even if they peer review an article, they still don’t get to see what issues other reviewers raised.

Peer review lacks transparency. If the review is biased or mediocre, the public can’t see that and point it out. Even other reviewers of other papers in the field, or the same paper, can’t see that and point it out. (Journal editors and the paper authors might be able to see all the peer reviews and get a more complete picture.)

Avoiding transparency avoids people seeing how bad your stuff is. In general, poor transparency is a red flag (a concerning indicator of potential danger or failure).

(Peer review processes vary some. They aren’t all equally good or bad, and I doubt everything I said applies in every case.)

According to publisher Elsevier (which, in their own words, “relies on the peer review process”):

Despite criticisms, peer review is still the only widely accepted method for research validation and has continued successfully with relatively minor changes for some 350 years.

Paths Forward, applied to draft articles, is a potential alternative to peer review.