Advocating Public Peer Review
I criticize peer review for lacking transparency. Then I talked about friendly, private editing, which is fine, and is usually used for papers before the peer review process. Private editing is fine because no one is asked to trust that it ensured high quality. It isn’t an attempt to have the important critical discussions in private.
Private editing doesn’t replace public, critical discussion. Peer review does replace a lot of public, critical discussion. It’s uncommon for academic journals to publish much debate or criticism of specific articles. I think, in a better world, the majority of academic papers would be written with the primary purpose of criticizing one or more other papers. Criticism is the key to rationality and papers should have to survive whatever criticism people can come up with.
Private, non-transparent peer review (as well as various informal, private discussions between people in the field) is in some ways the main process for criticism. I think that’s really bad. The main criticism process should be publicly readable and transparent.
Peer reviewed articles receive friendly private editing before being submitted for peer review. That’s fine. Then the peer review is an additional layer of editing that’s supposed to be critical, not friendly. But critical discussion is important to truth-seeking and should generally be public and transparent.
Letting anyone in the public participate (write criticisms) has upsides but also downsides. It’s hard to organize a discussion if hundreds of people want to participate. Letting anyone in the public read the critical discussions, by putting them on the internet, lacks that downside.
The downsides of doing peer review in public seem more about potentially harming the prestige of academia and revealing some academics as foolish, not about rationality. If academics are fragile, so they need a safer space to receive some criticism before they face the world with their idea, fine, but don’t count that as the main critical review or debate – follow it with publicly readable debate as the main place for important criticism. If authors can fix most of their errors before they get to the public part, OK, sure, good for them, but there should be publicly readable criticism for the primary critical discussion. We shouldn’t trust that the private, safe-space discussion involved such comprehensive, serious criticism that there’s no need for further criticism.
If the final, public version of something hasn’t received critical discussion in public, and you can’t read and be persuaded by responses to the criticisms, then don’t trust it. Even if they don’t let you participate in the critical discussion, they should at least let you read it and judge its quality. And if you think of some criticisms yourself, you should be able to check if any critics raised those criticisms already and got a response, or not.
I think peer review isn’t very good and uses privacy to hide how bad it is. The only people with much familiarity with the low quality of peer review are generally people who have invested many years of their life in academia. They are self-selected to be people who think academia is good, they have large career incentives to keep thinking that, and they are immersed in an environment with other academics who think academia is good (somewhat of an echo chamber).