Ignoring Criticism and Peer Review

In peer review, part of the idea is you (the article author) don’t get to just arbitrarily ignore the criticisms of the reviewer. You’re supposed to address the issues in some way.

But then once you publish, you mostly get to arbitrarily ignore your critics. The peer reviewers have a special status that later critics don’t.

This is sort of backwards. It’s especially public criticisms, after the article is finished, that shouldn’t be arbitrarily ignorable.

Instead of a few critics getting a special status, addressing criticism should be the norm. And instead of criticism being addressed primarily in private, it should be addressed primarily in public.

Answering public criticism is difficult to organize (what if lots of people say over one hundred total criticisms, and most of them are low quality?) but has very large upsides so it’s worth trying to do. I have some ideas about how to make it work (my writing about Paths Forward). Other people don’t seem to be working on the problem, which seems really bad even if the problem is very hard and my solutions wouldn’t work.

If people really wanted a solution to how to receive and address public criticism effectively, but they couldn’t figure it out, I’d respect that. If they tried a variety of things that didn’t work out, I’d respect that. But they broadly don’t seem to want it, which I can’t respect. They appear to be blocked by their bad values and irrationality, not by the difficulty of the problem. And the problem might turn out to be a lot easier than people think if they’d actually try. People seem to use it being hard as an excuse; if it were easy they’d presumably find a different excuse.

I could understand if people disagreed with my solutions and thought they weren’t going to work. I haven’t really received serious criticism refuting my solutions. But my point is, what seems really bad to me is people who reject my solutions while not working on any alternatives. If you don’t have a better idea, why not either try to create one or try some of my ideas? It seems like many people want there to be no solutions so they don’t have to address criticism from the public.

Even if only a few people (designated peer reviewers with special privileges) get their criticisms answered (in some manner, even if it’s not very good), that is the most important part of the debate about the ideas. The part where criticism is addressed instead of ignored is the key part of the debate. Having that key part take place in private means people can’t see how good the key part of the critical discussion is, and therefore don’t have the information required to judge it. If it is good, they can’t learn from it, or use it as an example to model their own debates after. When people hide the information you need to make an informed judgment, you should be distrustful and demand transparency, rather than trusting their expertise.

We the public should be able to read what sort of vigorous criticism of papers is happening, or not, from their peers. Just seeing final papers, but not the critical discussion, makes it much harder to judge how good some research is. Also, it’s common that people see flaws in papers and are told basically “It’s peer reviewed; if that was a real issue, peer reviewers would have caught it.”. People often ignore criticisms, without knowing a rebuttal or finding any rebuttal written in public, on the assumption that it was considered and appropriately addressed privately. Peer review makes that problem much worse.

The manner in which authors address criticisms from reviewers is also important. Sometimes they may do a good job and sometimes a terrible job. It’d be nice to know which authors are which. It’d also be nice to see the general patterns of what most authors are doing. Are conceptual disagreements usually being rationally resolved, or are people usually just unable to agree, so they compromise on some wording changes, or what? That’s really important to evaluating academia and various experts, so it shouldn’t be hidden. I have some guesses, but we should be able to read it for ourselves instead of guessing. I suspect they have some flaws to hide. I think people commonly make overly generous assumptions about hidden information instead of taking a harshly negative view due to the lack of transparency.