Paths Forward Could Replace Some Peer Review

Peer review lacks transparency. Paths Forward commonly involves rational, critical discussions on publicly-readable internet forums. That enables transparency and some other advantages like wider participation.

Article authors could be expected to address criticisms. This is like how people have recommended a norm of sharing raw data along with articles. I think it being standard to make raw data available would help. Giving counter-arguments to criticisms, post-publication, could also be a norm. If you don’t manage to address at least ten objections, maybe your article shouldn’t really be considered finished and people shouldn’t yet do other research based on it. Note that similar objections which involve the same conceptual issue should be addressed as a group, rather than addressing them individually to inflate the number of objections addressed.

What if no one happens to pay attention to an article so it doesn’t get much criticism? A few policies could address this. First, people who read or analyze an article but don’t come up with any criticisms could be expected to say that, so an article that’s flawless doesn’t appear ignored. Second, the forum management (basically the equivalent of journal editors) could assign some reviewers to each article to make sure it gets some attention. They should probably wait to see which articles haven’t received adequate criticism after a month, then assign people to those articles, to avoid unnecessary assignments for articles that would get enough attention anyway. Third, reviewers could have a culture of taking initiative to read articles, especially articles that aren’t getting enough review. Fourth, authors could be expected to attract reviewers, possibly by paying them. There could also be a system where authors could earn reviewing attention by doing reviews. It’s important that authors are not expected to attract reviewers by being popular and doing social climbing and social networking. We don’t want a popularity contest or a system that rewards high social status.

In general, if lots of people don’t pay much attention to something on the basis that it’s bad, wrong or low quality, but none of them will actually write any critical arguments, then they are a group of irrational haters. People who dislike something should be able to give or cite criticisms, and shouldn’t be satisfied with no written public criticism (that they consider correct) of something they think is bad.

I suggested articles receive and address ten objections before being treated as finished work with a conclusion anyone should accept yet. What if an article is so good that people struggle to come up with any objections? That’s unrealistic. People often make incorrect objections, so correctness shouldn’t really stop an article from receiving objections.

What if an article (or its author or conclusion) is so popular that no one wants to oppose it? People should be praised and socially rewarded for criticizing something popular that no one else criticized. We should have a culture that’s impressed by criticism in cases where everyone else was struggling to criticize (or choosing not to), rather than punishing critics as heretics. Also, we shouldn’t assume that if you criticize a point in an article that means you oppose the article’s conclusion. Critics can also say what they agree with before or after their criticisms, and be really explicit about which parts they’re criticizing.

If strong efforts aren’t enough to get any criticism, then just getting a bunch of experts to sign their name to say “I couldn’t come up with any criticism of this article” could be accepted as a substitute for addressing objections.

All this analysis mostly applies for articles making positive and potentially controversial claims which say something new. If an article criticizes another article, then soliciting criticism is less needed. The authors of the original article – as well as everyone who agrees with that article – can answer criticism of their position (or, in the alternative, change their minds to be unsure or to reject their prior beliefs).

In some fields, like math, some results (that come with formal proofs) bring less controversy and criticism than is typical of other fields like science. Controversy is also more common in the softer sciences than the hard sciences. Philosophy has an especially high amount of controversy – or in other words, it’s a field that people are especially bad at approaching objectively. Whereas they can approach math and the hard sciences more objectively. But, despite some differences, there’s plenty of room for and need for critical discussion in all fields. People could find more ways to critically discuss results in more objective fields if they valued critical discussion more. Questioning premises or potential applications or implications (that weren’t discussed in the paper) are some ways to open up more issues for critical analysis even when it’s hard to find errors directly in the text of a paper. In other words, an inter-disciplinary look at how claims relate to other fields (prior fields being built on or later fields being built to) can yield more criticism.