Objective Judgment, Chess Competition and How Science Is Failing

Summary: I discuss how the impact of bias is limited in chess competition. Then I discuss how bias is a bigger factor in philosophy and science. Then I discuss what chess is doing better and propose a solution for philosophy and science.

Bias in Chess

There is bias in the chess world, just like in approximately all fields. But I don't think any player who is in the top 20 in the world at chess exists but is unknown due to bias. I don't think someone like that gets missed or ignored. Chess has too many reasonably fair opportunities to enter fairly high level tournaments, win games, win the tournament, get a high rating, etc. And if you do that successfully enough, you will eventually get some opportunities to play in some top tournaments even if tournament organizers are biased against you. The severity and consequences of chess bias is limited.

There could be an exception for a really open, explicit bias. For example, a Russian player is being excluded from tournaments for supporting Russia's war on Ukraine. But no one credible claims he isn't a strong chess player; they just think he's immoral. (There are other Russian players who aren't excluded because they didn't make public statements favoring the war of aggression. Also players have to be able to travel to other countries to compete in tournaments, which is sometimes a practical issue but overall doesn't ruin chess competition. And there are online tournaments, though the most prestigious tournaments take place in person.)

There are also exceptions for cheaters. If you cheat but you're also really good, you could end up punished and excluded anyway, even if you stop cheating.

An interesting case was when the world's best player accused a player of cheating, but the player wasn't caught red handed. Overall, the evidence could be called ambiguous and circumstantial. This created a large bias against the possible cheater and cost him invitations to some tournaments. But despite this bias, he's still playing in a lot of tournaments, just not the most elite tournaments. He wasn't banned from the chess community because there was no proof that he cheated; there's just an unusually large bias against him now. Since he can still play in many tournaments, he has the opportunity to win them and get a higher rating. If he did well enough, he'd get more chances at more elite tournaments, despite the bias against him (albeit with some stricter anti-cheating precautions).

Although bias makes a significant difference, chess talent isn't massively overlooked. If you win enough, it overcomes bias. In the chess world, a top 20 player doesn't get accidentally viewed as the world's 5000th best player.

Bias in Other Fields

The situation is much worse in many other fields like medicine, philosophy or, I fear, every single science. One of the top 20 people active in the field can be rejected, mocked, and fired. They can die poor and disliked as punishment for their heretical innovations. There's widespread agreement that stuff like that has happened repeatedly in the past. Many people think we're better now, but they don't usually even try to point out specific, effective reforms. I haven't seen a rational case going over the reforms made and how and why they worked. As far as I can tell, the social hierarchies haven't fundamentally changed much and there aren't adequate precautions to prevent these kinds of biases from ruining the careers of some of the best thinkers today.

Sometimes someone is very successful on some issues but is still ignored and mocked on another issue. Being recognized as a great thinker in multiple fields isn't enough to get you a rational hearing on a separate issue. I suspect that's because rational hearings broadly aren't available, and the earlier success in other fields wasn't due to receiving a rational hearing in those cases either. (If an idea is genuinely great, and it's widely accepted and praised, that correct outcome doesn't imply that the process of acceptance was rational or good.)

Some people are recognized as great thinkers after they're dead but some aren't. What causes late recognition? Sometimes the field and its biases gradually change. If you're ahead of your time about something, and you're right, then if people eventually figure out that it's a good idea, they might notice that you wrote a book advocating it many years ago. At that point, they'll no longer be biased against you due to you having that belief because they now agree with that belief.

Difference Between Fields

Why is the situation better in chess? Because the goal of chess is very well-defined. There are exact rules for how to play chess and exact, uncontroversial rules determining who wins a game. There's no judge who decides how good you are; you're judged by whether you won, lost or drew. (Chess has tie games, called "draws".)

Some other sports also do a good job at objective rules, such as baseball or basketball. They define how you score points, how long the game last, and say the team with more points when the game ends is the winner. While they have referees to deal with some details, the referees are never tasked with deciding how good either team is or judging how well they played.

Other sports don't do as well, such as boxing or gymnastics, which have judges who score performances. But at least the judges have reasonably clearer guidelines for judging a performance than the sciences have. And at least there are people whose actual job it is to judge the performance. In science, you can figure something out and be ignored with no one in particular is assigned the job of objectively judging your work.

For philosophy, medicine and science to do better, they need more objective ways to judge which alleged discoveries are valuable. Discoveries shouldn't get or fail to get attention like a popularity contest. That'd help reduce bias and reduce the effects of social hierarchies (we want scientists judged by their work not by their social status). How can this be accomplished?

Rational Debate

I think science, medicine, philosophy and other intellectual fields would benefit massively from rational debate methodology. Debate methodology is parallel to the rules of chess, which specify the method of playing a chess game – who takes what actions when, what actions are disallowed, how to judge victory, etc.

Chess players play chess against each other to determine who is better. This works even if they hate each other because they have clear rules to follow. The rules are clear enough that even very biased audiences agree about who won a chess game. Scientists who disagree should debate using clear enough rules that the debate is productive even if they hate each other. Debate rules should be clear enough for biased audiences to agree about who won.

Fields need ways for people who disagree about the correct conclusions to debate – and for people to be able to tell who won the debate. The main issue is judging debates – if we could get it about as accurate as gymnastics, that would change the world. If we could get it as accurate as judging who won a chess game, that'd be even better. We can start with something easier, more equivalent to gymnastics judging, and if that becomes common then we'll gain experience with it and be in a better position to figure out something more advanced. I think the debate proposals I've already developed, if they were used and work as intended, would enable effective, objective debates. They are at least trying to solve that problem. I don't know of any serious proposals by anyone else that aim to solve that problem. I think people broadly aren't working on it and aren't interested in solutions to it.

Getting People to Debate

Besides solving the problem of creating rational debate methods, there's also a second problem of getting people to use the debate methods. If chess players just boasted about how great they were but didn't play chess games against each other, we wouldn't know who is a great player and who isn't. Similarly, we'd need a social expectation that anyone claiming to be great at science participate in debates (which the public can view) against people who disagree. There'd need to be some social norms about how much they're expected to debate and against who. A wide range of norms would be a huge improvement over the world today. We'd just need decent norms, not ideal norms.

In the chess world, people who are considered top players sometimes play in competitions where only top players are allowed to play. But they also play in some competitions which are open to many more players. Even if 80% of their games were in invite-only tournaments, and all the "top" players are actually bad, it would soon be discovered when they did poorly in open tournaments. Getting people to play/debate against a wide variety of non-insiders only 20% of the time is plenty to prevent a small group of insiders from only debating/playing against each other, claiming to be the best, and excluding many opponents from ever getting to play/debate.

How do you tell if judgements about winners are objective? A simple approach is to look at how controversial they are. There is approximately zero controversy about who won chess games. There is some controversy about winners in gymnastics or boxing, but I don't think it's a big problem most of the time (but I'm not very familiar with those sports, and I think my general point stands even if I'm incorrect about those particular examples). By contrast, there are ongoing, large controversies regarding political, philosophical, religious and scientific debates. For those topics, many generally smart, reasonable and well-informed people persistently and strongly disagree both about what the (current best guess at the) truth is and also about who won a particular debate between two people. So we can see that debates currently aren't objective enough.

I wrote some followup thoughts in Science Needs Rational Debate.