Yes or No Philosophy and Score Systems
Ideas don’t have degrees of goodness.
An idea is meant to solve a problem or, in other words, accomplish a goal. An idea either would succeed at its goal or it wouldn’t. The goal can be intellectual, e.g. answering a question or understanding an issue. It can also be more physical, e.g. getting $1,000,000, getting dinner, or getting married.
Rather than consider how good an idea is, we should consider if it works or does not work. This is a binary distinction rather than a matter of degree. There are only two conclusions to choose from. (You could also say something like “I don’t know” – you could refrain from reaching a conclusion.)
People think ideas/solutions partly work. This is due to vaguely defined problems/goals. A well-defined problem/goal specifies what constitutes success and what constitutes failure. Then, if it’s unambiguous, any outcome is one or the other.
What about partial credit for failures which are less severe? E.g. it’s better to fail at a project while wasting $1 than to fail at it while wasting $100.
Those failures are differentiated in terms of other problems/goals/objectives. If the problem is simply to achieve the goal, they both failed. If the problem is to try to achieve the goal while, in the case of failure, keeping losses under $50, then one approach failed and one succeeded.
In order to differentiate two ideas (so one is better than another), one must always specify a problem for which one is a success and the other a failure. One must always be able to draw a binary distinction between the two ideas in terms of some well-defined problem. Any preference for one idea over another can be translated into binary terms or else is incorrect. Putting it in binary terms clarifies it and better exposes it to criticism.
Fundamentally, we choose some ideas over others. We can’t get away from that. And it’s a binary choice. We accept some ideas and reject others. When we don’t put our reasoning in binary terms, our reasoning doesn’t correspond to reality as well.
There isn’t just one problem that matters. People often would like to solve problems A, B and C. Then what? Solving three problems is a single multi-part problem. Any group of problems can be treated as one complex problem.
What about partial credit for solving more problems? Say you list 100 problems and you want the idea which solves the most out of those 100 (they’re all equally important). Then an idea which solves 77 problems is a failure if you know of any idea which solves 80, because solving 77 is not the most. In this case, a tie is possible and all the tied ideas would be different and equal solutions (it’d take considering a different problem to differentiate them).
What if some problems on the list are more important than others? You can list them along with their point value, then say that whatever idea gets the highest point score is what you want. In that case, any idea with a lower point score than the max is simply refuted. If that’s not actually what you wanted – if you didn’t want to just disregard any option with a lower score – then you defined your problem/goal wrong.
What we often do, which works well, is use point scores as rough indicators. It’s very hard to do point scores in an exact or rigorous way to correspond exactly to what we want. But we can gather a list of solutions with high point scores – plus ignore the point scores to also include on the list a few other ideas with some kind of merit the point system failed to capture. The point scores help us find some candidates for the list that we might have otherwise missed. Then we differentiate the remaining solutions with binary distinctions – we come up with reasons to reject some, e.g. we decide that quality X is really important (and that there are good options which have it) and so reject all the ones without it. We have to come up with some way of deciding which makes hard distinctions.
If you try to use scores to the bitter end, instead of as initial approximations for exploratory brainstorming, you will think less effectively. Because, in the end, you’re picking one solution and rejecting the rest. You should do this due to reasons you came up with for picking it and not the others - clear differentiators. If you stick to scores, in the end what you say is “I will do this because it scored 92 while these other five ideas scored only in the 80-91 range.” Then you don’t actually know why you’re picking the one with a 92. It may not be the best at anything. It may also have a glaring flaw and not be the most well-rounded either. And it might be benefitting from an imperfection in the scoring system (such imperfections are, in general, in complex situations, unavoidable – we can make rough score systems but not exact ones). You should actually think through what your criteria for accepting and rejecting ideas are instead of dealing with it only indirectly via a scoring system.
Score systems try to convert qualitatively different factors into the same units (points, which sometimes also correspond to dollars or some other concrete unit). And they weight the factors, normally in a linear way. That can’t be done perfectly. It doesn’t perfectly correspond to people’s actual preferences which are routinely non-linear and involve complex interplays between factors.
It’s good to make decisions based on a small number of key factors that you understand. If there is a bunch of numerical analysis of factors which you want to take into account, make that, as a whole, one factor. You can be like “This idea scored 3% higher on the computer factor analysis spreadsheet thing. That’s my first reason for preferring it. Everything else being equal, and with no major reason to go against the numbers, I might as well use them. Additionally, I used my judgment to look at the key issues. I found this idea has no decisive flaws. Everything bad about it is a matter of degree like costing a bit more in some area or taking a bit longer for some parts. There’s no non-optional part with a fundamental flaw. A few optional parts just won’t work at all, but that’s OK, they were optional.”
I’ve been talking about complicated or hard projects. For simple and easy projects, it’s all the same but simpler and lower stakes and the problems are defined less aggressively (a lot more will satisfy us and constitute a solution). Problem definitions generally set lower bars for minor things that we don’t want to put work into optimizing. And then if we find ten solutions that all work, we can just pick one arbitrarily (or however else) rather than come up with some good way to choose between them. It’s OK if lots of solutions all succeed. We don’t need the “best” one. We don’t usually need to come up with tiebreakers. A working solution works. Success is success. We can move on and do something else. The case where we want to tiebreak and optimize between viable solutions is the minority case. For most stuff we just want any solution and to be done. Even using the highest scoring candidate on some rough metric is fine if the top five are all successful solutions of the problem and we don’t care about picking the best one, we just want to quickly pick anything that works.