Bryan Appleyard sums up a study by Berkley psychologist Philip Tetlock:
He studied pundits and discovered they were, to a rough approximation, always wrong when making predictions. He took 284 pundits and asked them questions about the future. Their performance was worse than chance. With three possible answers, they were right less than 33 per cent of the time. A monkey chucking darts would have done better.
Part of the problem arises from belief perseverance. Once we form a theory, we tend to stick to it, even in the face of contradictory information.
Low scoring forecasters exhibited that behavior in Tetlock’s study: they latched onto one big idea and blinded themselves to other drivers of history. Think about someone who peppers their speech with “moreover” and “all the more so,” continually stretching their idea and denying other interpretations. This over-reaching makes them prone to exaggeration, such as those who thought Quebec would secede and Canada would consequently disintegrate.
High scoring forecasters, on the other hand, recognized, even flaunted, a probability distribution of events. These are the people who qualify their arguments with “however” and “but,” those who accommodate opposing theories. They stitch together a framework of smaller ideas, rather then working down from a Grand Scheme.