It was run by an organisation called the Good Judgment Project. The GJP was set up by Philip Tetlock who, in 1984, had noticed something. The young psychologist had joined an expert committee set up by the National Academy of Sciences to help prevent nuclear war. The experts were divided. Conservatives wanted to maintain a tough line against the Soviet Union; liberals thought that line was strengthening Kremlin hardliners.
When, a few months later, Mikhail Gorbachev took office and started opening the USSR up, Tetlock was surprised to see that everyone thought this showed they’d been right all along. The conservatives thought the tough line had pushed the Kremlin into action; the liberals thought it would have happened anyway, and that the hard line had just slowed things down. People all thought they were right, whatever had actually happened.
So Tetlock set up an experiment to see how good experts were at predicting the future. First, he had to tie them to falsifiable predictions. Pundits had, and have, a tendency to vague answers that don’t really pin them to anything: “Food shortages could be likely”, and so on. Tetlock gave them specific questions with clear dates: “Will the yen be higher than it is now against the deutschmark in one month’s time?” He then asked the pundits to give percentage values for how likely that was: there is a 75% chance that this will happen, a 60% chance, etc. [...]
What he found was that pundits’ calibration, on average, was no better than random guessing – or, as Tetlock described it, “than a dart-throwing chimpanzee”. But some were significantly better. What predicted who was better was not whether they were conservative or liberal, or even, particularly, their expertise in the field, but how they approached the problem. People who assumed the world was simple and had simple solutions did badly; people who thought it was complex, who realised they could be wrong, and who learnt from mistakes did better. It also helped if you were good with numbers, and good at spotting patterns, as in IQ tests.
No comments:
Post a Comment