For the past three years, Rich and 3,000 other average people have been quietly making probability estimates about everything from Venezuelan gas subsidies to North Korean politics as part of the Good Judgment Project, an experiment put together by three well-known psychologists and some people inside the intelligence community.

. . . For most of his professional career, Tetlock studied the problems associated with expert decision making. His book Expert Political Judgment is considered a classic, and almost everyone in the business of thinking about judgment speaks of it with unqualified awe.

All of his studies brought Tetlock to at least two important conclusions.

First, if you want people to get better at making predictions, you need to keep score of how accurate their predictions turn out to be, so they have concrete feedback.

But also, if you take a large crowd of different people with access to different information and pool their predictions, you will be in much better shape than if you rely on a single very smart person, or even a small group of very smart people.

The wisdom of crowds is a very important part of this project, and it’s an important driver of accuracy, Tetlock said.

According to one report, the predictions made by the Good Judgment Project are often better even than intelligence analysts with access to classified information, and many of the people involved in the project have been astonished by its success at making accurate predictions.

. . . There’s a lot of noise, a lot of statistical random variation, Tetlock said. But it’s random variation around a signal, a true signal, and when you add all of the random variation on each side of the true signal together, you get closer to the true signal.

In other words, there are errors on every side of the mark, but there is a truth at the center that people are responding to, and if you average a large number of predictions together, the errors will end up canceling each other out, and you are left with a more accurate guess.

That is the wisdom of the crowd.

The point of the Good Judgment Project was to figure out if what was true for the dead ox is true for world events as well.

It is.

In fact, Tetlock and his team have even engineered ways to significantly improve the wisdom of the crowd — all of which greatly surprised Jason Matheny, one of the people in the intelligence community who got the experiment started.

They’ve shown that you can significantly improve the accuracy of geopolitical forecasts, compared to methods that had been the state of the art before this project started, he said.

What’s so challenging about all of this is the idea that you can get very accurate predictions about geopolitical events without access to secret information. In addition, access to classified information doesn’t automatically and necessarily give you an edge over a smart group of average citizens doing Google searches from their kitchen tables.

–Alix Spiegel, So You Think You’re Smarter Than A CIA Agent (2 April 2014), NPR Parallels

It’s nice to see closer attention to this in ongoing research. It ought to be completely unsurprising to hear these results, though. The basics of the wisdom-of-crowds findings are well-established by now, even old hat. So it should not be surprising that when you let a lot of people look at a problem, they often do better than the predictions of official experts. If folks are still agog that jes’ folks might do better at predicting than spies with access to classified information, then it may be because they have too much of a science-fiction picture of how intelligence agencies actually work, and too much of a picture of intelligence agencies as omnicompetent, omniscient secret conspiracies — as if the most significant information for predicting world events were some secret world of deep, dark esoteric truths that are locked away in classified files, which give them some immense and systematic advantage over jes’ folks without access to such files. But that’s not how it works. The overwhelming majority of intel analysis has nothing to do with classified information in the first place; it has to do with the completely unsexy reality of systematically gathering publicly available information from foreign newspapers, television, statistical abstracts, etc. There’s no reason to think that privileged government experts have an advantage in doing that; the image of spy agencies as omniscient forces in control of secret wisdom has much more to do with politics than it does with practical reality.

(Via William Gillis.)