Are "superforecasters" a real phenomenon?

The clear answer to the question posed, "do the performances of GJP participants follow a power-law distribution, such that the best 2% are significantly better than the rest" is yes - with a minor quibble, and a huge caveat. (Epistemic status: I'm very familiar with the literature, have personal experience as a superforecaster since the beginning, had discussions with Dan Gardner and the people running, have had conversations with the heads of Good Judgement Inc, etc.)

The minor quibble is identified in other comments, that it is unlikely that there is a sharp cutoff at 2%, there isn't a discontinuity, and power law is probably the wrong term. Aside from those "minor" issues, yes, there is a clear group of people who outperformed multiple years in a row, and this groups was fairly consistent from year to year. Not only that, but the order withing that group is far more stable than chance. That clearly validates the claim that "superforcasters are a real thing."

But the data that those people are better is based on a number of things, many of which aren't what you would think. First, the biggest difference between top forecasters and the rest is frequency of updates and a corresponding willingness to change their minds as evidence comes in. People who invest time in trying to forecast well do better than those who don't - to that extent, it's a skill like most others. Second, success at forecasting is predicted by most of the things that predict success at almost everything else - intelligence, time spent, and looking for ways to improve. Some of the techniques that Good Judgement advocates for superforecasters are from people who read Kahneman and Twersky, Tetlock, and related research, and tried to apply the ideas. The things that worked were adopted - but not everything helped. Other techniques were original to the participants - for instance, explicitly comparing your estimate for a question based on different timeframes, to ensure it is a coherent and reasonable probability. (Will X happen in the next 4 months? If we changed that to one month, would be estimate be about a quarter as high? What about if it were a year? If my intuition for the answer is about the same, I need to fix that.) Ideas like this are not natural ability, they are just applying intelligence to a problem they care about.

Also, many of the poorer performers were people who didn't continue forecasting, and their initial numbers got stale - they presumably would have updated. The best performers, on the other hand, checked the news frequently, and updated. At times, we would change a forecast once the event had / had not happened, a couple days before the question was closed, yielding a reasonably large "improvement" in our time-weighted score. This isn't a function of being naturally better - it's just the investment of time that helps. (This also explains a decent part of why weighting recency in aggregate scores is helpful - it removes stale forecasts.)

So in short, I'm unconvinced that superforecasters are a "real" thing, except in the sense that most people don't try, and people who do will do better, and improve over time. Given that, however, we absolutely should rely on superforecasters to make better predictions that the rest of people - as long as they continue doing the things that make them good forecasters.

I was going off absence of evidence (the paper didn't say anything other than taking the top 2%), so if anyone else has positive evidence that outweighs what I'm saying.

I don't see much disagreement between the two sources. The Vox article doesn't claim that there is much reason for selecting the top 2% rather than the top 1% or the top 4% or whatever. And the SSC article doesn't deny that the people who scored in the top 2% (and are thereby labeled "Superforecasters") systematically do better than most at forecasting.

I'm puzzled by the use of the term "power law distribution". I think that the GJP measured forecasting performance using Brier scores, and Brier scores are always between 0 and 1, which is the wrong shape for a fat-tailed distribution. And the next sentence (which begins "that is") isn't describing anything specific to power law distributions. So probably the Vox article is just misusing the term.

Just observing that the answer to this question should be more or less obvious from a histogram (assuming large enough N and a sufficient number of buckets), "Is there a substantial discontinuity at the 2% quantile?"

Power law behaviour is not necessary and arguably not sufficient for "superforecasters are a natural category" to win (e.g. it should win in a population in which 2% have a brier score of zero and the rest 1, which is not a power law).

By definition, the top 2% are always better than the other 98%.

Pre-selected so-called superforecasters were one of the groups asked to anticipate the outcome of UK's referendum on membership of the European Union and they gave a 23% chance of a vote against membership succeeding - the lowest % and worst performance of all groups asked to predict the result. For a forecasting system to be reliable, it has to be assessed as to its skill over a period of continuous forecasting that will tell you what % of forecasts are correct. To.be useful, this figure should be about 70% or more. The only forecasting system that exceeds this level of skill is the weather forecasting system. Superforecasting does not exist.