Brief Query- An Idea

Not at all a dumb question.

I assume the idea is to test "Rationality".

Basically, how good are LW readers at being right about tricky stuff relative to similarly intelligent folk who aren't LW readers?

I would think this would encompass (non-exhaustively) decision-making under risk and uncertainty, inference from incomplete information, making accurate predictions, and being well-calibrated.

Precisely what you would test (which may have been your point) is a very good question and is not at all obvious.

I'd try to answer that, but my knowledge of the scientific method isn't too good.

On a very similiar point though, a sufficently rigorous test to persuade scientists (rationally or irrationally) to conduct a professional test would be sufficent- as such a test would be the optimal result in theory if it could come about some other way, I think it would be a pragmatic choice.

Regarding 2, the reason to have a non-LWer is presumably because we are more likely to be biased and thus introduce subtle biases that favor LWians. Don't underestimate the human capacity for self-deception.

As suggested in the OP, they have to create the tests, not only evaluate their results. Even if average LWers want to find out whether LW memes are actually helpful, they are likely to be biased in choosing the criteria of rationality. For example, a test made by a LWer would more likely include a Newcombesque question where one-boxing would be classified as the rational answer, and since one-boxers are certainly more prevalent among LWers than in nearly any other group, the results would show that LW memes improve rationality. But the OP is not interested in testing whether LW memes improve LW-style extended rationality (it would be quite weird if they didn't) but a practical, real-life relevant rationality. We are not impartial judges when it comes to determining the boundary between these two.

Or more generally, you can never be too careful about possible biases. Not seeing a reason for a self-serving bias is a pretty weak evidence for its non-existence.

Probably we should have two or three different controls. One group of average humans, one group of scientists, day traders, and entrepreneurs, and one group of nerds on the internet.

where do you get a group of "people who haven't read LW yet, but are the sort of person who might read LW"?

They're called newbies. People who just recently started reading LW. Measure the improvement in rationality for the control group and for the experimental, newbie group.

Actually, the hard part may be finding a scientist willing to risk eir career and past work to admit that ey isn't a rationalist.