Rationality Verification Opportunity?

The field of psychometrics is all about this kind of thing. The keyword here is "practice effect". It sort of looks like Wikipedia's deletionists have trimmed their content on the concept down to two sentences in an article that's been nominated for deletion as non-notable, but if you hunt around you can find pre-existing content on ways to control for practice effects.

The unique thing about the situation with LW in this respect seems to me to be that there are a lot of people who tend to conceive and execute and publish polls with relatively sophisticated methodology for the internet in the complete absence of grants or formal publication or whatever. We're doing as a hobby for a community blog, (yes, a blog) what academics make an entire career out of! Maybe not fully solid with control groups yet, but we're kind of close to this already actually.

To make this sort of "surprising but casual competence" more dramatic and effective, it seems like it might be worthwhile to do a sequence on current best practices for community members to run studies on the community itself. Between the free polling technology available via the forms built into google docs and and chapters from psychometrics textbooks, I bet it wouldn't be that hard for someone to pull together such content for a sequence that makes it easier for people here to spice their efforts up with pretty solid techniques :-)

For example, we probably could get some interesting control data for practice effects the way Anna got control data in her recent poll via mechanical turk. If you've developed the quiz content you could ask LWers to take it and turkers to take it, and then get the same LWers and turkers to re-take it with some being exposed to whatever manipulation you tried on LW by posting content and others not... it wouldn't be a perfect control, but (ignoring the costs) it would be better than nothing...

...which naturally leads me to wonder. What is the value of information here? Is there some change in behavior that certain results would cause? What kind of increase in value could be expected from such a change in behavior? Anyone have guesses here?

Since you declared Crocker's rules on the writing so explicitly...

teqnuices

from first paragraph
techniques*

verification for new sequnces before thinking of any solutions

from third paragraph
sequences*

yourself(I know t

from second paragraph
This should probably have a space.

genreally

from fourth paragraph last sentence
generally*

reliablity

from fourth paragraph first sentence
reliability*

While there are some time and talent limitation I would be willing to help with creating the measures, collecting and interpeting the data and any other necessary steps.

limitations
interpreting

It seems that most writers have a decent idea of what benefits they expect people to gain from their sequences'

from fourth paragraph third sentence
There is no need for an apostrophe after the word "sequences."

Then before running the sequence main sequence they could put out a call for people to complete these measures and send them in

"Then before publishing the sequence they could request people complete these measures and send them in" would be a bit better.

Your post reads like spoken communication. Almost every sentence could be improved.

Please read this paper before you start down an unproductive path. It's in a different field but many of the same caveats apply.

The solution I came up with (when you insisted) was to verify rationality by observing the production of novel results. They don't have to be Nobel-prize grade, but someone who has actually understood some material presented here should be able to use it to generate thoughts that are not the result of a cache lookup.

In practical terms, this would look like a "meta-exercise" a la "The Five Second Level", asking the reader to first design an application exercise for the ideas newly assimilated, and then solve that exercise. The only requirement is that any solution should come as a surprise to the person proposing it.