Help create an instrumental rationality "stack ranking"?

Pretests. Pretests are critical.

This is something I've been thinking about for quite some time:

I had an idea for a web-based app for evaluating instrumental rationality techniques, something like Digg or a UserVoice-based forum where techniques get upvoted, downvoted, merged, separated and discussed. However, I don't currently have a solution for the problem of 'impulse upvoting' ("hey, this technique sounds cool, let's upvote it!") -- I don't know how to make the upvotes reflect long-term usefulness of the techniques.

My current best idea regarding the impulse upvote problem is "making techniques pay rent", literally:

Each app user has a fixed small number of homepage slots for techniques. If a technique doesn't work for a user, they can kick it out of the slot and replace it with another promising technique. Also, users can purchase more homepage slots for real-world money. Or the app can go even further: it can be free to download, but each slot will cost the user $0.99 per month.

This way we can rank the techniques based on their "rent time", i.e. the time they spend in their slot, which is only counted for active users of the app.

Usefulness of some techniques may depend on domain where one wants to use them. For example a technique "if you don't know something, use google and follow 3 highest links" depends on whether your problems are described on internet, and how much trustworthy are those answers. -- For "how do I join two strings in Python?" this technique works great. For financial questions, not so great, because website owners have a huge incentive to promote wrong answers.

Also, the same technique may have different results for different kinds of people, because of their environment, previous knowledge, personality, gender, social class, financial situation, or whatever. If you omit those details, you only get average results in general population, which is also not bad, but does not lead to optimal choice.

Measuring an impact of a technique is difficult. How much sure are you it was this technique that helped, and not something else? Maybe it was a placebo effect or just a coincidence. If we had hundreds of data points, the coincidences would average out, but we probably won't have so much data.

It always amazes me how a self proclaimed rationalist community who wouldn't accept a medical intervention based on anecdotal evidence have no problem accepting techniques for improving rationality based on anecdotal evidence.

I would like basic short answer/MC tests for Lesswrong content. I have not been through a lot of sequences but I have been around people that have and been in discussions about those sequences, and I have also read a lot of important posts from sequences that I haven't read entirely. I would like to know if I actually know the material. It can also be a good teaching tool in itself (see the comment on pretests), and it can also assess knowledge of Lesswrongian terms and phrases.

Even if it doesn't really test one's skill as a rationalist, they can still be incredibly useful.

This is a nicely written proposal for a practical, actionable idea. If you're not in tech, then you should consider doing this. It starts at a practical idea and has ways to branch out from the long term goal to other like including psychometrics and so forth.

I'm biased due to my open source project, but I think this is the kind of idea that fits well with cryptographically secure peer to peer systems that then aggregate into some groups, as the individual opinions are highly variable (correctly, as different brains need different training)