Startups as a Rationality Test Bed?

I have thought of this as well. A startup seems like a great test of rationality. Unfortunately, doing a successful startup is hard. It would be great to get rationality to the point of it empowering any old smart person to do a startup, but I think we will need smaller tests before then.

That said, I had not though of coming at it the other way like you did. Take potential startup people, teach rationality to half of them, and observe the outcome. What a great idea!

I think the best source of startup-caliber people who are about to start one is ycombinator. I wonder if we could convince paul graham to let us test some ideas on his founders. The information feedback time is very long tho. I don't think you could get definitive results for at least half a year. Testing rationality ideas on startup people would be a great way to solidly verify the methods, but it's not fast enough to develop them.

In case anyone missed it, from 2 days ago:

http://lesswrong.com/lw/9ik/a_word_to_the_resourceful/

I like this idea a lot, but I have a few questions.

1. Approximately how many hours of instruction are you imagining?
2. Would you do any placebo controlling? Maybe give a week of generic business classes to the control group?
3. How would you measure success? Just whether the company still exists? If they make a profit? Maybe some kind of subjective evaluation by the owners (i.e. do they consider the business to have met their own definition of success)?

The spinoff rationality training org is developing training materials, and their target level is that of the Silicon Valley startup crowd.

I'd say you're probably thinking along very similar lines as Eliezer et al.

The trouble is that you don't want to also measure a bunch of other stuff. Luck, connections, skills specific to your company, etc. This stuff makes your error bars really big. In order to get your error bars down, you might be able to measure some of these extra factors separately. But ultimately, I don't think running a startup will work as a standalone measure of rationality.

You could also do the lower-cost test of (simulated) venture-capital decisions, which presumabky tests a similar set of epistemic skills.

A cheap(er) way to get a less-clear look at the same information would be to find as many aspiring rationalist startup founders as possible (ones who have attended CFAR classes would be preferable, but we'd probably want to look at anyone associated with the rationalist community to have a larger sample), get data on startups without rationalist founders, and compare how well each group did. It would be really useful to know enough about the startups in the control group to control for things that could influence chances of success, especially things where rationalists tend to differ from the general population - IQ, autism spectrum quotient, etc.

So few startups succeed that it might not be possible to get a meaningful result out of this - I'm not sure how many rationalists have founded startups, but probably not enough to make a decent sample size.

Startups are hit and miss. There is a fair amount of luck involved, as well as other skills not related to rationality, and sometimes conflicting with it (using dark arts to successfully sell your idea/product, hire employees on a cheap with promise of a future big payout).

So no, it's not a good test, unless you control for for all other factors, which means you have to have many startups running at the same time and analyze the statistics after a few years.