Rationality, Singularity, Method, and the Mainstream

But "produc[ing] formidable rationalists" sounds like it's meant to make the world better in a generalized way, by producing people who can shine the light of rationality into every dark corner, et cetera.

Precisely. The Singularity Institute was founded due to Eliezer's belief that trying to build FAI was the best strategy for making the world a better place. That is the goal. FAI is just a sub-goal. There is still consensus that FAI is the most promising route, but it does not seem wise to put all of our eggs in one basket. We can't do all of the work that needs to be done within one organization and we don't plan to try.

Through programs like Rationality Boot Camp, we expect to identify people who really care about improving the world and radically increase their chances of coming to correct conclusions about what needs to be done and then actually doing so. Not only will more highly-motivated, rational people improve the world at a much faster rate, they will also serve as checks on our sanity. I don't expect that we are sufficiently sane at the moment to reliably solve the world's problems and we're really going to need to step up our game if we hope to solve FAI. This program is just the beginning. The initial investment is relatively small and, if we can actually do what we think we can, the program should pay for itself in the future. We'd have to be crazy not to try this. It may well be too confusing from a PR perspective to run future versions of the program within SingInst, but if so we can just turn it into its own organization.

If you have concrete proposals for valuable projects that you think we're neglecting and would like to help out with I would be happy to have a Skype chat and then put you in contact with Michael Vassar.

On a note related to "mission creep", I'd like to mention that the SL4 list has been lying dead for a long time, and that it would be nice to have an official successor. LW could be seen as that successor in that LW seems to have displaced the singularitarian community around SL4, but the latter had some virtues relative to the former, and I'm not sure this is on the whole a good situation.

What underlies your intuition that machine ethics will come to the rescue? Have you read the literature? It seems optimistic given what has been published to date. Most people are so fundamentally wrong that their presence in the field is net-negative. The handful of good researchers have to work extra hard to pull against the dead-weight of incompetence in this field already. If there was a "rapid growth" of this field, I would expect the few reasonable voices to be even more drowned out by the incoming rush of less-thoughtful colleagues.

Also, your analogy of x-rationality to Einstein's musing on life is eloquent but misleading. It actually is the case that scientists approaching FAI-level problems while working outside the framework of x-rationality are net-detrimental to progress. Having a 40 year career worth of bad intuitions driven by predictable biases doesn't help anyone find a solution.

At this time, my opinion stands with Jasen and SIAI and not with Mitchell. (But I haven't downvoted this post; it's worth discussing and is well-written.)

I think JoshuaFox hit several of the right points. Here are my main reasons for thinking the rationality boot camp is a good idea:

1. This is training to win at life.

Most people who go through the summer program will not end up working on AI directly. But this kind of bootcamp is training for winning at life, like a bootcamp on public speaking or social networking or dating skills. The boot camp Jasen is putting together will likely be more useful to more people than what SIAI has done in the past, and more useful to more people than a summer program all about AI.

2. This boot camp recruits the right people.

If you can't handle this kind of rigorous and specific rationality training, it's less likely you will be able to make useful, long-term contributions to the project of Friendly AI. If you're only up to the level of Traditional Rationality, you are not cut out to work on the single most important and difficult problem humanity has to face.

3. FAI doesn't allow experimentation. You have to be optimally rational.

SIAI must take steps to ensure that it's people are about as rational as humans are capable of being. One little bias could fuck the whole planet. There is no do-over after your experiment failed.

Maybe it's just me, but I think your writing has been getting much better in the last few weeks, especially your Vienna Circle comment and this post. Both have managed to change my opinions on several issues. Keep up the good work :-)

For what it's worth, my view is precisely the reverse: the Singularity is wishful thinking and trying to figure out Friendly AI today is like Roger Bacon trying to design a spam filter, but rationality is a worthwhile pursuit, the Sequences are some of the best writing on philosophy in the English language - I'd put them fully on a par with Hofstadter - and the rationality boot camp sounds like an interesting and potentially productive experiment.

Also, I'm far more impressed by the specific ideas Eliezer has come up with over the years - the concept of seed AI; the concept of Friendly AI; CEV; TDT - than by his ruminations about rationality in the Sequences.

The opposite for me.

I couldn't have put it better myself.

I do understand the SIAI's explanations for their rationality work.

1. It recruits good people.
2. FAI is a field that does not allow experimentation, and the stakes are high, and so top rationality skills are needed. And maybe:
3. Understanding correct rationality as well as human biases may cast light on the best architecture for an AGI.

But in the end, I quite agree with Mitchell's points.

Why did Eliezer take a detour from his AI work to write the sequences and create Less Wrong? Here is what he says in this interview with John Baez:

There are also an absolutely huge number of pitfalls that people stumble into when they try to think about, as I would put it, Friendly AI. Consider how many pitfalls people run into when they try to think about Artificial Intelligence. Next consider how many pitfalls people run into when they try to think about morality. Next consider how many pitfalls philosophers run into when they try to think about the nature of morality. Next consider how many pitfalls people run into when they try to think about hypothetical extremely powerful agents, especially extremely powerful agents that are supposed to be extremely good. Next consider how many pitfalls people run into when they try to imagine optimal worlds to live in or optimal rules to follow or optimal governments and so on.

Now imagine a subject matter which offers discussants a lovely opportunity to run into all of those pitfalls at the same time.

That’s what happens when you try to talk about Friendly Artificial Intelligence. And it only takes one error for a chain of reasoning to end up in Outer Mongolia. So one of the great motivating factors behind all the writing I did on rationality and all the sequences I wrote on Less Wrong was to actually make it possible, via two years worth of writing and probably something like a month’s worth of reading at least, to immunize people against all the usual mistakes.

So, in effect, the sequences constitute a training manual for new SIAI hires. And presumably Boot Camp consists of a trial run for an SIAI training program which involves some practical work.

I'll have to think about the subjective method part. In the meantime:

The mission, as I recall, was to design and code a Friendly artificial intelligence.

That'd be nice, sure. It technically seems less likely than the SIAI producing a detailed theory of Friendliness. So someone at some point may need to convince a lot of people to think about the issue in a certain (rational) way.

Here is a segment of the line of reasoning that I have been considering lately.

• The details of the creation of fAI are unknown; they must be discovered if we wish to avert the forecast catastrophe and reap the full benefits of AI.

• Further insights into decision-making are necessary for the creation of fAI.

• Teaching can be used as an efficient means of learning.

• Teaching rationality is expected to produce insights into decision-making.

• Insights into decision-making are what is missing from an understanding of fAI, AFAIK.

Furthermore, there will be significant amount of disutility before fAI is implemented. As utility is that thing which we wish to maximize, we ought to want to want to decrease disutility. The primary means we have at our disposal is rationality.

The caveat is that we must make sure that the benefits of teaching rationality in the present exceed the cost from delaying fAI.

My expectation is that the presently small fields of machine ethics and neuroscience of morality will grow rapidly and will come into contact, and there will be a distributed research subculture which is consciously focused on determining the optimal AI value system in the light of biological human nature.

Is SIAI working on trying to cause that?

It seems like it would do more harm than good, since it does a lot of work for FAI, and almost none for AI.

Funny, I just posted a shorter and less-specific discussion post for largely the same reason - my perception that LW is abandoning addressing these problems, in favor of rationality techniques.

My impression is that that was Eliezer's intention from the start. And, I love the idea of rationality bootcamp.

Michael Vassar has put a lot more thought into what SIAI should do now than I have, and I'm not going to disagree with him. Addressing this requires a financial analysis, not just picking what you think is the biggest problem and going straight at it. My impression is that SIAI operates like a college, which pays for operating expenses (neglecting grants and donations) by taking in a large number of undergraduates, in order to fund the work of grad students and professors.

I think LW is not going to be the place where we tackle these problems. That place does not yet exist. And I think LW should not be the place where we tackle these problems. It has already converged on a strong local attractor of opinion which it cannot escape from. It is too strongly-affiliated with a small number of people with largely similar viewpoints. We need some random restarts.

(I think the last 3 paragraphs of OP should be a separate post; their position at the conclusion of the post distracts greatly from the focus of the first part.)

...specific ideas Eliezer has come up with over the years - the concept of seed AI...

I don't want to nitpick on this again but when exactly did he come up with that concept? Because it is mentioned in Karl Schroeder's Ventus (Tor Books, 2000.) ISBN 978-0312871970.

Here are a few quotes from Ventus:

Look at it this way. Once long ago two kinds of work converged. We'd figured out how to make machines that could make more machines. And we'd figured out how to get machines to... not exactly think, but do something very much like it. So one day some people built a machine which knew how to build a machine smarter than itself. That built another, and that another, and soon they were building stuff the men who made the first machine didn't even recognize.

[...]

And, some of the mechal things kept developing, with tremendous speed, and became more subtle than life. Smarter than humans. Conscious of more. And, sometimes, more ambitious. We had little choice but to label them gods after we saw what they could do--namely, anything.

[...]

They did not command the wealth of nations, these researchers. Although their grants amounted to millions of Euros, they could never have funded a deep-space mission on their own, nor could they have built the giant machineries they conceived of. In order to achieve their dream, they built their prototypes only in computer simulation, and paid to have a commercial power satellite boost the Wind seeds to a fraction of light speed. [...] no one expected the Winds to bloom and grow the way they ultimately did.

It is further explained that the Winds were designed to evolve on their own so they are not mere puppets of human intentions but possess their own intrinsic architecture.

In other places in the book it is explained how humans did not create their AI Gods but that they evolved themselves from seeds designed by humans.