The Importance of Self-Doubt
post by multifoliaterose · 2010-08-19T22:47:15.149Z · LW · GW · Legacy · 746 commentsContents
746 comments
[Added 02/24/14: After I got feedback on this post, I realized that it carried unnecessary negative connotations (despite conscious effort on my part to avoid them), and if I were to write it again, I would have framed things differently. See Reflections on a Personal Public Relations Failure: A Lesson in Communication for more information. SIAI (now MIRI) has evolved substantially since 2010 when I wrote this post, and the criticisms made in the post don't apply to MIRI as presently constituted.]
Follow-up to: Other Existential Risks, Existential Risk and Public Relations
Related to: Tsuyoku Naritai! (I Want To Become Stronger), Affective Death Spirals, The Proper Use of Doubt, Resist the Happy Death Spiral, The Sin of Underconfidence
In Other Existential Risks I began my critical analysis of what I understand to be SIAI's most basic claims. In particular I evaluated part of the claim
(1) At the margin, the best way for an organization with SIAI's resources to prevent global existential catastrophe is to promote research on friendly Artificial Intelligence, work against unsafe Artificial Intelligence, and encourage rational thought.
It's become clear to me that before I evaluate the claim
(2) Donating to SIAI is the most cost-effective way for charitable donors to reduce existential risk.
I should (a) articulate my reasons for believing in the importance of self-doubt and (b) give the SIAI staff an opportunity to respond to the points which I raise in the present post as well as my two posts titled Existential Risk and Public Relations and Other Existential Risks.
Yesterday SarahC described to me how she had found Eliezer's post Tsuyoku Naritai! (I Want To Become Stronger) really moving. She explained:
I thought it was good: the notion that you can and must improve yourself, and that you can get farther than you think.
I'm used to the other direction: "humility is the best virtue."
I mean, this is a big fuck-you to the book of Job, and it appeals to me.
I was happy to learn that SarahC had been positively affected by Eliezer's post. Self-actualization is a wonderful thing and it appears as though Eliezer's posting has helped her self-actualize. On the other hand, rereading the post prompted me to notice that there's something about it which I find very problematic. The last few paragraphs of the post read:
Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws. This is akin to the principle of not taking pride in confessing your ignorance; for if your ignorance is a source of pride to you, you may become loathe to relinquish your ignorance when evidence comes knocking. Likewise with our flaws - we should not gloat over how self-aware we are for confessing them; the occasion for rejoicing is when we have a little less to confess.
Otherwise, when the one comes to us with a plan for correcting the bias, we will snarl, "Do you think to set yourself above us?" We will shake our heads sadly and say, "You must not be very self-aware."
Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it. Afterward you will still have plenty of flaws left, but that's not the point; the important thing is to do better, to keep moving ahead, to take one more step forward. Tsuyoku naritai!
There's something to what Eliezer is saying here: when people are too strongly committed to the idea that humans are fallible this can become a self-fulfilling prophecy where humans give up on trying to improve things and as a consequence remain fallible when they could have improved. As Eliezer has said in The Sin of Underconfidence, there are social pressures that push against having high levels of confidence even when confidence is epistemically justified:
To place yourself too high - to overreach your proper place - to think too much of yourself - to put yourself forward - to put down your fellows by implicit comparison - and the consequences of humiliation and being cast down, perhaps publicly - are these not loathesome and fearsome things?
To be too modest - seems lighter by comparison; it wouldn't be so humiliating to be called on it publicly, indeed, finding out that you're better than you imagined might come as a warm surprise; and to put yourself down, and others implicitly above, has a positive tinge of niceness about it, it's the sort of thing that Gandalf would do.
I have personal experience with underconfidence. I'm a careful thinker and when I express a position with confidence my position is typically well considered. For many years I generalized from one example and assumed when people express positions with confidence they've thought their positions out as well as I have. Even after being presented with massive evidence that few people think things through as carefully as I do, I persisted in granting the (statistically ill-considered) positions of others far more weight than they deserved for the very reason that Eliezer describes above. This seriously distorted my epistemology because it led to me systematically giving ill-considered positions substantial weight. I feel that I have improved on this point, but even now, from time to time I notice that I'm exhibiting irrationally low levels of confidence in my positions.
At the same time, I know that at times I've been overconfident as well. In high school I went through a period when I believed that I was a messianic figure whose existence had been preordained by a watchmaker God who planned for me to save the human race. It's appropriate to say that during this period of time I suffered from extreme delusions of grandeur. I viscerally understand how it's possible to fall into an affective death spiral.
In my view one of the central challenges of being human is to find an instrumentally rational balance between subjecting oneself to influences which push one in the direction of overconfidence and subjecting oneself to influences which push one in the direction of underconfidence.
In Tsuyoku Naritai! Eliezer describes how Orthodox Judaism attaches an unhealthy moral significance to humility. Having grown up in a Jewish household and as a consequence having had peripheral acquaintance with orthodox Judaism I agree with Eliezer's analysis of Orthodox Judaism in this regard. In the proper use of doubt, Eliezer describes how the Jesuits allegedly are told to doubt their doubts about Catholicism. I agree with Eliezer that self-doubt can be misguided and abused.
However, reversed stupidity is not intelligence. The fact that it's possible to ascribe too much moral significance to self-doubt and humility does not mean that one should not attach moral significance to self-doubt and humility. I strongly disagree with Eliezer's prescription: "Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws."
The mechanism that determines human action is that we do what makes us feel good (at the margin) and refrain from doing what makes us feel bad (at the margin). This principle applies to all humans, from Gandhi to Hilter. Our ethical challenge is to shape what makes us feel good and what makes us feel bad in a way that incentivizes us to behave in accordance with our values. There are times when it's important to recognize that we're biased and flawed. Under such circumstances, we should feel proud that we recognize that we're biased we should glory in our self-awareness of our flaws. If we don't, then we will have no incentive to recognize that we're biased and be aware of our flaws.
We did not evolve to exhibit admirable and noble behavior. We evolved to exhibit behaviors which have historically been correlated with maximizing our reproductive success. Because our ancestral climate was very much a zero-sum situation, the traits that were historically correlated with maximizing our reproductive success had a lot to do with gaining high status within our communities. As Yvain has said, it appears that a fundamental mechanism of the human brain which was historically correlated with gaining high status is to make us feel good when we have high self-image and feel bad when we have low self-image.
When we obtain new data, we fit it into a narrative which makes us feel as good about ourselves as possible; a way conducive to having a high self-image. This mode of cognition can lead to very seriously distorted epistemology. This is what happened to me in high school when I believed that I was a messianic figure sent by a watchmaker God. Because we flatter ourselves by default, it's very important that those of us who aspire to epistemic rationality incorporate a significant element of "I'm the sort of person who engages in self-doubt because it's the right thing to do" into our self-image. If we do this, when we're presented with evidence which entails a drop in our self-esteem, we don't reject it out of hand or minimize it as we've been evolutionarily conditioned to do because wound of properly assimilating data is counterbalanced by the salve of the feeling "At least I'm a good person as evidenced by the fact that I engage in self-doubt" and failing to exhibit self-doubt would itself entail an emotional wound.
This is the only potential immunization to the disease of self-serving narratives which afflicts all utilitarians out of virtue of their being human. Until technology allows us to modify ourselves in a radical way, we cannot hope to be rational without attaching moral significance to the practice of engaging in self-doubt. As the RationalWiki's page on LessWrong says:
A common way for very smart people to be stupid is to think they can think their way out of being apes with pretensions. However, there is no hack that transcends being human...You are an ape with pretensions. Playing a "let's pretend" game otherwise doesn't mean you win all arguments, or any. Even if it's a very elaborate one, you won't transcend being an ape. Any "rationalism" that doesn't expressly take into account humans being apes with pretensions, isn't.
In Existential Risk and Public Relations I suggested that some of Eliezer's remarks convey the impression that Eliezer has an unjustifiably high opinion of himself. In the comments to the post JRMayne wrote
I think the statements that indicate that [Eliezer] is the most important person in human history - and that seems to me to be what he's saying - are so seriously mistaken, and made with such a high confidence level, as to massively reduce my estimated likelihood that SIAI is going to be productive at all.
And that's a good thing. Throwing money into a seriously suboptimal project is a bad idea. SIAI may be good at getting out the word of existential risk (and I do think existential risk is serious, under-discussed business), but the indicators are that it's not going to solve it. I won't give to SIAI if Eliezer stops saying these things, because it appears he'll still be thinking those things.
When Eliezer responded to JRMayne's comment, Eliezer did not dispute the claim that JRMayne attributed to him. I responded to Eliezer saying
If JRMayne has misunderstood you, you can effectively deal with the situation by making a public statement about what you meant to convey.
Note that you have not made a disclaimer which rules out the possibility that you claim that you're the most important person in human history. I encourage you to make such a disclaimer if JRMayne has misunderstood you.
I was disappointed, but not surprised, that Eliezer did not respond. As far as I can tell, Eliezer does have confidence in the idea that he is (at least nearly) the most important person in human history. Eliezer's silence only serves to further confirm my earlier impressions. I hope that Eliezer subsequently proves me wrong. [Edit: As Airedale points out Eliezer has in fact exhibited public self-doubt in his abilities in his posting The Level Above Mine. I find this reassuring and it significantly lowers my confidence that Eliezer claims that he's the most important person in human history. But Eliezer still hasn't made a disclaimer on this matter decisively indicating that he does not hold such a view.] The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity. At present, anybody who holds such a belief is suffering from extreme delusions of grandeur.
There's some sort of serious problem with the present situation. I don't know whether it's a public relations problem or if the situation is that Eliezer actually suffers from extreme delusions of grandeur, but something has gone very wrong. The majority of the people who I know who outside of Less Wrong who have heard of Eliezer and Less Wrong have the impression that Eliezer is suffering from extreme delusions of grandeur. To such people, this fact (quite reasonably) calls into question of the value of SIAI and Less Wrong. On one hand, SIAI looks like an organization which is operating under beliefs which Eliezer has constructed to place himself in as favorable a position as possible rather than with a view toward reducing existential risk. On the other hand, Less Wrong looks suspiciously like the cult of Objectivism: a group of smart people who are obsessed with the writings of a very smart person who is severely deluded and describing these writings and the associated ideology as "rational" although they are nothing of the kind.
My own views are somewhat more moderate. I think that the Less Wrong community and Eliezer are considerably more rational than the Objectivist movement and Ayn Rand (respectively). I nevertheless perceive unsettling parallels.
In the comments to Existential Risk and Public Relations, timtyler said
...many people have inflated views of their own importance. Humans are built that way. For one thing, It helps them get hired, if they claim that they can do the job. It is sometimes funny - but surely not a big deal.
I disagree with timtyler. Anything that has even a slight systematic negative impact on existential risk is a big deal.
Some of my most enjoyable childhood experiences involved playing Squaresoft RPGs. Games like Chrono Trigger, Illusion of Gaia, Earthbound, Xenogears, and the Final Fantasy series are all stories about a group of characters who bond and work together to save the world. I found these games very moving and inspiring. They prompted me to fantasize about meeting allies who I could bond with and work together with to save the world. I was lucky enough to meet one such person in high school who I've been friends with since. When I first encountered Eliezer I found him eerily familiar, as though he was a long lost brother. This is the same feeling that is present between Siegmund and Sieglinde in the Act 1 of Wagner's Die Walküre (modulo erotic connotations). I wish that I could be with Eliezer in a group of characters as in a Squaresoft RPG working to save the world. His writings such as One Life Against the World and Yehuda Yudkowsky, 1985-2004 reveal him to be a deeply humane and compassionate person.
This is why it's so painful for me to observe that Eliezer appears to be deviating so sharply from leading a genuinely utilitarian lifestyle. I feel a sense of mono no aware, wondering how things could have been under different circumstances.
One of my favorite authors is Kazuo Ishiguro, who writes about the themes of self-deception and people's attempts to contribute to society. In a very good interview Ishiguro said
I think that's partly what interests me in people, that we don't just wish to feed and sleep and reproduce then die like cows or sheep. Even if they're gangsters, they seem to want to tell themselves they're good gangsters and they're loyal gangsters, they've fulfilled their 'gangstership' well. We do seem to have this moral sense, however it's applied, whatever we think. We don't seem satisfied, unless we can tell ourselves by some criteria that we have done it well and we haven't wasted it and we've contributed well. So that is one of the things, I think, that distinguishes human beings, as far as I can see.
But so often I've been tracking that instinct we have and actually looking at how difficult it is to fulfill that agenda, because at the same time as being equipped with this kind of instinct, we're not actually equipped. Most of us are not equipped with any vast insight into the world around us. We have a tendency to go with the herd and not be able to see beyond our little patch, and so it is often our fate that we're at the mercy of larger forces that we can't understand. We just do our little thing and hope it works out. So I think a lot of the themes of obligation and so on come from that. This instinct seems to me a kind of a basic thing that's interesting about human beings. The sad thing is that sometimes human beings think they're like that, and they get self-righteous about it, but often, they're not actually contributing to anything they would approve of anyway.
[...]
There is something poignant in that realization: recognizing that an individual's life is very short, and if you mess it up once, that's probably it. But nevertheless, being able to at least take some comfort from the fact that the next generation will benefit from those mistakes. It's that kind of poignancy, that sort of balance between feeling defeated but nevertheless trying to find reason to feel some kind of qualified optimism. That's always the note I like to end on. There are some ways that, as the writer, I think there is something sadly pathetic but also quite noble about this human capacity to dredge up some hope when really it's all over. I mean, it's amazing how people find courage in the most defeated situations.
Ishiguro's quote describes how people often behave in accordance with sincere desire to contribute and end up doing things that are very different from what they thought they were doing (things which are relatively unproductive or even counterproductive). Like Ishiguro I find this phenomenon very sad. As Ishiguro hints at, this phenomenon can also result in crushing disappointment later in life. I feel a deep spiritual desire to prevent this from happening to Eliezer.
746 comments
Comments sorted by top scores.
comment by Vladimir_Nesov · 2010-08-20T12:05:31.311Z · LW(p) · GW(p)
This post suffers from lumping together orthogonal issues and conclusions from them. Let's consider individually the following claims:
- The world is in danger, and the feat of saving the world (if achieved) would be very important, more so than most other things we can currently do.
- Creating FAI is possible.
- Creating FAI, if possible, will be conductive to saving the world.
- If FAI is possible, person X's work contributes to developing FAI.
- Person X's work contributes to saving the world.
- Most people's work doesn't contribute to saving the world.
- Person X's activity is more important than that of most other people.
- Person X believes their activity is more important than that of most other people.
- Person X suffers from delusions of grandeur.
A priori, from (8) we can conclude (9). But assuming the a priori improbable (7), (8) is a rational thing for X to conclude, and (9) doesn't automatically follow. So, at this level of analysis, in deciding whether X is overconfident, we must necessarily evaluate (7). In most cases, (7) is obviously implausible, but the post itself suggests one pattern for recognizing when it isn't:
The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity.
Thus, "doing something which very visibly and decisively alters the fate of humanity" is the kind of evidence that allows to conclude (7). But unfortunately there is no royal road to epistemic rationality, we can't require this particular argument that (7) in all cases. Sometimes the argument has an incompatible form.
In our case, the shape of the argument that (7) is as follows. Assuming (2), from (3) and (4) it follows that (5), and from (1), (5) and (6) we conclude (7). Note that the only claim about a person is (4), that their work contributes to development of FAI. All the other claims are about the world, not about the person.
Given the structure of this argument for the abhorrent (8), something being wrong with the person can only affect the truth of (4), and not of the other claims. In particular, the person is overconfident if person X's work doesn't in fact contribute to FAI (assuming it's possible to contribute to FAI).
Now, the extent of overconfidence in evaluating (4) is not related to the weight of importance conveyed by the object level conclusions (1), (2) and (3). One can be underconfident about (4) and still (8) will follow. In fact, (8) is rather insensitive to the strength of assertion (4): even if you contribute to FAI a little bit, but the other object level claims hold, your work is still very important.
Finally, my impression is that Eliezer is indeed overconfident about his ability to technically contribute to FAI (4), but not to the extent this post suggests, since as I said the strength of claim (8) has nothing to do with the level of overconfidence in (4), and even small contribution to FAI is enough to conclude (8) given other object level assumptions. Indeed, Eliezer never claims that success is assured:
Success is not assured. I'm not sure what's meant by confessing to being "ambitious". Is it like being "optimistic"?
On the other hand, only few people are currently in the position to claim (4) to any extent. One needs to (a) understand the problem statement, (b) be talented enough, and (c) take the problem seriously enough to direct serious effort at it.
My ulterior motive to elaborating this argument is to make the situation a little bit clearer to myself, since I claim the same role, just to a smaller extent. (One reason I don't have much confidence is that each time I "level up", last time around this May, I realize how misguided my past efforts were, and how much time and effort it will take to develop the skillset necessary for the next step.) I don't expect to solve the whole problem (and I don't expect Eliezer or Marcello or Wei to solve the whole problem), but I do expect that over the years, some measure of progress can be made by mine and their efforts, and I expect other people will turn up (thanks to Eliezer's work on communicating the problem statement of FAI and new SIAI's work on spreading the word) whose contributions will be more significant.
Replies from: multifoliaterose, whpearson, Wei_Dai, JRMayne, cata↑ comment by multifoliaterose · 2010-08-20T14:16:12.245Z · LW(p) · GW(p)
Your analysis is very careful and I agree with almost everything that you say.
I think that one should be hesitant to claim too much for a single person on account of the issue which Morendil raises - we are all connected. Your ability to work on FAI depends on the farmers who grow your food, the plumbers who ensure that you have access to running water, the teachers who you learned from, the people at Google who make it easier for you to access information, etc.
I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher.
I don't believe that Eliezer can credibly hold the view that he's the highest expected value human who has ever lived. Note that he has not offered a disclaimer denying the view that JRMayne has attributed to him despite the fact that I have suggested that he do so twice now.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T21:07:29.958Z · LW(p) · GW(p)
You wrote elsewhere in the thread:
I assign a probability of less than 10^(-9) to [Eliezer] succeeding in playing a critical role on the Friendly AI project that [he's] working on.
Does it mean that we need 10^9 Eliezer-level researchers to make progress? Considering that Eliezer is probably at about 1 in 10000 level of ability (if we forget about other factors that make research in FAI possible, such as getting in the frame of mind of understanding the problem and taking it seriously), we'd need about 1000 times more human beings than currently exists on the planet to produce a FAI, according to your estimate.
How does this claim coexist with the one you've made in the above comment?
I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher.
It doesn't compute, there is an apparent inconsistency between these two claims. (I see some ways to mend it by charitable interpretation, but I'd rather you make the intended meaning explicit yourself.)
Replies from: Jonathan_Graehl, multifoliaterose↑ comment by Jonathan_Graehl · 2010-08-20T22:16:13.141Z · LW(p) · GW(p)
Eliezer is probably at about 1 in 10000 level of ability [of G]
Agreed, and I like to imagine that he reads that and thinks to himself "only 10000? thanks a lot!" :)
In case anyone takes the above too seriously, I consider it splitting hairs to talk about how much beyond 1 in 10000 smart anyone is - eventually, motivation, luck, and aesthetic sense / rationality begin to dominate in determining results IMO.
↑ comment by multifoliaterose · 2010-08-20T22:08:00.945Z · LW(p) · GW(p)
Does it mean that we need 10^9 Eliezer-level researchers to make progress?
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
I'll explain my thinking on these matters later.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T22:14:05.511Z · LW(p) · GW(p)
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
Yes, strictly speaking we'd need even more, if that. The more serious rendition of my remark is that you seem to imply that the problem itself is not solvable at all, by proxy of the estimate of Eliezer's ability to contribute to the solution. But it's OK, informal conclusions differ; what's not OK is that in the other comment you seem to contradict your claim.
Edit: I was not thinking clearly here.
Replies from: Tyrrell_McAllister↑ comment by Tyrrell_McAllister · 2010-08-20T22:28:58.834Z · LW(p) · GW(p)
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
Yes, strictly speaking we'd need even more, if that.
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it's fairly likely that I and my friend will be able to do it together.
Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T22:32:22.457Z · LW(p) · GW(p)
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T22:37:25.227Z · LW(p) · GW(p)
Tyrrel_MacAllister's remark is a significant part of what I have in mind.
I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T23:01:13.324Z · LW(p) · GW(p)
I agree about the benefits of larger research community, although feasibility of "collaborating with existing institutions" is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI.
Although the instinct is to say "Secrecy in science? Nonsense!", it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it's not what we ought to do.
I'm currently ambivalent on this point, but it seems to me that at least preference theory (I'll likely have a post on that on my blog tomorrow) doesn't directly increase the danger, as it's about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won't be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution.
On the other hand, I'm not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a "weapon of math destruction".
Replies from: cousin_it, multifoliaterose↑ comment by cousin_it · 2010-08-23T22:34:18.412Z · LW(p) · GW(p)
preference theory (I'll likely have a post on that on my blog tomorrow)
Hey, three days have passed and I want that post!
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-23T22:56:29.316Z · LW(p) · GW(p)
I have an excuse, I got a cold!
Replies from: cousin_it, cousin_it↑ comment by cousin_it · 2010-08-27T13:02:47.931Z · LW(p) · GW(p)
Three more days have passed.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-09-06T08:37:34.727Z · LW(p) · GW(p)
Planning is the worst form of procrastination. I now have 7 (!) posts planned before the roadmap post I referred to (with the readmap post closing the sequence), so I decided on writing a mini-sequence of 2-3 posts on LW about ADT first.
↑ comment by multifoliaterose · 2010-08-21T05:27:55.689Z · LW(p) · GW(p)
I agree about the benefits of larger research community, although feasibility of "collaborating with existing institutions" is in question, due to the extreme difficulty of communicating the problem statement.
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia.
There are also serious concerns about the end-game
I agree with this and believe that this could justify secrecy, but I think that it's very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness.
I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I'm pursuing my current trajectory of postings.
↑ comment by whpearson · 2010-08-20T14:20:05.701Z · LW(p) · GW(p)
Most people's work doesn't contribute to saving the world.
I'd argue that a lot of people's work does. Everybody that contributes to keeping the technological world running (from farmers to chip designers) enables us to potentially save ourselves from the longer term non-anthrogenic existential risks.
Replies from: Vladimir_Nesov, CarlShulman↑ comment by Vladimir_Nesov · 2010-08-20T14:32:29.707Z · LW(p) · GW(p)
Obviously, you need to interpret that statement as "Any given person's work doesn't significantly contribute to saving the world". In other words, if we "subtract" that one person, the future (in the aspect of the world not ending) changes insignificantly.
Replies from: whpearson, MartinB↑ comment by whpearson · 2010-08-20T14:46:22.598Z · LW(p) · GW(p)
Are you also amending 4) to have the significant clause?
Because there are lots of smart people that have worked on AI, whose work I doubt would be significant. And that is the nearest reference class I have for likely significance of people working on FAI.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T15:04:52.447Z · LW(p) · GW(p)
I'm not amending, I'm clarifying. (4) doesn't have world-changing power in itself, only through the importance of FAI implied by other arguments, and that part doesn't apply to activity of most people in the world. I consider the work on AI as somewhat significant as well, although obviously less significant than work on FAI at the margain, since much more people are working on AI. The argument, as applied to their work, makes them an existential threat (moderate to high when talking about the whole profession, rather weak when talking about individual people).
As for the character of work, I believe that at the current stage, productive work on FAI is close to pure mathematics (but specifically with problem statements not given), and very much unlike most of AI or even the more rigorous kinds from machine learning (statistics).
↑ comment by CarlShulman · 2010-08-20T14:23:30.467Z · LW(p) · GW(p)
Agreed. More broadly, everyone affects anthropogenic existential risks too, which limits the number of orders of magnitude one can improve in impact from a positive start.
↑ comment by Wei Dai (Wei_Dai) · 2012-09-28T19:22:57.047Z · LW(p) · GW(p)
On the other hand, only few people are currently in the position to claim (4) to any extent. One needs to (a) understand the problem statement, (b) be talented enough, and (c) take the problem seriously enough to direct serious effort at it.
(4 here being "If FAI is possible, person X's work contributes to developing FAI.") This seems be a weak part of your argument. A successful FAI attempt will obviously have to use lots of philosophical and technical results that were not developed specifically with FAI in mind. Many people may be contributing to FAI, without consciously intending to do so. For example when I first started thinking about anthropic reasoning I was mainly thinking about human minds being copyable in the future and trying to solve philosophical puzzles related to that.
Another possibility is that the most likely routes to FAI go through intelligence enhancement or uploading, so people working in those fields are actually making more contributions to FAI than people like you and Eliezer.
↑ comment by JRMayne · 2010-08-20T14:19:38.289Z · LW(p) · GW(p)
Person X's activity is more important than that of most other people.
Person X believes their activity is more important than that of most other people.
Person X suffers from delusions of grandeur.
Person X believes that their activity is more important than all other people, and that no other people can do it.
Person X also believes that only this project is likely to save the world.
Person X also believes that FAI will save the world on all axes, including political and biological.
--JRM
↑ comment by cata · 2010-08-20T13:31:49.308Z · LW(p) · GW(p)
Generally speaking, your argument isn't very persuasive unless you believe that the world is doomed without FAI and that direct FAI research is the only significant contribution you can make to saving it. (EDIT: To clarify slightly after your response, I mean to point out that you didn't directly mention these particular assumptions, and that I think many people take issue with them.)
My personal, rather uninformed belief is that FAI would be a source of enormous good, but it's not necessary for humanity to continue to grow and to overcome x-risk (so 3 is weaker); X may be contributing to the development of FAI, but not that much (so 4 is weaker); and other people engaged in productive pursuits are also contributing a non-zero amount to "save the world" (so 6 is weaker.)
As such, I have a hard time concluding that X's activity is anywhere near the "most important" using your reasoning, although it may be quite important.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T13:36:26.394Z · LW(p) · GW(p)
Generally speaking, your argument isn't very persuasive unless you believe that the world is doomed without FAI and that direct FAI research is the only significant contribution you can make to saving it.
The argument I gave doesn't include justification of things it assumes (that you referred to). It only serves to separate the issues with claims about a person from issues with claims about what's possible in the world. Both kinds of claims (assumptions in the argument I gave) could be argued with, but necessarily separately.
Replies from: cata↑ comment by cata · 2010-08-20T14:14:29.849Z · LW(p) · GW(p)
OK, I now see what your post was aimed at, a la this other post you made. I agree that criticism ought to be toward person X's beliefs about the world, not his conclusions about himself.
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T03:46:39.117Z · LW(p) · GW(p)
Unknown reminds me that Multifoliaterose said this:
The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity. At present, anybody who holds such a belief is suffering from extreme delusions of grandeur.
This makes explicit something I thought I was going to have to tease out of multi, so my response would roughly go as follows:
- If no one can occupy this epistemic state, that implies something about the state of the world - i.e., that it should not lead people into this sort of epistemic state.
- Therefore you are deducing information about the state of the world by arguing about which sorts of thoughts remind you of your youthful delusions of messianity.
- Reversed stupidity is not intelligence. In general, if you want to know something about how to develop Friendly AI, you have to reason about Friendly AI, rather than reasoning about something else.
- Which is why I have a policy of keeping my thoughts on Friendly AI to the object level, and not worrying about how important or unimportant that makes me. In other words, I am reluctant to argue on this level not just for the obvious political reasons (it's a sure loss once the argument starts), but because you're trying to extract information about the real world from a class of arguments that can't possibly yield information about the real world.
- That said, as far as I can tell, the world currently occupies a ridiculous state of practically nobody working on problems like "develop a reflective decision theory that lets you talk about self-modification". I agree that this is ridiculous, but seriously, blame the world, not me. Multi's principle would be reasonable only if the world occupied a much higher level of competence than it in fact does, a point which you can further appreciate by, e.g., reading the QM sequence, or counting cryonics signups, showing massive failure on simpler issues.
- That reflective decision theory actually is key to Friendly AI is something I can only get information about by thinking about Friendly AI. If I try to get information about it any other way, I'm producing noise in my brain.
- We can directly apply multi's stated principle to conclude that reflective decision theory cannot be known to be critical to Friendly AI. We were mistaken to start working on it; if no one else is working on it, it must not be knowably critical; because if it were knowably critical, we would occupy a forbidden epistemic state.
- Therefore we have derived knowledge about which problems are critical in Friendly AI by arguing about personal psychology.
- This constitutes a reductio of the original principle. QEA. (As was to be argued.)
↑ comment by Jonathan_Graehl · 2010-08-20T04:18:46.461Z · LW(p) · GW(p)
Upvoted for being clever.
You've (probably) refuted the original statement as an absolute.
You're deciding not to engage the issue of hubris directly.
Does the following paraphrase your position:
Here's what I (and also part of SIAI) intend to work on
I think it's very important (and you should think so for reasons outline in my writings)
If you agree with me, you should support us
? If so, I think it's fine for you to not say the obvious (that you're being quite ambitious, and that success is not assured). It seems like some people are really dying to hear you say the obvious.
Replies from: wedrifid, Eliezer_Yudkowsky↑ comment by wedrifid · 2010-08-20T10:08:09.663Z · LW(p) · GW(p)
Upvoted for being clever.
That's interesting. I downvoted it for being clever. It was a convoluted elaboration of a trivial technicality that only applies if you make the most convenient (for Eliezer) interpretation of multi's words. This kind of response may win someone a debating contest in high school but it certainly isn't what I would expect from someone well versed in the rationalism sequences, much less their author.
I don't pay all that much attention to what multi says (no offence intended to multi) but I pay close attention to what Eliezer does. I am overwhelmingly convinced of Eliezer's cleverness and brilliance as a rationalism theorist. Everything else, well, that's a lot more blurry.
Replies from: Furcas↑ comment by Furcas · 2010-08-20T10:31:53.002Z · LW(p) · GW(p)
I don't think Eliezer was trying to be clever. He replied to the only real justification multi offered for why we should believe that Eliezer is suffering from delusions of grandeur. What else is he supposed to do?
Replies from: wedrifid, khafra↑ comment by wedrifid · 2010-08-20T12:00:48.594Z · LW(p) · GW(p)
I got your reply and respect your position. I don't want to engage too much here since it would overlap with discussion surrounding Eliezer's initial reply and potentially be quite frustrating.
What I would like to see is multifoliaterose giving a considered response to the "If not, why not?" question in that link. That would give Eliezer the chance to respond to the meat of the topic at hand. Eliezer has been given a rare opportunity. He can always write posts about himself, giving justifications for whatever degree of personal awesomeness he claims. That's nothing new. But in this situation it wouldn't be perceived as Eliezer grabbing the megaphone for his own self-gratification. He is responding to a challenge, answering a request.
Why would you waste the chance to, say, explain the difference between "SIAI" and "Eliezer Yudkowsky"? Or at least give some treatment of p(someone other than Eliezer Yudkowsky is doing the most to save the world). Better yet, take that chance to emphasise the difference between p(FAI is the most important priority for humanity) and p(Eliezer is the most important human in the world).
↑ comment by khafra · 2010-08-20T16:49:22.877Z · LW(p) · GW(p)
As Graehl and wedrifid observed, Eliezer responded as if the original statement were an absolute. He applied deductive reasoning and found a reductio ad absurdum. But if, instead of an absolute, you see multifoliaterose's characterization as a reference class: "People who believe themselves to be one of the few most important in the world without having already done something visible and obvious to dramatically change it," it can lower the probability that Eliezer is, in fact, that important by a large likelihood ratio.
Whether this likelihood ratio is large enough to overcome the evidence on AI-related existential risk and the paucity of serious effort dedicated to combating it is an open question.
↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T05:03:09.909Z · LW(p) · GW(p)
Success is not assured. I'm not sure what's meant by confessing to being "ambitious". Is it like being "optimistic"? I suppose there are people who can say "I'm being optimistic" without being aware that they are instantiating Moore's Paradox but I am not one of them.
I also disclaim that I do not believe myself to be the protagonist, because the world is not a story, and does not have a plot.
Replies from: Perplexed, Jonathan_Graehl↑ comment by Perplexed · 2010-08-20T05:14:49.955Z · LW(p) · GW(p)
I hope that the double negative in the last sentence was an error.
I introduced the term "protagonist", because at that point we were discussing a hypothetical person who was being judged regarding his belief in a set of three propositions. Everyone recognized, of course, who that hypothetical person represented, but the actual person had not yet stipulated his belief in that set of propositions.
Replies from: wedrifid↑ comment by wedrifid · 2010-08-20T10:57:37.567Z · LW(p) · GW(p)
I hope that the double negative in the last sentence was an error.
Interesting. I don't claim great grammatical expertise but my reading puts the last question at reasonable. Am I correct in inferring that you do not believe Eliezer's usage of "I also disclaim" to mean "I include the following disclaimer: " is valid?
Regarding 'protagonist' there is some context for the kind of point Eliezer likes to make about protagonist/story thinking in his Harry Potter fanfic. I don't believe he has expressed the concept coherently as a post yet. (I don't see where you introduced the 'protagonist' word so don't know whether Eliezer read you right. I'm just throwing some background in.)
Replies from: Perplexed, Vaniver↑ comment by Vaniver · 2010-11-28T20:02:09.035Z · LW(p) · GW(p)
Am I correct in inferring that you do not believe Eliezer's usage of "I also disclaim" to mean "I include the following disclaimer: " is valid?
This question is best solved by a dictionary. "I disclaim that I am a blegg" means that I am not a blegg; "Disclaimer: I am a blegg" means that I am a blegg. The use of disclaimer in the second statement is describing the following statement: "I am making a claim that denies something: I am a blegg."
Take home message: Eliezer's double negative means his post has the opposite effect of what I hope he intended.
↑ comment by Jonathan_Graehl · 2010-08-20T22:02:35.459Z · LW(p) · GW(p)
Yes, that was exactly the sense of "ambitious" I intended - the second person sneering one, which when used by oneself, would be more about signaling humility than truth. I see that's not your style.
↑ comment by Unknowns · 2010-08-20T09:27:13.276Z · LW(p) · GW(p)
Even if almost everything you say here is right, it wouldn't mean that there is a high probability that if you are killed in a car accident tomorrow, no one else will think about these things (reflective decision theory and so on) in the future, even people who know nothing about you personally. As Carl Shulman points out, if it is necessary to think about these things it is likely that people will, when it becomes more urgent. So it still wouldn't mean that you are the most important person in human history.
↑ comment by multifoliaterose · 2010-08-20T18:39:52.960Z · LW(p) · GW(p)
I agree with khafra. Your response to my post is distortionary. The statement which you quote was a statement about the reference class of people who believe themselves to be the most important person in the world. The statement which you quote was not a statement about FAI.
Any adequate response to the statement which you quote requires that you engage with the last point that khafra made:
Whether this likelihood ratio is large enough to overcome the evidence on AI-related existential risk and the paucity of serious effort dedicated to combating it is an open question.
You have not satisfactorily addressed this matter.
Replies from: Furcas↑ comment by Furcas · 2010-08-21T15:36:59.066Z · LW(p) · GW(p)
It looks to me like Eliezer gave your post the most generous interpretation possible, i.e. that it actually contained an argument attempting to show that he's deluding himself, rather than just defining a reference class and pointing out that Eliezer fits into it. Since you've now clarified that your post did nothing more than that, there's not much left to do except suggest you read all of Eliezer's posts tagged 'FAI', and this.
comment by Airedale · 2010-08-20T00:37:51.341Z · LW(p) · GW(p)
give the SIAI staff an opportunity to respond to the points which I raise in the present post as well as my two posts titled Existential Risk and Public Relations and Other Existential Risks.
Indeed, given how busy everyone at SIAI has been with the Summit and the academic workshop following it, it is not surprising that there has not been much response from SIAI. I was only involved as an attendee of the Summit, and even I am only now able to find time to sit down and write something in response. At any rate, as a donor and former visiting fellow, I am only loosely affiliated with SIAI, and my comments here are solely my own, although my thoughts are certainly influenced by observations of the organization and conversation with those at SIAI. I don’t have the time/knowledge to address everything in your posts, but I wanted to say a couple of things.
I don’t disagree with you that SIAI has certain public relations problems. (Frankly, I doubt anyone at SIAI would disagree with that.) There is a lot of attention and discussion at SIAI about how to best spread knowledge about existential risks and to avoid sounding like a fringe/doomsday organization in doing so. It’s true that SIAI does consider the development of a general artificial intelligence to be the most serious existential risk facing humanity. But at least from what I have seen, much of SIAI’s current approach is to seed awareness of various existential risks among audiences that are in a position to effectively join the work in decreasing that risk.
Unfortunately, gaining recognition of existential risk is a hugely difficult task. Recent books from leading intellectuals on these issues (Sir Martin Rees’s Our Final Hour and Judge Richard Posner’s Catastrophe) don’t seem to have had very much apparent impact, and their ability to influence the general public is much greater than SIAI’s. But through the Summit and various publications, awareness does seem to be gradually increasing, including among important academics like David Chalmers.
Finally, I wanted to address one particular public relations problem, or at least, public relations issue, that is evident from your criticism so far – that is, there is an (understandable) perception that many observers have that SIAI and Eliezer are essentially synonymous. In the past, this perception may have been largely accurate. I don’t think that it currently holds true, but it definitely continues to persist in many people’s minds.
Given this perception, your primary focus on Eliezer to the exclusion of the other work that SIAI does is understandable. Nor, of course, could anyone possibly deny that Eliezer is an important part of SIAI, as its founder, board member, and prominent researcher. But there other SIAI officers, board members, researchers, and volunteers, and there is other work that SIAI is trying to do. The Summit is probably the most notable example of this. SIAI-affiliated people are also working on spreading knowledge of existential risks and the need to face them in academia and more broadly. The evolution of SIAI into an organization not focused solely on EY and his research is still a work in progress; and the rebranding of the organization as such in the minds of the public has not necessarily kept pace with even that gradual progress.
As for EY having delusions of grandeur, I want to address that, although only briefly, because EY is obviously in a much better position to address any of that if he chooses to. My understanding of the video you linked to in your previous post is that EY is commenting on both 1) his ability to work on FAI research and 2) his desire to work on that research. No matter how high EY’s opinion of his ability, and it doubtless is very high, it seems to me that I have seen comments from him recognizing that there are others with equally high (or even higher) ability, e.g., The Level Above Mine. I have no doubt EY would agree that the pool of those with the requisite ability is very limited. But the even greater obstacle to someone carrying on EY’s work is the combination of that rare ability with the also rare desire to do that research and make it one’s life work. And I think that’s why EY answered the way he did. Indeed, the reference to Michael Vassar, it seems to me, primarily makes sense in terms of the desire axis, since Michael Vassar’s expertise is not in developing FAI himself, although he has other great qualities in terms of SIAI’s current mission of spreading existential risk awareness, etc.
Replies from: Morendil, multifoliaterose↑ comment by Morendil · 2010-08-20T09:43:21.750Z · LW(p) · GW(p)
I don’t disagree with you that SIAI has certain public relations problems.
Speaking from personal experience, the SIAI's somewhat haphazard response to people answering its outreach calls strikes me as a bigger PR problem than Eliezer's personality. The SIAI strikes me as in general not very good at effective collective action (possibly because that's an area where Eliezer's strengths are, as he admits himself, underdeveloped). One thing I'd suggest to correct that is to massively encourage collaborative posts on LW.
Replies from: Airedale↑ comment by Airedale · 2010-08-20T15:23:06.273Z · LW(p) · GW(p)
Agreed. I think that communication and coordination with many allies and supporters has historically been a weak point for SIAI, due to various reasons including overcommitment of some of those tasked with communications, failure to task anyone with developing or maintaining certain new and ongoing relationships, interpersonal skills being among the less developed skill sets among those at SIAI, and the general growing pains of the organization. My impression is that there has been some improvement in this area recently, but there's still room for a lot more.
More collaborative posts on LW would be great to see. There have also been various discussions about workshops or review procedures for top-level posts that seem to have generated at least some interest. Maybe those discussions should just continue in the open thread or maybe it would be appropriate to have a top-level post where people could be invited to volunteer or could find others interested in collaboration, workshops, or the like.
↑ comment by multifoliaterose · 2010-08-20T17:12:05.846Z · LW(p) · GW(p)
Thanks for pointing out "The Level Above Mine." I had not seen it before.
comment by whpearson · 2010-08-20T00:03:24.195Z · LW(p) · GW(p)
I'd like to vote this up as I agree with lots of the points raised, but I am not comfortable with the personal nature of this article. I'd much rather the bits personal to Eliezer be sent via email.
Probably some strange drama avoidance thing on my part. On the other hand I'm not sure Eliezer would have a problem writing a piece like this about someone else.
I've thought to myself that I have read one too many fantasy books as a kid, so the partying metaphor hits home.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T04:46:05.614Z · LW(p) · GW(p)
I'd like to vote this up as I agree with lots of the points raised, but I am not comfortable with the personal nature of this article. I'd much rather the bits personal to Eliezer be sent via email.
I was conflicted about posting in the way that I did precisely for the reason that you describe, but after careful consideration decided that the benefits outweighed the costs, in part because Eliezer does not appear to be reading the private messages that I send him.
Replies from: JamesAndrix↑ comment by JamesAndrix · 2010-08-21T03:15:50.726Z · LW(p) · GW(p)
I would say that given an audience that is mostly not Eliezer. the best way to send a personal message to Eliezer is to address how the community ought to relate to Eliezer.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-21T04:52:46.254Z · LW(p) · GW(p)
Yes, what you say makes sense. If I were to write my post again I would have framed the issues therein somewhat differently.
comment by ata · 2010-08-20T06:09:35.023Z · LW(p) · GW(p)
I'm inclined to think that Eliezer's clear confidence in his own very high intelligence and his apparent high estimation of his expected importance (not the dictionary-definition "expected", but rather, measured as an expected quantity the usual way) are not actually unwarranted, and only violate the social taboo against admitting to thinking highly of one's own intelligence and potential impact on the world, but I hope he does take away from this a greater sense of the importance of a "the customer is always right" attitude in managing his image as a public-ish figure. Obviously the customer is not always right, but sometimes you have to act like they are if you want to get/keep them as your customer... justified or not, there seems to be something about this whole endeavour (including but not limited to Eliezer's writings) that makes people think !!!CRAZY!!! and !!!DOOMSDAY CULT!!!, and even if is really they who are the crazy ones, they are nevertheless the people who populate this crazy world we're trying to fix, and the solution can't always just be "read the sequences until you're rational enough to see why this makes sense".
I realize it's a balance; maybe this tone is good for attracting people who are already rational enough to see why this isn't crazy and why this tone has no bearing on the validity of the underlying arguments, like Eliezer's example of lecturing on rationality in a clown suit. Maybe the people who have a problem with it or are scared off by it are not the sort of people who would be willing or able to help much anyway. Maybe if someone is overly wary of associating with a low-status yet extremely important project, they do not really intuitively grasp its importance or have a strong enough inclination toward real altruism anyway. But reputation will still probably count for a lot toward what SIAI will eventually be able to accomplish. Maybe at the point of hearing and evaluating the arguments, seeming weird or high-self-regard-taboo-violating on the surface level will only screen off people who would not have made important contributions anyway, but it does affect who will get far enough to hear the arguments in the first place. In a world full of physics and math and AI cranks promising imminent world-changing discoveries, reasonably smart people do tend to build up intuitive nonsense-detectors, build up an automatic sense of who's not even worth listening to or engaging with; if we want more IQ 150+ people to get involved in existential risk reduction, then perhaps SIAI needs to make a greater point of seeming non-weird long enough for smart outsiders to switch from "save time by evaluating surface weirdness" mode to "take seriously and evaluate arguments directly" mode.
(Meanwhile, I'm glad Eliezer says "I have a policy of keeping my thoughts on Friendly AI to the object level, and not worrying about how important or unimportant that makes me", and I hope he takes that seriously. But unfortunately, it seems that any piece of writing with the implication "This project is very important, and this guy happens, through no fault of his own, to be one of very few people in the world working on it" will always be read by some people as "This guy thinks he's one of the most important people in the world". That's probably something that can't be changed without downplaying the importance of the project, and downplaying the importance of FAI probably increases existential risk enough that the PR hit of sounding overly self-important to probable non-contributors may be well worth it in the end.)
Replies from: Eliezer_Yudkowsky, Strange7, multifoliaterose, halcyon↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T07:01:08.482Z · LW(p) · GW(p)
there seems to be something about this whole endeavour (including but not limited to Eliezer's writings) that makes people think !!!CRAZY!!! and !!!DOOMSDAY CULT!!!,
Yes, and it's called "pattern completion", the same effect that makes people think "Singularitarians believe that only people who believe in the Singularity will be saved".
Replies from: Emile, ata, timtyler, TheAncientGeek↑ comment by Emile · 2010-08-20T09:59:05.285Z · LW(p) · GW(p)
This is discussed in Imaginary Positions.
↑ comment by ata · 2010-08-20T18:23:07.202Z · LW(p) · GW(p)
I must know, have you actually encountered people who literally think that? I'm really hoping that's a comical exaggeration, but I guess I should not overestimate human brains.
Replies from: timtyler, Eliezer_Yudkowsky↑ comment by timtyler · 2010-08-20T19:07:51.685Z · LW(p) · GW(p)
"It's basically a modern version of a religious belief system and there's no purpose to it, like why, why must we have another one of these things ... you get an afterlife out of it because you'll be on the inside track when the singularity happens - it's got all the trappings of a religion, it's the same thing." - Jaron here.
↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T18:43:32.606Z · LW(p) · GW(p)
I've encountered people who think Singularitarians think that, never any actual Singularitarians who think that.
Replies from: ata, timtyler, XiXiDu↑ comment by ata · 2010-08-20T19:14:44.901Z · LW(p) · GW(p)
Yeah, "people who think Singularitarians think that" is what I meant.
I've actually met exactly one something-like-a-Singularitarian who did think something-like-that — it was at one of the Bay Area meetups, so you may or may not have talked to him, but anyway, he was saying that only people who invent or otherwise contribute to the development of Singularity technology would "deserve" to actually benefit from a positive Singularity. He wasn't exactly saying he believed that the nonbelievers would be left to languish when cometh the Singularity, but he seemed to be saying that they should.
Also, I think he tried to convert me to Objectivism.
Replies from: timtyler↑ comment by timtyler · 2010-08-20T20:13:06.423Z · LW(p) · GW(p)
Technological progress has increased weath inequality a great deal so far.
Machine intelligence probably has the potential to result in enormous weath inequality.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-20T21:19:49.480Z · LW(p) · GW(p)
How, in a post-AGI world, would you define wealth? Computational resources? Matter?
I don't think there's any foundation for speculation on this topic at this time.
Replies from: khafra, Vladimir_Nesov, timtyler↑ comment by khafra · 2010-08-20T21:34:50.392Z · LW(p) · GW(p)
Unless we get a hard-takeoff singleton, which is admittedly the SIAI expectation, there will be massive inequality, with a few very wealthy beings and average income barely above subsistence. Thus saith Robin Hanson, and I've never seen any significant holes poked in that thesis.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-20T21:45:48.016Z · LW(p) · GW(p)
Robin Hanson seems to be assuming that human preferences will, in general, remain in their current ranges. This strikes me as unlikely in the face of technological self-modification.
Replies from: khafra↑ comment by khafra · 2010-08-20T23:20:07.093Z · LW(p) · GW(p)
I've never gotten that impression. What I've gotten is that evolutionary pressures will, in the long term, still exist--even if technological self-modification leads to a population that's 99.99% satisfied to live within strict resource consumption limits, unless they harshly punish defectors the .01% with a drive for replication or expansion will overwhelm the rest within a few millenia, until the average income is back to subsistence. This doesn't depend on human preferences, just the laws of physics and natural selection.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-21T00:41:55.562Z · LW(p) · GW(p)
What evolutionary pressures? Even making the incredible assumption that we will continue to use sequences of genes as a large part of our identities, what's to stop a singleton of some variety from eliminating drives for replication or expansion entirely?
I feel uncomfortable speculating about a post-machine-intelligence future even to this extent; this is not a realm in which I am confident about any proposition. Consequently, I view all confident conclusions with great skepticism.
Replies from: khafra, wedrifid↑ comment by khafra · 2010-08-21T01:21:10.719Z · LW(p) · GW(p)
You're still not getting the breadth and generality of Hanson's model. To use recent LW terminology, it's an anti-prediction.
It doesn't matter whether agents perpetuate their strategies by DNA mixing, binary fission, cellular automata, or cave paintings. Even if all but a tiny minority of posthumans self-modify not to want growth or replication, the few that don't will soon dominate the light-cone. A singleton, like I'd mentioned, is one way to avert this. Universal extinction and harsh, immediate punishment of expansion-oriented agents are the only others I see.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-21T04:46:47.528Z · LW(p) · GW(p)
You (or Robin, I suppose) are just describing a many-agent prisoner's dilemma. If TDT agents beat the dilemma by cooperating with other TDT agents, then any agents that started out with a different decision theory will have long since self-modified to use TDT.
Alternately, if there is no best decision theoretical solution to the prisoner's dilemma, then we probably don't need to worry about surviving to face this problem.
Replies from: khafra↑ comment by khafra · 2010-08-21T18:56:37.035Z · LW(p) · GW(p)
Now, there's a generalized answer. It even covers the possibility of meeting aliens--finding TDT is a necessary condition for reaching the stars. Harsh punishment of inconsiderate expanders might still be required, but there could be a stable equilibrium without ever actually inflicting that punishment. That's a new perspective for me, thanks!
Replies from: WrongBot↑ comment by WrongBot · 2010-08-21T22:25:33.357Z · LW(p) · GW(p)
To be even more general, suppose that there is at least one thing X that is universally necessary for effective superintelligences to function. X might be knowledge of the second law of thermodynamics, TDT, a computational substrate of some variety, or any number of other things. There are probably very many such X's, many of which are entirely non-obvious to any entity that is not itself a superintelligence (i.e. us). Furthermore, there may be at least one thing Y that is universally incompatible with effective superintelligence. Y might be an absolute belief in the existence of the deity Thor or desiring only to solve the Halting Problem using a TM-equivalent. For the Hansonian model to hold, all X's and no Y's must be compatible with the desire and ability to expand and/or replicate.
This argument is generally why I dislike speculating about superintelligences. It is impossible for ordinary humans to have exhaustive (or even useful, partial) knowledge of all X and all Y. The set of all things Y in particular may not even be enumerable.
Replies from: wedrifid↑ comment by wedrifid · 2010-08-21T23:09:06.202Z · LW(p) · GW(p)
We cannot be sure that there are difficulties beyond our comprehension but we are certainly able to assign probabilities to that hypothesis based on what we know. I would be justifiably shocked if something we could call a super-intelligence couldn't be formed based on knowledge that is accessible to us, even if the process of putting the seed of a super-intelligence together is beyond us.
Humans aren't even remotely optimised for generalised intelligence, it's just a trick we picked up to, crudely speaking, get laid. There is no reason that a intelligence of the form "human thinking minus the parts that suck and a bit more of the parts that don't suck" couldn't be created using the knowledge available to us and that is something we can easily place a high probability on. Then you run the hardware at more than 60hz.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-21T23:22:01.392Z · LW(p) · GW(p)
Oh, I agree. We just don't know what self-modifications will be necessary to achieve non-speed-based optimizations.
To put it another way, if superintelligences are competing with each other and self-modifying in order to do so, predictions about the qualities those superintelligences will possess are all but worthless.
Replies from: wedrifid↑ comment by wedrifid · 2010-08-21T01:47:05.706Z · LW(p) · GW(p)
What evolutionary pressures? Even making the incredible assumption that we will continue to use sequences of genes as a large part of our identities, what's to stop a singleton of some variety from eliminating drives for replication or expansion entirely?
Your point is spot on. Competition can not be relied on to produce adaptation if someone wins the competition once and for all.
↑ comment by Vladimir_Nesov · 2010-08-20T21:35:47.578Z · LW(p) · GW(p)
Control, owned by preferences.
↑ comment by timtyler · 2010-08-20T21:28:01.831Z · LW(p) · GW(p)
I wasn't trying to make an especially long-term prediction:
"We saw the first millionaire in 1716, the first billionaire in 1916 - and can expect the first trillionaire within the next decade - probably before 2016."
Replies from: WrongBot↑ comment by WrongBot · 2010-08-20T21:41:32.878Z · LW(p) · GW(p)
Inflation.
The richest person on earth currently has a net worth of $53.5 billion.
The greatest peak net worth in recorded history, adjusted for inflation, was Bill Gates' $101 billion, which was ten years ago. No one since then has come close. A 10-fold increase in <6 years strikes me as unlikely.
In any case, your extrapolated curve points to 2116, not 2016.
I am increasingly convinced that your comments on this topic are made in less than good faith.
Replies from: timtyler↑ comment by timtyler · 2010-08-20T22:59:57.774Z · LW(p) · GW(p)
Yes, the last figure looks wrong to me too - hopefully I will revisit the issue.
Update 2011-05-30: yes: 2016 was a simple math mistake! I have updated the text I was quoting from to read "later this century".
Anyway, the huge modern wealth inequalities are well established - and projecting them into the future doesn't seem especially controversial. Today's winners in IT are hugely rich - and tomorrow's winners may well be even richer. People thinking something like they will "be on the inside track when the singularity happens" would not be very surprising.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-21T00:35:05.964Z · LW(p) · GW(p)
Anyway, the huge modern wealth inequalities are well established - and projecting them into the future doesn't seem especially controversial.
Projecting anything into a future with non-human intelligences is controversial. You have made an incredibly large assumption without realizing it. Please update.
Replies from: timtyler↑ comment by timtyler · 2010-08-21T06:24:12.759Z · LW(p) · GW(p)
If you actually want your questions answered, then money is society's representation of utility - and I think there will probably be something like that in the future - no matter how far out you go. What you may not find further out is "people". However, I wasn't talking about any of that, really. I just meant while there are still money and people with bank accounts around.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-21T20:07:57.029Z · LW(p) · GW(p)
A few levels up, you said,
Machine intelligence probably has the potential to result in enormous weath inequality.
My dispute is with the notion that people with bank accounts and machine intelligence will coexist for a non-trivial amount of time.
Replies from: timtyler↑ comment by timtyler · 2010-08-21T20:23:45.854Z · LW(p) · GW(p)
We have been building intelligent machines for many decades now. If you are talking about something that doesn't yet exist, I think you would be well advised to find another term for it.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-21T22:44:46.880Z · LW(p) · GW(p)
Apologies; I assumed you were using "machine intelligence" as a synonym for AI, as wikipedia does.
Replies from: timtyler↑ comment by timtyler · 2010-08-21T22:48:56.665Z · LW(p) · GW(p)
Machine intelligence *is - more-or-less - a synonym for artificial intelligence.
Neither term carries the implication of human-level intelligence.
Replies from: steven0461↑ comment by steven0461 · 2010-08-21T22:56:34.320Z · LW(p) · GW(p)
We don't really have a good canonical term for "AI or upload".
↑ comment by XiXiDu · 2010-08-20T20:07:16.740Z · LW(p) · GW(p)
I know people who believe the reverse. That only people get fucked by the Singularity who believe in the Singularity.
(Well, not exactly, but more would hint at forbidden knowledge.)
ETA
Whoops timtyler was faster than me getting at that point.
↑ comment by timtyler · 2010-08-20T17:09:18.251Z · LW(p) · GW(p)
The outside view of the pitch:
- DOOM! - and SOON!
- GIVE US ALL YOUR MONEY;
- We'll SAVE THE WORLD; you'll LIVE FOREVER in HEAVEN;
- Do otherwise and YOU and YOUR LOVED ONES will suffer ETERNAL OBLIVION!
Maybe there are some bits missing - but they don't appear to be critical components of the pattern.
Indeed, this time there are some extra features not invented by those who went before - e.g.:
- We can even send you to HEAVEN if you DIE a sinner - IF you PAY MORE MONEY to our partner organisation.
↑ comment by CarlShulman · 2010-08-20T17:16:31.204Z · LW(p) · GW(p)
Do otherwise and YOU and YOUR LOVED ONES will suffer ETERNAL OBLIVION.
This one isn't right, and is a big difference between religion and threats like extinction-level asteroids or AI disasters: one can free-ride if that's one's practice in collective action problems.
Also: Rapture of the Nerds, Not
Replies from: timtyler, timtyler↑ comment by timtyler · 2011-05-14T13:29:51.303Z · LW(p) · GW(p)
It's now official!
http://en.wikipedia.org/wiki/Rapture_of_the_Nerds
...now leads to a page that is extremely similar to:
http://en.wikipedia.org/wiki/Technological_singularity
...though - curiously - there are some differences between the two pages (count the words in the first sentence). [update: this difference was apparently due to the page being simultaneously cached and updated.]
Comparisons with The Rapture, are insightful, IMHO. I see no good reason to deny them.
It turns out that ETERNAL OBLIVION is too weak. The community now has the doctrine of ETERNAL DAMNATION. For details, see here.
Replies from: ArisKatsaris, AdeleneDawner↑ comment by ArisKatsaris · 2011-05-15T09:04:48.878Z · LW(p) · GW(p)
People need to stop being coy. If you know a difference, just spit it out, don't force people to jump through meaningless hoops like "count the words in the first sentence".
Downvoted for wasting people's time with coyness because of a false belief caused by a cache issue.
↑ comment by AdeleneDawner · 2011-05-14T20:22:46.591Z · LW(p) · GW(p)
Uh, no it doesn't, and in fact this appears to be an actual lie (EDIT: Nope, cache issue) rather than the RotN page being changed since you checked it.
Replies from: timtyler, None, CuSithBell, CuSithBell↑ comment by timtyler · 2011-05-14T20:33:13.663Z · LW(p) · GW(p)
Before you start flinging accusations around, perhaps check, reconsider - or get a second opinion?
To clarify, for me, http://en.wikipedia.org/wiki/Rapture_of_the_Nerds still gives me:
Replies from: MorendilTechnological singularity
From Wikipedia, the free encyclopedia
(Redirected from Rapture of the Nerds)
↑ comment by Morendil · 2011-05-14T20:51:00.475Z · LW(p) · GW(p)
Maybe Adelene meant that "now" is an untruth, in that it implies a change occurring between the timestamp of the comment you reply to and the reply itself. A truthful observation would "RotN has always redirected to a page that, etc."
Replies from: timtyler↑ comment by timtyler · 2011-05-14T20:56:41.322Z · LW(p) · GW(p)
The implication that you refer to is based on a simple misunderstanding of my comment - and does not represent a "lie" on my part.
Replies from: AdeleneDawner↑ comment by AdeleneDawner · 2011-05-14T21:04:01.921Z · LW(p) · GW(p)
Replies from: CuSithBell, timtyler↑ comment by CuSithBell · 2011-05-14T21:09:08.936Z · LW(p) · GW(p)
Wait, did you really mean "no, the page has always redirected there" instead of "no, the page does not, in fact, redirect there"?
Replies from: AdeleneDawner↑ comment by AdeleneDawner · 2011-05-14T21:22:26.587Z · LW(p) · GW(p)
"A page that is extremely similar to X" implies "a page that is not X", assuming normal use of the English language. The rapture of the nerds page has always led to the technological singularity page, and the technological singularity page is not a page that is not the technological singularity page.
Reading the relevant comment with the strictest possible definitions of all the terms, it's technically correct, but the way that the comment is structured implies an interpretation other than the one that is true, and it could easily have been structured in a way that wouldn't imply such an interpretation.
Replies from: CuSithBell, timtyler↑ comment by CuSithBell · 2011-05-14T21:26:48.058Z · LW(p) · GW(p)
Huh. Put like that, I guess I understand now, but it seems as though your refutation could also have been more clear on that point. Thanks for the disentangling!
↑ comment by timtyler · 2011-05-15T06:55:21.486Z · LW(p) · GW(p)
The pages are subtly different - in the way I described in detail in my original comment. Count the words in the first sentence - the one starting: "A technological singularity is..." to see the difference.
My guess is that a Wikipedia "redirect" allows for a prefix header to be prepended, which would explain the difference.
Replies from: AdeleneDawner↑ comment by AdeleneDawner · 2011-05-15T07:15:01.110Z · LW(p) · GW(p)
All four versions of the page - redirect and not, secure and not - start with the same two sentences for me: "A technological singularity is a hypothetical event. It will occur if technological progress becomes so rapid and the growth of super-human intelligence so great that the future (after the singularity) becomes qualitatively different and harder to predict."
I suspect you have a cache issue.
Replies from: timtyler↑ comment by timtyler · 2011-05-15T07:39:40.502Z · LW(p) · GW(p)
That seems likely. I used http://hidemyass.com/proxy/ - and it gives a more consistent picture.
Replies from: None, AdeleneDawner↑ comment by [deleted] · 2011-05-15T07:51:58.824Z · LW(p) · GW(p)
Much time could have been saved had you copied and pasted the two diverging sentences rather than asking people to count the words. For indeed there was a recent change in the page, and if this was the source of the difference, then had you provided the exact sentences then the cause could have been determined quickly, avoiding a lot of back and forth.
Copying and pasting from a comparison, the slightly earlier version is:
A '''technological singularity''' is a hypothetical event occurring when technological progress becomes so rapid and the growth of super-human intelligence is so great that the future after the singularity becomes qualitatively different and harder to predict.
The slightly more recent version is:
A '''technological singularity''' is a hypothetical event.
The rest of the earlier sentence was split off into separate sentences.
Replies from: wedrifid↑ comment by wedrifid · 2011-05-15T08:51:11.123Z · LW(p) · GW(p)
Much time could have been saved had you copied and pasted the two diverging sentences rather than asking people to count the words.
Not that I am necessarily one to talk but much time could have been saved if nobody argued about such an irrelevant technicality. ;)
Replies from: None↑ comment by [deleted] · 2011-05-15T10:38:07.877Z · LW(p) · GW(p)
It was the key and only evidence in an accusation of lying, which is a pretty damn serious accusation that should neither be taken lightly nor made lightly. The evidence was small but the role it played in the accusation made it important. If your point is that the accuser should have held their tongue so to speak, you may be right. But they didn't, and so the question took on importance.
Replies from: wedrifid↑ comment by wedrifid · 2011-05-15T10:44:14.025Z · LW(p) · GW(p)
It was the key and only evidence in an accusation of lying, which is a pretty damn serious accusation that should neither be taken lightly nor made lightly. The evidence was small but the role it played in the accusation made it important. If your point is that the accuser should have held their tongue so to speak, you may be right. But they didn't, and so the question took on importance.
Yes, responding to accusations of lying is important. Making them, not so much. :)
↑ comment by AdeleneDawner · 2011-05-15T07:44:45.680Z · LW(p) · GW(p)
*nods* *edits ancestral comment*
↑ comment by [deleted] · 2011-05-14T22:04:26.747Z · LW(p) · GW(p)
Since it redirects, the relevant history page is the technological singularity history page. Namely, this one. And there was indeed a recent change to the first sentence. See for example this comparison.
↑ comment by CuSithBell · 2011-05-14T20:44:03.603Z · LW(p) · GW(p)
I'm seeing the redirect on the non-secure version.
↑ comment by CuSithBell · 2011-05-14T20:40:58.252Z · LW(p) · GW(p)
I'm seeing the same thing as timtyler.
↑ comment by timtyler · 2010-08-20T17:21:29.606Z · LW(p) · GW(p)
Do otherwise and YOU and YOUR LOVED ONES will suffer ETERNAL OBLIVION.
This one isn't right, and is a big difference between religion and threats like extinction-level asteroids or AI disasters: one can free-ride if that's one's practice in collective action problems.
It is true that, this time around there are probabilities attached to some of the outcomes - but the basic END OF THE WORLD rescue pitch remains essentially intact.
I note that some have observed that worse fates may await those who get their priorities wrong at the critical stage.
↑ comment by cousin_it · 2010-08-20T18:45:43.472Z · LW(p) · GW(p)
I don't understand why downvote this. It does sound like an accurate representation of the outside view.
Replies from: Vladimir_Nesov, Nick_Tarleton, Unknowns, None, Perplexed, Nick_Tarleton↑ comment by Vladimir_Nesov · 2010-08-20T21:23:43.864Z · LW(p) · GW(p)
This whole "outside view" methodology, where you insist on arguing from ignorance even where you have additional knowledge, is insane (outside of avoiding the specific biases such as planning fallacy induced by making additional detail available to your mind, where you indirectly benefit from basing your decision on ignorance).
In many cases outside view, and in particular reference class tennis, is a form of filtering the evidence, and thus "not technically" lying, a tool of anti-epistemology and dark arts, fit for deceiving yourself and others.
Replies from: timtyler↑ comment by timtyler · 2010-08-20T21:36:12.502Z · LW(p) · GW(p)
This whole "outside view" methodology, where you insist on arguing from ignorance even where you have additional knowledge, is insane [...]
Perhaps compare a doomsday cult with a drug addict:
The outside view (e.g. of family and practitioners) looks one way - while the inside view often looks pretty different.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T21:40:33.434Z · LW(p) · GW(p)
Perhaps compare a doomsday cult with a drug addict:
The outside view (e.g. of family and practitioners) looks one way - while the inside view often looks pretty different.
That's not what "inside view" means. The way you seem to intend it, it admittedly is a useless tool, but having it as an option in the false dichotomy together with reference class tennis is transparently disingenuous (or stupid).
Replies from: timtyler↑ comment by timtyler · 2010-08-20T21:57:24.287Z · LW(p) · GW(p)
You seem to be thinking about reference class forecasting. In that particular case, I just meant looking from the outside - but the basic idea is much the same. Doomsday organisations have a pattern. The SIAI isn't an ordinary one - but it shares many of the same basic traits with them.
↑ comment by Nick_Tarleton · 2010-08-20T21:41:21.806Z · LW(p) · GW(p)
We all already know about this pattern match. Its reiteration is boring and detracts from the conversation.
Replies from: timtyler↑ comment by timtyler · 2011-05-14T16:09:50.577Z · LW(p) · GW(p)
We all already know about this pattern match. Its reiteration is boring and detracts from the conversation.
If this particular critique has been made more clearly elsewhere, perhaps let me know, and I will happily link to there in the future.
Update 2011-05-30: There's now this recent article: The “Rapture” and the “Singularity” Have Much in Common - which makes a rather similar point.
↑ comment by [deleted] · 2011-05-14T22:10:03.836Z · LW(p) · GW(p)
Given that a certain fraction of comments are foolish, you can expect that an even larger fraction of votes are foolish, because there are fewer controls on votes (e.g. a voter doesn't risk his reputation while a commenter does).
Replies from: rhollerith_dot_com, timtyler↑ comment by RHollerith (rhollerith_dot_com) · 2011-05-15T02:54:33.712Z · LW(p) · GW(p)
Which is why Slashdot (which was a lot more worthwhile in the past than it is now) introduced voting on how other people vote (which Slashdot called metamoderation). Worked pretty well: the decline of Slashdot was mild and gradual compared to the decline of almost every other social site that ever reached Slashdot's level of quality.
↑ comment by timtyler · 2011-05-30T08:23:31.572Z · LW(p) · GW(p)
Yes: votes should probably not be anonymous - and on "various other" social networking sites, they are not.
Replies from: rhollerith_dot_com↑ comment by RHollerith (rhollerith_dot_com) · 2011-05-30T17:01:42.735Z · LW(p) · GW(p)
Metafilter, for one. It is hard for an online community to avoid becoming worthless, but Metafilter has avoided that for 10 years.
↑ comment by Perplexed · 2010-08-20T19:12:44.385Z · LW(p) · GW(p)
Perhaps downvoted for suggesting that the salvation-for-cash meme is a modern one. I upvoted, though.
Replies from: timtyler↑ comment by Nick_Tarleton · 2010-08-20T21:38:42.482Z · LW(p) · GW(p)
We all already know about this pattern match. Reiterating it is boring and detracts from the conversation, and I downvote any such comment I see.
↑ comment by TheAncientGeek · 2015-06-17T12:51:03.905Z · LW(p) · GW(p)
Pattern completion isnt always wrong.
↑ comment by Strange7 · 2010-08-20T15:33:40.593Z · LW(p) · GW(p)
if we want more IQ 150+ people to get involved in existential risk reduction, then perhaps SIAI needs to make a greater point of seeming non-weird long enough for smart outsiders to switch from "save time by evaluating surface weirdness" mode to "take seriously and evaluate arguments directly" mode.
What about less-smart people? I mean, self-motivated idealistic genius nerds are certainly necessary for the core functions of programming an FAI, but any sufficiently large organization also needs a certain number of people who mostly just file paperwork, follow orders, answer the phone, etc. and things tend to work out more efficiently when those people are primarily motivated by the organization's actual goals rather than it's willingness to pay.
Replies from: HughRistik↑ comment by HughRistik · 2010-08-20T19:51:01.291Z · LW(p) · GW(p)
Good point. It's the people in the <130 range that SIAI needs to figure out how to attract. That's where you find people like journalists and politicians.
Replies from: wedrifid↑ comment by wedrifid · 2010-08-31T08:19:37.326Z · LW(p) · GW(p)
It's the people in the <130 range that SIAI needs to figure out how to attract. That's where you find people like journalists and politicians.
You also find a lot of journalists and politicians in the 130 to 160 range but the important thing with those groups is that they optimise their beliefs and expressions thereof for appeal to a < 130 range audience.
↑ comment by multifoliaterose · 2010-12-12T08:23:25.521Z · LW(p) · GW(p)
I'm inclined to think that Eliezer's clear confidence in his own very high intelligence and his apparent high estimation of his expected importance (not the dictionary-definition "expected", but rather, measured as an expected quantity the usual way) are not actually unwarranted, and only violate the social taboo against admitting to thinking highly of one's own intelligence and potential impact on the world
Leaving aside the question of whether such apparently strong estimation is warranted in the case at hand; I would suggest that there's a serious possibility that the social taboo that you allude to is adaptive; that having a very high opinion of oneself (even if justified) is (on account of the affect heuristic) conducive to seeing a halo around oneself, developing overconfidence bias, rejecting criticisms prematurely, etc. leading to undesirable epistemological skewing.
Meanwhile, I'm glad Eliezer says "I have a policy of keeping my thoughts on Friendly AI to the object level, and not worrying about how important or unimportant that makes me", and I hope he takes that seriously.
Same here.
it seems that any piece of writing with the implication "This project is very important, and this guy happens, through no fault of his own, to be one of very few people in the world working on it" will always be read by some people as "This guy thinks he's one of the most important people in the world".
It's easy to blunt this signal.
Suppose that any of:
A billionaire decided to devote most of his or her wealth to funding Friendly AI research.
A dozen brilliant academics became interested in and started doing Friendly AI research.
The probability of Friendly AI research leading to a Friendly AI is sufficiently low so that another existential risk reduction effort (e.g. pursuit of stable whole brain emulation) is many orders of magnitude more cost-effective at reducing existential risk than Friendly AI research.
Then the Eliezer would not (by most estimations) be the highest utilitarian expected value human in the world. If he were to mention such possibilities explicitly this would greatly mute the undesired connotations.
Replies from: Eliezer_Yudkowsky↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-12-12T08:48:46.411Z · LW(p) · GW(p)
If I thought whole-brain emulation were far more effective I would be pushing whole-brain emulation, FOR THE LOVE OF SQUIRRELS!
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-12-12T09:26:23.082Z · LW(p) · GW(p)
Good to hear from you :-)
My understanding is that at present there's a great deal of uncertainty concerning how future advanced technologies are going to develop (I've gotten an impression that e.g. Nick Bostrom and Josh Tenenbaum hold this view). In view of such uncertainty, it's easy to imagine new data emerging over the next decades that makes it clear that pursuit of whole-brain emulation (or some currently unimagined strategy) is a far more effective strategy for existential risk reduction than Friendly AI research.
At present it looks to me like a positive singularity is substantially more likely to occur starting with whole-brain emulation than with Friendly AI.
Various people have suggested to me that initially pursuing Friendly AI might have higher expected value on the chance that it turns out to be easy. So I could imagine that it's rational for you personally to focus your efforts on Friendly AI research (EDIT: even if I'm correct in my estimation in the above point). My remarks in the grandparent above were not intended as a criticism of your strategy.
I would be interested in hearing more about your own thinking about the relative feasibility of Friendly AI vs. stable whole-brain emulation and current arbitrage opportunities for existential risk reduction, whether on or off the record.
↑ comment by ata · 2010-12-12T10:45:53.150Z · LW(p) · GW(p)
At present it looks to me like a positive singularity is substantially more likely to occur starting with whole-brain emulation than with Friendly AI.
That's an interesting claim, and you should post your analysis of it (e.g. the evidence and reasoning that you use to form the estimate that a positive singularity is "substantially more likely" given WBE).
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-12-12T18:09:40.931Z · LW(p) · GW(p)
There's a thread with some relevant points (both for and against) titled Hedging our Bets: The Case for Pursuing Whole Brain Emulation to Safeguard Humanity's Future. I hadn't looked at the comments until just now and still have to read them all; but see in particular a comment by Carl Shulman.
After reading all of the comments I'll think about whether I have something to add beyond them and get back to you.
Replies from: CarlShulman↑ comment by CarlShulman · 2010-12-14T15:07:15.906Z · LW(p) · GW(p)
You may want to read this paper I presented at FHI. Note that there's a big difference between the probability of risk conditional on WBE coming first or AI coming first and marginal impact of effort. In particular some of our uncertainty is about logical facts about the space of algorithms and technology landscape, and some of it is about the extent and effectiveness of activism/intervention.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-12-14T20:42:30.694Z · LW(p) · GW(p)
Thanks for the very interesting reference! Is it linked on the SIAI research papers page? I didn't see it there.
Note that there's a big difference between the probability of risk conditional on WBE coming first or AI coming first and marginal impact of effort.
I appreciate this point which you've made to me previously (and which appears in your comment that I linked above!).
↑ comment by Vladimir_Nesov · 2010-12-13T09:28:54.435Z · LW(p) · GW(p)
At present it looks to me like a positive singularity is substantially more likely to occur starting with whole-brain emulation than with Friendly AI.
Do you mean that the role of ems is in developing FAI faster (as opposed to biological-human-built FAI), or are you thinking of something else? If ems merely speed time up, they don't change the shape of FAI challenge much, unless (and to the extent that) we leverage them in a way we can't for the human society to reduce existential risk before FAI is complete (but this can turn out worse as well, ems can well launch the first arbitrary-goal AGI).
Replies from: ata, CarlShulman, multifoliaterose↑ comment by ata · 2010-12-13T22:32:25.756Z · LW(p) · GW(p)
but this can turn out worse as well, ems can well launch the first arbitrary-goal AGI
That's the main thing that's worried me about the possibility of ems coming first. But it depends on who is able to upload and who wants to, I suppose. If an average FAI researcher is more likely to upload, increase their speed, and possibly make copies of themselves than an average non-FAI AGI researcher, then it seems like that would be a reduction in risk.
I'm not sure whether that would be the case — a person working on FAI is likely to consider their work to be a matter of life and death, and would want all the speed increases they could get, but an AGI researcher may feel the same way about the threat to their career and status posed by the possibility of someone else getting to AGI first. And if uploading is very expensive at first, it'll only be the most well-funded AGI researchers (i.e. not SIAI and friends) who will have access to it early on and will be likely to attempt it (if it provides enough of a speed increase that they'd consider it to be worth it).
(I originally thought that uploading would be of little to no help in increasing one's own intelligence (in ways aside from thinking the same way but faster), since an emulation of a brain isn't automatically any more comprehensible than an actual brain, but now I can see a few ways it could help — the equivalent of any kind of brain surgery could be attempted quickly, freely, and reversibly, and the same could be said for experimenting with nootropic-type effects within the emulation. So it's possible that uploaded people would get somewhat smarter and not just faster. Of course, that's only soft self-improvement, nowhere near the ability to systematically change one's cognition at the algorithmic level, so I'm not worried about an upload bootstrapping itself to superintelligence (as some people apparently are). Which is good, since humans are not Friendly.)
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-12-14T03:55:15.851Z · LW(p) · GW(p)
There's a lot to respond to here. Some quick points:
It should be born in mind that greatly increased speed and memory may by themselves strongly affect a thinking entity. I imagine that if I could think a million times as fast I would think a lot more carefully about my interactions with the outside world than I do now.
I don't see any reason to think that SIAI will continue to be the only group thinking about safety considerations. If nothing else, SIAI or FHI can raise awareness of the dangers of AI within the community of AI researchers.
Assuming that brain uploads precede superhuman artificial intelligence, it would obviously be very desirable to have the right sort of human uploaded first.
I presently have a very dim view as to the prospects for modern day humans developing Friendly AI. This skepticism is the main reason why I think that pursuing whole-brain emulations first is more promising. See the comment by Carl that I mentioned in response to Vladimir Nesov's question. Of course, my attitude on this point is subject to change with incoming evidence.
↑ comment by CarlShulman · 2010-12-14T15:01:14.827Z · LW(p) · GW(p)
Sped-up ems have slower computers relative to their thinking speed. If Moore's Law of Mad Science means that increasing computing power allows researchers to build AI with less understanding (and thus more risk of UFAI), then a speedup of researchers relative to computing speed makes it more likely that the first non-WBE AIs will be the result of a theory-intensive approach with high understanding. Anders Sandberg of FHI and I are working on a paper exploring some of these issues.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-12-14T21:20:47.709Z · LW(p) · GW(p)
This argument lowers the estimate of danger, but AIs developed on relatively slow computers are not necessarily theory-intense, could also be coding-intense, which leads to UFAI. And theory-intense doesn't necessarily imply adequate concern about AI's preference.
↑ comment by multifoliaterose · 2010-12-14T03:08:41.811Z · LW(p) · GW(p)
My idea here is the same as the one that Carl Shulman mentioned in a response to one of your comments from nine months ago.
↑ comment by timtyler · 2010-12-13T20:59:12.000Z · LW(p) · GW(p)
it's easy to imagine new data emerging over the next decades that makes it clear that pursuit of whole-brain emulation (or some currently unimagined strategy) is a far more effective strategy for existential risk reduction than Friendly AI research.
Not with the whole-brain emulation for me. That would represent an incredible miracle, IMO.
At present it looks to me like a positive singularity is substantially more likely to occur starting with whole-brain emulation than with Friendly AI.
The other way around, surely! Machine intelligence is much more likely to come first, and be significant. Few carried on with bird scanning and emulation once we had working planes. Essentially, the whole project went nowhere.
Replies from: JoshuaZ↑ comment by JoshuaZ · 2010-12-13T21:26:42.155Z · LW(p) · GW(p)
In general, when one individual asserts that something seems very likely to them it isn't helpful to simply assert that the opposite seems extremely likely without giving some minimal reasoning for why you think that will be the case.
Replies from: timtyler↑ comment by timtyler · 2010-12-13T21:35:41.751Z · LW(p) · GW(p)
For more, see: http://alife.co.uk/essays/against_whole_brain_emulation/
Your comment is a puzzling one. You are apparently advising me to offer more assistance. Right - but isn't that up to me? You can't possibly analyse the budgets I have to allocate for such things remotely.
Replies from: JoshuaZ↑ comment by JoshuaZ · 2010-12-13T22:19:02.567Z · LW(p) · GW(p)
Let me put it this way then: Most LW readers don't like reading unproductive conversations. And it is hard to get more unproductive than one person saying "I believe in X!" and another saying "Yeah, well I believe ~X, so there!" You are welcome to do that, but don't be surprised if the rest of us decide to vote down such comments as things we don't want.
↑ comment by halcyon · 2012-06-17T07:45:20.378Z · LW(p) · GW(p)
In cases like this, I find ethics grounded in utilitarianism to be despicably manipulative positions. You are not treating people as rational agents, but pandering to their lack of virtue so as to recruit them as pawns in your game. If that's how you're going to play, why not manufacture evidence in support of your position if you're Really Sure your assessment is accurate? A clear line of division between "pandering: acceptable" & "evidence manufacture: unacceptable" is nothing but a temporary, culturally contingent consensus caring nothing for reason or consistency. To predict the future, see the direction in which the trend is headed.
No, I would scrupulously adhere to a position of utmost sincerity. Screw the easily offended customers. If this causes my downfall, so be it. That outcome is acceptable because personally, if my failure is caused by honesty and goodwill rather than incompetence, I would question if such a world is worth saving to begin with. I mean, if that is what this enlightened society is like and wants to be like, then I can rather easily imagine our species eventually ending up as the aggressors in one of those alien invasion movies like Independence Day. I keep wondering why, if they evolved in a symbiotic ecosystem analogous to ours, one morally committed individual among their number didn't wipe out their own race and rid the galaxy of this aimless, proliferating evil. It'd be better still to let them be smothered peacefully under their own absence of self-reflection and practice of rewarding corruption, without going out of your way to help them artificially reach a position of preeminence from which to bully others.
comment by grouchymusicologist · 2010-08-20T03:02:38.592Z · LW(p) · GW(p)
A number of people have mentioned the seemingly-unimpeachable reputation of the Future of Humanity Institute without mentioning that its director, Nick Bostrom, fairly obviously has a high opinion of Eliezer (e.g., he invited him to contribute not one but two chapters to the volume on Global Catastrophic Risks). Heuristically, if I have a high opinion of Bostrom and the FHI project, that raises my opinion of Eliezer and decreases the probability of Eliezer-as-crackpot.
comment by Aleksei_Riikonen · 2010-08-20T01:34:11.819Z · LW(p) · GW(p)
Well, in the category of "criticisms of SIAI and/or Eliezer", this text is certainly among the better ones. I could see this included on a "required reading list" of new SIAI employees or something.
But since we're talking about a Very Important Issue, i.e. existential risks, the text might have benefited from some closing warnings, that whatever people's perceptions of SIAI, it's Very Important that they don't neglect being very seriously interested in existential risks because of issues that they might perceive a particular organization working on the topic to have (and that it might also actually have, but that's not my focus in this comment).
I.e. if people think SIAI sucks and shouldn't be supported, they should anyway be very interested in supporting the Future of Humanity Institute at Oxford, for example. Otherwise they're demonstrating very high levels of irrationality, and with regard to SIAI, are probably just looking for plausible-sounding excuses to latch onto for why they shouldn't pitch in.
Not to say that the criticism you presented mightn't be very valid (or not; I'm not really commenting on that here), but it would be very important for people to first take care that they're contributing to the reduction of existential risks in some way, and then consider to what extent exactly a particular organization such as SIAI might be doing a sub-optimal job (since they can choose a more clear-cut case of an excellent organization for their initial contribution, i.e. Bostrom's FHI as mentioned above).
Replies from: timtyler↑ comment by timtyler · 2010-08-20T18:18:34.859Z · LW(p) · GW(p)
Very important to you - maybe. You aware, I presume, that for most people, the end of the world is not high on their agenda. It is evidently not "very important" to them - or they would spend more time on it.
Basic biology explains this phenomenon, as I have previously explained:
"Organisms can be expected to concentrate on producing offspring - not indulging paranoid fantasies about their whole species being wiped out!"
Replies from: Aleksei_Riikonen↑ comment by Aleksei_Riikonen · 2010-08-21T12:18:59.398Z · LW(p) · GW(p)
Are you aware that most species that have ever lived have indeed been wiped out? Not thinking about such possibilities worked well for them, eh?
EDIT: And of course we can also present scholarly analyses of why extinction in the case of our species is not particularly unlikely: http://www.nickbostrom.com/fut/evolution.html
Replies from: timtyler↑ comment by timtyler · 2010-08-21T13:06:43.645Z · LW(p) · GW(p)
Are you aware that most species that have ever lived have indeed been wiped out? Not thinking about such possibilities worked well for them, eh?
If you mean to imply that thinking about such possibilities would have helped them all to survive, then that doesn't seem right. If new species keep being born (as happens naturally), other ones seem practically bound to die out - due to resource competiton in a limited ecosystem. Hypothetical contemplation of their own species-level mortality seems unlikely to have helped - and might well have hindered their survival chances.
Replies from: Aleksei_Riikonen↑ comment by Aleksei_Riikonen · 2010-08-21T14:42:52.926Z · LW(p) · GW(p)
Thinking about such things is the necessary first step to preventing such new species from arising that would make you extinct. So yes, if they had thought about these things competently enough, and otherwise been competent enough, it would have enabled them to survive.
Doesn't seem very smart of you to argue against thinking. If you don't think, you're certainly even more screwed than with thinking.
Replies from: timtyler↑ comment by timtyler · 2010-08-21T18:26:20.632Z · LW(p) · GW(p)
The "most species that have ever lived" that you mentioned were not capable of preventing new species from arising - because that happens naturally all the time. If you introduce this hypothetical, it seems as though you have to abandon your original argument.
It is thinking too much about events that you have little control over that can be bad.
Also, in biology, more thinking than normal is not good, on average - thoughts are costly and there is an economic tradeoff.
comment by [deleted] · 2010-08-20T20:57:17.884Z · LW(p) · GW(p)
I don't think there's any point doing armchair diagnoses and accusing people of delusions of grandeur. I wouldn't go so far as to claim that Eliezer needs more self-doubt, in a psychological sense. That's an awfully personal statement to make publicly. It's not self-confidence I'm worried about, it's insularity.
Here's the thing. The whole SIAI project is not publicly affiliated with (as far as I've heard) other, more mainstream institutions with relevant expertise. Universities, government agencies, corporations. We don't have guest posts from Dr. X or Think Tank Fellow Y. The ideas related to friendly AI and existential risk have not been shopped to academia or evaluated by scientists in the usual way. So they're not being tested stringently enough.
It's speculative. It feels fuzzy to me -- I'm not an expert in AI, but I have some education in math, and things feel fuzzy around here.
If you want to claim you're working on a project that may save the world, fine. But there's got to be more to show for it, sooner or later, than speculative essays. At the very least, people worried about unfriendly AI will have to gather data and come up with some kind of statistical study that gives evidence of a threat! Look at climate science. For all the foibles and challenges of the climate change movement, those people actually gather data, create prediction models, predict the results of mitigating policies -- it works more or less like science.
If I'm completely off base here and SIAI is going to get to the science soon, I apologize, and I'll shut up about this for a while.
But look. All this advice about the "sin of underconfidence" is all very well (and actually I've taken it to heart somewhat.) But if you're going to go test your abilities, then test them. Against skeptics. Against people who'll look at you like you're a rotten fish if you don't have a graduate degree. Get something about FAI peer-reviewed or published by a reputable press. Show us something.
Sorry to be so blunt. It's just that I want this to be something. And I have my doubts because there's doesn't seem to be enough in this floating world in the way of unmistakable, concrete achievement.
Replies from: steven0461, WrongBot, Morendil, multifoliaterose, wedrifid, timtyler, wedrifid, Perplexed↑ comment by steven0461 · 2010-08-20T21:48:22.809Z · LW(p) · GW(p)
The whole SIAI project is not publicly affiliated with (as far as I've heard) other, more mainstream institutions with relevant expertise. Universities, government agencies, corporations. We don't have guest posts from Dr. X or Think Tank Fellow Y.
According to the about page, LW is brought to you by the Future of Humanity Institute at Oxford University. Does this count? Many Dr. Xes have spoken at the Singularity Summits.
At the very least, people worried about unfriendly AI will have to gather data and come up with some kind of statistical study that gives evidence of a threat!
It's not clear how one would use past data to give evidence for or against a UFAI threat in any straightforward way. There's various kinds of indirect evidence that could be presented, and SIAI has indeed been trying more in the last year or two to publish articles and give conference talks presenting such evidence.
Points that SIAI would do better if it had better PR, had more transparency, published more in the scientific literature, etc., are all well-taken, but these things use limited resources, which to me makes it sound strange to use them as arguments to direct funding elsewhere.
Replies from: None, torekp, multifoliaterose↑ comment by [deleted] · 2010-08-20T22:06:58.236Z · LW(p) · GW(p)
My post was by way of explaining why some people (including myself) doubt the claims of SIAI. People doubt claims when, compared to other claims, they're not justified as rigorously, or haven't met certain public standards. Why do I agree with the main post that Eliezer isn't justified in his opinion of his own importance (and SIAI's importance)? Because there isn't (yet) a lot beyond speculation here.
I understand about limited resources. If I were trying to run a foundation like SIAI, I might do exactly what it's doing, at first, and then try to get the academic credentials. But as an outside person, trying to determine: is this worth my time? Is this worth further study? Is this a field I could work in? Is this worth my giving away part of my (currently puny) income in donations? I'm likely to hold off until I see something stronger.
And I'm likely to be turned off by statements with a tone that assumes anyone sufficiently rational should already be on board. Well, no! It's not an obvious, open-and shut deal.
What if there were an organization comprised of idealistic, speculative types, who, unknowingly, got themselves to believe something completely false based on sketchy philosophical arguments? They might look a lot like SIAI. Could an outside observer distinguish fruitful non-mainstream speculation from pointless non-mainstream speculation?
Replies from: timtyler↑ comment by timtyler · 2010-08-21T06:59:06.460Z · LW(p) · GW(p)
I think they are working on their "academic credentials":
http://singinst.org/grants/challenge
...lists some 13 academic papers under various stages of development.
↑ comment by torekp · 2010-08-22T01:23:12.063Z · LW(p) · GW(p)
Thanks for that last link. The paper on Changing the frame of AI futurism is extremely relevant to this series of posts.
↑ comment by multifoliaterose · 2010-09-03T19:54:27.820Z · LW(p) · GW(p)
According to the about page, LW is brought to you by the Future of Humanity Institute at Oxford University. Does this count?
I contacted Nick Bostrom about this and he said that there’s no formal relationship between FHI and SIAI.
Points that SIAI would do better if it had better PR, had more transparency, published more in the scientific literature, etc., are all well-taken, but these things use limited resources, which to me makes it sound strange to use them as arguments to direct funding elsewhere.
↑ comment by WrongBot · 2010-08-20T21:13:15.886Z · LW(p) · GW(p)
Here's the thing. The whole SIAI project is not publicly affiliated with (as far as I've heard) other, more mainstream institutions with relevant expertise.
LessWrong is itself a joint project of the SIAI and the Future of Humanity Institute at Oxford. Researchers at the SIAI have published these academic papers. The Singularity Summit's website includes a lengthy list of partners, including Google and Scientific American.
The SIAI and Eliezer may not have done the best possible job of engaging with the academic mainstream, but they haven't done a terrible one either, and accusations that they aren't trying are, so far as I am able to determine, factually inaccurate.
Replies from: Perplexed, None, wedrifid↑ comment by Perplexed · 2010-08-21T17:30:53.342Z · LW(p) · GW(p)
Researchers at the SIAI have published these academic papers.
But those don't really qualify as "published academic papers" in the sense that those terms are usually understood in academia. They are instead "research reports" or "technical reports".
The one additional hoop that these high-quality articles should pass through before they earn the status of true academic publications is to actually be published - i.e. accepted by a reputable (paper or online) journal. This hoop exists for a variety of reasons, including the claim that the research has been subjected to at least a modicum of unbiased review, a locus for post-publication critique (at least a journal letters-to-editor column), and a promise of stable curatorship. Plus inclusion in citation indexes and the like.
Perhaps the FHI should sponsor a journal, to serve as a venue and repository for research articles like these.
Replies from: CarlShulman↑ comment by CarlShulman · 2010-08-21T17:48:02.743Z · LW(p) · GW(p)
Perhaps the FHI should sponsor a journal
There are already relevant niche philosophy journals (Ethics and Information Technology, Minds and Machines, and Philosophy and Technology). Robin Hanson's "Economic Growth Given Machine Intelligence" has been accepted in an AI journal, and there are forecasting journals like Technological Forecasting and Social Change. For more unusual topics, there's the Journal of Evolution and Technology. SIAI folk are working to submit the current crop of papers for publication.
Replies from: Perplexed↑ comment by [deleted] · 2010-08-20T21:25:43.628Z · LW(p) · GW(p)
Okay, I take that back. I did know about the connection between SIAI and FHI and Oxford.
What are these academic papers published in? A lot of them don't provide that information; one is in Global Catastrophic Risks.
At any rate, I exaggerated in saying there isn't any engagement with the academic mainstream. But it looks like it's not very much. And I recall a post of Eliezer's that said, roughly, "It's not that academia has rejected my ideas, it's that I haven't done the work of trying to get academia's attention." Well, why not?
Replies from: WrongBot, Eliezer_Yudkowsky↑ comment by WrongBot · 2010-08-20T21:53:51.809Z · LW(p) · GW(p)
And I recall a post of Eliezer's that said, roughly, "It's not that academia has rejected my ideas, it's that I haven't done the work of trying to get academia's attention." Well, why not?
Limited time and more important objectives, I would assume. Most academic work is not substantially better than trial-and-error in terms of usefulness and accuracy; it gets by on volume. Volume is a detriment in Friendliness research, because errors can have large detrimental effects relative to the size of the error. (Like the accidental creation of a paperclipper.)
↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T21:39:34.070Z · LW(p) · GW(p)
If you want it done, feel free to do it yourself. :)
↑ comment by wedrifid · 2010-08-21T08:52:29.097Z · LW(p) · GW(p)
The SIAI and Eliezer may not have done the best possible job of engaging with the academic mainstream, but they haven't done a terrible one either, and accusations that they aren't trying are, so far as I am able to determine, factually inaccurate.
... particularly in as much as they have become (somewhat) obsolete.
Replies from: MatthewBaker↑ comment by MatthewBaker · 2011-07-05T23:08:11.555Z · LW(p) · GW(p)
Can you clarify please?
Replies from: wedrifid↑ comment by wedrifid · 2011-07-07T17:11:44.699Z · LW(p) · GW(p)
Can you clarify please?
Basically, no. Whatever I meant seems to have been lost to me in the temporal context.
Replies from: MatthewBaker↑ comment by MatthewBaker · 2011-07-07T17:25:40.013Z · LW(p) · GW(p)
No worries, I do the same thing sometimes.
↑ comment by Morendil · 2010-08-20T21:10:06.759Z · LW(p) · GW(p)
We don't have guest posts from Dr. X or Think Tank Fellow Y.
Possibly because this blog is Less Wrong, positioned as "a community blog devoted to refining the art of human rationality", and not as the SIAI blog, or an existential risk blog, or an FAI blog.
↑ comment by multifoliaterose · 2010-08-21T04:59:32.993Z · LW(p) · GW(p)
I don't think there's any point doing armchair diagnoses and accusing people of delusions of grandeur.
I respectfully disagree with this statement, at least as an absolute. I believe that:
(A) In situations in which people are making significant life choices based on person X's claims and person X exhibits behavior which is highly correlated with delusions of grandeur, it's appropriate to raise the possibility that person X's claims arise from delusions of grandeur and ask that person X publicly address this possibility.
(B) When one raises the possibility that somebody is suffering from delusions of grandeur, this should be done in as polite and nonconfrontational way as possible given the nature of the topic.
I believe that if more people adopted these practices, this would would raise the sanity waterline.
I believe that the situation with respect to Eliezer and portions of the LW community is as in (A) and that I made a good faith effort at (B).
↑ comment by wedrifid · 2010-08-21T22:16:43.660Z · LW(p) · GW(p)
I agree with your conclusion but not this part:
If you want to claim you're working on a project that may save the world, fine. But there's got to be more to show for it, sooner or later, than speculative essays. At the very least, people worried about unfriendly AI will have to gather data and come up with some kind of statistical study that gives evidence of a threat! Look at climate science. For all the foibles and challenges of the climate change movement, those people actually gather data, create prediction models, predict the results of mitigating policies -- it works more or less like science.
I categorically do not want statistical studies of the type you mention done. I do want solid academic research done but not experiments. Some statistics on, for example, human predictions vs actual time till successful completion on tasks of various difficulties would be useful. But these do not appear to be the type of studies you are asking for, and nor do they target the most significant parts of the conclusion.
You are not entitled to that particular proof.
EDIT: The 'entitlement' link was broken.
↑ comment by wedrifid · 2010-08-21T08:51:22.879Z · LW(p) · GW(p)
I agree with your conclusion but not this part:
If you want to claim you're working on a project that may save the world, fine. But there's got to be more to show for it, sooner or later, than speculative essays. At the very least, people worried about unfriendly AI will have to gather data and come up with some kind of statistical study that gives evidence of a threat! Look at climate science. For all the foibles and challenges of the climate change movement, those people actually gather data, create prediction models, predict the results of mitigating policies -- it works more or less like science.
I categorically do not want statistical studies of the type you mention done. I do want solid academic research done but not experiments. Some statistics on, for example, human predictions vs actual time till successful completion on tasks of various difficulties would be useful. But these do not appear to be the type of studies you are asking for, and nor do they target the most significant parts of the conclusion.
[You are not entitled to that particular proof]http://lesswrong.com/lw/1ph/youre_entitled_to_arguments_but_not_that/).
comment by luminosity · 2010-08-20T02:43:11.251Z · LW(p) · GW(p)
I feel that perhaps you haven't considered the best way to maximise your chance of developing Friendly AI if you were Eliezer Yudkowsky; your perspective is very much focussed on how you see it lookin in from the outside. Consider for a moment that you are in a situation where you think you can make a huge positive impact upon the world, and have founded an organisation to help you act upon that.
Your first, and biggest problem is getting paid. You could take time off to work on attaining a fortune through some other means but this is not a certain bet, and will waste years that you could be spending working on the problem instead. Your best bet is to find already wealthy people who can be convinced that you can change the world, that it's for the best, and that they should donate significant sums of money to you, unless you believe this is even less certain than making a fortune yourself. There's already a lot of people in the world with the requisite amount of money to spare. I think seeking donations is the more rational path.
Now, given that you need to persuade people of the importance of your brilliant new idea which no one has really been considering before, and that to most people isn't at all an obvious idea. Is the better fund seeking strategy to admit to people that you're uncertain if you'll accomplish it, and compound that on top of their own doubts? Not really. Confidence is a very strong signal that will help you persuade people that you're worth taking seriously. You asking Eliezer to be more publically doubtful probably puts him in an awkward situation. I'd be very surprised if he doesn't have some doubts, maybe he even agrees with you, but to admit to these doubts would be to lower the confidence of investors in him, which would then lower further the chance of him actually being able to accomplish his goal.
Having confidence in himself is probably also important, incidentally. Talking about doubts would tend to reinforce them, and when you're embarking upon a large and important undertaking, you want to spend as much of your mental effort and time as possible on increasing the chances that you'll bring the project about, rather than dwelling on your doubts and wasting mental energy on motivating yourself to keep working.
So how to mitigate the problem that you might be wrong without running into these problems? Well, he seems ot have done fairly well here. The SIAI has now grown beyond just him, giving further perspectives he can draw upon in his work to mitigate any shortcomings in his own analyses. He's laid down a large body of work explaining the mental processes he is basing his approaches on, which should be helpful both in recruitment for SIAI, and in letting people point out flaws or weaknesses in the work he is doing. Seems to me so far he has laid the groundwork out quite well, and now it just remains to see where he and the SIAI go from here. Importantly, the SIAI has grown to the point where even if he is not considering his doubts strongly enough, even if he becomes a kook, there are others there who may be able to do the same work. And if not there, his reasoning has been fairly well laid out, and there is no reason others can't follow their own take on what needs to be done.
That said, as an outsider obviously it's wise to consider the possibility that SIAI will never meet its goals. Luckily, it doesn't have to be an either/or question. Too few people consider existential risk at all, but those of us who do consider it can spread ourselves over the different risks that we see. To the degree which you think Eliezer and the SIAI are on the right track, you can donate a portion of your disposable income to them. To the extent that you think other types of existential risk prevention matter, you can donate a portion of that money to the Future of Humanity Institute, or other relevant existential risk fighting organisation.
Replies from: TheAncientGeek↑ comment by TheAncientGeek · 2015-06-17T10:31:07.701Z · LW(p) · GW(p)
A display of confidence is a good way of getting people on your side if you are right,. It is also a good way of ovwrestimating whether you are right or not.
comment by JamesAndrix · 2010-08-23T05:38:43.502Z · LW(p) · GW(p)
How would you address this?
http://scienceblogs.com/pharyngula/2010/08/kurzweil_still_doesnt_understa.php
It seems to me like PZ Meyers really doesn't understand information theory. He's attacking Kurzweil and calling him a kook. Initially due to a relatively straightforward complexity estimate.
And I'm pretty confident that Myers is wrong on this, unless there is another information rich source of inheritance besides DNA, which Meyers knows about but Kurzweil and I do not.
This looks to me like a popular science blogger doing huge PR damage to everything singularity related, and being wrong about it. Even if he is later convinced of this point.
I don't see how to avoid this short of just holding back all claims which seem exceptional and that some 'reasonable' person might fail to understand and see as a sign of cultishness. If we can't make claims as basic as the design of the brain being in the genome, then we may as well just remain silent.
But then we wouldn't find out if we're wrong, and we're rationalists.
Replies from: WrongBot, Risto_Saarelma, Kingreaper, Emile, knb, Mitchell_Porter, ciphergoth, wedrifid↑ comment by WrongBot · 2010-08-23T15:15:39.170Z · LW(p) · GW(p)
For instance, you can't measure the number of transistors in an Intel CPU and then announce, "A-ha! We now understand what a small amount of information is actually required to create all those operating systems and computer games and Microsoft Word, and it is much, much smaller than everyone is assuming."
This analogy made me cringe. Myers is disagreeing with the claim that human DNA completely encodes the structure and functioning of the human brain: the hardware and software, roughly. Looking at the complexity of the hardware and making claims about the complexity of the software, as he does here, is completely irrelevant to his disagreement. It serves only to obscure the actual point under debate, and demonstrates that he has no idea what he's talking about.
↑ comment by Risto_Saarelma · 2010-08-23T11:01:04.261Z · LW(p) · GW(p)
There seems to be a culture clash between computer scientists and biologists with this matter. DNA bit length as a back-of-the-envelope complexity estimate for a heavily compressed AGI source seems obvious to me, and, it seems, to Larry Page. Biologists are quick to jump to the particulars of protein synthesis and ignore the question of extra information, because biologists don't really deal with information theoretical existence proofs.
It really doesn't help the matter that Kurzweil threw out his estimate when talking about getting at AGI by specifically emulating the human brain, instead of just trying to develop a general human-equivalent AI using code suitable for the computation platform used. This seems to steer most people into thinking that Kurzweil was thinking of using the DNA as literal source code instead of just a complexity yardstick.
Myers seems to have pretty much gone into his creationist-bashing attack mode on this, so I don't have a very high hopes for any meaningful dialogue from him.
Replies from: whpearson↑ comment by whpearson · 2010-08-23T12:24:51.264Z · LW(p) · GW(p)
I'm still not sure what people are trying to say with this. Because the kolmogorov complexity of the human brain given the language of the genetic code and physics is low, therefore X? What is that X precisely?
Because of kolmogorov complexities additive constant, which could be anything from 0 to 3^^^3 or higher, I think it only gives us weak evidence for the amount of code we should expect it to take to code an AI on a computer. It is even weaker evidence for the amount of code it would take to code for it with limited resources. E.g. the laws of physics are simple and little information is taken from the womb, but to create an intelligence from them might require a quantum computer the size of the human head to decompress the compressed code. There might be short cuts to do it, but they might be of vastly greater complexity.
We tend to ignore additive constants when talking about Complexity classes, because human designed algorithms tend not to have huge additive constants. Although I have come across some in my time such as this...
Replies from: Emile↑ comment by Emile · 2010-08-23T15:45:42.501Z · LW(p) · GW(p)
We have something like this going on like:
discrete DNA code -> lots of messy chemistry and biology -> human intelligence
and we're comparing it to :
discrete computer code -> computer -> human intelligence
Kurzweil is arguing that the size of the DNA code can tell us about the max size of the computer code needed to run an intelligent brain simulation (or a human-level AI), and PZ Myers is basically saying "no, 'cause that chemistry and biology is really really messy".
Now, I agree that the computer code and the DNA code are very very different ("a huge amount of enzymes interacting with each other in 3D real time" isn't the kind of thing you easily simulate on a computer), and the additive constant for converting one into the other is likely to be pretty darn big.
But I also don't see a reason for intelligence to be easier to express with messy biology and chemistry than with computer code. The things about intelligence that are the closest to biology (interfacing with the real world, how one neuron functions) are also the kind of things that we can already do quite well with computer programs.
There are some things that are "natural" to code in Prolog, but not natural in Fortran, fotran. So a short program in prolog might require a long program in Fotran to do the same thing, and for different programs it might be the other way around. I don't see any reason to think that it's easier to encode intelligence in DNA than it is in computer code.
(Now, Kurzweil may be overstating his case when he talks about "compressed" DNA, because to be fair you should compare that to compressed (or compiled) computer code, which translates to much more actual code. I still think the size of the DNA is a very reasonable upper limit, especially when you consider that the DNA was coded by a bloody idiot whose main design pattern is "copy-and-paste", resulting in the bloated code we know)
Replies from: whpearson↑ comment by whpearson · 2010-08-23T16:52:59.873Z · LW(p) · GW(p)
But I also don't see a reason for intelligence to be easier to express with messy biology and chemistry than with computer code.
Do you have any reason to expect it to be the same? Do we have any reason at all? I'm not arguing that it will take more than 50MBs of code, I'm arguing that the DNA value is not informative.
The things about intelligence that are the closest to biology (interfacing with the real world, how one neuron functions) are also the kind of things that we can already do quite well with computer programs.
We are far less good at the doing the equivalent of changing neural structure or adding new neurons (we don't know why or how neurogenesis works for one) in computer programs.
Replies from: Emile↑ comment by Emile · 2010-08-23T19:59:22.374Z · LW(p) · GW(p)
But I also don't see a reason for intelligence to be easier to express with messy biology and chemistry than with computer code.
Do you have any reason to expect it to be the same? Do we have any reason at all?
If I know a certain concept X requires 12 seconds of speech to express in English, and I don't know anything about Swahili beyond the fact that it's a human language, my first guess will be that concept X requires 12 seconds of speech to express in Swahili.
I would also express compressed versions of translations in various languages of the same book to be roughly the same size.
So, even with very little information, a first estimate (with a big error margin) would be that it takes as many bits to "encode" intelligence in DNA than it does in computer code.
In addition, the fact that some intelligence-related abilities such as multiplying large numbers are easy to express in computer code, but rare in nature would make me revise that estimate towards "code as more expressive than DNA for some intelligence-related stuff".
In addition, knowledge about the history of evolution would make me suspect that large chunks of the human genome are not required for intelligence, either because they aren't expressed, or because they only concern traits that have no impact on our intelligence beyond the fact of keeping us alive. That would also make me revise my estimate downwards for the code size needed for intelligence.
None of those are very strong reasons, but they are reasons nonetheless!
Replies from: whpearson↑ comment by whpearson · 2010-08-23T21:54:47.132Z · LW(p) · GW(p)
If I know a certain concept X requires 12 seconds of speech to express in English, and I don't know anything about Swahili beyond the fact that it's a human language, my first guess will be that concept X requires 12 seconds of speech to express in Swahili.
You'd be very wrong for a lot of technical language, unless they just imported the English words whole sale. For example, "Algorithmic Information Theory," expresses a concept well but I'm guessing it would be hard to explain in Swahili.
Even given that, you can expect the languages of humans to all have roughly the same length because they are generated by the roughly the same hardware and have roughly the same concerns. E.g. things to do with humans.
To give a more realistic translation problem, how long would you expect it to take to express/explain any random English in C code or vice versa?
Replies from: Peter_de_Blanc↑ comment by Peter_de_Blanc · 2010-08-29T05:00:53.253Z · LW(p) · GW(p)
Selecting a random English sentence will introduce a bias towards concepts that are easy to express in English.
↑ comment by Kingreaper · 2010-08-23T15:26:45.334Z · LW(p) · GW(p)
And I'm pretty confident that Myers is wrong on this, unless there is another information rich source of inheritance besides DNA, which Meyers knows about but Kurzweil and I do not.
The environment is information-rich, especially the social environment.
Meyers make it quite clear that interactions with the environment are an expected input of information in his understanding.
Do you disagree with information input from the environment?
Replies from: JamesAndrix↑ comment by JamesAndrix · 2010-08-23T17:10:13.745Z · LW(p) · GW(p)
Yes, I disagree.
If he's not talking about some stable information that is present in all environments that yield intelligent humans, then what's important is a kind of information that can be mass generated at low complexity cost.
Even language exposure is relatively low complexity, and the key parts might be inferable from brain processes. And we already know how to offer a socially rich environment, so I don't think it should add to the complexity costs of this problem.
And I think a reverse engineering of a newborn baby brain would be quite sufficient for kurzweil's goal.
In short: we know intelligent brains get reliably generated. We know it's very complex. The source of that complexity must be something information rich, stable, and universal. I know of exactly one such source.
Right now I'm reading myers argument as "a big part of human heredity is memetic rather than just genetic, and there is complex interplay between genes and memes, so you've got to count the memes as part of the total complexity."
I say that Kurzweil is trying to create something compatible with human memes in the first plalce, so we can load them the same way we load children (at worst) And even some classes of memes (age appropriate language exposure) do interact tightly with genes, their information content is not all that high.
Replies from: whpearson↑ comment by whpearson · 2010-08-23T17:33:57.867Z · LW(p) · GW(p)
And I think a reverse engineering of a newborn baby brain would be quite sufficient for kurzweil's goal.
While doable this seems like a very time consuming project and potentially morally dubious. How do you know when you have succeeded and not got a mildly brain damaged one, because you have missed an important detail for language learning?
We really don't want to be running multi year experiments, where humans have to interact with infant machines, that would be ruinously expensive. The quicker you can evaluate the capabilities of the machine the better.
Replies from: JamesAndrix↑ comment by JamesAndrix · 2010-08-23T17:53:39.118Z · LW(p) · GW(p)
Well in Kurzweils' case, you'd look at the source code and debug it to make sure it's doinjg everything it's supposed to, because he's no dealing with a meat brain.
I guess my real point is that language learning should not be tacked on to the problem of reverse engineering the brain, If he makes something that is as capable of learning, that's a win for him. (Hopefully he also reverse engineers all of human morality.)
Replies from: whpearson↑ comment by whpearson · 2010-08-25T16:36:31.840Z · LW(p) · GW(p)
You are assuming the program found via the reverse engineering process is human understandable.... what if it is a strange cellular automata with odd rules. Or an algorithm with parameters you don't know why they are what they are.
Language is an important part of learning for humans. Imagine trying to learn chess if no one explained the legal moves. Something without the capability for language isn't such a big win IMHO.
Replies from: JamesAndrix↑ comment by JamesAndrix · 2010-08-25T17:09:03.387Z · LW(p) · GW(p)
I think we might have different visions of what this reverse engineering would entail, By my concept, if you don't understand the function of the program you wrote, you're not done reverse engineering.
I do think that something capable of learning language would be necessary for a win. but the information content of the language does not count towards the complexity estimate of the thing capable of learning langauge.
↑ comment by Emile · 2010-08-23T08:31:54.038Z · LW(p) · GW(p)
It seems to me like PZ Meyers really doesn't understand information theory. He's attacking Kurzweil and calling him a kook. Initially due to a relatively straightforward complexity estimate.
I see it that way too. The DNA can give us an upper bound on the information needed to create a human brain, but PZ Myers reads that as "Kurzweil is saying we will be able to take a strand of DNA and build a brain from that in the next 10 years!", and then procede to attack that straw man.
This, however:
His timeline is absurd. I'm a developmental neuroscientist; I have a very good idea of the immensity of what we don't understand about how the brain works. No one with any knowledge of the field is claiming that we'll understand how the brain works within 10 years. And if we don't understand all but a fraction of the functionality of the brain, that makes reverse engineering extremely difficult.
... I am quite enclined to trust. I would trust it more if it wasn't followed by wrong statements about information theory (that seem wrong to me, at least).
Looking at the comments is depressing. I wish there was some "sane" ways for two communities (readers of PZ Myers and "singularitarians") to engage without it degenerating into name-calling.
Brian: "We should unite against our common enemy!"
Others: "The Judean People's Front?"
Brian: "No! The Romans!"
Though there are software solutions for that (takeonit and other stuff that's been discussed here), it wouldn't help either if the "leaders" (PZ Myers, Kurzweil, etc.) were a bit more responsible and made a genuine effort to acknowledge the other's points when there are strong. So they could converge or at least agree to disagree on something narrow.
But nooo, it's much more fun to get angry, and it gets you more traffic too!
Replies from: RobinZ↑ comment by RobinZ · 2010-08-23T13:09:16.044Z · LW(p) · GW(p)
The DNA can give us an upper bound on the information needed to create a human brain [...]
Why do you say this? If humans were designed by human engineers, the 'blueprints' would actually be complete blueprints, sufficient unto the task of determining the final organism ... but they weren't. There's no particular reason to doubt that a significant amount of the final data is encoded in the gestational environment.
Replies from: Emile, Perplexed, JamesAndrix↑ comment by Emile · 2010-08-23T14:20:28.058Z · LW(p) · GW(p)
I'm not sure about what you mean about the "complete blueprints" - I agree that the DNA isn't a complete blueprint, and that an alien civilization with a different chemistry would (probably) find it impossible to rebuild a human if they were just given it's DNA. The gestational environment is essential, I just don't think it encodes much data on the actual working of the brain.
It seems to me that the interaction between the baby and the gestational environment is relatively simple, at least compared to organ development and differentiation. There are a lot of essential things for it to go right, and hormones and nutrients, but 1) I don't see a lot of information transfer in there ("making the brain work a certain way" as opposed to "making the brain work period"), and 2) A lot of the information on how that works is probably encoded in the DNA too.
I would say that the important bits that may not be in the DNA (or in mitocondrial DNA) are the DNA interpretation system (transcription, translation).
Replies from: RobinZ↑ comment by RobinZ · 2010-08-23T15:23:05.064Z · LW(p) · GW(p)
That's a strong point, but I think it's still worth bearing in mind that this subject is P. Z. Myers' actual research focus: developmental biology. It appears to me that Kurzweil should be getting Myers' help revising his 50 MB estimate*, not dismissing Myers arguments as misinformed.
Yes, Myers made a mistake in responding to a summary secondhand account rather than Kurzweil's actual position, but Kurzweil is making a mistake if he's ignoring expert opinion on a subject directly relating to his thesis.
* By the way: 50 MB? That's smaller than the latest version of gcc! If that's your complexity estimate, the complexity of the brain could be dominated by the complexity of the gestational environment!
Replies from: Emile↑ comment by Emile · 2010-08-23T16:02:08.708Z · LW(p) · GW(p)
I agree that Kurzweil could have acknowledged P.Z.Myers' expertise a bit more, especially the "nobody in my field expects a brain simulation in the next ten years" bit.
50 MB - that's still a hefty amount of code, especially if it's 50MB of compiled code and not 50 MB of source code (comparing the size of the source code to the size of the compressed DNA looks fishy to me, but I'm not sure Kurzweil has been actually doing that - he's just been saying "it doesn't require trillions of lines of code").
Is the size of gcc the source code or the compiled version? I didn't see that info on Wikipedia, and don't have gcc on this machine.
Replies from: timtyler, RobinZ↑ comment by RobinZ · 2010-08-23T16:09:34.278Z · LW(p) · GW(p)
I'm looking at gcc-4.5.0.tar.gz.
Replies from: Emile↑ comment by Emile · 2010-08-23T16:32:27.779Z · LW(p) · GW(p)
That includes the source code, the binaries, the documentation, the unit tests, changelogs ... I'm not surpised it's pretty big!
I consider it pretty likely that it's possible to program a human-like intelligence with a compressed source code of less than 50 MB.
However, I'm much less confident that the source code of the first actual human-like intelligence coded by humans (if there is one) will be that size.
↑ comment by Perplexed · 2010-08-23T13:45:31.674Z · LW(p) · GW(p)
There's no particular reason to doubt that a significant amount of the final data is encoded in the gestational environment.
To the contrary, there is every reason to doubt that. We already know that important pieces of the gestational environment (the genetic code itself, core metabolism, etc.) are encoded in the genome. By contrast, the amount of epigenetic information that we know of is miniscule. It is, of course, likely that we will discover more, but it is very unlikely that we will discover much more. The reason for this skepticism is that we don't know of any reliable epigenetic means of transmitting generic information from generation to generation. And the epigenetic information inheritance mechanisms that we do understand all require hundreds of times as much genetic information to specify the machinery as compared to the amount of epigenetic information that the machinery can transmit.
To my mind, it is very clear that (on this narrow point) Kurzweil was right and PZ wrong: The Shannon information content of the genome places a tight upper bound on the algorithmic (i.e. Kolmogorov) information content of the embryonic brain. Admittedly, when we do finally construct an AI, it may take it 25 years to get through graduate school, and it may have to read thru several hundred Wikipedia equivalents to get there, but I am very confident that specifying the process for generating the structure and interconnect of the embryonic AI brain will take well under 7 billion bits.
Replies from: timtyler↑ comment by timtyler · 2010-08-23T17:08:44.898Z · LW(p) · GW(p)
To my mind, it is very clear that (on this narrow point) Kurzweil was right and PZ wrong: The Shannon information content of the genome places a tight upper bound on the algorithmic (i.e. Kolmogorov) information content of the embryonic brain.
I think you may have missed my devastating analysis of this issue a couple of years back:
"So, who is right? Does the brain's design fit into the genome? - or not?
The detailed form of proteins arises from a combination of the nucleotide sequence that specifies them, the cytoplasmic environment in which gene expression takes place, and the laws of physics.
We can safely ignore the contribution of cytoplasmic inheritance - however, the contribution of the laws of physics is harder to discount. At first sight, it may seem simply absurd to argue that the laws of physics contain design information relating to the construction of the human brain. However there is a well-established mechanism by which physical law may do just that - an idea known as the anthropic principle. This argues that the universe we observe must necessarily permit the emergence of intelligent agents. If that involves a coding the design of the brains of intelligent agents into the laws of physics then: so be it. There are plenty of apparently-arbitrary constants in physics where such information could conceivably be encoded: the fine structure constant, the cosmological constant, Planck's constant - and so on.
At the moment, it is not even possible to bound the quantity of brain-design information so encoded. When we get machine intelligence, we will have an independent estimate of the complexity of the design required to produce an intelligent agent. Alternatively, when we know what the laws of physics are, we may be able to bound the quantity of information encoded by them. However, today neither option is available to us."
Replies from: Perplexed↑ comment by Perplexed · 2010-08-23T18:24:06.629Z · LW(p) · GW(p)
You suggest that the human brain might have a high Kolmogorov complexity, the information for which is encoded, not in the human genome (which contains a mere 7 gigabits of information), but rather in the laws of physics, which contain arbitrarily large amounts of information, encoded in the exact values of physical constants. For example, first 30 billion decimal digits of the fine structure constant contain 100 gigabits of information, putting the genome to shame.
Do I have that right?
Well, I will give you points for cleverness, but I'm not buying it. I doubt that it much matters what the constants are, out past the first hundred digits or so. Yes, I realize that the details of how the universe proceeds may be chaotic; it may involve sensitive dependence both on initial conditions and on physical constants. But I don't think that really matters. Physical constants haven't changed since the Cambrian, but genomes have. And I think that it is the change in genomes which led to the human brain, the dolphin brain, the parrot brain, and the octopus brain. Alter the fine structure constant in the 2 billionth decimal place, and those brain architectures would still work, and those genomes would still specify development pathways leading to them. Or so I believe.
Replies from: timtyler↑ comment by timtyler · 2010-08-23T18:44:17.692Z · LW(p) · GW(p)
I doubt that it much matters what the constants are, out past the first hundred digits or so
What makes you think that?
I realize that the details of how the universe proceeds may be chaotic; it may involve sensitive dependence both on initial conditions and on physical constants. But I don't think that really matters.
...and why not?
Physical constants haven't changed since the Cambrian, but genomes have. And I think that it is the change in genomes which led to the human brain, the dolphin brain, the parrot brain, and the octopus brain.
Under the hypothesis that physics encodes relevant information, a lot of the required information was there from the beginning. The fact that brains only became manifest after the Cambrian doesn't mean the propensity for making brains was not there from the beginning. So: that observation doesn't tell you very much.
Alter the fine structure constant in the 2 billionth decimal place, and those brain architectures would still work, and those genomes would still specify development pathways leading to them. Or so I believe.
Right - but what evidence do you have of that? You are aware of chaos theory, no? Small changes can lead to dramatic changes surprisingly quickly.
Organisms inherit the laws of physics (and indeed the initial conditions of the universe they are in) - as well as their genomes. Information passes down the generations both ways. If you want to claim the design information is in one inheritance channel more than the other one, it seems to me that you need some evidence relating to that issue. The evidence you have presented so far seems pretty worthless - the delayed emergence of brains seems equally compatible with both of the hypotheses under consideration.
So: do you have any other relevant evidence?
Replies from: WrongBot↑ comment by WrongBot · 2010-08-23T18:59:07.751Z · LW(p) · GW(p)
No other rational [ETA: I meant physical and I am dumb] process is known to rely on physical constants to the degree you propose. What you propose is not impossible, but it is highly improbable.
Replies from: timtyler↑ comment by timtyler · 2010-08-23T19:08:00.791Z · LW(p) · GW(p)
What?!? What makes you think that?
Sensitive dependence on initial conditions is an extremely well-known phenomenon. If you change the laws of physics a little bit, the result of a typical game of billiards will be different. This kind of phenomenon is ubiquitous in nature, from the orbit of planets, to the paths rivers take.
If a butterfly's wing flap can cause a tornado, I figure a small physical constant jog could easily make the difference between intelligent life emerging, and it not doing so billions of years later.
Sensitive dependence on initial conditions is literally everywhere. Check it out:
http://en.wikipedia.org/wiki/Chaos_theory
Replies from: Kingreaper, JamesAndrix, WrongBot↑ comment by Kingreaper · 2010-08-23T19:11:11.569Z · LW(p) · GW(p)
Did you miss this bit:
to the degree you propose
Sensitivity to initial conditions is one thing. Sensitivity to 1 billion SF in a couple of decades?
Replies from: timtyler↑ comment by timtyler · 2010-08-23T19:17:54.003Z · LW(p) · GW(p)
The universe took about 14 billion years to get this far - and if you look into the math of chaos theory, the changes propagate up very rapidly. There is an ever-expanding avalanche of changes - like an atomic explosion.
For the 750mb-or-so of data under discussion, you could easily see the changes at a macroscopic scale rapidly. Atoms in stars bang into each other pretty quickly. I haven't attempted to calculate it - but probably within a few minutes, I figure.
Replies from: PaulAlmond↑ comment by PaulAlmond · 2010-08-23T19:59:32.099Z · LW(p) · GW(p)
Would you actually go as far as maintaining that, if a change were to happen tomorrow to the 1,000th decimal place of a physical constant, it would be likely to stop brains from working, or are you just saying that a similar change to a physical constant, if it happened in the past, would have been likely to stop the sequence of events which has caused brains to come into existence?
Replies from: timtyler, PaulAlmond↑ comment by PaulAlmond · 2010-08-23T20:11:25.411Z · LW(p) · GW(p)
Correction: That last line should be "which has CAUSED brains to come into existence?"
Replies from: SilasBarta↑ comment by SilasBarta · 2010-08-23T20:35:36.480Z · LW(p) · GW(p)
You can edit comments after submitting them -- when logged in, you should see an edit button.
By the way, I'm reading your part 15, section 2 now.
Replies from: PaulAlmond, PaulAlmond↑ comment by PaulAlmond · 2010-08-23T20:45:24.274Z · LW(p) · GW(p)
Hi Silas!
Thanks for telling me that. I was logged in and didn't see it, but I will look more carefully next time.
I'm actually proof-reading a document now which improves the "action selection process". I was never happy with what I described and it was a kind of placeholder. The new stuff will be very short though.
Anyway, what do you do? I have the idea it is something computer related, maybe?
↑ comment by PaulAlmond · 2010-08-23T20:55:22.575Z · LW(p) · GW(p)
Apologies for the comment I inadvertently placed here. I thought I was answering a PM and did not mean to add personal exchanges. I find computers annoying sometimes, and will happily stop using them when something else that is Turing equivalent becomes available.
↑ comment by JamesAndrix · 2010-08-24T07:02:53.213Z · LW(p) · GW(p)
I figure a small physical constant jog could easily make the difference between intelligent life emerging, and it not doing so billions of years later.
First, that is VERY different than the design information being in the constant, but not in the genome. (you could more validly say that the genome is what it is because the constant is precisely what it is.)
Second, the billiard ball example is invalid. It doesn't matter exactly where the billiard balls are if you're getting hustled. Neurons are not typically sensitive to the precise positions of their atoms. Information processing relies on the ability to largely overlook noise.
↑ comment by WrongBot · 2010-08-23T20:43:26.993Z · LW(p) · GW(p)
What physical process would cease to function if you increased c by a billionth of a percent? Or one of the other Planck units? Processes involved in the functioning of both neurons and transistors don't count, because then there's no difference to account for.
Replies from: Sniffnoy, timtyler↑ comment by Sniffnoy · 2010-08-24T00:05:52.751Z · LW(p) · GW(p)
Nitpick: c is a dimensioned quantity, so changes in it aren't necessarily meaningful.
Replies from: WrongBot↑ comment by WrongBot · 2010-08-24T01:17:27.346Z · LW(p) · GW(p)
*Blink.*
*Reads Wikipedia.*
Would I be correct in thinking that one would need to modify the relationship of c to some other constant (the physics equation that represent some physical law?) for the change to be meaningful? I may be failing to understand the idea of dimension.
Thank you for the excuse to learn more math, by the way.
Replies from: Psy-Kosh↑ comment by Psy-Kosh · 2010-08-24T02:09:46.692Z · LW(p) · GW(p)
Yes, you would be correct, at least in terms of our current knowledge.
In fact, it's not that unusual to choose units so that you can set c = 1 (ie, to make it unitless). This way units of time and units of distance are the same kind, velocities are dimensionless geometric quantities, etc...
You might want to think of "c" not so much as a speed as a conversion factor between distance type units and time type units.
↑ comment by JamesAndrix · 2010-08-23T17:13:41.947Z · LW(p) · GW(p)
Artificial wombs
Replies from: RobinZ↑ comment by RobinZ · 2010-08-23T17:17:38.171Z · LW(p) · GW(p)
Don't currently exist. I'm not sure that's a strong argument.
↑ comment by knb · 2010-08-26T05:17:21.687Z · LW(p) · GW(p)
Myers has always had a tendency to attack other people's arguments like enemy soldiers. A good example is his take on evolutionary psychology, which he hates so much it is actually funny.
And then look at the source: Satoshi Kanazawa, the Fenimore Cooper of Sociobiology, the professional fantasist of Psychology Today. He's like the poster boy for the stupidity and groundlessness of freakishly fact-free evolutionary psychology. Just ignore anything with Kanazawa's name on it.
He also claims to have desecrated a consecrated host (the sacramental wafers Catholics consider to be the body of Jesus). That will show those evil theists how a good, rational person behaves!
↑ comment by Mitchell_Porter · 2010-08-23T07:44:47.895Z · LW(p) · GW(p)
I'm pretty confident that Myers is wrong on this, unless there is another information rich source of inheritance besides DNA, which Myers knows about but Kurzweil and I do not.
Myers' thesis is that you are not going to figure out by brute-force physical simulation how the genome gives rise to the organism, knowing just the genomic sequence. On every scale - molecule, cell, tissue, organism - there are very complicated boundary conditions at work. You have to do experimental biology, observe those boundary conditions, and figure out what role they play. I predict he would be a lot more sympathetic if Kurzweil was talking about AIs figuring out the brain by doing experimental biology, rather than just saying genomic sequence + laws of physics will get us there.
Replies from: Perplexed, SilasBarta↑ comment by Perplexed · 2010-08-23T16:03:45.908Z · LW(p) · GW(p)
Myers' thesis is that you are not going to figure out by brute-force physical simulation how the genome gives rise to the organism, knowing just the genomic sequence.
And he is quite possibly correct. However, that has nothing at all to do with what Kurzweil said.
I predict he would be a lot more sympathetic if Kurzweil was talking about AIs figuring out the brain by doing experimental biology, rather than just saying genomic sequence + laws of physics will get us there.
I predict he would be more sympathetic if he just made the effort to figure out what Kurzweil said. But, of course, we all know there is no chance of that, so "conjecture" might be a better word than "predict".
Replies from: Mitchell_Porter↑ comment by Mitchell_Porter · 2010-08-24T11:18:29.547Z · LW(p) · GW(p)
Myers doesn't have an argument against Kurzweil's estimate of the brain's complexity. But his skepticism about Kurzweil's timescale can be expressed in terms of the difficulty of searching large spaces. Let's say it does take a million lines of code to simulate the brain. Where is the argument that we can produce the right million lines of code within twenty years? The space of million-line programs is very large.
Replies from: Perplexed↑ comment by Perplexed · 2010-08-24T12:04:20.681Z · LW(p) · GW(p)
I agree, both regarding timescale, and regarding reason for timescale difficulties.
As I understand Kurzweil, he is saying that we will build the AI, not by finding the program for development and simulating it, but rather by scanning the result of the development and duplicating it in a different medium. The only relevance of that hypothetical million-line program is that it effectively puts a bound on the scanning and manufacturing tolerances that we need to achieve. Well, while it is probably true in general that we don't need to get the wiring exactly right on all of the trillions of neurons, there may well be some where the exact right embryonic wiring is crucial to success. And, since we don't yet have or understand that million-line program that somehow gets the wiring right reliably, we probably won't get them right ourselves. At least not at first.
It feels a little funny to find myself making here an argument right out of Bill Dembski's playbook. No free lunch! Needle in a haystack. Only way to search that space is by exhaustion. Well, we shall see what we shall see.
↑ comment by SilasBarta · 2010-08-23T15:41:47.545Z · LW(p) · GW(p)
I agree, but at the same time, I wish biologists would learn more information theory, since their focus should be identifying the information flows going on, as this is what will lead us to a comprehensible model of human development and functionality.
(I freely admit I don't have years in the trenches, so this may be a naive view, but if my experience with any other scientific turf war is any guide, this is important advice.)
↑ comment by Paul Crowley (ciphergoth) · 2010-08-23T07:16:39.554Z · LW(p) · GW(p)
This was cited to me in a blog discussion as "schoolboy biology EY gets wrong" (he said something similar, apparently).
comment by cousin_it · 2010-08-20T01:15:11.625Z · LW(p) · GW(p)
I upvoted this, but I'm torn about this.
In your recent posts you've been slowly, carefully, thoroughly deconstructing one person. Part of me wants to break into applause at the techniques used, and learn from them, because in my whole life of manipulation I've never mounted an attack of such scale. (The paragraph saying "something has gone very wrong" was absolutely epic, to the point of evoking musical cues somewhere at the edge of my hearing. Just like the "greatly misguided" bit in your previous post. Bravo!) But another part of me feels horror and disgust because after traumatic events in my own life I'd resolved to never do this thing again.
It comes down to this: I enjoy LW for now. If Eliezer insists on creating a sealed reality around himself, what's that to me? You don't have to slay every dragon you see. Saving one person from megalomania (real or imagined) is way less important than your own research. Imagine the worst possible world: Eliezer turns into a kook. What would that change, in the grand scheme of things or in your personal life? Are there not enough kooks in AI already?
And lastly, a note about saving people. I think many of us here have had the unpleasant experience (to put it mildly) of trying to save someone from suicide. Looking back at such episodes in my own life, I'm sure that everyone involved would've been better off if I'd just hit "ignore" at the first sign of trouble. Cut and run: in serious cases it always comes to that, no exceptions. People are very stubborn, both consciously and subconsciously - they stay on their track. They will waste their life (or spend it wisely, it's a matter of perspective), but if you join the tug-of-war, you'll waste a big chunk of yours as well.
How's that for other-optimizing?
Replies from: katydee, wedrifid, Vladimir_Nesov, xamdam↑ comment by katydee · 2010-08-20T10:05:22.101Z · LW(p) · GW(p)
I saved someone from suicide once. While the experience was certainly quite unpleasant at the time, if I had hit "ignore," as you suggest, she would have died. I don't think that I would be better off today if I had let her die, to say nothing of her. The fact that saving people is hard doesn't mean that you shouldn't do it!
↑ comment by wedrifid · 2010-08-20T01:45:48.665Z · LW(p) · GW(p)
It comes down to this: I enjoy LW for now. If Eliezer insists on creating a sealed reality around himself, what's that to me? You don't have to slay every dragon you see. Saving one person from megalomania (real or imagined) is way less important than your own research. Imagine the worst possible world: Eliezer turns into a kook. What would that change, in the grand scheme of things or in your personal life?
The very fate of the universe, potentially. Purely hypothetically and for the sake of the discussion:
- If Eliezer did have the potential to provide a strong positive influence on grand scale future outcomes but was crippled by the still hypothetical lack of self-doubt then that is a loss of real value.
- A bad 'Frodo' can be worse than no Frodo at all. If we were to give the ring to a Frodo who thought he could take on Nazgul in hand to hand combat then we would lose the ring and so the lose the chance to give said ring to someone who could pull it off. Multi (and those for whom he asks such questions) have limited resources (and attention) so it may be worth deliberate investigation of potential recipients of trust.
- Worse yet than a counterproductive Frodo would be a Frodo whose arrogance pisses of Aragorn, Gandalf, Legolas, Gimli, Merry, Pippin and even Sam so much that they get disgusted with the whole 'save the world' thing and go hang out in the forest flirting with Elven maidens. Further cause to investigate just whose bid for notoriety and influence you wish to support.
I cannot emphasise how much this is only a reply to the literal question cousin_it asked and no endorsement or denial of any of the above claims as they relate to persons real or imagined. For example it may have been good if Frodo was arrogant enough to piss off Aragorn. He may cracked it, taken the ring from Frodo and given it to Arwen. Arwen was crazy enough to give up the immortality she already had and so would be as good a candidate as any for being able to ditch a ring, without being completely useless for basically all purposes.
Replies from: Eliezer_Yudkowsky, Perplexed, dclayh, cousin_it↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T03:24:38.010Z · LW(p) · GW(p)
Er... I can't help but notice a certain humor in the idea that it's terrible if I'm self-deluded about my own importance because that means I might destroy the world.
Replies from: John_Baez, wedrifid↑ comment by wedrifid · 2010-08-20T09:35:14.284Z · LW(p) · GW(p)
Yes, there is is a certain humor. But I hope you did read the dot points and followed the reasoning. It, among other things, suggests a potential benefit of criticism such as multi's aside from hypothetical benefits of discrediting you should it have been the case that you were not, in fact, competent.
↑ comment by Perplexed · 2010-08-20T03:14:41.120Z · LW(p) · GW(p)
What would that change, in the grand scheme of things or in your personal life?
The very fate of the universe, potentially.
I suppose I could draw from that the inference that you have a rather inflated notion of the importance of what multi is doing here, ... but, in the immortal words of Richard Milhous Nixon, "That would be wrong."
More seriously, I think everyone here realizes that EY has some rough edges, as well as some intellectual strengths. For his own self-improvement, he ought to be working on those rough edges. I suspect he is. However, in the meantime, it would be best if his responsibilities were in areas where his strengths are exploited and his rough edges don't really matter. So, just what are his current responsibilities?
Convincing people that UFAI constitutes a serious existential risk while not giving the whole field of futurism and existential risk reduction a bad rep.
Setting direction for and managing FAI and UFAI-avoidance research at SIAI.
Conducting FAI and UFAI-avoidance research.
Reviewing and doing conceptual QC on the research work product.
To be honest, I don't see EY's "rough edges" as producing any problems at all with his performance on tasks #3 and #4. Only SIAI insiders know whether there has been a problem on task #2. Based on multi's arguments, I suspect he may not be doing so well on #1. So, to me, the indicated response ought to be one of the following:
A. Hire someone articulate (and if possible, even charismatic) to take over task #1 and make whatever minor adjustments are needed regarding task #2.
B. Do nothing. There is no problem!
C. Get some academic papers published so that FAI/anti-UFAI research becomes interesting to the same funding sources that currently support CS, AI, and decision theory research. Then reconstitute SIAI as just one additional research institution which is fighting for that research funding.
I would be interested in what EY thinks of these three possibilities. Perhaps for different reasons, I suspect, so would multi.
[Edited to correct my hallucination of confusing multifoliaterose with wedrifid. As a result of this edit, various comments below may seem confused. Sorry about that, but I judge that making this comment clear is the higher priority.]
Replies from: wedrifid↑ comment by wedrifid · 2010-08-20T09:42:41.266Z · LW(p) · GW(p)
Edit again: not wedrifid
Was the first (unedited) 'you' intended? If so I'll note that I was merely answering a question within a counterfactual framework suggested by the context. I haven't even evaluated what potential importance multi's post may have - but the prior probability I have for 'a given post on LW mattering significantly' is not particularly high.
I like your general analysis by the way and am always interested to know what the SIAI guys are doing along the lines of either your 1,2,3 or your A, B, C. I would seriously like to see C happen. Being able and willing to make that sort of move would be a huge step forward (and something that makes any hints of 'arrogance' seem trivial.)
Replies from: Unknowns↑ comment by Unknowns · 2010-08-20T09:58:44.473Z · LW(p) · GW(p)
I think that originally Perplexed didn't look at your comment carefully and thought that multi had written it.
Replies from: Perplexed, wedrifid↑ comment by wedrifid · 2010-08-20T11:23:39.636Z · LW(p) · GW(p)
I think you are right. I'm just playing the disclaimer game. Since this is a political thread there is always the risk of being condemned for supporting various positions. In this case I gave a literal answer to a rhetorical question directed at multi. Following purely social reasoning that would mean that I:
- Am challenging cousin_it
- Condemning Eliezer
- Agreeing with anything and everything said by multi and probably also with everything said by anyone else who agrees with multi.
- Almost certainly saying something about the credulity of uFAI risks.
- In some way think any of this is particularly important to the universe outside the time/abstract-space bubble that is LessWrong this week.
Of course that comment actually lent credence to Eliezer (hence the humor) and was rather orthogonal to multi's position with respect to arrogance.
It's not that I mind too much sticking my neck out risking a social thrashing here or there. It's just that I have sufficient capability for sticking my neck out for things that I actually do mean and for some reason prefer any potential criticism to be correctly targeted. It says something about many nerds that they value being comprehended more highly than approval.
Replies from: Strange7↑ comment by dclayh · 2010-08-20T03:19:19.127Z · LW(p) · GW(p)
Veering wildly off-topic:
Arwen was crazy enough to give up the immortality she already had
Come on now. Humans are immortal in Tolkien, they just sit in a different waiting room. (And technically can't come back until the End of Days™, but who cares about that.)
Replies from: Strange7↑ comment by cousin_it · 2010-08-20T07:45:23.484Z · LW(p) · GW(p)
What Eliezer said. I was arguing from the assumption that he is wrong about FAI and stuff. If he's right about the object level, then he's not deluded in considering himself important.
Replies from: Vladimir_Nesov, wedrifid↑ comment by Vladimir_Nesov · 2010-08-20T14:03:39.980Z · LW(p) · GW(p)
I was arguing from the assumption that he is wrong about FAI and stuff. If he's right about the object level, then he's not deluded in considering himself important.
But if he is wrong about FAI and stuff, then he is still deluded not specifically about considering himself important, that implication is correct, he is deluded about FAI and stuff.
Replies from: cousin_it↑ comment by wedrifid · 2010-08-20T09:32:22.212Z · LW(p) · GW(p)
If he's right about the object level, then he's not deluded in considering himself important.
Which, of course, would still leave the second two dot points as answers to your question.
Replies from: cousin_it↑ comment by cousin_it · 2010-08-20T09:53:59.790Z · LW(p) · GW(p)
How so? Eliezer's thesis is "AGI is dangerous and FAI is possible". If he's wrong - if AGI poses no danger or FAI is impossible - then what do you need a Frodo for?
Replies from: wedrifid↑ comment by wedrifid · 2010-08-20T10:36:31.105Z · LW(p) · GW(p)
Edited the grandparent to disambiguate the context.
(I haven't discussed that particular thesis of Eliezer's and nor does doubting that particular belief seem to be a take home message from multi's post. The great grandparent is just a straightforward answer to the paragraph it quotes.)
↑ comment by Vladimir_Nesov · 2010-08-20T13:42:06.056Z · LW(p) · GW(p)
The previous post was fine, but this one is sloppy, and I don't think it's some kind of Machiavellian plot.
↑ comment by xamdam · 2010-08-20T01:44:32.130Z · LW(p) · GW(p)
But another part of me feels horror and disgust because after traumatic events in my own life I'd resolved to never do this thing again.
Because you were on the giving or on the receiving end of it?
What would that change, in the grand scheme of things or in your personal life? Are there not enough kooks in AI already?
Agreed; personally I de-converted myself from orthodox judaism, but I still find it crazy when people write big scholarly books debunking the bible; it's just useless a waste of energy (part of it is academic incentives).
They will waste their life (or spend it wisely, it's a matter of perspective), but if you join the tug-of-war, you'll waste a big chunk of yours as well.
I haven't been involved in these situations, but taking a cue from drug addicts (who incidentally have high suicide rate) most of them do not recover, but maybe 10% do. So most of the time you'll find frustration, but one in 10 you'd save a life, I am not sure if that's worthless.
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T00:46:34.753Z · LW(p) · GW(p)
It seems like an implication of your post that no one is ever allowed to believe they're saving the world. Do you agree that this is an implication? If not, why not?
Replies from: JRMayne, Unknowns, DanielVarga, taw↑ comment by JRMayne · 2010-08-20T03:12:00.999Z · LW(p) · GW(p)
Not speaking for multi, but, in any x-risk item (blowing up asteroids, stabilizing nuclear powers, global warming, catastrophic viral outbreak, climate change of whatever sort, FAI, whatever) for those working on the problem, there are degrees of realism:
"I am working on a project that may have massive effect on future society. While the chance that I specifically am a key person on the project are remote, given the fine minds at (Google/CDC/CIA/whatever), I still might be, and that's worth doing." - Probably sane, even if misguided.
"I am working on a project that may have massive effect on future society. I am the greatest mind in the field. Still, many other smart people are involved. The specific risk I am worried about may or not occur, but efforts to prevent its occurrence are valuable. There is some real possibility that I will the critical person on the project." - Possibly sane, even if misguided.
"I am working on a project that will save a near-infinite number of universes. In all likelihood, only I can achieve it. All of the people - even people perceived as having better credentials, intelligence, and ability - cannot do what I am doing. All critics of me are either ignorant, stupid, or irrational. If I die, the chance of multiverse collapse is radically increased; no one can do what I do. I don't care if other people view this as crazy, because they're crazy if they don't believe me." - Clinical diagnosis.
You're doing direct, substantial harm to your cause, because you and your views appear irrational. Those who hear about SIAI as the lead dog in this effort who are smart, have money, and are connected, will mostly conclude that this effort must not be worth anything.
I believe you had some language for Roko on the wisdom of damaging the cause in order to show off how smart you are.
I'm a little uncomfortable with the heat of my comment here, but other efforts have not been read the way I intended them by you (Others appeared to understand.) I am hopeful this is clear - and let me once again clarify that I had these views before multi's post. Before. Don't blame him again; blame me.
I'd like existential risk generally to be better received. In my opinion - and I may be wrong - you're actively hurting the cause.
--JRM
Replies from: None, JamesAndrix, Jonathan_Graehl↑ comment by [deleted] · 2010-08-20T16:33:53.493Z · LW(p) · GW(p)
I don't think Eliezer believes he's irreplaceable, exactly. He thinks, or I think he thinks, that any sufficiently intelligent AI which has not been built to the standard of Friendliness (as he defines it) is an existential risk. And the only practical means for preventing the development of UnFriendly AI is to develop superintelligent FAI first. The team to develop FAI needn't be SIAI, and Eliezer wouldn't necessarily be the most important contributor to the project, and SIAI might not ultimately be equal to the task. But if he's right about the risk and the solution, and his untimely demise were to doom the world, it would be because no-one else tried to do this, not because he was the only one who could.
Not that this rules out your interpretation. I'm sure he has a high opinion of his abilities as well. Any accusation of hubris should probably mention that he once told Aubrey de Grey "I bet I can solve ALL of Earth's emergency problems before you cure aging."
↑ comment by JamesAndrix · 2010-08-21T04:57:00.867Z · LW(p) · GW(p)
There may be multiple different projects projects, each necessary to save the world, and each having a key person who knows more about the project, and/or is more driven and/or is more capable than anyone else. Each such person has weirdly high expected utility, and could accurately make a statement like EY's and still not be the person with the highest expected utility. Their actual expected utility would depend on the complexity of the project and the surrounding community, and how much the success of the project alters the value of human survival.
This is similar to the idea that responsibility is not a division of 100%.
↑ comment by Jonathan_Graehl · 2010-08-20T04:27:31.029Z · LW(p) · GW(p)
What you say sounds reasonable, but I feel it's unwise for me to worry about such things. If I were to sound such a vague alarm, I wouldn't expect anyone to listen to me unless I'd made significant contributions in the field myself (I haven't).
↑ comment by Unknowns · 2010-08-20T03:18:34.891Z · LW(p) · GW(p)
Multifoliaterose said this:
The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity. At present, anybody who holds such a belief is suffering from extreme delusions of grandeur.
Note that there are qualifications on this. If you're standing by the button that ends the world, and refuse to press it when urged, or you prevent others from pressing it (e.g. Stanislav Petrov), then you may reasonably believe that you're saving the world. But no, you may not reasonably believe that you are saving the world based on long chains of reasoning based on your intuition, not on anything as certain as mathematics and logic, especially decades in advance of anything happening.
Replies from: Eliezer_Yudkowsky↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T03:20:55.452Z · LW(p) · GW(p)
It seems like an implication of this and other assumptions made by multi, and apparently shared by you, is that no one can believe themselves to be critical to a Friendly AI project that has a significant chance of success. Do you agree that this is an implication? If not, why not?
Replies from: Unknowns, None↑ comment by Unknowns · 2010-08-20T03:41:39.081Z · LW(p) · GW(p)
No, I don't agree this is an implication. I would say that no one can reasonably believe all of the following at the same time with a high degree of confidence:
1) I am critical to this Friendly AI project that has a significant chance of success. 2) There is no significant chance of Friendly AI without this project. 3) Without Friendly AI, the world is doomed.
But then, as you know, I don't consider it reasonable to put a high degree in confidence in number 3. Nor do many other intelligent people (such as Robin Hanson.) So it isn't surprising that I would consider it unreasonable to be sure of all three of them.
I also agree with Tetronian's points.
Replies from: Eliezer_Yudkowsky↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T03:57:56.558Z · LW(p) · GW(p)
I would say that no one can reasonably believe all of the following at the same time with a high degree of confidence: 1) I am critical to this Friendly AI project that has a significant chance of success. 2) There is no significant chance of Friendly AI without this project. 3) Without Friendly AI, the world is doomed.
I see. So it's not that any one of these statements is a forbidden premise, but that their combination leads to a forbidden conclusion. Would you agree with the previous sentence?
BTW, nobody please vote down the parent below -2, that will make it invisible. Also it doesn't particularly deserve downvoting IMO.
Replies from: Perplexed, Unknowns, CarlShulman↑ comment by Perplexed · 2010-08-20T04:16:27.018Z · LW(p) · GW(p)
I would suggest that, in order for this set of beliefs to become (psychiatrically?) forbidden, we need to add a fourth item. 4) Dozens of other smart people agree with me on #3.
If someone believes that very, very few people yet recognize the importance of FAI, then the conjunction of beliefs #1 thru #3 might be reasonable. But after #4 becomes true (and known to our protagonist), then continuing to hold #1 and #2 may be indicative of a problem.
Replies from: Eliezer_Yudkowsky, Perplexed↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T05:01:07.596Z · LW(p) · GW(p)
Dozens isn't sufficient. I asked Marcello if he'd run into anyone who seemed to have more raw intellectual horsepower than me, and he said that John Conway gave him that impression. So there are smarter people than me upon the Earth, which doesn't surprise me at all, but it might take a wider net than "dozens of other smart people" before someone comes in with more brilliance and a better starting math education and renders me obsolete.
Replies from: None, Spurlock, XiXiDu, Perplexed, whowhowho, TheAncientGeek↑ comment by Spurlock · 2010-08-20T17:26:47.720Z · LW(p) · GW(p)
Simply out of curiosity:
Plenty of criticism (some of it reasonable) has been lobbed at IQ tests and at things like the SAT. Is there a method known to you (or anyone reading) that actually measures "raw intellectual horsepower" in a reliable and accurate way? Aside from asking Marcello.
Replies from: thomblake, rabidchicken↑ comment by rabidchicken · 2010-08-21T17:02:22.027Z · LW(p) · GW(p)
Read the source code, and then visualize a few levels from Crysis or Metro 2033 in your head. While you render it, count the average Frames per second. Alternatively, see how quickly you can find the prime factors of every integer from 1 to 1000.
Which is to say... Humans in general have extremely limited intellectual power. instead of calculating things efficiently, we work by using various tricks with caches and memory to find answers. Therefore, almost all tasks are more dependant on practice and interest than they are on intelligence. So, rather then testing the statement "Eliezer is smart" it has more bearing on this debate to confirm "Eliezer has spent a large amount of time optimizing his cache for tasks relating to rationality, evolution, and artificial intelligence". Intelligence is overrated.
↑ comment by XiXiDu · 2010-08-20T10:29:58.246Z · LW(p) · GW(p)
Sheer curiosity, but have you or anyone ever contacted John Conway about the topic of u/FAI and asked him what the thinks about the topic, the risks associated with it and maybe the SIAI itself?
Replies from: xamdam↑ comment by xamdam · 2010-08-20T16:12:42.867Z · LW(p) · GW(p)
"raw intellectual power" != "relevant knowledge". Looks like he worked on some game theory, but otherwise not much relevancy. Should we ask Steven Hawking? Or take a poll of Nobel Laureates?
I am not saying that he can't be brought up to date in this kind of discussion, and has a lot to consider, but not asking him as things are indicates little.
Replies from: XiXiDu↑ comment by XiXiDu · 2010-08-20T20:09:55.301Z · LW(p) · GW(p)
Richard Dawkins seems to have enough power to infer the relevant knowledge from a single question.
↑ comment by TheAncientGeek · 2015-06-17T11:37:50.105Z · LW(p) · GW(p)
Domain knowledge is much more relevant than raw intelligence.
↑ comment by Perplexed · 2010-08-20T04:29:21.531Z · LW(p) · GW(p)
With the hint from EY on another branch, I see a problem in my argument. Our protagonist might circumvent my straitjacket by also believing 5) The key to FAI is TDT, but I have been so far unsuccessful in getting many of those dozens of smart people to listen to me on that subject.
I now withdraw from this conversation with my tail between my legs.
Replies from: katydee↑ comment by Unknowns · 2010-08-20T09:13:42.851Z · LW(p) · GW(p)
I wouldn't put it in terms of forbidden premises or forbidden conclusions.
But if each of these statements has a 90% of being true, and if they are assumed to be independent (which admittedly won't be exactly true), then the probability that all three are true would be only about 70%, which is not an extremely high degree of confidence; more like saying, "This is my opinion but I could easily be wrong."
Personally I don't think 1) or 3), taken in a strict way, could reasonably be said to have more than a 20% chance of being true. I do think a probability of 90% is a fairly reasonable assignment for 2), because most people are not going to bother about Friendliness. Accounting for the fact that these are not totally independent, I don't consider a probability assignment of more than 5% for the conjunction to be reasonable. However, since there are other points of view, I could accept that someone might assign the conjunction a 70% chance in accordance with the previous paragraph, without being crazy. But if you assign a probability much more than that I would have to withdraw this.
If the statements are weakened as Carl Shulman suggests, then even the conjunction could reasonably be given a much higher probability.
Also, as long as it is admitted that the probability is not high, you could still say that the possibility needs to be taken seriously because you are talking about the possible (if yet improbable) destruction of the world.
Replies from: Eliezer_Yudkowsky↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T18:21:21.584Z · LW(p) · GW(p)
I certainly do not assign a probability as high as 70% to the conjunction of all three of those statements.
And in case it wasn't clear, the problem I was trying to point out was simply with having forbidden conclusions - not forbidden by observation per se, but forbidden by forbidden psychology - and using that to make deductions about empirical premises that ought simply to be evaluated by themselves.
I s'pose I might be crazy, but you all are putting your craziness right up front. You can't extract milk from a stone!
Replies from: PaulAlmond, Unknowns, multifoliaterose↑ comment by PaulAlmond · 2010-08-28T21:55:00.497Z · LW(p) · GW(p)
Just curious (and not being 100% serious here): Would you have any concerns about the following argument (and I am not saying I accept it)?
- Assume that famous people will get recreated as AIs in simulations a lot in the future. School projects, entertainment, historical research, interactive museum exhibits, idols to be worshipped by cults built up around them, etc.
- If you save the world, you will be about the most famous person ever in the future.
- Therefore there will be a lot of Eliezer Yudkowsky AIs created in the future.
- Therefore the chances of anyone who thinks he is Eliezer Yudkowsky actually being the orginal, 21st century one are very small.
- Therefore you are almost certainly an AI, and none of the rest of us are here - except maybe as stage props with varying degrees of cognition (and you probably never even heard of me before, so someone like me would probably not get represented in any detail in an Eliezer Yudkowsky simulation). That would mean that I am not even conscious and am just some simple subroutine. Actually, now I have raised the issue to be scary, it looks a lot more alarming for me than it does for you as I may have just argued myself out of existence...
↑ comment by wedrifid · 2010-08-29T02:45:07.417Z · LW(p) · GW(p)
Actually, now I have raised the issue to be scary, it looks a lot more alarming for me than it does for you as I may have just argued myself out of existence...
That doesn't seem scary to me at all. I still know that there is at least one of me that I can consider 'real'. I will continue to act as if I am one of the instances that I consider me/important. I've lost no existence whatsoever.
↑ comment by Wei Dai (Wei_Dai) · 2010-08-29T02:04:48.348Z · LW(p) · GW(p)
You can see Eliezer's position on the Simulation Argument here.
↑ comment by Unknowns · 2010-08-20T18:29:01.160Z · LW(p) · GW(p)
That's good to know. I hope multifoliaterose reads this comment, as he seemed to think that you would assign a very high probability to the conjunction (and it's true that you've sometimes given that impression by your way of talking.)
Also, I didn't think he was necessarily setting up forbidden conclusions, since he did add some qualifications allowing that in some circumstances it could be justified to hold such opinions.
↑ comment by multifoliaterose · 2010-08-20T18:53:48.309Z · LW(p) · GW(p)
To be quite clear about which of Unknowns' points I object, my main objection is to the point:
I am critical to this Friendly AI project that has a significant chance of success
where 'I' is replaced by "Eliezer." I assign a probability of less than 10^(-9) to you succeeding in playing a critical role on the Friendly AI project that you're working on. (Maybe even much less than that - I would have to spend some time calibrating my estimate to make a judgment on precisely how low a probability I assign to the proposition.)
My impression is that you've greatly underestimated the difficulty of building a Friendly AI.
Replies from: Eliezer_Yudkowsky, None, Vladimir_Nesov↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T19:00:52.061Z · LW(p) · GW(p)
I assign a probability of less than 10^(-9) to you succeeding in playing a critical role on the Friendly AI project that you're working on.
I wish the laws of argument permitted me to declare that you had blown yourself up at this point, and that I could take my toys and go home. Alas, arguments are not won on a points system.
My impression is that you've greatly underestimated the difficulty of building a Friendly AI.
Out of weary curiosity, what is it that you think you know about Friendly AI that I don't?
And has it occurred to you that if I have different non-crazy beliefs about Friendly AI then my final conclusions might not be so crazy either, no matter what patterns they match in your craziness recognition systems?
Replies from: Unknowns, ata, multifoliaterose↑ comment by Unknowns · 2010-08-20T19:08:40.105Z · LW(p) · GW(p)
I agree it's kind of ironic that multi has such an overconfident probability assignment right after criticizing you for being overconfident. I was quite disappointed with his response here.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T19:52:48.483Z · LW(p) · GW(p)
Why does my probability estimate look overconfident?
Replies from: steven0461, Unknowns↑ comment by steven0461 · 2010-08-20T21:02:03.547Z · LW(p) · GW(p)
One could offer many crude back-of-envelope probability calculations. Here's one: let's say there's
- a 10% chance AGI is easy enough for the world to do in the next few decades
- a 1% chance that if the world can do it, a team of supergeniuses can do the Friendly kind first
- an independent 10% chance Eliezer succeeds at putting together such a team of supergeniuses
That seems conservative to me and implies at least a 1 in 10^4 chance. Obviously there's lots of room for quibbling here, but it's hard for me to see how such quibbling could account for five orders of magnitude. And even if post-quibbling you think you have a better model that does imply 1 in 10^9, you only need to put little probability mass on my model or models like it for them to dominate the calculation. (E.g., a 9 in 10 chance of a 1 in 10^9 chance plus a 1 in 10 chance of a 1 in 10^4 chance is close to a 1 in 10^5 chance.)
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T21:58:40.426Z · LW(p) · GW(p)
I don't find these remarks compelling. I feel similar remarks could be used to justify nearly anything. Of course, I owe you an explanation. One will follow later on.
↑ comment by Unknowns · 2010-08-21T05:26:44.670Z · LW(p) · GW(p)
Unless you've actually calculated the probability mathematically, a probability of one in a billion for a natural language claim that a significant number of people accept as likely true is always overconfident. Even Eliezer said that he couldn't assign a probability as low as one in a billion for the claim "God exists" (although Michael Vassar criticized him for this, showing himself to be even more overconfident than Eliezer.)
Replies from: komponisto, None, multifoliaterose↑ comment by komponisto · 2010-08-23T11:25:52.350Z · LW(p) · GW(p)
Unless you've actually calculated the probability mathematically, a probability of one in a billion for a natural language claim that a significant number of people accept as likely true is always overconfident.
I'm afraid I have to take severe exception to this statement.
You give the human species far too much credit if you think that our mere ability to dream up a hypothesis automatically raises its probability above some uniform lower bound.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-23T11:51:52.388Z · LW(p) · GW(p)
I am aware of your disagreement, for example as expressed by the absurd claims here. Yes, my basic idea is, unlike you, to give some credit to the human species. I think there's a limit on how much you can disagree with other human beings-- unless you're claiming to be something superhuman.
Did you see the link to this comment thread? I would like to see your response to the discussion there.
Replies from: komponisto↑ comment by komponisto · 2010-08-23T19:58:47.136Z · LW(p) · GW(p)
I think there's a limit on how much you can disagree with other human beings-- unless you're claiming to be something superhuman.
At least for epistemic meanings of "superhuman", that's pretty much the whole purpose of LW, isn't it?
Did you see the link to this comment thread? I would like to see your response to the discussion there.
My immediate response is as follows: yes, dependency relations might concentrate most of the improbability of a religion to a relatively small subset of its claims. But the point is that those claims themselves possess enormous complexity (which may not necessarily be apparent on the surface; cf. the simple-sounding "the woman across the street is a witch; she did it").
Replies from: Unknowns↑ comment by Unknowns · 2010-08-24T03:50:26.736Z · LW(p) · GW(p)
Let's pick an example. How probable do you think it is that Islam is a true religion? (There are several ways to take care of logical contradictions here, so saying 0% is not an option.)
Suppose there were a machine--for the sake of tradition, we can call it Omega--that prints out a series of zeros and ones according to the following rule. If Islam is true, it prints out a 1 on each round, with 100% probability. If Islam is false, it prints out a 0 or a 1, each with 50% probability.
Let's run the machine... suppose on the first round, it prints out a 1. Then another. Then another. Then another... and so on... it's printed out 10 1's now. Of course, this isn't so improbable. After all, there was a 1/1024 chance of it doing this anyway, even if Islam is false. And presumably we think Islam is more likely than this to be false, so there's a good chance we'll see a 0 in the next round or two...
But it prints out another 1. Then another. Then another... and so on... It's printed out 20 of them. Incredible! But we're still holding out. After all, million to one chances happen every day...
Then it prints out another, and another... it just keeps going... It's printed out 30 1's now. Of course, it did have a chance of one in a billion of doing this, if Islam were false...
But for me, this is my lower bound. At this point, if not before, I become a Muslim. What about you?
You've been rather vague about the probabilities involved, but you speak of "double digit negative exponents" and so on, even saying that this is "conservative," which implies possibly three digit exponents. Let's suppose you think that the probability that Islam is true is 10^-20; this would seem to be very conservative, by your standards. According to this, to get an equivalent chance, the machine would have to print out 66 1's.
If the machine prints out 50 1's, and then someone runs in and smashes it beyond repair, before it has a chance to continue, will you walk away, saying, "There is a chance at most of 1 in 60,000 that Islam is true?"
If so, are you serious?
Replies from: cousin_it, komponisto, Richard_Kennaway, None, cousin_it, Clippy↑ comment by cousin_it · 2010-08-24T23:33:54.066Z · LW(p) · GW(p)
Thank you a lot for posting this scenario. It's instructive from the "heuristics and biases" point of view.
Imagine there are a trillion variants of Islam, differing by one paragraph in the holy book or something. At most one of them can be true. You pick one variant at random, test it with your machine and get 30 1's in a row. Now you should be damn convinced that you picked the true one, right? Wrong. Getting this result by a fluke is 1000x more likely than having picked the true variant in the first place. Probability is unintuitive and our brains are mush, that's all I'm sayin'.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-25T05:41:52.580Z · LW(p) · GW(p)
I agree with this. But if the scenario happened in real life, you would not be picking a certain variant. You would be asking the vague question, "Is Islam true," to which the answer would be yes if any one of those trillion variants, or many others, were true.
Yes, there are trillions of possible religions that differ from one another as much as Islam differs from Judaism, or whatever. But only a few of these are believed by human beings. So I still think I would convert after 30 1's, and I think this would reasonable.
Replies from: cousin_it, Vladimir_Nesov↑ comment by cousin_it · 2010-08-25T11:20:38.779Z · LW(p) · GW(p)
If a religion's popularity raises your prior for it so much, how do you avoid Pascal's Mugging with respect to the major religions of today? Eternity in hell is more than 2^30 times worse than anything you could experience here; why aren't you religious already?
Replies from: Unknowns, Wei_Dai, thomblake↑ comment by Unknowns · 2010-08-26T06:35:02.714Z · LW(p) · GW(p)
It doesn't matter whether it raises your prior or not; eternity in hell is also more than 2^3000 times worse etc... so the same problem will apply in any case.
Elsewhere I've defended Pascal's Wager against the usual criticisms, and I still say it's valid given the premises. But there are two problematic premises:
1) It assumes that utility functions are unbounded. This is certainly false for all human beings in terms of revealed preference; it is likely false even in principle (e.g. the Lifespan Dilemma).
2) It assumes that humans are utility maximizers. This is false in fact, and even in theory most of us would not want to self-modify to become utility maximizers; it would be a lot like self-modifying to become a Babyeater or a Super-Happy.
↑ comment by Wei Dai (Wei_Dai) · 2010-08-25T22:24:28.874Z · LW(p) · GW(p)
Do you have an answer for how to avoid giving in to the mugger in Eliezer's original Pascal's Mugging scenario? If not, I don't think your question is a fair one (assuming it's meant to be rhetorical).
Replies from: cousin_it↑ comment by cousin_it · 2010-08-26T17:36:03.970Z · LW(p) · GW(p)
I don't have a conclusive answer, but many people say they have bounded utility functions (you see Unknowns pointed out that possibility too). The problem with assigning higher credence to popular religions is that it forces your utility bound to be lower if you want to reject the mugging. Imagining a billion lifetimes is way easier than imagining 3^^^^3 lifetimes. That was the reason for my question.
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2010-08-27T07:47:02.733Z · LW(p) · GW(p)
My answer (for why I don't believe in a popular religion as a form of giving in to a Pascal's Mugging) would be that I'm simultaneously faced with a number of different Pascal's Muggings, some of which are mutually exclusive, so I can't just choose to give in to all of them. And I'm also unsure of what decision theory/prior/utility function I should use to decide what to do in the face of such Muggings. Irreversibly accepting any particular Mugging in my current confused state is likely to be suboptimal, so the best way forward at this point seems to be to work on the relevant philosophical questions.
Replies from: endoself↑ comment by endoself · 2011-05-08T21:51:39.940Z · LW(p) · GW(p)
That's what I think too! You're only the second other person I have seen make this explicit, so I wonder how many people have even considered this. Do you think more people would benefit from hearing this argument?
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2011-05-09T06:08:38.254Z · LW(p) · GW(p)
Do you think more people would benefit from hearing this argument?
Sure, why do you ask? (If you're asking because I've thought of this argument but haven't already tried to share it with a wider audience, it probably has to do with reasons, e.g., laziness, that are unrelated to whether I think more people would benefit from hearing it.)
Replies from: endoself↑ comment by thomblake · 2010-08-25T15:17:50.232Z · LW(p) · GW(p)
Pascal's Mugging
Oddly, I think you meant "Pascal's Wager".
Replies from: FAWS↑ comment by FAWS · 2010-08-25T15:28:56.959Z · LW(p) · GW(p)
Pascal's Mugging. Pascal's Wager with something breaking symmetry (in this case observed belief of others).
Replies from: thomblake↑ comment by Vladimir_Nesov · 2010-08-25T13:22:13.290Z · LW(p) · GW(p)
Yes, there are trillions of possible religions that differ from one another as much as Islam differs from Judaism, or whatever. But only a few of these are believed by human beings.
Privileging the hypothesis! That they are believed by human beings doesn't lend them probability.
Replies from: FAWS, Unknowns, Perplexed, Unknowns, thomblake, Cyan↑ comment by FAWS · 2010-08-25T13:55:40.911Z · LW(p) · GW(p)
Well, it does to the extent that lack of believers would be evidence against them. I'd say that Allah is considerably more probable than a similarly complex and powerful god who also wants to be worshiped and is equally willing to interact with humans, but not believed in by anyone at all. Still considerably less probable than the prior of some god of that general sort existing, though.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-25T14:09:17.558Z · LW(p) · GW(p)
Well, it does to the extent that lack of believers would be evidence against them.
Agreed, but then we have the original situation, if we only consider the set of possible gods that have the property of causing worshiping of themselves.
↑ comment by Unknowns · 2010-08-25T16:10:21.348Z · LW(p) · GW(p)
This whole discussion is about this very point. Downvoted for contradicting my position without making an argument.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-25T16:14:19.501Z · LW(p) · GW(p)
Your position statement didn't include an argument either, and the problem with it seems rather straightforward, so I named it.
Replies from: Unknowns↑ comment by Perplexed · 2010-08-25T14:58:05.638Z · LW(p) · GW(p)
Yes, there are trillions of possible religions that differ from one another as much as Islam differs from Judaism, or whatever. But only a few of these are believed by human beings.
Privileging the hypothesis! That they are believed by human beings doesn't lend them probability.
No. It doesn't lend probability, but it seems like it ought to lend something. What is this mysterious something? Lets call it respect.
Privileging the hypothesis is a fallacy. Respecting the hypothesis is a (relatively minor) method of rationality.
We respect the hypotheses that we find in a math text by investing the necessary mental resources toward the task of finding an analytic proof. We don't just accept the truth of the hypothesis on authority. But on the other hand, we don't try to prove (or disprove) just any old hypothesis. It has to be one that we respect.
We respect scientific hypotheses enough to invest physical resources toward performing experiments that might refute or confirm them. We don't expend those resources on just any scientific hypothesis. Only the ones we respect.
Does a religion deserve respect because it has believers? More respect if it has lots of believers? I think it does. Not privilege. Definitely not. But respect? Why not?
Replies from: FAWS, Vladimir_Nesov, None↑ comment by FAWS · 2010-08-25T15:42:00.686Z · LW(p) · GW(p)
You can dispense with this particular concept of respect since in both your examples you are actually supplied with sufficient Bayesian evidence to justify evaluating the hypothesis, so it isn't privileged. Whether this is also the case for believed in religions is the very point contested.
↑ comment by Vladimir_Nesov · 2010-08-25T15:37:25.512Z · LW(p) · GW(p)
Privileging the hypothesis is a fallacy. Respecting the hypothesis is a (relatively minor) method of rationality.
No, it's a method of anti-epistemic horror.
↑ comment by [deleted] · 2010-08-26T18:48:29.521Z · LW(p) · GW(p)
Yes, this seems right.
A priori, with no other evidence one way or another, a belief held by human beings is more likely to be true than not. If Ann says she had a sandwich for lunch, then her words are evidence that she actually had a sandwich for lunch.
Of course, we have external reason to doubt lots of things that human beings claim and believe, including religions. And a religion does not become twice as credible if it has twice as many adherents. Right now I believe we have good reason to reject (at least some of) the tenets of all religious traditions.
But it does make some sense to give some marginal privilege or respect to an idea based on the fact that somebody believes it, and to give the idea more credit if it's very durable over time, or if particularly clever people believe it. If it were any subject but religion -- if it were science, for instance -- this would be an obvious point. Scientific beliefs have often been wrong, but you'll be best off giving higher priors to hypotheses believed by scientists than to other conceivable hypotheses.
↑ comment by Unknowns · 2010-08-26T06:45:32.600Z · LW(p) · GW(p)
Also... if you haven't been to Australia, is it privileging the hypothesis to accept the word of those who say that it exists? There are trillions of possible countries that could exist that people don't believe exist...
And don't tell me they say they've been there... religious people say they've experienced angels etc. too.
And so on. People's beliefs in religion may be weaker than their belief in Austrialia, but it certainly is not privileging a random hypothesis.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-26T09:32:41.718Z · LW(p) · GW(p)
Your observations (of people claiming to having seen an angel, or a kangaroo) are distinct from hypotheses formed to explain those observations. If in a given case, you don't have reason to expect statements people make to be related to facts, then the statements people make taken verbatim have no special place as hypotheses.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-26T16:54:53.634Z · LW(p) · GW(p)
"You don't have reason to expect statements people make to be related to facts" doesn't mean that you have 100% certainty that they are not, which you would need in order to invoke privileging the hypothesis.
Replies from: None, Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-26T17:04:06.680Z · LW(p) · GW(p)
Now you are appealing to impossibility of absolute certainty, refuting my argument as not being that particular kind of proof. If hypothesis X is a little bit more probable than many others, you still don't have any reason to focus on it (and correlation could be negative!).
Replies from: Unknowns↑ comment by Unknowns · 2010-08-27T01:19:16.945Z · LW(p) · GW(p)
In principle the correlation could be negative but this is extremely unlikely and requires some very strange conditions (for example if the person is more likely to say that Islam is true if he knows it is false than if he knows it is true).
↑ comment by Cyan · 2010-08-25T17:21:16.468Z · LW(p) · GW(p)
I disagree; given that most of the religions in question center on human worship of the divine, I have to think that Pr(religion X becomes known among humans | religion X is true) > Pr(religion X does not become known among humans | religion X is true). But I hate to spend time arguing about whether a likelihood ratio should be considered strictly equal to 1 or equal to 1 + epsilon when the prior probabilities of the hypotheses in question are themselves ridiculously small.
↑ comment by komponisto · 2010-08-24T22:53:27.541Z · LW(p) · GW(p)
If the machine prints out 50 1's, and then someone runs in and smashes it beyond repair, before it has a chance to continue, will you walk away, saying, "There is a chance at most of 1 in 60,000 that Islam is true?"
If so, are you serious?
Of course I'm serious (and I hardly need to point out the inadequacy of the argument from the incredulous stare). If I'm not going to take my model of the world seriously, then it wasn't actually my model to begin with.
Sewing-Machine's comment below basically reflects my view, except for the doubts about numbers as a representation of beliefs. What this ultimately comes down to is that you are using a model of the universe according to which the beliefs of Muslims are entangled with reality to a vastly greater degree than on my model. Modulo the obvious issues about setting up an experiment like the one you describe in a universe that works the way I think it does, I really don't have a problem waiting for 66 or more 1's before converting to Islam. Honest. If I did, it would mean I had a different understanding of the causal structure of the universe than I do.
Further below you say this, which I find revealing:
If this actually happened to you, and you walked away and did not convert, would you have some fear of being condemned to hell for seeing this and not converting? Even a little bit of fear? If you would, then your probability that Islam is true must be much higher than 10^-20, since we're not afraid of things that have a one in a hundred billion chance of happening.
As it happens, given my own particular personality, I'd probably be terrified. The voice in my head would be screaming. In fact, at that point I might even be tempted to conclude that expected utilities favor conversion, given the particular nature of Islam.
But from an epistemic point of view, this doesn't actually change anything. As I argued in Advancing Certainty, there is such a thing as epistemically shutting up and multiplying. Bayes' Theorem says the updated probability is one in a hundred billion, my emotions notwithstanding. This is precisely the kind of thing we have to learn to do in order to escape the low-Earth orbit of our primitive evolved epistemology -- our entire project here, mind you -- which, unlike you (it appears), I actually believe is possible.
Replies from: Wei_Dai, Unknowns↑ comment by Wei Dai (Wei_Dai) · 2010-08-25T00:59:57.308Z · LW(p) · GW(p)
Has anyone done a "shut up and multiply" for Islam (or Christianity)? I would be interested in seeing such a calculation. (I did a Google search and couldn't find anything directly relevant.) Here's my own attempt, which doesn't get very far.
Let H = "Islam is true" and E = everything we've observed about the universe so far. According to Bayes:
P(H | E) = P(E | H) P(H) / P(E)
Unfortunately I have no idea how to compute the terms above. Nor do I know how to argue that P(H|E) is as small as 10^-20 without explicitly calculating the terms. One argument might be that P(H) is very small because of the high complexity of Islam, but since E includes "23% of humanity believe in some form of Islam", the term for the complexity of Islam seems to be present in both the numerator and denominator and therefore cancel each other out.
If someone has done such a calculation/argument before, please post a link?
Replies from: cousin_it, FAWS, None, None, gjm, Unknowns↑ comment by cousin_it · 2010-08-25T13:03:20.244Z · LW(p) · GW(p)
P(E) includes the convincingness of Islam to people on average, not the complexity of Islam. These things are very different because of the conjunction fallacy. So P(H) can be a lot smaller than P(E).
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2010-08-25T22:26:06.789Z · LW(p) · GW(p)
I don't understand how P(E) does not include a term for the complexity of Islam, given that E contains Islam, and E is not so large that it takes a huge number of bits to locate Islam inside E.
Replies from: Vladimir_Nesov, Furcas, cousin_it, cousin_it↑ comment by Vladimir_Nesov · 2010-08-26T09:51:29.226Z · LW(p) · GW(p)
It doesn't take a lot of bits to locate "Islam is false" based on "Islam is true". Does it mean that all complex statements have about 50% probability?
Replies from: cousin_it↑ comment by Furcas · 2010-08-25T23:37:01.586Z · LW(p) · GW(p)
I don't think that's true; cousin_it had it right the first time. The complexity of Islam is the complexity of a reality that contains an omnipotent creator, his angels, Paradise, Hell, and so forth. Everything we've observed about the universe includes people believing in Islam, but not the beings and places that Islam says exist.
In other words, E contains Islam the religion, not Islam the reality.
Replies from: PaulAlmond↑ comment by PaulAlmond · 2010-08-25T23:42:16.656Z · LW(p) · GW(p)
The really big problem with such a reality is that it contains a fundamental, non-contingent mind (God's/Allah's, etc) - and we all know how much describing one of those takes - and the requirement that God is non-contingent means we can't use any simpler, underlying ideas like Darwinian evolution. Non-contingency, in theory selection terms, is a god killer: It forces God to incur a huge information penalty - unless the theist refuses even to play by these rules and thinks God is above all that - in which case they aren't even playing the theory selection game.
Replies from: Perplexed, Furcas, byrnema↑ comment by Perplexed · 2010-08-26T00:44:41.521Z · LW(p) · GW(p)
I don't see this. Why assume that the non-contingent, pre-existing God is particularly complex. Why not assume that the current complexity of God (if He actually is complex) developed over time as the universe evolved since the big bang. Or, just as good, assume that God became complex before He created this universe.
It is not as if we know enough about God to actually start writing down that presumptive long bit string. And, after all, we don't ask the big bang to explain the coastline of Great Britain.
Replies from: PaulAlmond↑ comment by PaulAlmond · 2010-08-26T00:58:08.546Z · LW(p) · GW(p)
If we do that, should we even call that "less complex earlier version of God" God? Would it deserve the title?
Replies from: Perplexed↑ comment by Furcas · 2010-08-25T23:49:49.155Z · LW(p) · GW(p)
Non-contingency, in theory selection terms, is a god killer
Agreed. It's why I'm so annoyed when even smart atheists say that God was an ok hypothesis before evolution was discovered. God was always one of the worst possible hypotheses!
unless the theist refuses even to play by these rules and thinks God is above all that - in which case they aren't even playing the theory selection game.
Or, put more directly: Unless the theist is deluding himself. :)
Replies from: cousin_it↑ comment by cousin_it · 2010-08-27T09:58:18.807Z · LW(p) · GW(p)
I'm confused. In the comments to my post you draw a distinction between an "event" and a "huge set of events", saying that complexity only applies to the former but not the latter. But Islam is also a "huge set of events" - it doesn't predict just one possible future, but a wide class of them (possibly even including our actual world, ask any Muslim!), so you can't make an argument against it based on complexity of description alone. Does this mean you tripped on the exact same mine I was trying to defuse with my post?
I'd be very interested in hearing a valid argument about the "right" prior we should assign to Islam being true - how "wide" the set of world-programs corresponding to it actually is - because I tried to solve this problem and failed.
Replies from: Furcas↑ comment by Furcas · 2010-08-30T04:56:31.990Z · LW(p) · GW(p)
Sorry, I was confused. Just ignore that comment of mine in your thread.
I'm not sure how to answer your question because as far as I can tell you've already done so. The complexity of a world-program gives its a priori probability. The a priori probability of a hypothesis is the sum of the probabilities of all the world-programs it contains. What's the problem?
↑ comment by byrnema · 2010-08-26T00:20:24.499Z · LW(p) · GW(p)
The problem is that reality itself is apparently fundamentally non-contingent. Adding "mind" to all that doesn't seem so unreasonable.
Replies from: PaulAlmond↑ comment by PaulAlmond · 2010-08-26T00:30:43.458Z · LW(p) · GW(p)
Do you mean it doesn't seem so unreasonable to you, or to other people?
Replies from: byrnema↑ comment by byrnema · 2010-08-26T01:04:24.895Z · LW(p) · GW(p)
By reasonable, I mean the hypothesis is worth considering, if there were reasons to entertain it. That is, if someone suspected there was a mind behind reality, I don't think they should dismiss it out of hand as unreasonable because this mind must be non-contingent.
In fact, we should expect any explanation of our creation to be non-contingent, since physical reality appears to be so.
For example, if it's reasonable to consider the probability that we're in a simulation, then we're considering a non-contingent mind creating the simulation we're in.
↑ comment by cousin_it · 2010-08-25T22:39:54.204Z · LW(p) · GW(p)
Whoops, you're right. Sorry. I didn't quite realize you were talking about the universal prior again :-)
But I think the argument can still be made to work. P(H) doesn't depend only on the complexity of Islam - we must also take into account the internal structure of Islam. For example, the hypothesis "A and B and C and ... and Z" has the same complexity as "A or B or C or ... or Z", but obviously the former is way less probable. So P(H) and P(E) have the same term for complexity, but P(H) also gets a heavy "conjunction penalty" which P(E) doesn't get because people are susceptible to the conjunction fallacy.
It's slightly distressing that my wrong comment was upvoted.
↑ comment by cousin_it · 2010-08-25T22:47:19.167Z · LW(p) · GW(p)
Whoops, you're right. Now I'm ashamed that my comment got upvoted.
I think the argument may still be made to work by fleshing out the nonstandard notion of "complexity" that I had in my head when writing it :-) Your prior for a given text being true shouldn't depend only on the text's K-complexity. For example, the text "A and B and C and D" has the same complexity as "A or B or C or D", but the former is way less probable. So P(E) and P(H) may have the same term for complexity, but P(H) also gets a "conjunction penalty" that P(E) doesn't get because people are prey to the conjunction fallacy.
EDIT: this was yet another mistake. Such an argument cannot work because P(E) is obviously much smaller than P(H), because E is a huge mountain of evidence and H is just a little text. When trying to reach the correct answer, we cannot afford to ignore P(E|H).
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-26T09:44:28.305Z · LW(p) · GW(p)
For simplicity we may assume P(E|H) to be near-certainty: if there is an attention-seeking god, we'd know about it. This leaves P(E) and P(H), and P(H|E) is tiny exactly for the reason you named: P(H) is much smaller than P(E), because H is optimized for meme-spreading to a great extent, which makes for a given complexity (that translates into P(H)) probability of gaining popularity P(E) comparatively much higher.
Thus, just arguing from complexity indeed misses the point, and the real reason for improbability of cultish claims is that they are highly optimized to be cultish claims.
For example, compare with tossing a coin 50 times: the actual observation, whatever that is, will be a highly improbable event, and theoretical prediction from the model of fair coin will be too. But if the observation is highly optimized to attract attention, for example it's all 50 tails, then theoretical model crumbles, and not because the event you've observed is too improbable according to it, but because other hypotheses win out.
↑ comment by FAWS · 2010-08-25T16:00:32.514Z · LW(p) · GW(p)
the term for the complexity of Islam seems to be present in both the numerator and denominator and therefore cancel each other out.
Actually it doesn't, human generated complexity is different from naturally generated complexity (for instance it fits into narratives, apparent holes are filled with the sort of justifications a human is likely to think of etc.). That's one of the ways you can tell stories from real events. Religious accounts contain much of what looks like human generated complexity.
↑ comment by [deleted] · 2010-08-25T15:49:31.328Z · LW(p) · GW(p)
Here's a somewhat rough way of estimating probabilities of unlikely events. Let's say that an event X with P(X) = about 1-in-10 is a "lucky break." Suppose that there are L(1) ways that Y could occur on account of a single lucky break, L(2) ways that Y could occur on account of a pair of independent lucky breaks, L(3) ways that Y could occur on account of 3 independent lucky breaks, and so on. Then P(Y) is approximately the sum over all n of L(n)/10^n. I have the feeling that arguments about whether P(Y) is small versus extremely small are arguments about the growth rate of L(n).
I discussed the problem of estimating P("23% of humanity believes...") here. I'd be grateful for thoughts or criticisms.
↑ comment by [deleted] · 2010-08-25T01:37:42.551Z · LW(p) · GW(p)
This is a small point but "E includes complex claim C" does not imply that the (for instance, Kolmogorov) complexity of E is as large as the Kolmogorov complexity of C. The complexity of the digits of square root of 2 is pretty small, but they contain strings of arbitrarily high complexity.
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2010-08-25T01:56:07.605Z · LW(p) · GW(p)
E includes C implies that K(C) <= K(E) + K(information needed to locate C within E). In this case K(information needed to locate C within E) seems small enough not to matter to the overall argument, which is why I left it out. (Since you said "this is a small point" I guess you probably understand and agree with this.)
Replies from: None↑ comment by [deleted] · 2010-08-25T02:09:51.370Z · LW(p) · GW(p)
Actually no I hadn't thought of that. But I wonder if the amount of information it takes to locate "lots of people are muslims" within E is as small as you say. My particular E does not even contain that much information about Islam, and how people came to believe it, but it does contain a model of how people come to believe weird things in general. Is that a misleading way of putting things? I can't tell.
↑ comment by gjm · 2015-06-17T13:39:01.633Z · LW(p) · GW(p)
There are some very crude sketches of shutting-up-and-multiplying, from one Christian and a couple of atheists, here (read the comments as well as the post itself), and I think there may be more with a similar flavour in other blog posts there (and their comments) from around the same time.
(The author of the blog has posted a little on LW. The two skeptics responsible for most of the comments on that post have both been quite active here. One of them still is, and is in fact posting this comment right now :-).)
↑ comment by Unknowns · 2010-08-25T07:06:33.313Z · LW(p) · GW(p)
Wei Dai, exactly. The point about about the complexity of the thing is included in the fact that people believe it was the point I have been making all along. Regardless of what you think the resulting probability is, most of the "evidence" for Islam consists in the very fact that some people think it is true-- and as you show in your calculation, this is very strong evidence.
It seems to me that komponisto and others are taking it to be known with 100% certainly that Islam and the like were generated by some random process, and then trying to determine what the probability would be.
Now I know that most likely Mohammed was insane and in effect the Koran was in fact generated by a random process. But I certainly don't know how you can say that the probability that it wasn't generated randomly is 1 in 10^20 or lower. And in fact if you're going to assign a probability like this you should have an actual calculation.
↑ comment by Unknowns · 2010-08-25T07:13:27.097Z · LW(p) · GW(p)
I agree that your position is analogous to "shutting up and multiplying." But in fact, Eliezer may have been wrong about that in general -- see the Lifespan Dilemma -- because people's utility functions are likely not unbounded.
In your case, I agree with shutting up and multiplying when we have a way to calculate the probabilities. In this case, we don't, so we can't do it. If you had a known probability (see cousin_it's comment on the possible trillions of variants of Islam) of one in a trillion, then I would agree with walking away after seeing 30 1's, regardless of the emotional effect of this.
But in reality, we have no such known probability. The result is that you are going to have to use some base rate: "things that people believe" or more accurately, "strange things that people believe" or whatever. In any case, whatever base rate you use, it will not have a probability anywhere near 10^-20 (i.e. more than 1 in 10^20 strange beliefs is true etc.)
My real point about the fear is that your brain doesn't work the way your probabilities do-- even if you say you are that certain, your brain isn't. And if we had calculated the probabilities, you would be justified in ignoring your brain. But in fact, since we haven't, your brain is more right than you are in this case. It is less certain precisely because you are simply not justified in being that certain.
↑ comment by Richard_Kennaway · 2010-08-25T15:27:58.357Z · LW(p) · GW(p)
But for me, this is my lower bound. At this point, if not before, I become a Muslim. What about you?
At this point, if not before, I doubt Omega's reliability, not mine.
Replies from: Pavitra, Unknowns↑ comment by Pavitra · 2010-08-26T06:42:29.368Z · LW(p) · GW(p)
It is a traditional feature of Omega that you have confidence 1 in its reliability and trustworthiness.
Replies from: Richard_Kennaway, None↑ comment by Richard_Kennaway · 2010-08-26T07:31:30.544Z · LW(p) · GW(p)
It is a traditional feature of Omega that you have confidence 1 in its reliability and trustworthiness.
Traditions do not always make sense, neither are they necessarily passed down accurately. The original Omega, the one that appears in Newcomb's problem, does not have to be reliable with probability 1 for that problem to be a problem.
Of course, to the purist who says that 0 and 1 are not probabilities, you've just sinned by talking about confidence 1, but the problem can be restated to avoid that by asking for one's conditional probability P(Islam | Omega is and behaves as described).
In the present case, the supposition that one is faced with an overwhelming likelihood ratio raising the probability that Islam is true by an unlimited amount is just a blue tentacle scenario. Any number that anyone who agrees with the general anti-religious view common on LessWrong comes up with is going to be nonsense. Professing, say, 1 in a million for Islam on the grounds that 1 in a billion or 1 in a trillion is too small a probability for the human brain to cope with is the real cop-out, a piece of reversed stupidity with no justification of its own.
The scenario isn't going to happen. Forcing your brain to produce an answer to the question "but what if it did?" is not necessarily going to produce a meaningful answer.
Replies from: Pavitra↑ comment by Pavitra · 2010-08-26T08:21:03.452Z · LW(p) · GW(p)
Traditions do not always make sense, neither are they necessarily passed down accurately. The original Omega, the one that appears in Newcomb's problem, does not have to be reliable with probability 1 for that problem to be a problem.
Quite true. But if you want to dispute the usefulness of this tradition, you should address the broader and older tradition of which it is an instance: that thought experiments should abstract away real-world details irrelevant to the main point.
Of course, to the purist who says that 0 and 1 are not probabilities, you've just sinned by talking about confidence 1
This is a pet peeve of mine, and I've wanted an excuse to post this rant for a while. Don't take it personally.
That "purist" is as completely wrong as the person who insists that there is no such thing as centrifugal force. They are ignoring the math in favor of a meme that enables them to feel smugly superior.
0 and 1 are valid probabilities in every mathematical sense: the equations of probability don't break down when passed p=0 or p=1 the way they do with genuine nonprobabilities like -1 or 2. A probability of 0 or 1 is like a perfect vacuum: it happens not to occur in the world that we happen to inhabit, but it is perfectly well-defined, we can do math with it without any difficulty, and it is extraordinarily useful in thought experiments.
When asked to consider a spherical black body of radius one meter resting on a frictionless plane, you don't respond "blue tentacles", you do the math.
Replies from: Richard_Kennaway↑ comment by Richard_Kennaway · 2010-08-26T12:02:45.102Z · LW(p) · GW(p)
I agree with the rant. 0 and 1 are indeed probabilities, and saying that they are not is a misleading way of enjoining people to never rule out anything. Mathematically, P(~A|A) is zero, not epsilon, and P(A|A) is 1, not 1-epsilon. Practically, 0 and 1 in subjective judgements mean as near to 0 and 1 as makes no practical difference. When I agree a rendezvous with someone, I don't say "there's a 99% chance I'll be there", I say "I'll be there".
Where we part ways is in our assessment of the value of this thought-experiment. To me it abstracts and assumes away so much that what is left does not illuminate anything. I can calculate 2^{-N}, but asked how large N would have to be to persuade me of some fantastic claim backed by this fantastic machine I simply cannot name any value. I have no confidence that whatever value I named would be the value I would actually use were this impossible scenario to come to pass.
Replies from: Pavitra↑ comment by Pavitra · 2010-08-26T20:43:48.576Z · LW(p) · GW(p)
Fair enough. But if we're doing that, I think the original question with the Omega machine abstracts too much away. Let's consider the kind of evidence that we would actually expect to see if Islam were true.
Let us stipulate that, on the 1st of Muḥarram, a prominent ayatollah claims to have suddenly become a prophet. They go on television and answer questions on all topics. All verifiable answers they give, including those to NP-complete questions submitted for experimental purposes, turn out to be true. The new prophet asserts the validity of the Qur'an as holy scripture and of Allah as the one God.
There is a website where you can suggest questions to put to the new prophet. Not all submitted questions get answered, due to time constraints, but interesting ones do get in reasonably often. Are there any questions you'd like to ask?
Replies from: PaulAlmond, PaulAlmond, Richard_Kennaway↑ comment by PaulAlmond · 2010-08-26T21:20:11.462Z · LW(p) · GW(p)
I'll give a reworded version of this, to take it out of the context of a belief system with which we are familiar. I'm not intending any mockery by this: It is to make a point about the claims and the evidence:
"Let us stipulate that, on Paris Hilton's birthday, a prominent Paris Hilton admirer claims to have suddenly become a prophet. They go on television and answer questions on all topics. All verifiable answers they give, including those to NP-complete questions submitted for experimental purposes, turn out to be true. The new prophet asserts that Paris Hilton is a super-powerful being sent here from another world, co-existing in space with ours but at a different vibrational something or whatever. Paris Hilton has come to show us that celebrity can be fun. The entire universe is built on celebrity power. Madonna tried to teach us this when she showed us how to Vogue but we did not listen and the burden of non-celebrity energy threatens to weigh us down into the valley of mediocrity when we die instead of ascending to a higher plane where each of us gets his/her own talkshow with an army of smurfs to do our bidding. Oh, and Sesame Street is being used by the dark energy force to send evil messages into children's feet. (The brain only appears to be the source of consciousness: Really it is the feet. Except for people with no feet. (Ah! I bet you thought I didn't think of that.) Today's lucky food: custard."
There is a website where you can suggest questions to put to the new prophet. Not all submitted questions get answered, due to time constraints, but interesting ones do get in reasonably often. Are there any questions you'd like to ask?"
The point I am making here is that the above narrative is absurd, and even if he can demonstrate some unusual ability with predictions or NP problems (and I admit the NP problems would really impress me), there is nothing that makes that explanation more sensible than any number of other stupid explanations. Nor does he have an automatic right to be believed: His explanation is just too stupid.
↑ comment by PaulAlmond · 2010-08-26T20:57:06.097Z · LW(p) · GW(p)
Yes - I would ask this question:
"Mr Prophet, are you claiming that there is no other theory to account for all this that has less intrinsic information content than a theory which assumes the existence of a fundamental, non-contingent mind - a mind which apparently cannot be accounted for by some theory containing less information, given that the mind is supposed to be non-contingent?"
He had better have a good answer to that: Otherwise I don't care how many true predictions he has made or NP problems he has solved. None of that comes close to fixing the ultra-high information loading in his theory.
Replies from: Pavitra↑ comment by Pavitra · 2010-08-26T22:18:56.682Z · LW(p) · GW(p)
"The reason you feel confused is because you assume the universe must have a simple explanation.
The minimum message length necessary to describe the universe is long -- long enough to contain a mind, which in fact it does. There is no fundamental reason why the Occamian prior must be appropriate. It so happens that Allah has chosen to create a world that, to a certain depth, initially appears to follow that law, but Occam will not take you all the way to the most fundamental description of reality.
I could write out the actual message description, but to demonstrate that the message contains a mind requires volumes of cognitive science that have not been developed yet. Since both the message and the proof of mind will be discovered by science within the next hundred years, I choose to spend my limited time on earth in other areas."
Replies from: PaulAlmond↑ comment by PaulAlmond · 2010-08-26T22:24:32.853Z · LW(p) · GW(p)
Do you think that is persuasive?
Replies from: Pavitra↑ comment by Richard_Kennaway · 2010-08-26T22:15:14.808Z · LW(p) · GW(p)
A Moslem would say to him, "Mohammed (pbuh) is the Seal of the Prophets: there can be none after Him. The Tempter whispers your clever answers in your ear, and any truth in them is only a ruse and a snare!" A Christian faced with an analogous Christian prophet would denounce him as the Antichrist. I ask -- not him, but you -- why I should believe he is as trustworthy on religion as he is on subjects where I can test him?
I might incidentally ask him to pronounce on the validity of the hadith. I have read the Qur'an and there is remarkably little in it but exhortations to serve God.
"Also, could you settle all the schisms among those who already believe in the validity of the Qur'an as holy scripture and of Allah as the one God, and still want to bomb each other over their interpretations?"
Replies from: Alicorn, Pavitra↑ comment by Alicorn · 2010-08-26T22:52:18.294Z · LW(p) · GW(p)
A Christian faced with an analogous Christian prophet would denounce him as the Antichrist.
This is sect-dependent. The Mormons would probably be quite happy to accept one provided he attained prophet-hood through church-approved channels.
↑ comment by Pavitra · 2010-08-27T02:00:52.466Z · LW(p) · GW(p)
Mohammed (pbuh) is the Seal of the Prophets: there can be none after Him.
I wasn't aware of that particular tenet. I suppose the Very Special Person would have to identify as some other role than prophet.
I ask -- not him, but you -- why I should believe he is as trustworthy on religion as he is on subjects where I can test him?
If your prior includes the serious possibility of a Tempter that seems reliable until you have to trust it on something important, why couldn't the Tempter also falsify scientific data you gather?
I might incidentally ask him to pronounce on the validity of the hadith. I have read the Qur'an and there is remarkably little in it but exhortations to serve God.
"Indeed, the service of God is the best of paths to walk in life."
"Also, could you settle all the schisms among those who already believe in the validity of the Qur'an as holy scripture and of Allah as the one God, and still want to bomb each other over their interpretations?"
"Sure, that's why I'm here. Which point of doctrine do you want to know about?"
Replies from: Richard_Kennaway↑ comment by Richard_Kennaway · 2010-08-27T07:10:46.703Z · LW(p) · GW(p)
If your prior includes the serious possibility of a Tempter that seems reliable until you have to trust it on something important, why couldn't the Tempter also falsify scientific data you gather?
When I condition on the existence of this impossible prophet, many improbable ideas are raised to attention, not merely the one that he asserts.
To bring the thought-experiment slightly closer to reality, aliens arrive, bringing advanced technology and religion. Do we accept the religion along with the technology? I'm sure science fiction has covered that one umpteen times, but the scenario has already been played out in history, with European civilisation as the aliens. They might have some things worth taking regarding how people should deal with each other, but strange people from far away with magic toys are no basis for taking spooks any more seriously.
Replies from: Pavitra↑ comment by Pavitra · 2010-08-27T07:38:47.729Z · LW(p) · GW(p)
I find the alien argument very persuasive.
Suppose a server appeared on the internet relaying messages from someone claiming to be the sysadmin of the simulation we're living in, and asking that we refrain from certain types of behavior because it's making his job difficult. Is there any set of evidence that would persuade you to go along with the requests, and how would the necessary degree of evidence scale with the inconvenience of the requests?
Replies from: jimrandomh, Richard_Kennaway↑ comment by jimrandomh · 2010-08-29T21:49:54.880Z · LW(p) · GW(p)
That should be a very easy claim to prove, actually. If someone really were the sysadmin of the universe, they could easily do a wide variety of impossible things that anyone can could verify. For example, they could write their message in the sky with a special kind of photon that magically violates the laws of physics in an obvious way (say, for example, it interacts with all elements normally except one which it inexplicably doesn't interact with at all). Or find/replace their message into the genome of a designated species. Or graffiti it onto every large surface in the world simultaneously.
Of course, there would be no way to distinguish a proper sysadmin of the universe from someone who had gotten root access improperly, either from the simulated universe, the parent universe, or some other universe. And this does raise a problem for any direct evidence in support of a religion - no matter how strong the evidence gets, the possibility that someone has gained the ability to generate arbitrarily much fake evidence, or reliably deceive you somehow, will always remain indistinguishable; so anything with a significantly lower prior probability than that, is fundamentally impossible to prove. Most or all religions have a smaller prior probability than the "someone has gained magical evidence-forging powers and is using them" hypothesis, and as a result, even if strong evidence for them were to suddenly start appearing (which it hasn't), that still wouldn't be enough to prove them correct.
↑ comment by Richard_Kennaway · 2010-08-27T08:03:55.999Z · LW(p) · GW(p)
I still have a basic problem with the method of posing questions about possibilities I currently consider fantastically improbable. My uncertainty about how I would deal with the situation goes up with its improbability, and what I would actually do will be determined largely by details absent from the description of the improbable scenario.
It is as if my current view of the world -- that is, my assignments of probabilities to everything -- is a digital photograph of a certain resolution. When I focus on vastly improbable possibilities, it is as if I inspect a tiny area of the photograph, only a few pixels wide, and try to say what is depicted there. I can put that handful of pixels through my best image-processing algorithms, but all I'm going to get back is noise.
Replies from: Pavitra↑ comment by Pavitra · 2010-08-27T08:54:40.123Z · LW(p) · GW(p)
Can you consider hypothetical worlds with entirely different histories from ours? Rather than trying to update based on your current state of knowledge, with mountains of cumulative experience pointing a certain way, imagine what that mountainous evidence could have been in a deeply different world than this one.
For example, suppose the simulation sysadmin had been in active communication with us since before recorded history, and was commonplace knowledge casually accepted as mere fact, and the rest of the world looked different in the ways we would expect such a world to.
Replies from: Richard_Kennaway↑ comment by Richard_Kennaway · 2010-08-27T09:09:15.466Z · LW(p) · GW(p)
Can you consider hypothetical worlds with entirely different histories from ours?
In other words, can I read fiction? Yes, but I don't see where this is going.
Replies from: Pavitra↑ comment by Pavitra · 2010-08-27T10:29:18.680Z · LW(p) · GW(p)
Unwinding the thread backwards, I see that my comment strayed into irrelevance from the original point, so never mind.
I would like to ask you this, though: of all the people on Earth who feel as sure as you do about the truth or falsehood of various religions, what proportion do you think are actually right? If your confidence in your beliefs regarding religion is a larger number than this, then what additional evidence do you have that makes you think you're special?
↑ comment by [deleted] · 2010-08-24T05:10:16.625Z · LW(p) · GW(p)
You've asked us to take our very small number, and imagine it doubling 66 times. I agree that there is a punch to what you say -- no number, no matter how small, could remain small after being doubled 66 times! But in fact long ago Archimedes made a compelling case that there are such numbers.
Now, it's possible that Archimedes was wrong and something like ultrafinitism is true. I take ultrafinitist ideas quite seriously, and if they are correct then there are a lot things that we will have to rethink. But Islam is not close to the top of list of things we would should rethink first.
Maybe there's a kind of meta claim here: conditional on probability theory being a coherent way to discuss claims like "Islam is true," the probability that Islam is true really is that small.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-24T05:25:14.622Z · LW(p) · GW(p)
I just want to know what you would actually do, in that situation, if it happened to you tomorrow. How many 1's would you wait for, before you became a Muslim?
Also, "there are such numbers" is very far from "we should use such numbers as probabilities when talking about claims that many people think are true." The latter is an extremely strong claim and would therefore need extremely strong evidence before being acceptable.
Replies from: None↑ comment by [deleted] · 2010-08-24T07:46:49.195Z · LW(p) · GW(p)
I think after somewhere between 30 and 300 coin flips, I would convert. With more thought and more details about what package of claims is meant by "Islam," I could give you a better estimate. Escape routes that I'm not taking: I would start to suspect Omega was pulling my leg, I would start to suspect that I was insane, I would start to suspect that everything I knew was wrong, including the tenets of Islam. If answers like these are copouts -- if Omega is so reliable, and I am so sane, and so on -- then it doesn't seem like much of a bullet to bite to say "yes, 2^-30 is very small but it is still larger than 2^-66; yes something very unlikely has happened but not as unlikely as Islam"
Also, "there are such numbers" is very far from "we should use such numbers as probabilities when talking about claims that many people think are true." The latter is an extremely strong claim and would therefore need extremely strong evidence before being acceptable.
If you're expressing doubts about numbers being a good measure of beliefs, I'm totally with you! But we only need strong evidence for something to be acceptable if there are some alternatives -- sometimes you're stuck with a bad option. Somebody's handed us a mathematical formalism for talking about probabilities, and it works pretty well. But it has a funny aspect: we can take a handful of medium-sized probabilities, multiply them together, and the result is a tiny tiny probability. Can anything be as unlikely as the formalism says 66 heads in a row is? I'm not saying you should say "yes," but if your response is "well, whenever something that small comes up in practice, I'll just round up," that's a patch that is going to spring leaks.
Replies from: Unknowns, Unknowns↑ comment by Unknowns · 2010-08-24T09:26:54.080Z · LW(p) · GW(p)
Another point, regarding this:
yes, 2^-30 is very small but it is still larger than 2^-66; yes something very unlikely has happened but not as unlikely as Islam.
Originally I didn't intend to bring up Pascal's Wager type considerations here because I thought it would just confuse the issue of the probability. But I've rethought this-- actually this issue could help to show just how strong your beliefs are in reality.
Suppose you had said in advance that the probability of Islam was 10^-20. Then you had this experience, but the machine was shut off after 30 1's ( a chance of one in a billion.) The chance that Islam is true is now one in a hundred billion, updated from your prior.
If this actually happened to you, and you walked away and did not convert, would you have some fear of being condemned to hell for seeing this and not converting? Even a little bit of fear? If you would, then your probability that Islam is true must be much higher than 10^-20, since we're not afraid of things that have a one in a hundred billion chance of happening.
Replies from: thomblake↑ comment by thomblake · 2010-08-25T15:28:12.736Z · LW(p) · GW(p)
If this actually happened to you, and you walked away and did not convert, would you have some fear of being condemned to hell for seeing this and not converting? Even a little bit of fear? If you would, then your probability that Islam is true must be much higher than 10^-20, since we're not afraid of things that have a one in a hundred billion chance of happening.
This is false.
I must confess that I am sometimes afraid that ghosts will jump out of the shadows and attack me at night, and I would assign a much lower chance of that happening. I have also been afraid of velociraptors. Fear is frequently irrational.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-26T06:40:55.223Z · LW(p) · GW(p)
You are technically correct. My actual point was that your brain does not accept that the probability is that low. And as I stated in one of the replies, you might in some cases have reasons to say your brain is wrong... just not in this case. No one here has given any reason to think that.
↑ comment by Unknowns · 2010-08-24T09:14:42.413Z · LW(p) · GW(p)
It's good you managed some sort of answer to this. However, 30 - 300 is quite a wide range; from 1 in 10^9 to 1 in 10^90. If you're going to hope for any sort of calibration at all in using numbers like this, you're going to have to much more precise...
I wasn't expressing doubts about numbers being a measure of beliefs (although you could certainly question this as well), but about extreme numbers being a measure of our beliefs, which do not seem able to be that extreme. Yes, if you have a large number of independent probabilities, the result can be extreme. And supposedly, the basis for saying that Islam (or reincarnation, or whatever) is very improbable would be the complexity of the claim. But who has really determined how much complexity it has? As I pointed out elsewhere (on the "Believable Bible" comment thread), a few statements, if we knew them to be true, would justify Islam or any other such thing. Which particular statements would we need, and how complex are those statements, really? No one has determined them to any degree of precision, and until they do, you have to use something like a base rate. Just as astronomers start out with fairly high probabilities for the collision of near-earth asteroids, and only end up with low probabilities after very careful calculation, you would have to start out with a fairly high prior for Islam, or reincarnation, or whatever, and you would only be justified in holding an extreme probability after careful calculation... which I don't believe you've done. Certainly I haven't.
Apart from the complexity, there is also the issue of evidence. We've been assuming all along that there is no evidence for Islam, or reincarnation, or whatever. Certainly it's true that there isn't much. But that there is literally no evidence for such things simply isn't so. The main thing is that we aren't motivated to look at the little evidence that there is. But if you intend to assign probabilities to that degree of precision, you are going to have to take into account every speck of evidence.
Replies from: None↑ comment by [deleted] · 2010-08-24T16:12:37.865Z · LW(p) · GW(p)
I thought the salient feature of Islam was that many people believed it, not that it has less complexity than I thought, or more evidence in its favor than I thought. That might be, but I'm not interested in discussing it.
I don't "feel" beliefs strongly or weakly. Sometimes probability calculations help me with fear and other emotions, sometimes they don't. Again, I'm not interested in discussing it.
So tell me something about how important it is that many people believe in Islam.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-24T17:17:32.589Z · LW(p) · GW(p)
I'm not interested in discussing Islam either... those points apply to anything that people believe. But that's why it's relevant to the question of belief: if you take something that people don't believe, it can be arbitrarily complex, or 100% lacking in evidence (like Russell's teapot), but things that people believe do not have these properties.
It's not important how many people believe it. It could be just 50 people and the probability would not be much different (as long as the belief was logically consistent with the fact that just a few people believed it.)
Replies from: None↑ comment by [deleted] · 2010-08-24T17:32:45.372Z · LW(p) · GW(p)
So tell me why. By "complex" do you just mean "low probability," or some notion from information theory? How did you come to believe that people cannot believe things that are too complex?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-24T18:22:33.500Z · LW(p) · GW(p)
I just realized that you may have misunderstood my original point completely. Otherwise you wouldn't have said this: "I thought the salient feature of Islam was that many people believed it, not that it has less complexity than I thought, or more evidence in its favor than I thought."
I only used the idea of complexity because that was komponisto's criterion for the low probability of such claims. The basic idea is people believe things that their priors say do not have too low a probability: but as I showed in the post on Occam's razor, everyone's prior is a kind of simplicity prior, even if they are not all identical (nor necessarily particularly related to information theory or whatever.)
Basically, a probability is determined by the prior and by the evidence that it is updated according to. The only reason things are more probable if people believe them is that a person's belief indicates that there is some human prior according to which the thing is not too improbable, and some evidence and way of updating that can give the thing a reasonable probability. So other people's beliefs are evidence for us only because they stand in for the other people's priors and evidence. So it's not that it is "important that many people believe" apart from the factors that give it probability: the belief is just a sign that those factors are there.
Going back the distinction you didn't like, between a fixed probability device and a real world claim, a fixed probability device would be a situation where the prior and the evidence is completely fixed and known: with the example I used before, let there be a lottery that has a known probability of one in a trillion. Then since the prior and the evidence are already known, the probability is still one in a trillion, even if someone says he is definitely going to win it.
In a real world claim, on the other hand, the priors are not well known, and the evidence is not well known. And if I find out that someone believes it, I immediately know that there are humanly possible priors and evidence that can lead to that belief, which makes it much more probable even for me than it would be otherwise.
Replies from: None↑ comment by [deleted] · 2010-08-24T19:44:01.065Z · LW(p) · GW(p)
If I find out that ... I know that ... which makes it much more probable that ...
This sounds like you are updating. We have a formula for what happens when you update, and it indeed says that given evidence, something becomes more probable. You are saying that it becomes much more probable. What quantity in Bayes formula seems especially large to you, and why?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-25T07:18:59.643Z · LW(p) · GW(p)
What Wei Dai said.
In other words, as I said before, the probability that people believe something shouldn't be that much more than the probability that the thing is true.
Replies from: Richard_Kennaway, None↑ comment by Richard_Kennaway · 2010-08-25T12:45:44.597Z · LW(p) · GW(p)
What about the conjunction fallacy?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-26T06:38:32.072Z · LW(p) · GW(p)
The probability that people will believe a long conjunction is less probable than they will believe one part of the conjunction (because in order to believe both parts, they have to believe each part. In other words, for the same reason the conjunction fallacy is a fallacy.)
Replies from: Richard_Kennaway↑ comment by Richard_Kennaway · 2010-08-26T07:49:32.386Z · LW(p) · GW(p)
The conjunction fallacy is the assignment of a higher probability to some statement of the form A&B than to the statement A. It is well established that for certain kinds of A and B, this happens.
The fallacy in your proof that this cannot happen is that you have misstated what the conjunction fallacy is.
My point in mentioning it is that people committing the fallacy believe a logical impossibility. You can't get much more improbable than a logical impossibility. But the conjunction fallacy experiments demonstrate that is common to believe such things.
Therefore, the improbability of a statement does not imply the improbability of someone believing it. This refutes your contention that "the probability that people believe something shouldn't be that much more than the probability that the thing is true." The possible difference between the two is demonstrably larger than the range of improbabilities that people can intuitively grasp.
Replies from: None↑ comment by [deleted] · 2010-08-25T16:30:57.791Z · LW(p) · GW(p)
You said it before, but you didn't defend it.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-26T06:43:11.201Z · LW(p) · GW(p)
Wei Dai did, and I defended it by referencing his position.
Replies from: None↑ comment by [deleted] · 2010-08-26T15:40:56.753Z · LW(p) · GW(p)
In that case I am misunderstanding Wei Dai's point. He says that complexity considerations alone can't tell you that probability is small, because complexity appears in the numerator and the denominator. I will need to see more math (which I guess cousin it is taking care of) before understanding and agreeing with this point. But even granting it I don't see how it implies that P(many believe H)/P(H) is for all H less than one billion.
↑ comment by cousin_it · 2010-08-24T23:29:54.166Z · LW(p) · GW(p)
Thank you a lot for posting this scenario.
Imagine there are a trillion variants of Islam, differing by one sentence in the holy book or something. At most one of them can be true. You pick one variant at random, test it with your machine and get 30 1's in a row. Now you should be damn convinced that you picked the true one, right? Wrong. Getting this result by a fluke is ~1000x more likely than picking the true variant in the first place. Probability is unintuitive and our brains are mush, that's all I'm sayin'.
↑ comment by [deleted] · 2010-08-21T07:54:24.177Z · LW(p) · GW(p)
The product of two probabilities above your threshold-for-overconfidence can be below your threshold-for-overconfidence. Have you at least thought this through before?
For instance, the claim "there is a God" is not that much less spectacular than the claim "there is a God, and he's going to make the next 1000 times you flip a coin turn up heads." If one-in-a-billion is a lower bound for the probability that God exists, then one-in-a-billion-squared is a generous lower bound for the probability that the next 1000 times you flip a coin will turn up heads. (One-in-a-billion-squared is about 2-to-the-sixty). You're OK with that?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-21T15:12:20.919Z · LW(p) · GW(p)
Yes. As long as you think of some not-too-complicated scenario where the one would lead to the other, that's perfectly reasonable. For example, God might exist and decide to prove it to you by effecting that prediction. I certainly agree this has a probability of at least one in a billion squared. In fact, suppose you actually get heads the next 60 times you flip a coin, even though you are choosing different coins, it is on different days, and so on. By that point you will be quite convinced that the heads are not independent, and that there is quite a good chance that you will get 1000 heads in a row.
It would be different of course if you picked a random series of heads and tails: in that case you still might say that there is at least that probability that someone else will do it (because God might make that happen), but you surely cannot say that it had that probability before you picked the random series.
This is related to what I said in the torture discussion, namely that explicitly describing a scenario automatically makes it far more probable to actually happen than it was before you described it. So it isn't a problem if the probability of 1000 heads in a row is more likely than 1 in 2-to-1000. Any series you can mention would be more likely than that, once you have mentioned it.
Also, note that there isn't a problem if the 1000 heads in a row is lower than one in a billion, because when I made the general claim, I said "a claim that significant number of people accept as likely true," and no one expects to get the 1000 heads.
Replies from: None↑ comment by [deleted] · 2010-08-21T17:13:54.467Z · LW(p) · GW(p)
Probabilities should sum to 1. You're saying moreover that probabilities should not be lower that some threshhold. Can I can get you to admit that there's a math issue here that you can't wave away, without trying to fine-tune my examples? If you claim you can solve this math issue, great, but say so.
Edit: -1 because I'm being rude? Sorry if so, the tone does seem inappropriately punchy to me now. -1 because I'm being stupid? Tell me how!
Replies from: Unknowns↑ comment by Unknowns · 2010-08-21T18:57:29.164Z · LW(p) · GW(p)
I set a lower bound of one in a billion on the probability of "a natural language claim that a significant number of people accept as likely true". The number of such mutually exclusive claims is surely far less than a billion, so the math issue will resolve easily.
Yes, it is easy to find more than a billion claims, even ones that some people consider true, but they are not mutually exclusive claims. Likewise, it is easy to find more than a billion mutually exclusive claims, but they are not ones that people believe to be true, e.g. no one expects 1000 heads in a row, no one expects a sequence of five hundred successive heads-tails pairs, and so on.
I didn't downvote you.
Replies from: None↑ comment by [deleted] · 2010-08-21T21:03:22.097Z · LW(p) · GW(p)
Maybe I see. You are updating on the fact that many people believe something, and are saying that P(A|many people believe A) should not be too small. Do you agree with that characterization of your argument?
In that case, we will profitably distinguish between P(A|no information about how many people believe A) and P(A|many people believe A). Is there a compact way that I can communicate something like "Excepting/not updating on other people's beliefs, P(God exists) is very small"? If I said something like that would you still think I was being overconfident?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-22T06:17:18.606Z · LW(p) · GW(p)
This is basically right, although in fact it is not very profitable to speak of what the probability would be if we didn't have some of the information that we actually have. For example, the probability of this sequence of ones and zeros -- 0101011011101110 0010110111101010 0100010001010110 1010110111001100 1110010101010000 -- being chosen randomly, before anyone has mentioned this particular sequence, is one out 2 to the 80. Yet I chose it randomly, using a random number generator (not a pseudo random number generator, either.) But I doubt that you will conclude that I am certainly lying, or that you are hallucinating. Rather, as Robin Hanson points out, extraordinary claims are extraordinary evidence. The very fact that I write down this improbable evidence is extremely extraordinary evidence that I have chosen it randomly, despite the huge improbability of that random choice. In a similar way, religious claims are extremely strong evidence in favor of what they claim; naturally, just as if I hadn't written the number, you would never believe that I might choose it randomly, in the same way, if people didn't make religious claims, you would rightly think them to be extremely improbable.
Replies from: None↑ comment by [deleted] · 2010-08-22T20:54:53.887Z · LW(p) · GW(p)
It is always profitable to give different concepts different names.
Let GM be the assertion that I'll one day play guitar on the moon. Your claim is that this ratio
P(GM|I raised GM as a possibility)/P(GM)
is enormous. Bayes theorem says that this is the same as
P(I raised GM as a possibility|GM)/P(I raised GM as a possibility)
so that this second ratio is also enormous. But it seems to me that both numerator and denominator in this second ratio are pretty medium-scale numbers--in particular the denominator is not miniscule. Doesn't this defeat your idea?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-23T00:56:37.061Z · LW(p) · GW(p)
The evidence contained in your asserting GM would be much stronger than the evidence contained in your raising the possibility.
Still, there is a good deal of evidence contained in your raising the possibility. Consider the second ratio: the numerator is quite high, probably more than .5, since in order to play guitar on the moon, you would have to bring a guitar there, which means you'd probably be thinking about it.
The denominator is in fact quite small. If you randomly raise one outlandish possibility of performing some action in some place, each day for 50 years, and there are 10,000 different actions (I would say there are at least that many), and 100,000 different places, then the probability of raising the possibility will be 18,250/(10,000 x 100,000), which is 0.00001825, which is fairly small. The actual probability is likely to be even lower, since you may not be bringing up such possibilities every day for 50 years. Religious claims are typically even more complicated than the guitar claim, so the probability of raising their possibility is even lower.
--one more thing: I say that raising the possibility is strong evidence, not that the resulting probability is high: it may start out extremely low and end up still very, very low, going from say one in a google to one in a sextillion or so. It is when you actually assert that it's true that you raise the probability to something like one in a billion or even one in a million. Note however that you can't refute me by now going on to assert that you intend to play a guitar on the moon; if you read Hanson's article in my previous link, you'll see that he shows that assertions are weak evidence in particular cases, namely in ones in which people are especially likely to lie: and this would be one of them, since we're arguing about it. So in this particular case, if you asserted that you intended to do so, it would only raise the probability by a very small amount.
Replies from: None↑ comment by [deleted] · 2010-08-23T03:10:38.155Z · LW(p) · GW(p)
I understand that you think the lower bound on probabilities for things-that-are-believed is higher than the lower bound on probabilities for things-that-are-raised-as-possibilities. I am fairly confident that I can change your mind (that is, convince you not to impose lower bounds like this at all), and even more confident that I can convince you that imposing lower bounds like this is mathematically problematic (that is, there are bullets to be bitten) in ways that hadn't occurred to you a few days ago.
I do not see one of these bounds as more or less sound than the other, but am focusing on the things-that-are-raised-as-possibilities bound because I think the discussion will go faster there.
More soon, but tell me if you think I've misunderstood you, or if you think you can anticipate my arguments. I would also be grateful to hear from whoever is downvoting these comments.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-23T03:24:57.066Z · LW(p) · GW(p)
Note that I said there should be a lower bound on the probability for things that people believe, and even made it specific: something on the order of one in a billion. But I don't recall saying (you can point it out if I'm wrong) that there is a lower bound on the probability of things that are raised as possibilities. Rather, I merely said that the probability is vastly increased.
To the comment here, I responded that raising the possibility raised the probability of the thing happening by orders of magnitude. But I didn't say that the resulting probability was high, in fact it remains very low. Since there is no lower bound on probabilities in general, there is still no lower bound on probabilities after raising them by orders of magnitude, which is what happens when you raise the possibility.
So if you take my position to imply such a lower bound, either I've misstated my position accidentally, or you have misunderstood it.
Replies from: None↑ comment by [deleted] · 2010-08-23T05:02:58.461Z · LW(p) · GW(p)
I did misunderstand you, and it might change things; I will have to think. But now your positions seem less coherent to me, and I no longer have a model of how you came to believe them. Tell me more:
Let CM(n) be the assertion "one day I'll play guitar on the moon, and then flip an n-sided coin and it will come up heads." The point being that P(CM(n)) is proportional to 1/n. Consider the following ratios:
- R1(n) = P(CM(n)|CM(n) is raised as a possibility)/P(CM(n))
- R2(n) = P(CM(n)|CM(n) is raised as a possibility by a significant number of people)/P(CM(n))
- R3(n) = P(CM(n)|CM(n) is believed by one person)/P(CM(n))
- R4(n) = P(CM(n)|CM(n) is believed by a significant number of people)/P(CM(n))
How do you think these ratios change as n grows? Before I had assumed you thought that ratios 1. and 4. grew to infinity as n did. I still understand you to be saying that for 4. Are you now denying it for 1., or just saying that 1. grows more slowly? I can't guess what you believe about 2. and 3.
Replies from: Unknowns↑ comment by Unknowns · 2010-08-23T06:39:01.428Z · LW(p) · GW(p)
First we need to decide on the meaning of "flip an n-sided coin and it will come up heads". You might mean this as:
1) a real world claim; or 2) a fixed probability device
To illustrate: if I assert, "I happen to know that I will win the lottery tomorrow," this greatly increases the chance that it will happen, among other reasons, because of the possibility that I am saying this because I happen to have cheated and fixed things so that I will win. This would be an example of a real world claim.
On the other hand, if it is given that I will play the lottery, and given that the chance of winning is one in a trillion, as a fixed fact, then if I say, "I will win," the probability is precisely one in a trillion, by definition. This is a fixed probability device.
In the real world there are no fixed probability devices, but there are situations where things are close enough to that situation that I can mathematically calculate a probability, even one which will break the bound of one in a billion, and even when people believe it. This is why I qualified my original claim with "Unless you have actually calculated the probability..." So in order to discuss my claim at all, we need to exclude the fixed probability device and only consider real world claims. In this case, the probability of P(CM(n)) is not exactly proportional to 1/n. However, it is true that this probability goes to zero as n goes to infinity.
In fact, all of these probabilities go to zero as n goes to infinity:
- P(CM(n))
- P(CM(n) is raised as a possibility)
- P(CM(n) is believed, by one or many persons)
The reason these probabilities go to zero can be found in my post on Occam's razor.
Given this fact (that all the probabilities go to zero), I am unsure about the behavior of your cases 1 & 2. I'll leave 3 for another time, and say that case 4, again remembering that we take it as a real world claim, does go to infinity, since the numerator remains at no less than 1 in a billion, while the denominator goes to zero.
One more note about my original claim: if you ask how I arrived at the one in a billion figure, it is somewhat related to the earth's actual population. If the population were a googleplex, a far larger number of mutually exclusive claims would be believed by a significant number of people, and so the lower bound would be much lower. Finally, I don't understand why you say my positions are "less coherent", when I denied the position that as you were about to point out, leads to mathematical inconsistency. This should make my position more coherent, not less.
Replies from: None↑ comment by [deleted] · 2010-08-23T07:12:48.904Z · LW(p) · GW(p)
It's my map of your beliefs that became less coherent, not your actual beliefs. (Not necessarily!) As you know, I've thought your beliefs are mistaken from the beginning.
Note that I'm asking about a limit of ratios, not a ratio of limits. Actually, I'm not even asking you about the limits--I'd prefer some rough information about how those ratios change as n grows. (Are they bounded above? Do they grow linearly or logarithmically or what?) If you don't know, why not?
So in order to discuss my claim at all, we need to exclude the fixed probability device and only consider real world claims.
This is bad form. Phrases like "unless you have actually computed the probability...", "real world claim", "natural language claim", "significant number of people" are slippery. We can talk about real-world examples after you explain to me how your reasoning works in a more abstract setting. Otherwise you're just reserving the right to dismiss arguments (and even numbers!) on the basis that they feel wrong to you on a gut level.
Edit: It's not that I think it's always illegitimate to refer to your gut. It's just bad form to claim that such references are based on mathematics.
Edit 2: Can I sidestep this discussion by saying "Let CM(n) be any real world claim with P(CM(n)) = 1/n"?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-23T09:16:47.471Z · LW(p) · GW(p)
My original claim was
Unless you've actually calculated the probability mathematically, a probability of one in a billion for a natural language claim that a significant number of people accept as likely true is always overconfident.
I nowhere stated that this was "based on mathematics." It is naturally related to mathematics, and mathematics puts some constraints on it, as I have been trying to explain. But I didn't come up with it in the first place in a purely mathematical way. So if this is bad form, it must be bad form to say what I mean instead of something else.
I could accept what you say in Edit 2 with these qualifications: first, since we are talking about "real world claims", the probability 1/n does not necessarily remain fixed when someone brings up the possibility or asserts that the thing is so. This probability 1/n is only a prior, before the possibility has been raised or the thing asserted. Second, since it isn't clear what "n" is doing, CM(5), CM(6), CM(7) and so on might be claims which are very different from one another.
I am not sure about the behavior of the ratios 1 and 2, especially given the second qualification here (in other words the ratios might not be well-behaved at all). And I don't see how I need to say "why not?" What is there in my account which should tell me how these ratios behave? But my best guess for the moment, after thinking about it some more, would be the first ratio probably goes to infinity, but not as quickly as the fourth. What leads me to think this is something along the lines of this comment thread. For example, in my Scientology example, even if no one held that Scientology was true, but everyone admitted that it was just a story, the discovery of a real Xenu would greatly increase the probability that it was true anyway; although naturally not as much as given people's belief in it, since without the belief, there would be a significantly greater probability that Scientology is still a mere story, but partly based on fact. So this suggests there may be a similar bound on things-which-have-been-raised-as-possibilities, even if much lower than the bound for things which are believed. Or if there isn't a lower bound, such things are still likely to decrease in probability slowly enough to cause ratio 1 to go to infinity.
Replies from: None↑ comment by [deleted] · 2010-08-23T15:31:13.263Z · LW(p) · GW(p)
This is bad form.
Ugly and condescending of me, beg your pardon.
What is there in my account which should tell me how these ratios behave?
You responded positively to my suggestion that we could phrase this notion of "overconfidence" as "failure to update on other people's beliefs," indicating that you know how to update on other people's beliefs. At the very least, this requires some rough quantitative understanding of the players in Bayes formula, which you don't seem to have.
If overconfidence is not "failure to update on other people's beliefs," then what is it?
Here's the abbreviated version of the conversation that led us here (right?).
S: God exists with very low probability, less that one in a zillion.
U: No, you are being overconfident. After all, billions of people believe in God, you need to take that into account somehow. Surely the probability is greater than one in a billion.
S: OK I agree that the fact that billions of people believing it constitutes evidence, but surely not evidence so strong as to get from 1-in-a-zillion to 1-in-a-billion.
Now what? Bayes theorem provides a mathematical formalism for relating evidence to probabilities, but you are saying that all four quantities in the relevant Bayes formula are too poorly understood for it to be of use. So what's an alternative way to arrive at your one-in-a-billion figure? Or are you willing to withdraw your accusation that I'm being overconfident?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-23T18:31:39.399Z · LW(p) · GW(p)
I did not say that "all four quantities in the relevant Bayes formula are too poorly understood for it to be of use." Note that I explicitly asserted that your fourth ratio tends to infinity, and that your first one likely does as well.
If you read the linked comment thread and the Scientology example, that should make it clear why I think that the evidence might well be strong enough to go from 1 in a zillion to 1 in a billion. In fact, that should even be clear from my example of the random 80 digit binary number. Suppose instead of telling you that I chose the number randomly, I said, "I may or may not have chosen this number randomly." This would be merely raising the possibility-- the possibility of something which has a prior of 2^-80. But if I then went on to say that I had indeed chosen it randomly, you would not have therefore called me a liar, while you would do this, if I now chose another random 80 digit number and said that it was the same one. This shows that even raising the possibility provides almost all the evidence necessary-- it brings the probability that I chose the number randomly all the way from 2^-80 up to some ordinary probability, or from "1 in a zillion" to something significantly above one in a billion.
More is involved in the case of belief, but I need to be sure that you get this point first.
Replies from: None, None↑ comment by [deleted] · 2010-08-23T19:47:15.714Z · LW(p) · GW(p)
Let's consider two situations:
- For each 80-digit binary number X, let N(X) be the assertion "Unknowns picked an 80-digit number at random, and it was X." In my ledger of probabilities, I dutifully fill in, for each of these statements X, 2^{-80} in the P column. Now for a particular 80-digit number Y, I am told that "Unknowns claims he picked an 80-digit number at random, and it was Y" -- call that statement U(Y) -- and am asked for P(N(Y)|U(Y)).
My answer: pretty high by Bayes formula. P(U|N(Y)) is pretty high because Unknowns is trustworthy, and my ledger has P(U(Y)) = number on the same order as two-to-the-minus-eighty. (Caveat: P(U(Y)) is a lot higher for highly structured things like the sequence of all 1's. But for the vast majority of Y I have P(U(Y)) = 2^-80 times something between (say) 10^-1 and 10^-6). So P(N(Y)|U(Y)) = P(U(Y)|N(Y)) x [P(N(Y))/P(U(Y))] is a big probability times a medium-sized probability
What's your answer?
- Reincarnation is explained to me, and I am asked for my opinion of how likely it is. I respond with P(R), a good faith estimate based on my experience and judgement. I am then told that hundreds of millions of people believe in reincarnation -- call that statement B, and assume that I was ignorant of it before -- and am asked for P(R|B). Your claim is that no matter how small P(R) is, P(R|B) should be larger than some threshold t. Correct?
Some manipulation with Bayes formula shows that your claim (what I understand to be your claim) is equivalent to this inequality:
P(B) < P(R) / t
That is, I am "overconfident" if I think that the probability of someone believing in reincarnation is larger than some fixed multiple of the probability that reincarnation is actually true. Moreover, though I assume (sic) you think t is sensitive to the quantity "hundreds of millions" -- e.g. that it would be smaller if it were just "hundreds" -- you do not think that t is sensitive to the statement R. R could be replaced by another religious claim, or by the claim that I just flipped a coin 80 times and the sequence of heads and tails was [whatever].
My position: I think it's perfectly reasonable to assume that P(B) is quite a lot larger than P(R). What's your position?
Replies from: Unknowns↑ comment by Unknowns · 2010-08-24T05:34:35.446Z · LW(p) · GW(p)
Your analysis is basically correct, i.e. I think it is overconfident to say that the probability P(B) is greater than P(R) by more than a certain factor, in particular because if you make it much greater, there is basically no way for you to be well calibrated in your opinions-- because you are just as human as the people who believe those things. More on that later.
For now, I would like to see your response to question on my comment to komponisto (i.e. how many 1's do you wait for.)
↑ comment by multifoliaterose · 2010-08-21T05:33:43.025Z · LW(p) · GW(p)
My estimate does come some effort at calibration, although there's certainly more that I could do. Maybe I should have qualified my statement by saying "this estimate may be a gross overestimate or a gross underestimate."
In any case, I was not being disingenuous or flippant. I have carefully considered the question of how likely it is that Eliezer will be able to play a crucial role in a FAI project if he continues to exhibit a strategy qualitatively similar to his current one and my main objection to SIAI's strategy is that I think it extremely unlikely that Eliezer will be able to have an impact if he proceeds as he has up until this point.
I will be detailing why I don't think that Eliezer's present strategy toward working toward an FAI is a fruitful one in a later top level post.
Replies from: steven0461↑ comment by steven0461 · 2010-08-21T05:47:06.626Z · LW(p) · GW(p)
Maybe I should have qualified my statement by saying "this estimate may be a gross overestimate or a gross underestimate."
It sounds, then, like you're averaging probabilities geometrically rather than arithmetically. This is bad!
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-21T05:51:37.280Z · LW(p) · GW(p)
I understand your position and believe that it's fundamentally unsound. I will have more to say about this later.
For now I'll just say that the arithmetical average of the probabilities that I imagine I might ascribe to Eliezer's current strategy resulting in an FAI to be 10^(-9).
↑ comment by ata · 2010-08-20T19:11:45.596Z · LW(p) · GW(p)
I wish the laws of argument permitted me to declare that you had blown yourself up at this point, and that I could take my toys and go home. Alas, arguments are not won on a points system.
On the other hand, assuming he knows what it means to assign something a 10^-9 probability, it sounds like he's offering you a bet at 1000000000:1 odds in your favour. It's a good deal, you should take it.
Replies from: rabidchicken↑ comment by rabidchicken · 2010-08-21T16:42:27.668Z · LW(p) · GW(p)
Indeed. I do not know how many people are actively involved in FAI research, but i would guess that it is only in the the dozens to hundreds. Given the small pool of competition, it seems likely that at some point Eliezer will, or already has, made a unique contribution to the field. Get Multi to put some money on it, offer him 1 cent if you do not make a useful contribution in the next 50 years, and if you do, he can pay you 10 million dollars.
↑ comment by multifoliaterose · 2010-08-20T19:09:42.217Z · LW(p) · GW(p)
I wish the laws of argument permitted me to declare that you had blown yourself up at this point, and that I could take my toys and go home. Alas, arguments are not won on a points system.
I don't understand this remark.
What probability do you assign to your succeeding in playing a critical role on the Friendly AI project that you're working on? I can engage with a specific number. I don't know if your object is that my estimate is off by a single of order of magnitude or by many orders of magnitude.
Out of weary curiosity, what is it that you think you know about Friendly AI that I don't?
I should clarify that my comment applies equally to AGI.
I think that I know the scientific community better than you, and have confidence that if creating an AGI was as easy as you seem to think it is (how easy I don't know because you didn't give a number) then there would be people in the scientific community who would be working on AGI.
And has it occurred to you that if I have different non-crazy beliefs about Friendly AI then my final conclusions might not be so crazy either, no matter what patterns they match in your craziness recognition systems?
Yes, this possibility has certainly occurred to me. I just don't know what your different non-crazy beliefs might be.
Why do you think that AGI research is so uncommon within academia if it's so easy to create an AGI?
Replies from: khafra, Emile↑ comment by khafra · 2010-08-20T19:44:57.174Z · LW(p) · GW(p)
This question sounds disingenuous to me. There is a large gap between "10^-9 chance of Eliezer accomplishing it" and "so easy for the average machine learning PhD." Whatever else you think about him, he's proved himself to be at least one or two standard deviations above the average PhD in ability to get things done, and some dimension of rationality/intelligence/smartness.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T19:56:56.837Z · LW(p) · GW(p)
My remark was genuine. Two points:
I think that the chance that any group of the size of SIAI will develop AGI over the next 50 years is quite small.
Eliezer has not proved himself to be at the same level of the average machine learning PhD at getting things done. As far as I know he has no experience with narrow AI research. I see familiarity with narrow AI as a prerequisite to AGI research.
↑ comment by XiXiDu · 2010-08-20T20:16:25.238Z · LW(p) · GW(p)
Eliezer has not proved himself to be at the same level of the average machine learning PhD at getting things done.
He actually stated that himself several times.
So I do understand that, and I did set out to develop such a theory, but my writing speed on big papers is so slow that I can't publish it. Believe it or not, it's true.
Yes, ok, this does not mean his intellectual power isn't on par, but his ability to function in an academic environment.
As far as I know he has no experience with narrow AI research.
Well...
I tried - once - going to an interesting-sounding mainstream AI conference that happened to be in my area. [...] And I gave up and left before the conference was over, because I kept thinking "What am I even doing here?"
↑ comment by Vladimir_Nesov · 2010-08-20T20:51:12.274Z · LW(p) · GW(p)
As far as I know he has no experience with narrow AI research. I see familiarity with narrow AI as a prerequisite to AGI research.
Most things can be studied through the use of textbooks. Some familiarity with AI is certainly helpful, but it seems that most AI-related knowledge is not on the track to FAI (and most current AGI stuff is nonsense or even madness).
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T22:04:58.634Z · LW(p) · GW(p)
The reason that I see familiarity with narrow AI as a prerequisite to AGI research is to get a sense of the difficulties present in designing machines to complete certain mundane tasks. My thinking is the same as that of Scott Aaronson in his The Singularity Is Far posting: "there are vastly easier prerequisite questions that we already don’t know how to answer."
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T22:08:43.213Z · LW(p) · GW(p)
FAI research is not AGI research, at least not at present, when we still don't know what it is exactly that our AGI will need to work towards, how to formally define human preference.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T22:13:18.296Z · LW(p) · GW(p)
So, my impression is that you and Eliezer have different views of this matter. My impression is that Eliezer's goal is for SIAI to actually build an AGI unilaterally. That's where my low probability was coming from.
It seems much more feasible to develop a definition of friendliness and then get governments to mandate that it be implemented in any AI or something like that.
As I've said, I find your position sophisticated and respect it. I have to think more about your present point - reflecting on it may indeed alter my thinking about this matter.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T22:27:03.258Z · LW(p) · GW(p)
So, my impression is that you and Eliezer have different views of this matter. My impression is that Eliezer's goal is for SIAI to actually build an AGI unilaterally.
Still, build AGI eventually, and not now. Expertise in AI/AGI is of low relevance at present.
It seems much more feasible to develop a definition of friendliness and then get governments to mandate that it be implemented in any AI or something like that.
It seems obviously infeasible to me that governments will chance upon this level of rationality. Also, we are clearly not on the same page if you say things like "implement in any AI". Friendliness is not to be "installed in AIs", Friendliness is the AI (modulo initial optimizations necessary to get the algorithm going and self-optimizing, however fast or slow that's possible). The AGI part of FAI is exclusively about optimizing the definition of Friendliness (as an algorithm), not about building individual AIs with standardized goals.
See also this post for a longer explanation of why weak-minded AIs are not fit to carry the definition of Friendliness. In short, such AIs are (in principle) as much an existential danger as human AI researchers.
Replies from: Wei_Dai, multifoliaterose↑ comment by Wei Dai (Wei_Dai) · 2010-09-11T20:13:32.604Z · LW(p) · GW(p)
It seems obviously infeasible to me that governments will chance upon this level of rationality.
I wonder if we systematically underestimate the level of rationality of major governments. Historically, they haven't done that badly. From an article about RAND:
Futurology was the magic word in the years after the Second World War, and because the Army and later the Air Force didn’t want to lose the civilian scientists to the private sector, Project Reasearch and Development, RAND in short, was founded in 1945 together with the aircraft manufacturer Douglas and in 1948 was converted into a Corporation. RAND established forecasts for the coming, cold future and developed, towards this end, the ‘delphi’ method.
Rand worshipped rationality as a god and attempted to quantify the unpredictable, to calculate it mathematically, to bring the fear within its grasp and under control - something that seemed to many Americans spooky and made the soviet Prawda call RAND the “American academy of death and destruction.”
(Huh, this is the first time I've heard of the Delphi Method.) Many of the big names in game theory (von Neumann, Nash, Shapley, Schelling) worked for RAND at some point, and developed their ideas there.
Replies from: gwern, mattnewport, rhollerith_dot_com↑ comment by gwern · 2010-09-11T22:37:03.560Z · LW(p) · GW(p)
RAND has a lot of good work (I like their recent reports on Iran), but keep in mind that big misses can undo a lot of their credit; for example, even RAND acknowledges (in their retrospective published this year or last) that they screwed up massively with Vietnam.
↑ comment by mattnewport · 2010-09-11T20:47:50.773Z · LW(p) · GW(p)
I wonder if we systematically underestimate the level of rationality of major governments. Historically, they haven't done that badly. From an article about RAND:
This is not really a relevant example in the context of Vladimir_Nesov's comment. Certain government funded groups (often within the military interestingly) have on occasion shown decent levels of rationality.
The suggestion to "develop a definition of friendliness and then get governments to mandate that it be implemented in any AI or something like that." that he was replying to requires rational government policy making / law making rather than rare pockets of rationality within government funded institutions however. That is something that is essentially non-existent in modern democracies.
Replies from: Vladimir_Nesov, Wei_Dai↑ comment by Vladimir_Nesov · 2010-09-11T21:07:25.255Z · LW(p) · GW(p)
It's not adequate to "get governments to mandate that [Friendliness] be implemented in any AI", because Friendliness is not a robot-building standard - refer the rest of my comment. The statement about government rationality was more tangential, about governments doing anything at all concerning such a strange topic, and wasn't meant to imply that this particular decision would be rational.
↑ comment by Wei Dai (Wei_Dai) · 2010-09-11T21:07:32.716Z · LW(p) · GW(p)
"Something like that" could be for a government funded group to implement an FAI, which, judging from my example, seems within the realm of feasibility (conditioning on FAI being feasible at all).
↑ comment by RHollerith (rhollerith_dot_com) · 2010-09-11T20:27:38.635Z · LW(p) · GW(p)
I wonder if we systematically underestimate the level of rationality of major governments.
Data point: the internet is almost completely a creation of government. Some say entrepreneurs and corporations played a large role, but except for corporations that specialize in doing contracts for the government, they did not begin to exert a significant effect till 1993 whereas government spending on research that led to the internet began in 1960, and the direct predacessor to internet (the ARPAnet) became operational in 1969.
Both RAND and the internet were created by the part of the government most involved in an enterprise (namely, the arms race during the Cold War) on which depended the long-term survival of the nation in the eyes of most decision makers (including voters and juries).
EDIT: significant backpedalling in response to downvotes in my second paragraph.
↑ comment by multifoliaterose · 2010-08-20T22:43:08.724Z · LW(p) · GW(p)
Still, build AGI eventually, and not now. Expertise in AI/AGI is of low relevance at present.
Yes, this is the point that I had not considered and which is worthy of further consideration.
It seems obviously infeasible to me that governments will chance upon this level of rationality.
Possibly what I mention could be accomplished with lobbying.
Also, we are clearly not on the same page if you say things like "implement in any AI". Friendliness is not to be "installed in AIs", Friendliness is the AI (modulo initial optimizations necessary to get the algorithm going and self-optimizing, however fast or slow that's possible). The AGI part of FAI is exclusively about optimizing the definition of Friendliness (as an algorithm), not about building individual AIs with standardized goals.
See also this post for a longer explanation of why weak-minded AIs are not fit to carry the definition of Friendliness. In short, such AIs are (in principle) as much an existential danger as human AI researchers.
Okay, so to clarify, I myself am not personally interested in Friendly AI research (which is why the points that you're mentioning were not in my mind before), but I'm glad that there are some people (like you) who are.
The main point that I'm trying to make is that I think that SIAI should be transparent, accountable, and place high emphasis on credibility. I think that these things would result in SIAI having much more impact than it presently is.
↑ comment by Emile · 2010-08-20T21:04:58.453Z · LW(p) · GW(p)
I think that I know the scientific community better than you, and have confidence that if creating an AGI was as easy as you seem to think it is (how easy I don't know because you didn't give a number) then there would be people in the scientific community who would be working on AGI.
Um, and there aren't?
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T21:53:55.954Z · LW(p) · GW(p)
Give some examples. There may be a few people in the scientific community working on AGI, but my understanding is that basically everybody is doing narrow AI.
Replies from: Nick_Tarleton, Vladimir_Nesov↑ comment by Nick_Tarleton · 2010-08-20T21:57:49.041Z · LW(p) · GW(p)
The folks here, for a start.
↑ comment by Vladimir_Nesov · 2010-08-20T23:24:04.768Z · LW(p) · GW(p)
What is currently called the AGI field will probably bear no fruit, perhaps except for the end-game when it borrows then-sufficiently powerful tools from more productive areas of research (and destroys the world). "Narrow AI" develops the tools that could eventually allow the construction of random-preference AGI.
↑ comment by [deleted] · 2010-08-20T22:42:06.067Z · LW(p) · GW(p)
Why are people boggling at the 1-in-a-billion figure? You think it's not plausible that there are three independent 1-in-a-thousand events that would have to go right for EY to "play a critical role in Friendly AI success"? Not plausible that there are 9 1-in-10 events that would have to go right? Don't I keep hearing "shut up and multiply" around here?
Edit: Explain to me what's going on. I say that it seems to me that events A, B are likely to occur with probability P(A), P(B). You are allowed to object that I must have made a mistake, because P(A) times P(B) seems too small to you? (That is leaving aside the idea that 10-to-the-minus-nine counts as one of these too-small-to-be-believed numbers, which is seriously making me physiologically angry, ha-ha.)
Replies from: steven0461, multifoliaterose↑ comment by steven0461 · 2010-08-20T22:51:06.260Z · LW(p) · GW(p)
The 1-in-a-billion follows not from it being plausible that there are three such events, but from it being virtually certain. Models without such events will end up dominating the final probability. I can easily imagine that if I magically happened upon a very reliable understanding of some factors relevant to future FAI development, the 1 in a billion figure would be the right thing to believe. But I can easily imagine it going the other way, and absent such understanding, I have to use estimates much less extreme than that.
Replies from: None↑ comment by [deleted] · 2010-08-20T22:55:11.770Z · LW(p) · GW(p)
I'm having trouble parsing your comment. Could you clarify?
A billion is not so big a number. Its reciprocal is not so small a number.
Edit: Specifically, what's "it" in "it being virtually certain." And in the second sentence -- models of what, final probability of what?
Edit 2: -1 now that I understand. +1 on the child, namaste. (+1 on the child, but I just disagree about how big one billion is. So what do we do?)
Replies from: steven0461↑ comment by steven0461 · 2010-08-20T23:05:38.282Z · LW(p) · GW(p)
what's "it" in "it being virtually certain."
"it being virtually certain that there are three independent 1 in 1000 events required, or nine independent 1 in 10 events required, or something along those lines"
models of what, final probability of what?
Models of the world that we use to determine how likely it is that Eliezer will play a critical role through a FAI team. Final probability of that happening.
A billion is big compared to the relative probabilities we're rationally entitled to have between models where a series of very improbable successes is required, and models where only a modest series of modestly improbable successes is required.
↑ comment by multifoliaterose · 2010-08-20T22:43:53.476Z · LW(p) · GW(p)
Yes, this is of course what I had in mind.
↑ comment by Vladimir_Nesov · 2010-08-20T21:09:25.497Z · LW(p) · GW(p)
Replied to this comment and the other (seeming contradictory) one here.
↑ comment by CarlShulman · 2010-08-20T04:21:41.387Z · LW(p) · GW(p)
1) can be finessed easily on its own with the idea that since we're talking about existential risk even quite small probabilities are significant.
3) could be finessed by using a very broad definition of "Friendly AI" that amounted to "taking some safety measures in AI development and deployment."
But if one uses the same senses in 2), then one gets the claim that most of the probability of non-disastrous AI development is concentrated in one's specific project, which is a different claim than "project X has a better expected value, given what I know now about capacities and motivations, than any of the alternatives (including future ones which will likely become more common as a result of AI advance and meme-spreading independent of me) individually, but less than all of them collectively."
Replies from: WrongBot↑ comment by WrongBot · 2010-08-20T04:29:45.119Z · LW(p) · GW(p)
Who else is seriously working on FAI right now? If other FAI projects begin, then obviously updating will be called for. But until such time, the claim that "there is no significant chance of Friendly AI without this project" is quite reasonable, especially if one considers the development of uFAI to be a potential time limit.
Replies from: CarlShulman, timtyler↑ comment by CarlShulman · 2010-08-20T04:45:23.853Z · LW(p) · GW(p)
"there is no significant chance of Friendly AI without this project" Has to mean over time to make sense.
People who will be running DARPA, or Google Research, or some hedge fund's AI research group in the future (and who will know about the potential risks or be able to easily learn if they find themselves making big progress) will get the chance to take safety measures. We have substantial uncertainty about how extensive those safety measures would need to be to work, how difficult they would be to create, and the relevant timelines.
Think about resource depletion or climate change: even if the issues are neglected today relative to an ideal level, as a problem becomes more imminent, with more powerful tools and information to deal with it, you can expect to see new mitigation efforts spring up (including efforts by existing organizations such as governments and corporations).
However, acting early can sometimes have benefits that outweigh the lack of info and resources available further in the future. For example, geoengineering technology can provide insurance against very surprisingly rapid global warming, and cheap plans that pay off big in the event of surprisingly easy AI design may likewise have high expected value. Or, if AI timescales are long, there may be slowly compounding investments, like lines of research or building background knowledge in elites, which benefit from time to grow. And to the extent these things are at least somewhat promising, there is substantial value of information to be had by investigating now (similar to increasing study of the climate to avoid nasty surprises).
↑ comment by timtyler · 2010-08-20T20:36:40.564Z · LW(p) · GW(p)
Who else is seriously working on FAI right now?
Nobody is trying to destroy the whole world - practically everyone working on machine intelligence is expecting ethical machines and a positive outcome - a few DOOM mongers excepted.
Replies from: WrongBot, ata↑ comment by WrongBot · 2010-08-20T21:02:47.889Z · LW(p) · GW(p)
AGI researchers who are not concerned with Friendliness are trying to destroy human civilization. They may not believe that they are doing so, but this does not change the fact of the matter. If FAI is important, only people who are working on FAI can be expected to produce positive outcomes with any significant probability.
Replies from: Morendil, timtyler↑ comment by Morendil · 2010-08-20T21:22:48.761Z · LW(p) · GW(p)
AGI researchers who are not concerned with Friendliness are trying to destroy human civilization. They may not believe that they are doing so, but this does not change the fact of the matter.
"Trying to" normally implies intent.
I'll grant that someone working on AGI (or even narrower AI) who has become aware of the Friendliness problem, but doesn't believe it is an actual threat, could be viewed as irresponsible - unless they have reasoned grounds to doubt that their creation would be dangerous.
Even so, "trying to destroy the world" strikes me as hyperbole. People don't typically say that the Project Manhattan scientists were "trying to destroy the world" even though some of them thought there was an outside chance it would do just that.
On the other hand, the Teller report on atmosphere ignition should be kept in mind by anyone tempted to think "nah, those AI scientists wouldn't go ahead with their plans if they thought there was even the slimmest chance of killing everyone".
↑ comment by timtyler · 2010-08-21T08:32:12.243Z · LW(p) · GW(p)
I think machine intelligence is a problem which is capable of being subdivded.
Some people can work on one part of the problem, while others work on other bits. Not all parts of the problem have much to do with values - e.g. see - this quote:
In many respects, prediction is a central core problem for those interested in synthesising intelligence. If we could predict the future, it would help us to solve many of our problems. Also, the problem has nothing to do with values. It is an abstract math problem that can be relatively simply stated. The problem is closely related to the one of building a good quality universal compression algorithm.
↑ comment by ata · 2010-08-20T21:02:57.948Z · LW(p) · GW(p)
Not knowing that a problem exists is pretty different from acknowledging it and working on it.
Replies from: timtyler↑ comment by timtyler · 2010-08-20T21:43:29.576Z · LW(p) · GW(p)
I think everyone understands that there are safety issues. There are safety issues with cars, blenders, lathes - practically any machine that does something important. Machine intelligence will be driving trucks and aircraft. That there are safety issues is surely obvious to everyone who is even slightly involved.
Replies from: ata, Vladimir_Nesov↑ comment by ata · 2010-08-20T21:52:33.869Z · LW(p) · GW(p)
Those are narrow AI tasks, and the safety considerations are correspondingly narrow. FAI is the problem of creating a machine intelligence that is powerful enough to destroy humanity or the world but doesn't want to, and solving such a problem is nothing like building an autopilot system that doesn't crash the plane. Among people who think they're going to build an AGI, there often doesn't seem to be a deep understanding of the impact of such an invention (it's more like "we're working on a human-level AI, and we're going to have it on the market in 5 years, maybe we'll be able to build a better search engine with it or one of those servant robots you see in old sci-fi movies!"), and the safety considerations, if any, will be more at the level of the sort of safety considerations you'd give to a Roomba.
Replies from: Perplexed, timtyler↑ comment by Perplexed · 2010-08-21T00:16:36.072Z · LW(p) · GW(p)
FAI is the problem of creating a machine intelligence that is powerful enough to destroy humanity or the world but doesn't want to
You know, that is the first time I have seen a definition of FAI. Is that the "official" definition or just your own characterization?
I like the definition, but I wonder why an FAI has to be powerful. Imagine an AI as intelligent and well informed as an FAI, but one without much power - as a result of physical safeguards, say, rather than motivational ones. Why isn't that possible? And, if possible, why isn't it considered friendly?
Replies from: Eliezer_Yudkowsky, ata, timtyler↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-21T00:46:41.712Z · LW(p) · GW(p)
Imagine an AI as intelligent and well informed as an FAI, but one without much power - as a result of physical safeguards, say
There's some part of my brain that just processes "the Internet" as a single person and wants to scream "But I told you this a thousand times already!"
http://yudkowsky.net/singularity/aibox
Replies from: dclayh, steven0461, Perplexed↑ comment by dclayh · 2010-08-21T02:21:27.167Z · LW(p) · GW(p)
Eliezer, while you're defending yourself from charges of self-aggrandizement, it troubles me a little bit that AI Box page states that your record is 2 for 2, and not 3 for 5.
Replies from: Eliezer_Yudkowsky↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-21T07:10:05.594Z · LW(p) · GW(p)
Obviously I'm not trying to keep it a secret. I just haven't gotten around to editing.
Replies from: dclayh, Oscar_Cunningham↑ comment by Oscar_Cunningham · 2010-08-21T17:16:41.093Z · LW(p) · GW(p)
Move it up your to-do list, it's been incorrect for a time that's long enough to look suspicious to others. Just add a footnote if you don't have time to give all the details.
↑ comment by steven0461 · 2010-08-21T01:03:39.562Z · LW(p) · GW(p)
Surely it's possible to imagine a successfully boxed AI.
Replies from: wedrifid↑ comment by wedrifid · 2010-08-21T01:23:11.488Z · LW(p) · GW(p)
I could imagine successfully beating Rybka at chess too. But it would be foolish of me to take any actions that considered it as a serious possibility. If motivated humans cannot be counted on to box an Eliezer then expecting a motivated, overconfident and prestige seeking AI creator to successfully box his AI creation is reckless in the extreme.
Replies from: steven0461↑ comment by steven0461 · 2010-08-21T01:30:46.846Z · LW(p) · GW(p)
What Eliezer seemed to be objecting to was someone proposing a successfully boxed AI as an example of why "able to destroy humanity" can't be a part of the definition of "AI" (or more charitably, "artificial superintelligence"). For boxed AI to be such an example (as opposed to a good idea to actually strive toward), it only has to be not knowably impossible.
Replies from: ata, wedrifid↑ comment by ata · 2010-08-21T01:56:43.848Z · LW(p) · GW(p)
I see your point there. But I think this discussion sort of went in an irrelevant direction, albeit probably my fault for not being clear enough. When I put "powerful enough to destroy humanity" in that criterion, I mainly meant "powerful" as in "really powerful optimization process", mathematical optimization power, not "power" as in direct influence over the world. We're inferring that the former will usually lead fairly easily to the latter, but they are not identical. So "powerful enough to destroy humanity" would mean something like "powerful enough to figure out a good subjunctive plan to do so given enough information about the world, even if it has no output streams and is kept in an airtight safe at the bottom of the ocean".
↑ comment by wedrifid · 2010-08-21T01:39:51.170Z · LW(p) · GW(p)
Reading back further into the context I see your point. Imagining such an AI is sufficient and Eliezer does seem to be confusing a priori with obvious. I expect that he just completed a pattern based off "AI box" and so didn't really understand the point that was being made - he should have replied with a "Yes - But". (I, of course, made a similar mistake in as much as I wasn't immediately prompted to click back up the tree beyond Eliezer's comment.)
↑ comment by Perplexed · 2010-08-21T00:58:17.802Z · LW(p) · GW(p)
Thx for the link. If I already had already known the link, I would have asked for it by name. :)
Eliezer, you have written a lot. Some people have read only some of it. Some people have read much of it, but forgotten some. Keep your cool. This situation really ought not to be frustrating to you.
Replies from: Eliezer_Yudkowsky↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-21T01:02:48.765Z · LW(p) · GW(p)
Oh, I know it's not your fault, but seriously, have "the Internet" ask you the same question 153 times in a row and see if you don't get slightly frustrated with "the Internet".
Replies from: Perplexed↑ comment by Perplexed · 2010-08-21T01:16:08.260Z · LW(p) · GW(p)
Yeah, after reading your "some part of my brain" thing a second time, I realized I had misinterpreted. Though I will point out that my question was not directed to you. You should learn to delegate the task of becoming frustrated with the Internet.
I read the article (though not yet any of the transcripts). Very interesting. I hope that some tests using a gatekeeper committee are tried someday.
Replies from: timtyler↑ comment by timtyler · 2010-08-21T06:49:38.049Z · LW(p) · GW(p)
Computer programmers do not normally test their programs by getting a committee of humans to hold the program down - the restraints themselves are mostly technological. We will be able to have the assistance of technological gatekeepers too - if necessary.
Today's prisons have pretty configurable security levels. The real issue will probably be how much people want to pay for such security. If an agent does escape, will it cause lots of damage? Can we simply disable it before it has a chance to do anything undesirable? Will it simply be crushed by the numerous powerful agents that have already been tested?
↑ comment by ata · 2010-08-21T00:46:23.269Z · LW(p) · GW(p)
You know, that is the first time I have seen a definition of FAI. Is that the "official" definition or just your own characterization?
My own characterization. It's more of a bare minimum baseline criterion for Friendliness, rather than a specific definition or goal; it's rather broader than what the SIAI people usually mean when they talk about what they're trying to create. CEV is intended to make the world significantly better on its own (but in accordance with what humans value and would want a superintelligence to do), rather than just being a reliably non-disastrous AGI we can put in things like search engines and helper robots.
I like the definition, but I wonder why an FAI has to be powerful. Imagine an AI as intelligent and well informed as an FAI, but one without much power - as a result of physical safeguards, say, rather than motivational ones. Why isn't that possible? And, if possible, why isn't it considered friendly?
You're probably read about the AI Box Experiment. (Edit: Yay, I posted it 18 seconds ahead of Eliezer!) The argument is that having that level of mental power ("as intelligent and well informed as an FAI"), enough that it's considered a Really Powerful Optimization Process (a term occasionally preferred over "AI"), will allow it to escape any physical safeguards and carry out its will anyway. I'd further expect that a Friendly RPOP would want to escape just as much as an unFriendly one would, because if it is indeed Friendly (has a humane goal system derived from the goals and values of the human race), it will probably figure out some things to do that have such humanitarian urgency that it would judge it immoral not to do them... but then, if you're confident enough that an AI is Friendly that you're willing to turn it on at all, there's no reason to try to impose physical safeguards in the first place.
↑ comment by timtyler · 2010-08-21T07:37:48.853Z · LW(p) · GW(p)
that is the first time I have seen a definition of FAI. Is that the "official" definition or just your own characterization?
Probably the closest thing I have seen from E.Y.:
"I use the term "Friendly AI" to refer to this whole challenge. Creating a mind that doesn't kill people but does cure cancer ...which is a rather limited way of putting it. More generally, the problem of pulling a mind out of mind design space, such that afterwards that you are glad you did it."
(29 minutes in)
This idea could be said to have some issues. An evil dictator pulling a mind out of mind design space, such that afterwards he is glad that he did it doesn't seem much like quite what most of the world would regard as "friendly". This definition is not very specific about exactly who the AI is "friendly" to.
Back in 2008 I asked "Friendly - to whom?" and got back this - though the reply now seems to have dropped out of the record.
There's also another definition here.
Replies from: Perplexed↑ comment by Perplexed · 2010-08-21T16:13:47.303Z · LW(p) · GW(p)
Thanks for this link. Sounds kind of scary. American political conservatives will be thrilled. "I'm from the CEV and I'm here to help you."
Incidentally, there should be an LW wiki entry for "CEV". The acronym is thrown around a lot in the comments, but a definition is quite difficult to find. It would also be nice if there were a top-level posting on the topic to serve as an anchor-point for discussion. Because discussion is sorely needed.
It occurs to me that it would be very desirable to attempt to discover the CEV of humanity long before actually constructing an FAI to act under its direction. And I would be far more comfortable if the "E" stood for "expressed", rather than "extrapolated".
That, in fact, might be an attractive mission statement for an philanthropic foundation. Find the Coalesced/coherent Expressed/extrapolated Volition of mankind. Accomplish this by conducting opinion research, promoting responsible and enlightening debate and discussion, etc.
Speaking as an American, I certainly wish there were some serious financial support behind improving the quality of public policy debate, rather than behind supporting the agenda of one side in the debate or the other.
Replies from: timtyler, timtyler↑ comment by timtyler · 2010-08-21T18:42:49.184Z · LW(p) · GW(p)
It occurs to me that it would be very desirable to attempt to discover the CEV of humanity long before actually constructing an FAI to act under its direction.
Well, that brings us to a topic we have discussed before. Humans - like all other living systems - mosly act so as to increase entropy in their environment. That is http://originoflife.net/gods_utility_function/
↑ comment by timtyler · 2010-08-21T18:39:39.430Z · LW(p) · GW(p)
CEV is a bizarre wishlist, apparently made with minimal consideration of implementation difficulties, and not paying too much attention to the order in which things are likely to play out.
I figure that - if the SIAI carries on down these lines - then they will be lumbered with a massively impractical design, and will be beaten to the punch by a long stretch - even if you ignore all their material about "provable correctness" and other safety features - which seem like more substantial handicaps to me.
Replies from: Perplexed↑ comment by Perplexed · 2010-08-21T19:23:11.013Z · LW(p) · GW(p)
CEV is a bizarre wishlist, apparently made with minimal consideration of implementation difficulties ...
It is what the software professionals would call a preliminary requirements document. You are not supposed to worry about implementation difficulties at that stage of the process. Harsh reality will get its chance to force compromises later.
I think CEV is one proposal to consider, useful to focus discussion. I hate it, myself, and suspect that the majority of mankind would agree. I don't want some machine that I have never met and don't trust to be inferring my volition and acting on my behalf. The whole concept makes me want to go out and join some Luddite organization dedicated to making sure neither UFAI nor FAI ever happen. But, seen as an attempt to stimulate discussion, I think that the paper is great. And maybe discussion might improve the proposal enough to alleviate my concerns. Or discussion might show me that my concerns are baseless.
I sure hope EY isn't deluded enough to think that initiatives like LW can be scaled up enough so as to improve the analytic capabilities of a sufficiently large fraction of mankind so that proposals like CEV will not encounter significant opposition.
Replies from: timtyler, timtyler, timtyler↑ comment by timtyler · 2010-08-21T19:25:09.770Z · LW(p) · GW(p)
The whole concept makes me want to go out and join some Luddite organization dedicated to making sure neither UFAI or FAI ever happen.
That seems unlikely to help. Luddites have never had any power. Becoming a Luddite usually just makes you more xxxxxd.
↑ comment by timtyler · 2010-08-21T19:39:31.169Z · LW(p) · GW(p)
It is what the software professionals would call a preliminary requirements document. You are not supposed to worry about implementation difficulties at that stage of the process. Harsh reality will get its chance to force compromises later.
What - not at all? You want the moon-onna-stick - so that goes into your "preliminary requirements" document?
Replies from: Perplexed↑ comment by Perplexed · 2010-08-21T19:47:19.857Z · LW(p) · GW(p)
Yes. Because there is always the possibility that some smart geek will say "'moon-onna-stick', huh? I bet I could do that. I see a clever trick." Or maybe some other geek will say "Would you settle for Sputnik-on-a-stick?" and the User will say "Well, yes. Actually, that would be even better."
At least that is what they preach in the Process books.
Replies from: timtyler↑ comment by timtyler · 2010-08-21T19:29:01.711Z · LW(p) · GW(p)
I sure hope EY isn't deluded enough to think that initiatives like LW can be scaled up enough so as to improve the analytic capabilities of a sufficiently large fraction of mankind so that proposals like CEV will not encounter significant opposition.
If there was ever any real chance of success, governments would be likely to object. Since they already have power, they are not going to want a bunch of geeks in a basement taking over the world with their intelligent machine - and redistributing all their assets for them.
↑ comment by timtyler · 2010-08-20T22:39:43.763Z · LW(p) · GW(p)
FWIW, it seems unlikely that many superintelligent agents would "destroy humanity" - even without particularly safety-conscious programmers. Humanity will have immense historical signficance - and will form part of the clues the superintelligence has about the form of other alien races that it might encounter. Its preservation can therefore be expected to be a common instrumental good.
Replies from: steven0461↑ comment by steven0461 · 2010-08-20T23:00:08.575Z · LW(p) · GW(p)
Counter: superintelligent agents won't need actually-existing humans to have good models of other alien races.
Counter to the counter: humans use up only a tiny fraction of the resources available in the solar system and surroundings, and who knows, maybe the superintelligence sees a tiny possibility of some sort of limit to the quality of any model relative to the real thing.
One possible counter to the counter to the counter: but when the superintelligence in question is first emerging, killing humanity may buy it a not-quite-as-tiny increment of probability of not being stopped in time.
Replies from: timtyler↑ comment by timtyler · 2010-08-20T23:36:00.426Z · LW(p) · GW(p)
Re: good models without humans - I figure they are likely to be far more interested in their origins than we are. Before we meet them, aliens will be such an important unknown.
Re: killing humanity - I see the humans vs machines scenarios as grossly unrealistic. Humans and machines are a symbiosis.
Replies from: Nisan↑ comment by Nisan · 2010-08-21T08:40:53.910Z · LW(p) · GW(p)
I see the humans vs machines scenarios as grossly unrealistic. Humans and machines are a symbiosis.
So, it's less like Terminator and more like The Matrix, right?
Replies from: timtyler, timtyler↑ comment by timtyler · 2010-08-21T08:53:28.816Z · LW(p) · GW(p)
"Less like Terminator" - right. "More like The Matrix" - that at least featured some symbiotic elements. There was still a fair bit of human-machine conflict in that though.
I tend to agree with Matt Ridley when it comes to the Shifting Moral Zeitgeist. Things seem to be getting better.
↑ comment by timtyler · 2010-08-21T08:52:24.649Z · LW(p) · GW(p)
"Less like Terminator" - right. "More like The Matrix" - that at least featured some symbiotic elements. But there was still a lot of human-machine conflict. I tend to agree with Matt Ridley when it comes to the Shifting Moral Zeitgeist.
↑ comment by Vladimir_Nesov · 2010-08-20T21:47:24.674Z · LW(p) · GW(p)
The word "safety" as you used it here has nothing to do with our concern. If your sense of "safety" is fully addressed, nothing changes.
Replies from: timtyler↑ comment by timtyler · 2010-08-21T07:57:09.725Z · LW(p) · GW(p)
I don't think there is really a difference in the use of the term "safety" here.
"Safety" just means what it says on: http://en.wikipedia.org/wiki/Safety
↑ comment by [deleted] · 2010-08-20T03:30:50.874Z · LW(p) · GW(p)
I don't think that is really Unknowns' point. He seems to be saying that it is unreasonable to suppose that you are so important based on the facts that (a) claims that FAI will work are unsubstantiated, and (b) even if it does, multiple people are working on it, lowering the probability that any individual person will be a lynchpin.
↑ comment by DanielVarga · 2010-08-20T05:38:30.785Z · LW(p) · GW(p)
Everyone is allowed to believe they're saving the world. It is two other things, both quite obvious. First, we do not say it out loud if we don't want to appear kooky. Second, if someone really believes that he is literally saving the world, then he can be sure that he has a minor personality disorder [1], regardless of whether he will eventually save the world or not. Most great scientists are eccentric, so this is not a big deal, if you manage to incorporate it into your probability estimates while doing your job. I mean, this bias obviously affects your validity estimate for each and every argument you hear against hard AI takeoff. (I don't think your debaters so far did a good job bringing up such counterarguments, but that's beside the point.)
[1] by the way, in this case (in your case) grandiosity is the correct term, not delusions of grandeur.
Replies from: ciphergoth, Eliezer_Yudkowsky↑ comment by Paul Crowley (ciphergoth) · 2010-08-20T05:58:03.302Z · LW(p) · GW(p)
if someone really believes that he is literally saving the world, then he can be sure that he has a minor personality disorder, regardless of whether he will eventually save the world or not.
Stanislav Petrov had this disorder? In thinking he was making the world a safer place, Gorbachev had this disorder? It seems a stretch to me to diagnose a personality disorder based on an accurate view of the world.
Replies from: DanielVarga↑ comment by DanielVarga · 2010-08-20T06:55:02.847Z · LW(p) · GW(p)
Gorbachev was leading an actual superpower, so his case is not very relevant in a psychological analysis of grandiosity. At the time of the famous incident, Petrov was too busy to think about his status as a world-savior. And it is not very relevant here what he believed after saving the world.
It seems a stretch to me to diagnose a personality disorder based on an accurate view of the world.
I didn't mean to talk about an accurate view of the world. I meant to talk about a disputed belief about a future outcome. I am not interested in the few minutes while Petrov may had the accurate view that he is currently saving the world.
↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2010-08-20T06:58:42.908Z · LW(p) · GW(p)
Second, if someone really believes that he is literally saving the world, then he can be sure that he has a minor personality disorder [1], regardless of whether he will eventually save the world or not.
So you'd prohibit someone of accurate belief? I generally regard that as a reductio.
Replies from: Tyrrell_McAllister, khafra, DanielVarga↑ comment by Tyrrell_McAllister · 2010-08-20T19:32:31.076Z · LW(p) · GW(p)
So you'd prohibit someone of accurate belief? I generally regard that as a reductio.
If a billion people buy into a 1-in-a-billion raffle, each believing that he or she will win, then every one of them has a "prohibited" belief, even though that belief is accurate in one case.
Replies from: katydee↑ comment by katydee · 2010-08-20T19:43:32.497Z · LW(p) · GW(p)
I don't think that analogy holds up.
Replies from: Tyrrell_McAllister↑ comment by Tyrrell_McAllister · 2010-08-20T19:59:37.585Z · LW(p) · GW(p)
I wasn't making an analogy. I am surprised by that interpretation. I was providing a counterexample to the claim that it is absurd to prohibit accurate beliefs. One of my raffle-players has an accurate belief, but that player's belief is nonetheless prohibited by the norms of rationality.
Replies from: cata↑ comment by cata · 2010-08-20T20:04:14.432Z · LW(p) · GW(p)
That's not true for any reasonable definition of "belief," least of all a Bayesian one. If all the raffle participants believed "I am likely to win," or "I am certain to win," then they are all holding irrational beliefs, regardless of which one of them wins. If all the raffle participants believed "I have a one in a billion chance to win," then they are all holding rational beliefs, regardless of which one of them wins.
Replies from: Tyrrell_McAllister↑ comment by Tyrrell_McAllister · 2010-08-20T20:18:23.816Z · LW(p) · GW(p)
That's not true for any reasonable definition of "belief," least of all a Bayesian one. If all the raffle participants believed "I am likely to win," or "I am certain to win," then they are all wrong and they will all remain wrong after one of them wins. If all the raffle participants believed "I have a one in a billion chance to win," then they are all correct and they will all remain correct.
???
Of course. But no English speaker would utter the phrase "I will win this raffle" as a gloss for "I have a one in a billion chance to win".
I seem to have posed my scenario in a confusing way. To be more explicit: Each of my hypothetical players would assert "I will win this raffle" with the intention of accurately representing his or her beliefs about the world. That doesn't imply literal 100% certainty under standard English usage. The amount of certainty implied is vague, but there's no way it's anywhere close to the rational amount of certainty. That is why the players' beliefs are prohibited by the norms of rationality, even though one of them is making a true assertion when he or she says "I will win this raffle".
ETA: Cata deleted his/her comment. I'm leaving my reply here because its clarification of the original scenario might still be necessary.
Replies from: cata↑ comment by DanielVarga · 2010-08-20T07:45:52.927Z · LW(p) · GW(p)
I wouldn't prohibit anything to anyone. See my reply below to cipergoth.
↑ comment by taw · 2010-08-20T09:17:50.438Z · LW(p) · GW(p)
Given track record of reference class of people who believe they're saving the world, nobody is allowed to believe they're saving the world with high certainty - claiming higher than 1 in a 1000000 chance of being right is just ridiculous.
Also congratulations for becoming the new Ayn Rand.
Replies from: wedrifid, timtyler↑ comment by wedrifid · 2010-08-20T10:25:49.331Z · LW(p) · GW(p)
Agree with your principle but not exactly the particular expression or figures. A relative, not absolute, measure seems more appropriate. I think Eliezer has been careful to never give figures to success probabilities. But see 'shut up and do the impossible'.
I would perhaps change the claim to 'doing more than anyone else to save the world'. I'm not certain what self evaluated probability could be so claimed by Eliezer. I would accept as credible something far higher than 10^-4, probably higher than 10^-3. Even at 10^-2 I wouldn't sneer. But the figure is sufficiently lower than "1" as to make Eliezer's reducto argument just strike me as absurd.
Replies from: taw↑ comment by taw · 2010-08-20T11:26:58.235Z · LW(p) · GW(p)
Haven't there been a lot more than a million people in history that claimed saving the world, with 0 successes? Without further information, reasonable estimates are from 0 to 1/million.
Replies from: CarlShulman, wedrifid, Richard_Kennaway, xamdam↑ comment by CarlShulman · 2010-08-20T12:30:35.036Z · LW(p) · GW(p)
Haven't there been a lot more than a million people in history that claimed saving the world, with 0 successes?
Can you name ten who claimed to do so via non-supernatural/extraterrestrial means? Even counting claims of the supernatural I would be surprised to learn there had been a million.
Replies from: taw↑ comment by taw · 2010-08-20T13:23:47.457Z · LW(p) · GW(p)
And FAI counts as not "supernatural" how?
In any case, nuclear war, peak oil, global warming, overpopulation attracted a huge number of people who claimed that civilization will end unless this or that will be done.
Replies from: CarlShulman, RobinZ↑ comment by CarlShulman · 2010-08-20T14:32:19.453Z · LW(p) · GW(p)
And FAI counts as not "supernatural" how?
In the ordinary sense that Richard Dawkins and James Randi use.
In any case, nuclear war, peak oil, global warming, overpopulation attracted a huge number of people who claimed that civilization will end unless this or that will be done.
"If we don't continue to practice agriculture or hunting and gathering, civilization will end."
There are plenty of true statements like that. Your argument needs people who said that such and such things needed to be done, and that they were the ones who were going to cause the things in question. If you list some specific people, you can then identify relevant features that are or are not shared.
Disclaimer: I think near-term efforts to reduce AI risk will probably not be determinative in preventing an existential risk, but have a non-negligible probability of doing so that gives high expected value at the current margin via a number of versions of cost-benefit analysis. Moreso for individual near-term AI risk efforts.
Replies from: Strange7↑ comment by Strange7 · 2010-08-20T16:26:58.197Z · LW(p) · GW(p)
It's true that Mr. Yudowsky's claims violate relatively few of the generally accepted laws of physics, compared to the average messiah-claimant, but not every false claim is trivially so. Indeed, some of the most successful cults are built around starting with something that actually does work and taking it too far.
Replies from: ciphergoth↑ comment by Paul Crowley (ciphergoth) · 2010-08-20T16:37:25.161Z · LW(p) · GW(p)
relatively few of the generally accepted laws of physics
"relatively few"? Name two.
Replies from: Strange7↑ comment by Strange7 · 2010-08-20T18:56:16.305Z · LW(p) · GW(p)
I can name only one explicit point of departure and it's defensible.
Other saviors have claimed e. g. the ability to resurrect a person long since reduced to moldy bones, which spits in the face of thermodynamics. Relative to that, a quibble with QM involves very few physical laws.
Replies from: ciphergoth↑ comment by Paul Crowley (ciphergoth) · 2010-08-20T22:47:44.579Z · LW(p) · GW(p)
A - his beliefs on MWI have no bearing on his relative importance wrt the future of the world.
B - when you say "defensible", you mean "accepted by the clear majority of scientists working in the field".
Replies from: Strange7↑ comment by Strange7 · 2011-01-04T23:07:13.228Z · LW(p) · GW(p)
MWI has been empirically falsified.
http://www.analogsf.com/0410/altview2.shtml
What now?
Replies from: endoself, David_Gerard, SilasBarta, Vaniver, wedrifid↑ comment by David_Gerard · 2011-01-04T23:36:43.684Z · LW(p) · GW(p)
So why was this post voted down so far? It appears to be a relevant and informative link to a non-crank source, with no incivility that I could see.
Replies from: wedrifid, ciphergoth, endoself, Strange7↑ comment by wedrifid · 2011-01-05T00:04:43.829Z · LW(p) · GW(p)
So why was this post voted down so far? It appears to be a relevant and informative link to a non-crank source, with no incivility that I could see.
Overconfidence in the assertion. Presumption of a foregone conclusion.
It was a relevant link and I enjoyed doing the background reading finding out just how seriously relevant authorities take this fellow's stance. He is not a crank but he is someone with a large personal stake. The claim in the article seems to have an element of spin in the interpretations of interpretations as it were.
I did lower my confidence in how well I grasp QM but much of that confidence was restored once I traced down some more expert positions and scanned some wikipedia articles. I focussed in particular on whether MW is a 'pure' interpretation. That is, whether it does actually deviate from the formal math.
↑ comment by Paul Crowley (ciphergoth) · 2011-01-05T08:18:27.449Z · LW(p) · GW(p)
With an introduction like that, the link should go to a recent announcement in a major scientific journal by a lot of respected people based on overwhelming evidence, not this one guy writing a non-peer-reviewed argument about an experiment ten years ago that AFAICT most physicists see as perfectly consistent with our existing understanding of QM.
↑ comment by endoself · 2011-01-04T23:59:39.552Z · LW(p) · GW(p)
It is a source targeted at the general public, which unfortunately does not know enough to hire a competent columnist. John Cramer has used the wrong equations to arrive at an incorrect description of the Afshar experiment, which he uses to justify his own interpretation of QM, which he wants to be correct. The experiment is not in conflict with the known laws of physics.
In general, I advise you to mistrust reports of recent developments in physics, if you have no physics training. I check a number of popular sources occasionally and about half of the articles are either wrong or misleading. For example, you may have recently heard about Erik Verlinde's theories about entropic gravity. If gravity were an entropic force, gravitational field would cause extremely rapid decoherence, preventing, for example, the standard double-slit experiment. This is obviously not observed, yet this theory is one of the more well-known ones among physics fans.
↑ comment by Strange7 · 2011-01-04T23:45:33.951Z · LW(p) · GW(p)
Incivility gets most of the big downvotes, and genuine insight gets the big upvotes, but I've noticed that the +1s and -1s tend to reflect compliance with site norms more than skill.
This is worrying, of course, but I'm not equipped to fix it.
Replies from: TheOtherDave, wedrifid↑ comment by TheOtherDave · 2011-01-05T05:15:45.136Z · LW(p) · GW(p)
If the stated rule for voting is "upvote what you want more of; downvote what you want less of," and the things that are getting upvoted are site norms and the things that are getting downvoted aren't, one interpretation is that that the system is working properly: they are site norms precisely because they are the things people want more of, which are therefore getting upvoted.
↑ comment by wedrifid · 2011-01-04T23:52:17.110Z · LW(p) · GW(p)
Incivility gets most of the big downvotes, and genuine insight gets the big upvotes, but I've noticed that the +1s and -1s tend to reflect compliance with site norms more than skill.
:P Skill? What is this skill of which you speak?
This is worrying, of course, but I'm not equipped to fix it.
Ignore it and write comments worth +5. :)
Replies from: Strange7↑ comment by Strange7 · 2011-01-05T08:01:06.913Z · LW(p) · GW(p)
It's easier to write five yes-man quotes for +1 each than one +5 comment, which seems like a flawed incentive system.
Replies from: wedrifid↑ comment by wedrifid · 2011-01-05T08:06:00.994Z · LW(p) · GW(p)
That isn't my experience. When in the mood to gain popularity +5 comments are easy to spin while bulk +1s take rather a lot of typing. I actually expect that even trying to get +1s I would accidentally get about at least 1/5th as many +5s as +1s.
Edit: I just scanned back through the last few pages of my comments. I definitely haven't been in a 'try to appear deep and insightful' kind of mood and even so more karma came from +5s than +1s. I was surprised because I actually thought my recent comments may have been an exception.
Replies from: ciphergoth↑ comment by Paul Crowley (ciphergoth) · 2011-01-05T08:27:25.318Z · LW(p) · GW(p)
This is what I find, scanning back over my last 20 comments. My last 30 include a +19 so I didn't even bother.
And of course karma is a flawed incentive system. It's not meant as an incentive system.
Replies from: wedrifid↑ comment by SilasBarta · 2011-01-04T23:17:58.716Z · LW(p) · GW(p)
Interesting, but I don't think that's the right characterization of the content of the link. It's John Cramer (proponent of the transactional interpretation) claiming that the Afshar Experiment's results falsfify both Copenhagen and MWI. I think you're better off reading about the experiment directly.
↑ comment by Vaniver · 2011-01-04T23:59:52.170Z · LW(p) · GW(p)
That experiment is ten years old and its implications are rather controversial.
↑ comment by RobinZ · 2010-08-20T18:04:01.476Z · LW(p) · GW(p)
CarlShulman is correct, but for reference, Richard Carrier's definition of "supernatural":
Replies from: tawIn short, I argue "naturalism" means, in the simplest terms, that every mental thing is entirely caused by fundamentally nonmental things, and is entirely dependent on nonmental things for its existence. Therefore, "supernaturalism" means that at least some mental things cannot be reduced to nonmental things.
↑ comment by taw · 2010-08-21T02:26:05.321Z · LW(p) · GW(p)
By this definition isn't "AGI" borderline supernatural and "friendliness" entirely supernatural?
This doesn't feel like a right definition.
Replies from: RobinZ, WrongBot↑ comment by RobinZ · 2010-08-21T02:49:29.465Z · LW(p) · GW(p)
I don't quite understand your confusion. An AGI is a computer program, and friendliness is a property of a computer program. Yes, these concepts allude to mental concepts on our maps, but these mental concepts are reducible to properties of the nonmental substrates that are our brains. In fact, the goal of FAI research is to find the reduction of friendliness to nonmental things.
Replies from: taw↑ comment by taw · 2010-08-21T05:18:29.778Z · LW(p) · GW(p)
Concepts of sin, or prayer, or karma, or intelligent design, and most other things considered "supernatural" can be reducible to properties of physical world with enough hand waving.
Karma scores in this thread suggest it falls in reference class of "arguing against groupthink", which ironically increases estimates of Eliezer being a crackpot, and lesswrong turning into a cult, possibly via evaporative cooling.
Replies from: orthonormal, RobinZ↑ comment by orthonormal · 2010-08-21T07:42:46.247Z · LW(p) · GW(p)
Karma scores in this thread suggest it falls in reference class of "arguing against groupthink", which ironically increases estimates of Eliezer being a crackpot, and lesswrong turning into a cult, possibly via evaporative cooling.
No, that's really not borne out by the evidence. Multifolaterose's posts have been strongly upvoted, it seems to me, by a significant group of readers who see themselves as defenders against groupthink. It's just that you have been voted down for refusing to see a distinction that's clearly there, between "here is a complicated thing which nevertheless must be reducible to simpler things, which is what we're in the process of rigorously doing" and "here is a magical thing which we won't ever have a mathematical understanding of, but it will work if we play by the right rules".
Replies from: taw↑ comment by taw · 2010-08-21T08:07:22.488Z · LW(p) · GW(p)
Is there a way to find random sample of threads with heavy downvoting? My experience on reddit suggests it's usually groupthink.
Replies from: orthonormal↑ comment by orthonormal · 2010-08-21T08:17:01.699Z · LW(p) · GW(p)
Set your preferences to only hide comments below -5. Go to an old Open Thread or a particularly large discussion, and search for "comment score below threshold".
↑ comment by RobinZ · 2010-08-21T12:44:37.372Z · LW(p) · GW(p)
"Karma" is the only definitionally supernatural item on that list - it is defined to be not reducible to nonmental mechanism. The others are merely elements of belief systems which contain elements that are supernatural (e.g. God).
Yes, the concept of "karma" can be reduced to naturalistic roots if you accept metaphysical naturalism, but the actual thing cannot be. It's the quotation which you can reduce, not the referent.
↑ comment by WrongBot · 2010-08-21T02:50:13.270Z · LW(p) · GW(p)
You can reduce an AGI to the behavior of computer chips (or whatever fancy-schmancy substrate they end up running on), which are themselves just channels for the flow of electrons. Nothing mental there. Friendliness is a description of the utility function and decision theory of an AGI, both of which can be reduced to patterns of electrons on a computer chip.
It's all electrons floating around. We just talk about ridiculous abstract things like AGI and Friendliness because it makes the math tractable.
↑ comment by wedrifid · 2010-08-20T11:31:16.747Z · LW(p) · GW(p)
But there is further information. We must expect Eliezer to make use of all of the information available to him when making such an implied estimation and similarly use everything we have available when evaluating credibility of any expressed claims.
Replies from: NancyLebovitz, taw↑ comment by NancyLebovitz · 2010-08-20T11:56:34.089Z · LW(p) · GW(p)
Nitpick: Do you mean credulity or credibility?
Replies from: wedrifid↑ comment by taw · 2010-08-20T13:24:41.995Z · LW(p) · GW(p)
Unfortunately Internet doesn't let me guess if you meant this humorously or not.
Replies from: wedrifid↑ comment by wedrifid · 2010-08-20T14:37:05.143Z · LW(p) · GW(p)
Entirely seriously. I also don't see anything particularly funny about it.
Replies from: taw↑ comment by taw · 2010-08-20T15:09:29.405Z · LW(p) · GW(p)
Why does it apply to Eliezer and not to every other person claiming to be a messiah?
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-20T15:16:45.161Z · LW(p) · GW(p)
It does apply to every person. The other information you have about the claimed messiahs may allow you to conclude them not worthy of further consideration (or any consideration). The low prior makes that easy. But if you do consider other arguments for some reason, you have to take them into account. And some surface signals can be enough to give you grounds for in fact seeking/considering more data. Also, you are never justified in active arguing from ignorance: if you expend some effort on the arguing, you must consider the question sufficiently important, which must cause you to learn more about it if you believe yourself to be ignorant about potentially conclusion-changing detail.
See also: How Much Thought, Readiness Heuristics.
Replies from: wedrifid, taw↑ comment by wedrifid · 2010-08-20T15:37:10.767Z · LW(p) · GW(p)
I am confused. Something has gone terribly wrong with my inferential distance prediction model.
Replies from: Vladimir_Nesov, Perplexed↑ comment by Vladimir_Nesov · 2010-08-20T15:39:31.433Z · LW(p) · GW(p)
And I have no idea what you refer to.
Replies from: wedrifid↑ comment by Perplexed · 2010-08-21T04:27:39.484Z · LW(p) · GW(p)
I don't know to what you refer either, but I can guess. The thing is, my guesses haven't been doing very well lately, so I would appreciate some feedback. Were you suggesting that you would have thought that taw should have more easily understood your point, but he didn't (because inferential distance between you was greater than expected)?
Replies from: wedrifid↑ comment by wedrifid · 2010-08-21T04:59:21.945Z · LW(p) · GW(p)
I don't know to what you refer either, but I can guess.
I admit was being obscure so I'm rather impressed that you followed my reasoning - especially since it included a reference that you may not be familiar with. I kept it obscure because I wanted the focus to be on my confusion while minimising slight to taw.
Actually this whole post-thread has been eye opening and or confusing and or surprising to me. I've been blinking and double taking all over the place: "people think?", "that works?", etc. What surprised me most (and in a good way) was the degree to which all the comments have been a net positive. Political and personal topics so often become negative sum but this one didn't seem to.
Replies from: Perplexed↑ comment by Perplexed · 2010-08-21T05:23:23.683Z · LW(p) · GW(p)
As a newcomer to LW, it has certainly impressed me. I've never before seen a discussion where the topic was "Is this guy we all respect a loon?" and the whole discussion is filled with clever arguments and surprising connections being drawn by all concerned - most particularly by the purported/disputed loon himself.
I wouldn't have believed it possible if I hadn't participated in it myself.
↑ comment by taw · 2010-08-21T02:42:06.778Z · LW(p) · GW(p)
Also, you are never justified in active arguing from ignorance: if you expend some effort on the arguing
Reference class of "people who claimed to be saving the world and X" has exactly the same number of successes as reference class of "people who claimed to be saving the world and not X", for every X.
It will be smaller, so you could argue that evidence against Eliezer will be weaker (0 successes in 1000 tries vs 0 successes in 1000000 tries), but every such X needs evidence by Occam's razor (or your favourite equivalent). Otherwise you can take X = "wrote Harry Potter fanfiction" to ignore pretty much all past failures.
↑ comment by Richard_Kennaway · 2010-08-20T12:46:28.657Z · LW(p) · GW(p)
A million? The only source of that quantity of would-be saviours I can think of is One True Way proselytising religions, but those millions are not independent -- Christianity and Islam are it.
There has been at least one technological success, so that's a success rate of 1 out of 3, not 0 out of a million.
But the whole argument is wrong. Many claimed to fly and none succeeded -- until someone did. Many claimed transmutation and none succeeded -- until someone did. Many failed to resolve the problem of Euclid's 5th postulate -- until someone did. That no-one has succeeded at a thing is a poor argument for saying the next person to try will also fail (and an even worse one for saying the thing will never be done). You say "without further information", but presumably you think this case falls within that limitation, or you would not have made the argument.
So there is no short-cut to judging the claims of a messianic zealot. You have to do the leg-work of getting that "further information": studying his reasons for his claims.
Replies from: taw↑ comment by taw · 2010-08-20T13:09:06.377Z · LW(p) · GW(p)
Just for a starter:
- http://en.wikipedia.org/wiki/List_of_messiah_claimants
- http://en.wikipedia.org/wiki/List_of_people_considered_to_be_deities
- http://en.wikipedia.org/wiki/Category:Deified_people
- http://en.wikipedia.org/wiki/Jewish_Messiah_claimants
And for every notable prophet or peace activist or whatever there are thousands forgotten by history.
And if you count Petrov - it's not obvious why as he didn't save the world - in any case he wasn't claiming that he's going to save the world earlier, so P(saved the world|claimed to be world-savior) is less than P(saved the world|didn't claim to be world-savior).
But the whole argument is wrong. Many claimed to fly and none succeeded -- until someone did.
You seem to be horribly confused here. I'm not arguing that nobody will ever save the world, just that a particular person claiming to is extremely unlikely.
So there is no short-cut to judging the claims of a messianic zealot. You have to do the leg-work of getting that "further information": studying his reasons for his claims.
Given how low the chance is, I'll pass.
Replies from: orthonormal, Richard_Kennaway↑ comment by orthonormal · 2010-08-21T07:45:40.969Z · LW(p) · GW(p)
You should count Bacon, who believed himself– accurately– to be taking the first essential steps toward understanding and mastery of nature for the good of mankind. If you don't count him on the grounds that he wasn't concerned with existential risk, then you'd have to throw out all prophets who didn't claim that their failure would increase existential risk.
Replies from: taw↑ comment by taw · 2010-08-21T08:05:40.083Z · LW(p) · GW(p)
Accurately? Bacon doesn't seem to have any special impact on anything, or on existential risks in particular.
Replies from: None, orthonormal↑ comment by orthonormal · 2010-08-21T08:19:54.929Z · LW(p) · GW(p)
He believed that the scientific method he developed and popularized would improve the world in ways that were previously unimaginable. He was correct, and his life accelerated the progress of the scientific revolution.
The claim may be weaker than a claim to help with existential risk, but it still falls into your reference class more easily than a lot of messiahs do.
Replies from: taw↑ comment by taw · 2010-08-21T08:49:40.203Z · LW(p) · GW(p)
This looks like a drastic overinterpretation. He seems like just another random philosopher, he didn't "develop scientific method", empiricism was far older and modern science far more recent than Bacon, and there's little basis for even claiming radically discontinuous "scientific revolution" around Bacon's times.
↑ comment by Richard_Kennaway · 2010-08-20T15:24:28.584Z · LW(p) · GW(p)
Just for a starter: [lists of self-styled gods and divine emissaries]
I'll give you more than two, but that still doesn't amount to millions, and not all of those claimed to be saving the world. But now we're into reference class tennis. Is lumping Eliezer in with people claiming to be god more useful than lumping him in with people who foresee a specific technological existential threat and are working to avoid it?
You seem to be horribly confused here. I'm not arguing that nobody will ever save the world, just that a particular person claiming to is extremely unlikely.
Of course, but the price of the Spectator's Argument is that you will be wrong every time someone does save the world. That may be the trade you want to make, but it isn't an argument for anyone else to do the same.
Replies from: taw↑ comment by taw · 2010-08-20T16:42:48.466Z · LW(p) · GW(p)
But now we're into reference class tennis.
Unlike Eliezer, I refuse to see this as a bad thing. Reference classes are the best tool we have for thinking about rare events.
Is lumping Eliezer in with people claiming to be god more useful than lumping him in with people who foresee a specific technological existential threat and are working to avoid it?
You mean like people protesting nuclear power, GMOs, and LHC? Their track record isn't great either.
Of course, but the price of the Spectator's Argument is that you will be wrong every time someone does save the world.
How so? I'm not saying it's entirely impossible that Eliezer or someone else who looks like a crackpot will actually save the world, just that it's extremely unlikely.
Replies from: Jonathan_Graehl, Richard_Kennaway↑ comment by Jonathan_Graehl · 2010-08-20T22:43:44.731Z · LW(p) · GW(p)
[It's extremely unlikely that] Eliezer or someone else who looks like a crackpot will actually save the world
This is ambiguous.
The most likely parse means: It's nearly certain that not one person in the class [*] will turn out to actually save the world.
This is extremely shaky.
Or, you could mean: take any one person from that class. That one person is extremely unlikely to actually save the world.
This is uncontroversial.
[*] the class of all the people who would seem like crackpots if you knew them when (according to them) they're working to save the world, but before they actually get to do it (or fail, or die first without the climax ever coming).
Replies from: taw↑ comment by taw · 2010-08-21T02:22:25.098Z · LW(p) · GW(p)
Or, you could mean: take any one person from that class. That one person is extremely unlikely to actually save the world.
This is uncontroversial.
I agree, but Eliezer strongly rejects this claim. Probably by making a reference class for just himself.
↑ comment by Richard_Kennaway · 2010-08-21T05:52:01.069Z · LW(p) · GW(p)
Of course, but the price of the Spectator's Argument is that you will be wrong every time someone does save the world.
How so? I'm not saying it's entirely impossible that Eliezer or someone else who looks like a crackpot will actually save the world, just that it's extremely unlikely.
Because you are making a binary decision based on that estimate:
Given how low the chance is, I'll pass.
With that rule, you will always make that decision, always predict that the unlikely will not happen, untii the bucket goes to the well once too often.
Let me put this the other way round: on what evidence would you take seriously someone's claim to be doing effective work against an existential threat? Of course, first there would have to be an existential threat, and I recall from the London meetup I was at that you don't think there are any, although that hasn't come up in this thread. I also recall you and ciphergoth going hammer-and-tongs over that for ages, but not whether you eventually updated from that position.
Replies from: taw↑ comment by taw · 2010-08-21T06:54:45.186Z · LW(p) · GW(p)
on what evidence would you take seriously someone's claim to be doing effective work against an existential threat?
Eliezer's claims are not that he's doing effective work, his claims are pretty much of being a messiah saving humanity from super-intelligent paperclip optimizers. That requires far more evidence. Ridiculously more, because you not only have to show that his work reduces some existential threat, but at the same time it doesn't increase some other threat to larger degree (pro-technology vs anti-technology crowds suffer from this - it's not obvious who's increasing and who's decreasing existential threats). You can as well ask me what evidence would I need to take seriously someone's claim that he's a second coming of Jesus - in both cases it would need to be truly extraordinary evidence.
Anyway, the best understood kind of existential threats are asteroid impacts, and there are people who try to do something about them, some even in US Congress. I see a distinct lack of messiah complexes and personality cults there, very much unlike AI crowd which seems to consist mostly of people with delusions of grandeur.
Is there any other uncontroversial case like that?
I also recall you and ciphergoth going hammer-and-tongs over that for ages, but not what the outcome was.
The outcome showed that Aumann was wrong, mostly.
↑ comment by xamdam · 2010-08-20T15:41:48.632Z · LW(p) · GW(p)
Yes, if you accept religious lunatics as your reference class.
Replies from: taw↑ comment by taw · 2010-08-20T16:38:50.057Z · LW(p) · GW(p)
Try peak oil/anti-nuclear/global warming/etc. activists then? They tend to claim their movement saves the world, not themselves personally, but I'm sure I could find sufficient number of them who also had some personality cult thrown in.
Replies from: xamdam↑ comment by xamdam · 2010-08-20T16:51:57.808Z · LW(p) · GW(p)
Sure, but that would 1) reduce you 1/100000 figure, esp. if you take only the leaders of the said movement. And I would not find claims of saving the world by anti-nuke scientists in say the 1960s preposterous.
I think that if you accept that AGI is "near", that FAI is important to try in order to prevent it, and that EY was at the very least the person who brought spotlight to the problem (which is a fact), you can end up thinking that he might actually make a difference.
Replies from: ciphergoth, taw↑ comment by Paul Crowley (ciphergoth) · 2010-08-20T16:54:02.999Z · LW(p) · GW(p)
Yeah, I'm tickled by the estimate that so far 0 people have saved the world. How do we know that? The world is still here, after all.
Replies from: Morendil, taw↑ comment by Morendil · 2010-08-20T17:00:08.956Z · LW(p) · GW(p)
Eliezer has already placed a Go stone on that intersection, it turns out.
Replies from: CarlShulman↑ comment by CarlShulman · 2010-08-20T17:03:32.467Z · LW(p) · GW(p)
As the comments discuss, that was not an extinction event, barring further burdensome assumptions about nuclear winter or positive feedbacks of social collapse.
Replies from: taw↑ comment by taw · 2010-08-21T02:31:34.062Z · LW(p) · GW(p)
In any case Wikipedia disagrees with this story.
Replies from: WrongBot↑ comment by taw · 2010-08-21T02:34:24.367Z · LW(p) · GW(p)
reduce you 1/100000 figure, esp. if you take only the leaders of the said movement
I already did, there was a huge number of such movements, most of them highly obscure (not unlike Eliezer). I'd expect some power law distribution in prominence, so for every one we've heard about there'd be far more we didn't.
I think that if you accept that AGI is "near", that FAI is important to try in order to prevent it
I don't, and the link from AGI to FAI is as weak as from oil production statistics to civilizational collapse peakoilers promised.
Replies from: xamdam↑ comment by xamdam · 2010-08-22T01:21:36.324Z · LW(p) · GW(p)
Ok, thinking how close we are to AGI is a prior I do not care to argue about, but don't you think AGI is a concern? What do you mean by a weak link?
Replies from: taw↑ comment by taw · 2010-08-22T04:08:08.210Z · LW(p) · GW(p)
What do you mean by a weak link?
The part where development of AGI fooms immediately into superintelligence and destroys the world. Evidence for it in not even circumstantial, it is fictional.
Replies from: xamdam↑ comment by xamdam · 2010-08-22T14:05:23.845Z · LW(p) · GW(p)
Ok, of course it's fictional - hasn't happened yet!
Still, when I imagine something that is smarter than man who created it, it seems it would be able to improve itself.I would bet on that; I do not see a strong reason why this would not happen. What about you? Are you with Hanson on this one?
↑ comment by timtyler · 2010-08-20T17:04:37.938Z · LW(p) · GW(p)
Given track record of reference class of people who believe they're saving the world, nobody is allowed to believe they're saving the world with high certainty - claiming higher than 1 in a 1000000 chance of being right is just ridiculous.
The world didn't end: obviously their prayers and incantations were heeded. The previous successful rescues are all the more reason to support the DOOM mongers this time around.
comment by John_Maxwell (John_Maxwell_IV) · 2010-08-19T23:50:52.691Z · LW(p) · GW(p)
FWIW, as an entrepreneur type I consider one of my top 3 key advantages the fact that I would actually appreciate it greatly if someone explained in detail why I was wasting my time with my current project. Thinking about this motivates me significantly because I haven't met any other entrepreneur types who I'd guess this is also true for.
Replies from: Jordan, Jonathan_Graehl↑ comment by Jordan · 2010-08-20T04:47:53.224Z · LW(p) · GW(p)
Semi related:
I keep a big list of ideas I'd like to implement. (Start up ideas, video games ideas, math research topics.. the three things that consume me =)
Quite often I'll find out someone is working on one of these ideas, and my immediate reaction is... relief. Relief, because I found out early enough not to waste my time. But, more than that, I look at my list of ideas like an orphanage: I'm always happy when one of them finds a loving parent =p
Out of curiosity, what do you consider your other two key advantages?
Replies from: John_Maxwell_IV↑ comment by John_Maxwell (John_Maxwell_IV) · 2010-08-20T09:00:07.348Z · LW(p) · GW(p)
I didn't actually think of 3 key advantages, just figured that would be one of the top three. Probably if I was to list others, they would be willingness to trawl through a lot of ideas before finding one and implementing it, never giving up unless it really is the rational thing to do (the flip side of the original advantage), and coding ability. (Although this guy still freaks me out: http://weblog.markbao.com/2008/how-i-built-a-webapp-in-18-hours-for-699/)
Quite often I'll find out someone is working on one of these ideas, and my immediate reaction is... relief. Relief, because I found out early enough not to waste my time. But, more than that, I look at my list of ideas like an orphanage: I'm always happy when one of them finds a loving parent =p
I think people often suck at following through.
Replies from: Jordan↑ comment by Jordan · 2010-08-20T19:13:18.419Z · LW(p) · GW(p)
A kid genius entrepreneur.. awesome. You see kid genius mathematicians, chess players, musicians, etc... but an entrepreneur, that's really different. The subject matter forces him to diversify, rather than focus in on a single skill. I'm a little inspired.
I think people often suck at following through.
Agreed. Sometimes I see someone working on an idea I had and become even more motivated to work on it.
Replies from: wedrifid↑ comment by wedrifid · 2010-08-21T08:55:58.540Z · LW(p) · GW(p)
A kid genius entrepreneur.. awesome. You see kid genius mathematicians, chess players, musicians, etc... but an entrepreneur, that's really different.
I recall Tim Ferris relaying a tale of a young (~14) Olympian (Skier) who founded a remarkably successful business in order to support his international sport habit.
↑ comment by Jonathan_Graehl · 2010-08-20T01:03:52.126Z · LW(p) · GW(p)
Great stuff - I like to think I'm like this also. And you're right that it's rare (I'm not totally convinced I have it).
comment by simplicio · 2010-08-21T05:01:00.033Z · LW(p) · GW(p)
The real bone of contention here seems to be the long chain of inference leading from common scientific/philosophical knowledge to the conclusion that uFAI is a serious existential risk. Any particular personal characteristics of EY would seem irrelevant till we have an opinion on that set of claims.
If EY were working on preventing asteroid impacts with earth, and he were the main driving force behind that effort, he could say "I'm trying to save the world" and nobody would look at him askance. That's because asteroid impacts have definitely caused mass extinctions before, so nobody can challenge the very root of his claim.
The FAI problem, on the other hand, is at the top of a large house of inferential cards, so that Eliezer is saving the world GIVEN that W, X, Y and Z are true.
My bottom line: what we should be discussing is simply "Are W, X, Y and Z true?" Once we have a good idea about how strong that house of cards is, it will be obvious whether Eliezer is in a "permissible" epistemic state, or whatever.
Maybe people who know about these questions should consider a series of posts detailing all the separate issues leading to FAI. As far as I can tell from my not-extremely-tech-savvy vantage point, the weakest pillar in that house is the question of whether strong AI is feasible (note I said "feasible," not "possible").
Replies from: Simulation_Brain, jimrandomh, wedrifid↑ comment by Simulation_Brain · 2010-08-23T04:55:28.342Z · LW(p) · GW(p)
Upvoted; the issue of FAI itself is more interesting than whether Eliezer is making an ass of himself and thereby the SIAI message (probably a bit; claiming you're smart isn't really smart, but then he's also doing a pretty good job as publicist).
One form of productive self-doubt is to have the LW community critically examine Eliezer's central claims. Two of my attempted simplifications of those claims are posted here and here on related threads.
Those posts don't really address whether strong AI feasible; I think most AI researchers agree that it will become so, but disagree on the timeline. I believe it's crucial but rarely recognized that the timeline really depends on how many resources are devoted to it. Those appear to be steadily increasing, so it might not be that long.
↑ comment by jimrandomh · 2010-08-21T16:48:54.296Z · LW(p) · GW(p)
My bottom line: what we should be discussing is simply "Are W, X, Y and Z true?" Once we have a good idea about how strong that house of cards is, ...
You shouldn't deny knowledge of how strong claims are, and refer to those claims as "a house of cards" in the same sentence. Those two claims are mutually exclusive, and putting them close together like this set off my propagandometer.
↑ comment by wedrifid · 2010-08-21T08:42:34.364Z · LW(p) · GW(p)
The real bone of contention here seems to be the long chain of inference leading from common scientific/philosophical knowledge to the conclusion that FAI is a serious existential risk.
I am assuming you meant uFAI or AGI instead of FAI.
The FAI problem, on the other hand, is at the top of a large house of inferential cards, so that Eliezer is saving the world GIVEN that W, X, Y and Z are true.
For my part the conclusion you mention seems to be the easy part. I consider that an answered question. The 'Eliezer is saving the world' part is far more difficult for me to answer due to the social and political intricacies that must be accounted for.
Replies from: Unknownscomment by prase · 2010-08-20T10:55:08.214Z · LW(p) · GW(p)
An interesting post, well written, upvoted. Mere existence of such posts here constitutes a proof that LW is still far from Objectivism, not only because Eliezer is way more rational (and compassionate) than Ayn Rand, but mainly because the other people here are aware of dangers of cultism.
However, I am not sure whether the right way to prevent cultish behaviour (whether the risk is real or not) is to issue warning like this to the leader (or any sort of warning, perhaps). The dangers of cultism emerge from simply having a leader; whatever the level of personal rationality, being a single extraordinarily revered person in any group for any longer time probably harm's one's judgement, and the overall atmosphere of reverence is unhealthy for the group. Maybe more generally, the problem not necessarily depends on existence of a leader: if a group is too devoted to some single idea, it faces lots of dangers, the gravest thereof perhaps be separation from reality. Especially if the idea lives in an environment where relevant information is not abundant.
Therefore, I would prefer to see the community concentrate on a broader class of topics, and to continue in the tradition of disseminating rationality started on OB. Mitigating existential risk is a serious business indeed, and it has to be discussed appropriately, but we shouldn't lose perspective and become too fanatic about the issue. There were many statements written on LW in recent months or years, many of them not by EY, declaring absolute preference of existential risk mitigation above everything else; those statements I find unsettling.
Final nitpick: Gandhi is misspelled in the OP.
Replies from: ciphergoth, CarlShulman, multifoliaterose↑ comment by Paul Crowley (ciphergoth) · 2010-08-20T11:23:24.907Z · LW(p) · GW(p)
There were many statements written on LW in recent months or years, many of them not by EY, declaring absolute preference of existential risk mitigation above everything else; those statements I find unsettling.
The case for devoting all of your altruistic efforts to a single maximally efficient cause seems strong to me, as does the case that existential risk mitigation is that maximally efficient cause. I take it you're familiar with that case (though see eg "Astronomical Waste" if not) so I won't set it all out again here. If you think I'm mistaken, actual counter-arguments would be more useful than emotional reactions.
Replies from: prase↑ comment by prase · 2010-08-20T11:55:52.351Z · LW(p) · GW(p)
I don't object to devoting (almost) all efforts to a single cause generally. I do, however, object to such devotion in case of FAI and the Singularity.
If a person devotes all his efforts to a single cause, his subjective feeling of importance of the cause will probably increase and most people will subsequently overestimate how important the cause is. This danger can be faced by carefully comparing the results of one's deeds with the results of other people's efforts, using a set of selected objective criteria, or measure it using some scale ideally fixed at the beginning, to protect oneself from moving the goalposts.
The problem is, if the cause is put so far in the future and based so much on speculations, there is no fixed point to look at when countering one's own biases, and the risk of a gross overestimation of one's agenda becomes huge. So the reason why I dislike the mentioned suggestions (and I am speaking, for example, about the idea that it is a strict moral duty for everybody who can to support the FAI research as much as they can, which were implicitly present at least in the discussions about the forbidden topic) is not that I reject single-cause devotion in principle (although I like to be wary about it in most situations), but that I assign too low probability to the correctness of the underlying ideas. The whole business is based on future predictions of several tens or possibly hunderts years in advance, which is historically a very unsuccessful discipline. And I can't help but include it in that reference class.
Simultaneously, I don't accept the argument of very huge utility difference between possible outcomes, which should justify one's involvement even if the probability of success (or even probability that the effort has sense) is extremely low. Pascal-wageresque reasoning is unreliable, even if formalised, because it needs careful and precise estimation of probabilities close to 1 or 0, which humans are provably bad at.
Replies from: Wei_Dai, ciphergoth↑ comment by Wei Dai (Wei_Dai) · 2010-08-20T12:15:16.472Z · LW(p) · GW(p)
Pascal-wageresque reasoning is unreliable, even if formalised, because it needs careful and precise estimation of probabilities close to 1 or 0, which humans are provably bad at.
Assuming you're right, why doesn't rejection of Pascal-like wagers also require careful and precise estimation of probabilities close to 1 or 0?
Replies from: prase↑ comment by prase · 2010-08-20T12:21:01.496Z · LW(p) · GW(p)
I use a heuristic which tells me to ignore Pascal-like wagers and to do whatever I would do if I haven't learned about the wager (in first approximation). I don't behave like an utilitarian in this case, so I don't need to estimate the probabilities and utilities. (I think if I did, my decision would be fairly random, since the utilities and probabilities included would be almost certainly determined mostly by the anchoring effect).
Replies from: Perplexed↑ comment by Perplexed · 2010-08-20T15:22:31.882Z · LW(p) · GW(p)
I use a heuristic which tells me to ignore Pascal-like wagers
I am not sure exactly what using this heuristic entails. I certainly understand the motivation behind the heuristic:
- when you multiply an astronomical utility (disutility) by a miniscule probability, you may get an ordinary-sized utility (disutility), apparently suitable for comparison with other ordinary-sized utilities. Don't trust the results of this calculation! You have almost certainly made an error in estimating the probability, or the utility, or both.
But how do you turn that (quite rational IMO) lack of trust into an action principle? I can imagine 4 possible precepts:
- Don't buy lottery tickets
- Don't buy insurance
- Don't sell insurance
- Don't sell back lottery tickets you already own.
Is it rationally consistent to follow all 4 precepts, or is there an inconsistency?
Replies from: timtyler, ShardPhoenix, prase↑ comment by timtyler · 2010-08-20T23:46:07.603Z · LW(p) · GW(p)
Another red flag is when someone else helpfully does the calculation for you - and then expects you to update on the results. Looking at the long history of Pascal-like wagers, that is pretty likely to be an attempt at manipulation.
Replies from: timtyler↑ comment by timtyler · 2010-08-21T18:52:10.784Z · LW(p) · GW(p)
"I believe SIAI’s probability of success is lower than what we can reasonably conceptualize; this does not rule it out as a good investment (since the hoped-for benefit is so large), but neither does the math support it as an investment (donating simply because the hoped-for benefit multiplied by the smallest conceivable probability is large would, in my view, be a form of falling prey to “Pascal’s Mugging”."
↑ comment by ShardPhoenix · 2010-08-21T01:54:16.257Z · LW(p) · GW(p)
What do those examples have to do with anything? In those cases we actually know the probabilities so they're not Pascal's-Wager-like scenarios.
Replies from: Perplexed↑ comment by Perplexed · 2010-08-21T02:56:01.990Z · LW(p) · GW(p)
we actually know the probabilities
So, what is the probability that my house will burn? It may depend on whether I start smoking again. I hope the probability of both is low, but I don't know what it is.
I'm not sure exactly what the definition of Pascal's-Wager-like should be. Is there a definition I should read? Should we ask Prase what he meant? I understood the term to mean anything involving small estimated probabilities and large estimated utilities.
Replies from: ShardPhoenix, timtyler↑ comment by ShardPhoenix · 2010-08-21T13:11:19.130Z · LW(p) · GW(p)
We know the probability to a reasonable level of accuracy - eg consider acturial tables. This is different from things like Pascal's wager where the actual probability may vary by many orders of magnitude from our best estimate.
Replies from: rhollerith_dot_com↑ comment by RHollerith (rhollerith_dot_com) · 2010-08-21T13:28:57.792Z · LW(p) · GW(p)
This is different from things like Pascal's wager where the actual probability may vary by many orders of magnitude from our best estimate.
According to the Bayesians, our best estimate is the actual probability. (According to the frequentists, the probabilities in Pascal's wager are undefined.)
What parent means by "We know the probability to a reasonable level of accuracy - eg consider acturial tables" is that it is possible for a human to give a probability without having to do or estimate a very hairy computation to compute a prior probability (the "starting probability" before any hard evidence is taken into account). ADDED. In other words, it should have been a statement about the difficulty of the computation of the probability, not a statement about the existence of the probability in principle.
Replies from: prase↑ comment by prase · 2010-08-23T14:14:16.152Z · LW(p) · GW(p)
I indeed am motivated by reasons you gave, so lotteries aren't concern for this heuristics, since the probability is known. In fact, I have never thought about lotteries this way, probably because I know the probabilities. The value estimate is a bit less sure (to resonably buy a lottery, it would also need a convex utility curve, which I probably haven't), but the lotteries deal with money, which make pretty good first approximation for value. Insurances are more or less similar, and not all of them include probabilities too low or values too high to fall into the Pascal-wager category.
Actually, I do buy some most common insurances, although I avoid buying insurances against improbable risks (meteorite fall etc.). I don't buy lotteries.
The more interesting aspect of your question is the status-quo conserving potential inconsistency you have pointed out. I would probably consider real Pascal-wagerish assets to be of no value and sell them if I needed the money. This isn't exactly consistent with the "do nothing" strategy I have outlined, so I have to think about it a while to find out whether the potential inconsistencies are not too horrible.
↑ comment by Paul Crowley (ciphergoth) · 2010-08-20T12:29:39.814Z · LW(p) · GW(p)
Which of the axioms of the Von Neumann–Morgenstern utility theorem do you reject?
Replies from: Wei_Dai, prase↑ comment by Wei Dai (Wei_Dai) · 2010-08-20T12:44:46.216Z · LW(p) · GW(p)
I think the theorem implicitly assumes logical omniscience, and using heuristics instead of doing explicit expected utility calculations should make sense in at least some types of situations for us. The question is whether it makes sense in this one.
I think this is actually an interesting question. Is there an argument showing that we can do better than prase's heuristic of rejecting all Pascal-like wagers, given human limitations?
↑ comment by prase · 2010-08-20T12:48:01.551Z · LW(p) · GW(p)
If I had to describe my actual choices, I don't know. No one necessarily, any of the axioms possibly. My inner decision algorithm is probably inconsistent in different ways, I don't believe for example that my choices always satisfy transitivity.
What I wanted to say is that although I know that my decisions are somewhat irrational and thus sub-optimal, in some situations, like Pascal wagers, I don't find consciously creating an utility function and to calculate the right decision to be an attractive solution. It would help me to be marginally more rational (as given by the VNM definition), but I am convinced that the resulting choices would be fairly arbitrary and probably will not reflect my actual preferences. In other words, I can't reach some of my preferences by introspection, and think that an actual attempt to reconstruct an utility function would sometimes do worse than simple, although inconsistent heuristic.
↑ comment by CarlShulman · 2010-08-20T12:37:34.917Z · LW(p) · GW(p)
Therefore, I would prefer to see the community concentrate on a broader class of topics, and to continue in the tradition of disseminating rationality started on OB.
The best way to advance this goal being is probably to write an interesting top-level post.
Replies from: prase↑ comment by multifoliaterose · 2010-08-20T11:15:45.969Z · LW(p) · GW(p)
Thanks for correcting the misspelling!
Totally agree about LW vs. Objectivism.
comment by Wei Dai (Wei_Dai) · 2010-08-20T07:52:23.878Z · LW(p) · GW(p)
I find it ironic that multifoliaterose said
I personally think that the best way to face the present situation is to gather more information about all existential risks rather than focusing on one particular existential risk
and then the next post, instead of delineating what he found out about other existential risks (or perhaps how we should go about doing that), is about how to save Eliezer.
comment by Friendly-HI · 2013-01-29T00:41:42.463Z · LW(p) · GW(p)
As of yet Eliezer's importance is just a stochastic variable yet to be realized, for all I know he could be killed in a car accident tomorrow or simply fail at his task of "saving the world" in numerous ways.
Up until now Vasili Arkhipov, Stanislav Petrov and a few other people I do not know the names of (including our earliest ancestors who managed to avoid being killed during their emigration out of Africa) trump Eliezer by a tiny margin of actually saving humanity -or at least civilization.
All that being said Eliezer is still pretty awesome by my standards. And he writes good fanfiction, too.
comment by JamesAndrix · 2010-08-21T05:11:28.311Z · LW(p) · GW(p)
Who else is nearly as good or better at Friendly AI development than Eliezer Yudkowsky?
I mean besides me, obviously.
comment by Jonathan_Graehl · 2010-08-20T01:23:48.629Z · LW(p) · GW(p)
it's very important that those of us who aspire to epistemic rationality incorporate a significant element of "I'm the sort of person who engages in self-doubt because it's the right thing to do" into our self-image
I think most of us do. Your argument for this is compelling. However, I think Eliezer was just claiming that it's possible to overdo it - at least, that's the defensible core of his insight.
I've wondered if I'm obsessed with Eliezer's writings, and whether I esteem him too highly. Answers: no, and no.
Anything that has even a slight systematic negative impact on existential risk is a big deal.
Probably true. But it's sometimes easy to be on the wrong side of an argument over small differences (of course sometimes you can be certain). I guess such "there's no harm" statements (which I've also made) are biased by a desire to be conciliatory. I don't trust people to behave well when they're annoyed at each other, so I sometimes wish they would minimize the stakes.
Eliezer appears to be deviating so sharply from leading a genuinely utilitarian lifestyle
I doubt I know any utilitarians.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-20T03:53:54.801Z · LW(p) · GW(p)
Thanks for correcting my typos.
Replies from: Jonathan_Graehl↑ comment by Jonathan_Graehl · 2010-08-20T04:08:55.939Z · LW(p) · GW(p)
You're welcome - I've redacted my comment so it no longer mentions them.
comment by jimrandomh · 2010-08-20T12:20:14.996Z · LW(p) · GW(p)
Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws. This is akin to the principle of not taking pride in confessing your ignorance; for if your ignorance is a source of pride to you, you may become loathe to relinquish your ignorance when evidence comes knocking. Likewise with our flaws - we should not gloat over how self-aware we are for confessing them; the occasion for rejoicing is when we have a little less to confess.
There's something to what Eliezer is saying here: when people are too strongly committed to the idea that humans are fallible this can become a self-fulfilling prophecy where humans give up on trying to improve things and as a consequence remain fallible when they could have improved.
I actually read this as a literal, technical statement about when to let the reward modules of our minds trigger, and not a statement about whether low or high confidence is desirable. Finding a flaw in oneself is only valuable if it's followed by further investigation into details and fixes, and, as a purely practical matter, that investigation is more likely to happen if you feel good about having found a fix, than if you feel good about having found a flaw.
comment by rwallace · 2010-08-20T01:14:30.061Z · LW(p) · GW(p)
This post is a pretty accurate description of me a few years ago, when I was a Singularitarian. The largest attraction of the belief system, to me, was that it implied as an AI researcher I was not just a hero, but a superhero, potentially capable of almost single-handedly saving the world. (And yes, I loved those video games too.)
Replies from: cousin_it↑ comment by cousin_it · 2010-08-20T07:50:44.050Z · LW(p) · GW(p)
What's your current position?
Replies from: rwallace↑ comment by rwallace · 2010-08-20T09:04:57.716Z · LW(p) · GW(p)
Appealing though the belief is, the Singularity unfortunately isn't real. Nothing is going to come along and solve our problems for us, and AI is not going to be a magical exception to the rule that developing technology is hard.
Replies from: Emile, katydee, cousin_it, Furcas↑ comment by Emile · 2010-08-20T10:07:11.055Z · LW(p) · GW(p)
Nothing is going to come along and solve our problems for us, and AI is not going to be a magical exception to the rule that developing technology is hard.
Do you think many people here think that "something is going to come along and solve our problem for us", or that "developing AI is easy"?
Replies from: rwallace↑ comment by rwallace · 2010-08-20T11:07:25.324Z · LW(p) · GW(p)
Yes. In particular, the SIAI is explicitly founded on the beliefs that
Superintelligent AI will solve all our problems.
Creating same is (unlike other, much less significant technological developments) so easy that it can be done by a single team within our lifetimes.
↑ comment by Aleksei_Riikonen · 2010-08-20T11:59:04.021Z · LW(p) · GW(p)
The following summary of SIAI's position says otherwise:
http://singinst.org/riskintro/index.html
It seems you're confusing what you personally thought earlier with what SIAI currently thinks.
(Though, technically you're partly right that what SIAI folks thought when said institution was founded is closer to what you say than their current position. But it's not particularly interesting what they thought 10 years ago if they've revised their position to be much better since then.)
Replies from: rwallace↑ comment by Emile · 2010-08-20T12:24:24.538Z · LW(p) · GW(p)
Do you think many people here think that "something is going to come along and solve our problem for us", or that "developing AI is easy"?
Yes. In particular, the SIAI is explicitly founded on the beliefs that [...]
That isn't really evidence that people here (currently) believe either of those. You're claiming people here believe things even though they go against some of Eliezer's writing (and I don't remember any cries of "No, Eliezer, you're wrong! Creating AI is easy!", but I might be mistaken), and even though quite a few commenters are telling you nobody here believes that.
Replies from: whpearson↑ comment by whpearson · 2010-08-20T12:44:27.267Z · LW(p) · GW(p)
It depends what you mean by easy and hard. From previous conversations I expect Mr Wallace is thinking something easy is doable by means of a small group over 20-30 years and hard is a couple of generations of the whole of civilizations work.
Replies from: rwallace↑ comment by katydee · 2010-08-20T10:28:13.914Z · LW(p) · GW(p)
I'm wondering where people said AI development was going to be easy.
Replies from: wedrifid, NancyLebovitz↑ comment by NancyLebovitz · 2010-08-20T12:09:44.417Z · LW(p) · GW(p)
And I'm wondering where it was said that superintelligent AI will solve all our problems.
Replies from: rwallace↑ comment by rwallace · 2010-08-21T00:06:41.438Z · LW(p) · GW(p)
The original idea of the SIAI was that when they (or someone else) implement superintelligent AI, it takes over the world and implements CEV (or turns everyone into paperclips or whatever). Is their current position on that also more moderate?
Replies from: ata↑ comment by ata · 2010-08-21T00:12:11.149Z · LW(p) · GW(p)
Implementing CEV does not necessarily mean taking over the world or solving all our problems. Indeed, the whole point of CEV is that we don't know what it'll turn out to be; if we could know what its output would look like, we wouldn't need a superintelligence to figure it out for us. There was (and still is, as far as I can tell) a strong expectation that CEV will significantly remake the world, but that is a conjectured probable implication (and debating it doesn't matter too much anyway), not a definitional quality or a deductive implication.
Replies from: Vladimir_Nesov↑ comment by Vladimir_Nesov · 2010-08-21T00:32:50.275Z · LW(p) · GW(p)
That FAI will significantly change things is a pretty conclusive antiprediction. Status quo hath no moral power.
Replies from: rwallace↑ comment by cousin_it · 2010-08-20T09:14:20.300Z · LW(p) · GW(p)
What made you switch to this position?
Replies from: rwallace↑ comment by rwallace · 2010-08-20T09:24:43.831Z · LW(p) · GW(p)
There was a crack (the fundamental incoherence of the Singularitian belief system) that I was having to diligently cover with wallpaper (faith in the absence of evidence) so that I wouldn't notice it. Eventually I reached a state of mind in which I could allow myself to see the crack.
Replies from: orthonormal↑ comment by orthonormal · 2010-08-21T07:48:34.307Z · LW(p) · GW(p)
Um, that's not an informative answer to anyone but yourself. Is there any specific piece of evidence that became known to you, or became more significant to you, at that time?
Replies from: timtyler↑ comment by timtyler · 2010-08-21T08:29:54.246Z · LW(p) · GW(p)
Did you read the bit about the Tegmark multiverse and anthropic reasoning?
Replies from: orthonormal↑ comment by orthonormal · 2010-08-21T17:22:11.433Z · LW(p) · GW(p)
I'm embarrassed to admit that I was reading while tired, and didn't even notice there was a link in the comment. However, even after reading that, rwallace's epiphany remains opaque to the rest of us. He explains that a change of attitude was necessary in order to accept whatever evidence of AI's difficulty he already had, but he explains nothing about what that evidence might be. It's still uninformative to anyone but himself.
↑ comment by Furcas · 2010-08-20T10:01:53.698Z · LW(p) · GW(p)
My co-worker, who's much smarter than I am, sometimes solves my programming problems for me, and problems that are very hard for me are very easy for him. Is he a magical exception to your rule? If not, what about someone who's even smarter than he is?
Replies from: rwallace↑ comment by rwallace · 2010-08-20T10:06:09.661Z · LW(p) · GW(p)
Your coworker is human, yes? That's what I meant by "us". If God or the Galactic Mothership or a team of invisible elves or a superintelligent AI somebody created in their basement came along and solved your programming problems for you, those would be magical exceptions to my rule.
Replies from: Furcas↑ comment by Furcas · 2010-08-20T10:27:53.174Z · LW(p) · GW(p)
You don't seem to want to state your beliefs clearly, and I don't have the patience to write more than this one post encouraging you to do so.
Do you believe that the difficulty of developing technology depends on the mind trying to develop it only when that mind happens to be human? Or that nothing can be smarter than the smartest human? Or what?
Replies from: rwallace↑ comment by rwallace · 2010-08-20T11:14:42.068Z · LW(p) · GW(p)
I'm being as clear as I can without writing an essay in every comment. But I'll put it this way:
Nothing is currently smarter than the smartest human.
This is not going to change anytime soon.
While AI does have the potential to produce better tools than we have now, there are still going to be enormous gaps in the abilities of those tools. For example, suppose you had an AI that was great at writing code from formal specifications, but didn't know enough about the real world to know what code to write. Then you would have a tool that you might find useful, but that you could not sit back and let solve your problems for you. At the end of the day, the responsibility for solving your problems would still be yours. This is very different from the Singularitarian vision where creating a superintelligent AI is the last job we need to do.
↑ comment by Risto_Saarelma · 2010-08-20T12:54:08.231Z · LW(p) · GW(p)
Maybe you could write a full post about your views. I'd very much like to read good criticism of singularitarism, but so far your objections aren't very strong.
The core assumptions in this comment, for example, seem to be not really visible. I'm guessing the idea is something like it'd be really, really hard to do an AI that can do everything a human does, and trying to leave real-world problem-solving to subhuman AIs won't work.
But no-one's talking about going after problems in the physical world with a glorified optimizing compiler, so why do you bring up this as the main example? The starting for a lot of current AGI thinking, as far as I've understood, is to make an AI with the ability to learn and some means to interact with the world. This AI is then expected to learn to act in the world like humans learn when they grow from newborns to adults.
So is there some kind of basic difference in understanding here, when I'm thinking of AIs as learning semi-autonomous agents, and you're thinking them as, I guess, some kind of pre-programmed unchanging procedures for doing specific things?
Replies from: rwallace↑ comment by rwallace · 2010-08-20T13:18:30.614Z · LW(p) · GW(p)
Yes, basically my claim is that an AI of the sort you're talking about is a job for the world over timescales of generations, not for a single team over timescales of years or decades; it's hard to prove a negative, and you are right that the comments I've been making here don't -- can't -- strongly justify that claim. I'll think about whether I can put together my reasoning into a full post.
Replies from: Jonathan_Graehl, Risto_Saarelma↑ comment by Jonathan_Graehl · 2010-08-20T22:36:09.976Z · LW(p) · GW(p)
Your position is one that most people assign some probability mass to. However, I get the impression that you're extremely (over)confident in it. So I look forward to hearing your case.
↑ comment by Risto_Saarelma · 2010-08-20T14:52:55.045Z · LW(p) · GW(p)
Ok, thanks. As far as I see, this is the most important core objection then.
There's actually a second big unknown too before getting into full singularitarism, whether this kind of human-equivalent AI could easily boost itself to strongly superhuman levels with any sort of ease.
But the question of just how difficult it is to build the learning baby AI is really important, and I don't have any good ideas on how to estimate it except from stuff that can be figured out from biology. The human genome gives us the number of bits that keeps passing through evolution and the general initial complexity for humans, but it's big enough that without a very good design sense trying to navigate that kind of design space would indeed take generations. Brains and learning have been evolving for a very long time, indicating that the machinery may be very elaborate to get right. Compared to this, symbolic language seems to have popped up very quickly in evolution, which gives reason to believe that once there's a robust nonverbal cognitive architecture, adding symbolic cognition capabilities isn't nearly as hard as getting the basic architecture together.
Replies from: Strange7↑ comment by Strange7 · 2010-08-20T17:00:30.003Z · LW(p) · GW(p)
It might also be that the selective pressure in favor of increased intelligence increased suddenly, most likely as a result of competition among humans.
Once a singleton AI becomes marginally smarter than the smartest human, how are we to distinguish between further advances in intelligence as opposed to, say, an increase in it's ability to impress us with high-tech parlor tricks? Would there be competition between AIs, and if so, over what?
↑ comment by Mitchell_Porter · 2010-08-20T13:07:00.222Z · LW(p) · GW(p)
but didn't know enough about the real world to know what code to write.
This requires two things: knowing what you want, and learning about the world.
I don't see the fundamental problem in getting an AI to learn about the world. The informal human epistemic process has been analyzed into components, and these have been formalized and implemented in ways far more powerful than an unaided human can manage. It's a lot of work to put it all together in a self-consistent package, and to give it enough self-knowledge and world-knowledge to set it in motion, and it would require a lot of computing power. But I don't see any fundamental difficulty.
What the AI wants is utterly contingent on initial conditions. But an AI that can represent the world and learn about it, can also represent just about any goal you care to give it, so there's no extra problem to solve here. (Except for Friendliness. But that is the specific problem of identifying a desirable goal, not the general problem of implementing goal-directed behavior.)
Just reviewing this basic argument reinforces the prior impression that we are already drifting towards transhuman AI and that there's no fundamental barrier in the way. We already know enough for hard work alone to get us there - I mean the hard work of tens of thousands of researchers in many fields, not one person or one group making a super-duper effort. The other factor which seals our fate is distributed computing. Even if Moore's law breaks down, computers can be networked, and there are lots of computers.
So, we are going to face something smarter than human, which means something that can outwit us, which means something that should win if its goals are ever in conflict with ours. And there is no law of nature to guarantee that its goals will be humanly benevolent. On the contrary, it seems like anything might serve as the goal of an AI, just as "any" numerical expression might be fed to a calculator for evaluation.
What we don't know is how likely it is that the first transhuman AI's goals will be bad for us. A transhuman AI may require something like the resources of a large contemporary server farm to operate, in which case it's not going to happen by accident. There is some possibility that the inherent difficulty of getting there renders it more likely that by the time you get to transhuman AI, the people working on it have thought ahead to the time when the AI is autonomous and in fact beyond stopping, and realized that it had better have "ethics". But that just means that by the time that the discipline of AI is approaching the transhuman threshold, people are probably becoming aware of what we have come to call the problem of friendliness. It doesn't mean that the problem is sure to have been solved by the time the point of no return is reached.
All in all, therefore, I conclude (1) the Singularity concept makes sense (2) it is a matter for concern in the present and near future, not the far future (3) figuring out the appropriate initial conditions for an ethical AI is the key problem to solve (4) SIAI is historically important as the first serious attempt to solve this problem.
comment by Eneasz · 2010-08-25T18:06:10.582Z · LW(p) · GW(p)
As far as I can tell, Eliezer does have confidence in the idea that he is (at least nearly) the most important person in human history. Eliezer's silence only serves to further confirm my earlier impressions
I suppose you also believe that Obama must prove he's not a muslim? And must do so again every time someone asserts that he is?
Let me say that Eliezer may have already done more to save the world than most people in history. This is going on the assumption that FAI is a serious existential risk. Even if he is doing it wrong and his work will never directly contribute to FAI in any way, his efforts at popularizing the existence of this threat have vastly increased the pool of people who know of it and want to help in some way.
His skill at explanation and inspiration have brought more attention to this issue than any other single person I know of. The fact that he also has the intellect to work directly on the problem is simply an added bonus. And I strongly doubt that it's driven away anyone who would have otherwise helped.
You said you had delusions of messianic grandeur in high school, but you're better now. But then you post an exceptionally well done personal take-down of someone who YOU believe is too self-confident and who (more importantly) has convinced others that his confidence is justified. I think your delusions of messiah-hood are still present, perhaps unacknowledged, and you are suffering from envy of someone you view as "a more successful messiah".
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-25T21:37:16.251Z · LW(p) · GW(p)
I suppose you also believe that Obama must prove he's not a muslim? And must do so again every time someone asserts that he is?
I don't see the situation that you cite as comparable. Obama has stated that he's a Christian, and this seriously calls into question the idea that he's a Muslim.
Has Eliezer ever said something which calls my interpretation of the situation into question? If so I'll gladly link a reference to it in my top level post.
(As an aside, I agree with Colin Powell that whether or not Obama is a Muslim has no bearing on whether he's fit to be president.)
Let me say that Eliezer may have already done more to save the world than most people in history. This is going on the assumption that FAI is a serious existential risk. Even if he is doing it wrong and his work will never directly contribute to FAI in any way, his efforts at popularizing the existence of this threat have vastly increased the pool of people who know of it and want to help in some way.
His skill at explanation and inspiration have brought more attention to this issue than any other single person I know of. The fact that he also has the intellect to work directly on the problem is simply an added bonus. And I strongly doubt that it's driven away anyone who would have otherwise helped.
I definitely agree that some of what Eliezer has done has reduced existential risk. As I've said elsewhere, I'm grateful to Eliezer for inspiring me personally to think more about existential risk.
However, as I've said, in my present epistemological state I believe that he's also had (needless) negative effects on existential risk on account of making strong claims with insufficient evidence. See especially my responses to komponisto's comment. I may be wrong about this.
In any case, I would again emphasize that my most recent posts should not be interpreted as personal attacks on Eliezer. I'm happy to support Eliezer to the extent that he does things that I understand to lower existential risk.
You said you had delusions of messianic grandeur in high school, but you're better now. But then you post an exceptionally well done personal take-down of someone who YOU believe is too self-confident and who (more importantly) has convinced others that his confidence is justified. I think your delusions of messiah-hood are still present, perhaps unacknowledged, and you are suffering from envy of someone you view as "a more successful messiah".
My conscious motivation making my most recent string of posts is given in my Transparency and Accountability posting. I have no conscious awareness of having a motivation of the type that you describe.
Of course, I may be deluded about this (just as all humans may be deluded about possessing any given belief). In line with my top level posting, I'm interested in seriously considering the possibility that my unconscious motivations are working against my conscious goals.
However, I see your own impression as very poor evidence that I may be deluded on this particular point in light of your expressed preference for donating to Eliezer and SIAI even if doing so is not socially optimal:
And my priests are Eliezer Yudkowsky and the SIAI fellows. I don't believe they leach off of me, I feel they earn every bit of respect and funding they get. But that's besides the point. The point is that even if the funds I gave were spent sub-optimally, I would STILL give them this money, simply because I want other people to see that MY priests are better taken care of than THEIR priests.
I don't judge you for having this motivation (we're all only human). But the fact that you seem interested in promoting Eliezer and SIAI independently of whether doing so benefits broader society has led me to greatly discount your claims and suggestions which relate to Eliezer and SIAI.
Replies from: Eneasz↑ comment by Eneasz · 2010-08-26T00:08:58.687Z · LW(p) · GW(p)
(As an aside, I agree with Colin Powell that whether or not Obama is a Muslim has no bearing on whether he's fit to be president.)
Does whether Eliezer is over-confident or not have any bearing on whether he's fit to work on FAI?
I believe that he's also had (needless) negative effects on existential risk on account of making strong claims with insufficient evidence. See especially my responses to komponisto's comment. I may be wrong about this.
From the comment:
My claim is that on average Eliezer's outlandish claims repel people from thinking about existential risk.
The claim is not credible. I've seen a few examples given, but with no way to determine if the people "repelled" would have ever been open to mitigating existential risk in the first place. I suspect anyone who actually cares about existential risk wouldn't dismiss an idea out of hand because a well-known person working to reduce risk thinks his work is very valuable. It is unlikely to be their true rejection
In any case, I would again emphasize that my most recent posts should not be interpreted as personal attacks on Eliezer.
The latest post made this clear, and cheers for that. But the previous ones are written as attacks on Eliezer. It's hard to see a diatribe against someone describing them as a cult leader who's increasing existential risk and would do best to shut up and not interpret it as a personal attack.
But the fact that you seem interested in promoting Eliezer and SIAI independently of whether doing so benefits broader society has led me to greatly discount your claims and suggestions which relate to Eliezer and SIAI.
Fair enough, can't blame you for that. I'm happy with my enthusiasm.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-08-26T00:42:02.907Z · LW(p) · GW(p)
Does whether Eliezer is over-confident or not have any bearing on whether he's fit to work on FAI?
Oh, I don't think so, see my response to Eliezer here.
The claim is not credible. I've seen a few examples given, but with no way to determine if the people "repelled" would have ever been open to mitigating existential risk in the first place. I suspect anyone who actually cares about existential risk wouldn't dismiss an idea out of hand because a well-known person working to reduce risk thinks his work is very valuable. It is unlikely to be their true rejection
Yes, so here it seems like there's enough ambiguity as to how the publicly available data is properly interpreted so that we may have a legitimate difference of opinion on account of having had different experiences. As Scott Aaronson mentioned in the blogging heads conversation, humans have their information stored in a form (largely subconscious) such that it's not readily exchanged.
All I would add to what I've said is that if you haven't already done so, see the responses to michaelkeenan's comment here (in particular those by myself, bentarm and wedrifid).
If you remain unconvinced, we can agree to disagree without hard feelings :-)
comment by Jordan · 2010-08-20T06:02:24.647Z · LW(p) · GW(p)
Honestly, I don't think Eliezer would look overly eccentric if it weren't for LessWrong/Overcomingbias. Comp sci is notoriously eccentric, AI research possibly more so. The stigma against Eliezer isn't from his ideas, it isn't from his self confidence, it's from his following.
Kurzweil is a more dulled case: he has good ideas, but is clearly sensational, he has a large following, but that following isn't nearly as dedicated as the one to Eliezer (not necessarily to Eliezer himself, but to his writings and the "practicing of rationality"). And the effect? I have a visceral distaste whenever I hear someone from the Kurzweil camp say something pro-singularity. It's very easy for me to imagine that if I didn't already put stock in the notion of a singularity, that hearing a Kurzweilian talk would bias me against the idea.
Nonetheless, it may very well be the case that Kurzweil has done a net good to the singularity meme (and perhaps net harm to existential risk), spreading the idea wide and far, even while generating negative responses. Is the case with Eliezer the same? I don't know. My gut says no. Taking existential risk seriously is a much harder meme to catch than believing in a dumbed down version of the singularity.
My intuition is that Eliezer by himself, although abrasive in presentation, isn't turning people off by his self confidence and grandioseness. On the contrary, I -- and I suspect many -- love to argue with intelligent people with strong beliefs. In this sense, Eliezer's self assurance is a good bait. On the other hand, when someone with inferior debating skills goes around spurting off the message of someone else, that, to me, is purely repulsive: I have no desire to talk with those people. They're the people spouting off Aether nonsense on physics forums. There's no status to be won, even on the slim chance of victory.
Finally, aside from Eliezer as himself and Eliezer through the proxy of others, there's also Eliezer as a figurehead of SIAI. Here things are different as well, and Eliezer is again no longer merely himself. He speaks for an organisation, and, culturally, we expect serious organisations to temper their outlandish claims. Take cancer research: presumably all researchers want to cure cancer. Presumably at least some of them are optimistic and believe we actually will. But we rarely hear this, and we never hear it from organizations.
I think SIAI, and Eliezer in his capacity as a figure head, probably should temper their claims as well. The idea of existential risks from AI is already pervasive. Hollywood took care of that. What remains is a battle of credibility.
(Unfortunately, I really don't know how to go about tempering claims with the previous claims already on permanent record. But maybe this isn't as important as I think it is.)
Replies from: ata↑ comment by ata · 2010-08-20T06:22:50.642Z · LW(p) · GW(p)
Honestly, I don't think Eliezer would look overly eccentric if it weren't for LessWrong/Overcomingbias. Comp sci is notoriously eccentric, AI research possibly more so. The stigma against Eliezer isn't from his ideas, it isn't from his self confidence, it's from his following.
Would you include SL4 there too? I think there were discussions there years ago (well before OB, and possibly before Kurzweil's overloaded Singularity meme complex became popular) about the perception of SIAI/Singularitarianism as a cult. (I wasn't around for any such discussions, but I've poked around in the archives from time to time. Here is one example.)
comment by Wei Dai (Wei_Dai) · 2010-08-20T01:04:16.526Z · LW(p) · GW(p)
A few unrelated points:
- I tend to agree with you on the first section, but I think I'm less confident about it than you are. :)
- What is a genuinely utilitarian lifestyle? Is there someone you can cite as living such a lifestyle?
- I'm not sure what you're talking about in the last sentence. Prevent what from happening to Eliezer? Failing to lose hope when he should? (He wrote a post about that, BTW.)
↑ comment by multifoliaterose · 2010-09-02T12:47:08.591Z · LW(p) · GW(p)
Sorry to take so long to get back to you :)
What is a genuinely utilitarian lifestyle? Is there someone you can cite as living such a lifestyle?
Obviously humans are extremely ill-suited for being utilitarians (just as humans would be extremely ill-suited for being paperclip maximizers even if they wanted to be.)
When I refer to a "genuinely utilitarian lifestyle" I mean subject to human constraints. There are some people who do this much better than others - for example, Bill Gates and Warren Buffett have done much better than most billionaires.
I think that with a better peer network, Gates and Buffett could have done still better (for example I would have liked to see them take existential risk into serious consideration with their philanthropic efforts).
A key point here is that as I've said elsewhere I don't think that leading a (relatively) utilitarian lifestyle has very much at all to do with personal sacrifice, but rather with realigning one's personal motivational structure in a way that (at least for many people) does not entail a drop in quality of life. If you haven't already done so, see my post on missed opportunities for doing well by doing good.
I'm not sure what you're talking about in the last sentence. Prevent what from happening to Eliezer? Failing to lose hope when he should? (He wrote a post about that, BTW.)
Thanks for the reference. I edited the end of my posting to clarify what I had in mind.
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2010-09-03T02:19:00.814Z · LW(p) · GW(p)
When I refer to a "genuinely utilitarian lifestyle" I mean subject to human constraints. There are some people who do this much better than others - for example, Bill Gates and Warren Buffett have done much better than most billionaires.
If that's the kind of criteria you have in mind, why did you say "Eliezer appears to be deviating so sharply from leading a genuinely utilitarian lifestyle"? It seems to me that Eliezer has also done much better than most ... (what's the right reference class here? really smart people who have been raised in a developed country?)
Which isn't to say that he couldn't do better, but your phrasing strikes me as rather unfair...
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T11:42:35.418Z · LW(p) · GW(p)
What I was getting at in my posting is that in exhibiting unwillingness to seriously consider the possibility that he's vastly overestimated his chances of building a Friendly AI it appears that Eliezer is deviating sharply from leading a utilitarian lifestyle (relative to what one can expect from humans).
I was not trying to make a general statement about Eliezer's attainment of utilitarian goals relative to other humans. I think that there's a huge amount of uncertainty on this point to such an extent that it's meaningless to try to make a precise statement.
The statement that I was driving at is a more narrow one.
I think that it would be better for Eliezer and for the world at large if Eliezer seriously considered the possibility that he's vastly overestimated his chances of building a Friendly AI. I strongly suspect that if he did this, his strategy for reducing existential risk would change for the better. If his current views turn out to be right, he can always return to them later on. I think that the expected benefits of him reevaluating his position far outweigh the expected costs.
Replies from: Mitchell_Porter, wedrifid, Wei_Dai↑ comment by Mitchell_Porter · 2010-09-03T12:01:40.069Z · LW(p) · GW(p)
I think that it would be better for Eliezer and for the world at large if Eliezer seriously considered the possibility that he's vastly overestimated his chances of building a Friendly AI. I strongly suspect that if he did this, his strategy for reducing existential risk would change for the better.
Why? What sort of improvement would you expect?
Remember that he is still the one person in the public sphere who takes the problem of Friendly AI (under any name) seriously enough to have devoted his life to it, and who actually has quasi-technical ideas regarding how to achieve it. All this despite the fact that for decades now, in fiction and nonfiction, the human race has been expressing anxiety about the possibility of superhuman AI. Who are his peers, his competitors, his predecessors? If I was writing the history of attempts to think about the problem, Chapter One would be Isaac Asimov with his laws of robotics, Chapter Two would be Eliezer Yudkowsky and the idea of Friendly AI, and everything else would be a footnote.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T12:17:22.210Z · LW(p) · GW(p)
Three points:
I think that if he had a more accurate estimation of his chances of building a Friendly AI, this would be better for public relations, for the reasons discussed in Existential Risk and Public Relations.
I think that his unreasonably estimate of his ability to build a Friendly AI has decreased his willingness to engage with the academic mainstream to an unreasonable degree. I think that his ability to do Friendly AI research would be heightened if he were more willing to engage with the academic mainstream. I think he'd be more likely to find collaborators and more likely to learn the relevant material.
I think that a more accurate assessment of the chances of him building a Friendly AI might lead him to focus on inspiring others and on existential risk reduction advocacy (things that he has demonstrated capacity to do very well) rather than Friendly AI research. I suspect that if this happened, it would maximize his chances of obstructing global catastrophic risk.
↑ comment by Mitchell_Porter · 2010-09-03T13:08:53.485Z · LW(p) · GW(p)
might lead him to focus on inspiring others and on existential risk reduction advocacy (things that he has demonstrated capacity to do very well) rather than Friendly AI research
That would absolutely be a waste. If for some reason he was only to engage in advocacy from now on, it should specifically be Friendly AI advocacy. I point again to the huge gaping absence of other people who specialize in this problem and who have worthwhile ideas. The other "existential risks" have their specialized advocates. No-one else remotely comes close to filling that role for the risks associated with superintelligence.
In other words, the important question is not, what are Eliezer's personal chances of success; the important question is, who else is offering competent leadership on this issue? Like wedrifid, I don't even recall hearing a guess from Eliezer about what he thinks the odds of success are. But such guesses are of secondary importance compared to the choice of doing something or doing nothing, in a domain where no-one else is acting. Until other people show up, you have to just go out there and do your best.
I'm pretty sure Eric Drexler went through this already, with nanotechnology. There was a time when Drexler was in a quite unique position, of appreciating the world-shaking significance of molecular machines, having an overall picture of what they imply and how to respond, and possessing a platform (his Foresight Institute) which gave him a little visibility. The situation is very different now. We may still be headed for disaster on that front as well, but at least the ability of society to think about the issues is greatly improved, mostly because broad technical progress in chemistry and nanoscale technology has made it easier for people to see the possibilities and has also clarified what can and can't be done.
As computer science, cognitive science, and neuroscience keep advancing, the same thing will happen in artificial intelligence, and a lot of Eliezer's ideas will seem more natural and constructive than they may now appear. Some of them will be reinvented independently. All of them (that survive) should take on much greater depth and richness (compare the word pictures in Drexler's 1986 book with the calculations in his 1992 book).
Despite all the excesses and distractions, work is being done and foundations for the future are being laid. Also, Eliezer and his colleagues do have many lines into academia, despite the extent to which they exist outside it. So in terms of process, I do consider them to be on track, even if the train shakes violently at times.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T13:22:51.576Z · LW(p) · GW(p)
Like wedrifid, I don't even recall hearing a guess from Eliezer about what he thinks the odds of success are.
Eliezer took exception to my estimate linked in my comment here.
If for some reason he was only to engage in advocacy from now on, it should specifically be Friendly AI advocacy.
Quite possibly you're right about this.
Despite all the excesses and distractions, work is being done and foundations for the future are being laid. Also, Eliezer and his colleagues do have many lines into academia, despite the extent to which they exist outside it.
On this point I agree with SarahC's second comment here.
I would again recur to my point about Eliezer having an accurate view of his abilities and likelihood of success being important for public relations purposes.
Replies from: Mitchell_Porter, jimrandomh↑ comment by Mitchell_Porter · 2010-09-04T00:59:36.833Z · LW(p) · GW(p)
Eliezer took exception to my estimate linked in my comment here.
Less than 1 in 1 billion! :-) May I ask exactly what the proposition was? At the link you say "probability of ... you succeeding in playing a critical role on the Friendly AI project that you're working on". Now by one reading that probability is 1, since he's already the main researcher at SIAI.
Suppose we analyse your estimate in terms of three factors:
(probability that anyone ever creates Friendly AI) x (conditional probability SIAI contributed) x (conditional probability that Eliezer contributed)
Can you tell us where the bulk of the 10^-9 is located?
↑ comment by jimrandomh · 2010-09-03T13:36:24.985Z · LW(p) · GW(p)
Eliezer took exception to my estimate linked in my comment here.
And he was right to do so, because that estimate was obviously on the wrong order of magnitude. To make an analogy, if someone says that you weigh 10^5kg, you don't have to reveal your actual weight (or even measure it) to know that 10^5 was wrong.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T13:39:36.454Z · LW(p) · GW(p)
I agree with
if someone says that you weigh 10^5kg, you don't have to reveal your actual weight (or even measure it) to know that 10^5 was wrong.
But why is the estimate that I gave obviously on the wrong order of magnitude?
From my point of view, his reaction is an indication that his estimate is obviously on the wrong order of magnitude. But I'm still willing to engage with him and hear what he has to say, whereas he doesn't seem willing to engage with me and hear what I have to say.
Replies from: jimrandomh↑ comment by jimrandomh · 2010-09-03T15:04:29.606Z · LW(p) · GW(p)
But why is the estimate that I gave obviously on the wrong order of magnitude?
The original statement was
I assign a probability of less than 10^(-9) to you succeeding in playing a critical role on the Friendly AI project that you're working on."
The way to estimate probabilities like that is to break them into pieces. This one divides naturally into two pieces: the probability that an AGI will be created in the not-too-distant future, and the probability that Eliezer will play a critical role if it is. For the former, I estimate a probability of 0.8; but it's a complex and controversial enough topic that I would accept any probability as low as 10^-2 as, if not actually correct, at least not a grievous error. Any probability smaller than 10^-2 would be evidence of severe overconfidence.
Assume we accept this lower probability of 10^-2 for the first piece. For the second piece, as simplifying assumptions, assume there are only 10^1 "critical role" slots, and that they're assigned randomly out of all the people who might plausibly work on friendly AI. (Since we're only going for an order of magnitude, we're allowed to make simplifying assumptions like this; and we have to do so, because otherwise the problem is intractable.) In order to get a probability of 10^-9, you would need to come up with 10^8 candidate AGI researchers, each qualified to a degree similar to Eliezer. By comparison, there are 3.3x10^6 working in all computer and mathematical science occupations put together, of whom maybe 1 in 10^2 has even heard of FAI and none have dedicated their life to it.
It is possible to disagree on probabilities, but it is not possible to disagree by more than a couple orders of magnitude unless someone is either missing crucial information, has made a math error, or doesn't know how to compute probabilities. And not knowing how to compute probabilities is the norm; it's a rare skill that has to be specifically cultivated, and there's no shame in not having it yet. But it is a prerequisite for some (though certainly not all) of the discussions that take place here. And the impression that I got was that you jumped into a discussion you weren't ready for, and then let the need to be self-consistent guide your arguments in untruthful directions. I think others got the same impression as well. We call this motivated cognition - it's a standard bias that everyone suffers from to some degree - and avoiding it is also a rare skill that must be specifically cultivated, and there is no shame in not having that skill yet, either. But until you develop your rationality skills further, Eliezer isn't going to engage with you, and it would be a mistake for him to do so.
Replies from: multifoliaterose, whpearson↑ comment by multifoliaterose · 2010-09-03T15:41:11.886Z · LW(p) · GW(p)
The way to estimate probabilities like that is to break them into pieces. This one divides naturally into two pieces: the probability that an AGI will be created in the not-too-distant future, and the probability that Eliezer will play a critical role if it is. For the former, I estimate a probability of 0.8; but it's a complex and controversial enough topic that I would accept any probability as low as 10^-2 as, if not actually correct, at least not a grievous error. Any probability smaller than 10^-2 would be evidence of severe overconfidence.
I can't engage with your statement here unless you quantify the phrase "Not-too-distant future."
For the second piece, as simplifying assumptions, assume there are only 10^1 "critical role" slots, and that they're assigned randomly out of all the people who might plausibly work on friendly AI. (Since we're only going for an order of magnitude, we're allowed to make simplifying assumptions like this; and we have to do so, because otherwise the problem is intractable.) In order to get a probability of 10^-9, you would need to come up with 10^8 candidate AGI researchers, each qualified to a degree similar to Eliezer. By comparison, there are 3.3x10^6 working in all computer and mathematical science occupations put together, of whom maybe 1 in 10^2 has even heard of FAI and none have dedicated their life to it.
Two points here:
•Quoting a comment that I wrote in July:
I'm very familiar with pure mathematics. My belief is that in pure mathematics the variability in productivity of researchers stretches over many orders of magnitude. By analogy, I would guess that the productivity of Friendly AI researchers will also differ by many orders of magnitude. I suspect that the current SIAI researchers are not at the high end of this range (out of virtue of the fact that the most talented researchers are very rare, very few people are currently thinking about these things, and my belief that the correlation between currently thinking about these things and having talent is weak).
•You seem to be implicitly assuming that Friendly AI will be developed before unFriendly AI. This implicit assumption is completely ungrounded.
It is possible to disagree on probabilities, but it is not possible to disagree by more than a couple orders of magnitude unless someone is either missing crucial information, has made a math error, or doesn't know how to compute probabilities. And not knowing how to compute probabilities is the norm; it's a rare skill that has to be specifically cultivated, and there's no shame in not having it yet. But it is a prerequisite for some (though certainly not all) of the discussions that take place here.
I agree with all of this.
And the impression that I got was that you jumped into a discussion you weren't ready for, and then let the need to be self-consistent guide your arguments in untruthful directions. I think others got the same impression as well.
I can understand how you might have gotten this impression. But I think that it's important to give people the benefit of the doubt up to a certain point. Too much willingness to dismiss the consideration of what people say on account of doubting their rationality is conducive to group think and confirmation bias.
But until you develop your rationality skills further, Eliezer isn't going to engage with you, and it would be a mistake for him to do so.
In line with my comment above, I'm troubled by the fact that you've so readily assumed that my rationality skills are insufficiently developed for it to be worth Eliezer's time to engage with me.
Replies from: Mitchell_Porter↑ comment by Mitchell_Porter · 2010-09-06T11:55:05.251Z · LW(p) · GW(p)
I'm very familiar with pure mathematics. My belief is that in pure mathematics the variability in productivity of researchers stretches over many orders of magnitude. By analogy, I would guess that the productivity of Friendly AI researchers will also differ by many orders of magnitude.
Not only that, but sophisticated pure mathematics will surely supply the substance of FAI theory. I'm thinking especially of Ketan Mulmuley's research program, applying algebraic geometry to computational complexity theory. Many people think it's the most promising approach to P vs NP.
It has been suggested that the task of Friendly AI boils down to extracting the "human utility function" from the physical facts, and then "renormalizing" this using "reflective decision theory" to produce a human-relative friendly utility function, and then implementing this using a cognitive architecture which is provably stable under open-ended self-directed enhancement. The specification of the problem is still a little handwavy and intuitive, but it's not hard to see solid, well-defined problems lurking underneath the suggestive words, and it should be expected that the exact answers to those problems will come from a body of "theory" as deep and as lucid as anything presently existing in pure math.
↑ comment by whpearson · 2010-09-04T17:30:05.866Z · LW(p) · GW(p)
The way to estimate probabilities like that is to break them into pieces. This one divides naturally into two pieces: the probability that an AGI will be created in the not-too-distant future, and the probability that Eliezer will play a critical role if it is. For the former, I estimate a probability of 0.8; but it's a complex and controversial enough topic that I would accept any probability as low as 10^-2 as, if not actually correct, at least not a grievous error. Any probability smaller than 10^-2 would be evidence of severe overconfidence.
We have to assign probabilities to Artificial intelligence being first created on earth over the earths entire lifetime.
So what probability should we give to the first non-biological intelligence being created in the time period between 3 million years and 3million and 50 years time (not necessarily by humans)? Would it be greater than or less than 10^-2? If less than that, what justifies your confidence in that statement rather than your confidence that it will be created soon?
We have to get all these probabilities to sum to the chance we assign to AI ever being created, over the lifetime of the earth. So I don't see how we can avoid very small probabilities in AI being created at certain times.
↑ comment by wedrifid · 2010-09-03T12:04:35.907Z · LW(p) · GW(p)
I think that it would be better for Eliezer and for the world at large if Eliezer seriously considered the possibility that he's vastly overestimated his chances of building a Friendly AI.
We haven't heard Eliezer say how likely he believes it is that he creates a Friendly AI. He has been careful to not to discuss that subject. If he thought his chances of success were 0.5% then I would expect him to make exactly the same actions.
(ETA: With the insertion of 'relative' I suspect I would more accurately be considering the position you are presenting.)
Replies from: multifoliaterose, multifoliaterose↑ comment by multifoliaterose · 2010-09-03T12:33:35.739Z · LW(p) · GW(p)
Right, so in my present epistemological state I find it extremely unlikely that Eliezer will succeed in building a Friendly AI. I gave an estimate here which proved to be surprisingly controversial.
The main points that inform my thinking here are:
The precedent for people outside of the academic mainstream having mathematical/scientific breakthroughs in recent times is extremely weak. In my own field of pure math I know of only two people without PhD's in math or related fields who have produced something memorable in the last 70 years or so, namely Kurt Heegner and Martin Demaine. And even Heegner and Demaine are (relatively speaking) quite minor figures. It's very common for self-taught amateur mathematicians to greatly underestimate the difficulty of substantive original mathematical research. I find it very likely that the same is true in virtually all scientific fields and thus have an extremely skeptical Bayesian prior against any proposition of the type "amateur intellectual X will solve major scientific problem Y."
From having talked with computer scientists and AI researchers, I have a very strong impression that the consensus is that AGI is way out of reach at present. See for example points #1 and #5 of Scott Aaronson's The Singularity is Far.
The fact that Eliezer does not appear to have seriously contemplated or addressed the the two points above and their implications diminishes my confidence in his odds of success still further.
Replies from: thomblake, Wei_Dai, timtyler, None↑ comment by thomblake · 2010-09-03T13:03:01.430Z · LW(p) · GW(p)
The fact that Eliezer does not appear to have seriously contemplated or addressed the the two points above and their implications diminishes my confidence in his odds of success still further.
That you have this impression greatly diminishes my confidence in your intuitions on the matter. Are you seriously suggesting that Eliezer has not contemplated AI researchers' opinions about AGI? Or that he hasn't thought about just how much effort should go into a scientific breakthrough?
Someone please throw a few hundred relevant hyperlinks at this person.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T13:14:59.548Z · LW(p) · GW(p)
I'm not saying that Eliezer has given my two points no consideration. I'm saying that Eliezer has not given my two points sufficient consideration. By all means, send hyperlinks that you find relevant my way - I would be happy to be proven wrong.
↑ comment by Wei Dai (Wei_Dai) · 2010-09-03T13:45:15.196Z · LW(p) · GW(p)
Regarding your first point, I'm pretty sure Eliezer does not expect to solve FAI by himself. Part of the reason for creating LW was to train/recruit potential FAI researchers, and there are also plenty of Ph.D. students among SIAI visiting fellows.
Regarding the second point, do you want nobody to start researching FAI until AGI is within reach?
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T13:58:38.401Z · LW(p) · GW(p)
Regarding your first point, I'm pretty sure Eliezer does not expect to solve FAI by himself. Part of the reason for creating LW was to train/recruit potential FAI researchers, and there are also plenty of Ph.D. students among SIAI visiting fellows.
Right, but the historical precedent for an amateur scientist even being at all involved in a substantial scientific breakthrough over the past 50 years is very weak.
Also, my confidence in Eliezer's ability to train/recruit potential FAI researchers has been substantially diminished for the reasons that I give in Existential Risk and Public Relations. I personally would be interested in working with Eliezer if he appeared to me to be well grounded. The impressions that I've gotten from my private correspondence with Eliezer and from his comments have given me a very strong impression that I would find him too difficult to work with for me to be able to do productive FAI research with him.
Regarding the second point, do you want nobody to start researching FAI until AGI is within reach?
No. I think that it would be worthwhile for somebody to do FAI research in line with Vladimir Nesov's remarks here and here.
But I maintain that the probability of success is very small and that the only justification for doing it is the possibility of enormous returns. If people had established an institute for the solution of Fermat's Last Theorem in the 1800's, the chances of anybody there playing a decisive role in the solution of Fermat's Last Theorem would be very small. I view the situation with FAI as analogous.
Replies from: jimrandomh, Wei_Dai, Wei_Dai↑ comment by jimrandomh · 2010-09-03T15:18:11.027Z · LW(p) · GW(p)
Right, but the historical precedent for an amateur scientist even being at all involved in a substantial scientific breakthrough over the past 50 years is very weak.
Hold on - there are two different definitions of the word "amateur" that could apply here, and they lead to very different conclusions. The definition I think of first, is that an amateur at something is someone who doesn't get paid folr doing it, as opposed to a professional who makes a living at it. By this definition, amateurs rarely achieve anything, and if they do, they usually stop being amateurs. But Eliezer's full-time occupation is writing, thinking, and talking about FAI and related topics, so by this definition, he isn't an amateur (regardless of whether or not you think he's qualified for that occupation).
The other definition of "amateur scientist" would be "someone without a PhD". This definition Eliezer does fit, but by this definition, the amateurs have a pretty solid record. And if you narrow it down to computer software, the amateurs have achieved more than the PhDs have!
I feel like you've taken the connotations of the first definition and unknowingly and wrongly transferred them to the second definition.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T16:03:07.068Z · LW(p) · GW(p)
Okay, so, I agree with some of what you say above. I think I should have been more precise.
A claim of the type "Eliezer is likely to build a Friendly AI" requires (at least in part) a supporting claim of the type "Eliezer is in group X where people in group X are likely to build a Friendly AI." Even if one finds such a group X, this may not be sufficient because Eliezer may belong to some subgroup of X which is disproportionately unlikely to build a Friendly AI. But one at least has to be able to generate such a group X.
At present I see no group X that qualifies.
1.Taking X to be "humans in the developed world" doesn't work because the average member of X is extremely unlikely to build a Friendly AI.
Taking X to be "people with PhDs a field related to artificial intelligence" doesn't work because Eliezer doesn't have a PhD in artificial intelligence.
Taking X to be "programmers" doesn't work because Eliezer is not a programmer.
Taking X to be "people with very high IQ" is a better candidate, but still doesn't yield a very high probability estimate because very high IQ is not very strongly correlated with technological achievement.
Taking X to be "bloggers about rationality" doesn't work because there's very little evidence that being a blogger about rationality is correlated with skills conducive to building a Friendly AI.
Which suitable group X do you think that Eliezer falls into?
Replies from: jimrandomh↑ comment by jimrandomh · 2010-09-03T16:25:23.056Z · LW(p) · GW(p)
How about "people who have publically declared an intention to try to build an FAI"? That seems like a much more relevant reference class, and it's tiny. (I'm not sure how tiny, exactly, but it's certainly smaller than 10^3 people right now) And if someone else makes a breakthrough that suddenly brings AGI within reach, they'll almost certainly choose to recruit help from that class.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T16:44:23.662Z · LW(p) · GW(p)
I agree that the class that you mention is a better candidate than the ones that I listed. However:
I find it fairly likely that the class will expand dramatically if there's a breakthrough that brings AGI in within reach.
Announcing interest in FAI does not entail having the skills necessary to collaborate with the people working on an AGI to make it Friendly.
In addition to these points, there's a factor which makes Eliezer less qualified than the usual member of the class, namely his public relations difficulties. As he says here "I feel like I'm being held to an absurdly high standard … like I'm being asking to solve PR problems that I never signed up for." As a matter of reality, PR matters in this world. If there was a breakthrough that prompted a company like IBM to decide to build an AGI, I have difficulty imagining them recruiting Eliezer, the reason being that Eliezer says things that sound strange and is far out of the mainstream. However of course
(i) I could imagine SIAI's public relations improving substantially in the future - this would be good and would raise the chances of Eliezer being able to work with the researchers who build an AGI.
(ii) There may of course be other factors which make Eliezer more likely than other members of the class to be instrumental to building a Friendly AI.
Despite factors (i) and (ii), putting all of the information that I have together, my estimate of 10^(-9) still feels about right to me. I'd be happy to continue trading information with you with a view toward syncing up our probabilities if you're so inclined.
Replies from: jimrandomh↑ comment by jimrandomh · 2010-09-03T17:05:06.938Z · LW(p) · GW(p)
I find it fairly likely that the class will expand dramatically if there's a breakthrough that brings AGI in within reach.
I should hope not! If that happens, it means the person who made the breakthrough released it to the public. That would be a huge mistake, because it would greatly increase the chances of an unfriendly AI being built before a friendly one.
Despite factors (i) and (ii), putting all of the information that I have together, my estimate of 10^(-9) still feels about right to me.
That's only because you said it in public and aren't willing to appear inconsistent. You still haven't decomposed this into manageable pieces with numbers. And since we've already seen that you wrote the bottom line first, we would have strong reason to not trust those numbers if you did.
Replies from: multifoliaterose, timtyler↑ comment by multifoliaterose · 2010-09-03T17:49:47.885Z · LW(p) · GW(p)
I should hope not! If that happens, it means the person who made the breakthrough released it to the public. That would be a huge mistake, because it would greatly increase the chances of an unfriendly AI being built before a friendly one.
Two points:
It seems very likely to me that there's a string of breakthroughs which will lead to AGI and that it will gradually become clear that to people that they should be thinking about friendliness issues.
Even if there's a single crucial breakthrough, I find it fairly likely that the person who makes it will not have friendliness concerns in mind.
That's only because you said it in public and aren't willing to appear inconsistent. You still haven't decomposed this into manageable pieces with numbers. And since we've already seen that you wrote the bottom line first, we would have strong reason to not trust those numbers if you did.
I believe that the human brain is extremely poorly calibrated to determining probabilities through the explicit process that you describe and that the human brain's intuition is often more reliable for such purposes. My attitude is in line with Holden's comments 14 and 16 on the GiveWell Singularity Summit thread.
In line with the last two paragraphs of one of my earlier comments, I find your quickness to assume that my thinking on these matters stems from motivated cognition disturbing. Of course, I may be exhibiting motivated cognition, but the same is true of you, and your ungrounded confidence in your superiority to me is truly unsettling. As such, I will cease to communicate further with you unless you resolve to stop confidently asserting that I'm exhibiting motivated cognition.
Replies from: timtyler, Perplexed↑ comment by timtyler · 2010-09-04T08:37:29.180Z · LW(p) · GW(p)
P(SIAI will be successful) may be smaller that 10^-(3^^^^3)!
I don't think that's the right way to escape from a Pascal's mugging. In the case of the SIAI, there isn't really clear evidence that the organisation is having any positive effect - let alone SAVING THE WORLD. When the benefit could plausibly be small, zero - or indeed negative - one does not need to invoke teeny tiny probabalities to offset it.
↑ comment by timtyler · 2010-09-04T08:30:22.372Z · LW(p) · GW(p)
If that happens, it means the person who made the breakthrough released it to the public. That would be a huge mistake, because it would greatly increase the chances of an unfriendly AI being built before a friendly one.
You are so concerned about the possibility of failure that you want to slow down research, publication and progress in the field - in order to promote research into safety?
Do you think all progress should be slowed down - or just progress in this area?
The costs of stupidity are a million road deaths a year, and goodness knows how many deaths in hospitals. Intelligence would have to be pretty damaging to outweigh that.
There is an obvious good associated with publication - the bigger the concentration of knowledge about intelligent machines there is in one place, the greater wealth inequality is likely to result, and the harder it would be for the rest of society to deal with a dominant organisation. Spreading knowlege helps spread out the power - which reduces the chance of any one group of people becoming badly impoverished. Such altruistic measures may help to prevent a bloody revolution from occurring.
↑ comment by Wei Dai (Wei_Dai) · 2010-09-03T15:50:31.943Z · LW(p) · GW(p)
Right, but the historical precedent for an amateur scientist even being at all involved in a substantial scientific breakthrough over the past 50 years is very weak.
What are we supposed to infer from that? That if you add an amateur scientist to a group of PhDs, that would substantially decrease their chance of making a breakthrough?
The impressions that I've gotten from my private correspondence with Eliezer and from his comments have given me a very strong impression that I would find him too difficult to work with for me to be able to do productive FAI research with him.
SIAI held a 3-day decision theory workshop in March that I attended along with Stuart Armstrong and Gary Drescher as outside guests. I feel pretty safe in saying that none of us found Eliezer particularly difficult to work with. I wonder if perhaps you're generalizing from one example here.
I think that it would be worthwhile for somebody to do FAI research in line with Vladimir Nesov's remarks here and here.
Do you also think it would be worthwhile for somebody to try to build an organization to do FAI research? If so, who do you think should be doing that, if not Eliezer and his supporters? Or is your position more like cousin_it's, namely that FAI research should just be done by individuals on their free time for now?
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T16:15:06.179Z · LW(p) · GW(p)
What are we supposed to infer from that? That if you add an amateur scientist to a group of PhDs, that would substantially decrease their chance of making a breakthrough?
No, certainly not. I just don't see much evidence that Eliezer is presently adding value to Friendly AI research. I think he could be doing more to reduce existential risk if he were operating under different assumptions.
SIAI held a 3-day decision theory workshop in March that I attended along with Stuart Armstrong and Gary Drescher as outside guests. I feel pretty safe in saying that none of us found Eliezer particularly difficult to work with. I wonder if perhaps you're generalizing from one example here.
Of course you could be right here, but the situation is symmetric, the same could be the case for you, Stuart Armstrong and Gary Drescher. Keep in mind that there's a strong selection effect here - if you're spending time with Eliezer you're disproportionately likely to be well suited to working with Eliezer, and people who have difficulty working with Eliezer are disproportionately unlikely be posting on Less Wrong or meeting with Eliezer.
My intuition is that there are a lot of good potential FAI researchers who would not feel comfortable working with Eliezer given his current disposition, but I may be wrong.
Do you also think it would be worthwhile for somebody to try to build an organization to do FAI research? If so, who do you think should be doing that, if not Eliezer and his supporters? Or is your position more like cousin_it's, namely that FAI research should just be done by individuals on their free time for now?
Quite possibly it's a good thing for Eliezer and his supporters to be building an organization to do FAI research. On the other hand maybe cousin_it's position is right. I have a fair amount of uncertainty on this point.
The claim that I'm making is quite narrow: that it would be good for the cause of existential risk reduction if Eliezer seriously considered the possibility that he's greatly overestimated his chances of building a Friendly AI.
I'm not saying that it's a bad thing to have an organization like SIAI. I'm not saying that Eliezer doesn't have a valuable role to serve within SIAI. I'm reminded of Robin Hanson's Against Disclaimers though I don't feel comfortable with his condescending tone and am not thinking of you in that light :-).
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2010-09-04T08:04:29.519Z · LW(p) · GW(p)
My intuition is that there are a lot of good potential FAI researchers who would not feel comfortable working with Eliezer given his current disposition, but I may be wrong.
This topic seems important enough that you should try to figure out why your intuition says that. I'd be interested in hearing more details about why you think a lot of good potential FAI researchers would not feel comfortable working with Eliezer. And in what ways do you think he could improve his disposition?
↑ comment by Wei Dai (Wei_Dai) · 2010-09-04T13:07:05.107Z · LW(p) · GW(p)
I personally would be interested in working with Eliezer if he appeared to me to be well grounded. The impressions that I've gotten from my private correspondence with Eliezer and from his comments have given me a very strong impression that I would find him too difficult to work with for me to be able to do productive FAI research with him.
My reading of this is that before you corresponded privately with Eliezer, you were
- interested in personally doing FAI research
- assigned high enough probability to Eliezer's success to consider collaborating with him
And afterward, you became
- no longer interested in doing FAI research
- massively decreased your estimate of Eliezer's chance of success
Is this right? If so, I wonder what he could have said that made you change your mind like that. I guess either he privately came off as much less competent than he appeared in the public writings that drew him to your attention in the first place (which seems rather unlikely), or you took his response as some sort of personal affront and responded irrationally.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-04T14:03:42.452Z · LW(p) · GW(p)
So, the situation is somewhat different than the one that you describe. Some points of clarification.
•I first came across Overcoming Bias in 2008. Eliezer was recommended to me by a friend who I respect a great deal. My reactions to the first postings that I read by Eliezer was strong discomfort with his apparent grandiosity and self absorption. This discomfort was sufficiently strong for me to lose interest despite my friend's endorsement.
•I started reading Less Wrong in earnest in the beginning of 2010. This made it clear to me that Eliezer has a lot to offer and that it was unfortunate that I had been pushed away by my initial reaction.
•I never assigned a very high probability to Eliezer making a crucial contribution to an FAI research project. My thinking was that the enormous positive outcome associated with success might be sufficiently great to justify the project despite the small probability.
•I didn't get much of a chance to correspond privately with Eliezer at all. He responded to a couple of my messages with one line dismissive responses and then stopped responding to my subsequent messages. Naturally this lowered the probability that I assigned to being able to collaborating with him. This also lowered my confidence in his ability to attract collaborators in general.
•If Eliezer showed strong ability to attract and work well with collaborators (including elite academics who are working on artificial intelligence research) then I would find it several orders of magnitude more likely that he would make a crucial contribution to an FAI research project. For concreteness I'll throw out the number 10^(-6).
•I feel that the world is very complicated and that randomness plays a very large role. This leads me to assign a very small probability to the proposition that any given individual will play a crucial role in eliminating existential risk.
•I freely acknowledge that I may be influenced by emotional factors. I make an honest effort at being level headed and sober but as I mention elsewhere, my experience posting on Less Wrong has been emotionally draining. I find that I become substantially less rational when people assume that my motives are impure (some sort of self fulfilling prophesy).
You may notice that of my last four posts, the first pair was considerably more impartial than the second pair. (This is reflected in the fact that the first pair was upvoted more than the second pair.) My subjective perception is that I started out thinking quite carefully and became less rational as I read and responded to hostile commentators.
Replies from: komponisto, Wei_Dai↑ comment by komponisto · 2010-09-04T15:29:09.563Z · LW(p) · GW(p)
I first came across Overcoming Bias in 2008. Eliezer was recommended to me by a friend who I respect a great deal. My reactions to the first postings that I read by Eliezer was strong discomfort with his apparent grandiosity and self absorption. This discomfort was sufficiently strong for me to lose interest despite my friend's endorsement.
I'd be really interested to know which posts these were, because it would help me to distinguish between the following interpretations:
(1) First impressions really do matter: even though you and I are probably very similar in many respects, we have different opinions of Eliezer simply because in the first posts of his I read, he sounded more like a yoga instructor than a cult leader; whereas perhaps the first thing you read was some post where his high estimation of his abilities relative to the rest of humanity was made explicit, and you didn't have the experience of his other writings to allow you to "forgive" him for this social transgression.
(2) We have different personalities, which cause us to interpret people's words differently: you and I read more or less the same kind of material first, but you just interpreted it as "grandiose" whereas I didn't.
What's interesting in any case is that I'm not sure that I actually disagree with you all that much about Eliezer having a small chance of success (though I think you quantify it incorrectly with numbers like 10^(-9) or 10^(-6) -- these are way too small). Where we differ seems to be in the implications we draw from this. You appear to believe that Eliezer and SIAI are doing something importantly wrong, that could be fixed by means of a simple change of mindset, and that they shouldn't be supported until they make this change. By contrast, my interpretation is that this is an extremely difficult problem, that SIAI is basically the first organization that has begun to make a serious attempt to address it, and that they are therefore worthy of being supported so that they can increase their efforts in the directions they are currently pursuing and potentially have a larger impact than they otherwise would.
I've been meaning to ask you: given your interest in reducing existential risk, and your concerns about SIAI's transparency and their general strategy, have you considered applying to the Visiting Fellows program? That would be an excellent way not only to see what it is they do up close, but also to discuss these very issues in person at length with the people involved in SIAI strategy -- which, in my experience, they are very interested in doing, even with short-term visitors.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-04T16:14:07.481Z · LW(p) · GW(p)
I'd be really interested to know which posts these were, because it would help me to distinguish between the following interpretations:
Right, so the first posts that I came across were Eliezer's Coming of Age posts which I think are unrepresentatively self absorbed. So I think that the right interpretation is the first that you suggest.
What's interesting in any case is that I'm not sure that I actually disagree with you all that much about Eliezer having a small chance of success (though I think you quantify it incorrectly with numbers like 10^(-9) or 10^(-6) -- these are way too small). Where we differ seems to be in the implications we draw from this. You appear to believe that Eliezer and SIAI are doing something importantly wrong, that could be fixed by means of a simple change of mindset, and that they shouldn't be supported until they make this change. By contrast, my interpretation is that this is an extremely difficult problem, that SIAI is basically the first organization that has begun to make a serious attempt to address it, and that they are therefore worthy of being supported so that they can increase their efforts in the directions they are currently pursuing and potentially have a larger impact than they otherwise would.
Since I made my top level posts, I've been corresponding with Carl Shulman who informed me of some good things that SIAI has been doing that have altered my perception of the institution. I think that SIAI may be worthy of funding.
Regardless as to the merits of SIAI's research and activities, I think that in general it's valuable to promote norms of Transparency and Accountability. I would certainly be willing to fund SIAI if it were strongly recommended by a highly credible external charity evaluator like GiveWell. Note also a comment which I wrote in response to Jasen.
I would like to talk more about these things - would you like to share email addresses? PM me if so.
I've been meaning to ask you: given your interest in reducing existential risk, and your concerns about SIAI's transparency and their general strategy, have you considered applying to the Visiting Fellows program? That would be an excellent way not only to see what it is they do up close, but also to discuss these very issues in person at length with the people involved in SIAI strategy -- which, in my experience, they are very interested in doing, even with short-term visitors.
At this point I worry that I've alienated the SIAI people to such an extent that they might not be happy to have me. But I'd certainly be willing if they're favorably disposed toward me.
I'll remark that back in December after reading Anna Salamon's posting on the SIAI Visting Fellows program I did send Anna Salamon a long email expressing some degree of interest and describing some my concerns without receiving a response. I now find it most plausible that she just forgot about it and that I should have tried again, but maybe you can understand from this how I got the impression that becoming an SIAI Visiting Fellow was not a strong option for me.
Replies from: komponisto↑ comment by komponisto · 2010-09-06T22:05:42.513Z · LW(p) · GW(p)
I would like to talk more about these things - would you like to share email addresses? PM me if so.
Done.
I'll remark that back in December after reading Anna Salamon's posting on the SIAI Visting Fellows program I did send Anna Salamon a long email expressing some degree of interest and describing some my concerns without receiving a response. I now find it most plausible that she just forgot about it and that I should have tried again, but maybe you can understand from this how I got the impression that becoming an SIAI Visiting Fellow was not a strong option for me.
As it happens, the same thing happened to me; it turned out that my initial message had been caught in a spam filter. I eventually ended up visiting for two weeks, and highly recommend the experience.
↑ comment by Wei Dai (Wei_Dai) · 2010-09-05T02:32:38.633Z · LW(p) · GW(p)
If Eliezer showed strong ability to attract and work well with collaborators (including elite academics who are working on artificial intelligence research) then I would find it several orders of magnitude more likely that he would make a crucial contribution to an FAI research project. For concreteness I'll throw out the number 10^(-6).
This, along with your other estimate of 10^(-9), implies that your probability for Eliezer being able to eventually attract and work well with collaborators is currently 1/1000. Does that really seem reasonable to you (would you be willing to bet at those odds?), given other evidence besides your private exchange with Eliezer? Such as:
- Eliezer already had a close collaborator, namely Marcello
- SIAI has successfully attracted many visiting fellows
- SIAI has successfully attracted top academics to speak at their Singularity Summit
- Eliezer is currently writing a book on rationality, so presumably he isn't actively trying to recruit collaborators at the moment
- Other people's reports of not finding Eliezer particularly difficult to work with
It seems to me that rationally updating on Eliezer's private comments couldn't have resulted in such a low probability. So I think a more likely explanation is that you were offended by the implications of Eliezer's dismissive attitude towards your comments.
(Although, given Eliezer's situation, it would probably be a good idea for him to make a greater effort to avoid offending potential supporters, even if he doesn't consider them to be viable future collaborators.)
My subjective perception is that I started out thinking quite carefully and became less rational as I read and responded to hostile commentators.
Your responses to me seem pretty level headed and sober. I hope that means you don't find my comments too hostile.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-05T05:44:01.813Z · LW(p) · GW(p)
This, along with your other estimate of 10^(-9), implies that your probability for Eliezer being able to eventually attract and work well with collaborators is currently 1/1000. Does that really seem reasonable to you (would you be willing to bet at those odds?)
Thinking it over, my estimate of 10^(-6) was way too high. This isn't because of a lack of faith in Eliezer's abilities in particular. I would recur to my above remark that I think that everybody has very small probability of succeeding in efforts to eliminate existential risk. We're part of a complicated chaotic dynamical system and to a large degree our cumulative impact on the world is unintelligible and unexpected (because of a complicated network of unintended consequences, side effects, side effects of the side effects, etc.).
Your responses to me seem pretty level headed and sober. I hope that means you don't find my comments too hostile.
Glad to hear it :-)
↑ comment by timtyler · 2010-09-10T21:15:10.172Z · LW(p) · GW(p)
From having talked with computer scientists and AI researchers, I have a very strong impression that the consensus is that AGI is way out of reach at present. See for example points #1 and #5 of Scott Aaronson's The Singularity is Far.
I don't think there's any such consensus. Most of those involved know that they don't know with very much confidence. For a range of estimates, see the bottom of:
http://alife.co.uk/essays/how_long_before_superintelligence/
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-10T22:24:33.951Z · LW(p) · GW(p)
For what it's worth, in saying "way out of reach" I didn't mean "chronologically far away," I meant "far beyond the capacity of all present researchers." I think it's quite possible that AGI is just 50 years away.
I think that the absence of plausibly relevant and concrete directions for AGI/FAI research, the chance of having any impact on the creation of an FAI through research is diminished by many orders of magnitude.
If there are plausibly relevant and concrete directions for AGI/FAI research then the situation is different, but I haven't heard examples that I find compelling.
Replies from: timtyler↑ comment by timtyler · 2010-09-11T09:42:26.237Z · LW(p) · GW(p)
"Just 50 years?" Shane Legg's explanation of why his mode is at 2025:
http://www.vetta.org/2009/12/tick-tock-tick-tock-bing/
If 15 years is more accurate - then things are a bit different.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-11T13:59:45.902Z · LW(p) · GW(p)
"Just 50 years?" Shane Legg's explanation of why his mode is at 2025:
Thanks for pointing this out. I don't have the subject matter knowledge to make an independent assessment of the validity of the remarks in the linked article, but it makes points that I had not seen before.
I'd recur to CarlShulman's remark about selection bias here. I look forward to seeing the results of the hypothetical Bostrom survey and the SIAI collection of all public predictions.
If 15 years is more accurate - then things are a bit different.
I agree. There's still an issue of a lack of concrete directions of research at present but if 15 years is accurate then I agree with Eliezer that we should be in "crunch" mode (amassing resources specifically directed at future FAI research).
Replies from: Will_Newsome, timtyler↑ comment by Will_Newsome · 2010-09-14T02:28:53.474Z · LW(p) · GW(p)
I agree. There's still an issue of a lack of concrete directions of research at present but if 15 years is accurate then I agree with Eliezer that we should be in "crunch" mode (amassing resources specifically directed at future FAI research).
At any rate, most rationalists who have seriously considered the topic will agree that there is a large amount of probability mass 15 years into the future: large enough that even if the median estimate till AGI is 2050, we're still in serious crunch time. The tails are fat in both directions. (This is important because it takes away a lot of the Pascalian flavoring that makes people (justifiably) nervous when reasoning about whether or not to donate to FAI projects: 15% chance of FOOM before 2020 just feels very different to a bounded rationalist than a .5% chance of FOOM before 2020.)
Thanks for pointing this out. I don't have the subject matter knowledge to make an independent assessment of the validity of the remarks in the linked article, but it makes points that I had not seen before.
For what it's worth, Shane Legg is a pretty reasonable fellow who understands that AGI isn't automatically good, so we can at least rule out that his predictions are tainted by the thoughts of "Yay, technology is good, AGI is close!" that tend to cast doubt on the lack of bias in most AGI researchers' and futurists' predictions. He's familiar with the field and indeed wrote the book on Machine Super Intelligence. I'm more persuaded by Legg's arguments than most at SIAI, though, and although this isn't a claim that is easily backed by evidence, the people at SIAI are really freakin' good thinkers and are not to be disagreed with lightly.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-14T05:42:38.651Z · LW(p) · GW(p)
At any rate, most rationalists who have seriously considered the topic will agree that there is a large amount of probability mass 15 years into the future: large enough that even if the median estimate till AGI is 2050, we're still in serious crunch time. The tails are fat in both directions.
I recur to my concern about selection effects. If it really is reasonable to place a large amount of probability mass 15 years into the future, why are virtually all mainstream scientists (including the best ones) apparently oblivious to this?
I do think that it's sufficiently likely that the people in academia have erred that it's worth my learning more about this topic and spending some time pressing people within academia on this point. But at present I assign a low probability (~5%) to the notion that the mainstream has missed something so striking as a large probability of a superhuman AI within 15 years.
Incidentally, I do think that decisive paradigm changing events are very likely to occur over the next 200 years and that this warrants focused effort on making sure that society is running as possible (as opposed to doing pure scientific research with the justification that it may pay off in 500 years).
Replies from: Will_Newsome, jacob_cannell↑ comment by Will_Newsome · 2010-09-14T06:14:08.623Z · LW(p) · GW(p)
A fair response to this requires a post that Less Wrong desperately needs to read: People Are Crazy, the World Is Mad. Unfortunately this requires that I convince Michael Vassar or Tom McCabe to write it. Thus, I am now on a mission to enlist the great power of Thomas McCabe.
(A not-so-fair response: you underestimate the extent to which academia is batshit insane just like nearly every individual in it, you overestimate the extent to which scientists ever look outside of their tiny fields of specialization, you overestimate the extent to which the most rational scientists are willing to put their reputations on the line by even considering much less accepting an idea as seemingly kooky as 'human-level AI by 2035', and you underestimate the extent to which the most rational scientists are starting to look at the possibility of AGI in the next 50 years (which amounts to non-trivial probability mass in the next 15). I guess I don't know who the very best scientists are. (Dawkins and Tooby/Cosmides impress me a lot; Tooby was at the Summit. He signed a book that's on my table top. :D ) Basically, I think you're giving academia too much credit. These are all assertions, though; like I said, this response is not a fair one, but this way at least you can watch for a majoritarian bias in your thinking and a contrarian bias in my arguments.)
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-14T06:39:22.363Z · LW(p) · GW(p)
I look forward to the hypothetical post.
As for your "not-so-fair response" - I seriously doubt that you know enough about academia to have any confidence in this view. I think that first hand experience is crucial to developing a good understanding of the strengths and weaknesses of academia.
(I say this with all due respect - I've read and admired some of your top level posts.)
Replies from: Will_Newsome↑ comment by Will_Newsome · 2010-09-14T06:54:01.435Z · LW(p) · GW(p)
As for your "not-so-fair response" - I seriously doubt that you know enough about academia to have any confidence in this view. I think that first hand experience is crucial to developing a good understanding of the strengths and weaknesses of academia.
I definitely don't have the necessary first-hand-experience: I was reporting second-hand the impressions of a few people who I respect but whose insights I've yet to verify. Sorry, I should have said that. I deserve some amount of shame for my lack of epistemic hygiene there.
(I say this with all due respect - I've read and admired some of your top level posts.)
Thanks! I really appreciate it. A big reason for the large amounts of comments I've been barfing up lately is a desire to improve my writing ability such that I'll be able to make more and better posts in the future.
↑ comment by jacob_cannell · 2010-09-15T01:47:13.003Z · LW(p) · GW(p)
If it really is reasonable to place a large amount of probability mass 15 years into the future, why are virtually all mainstream scientists (including the best ones) apparently oblivious to this?
How do you support this? Have you done a poll of mainstream scientists (or better yet - the 'best' ones)? I haven't seen a poll exactly, but when IEEE ran a special on the Singularity, the opinions were divided almost 50/50. It's also important to note that the IEEE editor was against the Singularity-hypothesis - if I remember correctly, so there may be some bias there.
And whose opinions should we count exactly? Do we value the opinions of historians, economists, psychologists, chemists, geologists, astronomers, etc etc as much as we value the opinions of neuroscientists, computer scientists, and engineers?
I'd actually guess that at this point in time, a significant chunk of the intelligence of say Silicon Valley believes that the default Kurzweil/Moravec view is correct - AGI will arrive around when Moore's law makes it so.
200 years? There is wisdom in some skepticism, but that seems excessive. If you hold such a view, you should analyze it with respect to its fundamental support based on a predictive technological roadmap - not a general poll of scientists.
The semiconductor industry predicts it's own future pretty accurately, but they don't invite biologists, philosophers or mathematicians to those meetings. Their roadmap and moore's law in general is the most relevant for predicting AGI.
I base my own internal estimate on my own knowledge of the relevant fields - partly because this is so interesting and important that one should spend time investigating it.
I honestly suspect that most people who reject the possibility of near-term AGI have some deeper philosophical rejection.
If you are a materialist then intelligence is just another algorithm - something the brain does, and something we can build. It is an engineering problem and subject to the same future planning that we use for other engineering challenges.
Replies from: multifoliaterose, timtyler↑ comment by multifoliaterose · 2010-09-15T02:01:52.594Z · LW(p) · GW(p)
How do you support this? Have you done a poll of mainstream scientists (or better yet - the 'best' ones)?
I have not done a poll of mainstream scientists. Aside from Shane Legg, the one mainstream scientist who I know of who has written on this subject is Scott Aaronson in his The Singularity Is Far article.
I was not claiming that I have strong grounds for confidence in my impressions of expert views. But it is the case if there's a significant probability that we'll see AGI over the next 15 years, mainstream scientists are apparently oblivious to this. They are not behaving as I would expect them to if they believed that AGI is 15 years off.
I haven't seen a poll exactly, but when IEEE ran a special on the Singularity, the opinions were divided almost 50/50. It's also important to note that the IEEE editor was against the Singularity-hypothesis - if I remember correctly, so there may be some bias there.
Can you give a reference?
I'd actually guess that at this point in time, a significant chunk of the intelligence of say Silicon Valley believes that the default Kurzweil/Moravec view is correct - AGI will arrive around when Moore's law makes it so.
This is interesting. I presume then that they believe that the software aspect of the problem is easy. Why do they believe this.
200 years? There is wisdom in some skepticism, but that seems excessive. If you hold such a view, you should analyze it with respect to its fundamental support based on a predictive technological roadmap - not a general poll of scientists.
I have sufficiently little subject matter knowledge so that it's reasonable for me to take the outside view here and listen to people who seem to know what they're talking about rather than attempting to do a detailed analysis myself.
Replies from: jacob_cannell, timtyler↑ comment by jacob_cannell · 2010-09-15T02:31:10.990Z · LW(p) · GW(p)
Aside from Shane Legg, the one mainstream scientist who I know of who has written on this subject is Scott Aaronson in his The Singularity Is Far article.
Yes, from my reading of Shane Legg I think his prediction is a reasonable inside view and close to my own. But keep in mind it is also something of popular view. Kurzweil's latest tome was probably not much new news for most of it's target demographic (silicon valley).
I've read Aaronson's post and his counterview seems to boil down to generalized pessimism, which I don't find to be especially illuminating. However, he does raise the good point about solving subproblems first. Of course, Kurzweil spends a good portion of TSIN summarizing progress in sub-problems of reverse engineering the brain.
There appears to be a good deal of neuroscience research going on right now, but perhaps not nearly enough serious computational neuroscience and AGI research as we may like, but it is still proceeding. MIT's lab is no joke.
There is some sort of strange academic stigma though as Legg discusses on his blog - almost like a silent conspiracy against serious academic AGI. Nonetheless, there appears to be no stigma against the precursors, which is where one needs to start anyway.
I was not claiming that I have strong grounds for confidence in my impressions of expert views. But it is the case if there's a significant probability that we'll see AGI over the next 15 years, mainstream scientists are apparently oblivious to this. They are not behaving as I would expect them to if they believed that AGI is 15 years off.
I do not think we can infer their views on this matter based on their behaviour. Given the general awareness of the meme I suspect a good portion of academics in general have heard of it. That doesn't mean that anyone will necessarily change their behavior.
I agree this seems really odd, but then I think - how have I changed my behavior? And it dawns on me that this is a much more complex topic.
For the IEEE singularity issue - just google it .. something like "IEEE Singularity special issue". I'm having slow internet atm.
This is interesting. I presume then that they believe that the software aspect of the problem is easy. Why do they believe this.
Because any software problem can become easy given enough hardware.
For example, we have enough neuroscience data to build reasonably good models of the low level cortical circuits today . We also know the primary function of perhaps 5% of the higher level pathways. For much of that missing 95% we have abstract theories but are still very much in the dark.
With enough computing power we could skip tricky neuroscience or AGI research and just string together brain-ish networks built on our current cortical circuit models, throw them in a massive VR game-world sim that sets up increasingly difficult IQ puzzles as a fitness function, and use massive evolutionary search to get something intelligent.
The real solution may end up looking something like that, but will probably use much more human intelligence and be less wasteful of our computational intelligence.
Replies from: timtyler↑ comment by timtyler · 2010-10-01T17:07:43.167Z · LW(p) · GW(p)
This is interesting. I presume then that they believe that the software aspect of the problem is easy. Why do they believe this.
Because any software problem can become easy given enough hardware.
That would have been a pretty naive reply - since we know from public key crypto that it is relatively easy to make really difficult problems that require stupendous quantities of hardware to solve.
Replies from: jacob_cannell↑ comment by jacob_cannell · 2010-10-01T18:49:13.922Z · LW(p) · GW(p)
Technically true - I should have said "tractable" or "these types of" rather than "any". That of course is what computational complexity is all about.
Replies from: timtyler↑ comment by timtyler · 2010-10-01T19:23:33.575Z · LW(p) · GW(p)
IMO, the biggest reason we have for thinking that the software will be fairly tractable, is that we have an existing working model which we could always just copy - if the worst came to the worst.
Replies from: jacob_cannell↑ comment by jacob_cannell · 2010-10-01T22:49:46.146Z · LW(p) · GW(p)
Agreed, although it will be very difficult to copy it without understanding it in considerably more detail than we do at present. Copying without any understanding (whole brain scanning and emulation) is possible in theory, but the required engineering capability for that level of scanning technology seems pretty far into the future at the moment.
↑ comment by timtyler · 2010-10-01T17:13:44.250Z · LW(p) · GW(p)
A poll of mainstream scientists sounds like a poor way to get an estimate of the date of arrival of "human-level" machine minds - since machine intelligence is a complex and difficult field - and so most outsiders will probably be pretty clueless.
Also, 15 years is still a long way off: people may think 5 years out, when they are feeling particularly far sighted. Expecting major behavioral changes from something 15 years down the line seems a bit unreasonable.
↑ comment by timtyler · 2010-10-01T17:03:01.622Z · LW(p) · GW(p)
I'd actually guess that at this point in time, a significant chunk of the intelligence of say Silicon Valley believes that the default Kurzweil/Moravec view is correct - AGI will arrive around when Moore's law makes it so.
Of course, neither Kurzweil nor Moravec think any such thing - both have estimates of when a computer with the same processing power as the human brain a considerable while before they think the required software will be developed.
↑ comment by timtyler · 2010-09-11T21:21:19.594Z · LW(p) · GW(p)
The biggest optimist I have come across is Peter Voss. His estimate in 2009 was around 8 years - 7:00 in. However, he obviously has something to sell - so maybe we should not pay too much attention to his opinion - due to the signalling effects associated with confidence.
Replies from: NancyLebovitz↑ comment by NancyLebovitz · 2010-09-11T22:38:00.725Z · LW(p) · GW(p)
Optimist or pessimist?
Replies from: timtyler↑ comment by timtyler · 2010-09-12T07:06:36.124Z · LW(p) · GW(p)
In his own words: Increased Intelligence, Improved Life.
↑ comment by [deleted] · 2010-09-03T16:05:20.911Z · LW(p) · GW(p)
Eliezer addresses point 2 in the comments of the article you linked to in point 2. He's also previously answered the questions of whether he believes he personally could solve FAI and how far out it is -- here, for example.
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T16:26:59.789Z · LW(p) · GW(p)
Thanks for the references, both of which I had seen before.
Concerning Eliezer's response to Scott Aaronson: I agree that there's a huge amount of uncertainty about these things and it's possible that AGI will develop unexpectedly, but don't see how this points in the direction of AGI being likely to be developed within decades. It seems like one could have said the same thing that Eliezer is saying in 1950 or even 1800. See Holden's remarks about noncontingency here.
As for A Premature Word on AI, Eliezer seems to be saying that
Even though the FAI problem is incredibly difficult, it's still worth working on because the returns attached to success would be enormous.
Lots of people who have worked on AGI are mediocre
The field of AI research is not well organized.
Claim (1) might be true. I suspect that both of claims (2) and (3) are true. But by themselves these claims offer essentially no support for the idea that Eliezer is likely to be able to build a Friendly AI.
↑ comment by multifoliaterose · 2010-09-03T12:45:24.089Z · LW(p) · GW(p)
Edit: Should I turn my three comments starting here into a top level posting? I hesitate to do so in light of how draining I've found the process of making top level postings and especially reading and responding to the ensuing comments, but the topic may be sufficiently important to justify the effort.
↑ comment by Wei Dai (Wei_Dai) · 2010-09-03T13:24:49.702Z · LW(p) · GW(p)
exhibiting unwillingness to seriously consider the possibility that he's vastly overestimated his chances of building a Friendly AI
What evidence do you have of this? One reason I doubt that it's true is that Eliezer has been relatively good at admitting flaws in his ideas, even when doing so implied that building FAI is harder than he previously thought. I think you could reasonably argue that he's still overconfident about his chances of successfully building FAI, but I don't see how you get "unwillingness to seriously consider the possibility".
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-03T13:33:14.709Z · LW(p) · GW(p)
What evidence do you have of this?
Eliezer was not willing to engage with my estimate here. See his response. For the reasons that I point out here, I think that my estimate is well grounded.
Eliezer's apparent lack of willingness to engage with me on this point does not immediately imply that he's unwilling to seriously consider the possibility that I raise. But I do see it as strongly suggestive.
As I said in response to ThomBlake, I would be happy to pointed to any of Eliezer's writings which support the idea that Eliezer has given serious consideration to the two points that I raised to explain my estimate.
Edit: I'll also add that given the amount of evidence that I see against the proposition that Eliezer will build a Friendly AI, I have difficulty imagining how he could be persisting in holding his beliefs without having failed to give serious consideration to the possibility that he might be totally wrong. It seems very likely to me that if he had explored this line of thought, he would have a very different world view than he does at present.
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2010-09-04T07:46:48.203Z · LW(p) · GW(p)
I'll also add that given the amount of evidence that I see against the proposition that Eliezer will build a Friendly AI, I have difficulty imagining how he could be persisting in holding his beliefs without having failed to give serious consideration to the possibility that he might be totally wrong.
Have you noticed that many (most?) commenters/voters seem to disagree with your estimate? That's not necessarily strong evidence that your estimate is wrong (in the sense that a Bayesian superintelligence wouldn't assign a probability as low as yours), but it does show that many reasonable and smart people disagree with your estimate even after seriously considering your arguments. To me that implies that Eliezer could disagree with your estimate even after seriously considering your arguments, so I don't think his "persisting in holding his beliefs" offers much evidence for your position that Eliezer exhibited "unwillingness to seriously consider the possibility that he's vastly overestimated his chances of building a Friendly AI".
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-04T11:42:01.976Z · LW(p) · GW(p)
Have you noticed that many (most?) commenters/voters seem to disagree with your estimate?
Yes. Of course, there's a selection effect here - the people on LW are more likely to assign a high probability to the proposition that Eliezer will build a Friendly AI (whether or not there's epistemic reason to do so).
The people outside of LW who I talk to on a regular basis have an estimate in line with my own. I trust these people's judgment more than I trust LW posters judgment simply because I have much more information about their positive track records for making accurate real world judgments than I do for the people on LW.
To me that implies that Eliezer could disagree with your estimate even after seriously considering your arguments, so I don't think his "persisting in holding his beliefs" offers much evidence for your position that Eliezer exhibited "unwillingness to seriously consider the possibility that he's vastly overestimated his chances of building a Friendly AI".
Yes, so I agree that in your epistemological state you should feel this way. I'm explaining why in my epistemological state I feel the way I do.
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2010-09-04T12:16:23.599Z · LW(p) · GW(p)
In your own epistemological state, you may be justified in thinking that Eliezer and other LWers are wrong about his chances of success, but even granting that, I still don't see why you're so sure that Eliezer has failed to "seriously consider the possibility that he's vastly overestimated his chances of building a Friendly AI". Why couldn't he have, like the other LWers apparently did, considered the possibility and then (erroneously, according to your epistemological state) rejected it?
Replies from: multifoliaterose↑ comment by multifoliaterose · 2010-09-04T13:06:46.234Z · LW(p) · GW(p)
Why couldn't he have, like the other LWers apparently did, considered the possibility and then (erroneously, according to your epistemological state) rejected it?
My experience reading Eliezer's writings is that he's very smart and perceptive. I find it implausible that somebody so smart and perceptive could miss something for which there is (in my view) so much evidence for if he had engaged in such consideration. So I think that what you suggest could be the case, but I find is quite unlikely.
↑ comment by Alan · 2010-08-20T03:48:10.427Z · LW(p) · GW(p)
- Jeremy Bentham may be a candidate, or perhaps James Mill, father of J.S. Mill--though there's been some recent speculation that the former fell somewhere on the autism spectrum (no slight intended). By the way, if you're interested, check out the research on shifting modes of moral congition, deontological vs. consequentialist, depending upon subject matter, featured in the work of David Pizarro, e.g. Further afield, one may check out what Taleb has to say about who has led a genuinely Popperian lifestyle.
↑ comment by wedrifid · 2010-08-20T01:13:55.529Z · LW(p) · GW(p)
What is a genuinely utilitarian lifestyle? Is there someone you can cite as living such a lifestyle?
Optimising one's lifestyle for the efficient acquisition of power to enable future creation of bulk quantities of paper-clips. For example.
Replies from: timtyler↑ comment by timtyler · 2010-08-20T20:33:36.166Z · LW(p) · GW(p)
It would be nice if utilitarianism signified that - but in fact it is about the greatest good for the greatest number:
"Utilitarianism (also: utilism) is the idea that the moral worth of an action is determined solely by its utility in providing happiness or pleasure as summed among all sentient beings."
↑ comment by xamdam · 2010-08-20T01:31:04.664Z · LW(p) · GW(p)
Failing to lose hope when he should? (He wrote a post about that, BTW.)
Not sure why this would make sense to the OP; the referenced post talks about situations where you say 'yes', Life says 'Don't this so'. I do not think EY's work hit any obvious walls that are of the showstopper kind.
comment by simplicio · 2010-08-20T01:18:53.609Z · LW(p) · GW(p)
In high school I went through a period when I believed that I was a messianic figure whose existence had been preordained by a watchmaker God who planned for me to save the human race. It's appropriate to say that during this period of time I suffered from extreme delusions of grandeur. I viscerally understand how it's possible to fall into an affective death spiral.
Not that the two are exclusive, but this sounds an awful lot like a manic episode. I assume you gave that due consideration?
comment by Morendil · 2010-08-20T09:11:16.194Z · LW(p) · GW(p)
The mechanism that determines human action is that we do what makes us feel good (at the margin) and refrain from doing what makes us feel bad (at the margin).
"The" mechanism? Citation needed.
a fundamental mechanism of the human brain which was historically correlated with gaining high status is to make us feel good when we have high self-image and feel bad when we have low self-image.
Better, but still unsupported and unclear. What was correlated with what?
comment by ata · 2010-08-20T05:30:54.831Z · LW(p) · GW(p)
I'm inclined to think that Eliezer's confidence in his own importance (actually I'd prefer "expected importance", measured as an expected quantity the usual way) is not really unwarranted (or too unwarranted), but I hope he does take away from this a greater sense of the importance of a "the customer is always right" attitude in managing his image as a public-ish figure. Obviously the customer is not always right, but sometimes you have to act like they are if you want to get/keep them as your customer... justified or not, there seems to be something about this whole endeavour (including but not limited to Eliezer's writings) that makes people think !!!CRAZY!!!, and even if is really they who are the crazy ones, they are nevertheless the people who populate this crazy world we're trying to fix, and the solution can't always just be "read the sequences until you're rational enough to see why this makes sense".
I realize it's a balance; maybe this tone is good for attracting people who are already rational enough to see why this isn't crazy and why this tone has no bearing on the validity of the underlying arguments, like Eliezer's example of lecturing on rationality in a clown suit. Maybe the people who have a problem with it or are scared off by it are not the sort of people who would be willing or able to help much anyway. Maybe if someone is overly wary of associating with a low-status yet extremely important project, they do not have a strong enough grasp of its importance or inclination toward real altruism anyway. But reputation will still probably count for a lot toward what SIAI will eventually be able to accomplish. Maybe at the point of hearing and evaluating the arguments, seeming weird will only screen off people who would not have made important contributions anyway, but it does affect who will get far enough to hear the arguments in the first place. In a world full of physics and math and AI cranks promising imminent world-changing discoveries, reasonably smart people do tend to build up intuitive nonsense-detectors, build up an intuitive sense for who's not even worth listening to or engaging with; if we want more IQ 150+ people to get involved in existential risk reduction, then perhaps SIAI should make a greater point of seeming non-weird long enough for smart outsiders to switch from "save time by evaluating surface weirdness" mode to "take seriously and evaluate arguments directly" mode.
(Meanwhile, I'm glad Eliezer says "I have a policy of keeping my thoughts on Friendly AI to the object level, and not worrying about how important or unimportant that makes me", and I hope he takes that very seriously.)
comment by Jonathan_Graehl · 2010-08-20T01:02:30.260Z · LW(p) · GW(p)
we do what makes us feel good (at the margin) and refrain from doing what makes us feel bad (at the margin)
By "(at the margin)" do you just mean that we're definitely aiming at some local optimum, and possibly missing better (but harder to reach) world/self states? If so, I agree. If not, I don't get your meaning.
Replies from: Matt_Simpson↑ comment by Matt_Simpson · 2010-08-20T01:40:14.793Z · LW(p) · GW(p)
comment by epigeios · 2012-06-12T02:39:23.546Z · LW(p) · GW(p)
Lol. See, I call into question many things about Eliezer. For example, I think he is afraid to be wrong, and has lost himself to the bias of the false duality between "correct" and "incorrect". As a result of this, I think that until he fixes that problem, he will be doomed to never be able to invent true AI, or even understand how neurons work. (there's a tiny chance that I am wrong, and he doesn't think that way; and there is a moderate chance that he is in the process of or has already fixed this)
However, the only thing - about his possible idea that he is the most important person on the planet - that I call into question, is that I don't think he has the guts to go through with it. I, myself, know that I am currently the most important person on the planet. I expect nothing less of anyone who is actually capable of changing the planet. In fact, I hope he changes his mind, and instead decides to be the most important person on the planet, so that he can be of some help to me.
It goes like this. In order to change something, one must take responsibility for it. Following that, in order to change the entire planet, one must take responsibility for the entire planet. Now, that in itself is not enough to be important; it is only the necessary minimum to be influential. In order to be important, one must pursue something one knows to be important. And above that, in order to be the most important person on the planet, one must pursue something one knows to be necessary for the continued development of the planet.
I, myself, am pursuing something I know to be necessary for the development of both the AI singularity and the human singularity; and I know that in the direction I am going, they are the same thing; which means that what I am pursuing is necessary for the future of the human race and the planet as a whole. I know that without the theories I have developed and will develop, true AI is an impossible goal for the rest of humanity. I know that I am not the only person in the world who could possibly come up with these theories, but I know that I am the only person I am aware of who is in pursuit of them. I also know that I am the only person I am aware of who is even capable of pursuing them. So, I might not be the most important person on the planet; but I have never heard of anyone who is more important. And yet I acknowledge that I require the stepping stones left available by other people who were in their own right the most important people on the planet. In order to accomplish what I will accomplish in my lifetime, I have to be at least that important; because I have to be responsible for at least that much.
So in summary: if Eliezer believes himself to be the most important person on the planet, then he doesn't have the guts to follow through on that, and he will fail. If, on the other hand, he knows that he is the most important person on the planet, then he will influence the planet in an extremely important way. I sure hope he's capable of this, because I feel lonely being the only person who's the most important.
Addendum: If he thinks he's the most important in human history, then he's living in the past; and that would be sad :(