"Dangers of AI and the End of Human Civilization" Yudkowsky on Lex Fridman

Given the size and demographics of Lex's audience, I think this is more likely to bring a large influx of new users to LW and adjacent communities than either the Bankless podcast or the TIME article.

In the long term, that might be a good thing, but in the meantime, it's probably a good idea to keep this classic post [LW · GW] in mind.

Sheesh. Wild conversation. While I felt Lex was often missing the points Eliezer was saying, I'm glad he gave him the space and time to speak. Unfortunately, it felt like the conversation would keep moving towards reaching a super critical important insight that Eliezer wanted Lex to understand, and then Lex would just change the topic onto something else, and then Eliezer just had to begin building towards a new insight. Regardless, I appreciate that Lex and Eliezer thoroughly engaged with each other; this will probably spark good dialogue and get more people interested in the field. I'm glad it happened. 

For those who are time constrained and wondering what is in it: Lex and Eliezer basically cover a whole bunch of high-level points related to AI not-kill-everyone-ism, delving into various thought experiments and concepts which formulate Eliezer's worldview. Nothing super novel that you probably haven't heard of if you've been following the field for some time. 

I think this was good, but I really think laying out the basic arguments for convergent instrumental goals is a foundational part of introducing the topic to a new audience (which I expect is the case for most of Lex's listeners) which wasnt sufficiently explained.

Making it clear that most innocuous goals beget resource acquisition and self preservation which is what puts an agentic AI in conflict with humans by default is what really makes the concern concrete for many people. Otherwise I think there is a tendancy to assume that some leap is required for a system to be in conflict which is much harder to picture and seems intuitively more preventable.

I disagree with Eliezer Yudkowsky on a lot, but one thing I can say for his credibility is that in possible futures where he's right, nobody will be around to laud his correctness, and in possible futures where he's wrong, it will arguably be very clear how wrong his views were. Even if he has a big ego (as Lex Fridman suggested), this is a good reason to view his position as sincere and—dare I say it—selfless.

LEX: Is there a way to measure general intelligence? I mean, I could ask that question in a million ways but basically, will you know it when you see it? It being in an AGI system?

YUD: Heh. If you boil a frog gradually enough, if you zoom in far enough, it's always hard to tell around the edges. GPT-4, people are saying right now, "Like, this looks to us like a spark of general intelligence. It is like able to do all these things it was not explicitly optimized for." Other people are being like, "No its too early, its like, like fifty years off." And you know if they say that they're kind of whack because how could they possibly know that even if it were true. But uh, you know - not to strawman - but some people may say like "That's not general intelligence" and not furthermore append "It's 50 years off." 

Uhm, or they may be like, "It's only a very tiny amount." And you know, the thing I would worry about is if is this how things are scaling then - jumping out ahead and trying not to be wrong in the same way that I've been wrong before - maybe GPT-5 is more unambiguously a general intelligence and maybe that is getting to a point where it is like even harder to turn back. Not that it would be easy to turn back now but you know maybe if you let, if you like start integrating GPT-5 into the economy it is even hard to turn back past there.

LEX: Isn't it possible that there's a you know, with the frog metaphor, you can kiss the frog and turn it into a prince as you're boiling it? Could there be a phase shift in the frog where it's unambiguous as you're saying?

YUD: I was expecting more of that. I was . . . like the fact that GPT-4 is kind of on the threshold and neither here nor there. Like that itself is like not the sort of thing, that's not quite how I expected it to play out. I was expecting there to be more of an issue, more of a sense of like, different discoveries. Like the discovery of transformers where you would stack them up, and there would be like a final discovery, and then you would like get something that was like more clearly general intelligence. So the the way that you are like taking what is probably basically the same architecture in GPT-3 and throwing 20 times as much compute at it, probably, and getting out GPT-4.  And then it's like maybe just barely a general intelligence or like a narrow general intelligence or you know something we don't really have the words for. Uhm yeah,that is not quite how I expected it to play out.

This somewhat confusing exchange is another indicator that the "general intelligence" component of AGI functions a lot better as a stylized concept than it does when applied to the real world. But it is also a clear indicator that EY thinks that recent AI developments and deployments look more like slow takeoff than fast takeoff, at least relative to his expectations.

Fantastic interview so far, this part blew my mind:

@15:50 "There's another moment where somebody is asking Bing about: I fed my kid green potatoes and they have the following symptoms and Being is like that's solanine poisoning. Call an ambulance! And the person is like I can't afford an ambulance, I guess if this is time for my kid to go that's God's will and the main Bing thread gives the message of I cannot talk about  this anymore"  and the suggested replies to it say  "please don't give up on your child, solanine poisoning can be treated if caught early"

I would normally dismiss such story as too unlikely to be true and hardly worth considering, but I don't think Eliezer would chose to mention it if he didn't think there was at least some chance of it being true. I tried to google it and was unable to find anything about it. Does anyone have a link to it?

Also does anyone know which image he's referring to in this part: @14:00 "Somebody asked Bing Sydney to describe herself and fed the resulting description into one of the stable diffusion" [...] "the pretty picture of the girl with the with the steampunk goggles on her head if I'm remembering correctly"

Around the 1:25:00 mark, I'm not sure I agree with Yudkowsky's point about AI not being able to help with alignment only(?) because those systems will be trained to get the thumbs up from the humans and not to give the real answers. 

For example, if the Wright brothers had asked me about how wings produce lift, I may have only told them "It's Bernoulli's principle, and here's how that works..." and spoken nothing about the Coanda effect - which they also needed to know about - because it was just enough to get the thumbs up from them. But...

But that still would've been a big step in the right direction for them. They could've then run experiments and seen that Bernoulli's principle doesn't explain the full story, and then asked me for more information, and at that point I would've had to have told them about the Coanda effect. 

There's also the possibility that what gets the thumbs up from the humans actually just is the truth. 

For another example, if I ask a weak AGI for the cube root of 148,877 the only answer that gets a thumbs up is going to be 53, because I can easily check that answer. 

So long as you remain skeptical and keep trying to learn more, I'm not seeing the issue. And of course, hanging over your head the entire time is the knowledge of exactly what the AGI is doing, so anyone with half a brain WOULD remain skeptical. 

This could potentially also get you into a feedback loop of the weak explanations allowing you to slightly better align the AGI you're using, which can then make it give you better answers. 

Yudkowsky may have other reasons for thinking that weak AGI can't help us in this way though, so IDK. 

Hi, I'm new here so I bet I'm missing some important context. I listen to Lex's podcast and have only engaged with a small portion of Yud's work. But I wanted to make some comments on the analogy of a fast human in a box vs. the alien species. Yud said he's been workshopping this analogy for a while, so I thought I would leave a comment on what I think the analogy is still missing for me. In short, I think the human-in-a-box-in-an-alien-world analogy smuggles in an assumption of alienness and I'd like to make this assumption more explicit.

Before I delve into any criticism of the analogy, I'd like to give credit where it's due! I think the analogy is great as a way to imagine a substantial difference in intelligence, which (I think?) was the primary goal. It is indeed much more concrete and helpful than trying to imagine something several times more intelligent than von Neumann, which is hard and makes my brain shut off.

Now, let me provide some context from the conversation. The most relevant timestamp to illustrate this is around 2:49:50. Lex tries to bring the conversation to the human data used to train these models, which Yud discounts as a mere "shadow" of real humanness. Lex pushes back against this a bit, possibly indicating he thinks there is more value to be derived from the data ("Don't you think that shadow is a Jungian shadow?") but Yud insists that this would give an alien a good idea of what humans are like inside, but "this does not mean that if you have a loss function of predicting the next token from that dataset, the mind picked out by gradient descent... is itself a human." To me, this is a fair point. Lex asks whether those tokens have a deep humanness in them, and Yud goes back to a similar person-in-a-box analogy: "I think if you sent me to a distant galaxy with aliens that are much stupider than I am..." 

Okay, that should be enough context. Basically, I think Yud has an intuition that artificial intelligence will be fundamentally alien to us. Evidence provided for this intuition I heard in the conversation is that gradient descent is different than natural selection. More evidence is the difference between human brain function and large matrices + linear algebra approaches to problem solving.

I, who have not thought about this anywhere close to as much as Yud but insist on talking anyway, don't share these intuitions about AI, because I don't see how the differences in substrate/instantiation of problem-solving mechanism or choice of optimizer would fundamentally affect the outcome. For example, if I want to find the maxima of a function, it doesn't matter if I use conjugate gradient descent or Newton's method or interpolation methods or whatever, they will tend to find the same maxima assuming they are looking at the same function. (Note that there are potential issues here because some techniques are better suited to certain types of functions, and I could see an argument that the nature of the function is such that different optimization techniques would find different maxima. If you think that, I'd love to hear more about why you think that is the case!). As far as substrate independence, I don't have any strong evidence for this, other than saying that skinning a cat is skinning a cat.

I tend to think that the data is more important when considering how the AI is likely to behave, and as Lex points out, the data is all deeply human. The data contains a lot of what it means to be human, and training an AI on this data would only cause an alien actress (as Yud puts it) to fake humanness and kill us all if it is an alien. But really, would it be an alien? In a sense, it was "raised" by humans, using only human stuff. It's framework is constructed of human concepts and perspectives. To me, based on the huge amount of human data used to train it, it seems more parsimonious that it would be extra human, not alien at all, really. 

I think a better analogy than human-in-a-box-in-an-alien-world is human-in-a-box-in-a-HUMAN-world. Following the original analogy, let's put sped-up von Neumann back in the box, but the box isn't in a world full of stupid aliens, it's full of stupid humans. I don't think von Neumann (or an army of hundreds of von Neumann's, etc.) would try to kill everyone even if they disagreed with factory farming, or whatever cause they have to try to control the world and make it different than it is. I think the von Neumann army would see us as like them, not fundamentally alien.

Thanks for reading this long post, I'd be interested to see what you all think about this. As I mentioned, there's certainly a good deal of context backing up Yud's intuitions that I'm missing.

I don't understand why Eliezer changed his perspective about the current approach of Transformer next-token prediction not being the path towards AGI. It should not be surprising that newer versions of GPT will asymptotically approach (mimicry) of AGI, but that shouldn't convince anyone that they are going to break through that barrier without a change in paradigm. All the intelligent organisms we know of do not have imitation as their primary optimization objective - their objective function is basically to survive or avoid pain. As a result, they of course form sub-goals which might include imitation, but only to the extent that it is instrumental to their survival. Optimizing 100% for imitation does not lead to AGI, because how can novelty emerge from nothing but imitation.