Bankless Podcast: 159 - We’re All Gonna Die with Eliezer Yudkowsky

I can't even get a good answer of "What's the GiveWell of AI Safety" so I can quickly donate to a very reputable and agreed upon option with little thinking without at best getting old lists to a ton of random small orgs and giving up. I'm not very optimistic ordinary less convinced people who want to help are having an easier time.

I’d encourage you to do that.

I've thought for awhile here that the primary problem in alarmism (I'm one) is painting a concise, believable picture.  It takes a willful effort and  open mind to build a "realistic" picture of this here-to-fore unknown mechanism/organism for oneself, and is near impossible to do for someone who is skeptical or opposed to the potential conclusion.

Being a web dev I've brainstormed on occasion ways to build short, crowd-ranked chains of causal steps which people could evaluate for themselves, with various doubts and supporting evidence given to each.   It's still a vague vision which is why I raise it here, to see if anyone else wants to get involved on that abstract design level.  ( hmu )

I think the general problem of painting a believable picture could be solved by a number of different mediums though.  Unfortunately we're drowning in inaccurate or indulgent dystopias which end up creating a "boy who cried wolf" effect for earnest ones.

A superintelligent AI could fail to achieve its goals and still get us all killed. For example, it could design a virus that kills all humans, and then find out that there were some parts of economy it did not understand, so it runs out of resources and dies, without converting the galaxy to paperclips. Nonetheless, humans remain dead.

It's hard to manufacture things, but it's not that hard to do so in a way that pretty much can't kill you. Just keep the computation that is you at somewhat of a distance from the physical experiment, and don't make stuff that might consume the whole earth. Making a general intelligence is an extraordinary special case: if you're actually doing it, it might self-improve and then kill you.

It is the very same rationale that stands behind assumptions like "why Stockfish won't execute losing set of moves" - it is just that good at chess. Or better - it is just that smart when it come down to chess.

In this thought experiment the way to go is not to "i see that AGI could likely fail at this step, therefore it will fail" but to keep thinking and inventing better moves for AGI to execute, which won't be countered as easily. It is an important part of "security mindset" and probably major reason why Eliezer speaks about lack of pessimism in the field.

That part was a bit unclear. I guess he could work with redwood/conjecture without necessarily quitting his MIRI position?

We simply don't know how things are going to turn out

His model does say how things are going to turn out: with everyone dying.

Seriously, people are looking up to you. That's not how you treat them.

What's the preferred alternative? Lying or withholding relevant arguments, out of general principle, without even an expected benefit to anyone?

I think there's a kind of division of labor going on, and I'm going to use a software industry metaphor. If you're redteaming, auditing, or QAing at a large org, you should really be maxing out on paranoia, being glass half empty, etc. because you believe that elsewhere in the institution, other peoples' jobs are to consider your advice and weigh it against the risk tolerance implied by the budget or by regulation or whatever. Whereas I think redteaming, auditing, or QAing at a small org you kind of take on some of the responsibility of measuring threatmodels against given cost constraints. It's not exactly obvious that someone else in the org will rationally integrate information you provide into the organization's strategy and implementation, you want them to follow your recommendations in a way that makes business sense.

My guess is that being a downer comes from this large org register of a redteam's job description being literally just redteam, whereas it might make sense for other researchers or communicators to take a more small org approach where the redteam is probably multitasking in some way.

Intuition pump: I don't really know a citation, but I once saw a remark that the commercial airline crash rate in the late soviet union was plausibly more rational than the commercial airline crash rate in the US. Airplane risk intolerance in the US is great for QA jobs, but that doesn't mean it's based on an optimal tradeoff between price and safety with respect to stakeholder preferences (if you could elicit them in some way). Economists make related remarks re nuclear energy.

I just posted a full transcript on LW here! [LW · GW]

click on 3 dots (if on a browser) and there is a transcript option in the pop-up menu.

See https://www.lesswrong.com/posts/8gqrbnW758qjHFTrH/security-mindset-and-ordinary-paranoia [LW · GW] and https://www.lesswrong.com/posts/cpdsMuAHSWhWnKdog/security-mindset-and-the-logistic-success-curve [LW · GW] for Yudkowsky's longform explanation of the metaphor.

I believe you are correct, however, EY argues that a sufficiently intelligent AGI might be able to hack biology and use ribosomes and protein machinery to skip straight to diamondoid self replicating nanotechnology.

I suspect this is impossible - that is, there may exist a sequence of steps that would work and achieve this, but it does not mean that the information exists within the pool of [all scientific data collected by humans] to calculate what those steps are.  

Instead you would have to do this iteratively, similar to how humans would do it.  Secure a large number of STM microscopes and vacuum chambers.  Build, using electron beam lathes or other methods, small parts to test nanoscale bonding strategies.  Test many variants and develop a simulation sufficient to design nanoscale machine parts.  Iteratively use the prior data to design and test ever larger and more sophisticated assemblies.  Then, once you have a simulated path to success and high enough confidence, bootstrap a nanoforge.  (bootstrap means you try to choose a path where prior steps on the path make future steps easier)

An ASI holding everyone hostage isn't a winning scenario for the ASI.  Humans are just going to pull the trigger on their own nuclear guns in such a scenario.  

The EY scenarios where the ASI wins generally involve deception.  Everything is fine, until everyone dies all at the same time from some type of immediately lethal but delayed bioweapon or nanotechnology based weapon.  

Botulism toxin is one way this is theoretically achievable - it takes a very small quantity to kill a human.  So a 'time bomb' of a capsule of it, injected painlessly into most humans using nanotechnology, or a virus that edits our genome and inserts the botulism gene and some mechanism to prevent expression for a few months, or similar method.  For one thing, botulism toxin is a protein and is probably much larger than it needs to be...

More specifically, there's something pretty valuable about conversations like this. Many "reputable / intelligent" people will falsely believe that they understand Eliezer's position. Such people will often direct the conversation towards non-central points, and generally be less available to change their minds and attend to central points. In a conversation with "dim-witted" or naive hosts, the key points are expressed clearly and repeatedly. That sort of thing often gets through to "reputable / intelligent" listeners better than more advanced conversations.

Ultimately not constructive 

Consider the target audience of this podcast.

I think there's something you're right about: Some people more familiar with the arguments could've given better pushback to the positions, or probed in a more critical way. Perhaps the hosts were chosen for having had little exposure to the existing arguments.