Carl Shulman on The Lunar Society (7 hour, two-part podcast)

Carl [LW · GW], your vision (which I guess is fairly mainstream in AI alignment these days) involves having a powerful civilization be controlled by something weak and fragile (humans), but I worry that humans are both prone to responding in crazy ways to many kinds of inputs^[1] and being optimized against by superintelligences (since they are rewarded/reinforced by evaluations from humans). I feel like a lot of the stuff you talk about in this podcast (mind reading, adversarial training) as being responsible for your optimism mostly just shifts probability mass from the more obvious bad outcomes (deception, takeover) to less obvious bad outcomes (humans going crazy, failing to converge to correct values). It's hard for me to get nearly as hopeful as you seem, without being able to see some kind of solution that has the shape of "here are all the ways that humans are defective/unsafe, and ways to patch those holes or avoid them in human-AI interactions" (or some other kind of solution that avoids depending on humans for x-safety at all), unless "hopeful" is defined narrowly as solving intent-alignment or avoiding takeover.

To put it another way, being easily deceived is just one way that humans are defective/unsafe, but we (i.e., people in the AI alignment/x-risk space) seem to be focused on it almost to the complete exclusion of other human-AI safety problems, which seems bad from both a technical perspective (how do we solve the other problems if nobody is working on them) and social/governance perspective (society is liable to be too optimistic about the AI transition going well because nobody hears about the less obvious problems). As evidence for the neglect, I note that interstice's comment [LW(p) · GW(p)] to Paul Christiano's My views on “doom” [LW · GW]remains unanswered:

Interesting that you think the risk of us “going crazy” after getting AI in some way is roughly comparable to overall AI takeover risk. I’d be interested to hear more if you have more detailed thoughts here. On this view it also seems like it could be a great x-risk-reduction opportunity if there are any tractable strategies, given how neglected it is compared to takeover risk.

1. ^{^}

   for example, getting addicted, believing in obviously crazy conspiracy theories, refusing to speak or act in accordance with obvious truths [LW(p) · GW(p)] because they're low status / outside the overton window, getting sucked into non-effective or counterproductive causes

Here how you can talk about bombing unlicensed datacenters without using "strike" and "bombing".

If we can probe the thoughts and motivations of an AI and discover wow, actually GPT-6 is planning to takeover the world if it ever gets the chance. That would be an incredibly valuable thing for governments to coordinate around because it would remove a lot of the uncertainty, it would be easier to agree that this was important, to have more give on other dimensions and to have mutual trust that the other side actually also cares about this because you can't always know what another person or another government is thinking but you can see the objective situation in which they're deciding. So if there's strong evidence in a world where there is high risk of that risk because we've been able to show actually things like the intentional planning of AIs to do a takeover or being able to show model situations on a smaller scale of that I mean not only are we more motivated to prevent it but we update to think the other side is more likely to cooperate with us and so it's doubly beneficial.

Here is alternative to dangerous experiments to develop enhanced cognition in humans. Sounds less extreme and little more doable.

Just going from less than 1% of the effort being put into AI to 5% or 10% of the effort or 50% or 90% would be an absolutely massive increase in the amount of work that has been done on alignment, on mind reading AIs in an adversarial context. 

If it's the case that as more and more of this work can be automated and say governments require that you put 50% or 90% of the budget of AI activity into these problems of make this system one that's not going to overthrow our own government or is not going to destroy the human species then the proportional increase in alignment can be very large even just within the range of what we could have done if we had been on the ball and having humanity's scientific energies going into the problem. Stuff that is not incomprehensible, that is in some sense is just doing the obvious things that we should have done.

Also pretty bizarre that in response to 

Dwarkesh Patel 02:18:27

So how do we make sure it's not the thing it learns is not to manipulate us into rewarding it when we catch it not lying but rather to universally be aligned.

Carl Shulman 02:18:41

Yeah, so this is tricky. Geoff Hinton was recently saying there is currently no known solution for this. 

The answer was: yes, but we are doing it anyway. But with a twists like adversarial examples, adversarial training and simulations. If Shulman had THE ANSWER to Alignment problem then he would not kept it secret, but i cant help but feel some disappointment, because he sounds SO hopeful and confident. I somehow expected something different than variation of "we are going to us weaker AIs to help us to align stronger AIs while trying to outrun capabilities research teams". Even if this variation (in his description) seems very sophisticated with mind reading and inducing hallucinations.

I wanted to read through the transcript but found it a bit overwhelming. I made two google docs for the Pt 1 and Pt 2 transcript, where people can comment on bits that seemed particularly interesting. This seemed maybe helpful for an impromptu reading group.

Part 1

Part 2

I've only listened to part 1 so far, and I found the discussion of intelligence explosion to be especially fresh. (That's hard to do given the flood of AI takes!) In particular (from memory, so I apologize for errors):

• By analogy to chip compute scaling as a function of researcher population, it makes super-exponential growth seem possible if AI-compute-increase is substituted for researcher-population-increase. A particularly interesting aspect of this is that the answer could have come out the other way if the numbers had worked out differently as Moore's law progressed. (It's always nice to give reality a chance to prove you wrong.)
• The intelligence explosion starts before human-level AI. But I was left wanting to know more: if so, how do we know when we've crossed the inflection point into the intelligence explosion? Is it possible that we're already in an intelligence explosion, since AlexNet, or Google's founding, or the creation of the internet, or even the invention of digital computers? And I thought Patel's point about the difficulty of automating a "portfolio of tasks" was great and not entirely addressed.
• The view of intelligence explosion as consisting concretely of increases in AI researcher productivity, though I've seen it observed elsewhere, was good to hear again. It helps connect the abstract concept of intelligence explosion to how it could play out in the real world.