On presenting the case for AI risk

Strong upvote; agree with most / all of what you wrote. Having said that:

Isn't this immediately falsified by human beings? ... And isn't it a bit concerning if your alleged generalization breaks down hardest on the most relevant data point we have for trying to predict the impact of automating general intelligence?

I'm not sure how Conor would reply to this, but my models of Paul Christiano and Robin Hanson have some things to say in response. My Paul model says:

Humans were preceded on the evolutionary tree by a number of ancestors, each of which was only slightly worse along the relevant dimensions. It's true that humans crossed something like a supercriticality threshold, which is why they managed to take over the world while e.g. the Neanderthals did not, but the underlying progress curve humans emerged from was in fact highly continuous with humanity's evolutionary predecessors. Thus, humans do not represent a discontinuity in the relevant sense.

To this my Robin model adds:

In fact, even calling it a "supercriticality threshold" connotes too much; the actual thing that enabled humans to succeed where their ancestors did not, was not their improved (individual) intelligence relative to said ancestors, but their ability to transmit discoveries from one generation to the next. This ability, "cultural evolution", permits faster iteration on successful strategies than does the mutation-and-selection procedure employed by natural selection, and thus explains the success of early humans -- but it does not permit for a new-and-improved AGI to come along and obsolete humans in the blink of an eye.

Of course, I have to give my Eliezer model (who I agree with more than either of the above) a chance to reply:

Paul: It's all well and good to look back in hindsight and note that some seemingly discontinuous outcome emerged from a continuous underlying process, but this does not weaken the point -- if anything, it strengthens it. The fact that a small, continuous change to underlying genetic parameters resulted in a massive increase in fitness shows that the function mapping design space to outcome space is extremely jumpy, which means that roughly continuous progress in design space does not imply a similarly continuous rate of change in real-world impact; and the latter is what matters for AGI.

Robin: From an empirical standpoint, AlphaGo Zero is already quite a strong mark against the "cultural evolution" hypothesis. But from a more theoretical standpoint, note that (according to your own explanation!) the reason "cultural evolution" outcompetes natural selection is because the former iterates more quickly than the latter; this means that it is speed of iteration that is the real underlying driver of progress. Then, if there exists a process that permits yet faster iteration, it stands to reason that that process would outcompete "cultural evolution" in precisely the same way. Thinking about "cultural evolution" gives you no evidence either way as to whether such a faster process exists, which essentially means the "cultural evolution" hypothesis tells you nothing about whether / how quickly AGI can surpass the sum total of humanity's ability / knowledge, after being created.

Human beings entered into a world without intelligence, but machine intelligence will be entering a world where humans, corporations, governments and societies will be doing everything they can to control and monitor the AI. It took human beings hundreds of thousands of years to go from knowing nothing other than how to get food to knowing enough and being powerful enough to really start threatening e.g. the biosphere. AIs could go from pretty dumb to super intelligent very quickly, sure, but how long will it take to go from powerless to world domination? With humans doing our best to control AIs and resist?

↑ comment by Ben Pace (Benito) · 2022-03-13T17:07:46.699Z · LW(p) · GW(p)

a world where humans, corporations, governments and societies will be doing everything they can to control and monitor the AI

I think I have to claim this as wishful thinking. Were humans doing everything they could to control and monitor the coronavirus? No, and to say such a thing is to be telling fairy-tale stories, not describing the current human world.

Replies from: conor-sullivan

↑ comment by Lone Pine (conor-sullivan) · 2022-03-16T00:54:41.105Z · LW(p) · GW(p)

I think you and I have a very different impression of the pandemic then. No pandemic in history was more closely monitored. We did more to try to control this pandemic than any health event ever. Also, humanity didn't literally go extinct, as is claimed for alignment.

Replies from: Benito

↑ comment by Ben Pace (Benito) · 2022-03-16T01:37:41.310Z · LW(p) · GW(p)

Here is a quick drawing I did to communicate my point, and in particular to show the chasm between "best pandemic-response so far" and "doing everything we can".

I believe with quite high confidence that with a bit of test&trace and some challenge trials, this pandemic could've been over in 3-5 months, instead of over 2 years. Every part of this seems simple to me (the mRNA vaccines were invented in <48 hours in January, challenge trials require only like 100s of people for ~2 weeks to get confident results, I think a company with the logistical competence of Amazon could've gotten a country vaccinated in just a couple of months, etc). So it looks to me like we're very very far from "Doing Everything We Can", so even if as you say we did better than ever before, we still didn't get a passing grade according to me.

Recap: I'm making this point because you said you're expecting a world where we're doing everything we can, I gave the counterexample of Covid for our collective competence, and you said you thought we did better than any other pandemic. This isn't a crux for me because there were lots of major easy wins we could've had which we did not. Our response looks to me like "not getting a passing grade" with the resources we had, and not really using most of the resources we had, while scoring lots of own-goals in the process.

AI research will progress with or without one man's ideas. EY is impeding alignment research by choking out object-level discussion, without any benefit. Him just entertaining the idea of AGI following one particular path, be it deep learning or bayes nets or whole brain emulation, would give the discussion a basis to start from. Just sharing ideas wouldn't do anything to push forward capabilities research unless he inadvertently convinced researchers put in the work to implement his ideas. Ideas don't implement themselves.

I think the "step-function" thing is more that he believes that these things will follow exponential/logistic curves, which can easily look like step-functions to humans, but are still perfectly lawful and continuous.

This is completely different from my Eliezer-model. My Eliezer-model says:

• Every invention is a step change, if it involves a step where you go from 'not knowing how to get the thing working' to 'knowing how to get the thing working'; or if it involves a step where the thing goes from 'not working' to 'working'.
• Physics is continuous, so there are always ways to look at step changes like these and find some continuous process underlying it, with the benefit of hindsight. But this isn't very useful, because it's not something we're good at doing with foresight. (If it were, there would be someone in the world with a good empirical track record of timing when different inventions will happen years in advance.)
• Macroeconomic growth isn't a smooth curve because physics is continuous; nor is it a smooth curve because there's some law of nature saying 'everything has to be continuous'. Rather, it's a smooth curve because it's built out of inventions, trades, etc. that are individually low-impact, so no one event dominates. Zoom out on a large number of minor step changes and you get something that isn't itself a step change.
• The difference between AGI and nukes on the one hand, and most other inventions on the other hand, isn't that AGI or nukes are 'more step-function-y' than Microsoft Word, the Wright Flyer, reggae music, etc. It's that AGI and nukes are higher-impact than Microsoft Word, the Wright Flyer, etc., so the same zero-to-one leap that produces a small impact when it's 'you've finally gotten your word processor to start displaying text', produces a large impact when it's 'you've finally gotten your AI to start modeling messy physical paths-through-time'.
• And the reason inventing the first AGI or the first nuke has a larger impact than inventing the first reggae song, is because the physics of general intelligence and of nuclear chain reactions just entails they're very powerful processes. If you invent a weapon that increases energy yield by 2x, then you'll have a 2x-ish impact on the world; if you invent a weapon that increases energy yield by 1000x, then you'll have a 1000x-ish impact on the world.

Quoting [LW · GW] EY:

[...] Before anybody built the first critical nuclear pile in a squash court at the University of Chicago, was there a pile that was almost but not quite critical? Yes, one hour earlier. Did people already build nuclear systems and experiment with them? Yes, but they didn't have much in the way of net power output. Did the Wright Brothers build prototypes before the Flyer? Yes, but they weren't prototypes that flew but 80% slower.

I guarantee you that, whatever the fast takeoff scenario, there will be some way to look over the development history, and nod wisely and say, "Ah, yes, see, this was not unprecedented, here are these earlier systems which presaged the final system!" Maybe you could even look back to today and say that about GPT-3, yup, totally presaging stuff all over the place, great. But it isn't transforming society because it's not over the social-transformation threshold.

AlphaFold presaged AlphaFold 2 but AlphaFold 2 is good enough to start replacing other ways of determining protein conformations and AlphaFold is not; and then neither of those has much impacted the real world, because in the real world we can already design a vaccine in a day and the rest of the time is bureaucratic time rather than technology time, and that goes on until we have an AI over the threshold to bypass bureaucracy.

[...] There is not necessarily a presage of 9/11 where somebody flies a small plane into a building and kills 100 people, before anybody flies 4 big planes into 3 buildings and kills 3000 people; and even if there is some presaging event like that, which would not surprise me at all, the rest of the world's response to the two cases was evidently discontinuous. You do not necessarily wake up to a news story that is 10% of the news story of 2001/09/11, one year before 2001/09/11, written in 10% of the font size on the front page of the paper.

Physics is continuous but it doesn't always yield things that "look smooth to a human brain". Some kinds of processes converge to continuity in strong ways where you can throw discontinuous things in them and they still end up continuous, which is among the reasons why I expect world GDP to stay on trend up until the world ends abruptly; because world GDP is one of those things that wants to stay on a track, and an AGI building a nanosystem can go off that track without being pushed back onto it.

I guess the crux here is that I don't think HLMI need be so narrowly focused on mathematically specified goals. I think there is a kind of intelligence that doesn't need goals to operate, and also a kind of intelligence that can actually understand that there is a "goal behind every goal" (aka theory of mind). A non-goal directed AI might be driven by creativity or a desire to learn. It might be like GPT-3 where it's just trying to predict some complex input, but not in a goal-y way. You can obviously frame these things as goals, like in DeepMind's paper "Reward is Enough", but I do think there is a sense that intelligence doesn't absolutely need to be goal-y. Yes, this gets into the oracle vs task vs agent AI debate.

Alignment arguments seem to imagine an "asocial AI" which can't understand humans but can perform marvelously at narrowly focused mathematically specified goals. For example, "design a nanotech pathogen to kill every human on the planet" but not "convince every human on the planet to believe in my favored political ideology." Such an AI would certainly be dangerous. The existence proof that intelligence exists, the human brain, also gives us strong reason to believe that an AI could be built which is not asocial and also not sociopathic. The big American tech companies are most interested in this kind of highly social AI, since they are soft powers and need charasmatic AI which people will ultimately like and not reject.

In my opinion the asocial AI is not "real intelligence" and that's what average people and tech company CEOs will say about it. A "real AI" can have a conversation with a person, and if the person says "your goal is bad, here's why, please do this instead," the AI would switch goals. Perhaps that can be framed as a meta-goal such as "do what humans say (or mean)", but I expect (with some probability) that by the time we get to "real AI" we'll have some out-of-the-box thinking and have a much better idea of how to make corrigible AI.

In this framework, if we can make highly social AI before we get extremely powerful asocial AI, then we are safe. But if extremely powerful AI comes before we teach AI to be social, then there is big danger of an alignment problem.

Why would a program that’s been built with goal function A suddenly switch over to using goal function B just because it’s become smart enough to understand goal function B

Why would a programme have a goal function that's complete separate from everything else? Our current most advanced AIs don't. If it did, why would you want one implementation of human semantics in the goal function, and another one in the implementation function? Why reduplicate the effort?