AI Views Snapshots

Thank you for this. Here's mine:

1. It’s physically possible to build STEM+ AI (though it's OK if we collectively decide not to build it.)
2. STEM+ AI will exist by the year 2035 but not by 2100 (human scientists will improve significantly thanks to cohering blending volition [? · GW], aka ⿻Plurality).
3. If STEM+AI is built, within 10 years AIs will be able to disempower humanity (but won't do it.)
4. The future will be better if AI does not wipe out humanity.
5. Given sufficient technical knowledge, humanity could in principle build vastly superhuman AIs that reliably produce very good outcomes.
6. Researchers will produce good enough processes for continuous alignment.
7. Technical research and policy work are ~equally useful.
8. Governments will generally be reasonable in how they handle AI risk.

Always glad to update as new evidence arrives.

Mine:

My answer to "If AI wipes out humanity and colonizes the universe itself, the future will go about as well as if humanity had survived (or better)" is pretty much defined by how the question is interpreted. It could swing pretty wildly, but the obvious interpretation seems ~tautologically bad.

Great idea! Here's mine:
 

"it's physically impossible" should probably be "it's physically possible" to avoid an unnecessary inversion. also, it needs something about "aligned to whom", as I and many others believe it is either very important or the same technical problem. I added brightness colorcoding to help our mostly-brightness-sensitive vision systems differentiate. It would be good to have an easy to use picker page that would produce something like this, eg perhaps with coda.

here's mine

Nice. One thing: initially I couldn't figure out how to read this because I didn't see the key at the top. I think the key is a bit too easy to miss if you are zooming in to look at the image on mobile. Maybe make it more prominent?

Here's a spreadsheet version you can copy. Fill your answers in the "answers" tab, make your screenshot from the "view" tab.

I plan to add more functionality to this (especially comparison mode, as I collect some answers found on the Internet). You can now compare between recorded answers! Including yours, if you have filled them!

I will attempt to collect existing answers, from X and LW/EA comments.

If ASIs don't initially disempower humanity, even though they could, uplifted humans or AI-developed interventions might then make it impossible for AIs to disempower humanity. This makes the statement "within 10 years AIs will be able to disempower humanity" ambiguous between there being a point within the 10 year interval where this holds, and this holding at the end of the interval.

Seems interesting! Here are my answers, along with a couple comments on things I was confused about.

I really like this! (here's mine [LW(p) · GW(p)])

A few questions:

• The first time AI reaches STEM+ capabilities (if that ever happpens), it will disempower humanity within three months

  So this is asking for P(fasttakeoff and unaligned | STEM+) ? It feels weird that it's asking for both. Unless you count aligned-AI-takeover as "disempowering" humanity. Asking for either P(fasttakeoff | STEM+) or P(fasttakeoff | unaligned and STEM+) would make more sense, I think.

• Do you count aligned-AI-takeover (where an aligned AI takes over everything and creates an at-least-okay [LW · GW] utopia) as "disempowering humanity"?

• "reasonable and informed" is doing a lot of work here — is that left to be left to the reader, or should there be some notion of what rough amount of people you expect that to be? I think that, given the definitions I filled my chart with, I would say that there are <1000 people on earth right now who fit this description (possibly <100).

I think I'm going to put a low probability on "within 10 years AIs will be able to disempower humanity".

i think merely being good at science is in no way sufficient to be able to do that; would require a bunch of additional factors (e.g. power seeking, ability to persuade humans, etc etc)

but on the other hand, I think Yann Lecun is way too complacent when he imagines that intelligent AIs will be just like high IQ humans employed by corporations. At a slight risk of being constraversial,  i would suggest that e.g. Hamas members are within the range of human behaviours within the training set that an AI could choose to emulate.

Here's mine

I'm most willing to hear meta-level arguments about internal consistency, or specific existing evidence that I don't know about (especially "secret" evidence). Less certain about the governance sections and some of the exact-wordings.

Alright, I finally have my AI snapshot, so I'll post it below:

https://docs.google.com/drawings/d/1mx2mcblkC0w74hnjN_fPMnDgk4jMG1iu5H_LITw7lVs/edit?usp=sharing

Since I can't edit the spreadsheet, here are my answers to the questions posed in a comment:

It is physically impossible to ever build STEM+ AI: less than 1%, probably far lower, but at any rate I don't currently see a reasonable way to get to any other probability that doesn't involve "known science is wrong only in the capabilities of computation." And I suspect the set of such worlds that actually exist is ~empty, for many reasons, so I see no reason to privilege the hypothesis.

STEM+ AI will exist by the year 2035: I'll be deferring to this post for my model and all probabilities generated by it, since I think it's good enough for the effort it made:

https://www.lesswrong.com/posts/3nMpdmt8LrzxQnkGp/ai-timelines-via-cumulative-optimization-power-less-long [LW · GW]

STEM+ AI will exist by the year 2100: I'll be deferring to this post for my model and all probabilities generated by it, since I think it's good enough for the effort it made:

https://www.lesswrong.com/posts/3nMpdmt8LrzxQnkGp/ai-timelines-via-cumulative-optimization-power-less-long [LW · GW]

If STEM+AI is built, within 10 years AIs will be (individually or collectively) able to disempower humanity: I'd say that a lot of probability ranges are reasonable, though I tend to be a bit on the lower end of 1-50% because of time and regulatory constraints.

If STEM+AI is built, within 10 years AIs will disempower humanity: I'd put it at less than 1% chance, I'm confident that 10% probability is wrong, but I could see some reasonable person holding it, and flat out don't see a way to get to 10-90%+ for a reasonable person, primarily because I have a cluster of beliefs of "alignment is very easy" combined with "intelligence isn't magic" and regulatory constraints from society.

The first time an AI reaches STEM+ capabilities (if that ever happens), it will disempower humanity within three months: Almost certainly below 1%, and I do not see a way to get a reasonable person to hold this view that doesn't rely on either FOOM models of AI progress or vastly unrealistic models of society. This is far too fast for any big societal change. I'd put it at a decade minimum. It's also obviated by my general belief that alignment is very easy.

If AI wipes out humanity and colonizes the universe itself, the future will go about as well as if humanity had survived (or better).

Almost any probability could be entered by a reasonable person, so I elect to maintain all probability ranges between 0 and 1 as options.

Given sufficient technical knowledge, humanity could in principle build vastly superhuman AIs that reliably produce very good outcomes: Over 99%, and indeed I believe a stronger statement that will be elucidated in the next question.

Researchers will solve enough of the alignment problem to prevent world-endangering AI from ever being built:

Over 99% is my odds right now, I could see a reasonable person having 50%-99% credences, but not lower than that.

My main reasons for this come down to my deep skepticism around deceptive alignment, to use a terminology Evan Hubinger used, I think were either not underwater at all, or only underwater by only 1-1000 bits of data, which is tiny compared to the data around human values. I basically entirely disbelieve the memeplex LW developed that human values are complicated and totally arbitrary, or at least the generators of human values, and a partial reason for that is that the simplicity bonuses offered by good world-modeling also bleed over a lot into value formation, and while there is a distinction between values and capabilities, I think it's not nearly as sharp or as different in complexity as LWers think.

This was an important scenario to get out of the way, because my next argument wouldn't work if deceptive alignment happened.

Another argument that Jaime Sevilla used, and that I tend to agree with, is that it's fundamentally profitable for companies to control AI, primarily because control research is both cheaper and more valuable for AIs, and it's not subject to legal restrictions, and they internalize a lot more of the risks of AI going out of control, which is why I expect problems like Goodharting to mostly go away by default, because I expect the profit incentive to be quite strong and positive for alignment in general. This also implies that a lot of LW work is duplicative at best, so that's another point.

There are of course other reasons, but this comment would be much longer.

For the purpose of preventing AI catastrophes, technical research is more useful than policy work: I'd say I have a wide range of probabilities, but I do think that it's not going to be 90%+, or even 50%+, and a big part of this reason is I'm focused on fairly different AI catastrophes, like this story, where the companies have AI that is controllable and powerful enough to make humans mostly worthless, basically removing all incentive for capitalists to keep humans alive or well-treated. This means I'm mostly not caring about technical research that rely on controlling AIs by default.

https://www.lesswrong.com/posts/2ujT9renJwdrcBqcE/the-benevolence-of-the-butcher [LW · GW]

Governments will generally be reasonable in how they handle AI risk: 10-90% is my probability for now, with a wide distribution. Right now, AI risk is basically talked about, and one of the best things is that a lot of the regulation is pretty normalish. I would worry a little more if pauses are seriously considered, because I'd worry that the reason for the pause is to buy time for safety, but in my models of alignment, we don't need that. I'd say the big questions are how much would rationalists gain power over the government, and what the balance of pro to anti-pause politics looks like.

It would be an unprecedentedly huge tragedy if we never built STEM+ AI: <<<1-40% chance IMO, and a potential difference from most AI optimists/pro-progress people is that if we ignore long-term effects, and ignore long-termism, it's likely that we will muddle along if AI is severely restricted, and it would be closer to the nuclear case, where it's clearly bad that it was slowed down, but it wasn't catastrophic.

Over the long-term, it would be an unprecedently huge tragedy, mostly due to base-rate risk and potential loss of infinite utility (depending on the physics).

I'd mostly agree with Jeffrey Heninger's post on how muddling along is more likely than dystopia here, if it wasn't for who's trying to gain power, and just how much they veer toward some seriously extreme actions.

https://www.lesswrong.com/posts/pAnvMYd9mqDT97shk/muddling-along-is-more-likely-than-dystopia [LW · GW]

It's admittedly an area where I have the weakest evidence on, but a little of my probability is based on worrying about how rationalists would slow down AI leading to extremely bad outcomes.

Props for doing this! Mine:

I do feel like "disempower humanity" is a slightly odd framing. I'm operationalizing "humanity remains in power" as something along the lines of "most human governments continue collecting taxes and using those taxes on things like roads and hospitals, at least half of global energy usage is used in the process of achieving ends that specific humans want to achieve", and "AI disempowers humans" as being that "humanity remains in power" becomes false specifically due to AI.

But there's another interpretation that goes something like "the ability of each human to change their environment is primarily deployed to intentionally improve the long-term prospects for human flourishing", and I don't think humanity has ever been empowered by that definition (and I don't expect it to start being empowered like that).

Similar ambiguity around "world-endangering AI" -- I'm operationalizing that as "any ML system, doesn't have to be STEM+AI specifically, that could be part of a series of actions or events leading to a global catastrophe".

For "as well as if humanity had survived" I'm interpreting that as "survived the dangers of AI specifically".

It’s physically impossible to ever build STEM+ AI.

This is tautological, it is impossible by known physics to not be able to do this

STEM+ AI will exist by the year 2100.

Similarly tautological, a machine that is STEM+ can be developed by various recursive methods

If STEM+AI is built, within 10 years AIs will disempower humanity.

This is orthogonal, a STEM+ machine need have no goals of its own

If STEM+AI is built, within 10 years AIs will be (individually or collectively) able to disempower humanity.

Humans have to make some catastrophically bad choices for this to even be possible.

The first time an AI reaches STEM+ capabilities (if that ever happens),

it will disempower humanity within three months.

foom requires physics to allow this kind of doubling rate.  It likely doesn't, at least starting with human level tech

Given sufficient technical knowledge, humanity could in principle build vastly superhuman AIs that reliably produce very good outcomes.

No state no problems.  I define "good" as "aligned with the user".

It would be an unprecedentedly huge tragedy if we never built STEM+ AI.

This is the early death of every human who will ever live for all time.  Medical issues are too complex for human brains to ever solve them reliably. Adding some years with drugs or gene hacks, sure.  Stopping every possible way someone can die, so that they witness their 200th and 2000th birthday?  No way.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

me:

Great stuff! I made a black-and-white printable version here. Maybe this could be a fun coloring-bonding activity at some meetups?:)

I find one consistent crux I have with people not concerned about AI risk is that they believe massively more resources will be invested into technical safety before AGI is developed.

In the context of these statements, I would put it as something like "The number of people working full-time on technical AI Safety will increase by an order of magnitude by 2030".

There's a hole in this, and it has to do the belief structure here.

The first time an AI reaches STEM+ capabilities (if that ever happens), it will disempower humanity within three months.  

If STEM+AI is built, within 10 years AIs will disempower humanity.

What is an "AI"?  A program someone runs on their computer?  An online service that runs on a cluster of computers in the cloud?  

Implicitly you're assuming the AI isn't just competent at "STEM+", it has motivations.  It's not just a neural network that the training algorithm adjusted until the answers are correct on some benchmark/simulator that evaluates the model's ability at "STEM+", but some entity that...runs all the time?  Why?  Has local memory and continuity of existence?  Why does it have this?

Why are humans going to give most AI systems any of this?  

Is the AI just going to evolve these systems as a side effect of RSI environments, where humans didn't design the highest end AI models but they were architected by other AI models?  That might result in this kind of behavior.

Memory/continuity of existence/personal goals for the AI are not simple things.  They won't be features added by accident.  Eliezer and others have argued that maybe llms could exhibit some of these features spontaenously, or the inner:outer argument, but memory can't be added by anything but a purposeful choice.  (memory means that after a session, the model retains altered bits, and all sessions alter bits used by the model.  It's not the same as a model that gets to skim the prior session logs on a new run, since humans get to control which logs the model sees.  Continuity of existence requires memory, personal goals require memory to accomplish)

Maybe I'm badly off here but it almost sounds like we have the entire alignment argument exactly wrong.  That instead of arguing we shouldn't build AI, or we need to figure out alignment, maybe the argument should be that we shouldn't build stateful AI with a continuity of existence for however many decades/centuries it takes to figure out how to make stable ones.  Aligning a stateful machine that self modifies all the time is insanely difficult and may not actually be possible except by creating a world where the power balance is unfavorable to AI takeover.

 

I originally saw this on Twitter, and posted mine in response, but I feel in-depth discussion is probably more productive here, so I appreciate you cross-posting this :)