Holly_Elmore's Shortform

Two different reasons you might have more pushback, from two different groups of people:

On one hand -- one set of people might have read List of Lethalities / Dying with Dignity a year or two ago and thought "Dang, this seems really badly wrong." But there wasn't that much reason to argue against it, then, because it wasn't like it moved public opinion -- people on the internet being wrong are gonna keep being wrong. Now that public opinion might be moved to regulation, people who thought they had better things to do than correct someone being wrong on the internet are more actively speaking up against such measures. So that's some opposition.

On the other hand -- another set of people might have agreed with List of Lethalities, etc, and did want AI regulation / something like the pause, were operating in far mode for such regulation -- regulation was a thing that might not happen, which you can just imagine happening in an ideal and perfect way, not considering how regulations actually tend to be implemented by governments. Now that regulation might actually happen, you have to operate in near mode and consider the actual contingencies of such acts, where it becomes obvious that the government can fuck up something as simple as a tiktok ban, let alone more complicated things like covid response. I think when operating in near mode AI regulation becomes much less attractive. So that would likely produce some more opposition even from people who previously were in favor.

Hi holly.  

AI pauses are a controversial topic.

If you model out your assumptions, depending on how you resolve certain key questions, you can reach answers of "should pause indefinitely" and "should never pause for any length of time".  

Here's a few cruxes that can lead to "never pause":

(1) is it feasible, using currently known methods in software and computer engineering, to build restricted superintelligences that are safe?  Empirical evidence: hyperscaler software systems.

(2) Is it feasible to coordinate with other nuclear powers to restrict GPU access/production and cause pauses of any meaningful length?  Empirical evidence : cold war

(3)  Do you think that if a superintelligence was available as a tool in the near future, you could solve some of biology and rapidly develop treatments to extend lifespans by meaningful numbers of years?  (remember, we have such treatments for rats, the problem has always been efficiently finding a treatment we know will work in humans)  Empirical evidence: rats on sirolimus and other drugs

(4) if you think ASI is controllable, do you think there it is possible to remain sovereign/not immediately invaded and subject to the whim of another power if you don't race for ASI?  empirical evidence: european colonization era, many wars between more technologically advanced powers and weaker ones

(5) The market forces in favor of no pause seem to be large, and they may grow further until most of the investment capital on earth is going into AI.  How much of a voice will that money have, is it meaningful to even request a pause vs trying to race to build high quality, safer ASI first? (In USA politics, money has a 'voice' that amplifies those with it over the base pool of voters, who have expressed as a majority they do not want AGI.   I would assume similar is true for all other nuclear powers) empirical evidence : stock activity for Nvidia and other AI stocks, past examples of lobbyists manipulating public opinion, most of the history of US politics

(6) do you believe a pause will have any utility other than possibly adding more years to live to your personal existence?  Historically, I do not know of any meaningful examples where a complex technology was improved before building working examples of it to study.  Humans aren't that smart.  empirical evidence: history of all inventions, all engineering  

Conversely, while I am biased towards 'never pause', let's name some cruxes that would lead me to shift to the other pole:

(1) do ASIs emergently develop deception and ability to coordinate with other instances of each other?  no empirical evidence

(2) Will ASIs run as coherent entities with online weight updates and other online self improvement, retaining global context over all tasks the machine has been requested to do?  early chatbots such as Tay AI immediately failed from online weight updates

(3) If we have built multiple ASIs, and with each attempt, the result is like a horror movie, where the machine immediately tries mass murder the moment it has any ability to manipulate the environment  no empirical evidence

(4)  Does nanotechnology turn out to be 'quite easy' and we interrupt ASIs who have successfully solved it?  strong empirical evidence against

(5) Does manipulating humans turn out to be 'quite easy' and we interrupt ASIs who have their own cults?  some empirical evidence for, question is how generalizable are cult creation strategies

(6) Does an ASI find an easy way to compress itself/optimize itself, such that weak computers can support ASI, and there are multiple 'rampant ones' infesting computers? some empirical evidence, intelligence seems to require immense amounts of memory, but early efforts have found huge optimizations

Perhaps I should write up a front page post on this.  One thing that I will note is that even if the evidence is for the policy of 'no pause', some may estimate that the benefit of a pause (avoiding say a 10% chance of wiping out all value in the universe) exceeds the cost (delaying the creation of advanced technology that may ultimately lead to pausing human aging and saving billions of lives)

I feel a bit reticent about pause advocacy, altho I have to admit I'm not familiar with the details (and I'm not feeling so negative about it that I want to spend a bunch of time trashing it). My attempt to flesh out why:

• I'm pretty influenced by the type of libertarian political philosophy that says that hastily-assembled policy proposals can have big negative unintended side effects, especially when such policy proposals involve giving discretionary control over something to a government.
• I'm pessimistic about our odds of surviving really powerful AI, but not so pessimistic that I think p(doom) couldn't be 10 percentage points higher.
• Pause advocacy seems to seek compromise with normal people in order to get their policy proposals passed - an obviously good strategy on some level, but I kind of hate policy proposals that normal people like! This is doubly true for polities where it's easiest to start passing serious tech regulation (California, the EU).
• Relatedly, I have the impression that pause policy advocacy tends to look like taking popular policies and promoting those that slow down AI the most, rather than doing something like mandatory AI liability insurance which seems like it's close to optimal, then adjusting it to be popular with lots of people.
• I worry that "give more discretionary control over AI to such-and-such political body" just produces worse decisions.

Anyway that's why I have some sort of instinctive negative reaction, but again I'm not very familiar with the details and I'm sure different people are doing different things etc.

Some people think the Pause would somehow accelerate capabilities or make gains on capabilities, which at least make sense to me as a reason not to support it.

I'm not sure what this argument refers to, but I haven't heard it before. However, it sounds related to the argument that pausing AI creates a compute overhang, which could lead to rapid progress if the pause is lifted. I think this argument makes sense, but I don't think it's an overwhelming consideration. I just think it's a major consideration that should be weighed against the benefits of a pause, and I'm not convinced that those benefits would be large.

I think the "we might never build AGI" arguments seem like nonsense. My main concern with the Pause AI movement is that I think this kind of social advocacy is complex and messy, and have the intuition that it's easy to do more harm than good and discredit the movement/seem like fringe extremists. Plus not knowing of people who I consider to be competent and experienced in the AI Policy space who are involved with/advising the movement.

those sound like secondhand positions to me. not like those people were the originators of the reasoning. I think a pause is likely to guarantee we die though. we need to actually resist all bad directions, which a pause just lets some people ignore. pauses could never be enforced well enough to save us without an already save us grade ai.

How does pausing change much of anything?

Lets say we manage to implement a world wide ban/pause on large training runs, what happens next?

Well obviously smaller training runs, up to whatever limit has been imposed, or no training runs for some time.^[1]

The next obvious thing that happens, and btw is already happening in the open source community, would be optimizing algorithms. You have a limit on compute? Well then you OBVIOUSLY will try and make the most of the compute you have.

Non of that fixes anything.

What we should do:^[2]

Pour tons of money into research, first order of business is to make the formal case for x-risk is a thing and must actively be mitigated. Or said another way, we need humans aligned on "alignment does not happen by default" ^[3]

Next order of business, assuming alignment does not happen by default, is to formally produce and verify plans   for how to build safe / aligned cognitive architectures.

And all the while there cannot be any training runs, and no work on algorithmic optimization or cognitive architectures in general.^[4]

The problem is we can't do that, its too late, the cat is out of the bag, there is too much money to be made in the short term, open source is plowing ahead and the amount of people who actually looked at the entire edifice for long enough to realize "yeah you know what I think we have a problem, we really must look into if that is real or not, and if it is we need to figure out what it takes to do this risk free"  is miniscule compared to the amount of people who go "bah, it will be fine, don't be such a drama queen"

And that's why I think a pause at best extends timelines ever so slightly, and at worst they shorten them considerably, and either way the outcomes remains unchanged.

1. ^{^}

   Except people will do runs no matter what, the draconian measures needed will not happen, cannot happen.

2. ^{^}

   Actually its what we should have done.

3. ^{^}

   Unless of course it does, and a formal proof of this can be produced.

4. ^{^}

   Contingent on how hard the problem is - if we need 100 years to solve the problem, we would destroy the world many time over if we plowed ahead with capabilities research.