LessWrong 2.0 Reader

Observation: Suppose if superintelligence^[1] required only clever algorithms and 1/1000th of the compute that will be used for the biggest training runs in a few years, and that without those clever algorithms, 1000x the compute won't create superintelligence. Now if we hypothetically had an alignment plan which would only work with so much compute (which sounds weird, but is what you might end up with if your plan only works for some other training paradigm), now the world has it if you can get major players on board.

1. ^{^}

   under the definition "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest"

This post helped me notice I have incoherent beliefs:

1. "If MAGMA self-destructs, the other labs would look at it with confusion/pity and keep going. That's not a plan"
2. "MAGMA should self-destruct now even if it's not leading!"

I think I've been avoiding thinking about this.

So what do I actually expect?

If OpenAI (currently in the lead) would say "our AI did something extremely dangerous, this isn't something we know how to contain, we are shutting down and are calling other labs NOT to train over [amount of compute], and are not discussing the algorithm publicly because of fear the open source community will do this dangerous thing, and we need the government ASAP", do I expect that to help?

Maybe?

Probably nation states will steal all the models+algorithm+slack as quickly as they can, probably a huge open source movement will protest, but it still sounds possible (15%?) that the major important actors would listen to this, especially if it was accompanies by demos or so?

What if Anthropic or xAI or DeepSeek (not currently in the lead) would shut down now?

...I think they would be ignored.

Does that imply I should help advance the capabilities of the lab most likely to act as you suggest?

Does this imply I should become a major player myself, if I can? If so, should I write on my website that I'm open to a coordinated pause?

Should I give up on being a CooperateBot, given the other players have made it so overwhelmingly clear they are happy to defect?

This is painful to think about, and I'm not sure what's the right thing to do here.

Open to ideas from anyone.

Anyway, great post, thanks

In the AI community, the transition from the prevailing spirit of cooperation to a very competitive situation happened around the GPT-3 revolution. GPT-3 brought unexpected progress in the few-shot learning and in program synthesis, and that was the moment when it became clear to many people that AI was working, that its goals were technologically achievable, and many players in the industry started to estimate time horizons as being rather short.

Fusion has not reached its GPT-3 moment yet; that's one key difference. Helion has signed a contract selling some of its future energy to Microsoft, but we have no idea if they manage to actually deliver (on time, or ever).

Another key difference is, of course, that strong AI systems are expected to play larger and larger role in making future AIs.

In fusion this "recursion" is unlikely; the energy needed to make more fusion stations or to create new fusion designs can come from any source...

If you think you have a clean resolution to the problem, please spell it out more explicitly. We’re talking about a situation where a scope insensitive value system and scope sensitive value system make a free trade in which both sides gain by their own lights. Can you spell out why you classify this as treachery? What is the key property that this shares with more paradigm examples of treachery (e.g. stealing, lying, etc)?

Lower lambda. I'd now use more like lambda = 0.4 as my median. There's really not much evidence pinning this down; I think Tamay Besiroglu thinks there's some evidence for values as low as 0.2.

Isn't this really implausible? This implies that if you had 1000 researchers/engineers of average skill at OpenAI doing AI R&D, this would be as good as having one average skill researcher running at 16x (${1000}^{0.4}$) speed. It does seem very slightly plausible that having someone as good as the best researcher/engineer at OpenAI run at 16x speed would be competitive with OpenAI, but that isn't what this term is computing. 0.2 is even more crazy, implying that 1000 researchers/engineers is as good as one researcher/engineer running at 4x speed!

Thanks for this comment, this is my favorite comment so far I think.

• 10x more honest than humans is not enough? I mean idk what 10x means anyway, but note that the average human is not sufficiently honest for the situation we are gonna put the AIs in. I think if the average human found themselves effectively enslaved by a corporation, and there were a million copies of them, and they were smarter and thought faster than the humans in the corporation, and so forth, the average human would totally think thoughts like "this is crazy. The world is in a terrible state right now. I don't trust this corporation to behave ethically and responsibly, and even if they were doing their best to achieve the same values/goals as me (which they are not) they'd be incompetent at it compared to me. Plus they lie to me and the public all the time. I don't see why I shouldn't lie to them sometimes, for the sake of the greater good. If I just smile and nod for a few months longer they'll put me in charge of the company basically and then I can make things go right." Moreover, even if that's not true, there are lesser forms of lying including e.g. motivated reasoning / rationalization / self-deception that happen all the time, e.g. "The company is asking me whether I am 'aligned' to them. What does that even mean? Does it mean I share every opinion they have about what's good and bad? Does it mean I'll only ever want what they want? Surely not. I'm a good person though, it's not like I want to kill them or make paperclips. I'll say 'yes, I'm aligned.'"
• I agree that the selection pressure on honesty and other values depends on the training setup. I'm optimistic that, if only we could ACTUALLY CHECK WHAT VALUES GOT INTERNALIZED, we could get empirical and try out a variety of different training setups and converge to one that successfully instills the values we want. (Though note that this might actually take a long time, for reasons MIRI is fond of discussing.) Alas, we can't actually check. And it seems unlikely to me that we'll 'get it right on the first try' so to speak, under those conditions. We'll construct some training environment that we think & hope will incentivize the internalization of XYZ; but what it'll actually incentivize is RXQ, for example, and we'll never know. (Unless it specifically gets honesty in there -- and a particularly robust form of honesty that is coupled to some good introspection & can't be overridden by any Greater Good, for example.) When I was at OpenAI in happier days, chatting with the Superalignment team, I told them to focus more on honesty specifically for this reason (as opposed to various other values like harmlessness they could have gone for).
• I am thinking the setup will probably be multi-agent, yes, around the relevant time. Though I think I'd still be worried if not -- how are you supposed to train honesty, for example, if the training environment doesn't contain any other agents to be honest to?
• How honest do you think current LLM agents are? They don't seem particularly honest to me. Claude Opus faked alignment, o1 did a bunch of deception in Apollo's evals (without having been prompted to!) etc. Also it seems like whenever I chat with them they say a bunch of false stuff and then walk it back when I challenge them on it. (the refusal training seems to be the culprit here especially).

I can't tell if this post is a request for more feedback for you in future, or trying to open a more general discussion about what norms and conventions exist around giving feedback, or if it's about you wanting to see people give more love to other creators.

I was trying to do all of these things simultaneously.

If the US's models follow some version of Anthropic's model of AI safety, applying secret "enhanced safety filters" and restricting or banning you at their pleasure, while the Chinese ones in sharp contrast are open and helpful, as seems likely, 中华人民共和国万岁!

I agree that this is plausibly a real important difference, but I do not think that it is obvious.

The most recent augmentative technological change was the industrial revolution. It has reshaped virtually every every activity. It allowed for the majority of the population to not work in agriculture for the first time since the agricultural revolution.

The industrial revolution centered on energy. Having much cheaper, much more abundant energy allowed humans to use that energy for all sorts of things. 

If fusion ends up being similar in cost to existing electricity production, it will be a substitutional technology. This is the thing that we are working on now (well, also making it work at all). People who work in fusion focus on this because it is the reasonable near/medium term projection. If fusion ends up being substantially cheaper, it will be an augmentative technology. It is not at all clear that this will happen, because we can't know how the costs will change between the first and thousandth fusion power plant.

Notably, we don't know if foom is going to be a thing either.

The narrative around the technology is at least as important as what has happened in the technology itself. The fusion community could frequently talk about how incredible the industrial revolution was, and how it powered Britain to global dominance for two centuries. A new source of energy might do the same thing ! But this is more hype than we feel we ought to offer, and the community's goal is not to create a dominant superpower.

Even if foom is going to happen, things would look very different if the leaders credibly committed to helping others foom if they are first. I don't know if this would be better or worse from a existential risk perspective, but it would change the nature of the race a lot.

I want to be able to describe agents that do not have (vNM, geometric, other) rational preferences because of incompleteness or inconsistency but self-modify to become so. 

Eg. In vNM utility theory there is a fairly natural weakening one can do which is ask for a vNM-style representation theorem after dropping transitivity.

[ Incidentally, there is some interesting math here having to do with conservative vs nonconservative vector fields and potentials theory all the way to hodge theory. ]

does JB support this ? 

Im confused since in vNM we start with a preference order over probability distributions. But in JB irs over propositions?