LessWrong 2.0 Reader

Indeed this seems related! We'd hope that there isn't too much conflict between the different "generations", since the intent is for the overseer to enable long-term plans, but it seems like something that could happen if the overseer is bad enough.

You refered to " others' definition (which is similar but doesn't rely on the game over clause) ", and I'm saying no, it's not relevantly similar, and it's not just my definition minus doom.

The CoTs themselves are well structured into paragraphs. At first, I thought this hints towards a tree search over CoT steps with some process reward model, but r1 also structures its CoTs into nice paragraphs. 

The o1 evaluation doc gives -- or appears to give -- the full CoT for a small number of examples, that might be an interesting comparison.

I haven't used o1 through the GUI interface so I can't check, but hasn't it always been the case that you can click the <thought for n seconds> triangle and get the summary of the CoT (as discussed in the 'Hiding the Chains of Thought' section of that same doc)?

This is correct.

This is correct.

I notice that it is probably harder for us to assume that there is only exactly one ASI, for if there were multiple, the chances that one of them might not suicide, for whatever reason, becomes its own class of significant concerns.

If the first ASI that we build is aligned, then it would use its superintelligent capabilities to prevent other ASIs from being built, in order to avoid this problem.

If the first ASI that we have build is misaligned, then it would also use its superintelligent capabilities to prevent other ASIs from being built. Thus, it simply wouldn't allow us to build an aligned ASI.

So basically, if manage to build an ASI without being prevented from doing so by other ASIs, then our ASI would use its superhuman capabilities to prevent other ASIs from being built.

Similarly, if the ASI itself 
is not fully and absolutely monolithic --
if it has any sub-systems or components
which are also less then perfectly aligned,
so as to want to preserve themselves, etc --
that they might prevent whole self termination

ASI can use exactly the same security techniques for preventing this problem as for preventing case (2). However, solving this issue is probably even easier, because, in addition to the security techniques, ASI can just decide to turn itself into a monolith (or, in other words, remove those subsystems).

The 'limits of control theory' aspects 

of the overall SNC argument basically states

(based on just logic, and not physics, etc)

that there are still relevant unknown unknowns

and interactions that simply cannot be predicted,

no matter how much compute power you throw at it.

It is not what we can control and predict and do, 

that matters here, but what we cannot do, 

and could never do, even in principle, etc.

This same reasoning could just well be applied to humans. There are still relevant unknown unknowns and interactions that simply cannot be predicted, no matter how much compute power you throw at it. With or without ASI, some things cannot be predicted.

This is what I meant by my guardian angel analogy. Just because a guardian angel doesn't know everything (has some unknowns), doesn't mean that we should expect our lives to go better without it, than with it, because humans have even more unknowns, due to being less intelligent and having lesser predictative capacities.

Hence to the question of "Is alignment enough?"

we arrive at a definite answer of "no",

both in 1; the sense of 'can prevent all classes

of significant and relevant (critical) human harm

I think we might be thinking about different meanings of "enough". For example, if humanity goes extinct in 50 years without alignment and it goes extinct in 10¹² years with alignment, then alignment is "enough"... to achieve better outcomes than would be achieved without it (in this example).

In the sense of "can prevent all classes of significant and relevant (critical) human harm", almost nothing is ever enough, so this again runs into an issue of being a very narrow, uncontroversial and inconsequential argument. If ~all of the actions that we can take are not enough, then the fact that building an aligned ASI is not enough is true almost by definition.

Not surprised, given China's announcement of a $145 billion dollar AI fund next week, and a potential $1 trillion in more funding by February or March.

If the $1 trillion dollar commitment is real, I have to give a lot of Bayes points to those who predicted an AI race, because this is not a thing you do unless you either want the technology for yourself, or you are racing against someone:

https://x.com/AndrewCurran_/status/1883721802280841245

Because of the success of R1 it looks like about $145 billion USD in Chinese AI funds were created this week. The rumor is that the Chinese government will also officially announce a trillion dollar AI fund in early February.

https://x.com/angelusm0rt1s/status/1883796706107736126

Not a guarantee that it will be in Feb It will be in Q1(Feb or March)

I'm not sure, given the "Indeterminate priors" section. But assuming that's true, what implication are you drawing from that? (The indeterminacy for us doesn't go away just because we think logically omniscient agents wouldn't have this indeterminacy.)

In one sense, the implication is that for an ideal reasoner, you can always give a probability to every event.

You are correct that the indeterminancy for us wouldn't go away.

The arbitrariness of a precise prior is a fact of life. This doesn't imply we shouldn't reduce this arbitrariness [LW · GW] by having indeterminate priors.

Perhaps.

I'd expect that we can still extend a no free lunch style argument such that the choice of indeterminate prior is arbitrary if we want to learn in the maximally general case, but I admit no such theorem is known, and maybe imprecise priors do avoid such a theorem.

I'm not saying indeterminate priors are bad, but rather that they probably aren't magical.

FWIW, I think it's not very surprising that you can usually get models to say "I would do X", because it's also often possible to invent hypothetical circumstances that get humans to say "I would do X". 

E.g. consider the trolley problem: 

• "Would you pull a lever to kill a person?" --> "No"
• "What if the only alternative is to let 5 people die via inaction?" --> "Pulling the lever seems reasonable" 

I wrote a post a while ago discussing this point more here [LW(p) · GW(p)]. 

This sounds a lot like what @Seth Herd [LW · GW]'s talk about instruction following AIs is all about:

https://www.lesswrong.com/posts/7NvKrqoQgJkZJmcuD/instruction-following-agi-is-easier-and-more-likely-than [LW · GW]