LessWrong 2.0 Reader

China is producing research in a number of areas right now that is surpassing the West and arguably more impressive scientifically than producing top LLMs.

A big reason China is lagging a little bit might be political interference at major tech companies. Xi Jinping instigated a major crackdown recently. There is also significantly less Chinese text data. I am not a China or tech expert so these sre just guesses.

In any case, I wouldn't assign it to much significance. The AI space is just moving so quickly that even a minor year delay can seem like lightyears. But that doesnt mean that Chinese companies cant so it or that a country-continent with 1,4 billion people and a history of many technological firsts cant scale up a transformer.

In which worlds would AI Control prevent significant harm?

When I bring up the issue of AI model security to people working in AI safety, I’m often met with something of the form “yes, this is a problem. It’s important that people work hard on securing AI models. But it doesn’t really affect my work”.

Using AI Control [LW · GW] (an area which has recently excited many [LW(p) · GW(p)] in the field) as an example, I lay out an argument for why it might not be as effective an agenda as one might think after considering the realities of our cyber security situation.

1. AI Control concerns itself with models that intentionally try to subvert its developers.
2. These models are likely to be very generally capable and capable of causing significant harm without countermeasures.
3. Leading cyber-capable institutions would likely expend significant resources and political capital to steal these models or steal enough insights to reproduce such models.
4. If the weights or insights are stolen, work on AI control will not prevent these models from causing significant harm.
5. Current AI developers are not on track to be able to defend against high-priority operations from leading cyber-capable institutions in the coming years.
6. Therefore,  AI control will only be useful in the coming years under one (or more) of these conditions:
   1. Models that scheme are unlikely to be generally capable/dangerous enough to be a high-priority target for leading cyber-capable institutions.
   2. Models that scheme are only developed by actors that can thwart high-priority operations from leading cyber-capable institutions (which precludes current AI developers for at least several years).
   3. AI Control won’t be directly useful in the coming years but it will be indirectly useful to progress the field for when models are developed by actors capable of thwarting top cyber operations.
   4. Even if the model was stolen and caused significant harm, there would still be less harm overall than if the model undergoing AI control measures also broke out. Of course, this argument is much weaker for models capable of existential harm.
   5. Actors that steal the model would also implement strong-enough AI control measures and would not misuse the AI to cause significant harm.

There are of course other arguments against working on AI control. E.g. it may encourage the development and use of models that are capable of causing significant harm. This is an issue if the AI control methods fail or if the model is stolen. So one must be willing to eat this cost or argue that it’s not a large cost when advocating for AI Control work.

This isn’t to say that AI Control isn’t a promising agenda, I just think people need to carefully consider the cases in which their agenda falls down for reasons that aren’t technical arguments about the agenda itself.

I’m also interested to hear takes from those excited by AI Control on which conditions listed in #6 above that they expect to hold (or to otherwise poke holes in the argument).

This seems like the sort of R&D that China is good at: research that doesn't need superstar researchers and that is mostly made of incremental improvements. But yet they don't seem to be producing top LLMs. Why is that?

The way I do this is use the Print as PDF functionality in the browser on every single post, and then concatenate them using pdfunite.

I interpreted Eliezer as writing from the assumption that the superintelligence(s) in question are in fact not already aligned to maximize whatever it is that humanity needs to survive, but some other goal(s), which diverge from humanity's interests once implemented.

He explicitly states that the essay's point is to shoot down a clumsy counterargument (along "it wouldn't cost the ASI a lot to let us live, so we should assume they'd let us live"). So the context (I interpret) is that such requests, however sympathetic, have not been ingrained into the ASI:s goals. Using a different example would mean he was discussing something different.

That is, "just because it would make a trivial difference from the ASI:s perspective to let humanity thrive, whereas it would make an existential difference from humanity's perspective, doesn't mean ASIs will let humanity thrive", assuming such conditions aren't already baked into their decision-making.

I think Eliezer spends so much time on working from these premises because he believes 1) an unaligned ASI to be the default outcome of current developments, and 2) that all current attempts at alignment will necessarily fail.

What did you mean exactly in 2016 by the scaling hypothesis ?

Something like 'we could have AGI just by scaling up deep learning / deep RL, without any need for major algorithmic breakthroughs'.

Having past 2035 timelines and believing in the pure scaling maximalist hypothesis (which fwiw i don't believe in for reasons i have explained elsewhere) are in direct conflict so id be curious if you could more exactly detail your beliefs back then.

I'm not sure this is strictly true, though I agree with the 'vibe'. I think there were probably a couple of things in play:

• I still only had something like 20% on scaling, and I expected much more compute would likely be needed, especially in that scenario, but also more broadly (e.g. maybe something like the median in 'bioanchors' - 35 OOMs of pretraining-equivalent compute, if I don't misremember; though I definitely hadn't thought very explicitly about how many OOMs of compute at that time) - so I thought it would probably take decades to get to the required amount of compute.
• I very likely hadn't thought hard and long enough to necessarily integrate/make coherent my various beliefs. 
• Probably at least partly because there seemed to be a lot of social pressure from academic peers against even something like '20% on scaling', and even against taking AGI and AGI safety seriously at all. This likely made it harder to 'viscerally feel' what some of my beliefs might imply, and especially that it might happen very soon (which also had consequences in delaying when I'd go full-time into working on AI safety; along with thinking I'd have more time to prepare for it, before going all in).

What did you mean exactly in 2016 by the scaling hypothesis ?

Having past 2035 timelines and believing in the pure scaling maximalist hypothesis (which fwiw i don't believe in for reasons i have explained elsewhere) are in direct conflict so id be curious if you could more exactly detail your beliefs back then.

Intersubjective agreement on what "is an alignment technique" means will be far worse than on "is a machine learning technique", and many implications of the first claim are far more contentious than of the second.

I think it is highly uncontroversial and even trivial to call RLHF an alignment technique, given that it is literally used to nudge the model away from "bad" responses and toward "good" responses. It seems the label "alignment technique" could only be considered inappropriate here for someone who has a nebulous science fiction idea of alignment as a technology that doesn't currently exist at all, like it was seen when Eliezer originally wrote the sequences. I think it's obvious that this view is outdated now.

Fwiw, in 2016 I would have put something like 20% probability on what became known as 'the scaling hypothesis'. I still had past-2035 median timelines, though. 

This is a great post.

It's a good pointer to a part of the art which I am aiming for.

Some instructions are a bit vague and I'd have found it even nicer if you had more precisely written what repeated practices (e.g. what sorts of meditation) one can do to e.g. become better embody the void. I think if you had I would probably disagree with some parts of the suggested path, but it'd still be nice to have something more concrete to take what is good from.

I feel like the "creating a rationality dojo" part sorta distracts from the really important pieces in this post. It might've been nicer to just do the main post on "towards becoming truly formidable in the art" and then a second independentish post on "creating a rationality dojo".

(It feels a bit like you're saying "if you want to create a good rationality dojo, you first need to become great in the art, and then only create a dojo when the art calls for you to do it". Might've been nicer if you had just been like "here are some pointers to what i can sense which might be useful for aspiring rationalists". I think for many aspiring beisutsukai it might be the case that their purpose won't call for them to create a dojo for quite a while. Also beisutsukai usually don't become beisutsukai because they just want to become great in the art, but rather as instrumental to their actual purpose. ("the art must have a purpose other than its own".))

It's amusing because I doubt this is my gift to give the world. I'm doing something closely related, but different. Far too mystical for the right aesthetic.

I'm curious, what do you think is your gift to give to the world? (Feel free to answer by PM.)