LessWrong 2.0 Reader

What did you mean exactly in 2016 by the scaling hypothesis ?

Having past 2035 timelines and believing in the pure scaling maximalist hypothesis (which fwiw i don't believe in for reasons i have explained elsewhere) are in direct conflict so id be curious if you could more exactly detail your beliefs back then.

Intersubjective agreement on what "is an alignment technique" means will be far worse than on "is a machine learning technique", and many implications of the first claim are far more contentious than of the second.

I think it is highly uncontroversial and even trivial to call RLHF an alignment technique, given that it is literally used to nudge the model away from "bad" responses and toward "good" responses. It seems the label "alignment technique" could only be considered inappropriate here for someone who has a nebulous science fiction idea of alignment as a technology that doesn't currently exist at all, like it was seen when Eliezer originally wrote the sequences. I think it's obvious that this view is outdated now.

Fwiw, in 2016 I would have put something like 20% probability on what became known as 'the scaling hypothesis'. I still had past-2035 median timelines, though. 

This is a great post.

It's a good pointer to a part of the art which I am aiming for.

Some instructions are a bit vague and I'd have found it even nicer if you had more precisely written what repeated practices (e.g. what sorts of meditation) one can do to e.g. become better embody the void. I think if you had I would probably disagree with some parts of the suggested path, but it'd still be nice to have something more concrete to take what is good from.

I feel like the "creating a rationality dojo" part sorta distracts from the really important pieces in this post. It might've been nicer to just do the main post on "towards becoming truly formidable in the art" and then a second independentish post on "creating a rationality dojo".

(It feels a bit like you're saying "if you want to create a good rationality dojo, you first need to become great in the art, and then only create a dojo when the art calls for you to do it". Might've been nicer if you had just been like "here are some pointers to what i can sense which might be useful for aspiring rationalists". I think for many aspiring beisutsukai it might be the case that their purpose won't call for them to create a dojo for quite a while. Also beisutsukai usually don't become beisutsukai because they just want to become great in the art, but rather as instrumental to their actual purpose. ("the art must have a purpose other than its own".))

It's amusing because I doubt this is my gift to give the world. I'm doing something closely related, but different. Far too mystical for the right aesthetic.

I'm curious, what do you think is your gift to give to the world? (Feel free to answer by PM.)

I think there is far too much focus on technical approaches, when what is needed is a more socio-political focus.  Raising money, convincing deep pockets of risks to leverage smaller sums, buying politicians, influencers and perhaps other groups that can be coopted and convinced of existential risk to put a halt to Ai dev.

It amazes me that there are huge, well financed and well coordinated campaigns for climate, social and environmental concerns, trivial issues next to AI risk, and yet AI risk remains strictly academic/fringe.  What is on paper a very smart community embedded in perhaps the richest metropolitan area the world has ever seen, has not been able to create the political movement needed to slow things up.  I think precisely because they pitching to the wrong crowd.

Dumb it down.  Identify large easily influenceable demographics with a strong tendency to anxiety that can be most readily converted - most obviously teenagers, particularly girls and focus on convincing them of the dangers, perhaps also teachers as a community - with their huge influence.  But maybe also the elederly - the other stalwart group we see so heavily involved in environmental causes.  It would have orders of magnitude more impact than current cerebral elite focus, and history is replete with revolutions borne out of targeting conversion of teenagers to drive them.

I don't see a significant difference in your distinction between alignment and control. If you say alignment is about doing what you want (which I strongly disagree with in its generality, e.g. when someone might want to murder or torture people or otherwise act unethically), that obviously includes your wanting to "be OK" when the AI didn't do exactly what you want. Alignment comes in degrees, and you merely seem to equate control with non-perfect alignment and alignment with perfect alignment. Or I might be misunderstanding what you have in mind.

I work for a startup that builds agents, and yes, we use the architecture described here — with the additional feature that we don't own the machines that the inference or execution mecahnisms run on: they're in separate datacenters owned and operated by other companies (for the inference servers, generally foundation model companies behind an API)

Cutting edge AI research seems remarkably and surprisingly easy compared to other forms of cutting edge science. Most things work on the first try, clever insights aren't required, it's mostly an engineering task of scaling compute. 

Oh no uh-oh I think I might have confused Shane Legg with Jan Leike

I basically agree, for three reasons:

1. The level of understanding of and caring about human values required to not kill everyone and be able to keep many humans alive, is actually pretty low (especially on the knowledge side).
2. That's also basically sufficient to motivate wanting to learn more about human values, and being able to, so then the Value Learning [? · GW] process then kicks in: a competent and caring alien zookeeper would want to learn more about their charges' needs.
3. We have entire libraries half of whose content is devoted to "how to make humans happy", and we already fed most of them into our LLMs as training material. On a factual basis, knowing how to make humans happy in quite a lot of detail (and for a RAG agent, looking up details they don't already have memorized) is clearly well within their capabilities. The part that concerns me is the caring side, and that's not conceptually complicated: roughly speaking, the question is how to ensure an agent's selfless caring for humans is consistently a significantly stronger motivation than various bad habits [LW · GW] like ambition, competitiveness, and powerseeking that it either picked up from us during the "distillation" of the base model, and/or learnt during RL training.

Also, a question for this quote is what's the assumed capability/compute level used in this thought experiment?

E.g. if the there was an guide for alien zookeepers (ones already familiar with Terran biochemistry) on how to keep humans, how long would it need to be for the humans to mostly survive? 

ASI, or high AGI: capable enough that we've lost control and alignment is an existential risk.