LessWrong 2.0 Reader

I'll come out to the door to let you in if you're not here yet!

How do I get inside Lighthaven?

I would think things are headed toward these companies fine tuning an open source near-frontier LLM. Cheaper than building one from scratch but with most of the advantages.

Following Emma's advice on FB I exploded the tracks, healed splits, sorted them by take, and then comped manually ignoring reaper's features that are supposed to make this easier. It worked great, and I now have a rough mix ready for Lily's feedback!

uspect[faul_sname] Humans do seem to have strong preferences over immediate actions.

[Jeremy Gillen] I know, sometimes they do. But if they always did then they would be pretty useless. Habits are another example. Robust-to-new-obstacles behavior tends to be is driven by the future goals.

Point of clarification: the type of strong preferences I'm referring to are more deontological-injunction shaped than they are habit-shaped. I expect that a preference not to exhibit the behavior of murdering people would not meaningfully hinder someone whose goal was to get very rich from achieving that goal. One could certainly imagine cases where the preference not to murder caused the person to be less likely to achieve their goals, but I don't expect that it would be all that tightly binding of a constraint in practice, and so I don't think pondering and philosophizing until they realize that they value one murder at exactly -$28,034,771.91 would meaningfully improve that person's ability to get very rich.

I think if you remove "at any cost", it's a more reasonable translation of "going hard". It's just attempting to achieve a long-term goal that is hard to achieve.

I think there's more to the Yudkowsky definition of "going hard" it than "attempting to achieve hard long-term goals". Take for example:

@ESYudkowsky Mossad is much more clever and powerful than novices implicitly imagine a "superintelligence" will be; in the sense that, when novices ask themselves what a "superintelligence" will be able to do, they fall well short of the actual Mossad.

@ESYudkowsky Why? Because Mossad goes hard; and people who don't go hard themselves, have no simple mental motion they can perform -- no simple switch they can access -- to imagine what it is actually like to go hard; and what options become available even to a mere human when you do.

My interpretation of the specific thing that made Mossad's actions an instance of "going hard" here was that they took actions that most people would have thought of as "off limits" in the service of achieving their goal (and that doing so actually helped them achieve their goal (and that it actually worked out for them - we don't generally say that Elizabeth Holmes "went hard" with Theranos). The supply chain attack in question does demonstrate significant technical expertise, but it also demonstrates a willingness to risk provoking parties that were uninvolved in the conflict in order to achieve their goals.

Perhaps instead of "attempting to achieve the goal at any cost" it would be better to say "being willing to disregard conventions and costs imposed on uninvolved parties, if considering those things would get in the way of the pursuit of the goal".

[faul_sname] It does seem to me that "we have a lot of control over the approaches the agent tends to take" is true and becoming more true over time.

[Jeremy Gillen] No!

I suspect we may be talking past each other here. Some of the specific things I observe:

• RLHF works pretty well for getting LLMs to output text which is similar to text which has been previously rated as good, and dissimilar to text which has previously been rated as bad. It doesn't generalize perfectly, but it does generalize well enough that you generally have to use adversarial inputs to get it to exhibit undesired behavior - we call them "jailbreaks" not "yet more instances of bomb creation instructions".
• Along those lines, RLAIF also seems to Just Work™.
• And the last couple of years have been a parade of "the dumbest possible approach works great actually" results, e.g.
  • "Sure, fine-tuning works, but what happens if we just pick a few thousand weights and only change those weights, and leave the rest alone?" (Answer: it works great)
  • "I want outputs that are more like thing A and less like thing B, but I don't want to spend a lot of compute on fine tuning. Can I just compute both sets of activations and subtract the one from the other?" (Answer: Yep! [LW · GW])
  • "Can I ask it to write me a web application from a vague natural language description, and have it make reasonable choices about all the things I didn't specify" (Answer: astonishing amounts of yes)
• Take your pick of the top chat-tuned LLMs. If you ask it about a situation and ask what a good course of action would be, it will generally give you pretty sane answers.

So from that, I conclude:

• We have LLMs which understand human values, and can pretty effectively judge how good things are according to those values, and output those judgements in a machine-readable format
• We are able to tune LLMs to generate outputs that are more like the things we rate as good and less like the things we rate as bad

Put that together and that says that, at least at the level of LLMs, we do in fact have AIs which understand human morality and care about it to the extent that "care about" is even the correct abstraction for the kind of thing they do.

I expect this to continue to be true in the future, and I expect that our toolbox will get better faster than the AIs that we're training get more capable.

I'm talking about stability properties like "doesn't accidentally radically change the definition of its goals when updating its world-model by making observations".

What observations lead you to suspect that this is a likely failure mode?

Algorithmic progress can use 1. high iteration speed 2. well-defined success metrics (scaling laws) 3.broad knowledge of the whole stack (Cuda to optimization theory to test-time scaffolds) 4. ...

Alignment broadly construed is less engineering and a lot more blue skies, long horizon, and under-defined (obviously for engineering heavy alignment sub-tasks like jailbreak resistance, and some interp work this isn't true).

Generally agree, but I do think prosaic alignment has quite a few advantages vs. prosaic capabilities (e.g. in the extra slides here) and this could be enough [LW(p) · GW(p)] to result in aligned (-enough) automated safety researchers which can be applied to the more blue skies parts of safety research. I would also very much prefer something like a coordinated pause around the time when safety research gets automated.

This line of reasoning is part of why I think it's valuable for any alignment researcher (who can) to focus on bringing the under-defined into a well-defined framework. Shovel-ready tasks will be shoveled much faster by AI shortly anyway.

Agree, I've written about (something related to) this very recently [LW(p) · GW(p)].

Multi voiced podcast episode for this post in all is two hours and thirty eight minute glory!

(Fun fact, the voice I gave Nate was the one created for contract devils in Planecrash)

https://open.substack.com/pub/dwatvpodcast/p/book-review-on-the-edge-the-gamblers

I appreciate it when people repost here things Eliezer has written on Twitter or Facebook because it makes it easier for me to stay away from Twitter and Facebook.

(OTOH, I'm grateful to Eliezer for participating on Twitter because posting on Twitter has much higher impact than posting here does.)

This relies on an assumption that you can make up for lack-of-intelligence by numbers or speed. Without that assumption, you could expect that AI research will be dominated by humans until AIs finally “get it”, after which they’ll take over with a huge margin.

I interpret the research program described here as aiming to make this assumption true.

My personal feeling is that those who emphasize the "spiritual" interpretations are often doing it as a dodge, to avoid the challenge of having to follow the non-spiritual interpretations.

That feels a bit contrived. Do you really suggest that the most natural reading of something like "poor in spirit" is... non-spiritual? Turning away from materialism may sure derive from that, but to claim that it was the main focus seems quite a stretch.