LessWrong 2.0 Reader

You'd probably get more enthusiasm here if you led the article with a clear statement of its application for safety. We on LW are typically not enthusiastic about capabilities work in the absence of a clear and strong argument for how it improves safety more than accelerates progress toward truly dangerous AGI. If you feel differently, I encourage you to look with an open mind at the very general argument for why creating entities smarter than us is a risky proposition.

(Gemini did actually write much of the Gemini_Plays_Pokemon scaffolding, but only in the sense of doing what David told it to do, not designing and testing it.)

I think you're probably right that a LLM coding its own scaffolding is probably more achievable than one playing the game like a human, but I don't think current models can do it - watching the streams, the models don't seem like they understand their own flaws, although admittedly they haven't been prompted to focus on this.

Couldn't we privately ask Sam Altman “I would do X if Dario and Demis also commit to the same thing”?

Seems like the obvious thing one might like to do if people are stuck in a race and cannot coordinate.

X could be implementing some mitigation measures, supporting some piece of regulation, or just coordinating to tell the president that the situation is dangerous and we really do need to do something.

What do you think?

(Note, this is different from “if‑then” commitments proposed by Holden, which are more about if we cross capability X then we need to do mitigation Y)

I think this is a pretty important question. Jailbreak resistance will play a pretty big role in how broadly advanced AI/AGI systems are deployed. That will affect public opinion, which probably affects alignment efforts significantly (although It's hard to predict exactly how).

I think that setups like you describe will make it substantially harder to jailbreak LLMs. There are many possible approaches, like having the monitor LLM read only a small chunk of text at a time so that the jailbreak isn't complete in any section, and monitoring all or some of the conversation to see if the LLM is behaving as it should or if it's been jailbroken. Having full text sent to the developer and analyzed for risks would problematic for privacy, but many would accept those terms to use a really useful system.

Kinda Contra Kaj on LLM Scaling [LW(p) · GW(p)]

I didn't see Kaj Sotala's "Surprising LLM reasoning failures make me think we still need qualitative breakthroughs for AGI [LW · GW]" until yesterday, or I would have replied sooner. I wrote a reply last night and today, which got long enough that I considered making it a post, but I feel like I've said enough top-level things on the topic until I have data to share (within about a month hopefully!).

But if anyone's interested to see my current thinking on the topic, here it is [LW(p) · GW(p)].

As defined, this is a little paradoxical: how could I convince a human like you to perceive domains of real improvement which humans do not perceive...?

Oops, yes. I was thinking "domains of real improvement which humans are currently perceiving in LLMs", not "domains of real improvement which humans are capable of perceiving in general". So a capability like inner-monologue or truesight, which nobody currently knows about, but is improving anyway, would certainly qualify. And the discovery of such a capability could be 'real' even if other discoveries are 'fake'.

That said, neither truesight nor inner-monologue seem uncoupled to the more common domains of improvement, as measured in benchmarks and toy models and people-being-scared. The latter, especially, I thought was popularized because it was so surprisingly good at improving benchmark performance. Truesight is narrower, but at the very least we'd expect it to correlate with skill in the common "write [x] in the style of [y]" prompt, right? Surely the same network of associations which lets it accurately generate "Eliezer Yudkowsky wrote this" after a given set of tokens, would also be useful for accurately finishing a sentence starting with "Eliezer Yudkowksy says...".

So I still wouldn't consider these things to have basically-nothing to do with commonly perceived domains of improvement.

This feels like debating a holocaust denier. We are moving from "it did not happen at all" to "maybe it wasn't six million Jews but only five million". ("You did not name a single historian, Greek city state, solitary event, or personality from history" -> "ancients simply did not keep accurate records ... what evidence we do have shows the numbers to be always exaggerated")

The argument by inaccurate records goes both ways. If there is a genocide today, we probably know about it, and someone at least makes a note in Wikipedia. In the past, ethnic groups could be erased with no one (other than the people involved in the war) noticing. The fact that the list of known genocides in 20th century is longer than the list of known genocides in e.g. 12th century is mostly because of better bookkeeping.

And yet, despite choosing a century randomly (if I tried on purpose, I could have chosen e.g. the 13th century with Albigenian Crusade as a good example), Wikipedia mentions "Massacre of the Latins" with about 60 000 dead in the 12th century. In a world where the population was not even 1/10 of what it is today, so relatively comparable with the numbers that you have mentioned. And we have no idea about what massacres might have happened in 12th century Africa.

So yes, today we have more victims in absolute numbers, but that's because we have larger populations and stronger weapons. When you have to kill your enemies using a hand axe, I guess you get quite tired after chopping off dozen heads. With a nuke, you just press a button and thousands die. And yet, despite the other side having nukes, most Japanese survived WW2. (Which is something they totally did not expect, given their usual behavior towards defeated enemies.) The people in the past were as efficient at killing their enemies with swords, as we are with the weapons of mass destruction today.

"Now go, attack the Amalekites and totally destroy all that belongs to them. Do not spare them; put to death men and women, children and infants, cattle and sheep, camels and donkeys." (1 Samuel 15:3) Tell me again how civilians were not considered valid targets in the past.

You mention compelling prisoners of war to labor, as an analogy to slavery. Yeah, but that was an exception during the war. (Except for the Soviets, who conveniently kept many of the prisoners of war long after the war was over.) Now compare to a situation thousand years ago, when the slave trade was a crucial part of European economy, comparable to oil trade today. The reason entire countries converted to Christianity was to stop the unending slave raids from their neighbors. (Christians had a taboo against enslaving each other. So did Muslims. Both of them considered it okay to enslave each other, and the pagans.) Or consider Africa: the first black slaves brought to America were legally bought in Africa from the local African slave traders. Americans did not invent slavery; they just provided a huge new market for it.

Sorry, I think it is you who needs to learn history. Yes, humans suck today; the "Noble Savages" were not any better, probably much worse.

Why isn't the fact software developers spend 3 years not learning all that much (far less than they would in 6 months on the job) not a problem?

Suppose you're in a setting where the world is so large that you will only ever experience a tiny fraction of it directly, and you have to figure out the rest via generalization. Then your argument doesn't hold up: shifting the mean might totally break your learning. But I claim that the real world is like this. So I am inherently skeptical of any result (like most convergence results) that rely on just trying approximately everything and gradually learning which to prefer and disprefer.

I think that there's an important difference between the claim I'm making and the kinds of claims that Marcus has been making.

I definitely didn't mean to sound like I was comparing your claims to Marcus's! I didn't take your claims that way at all (and in particular you were very clear that you weren't putting any long-term weight on those particular cases). I'm just saying that I think our awareness of the outside view should be relatively strong in this area, because the trail of past predictions about the limits of LLMs is strewn with an unusually large number of skulls.

Yeah I don't have any strong theoretical reason to expect that scaling should stay stopped. That part is based purely on the empirical observation that scaling seems to have stopped for now

My argument is that it's not even clear (at least to me) that it's stopped for now. I'm unfortunately not aware of a great site that keeps benchmarks up to date with every new model, especially not ones that attempt to graph against estimated compute -- but I've yet to see a numerical estimate that shows capabilities-per-OOM-compute slowing down. If you're aware of good data there, I'd love to see it! But in the meantime, the impression that scaling laws are faltering seems to be kind of vibes-based, and for the reasons I gave above I think those vibes may be off.