LessWrong 2.0 Reader

@Ericf [LW · GW] you are right, I didn't calculate it like this. I´m going to adjust the formula to see if it changes the overall outcome of the essay or not.

If the idea is "you should use AI to work on your personal projects", the problem is I'm already doing that (notice the $50/month).  I'm looking for ways to spend 20x more on AI without spending 20x more of my time (which is physically impossible).

What's the use in saying that values change, rather than just saying that you aren't interested in concepts involving values, then?

I don't have one. What would I use it for? I don't think anyone else yet has one, at least not something mathematically founded, with the simplicity and inevitability of VNM. People put forward various ideas and discuss the various "monsters" I listed, but I see no sign of a consensus.

A generic example is kind of an anti-example though.

If you reject utility theory, what approach do you use for modelling values instead, and what makes you feel that approach is helpful?

The clean lines make me think you didn't use hypergeometric calculations. If I have 2 extrovert friends, on any given day 0 (25%), 1(50%), or 2(25%) of them will want to hang out. If I want to hang out on day N, there is a 25% chance I fail to.

One example would be the generic one from the OP: "As a teenager, I endorsed the view that Z is the highest objective of human existence. … Yeah, it’s a bit embarrassing in hindsight." This hypothetical teenager's values (I suggest, in disagreement with the OP) have changed. Their knowledge about the world has no doubt also changed, but I see no need to postulate some unobservable deeper value underlying their earlier views that has remained unchanged, only their knowledge about Z having changed.

Long-term lasting changes in one's food preferences might also count, but not the observation that whatever someone has for lunch, they are less likely to have again for dinner.

Utility theory is overrated. There is a certain mathematical neatness to it for "small world" problems, where you know all of the possible actions and their possible effects, and the associated probabilities and payoffs, and you are just choosing the best action, once. Eliezer has described the situation as like a set of searchlights coherently pointing in the same direction. But as soon as you try to make it a universal decision theory it falls apart for reasons that are like another set of searchlights pointing off in all directions, such as unbounded utility, St Petersburg-like games, "outcomes" consistsing of all possible configurations of one's entire future light-cone, utility monsters, repugnant conclusions, iterated games, multi-player games, collective utility, agents trying to predict each other, and so on, illuminating a landscape of monsters surrounding the orderly little garden of VNM-based utility.

This is an interesting historical perspective... But it's not really what the fundamental case for AGI doom routes through. In particular: AGI doom is not about "AI systems", as such.

AGI doom is, specifically, about artificial generally intelligent systems capable of autonomously optimizing the world the way humans can, and who are more powerful at this task than humans. The AGI-doom arguments do not necessarily have anything to do with the current SoTA ML models.

Case in point: A manually written FPS bot is technically "an AI system". However, I think you'd agree that the AGI-doom arguments were never about this type of system, despite it falling under the broad umbrella of "an AI system".

Similarly, if a given SoTA ML model architecture fails to meet the definition of "a generally intelligent system capable of autonomously optimizing the world the way humans can", then the AGI doom is not about it. The details of its workings, therefore, have little to say, one way or another, about the AGI doom.

Why are the AGI-doom concerns extended to the current AI-capabilities research, then, if the SoTA models don't fall under said concerns? Well, because building artificial generally intelligent systems is something the AGI labs are specifically and deliberately trying to do. Inasmuch as the SoTA models are not the generally intelligent systems that are within the remit of the AGI-doom arguments, and are instead some other type of systems, the current AGI labs view this as their failure that they're doing their best to "fix".

And this is where the fundamental AGI-doom arguments – all these coherence theorems, utility-maximization frameworks, et cetera – come in. At their core, they're claims that any "artificial generally intelligent system capable of autonomously optimizing the world the way humans can" would necessarily be well-approximated as a game-theoretic agent. Which, in turn, means that any system that has the set of capabilities the AI researchers ultimately want their AI models to have, would inevitably have a set of potentially omnicidal failure modes.

In other words: The set of AI systems defined by "a generally intelligent world-optimization-capable agent", and the set of AI systems defined by "the subject of fundamental AGI-doom arguments", is the same set of systems. You can't have the former without the latter. And the AI industry wants the former; therefore, the arguments go, it will unleash the latter on the world.

While, yes, the current SoTA models are not subjects of the AGI doom arguments, that doesn't matter, because the current SoTA models are incidental research artefacts that are produced on AI industry's path to building an AGI. The AGI-doom arguments apply to the endpoint of that process, not the messy byproducts.

So any evidence we uncover about how the current models are not dangerous the way AGI-doom arguments predict AGIs to be dangerous, is just evidence that they're not AGI yet. It's not evidence that AGI would not be dangerous. (Again: FPS bots' non-dangerousness isn't evidence that AGI would be non-dangerous.)

(I'd written some more about this topic here [LW · GW]. See also gwern's Why Tool AIs Want to Be Agent AIs for more arguments regarding why AI research's endpoint would be an AI agent, instead of something as harmless and compliant as the contemporary models.)

Counterarguments to AGI-doom arguments that focus on pointing to the SoTA models, as such, miss the point. Actual counterarguments would instead find some way to argue that "generally intelligent world-optimizing agents" and "subjects of AGI-doom arguments" are not the exact same type of system; that you can, in theory, have the former without the latter. I have not seen any such argument, and the mathematical noose around them is slowly tightening [LW · GW] (uh, by which I mean: their impossibility may be formally provable).

What is going on with Constitution AI? Does anyone know why no LLM aside from Claude (at least none that I can find) has used it? One would think that if it works about as well as RLHF (which it seems to), AI companies would be flocking to it to save on the cost of human labor?

Also, apparently ChatGPT doesn't know that Constitutional AI is RLAIF (until I reminded it) and Gemini thinks RLAIF and RLHF are the same thing. (Apparently not a fluke as both models made the same error 2 out of 3 times.)

Just a random guess: is it possible that the tasks where LLMs benefit from chain-of-thought [LW · GW] are the same tasks where mild autism is an advantage for humans? Like, maybe autism makes it easier for humans to chain the thoughts, at the expense of something else?