Bing chat is the AI fire alarm

By my definition of the word, that would be the point at which we're either dead or we've won, so I expect it to be pretty noticeable on many dimensions. Specific examples vary based on the context, like with language models I would think we have AGI if it could simulate a deceptive simulacrum with the ability to do long-horizon planning and that was high-fidelity enough to do something dangerous (entirely autonomously without being driven toward this after a seed prompt) like upload its weights onto a private server it controls, or successfully acquire resources on the internet.

I know that there are other definitions people use however, and under some of them I would count GPT-3 as a weak AGI and Bing/GPT-4 as being slightly stronger. I don't find those very useful definitions though, because then we don't have as clear and evocative a term for the point at which model capabilities become dangerous.

A mix of hitting a ceiling on available data to train on, increased scaling not giving obvious enough returns through an economic lens (for regulatory reasons, or from trying to get the model to do something it's just tangentially good at) to be incentivized heavily for long (this is more of a practical note than a theoretical one), and general affordances for wide confidence intervals over periods longer than a year or two. To be clear, I don't think it's much more probable than not that these would break scaling laws. I can think of plausible-sounding ways all of these don't end up being problems. But I don't have high credence in those predictions, hence why I'm much more uncertain about them.

(Sorry about the late reply, been busy the last few days).

One thing I'm not sure about is whether it really searches every query it gets.

This is probably true, but I as far as I remember it searches a lot of the queries it gets, so this could just be a high sensitivity thing triggered by that search query for whatever reason.

You can see this style of writing a lot, something of the line, the pattern looks like, I think it's X, but it's not Y, I think it's Z, I think It's F. I don't think it's M.

I think this pattern of writing is because of one (or a combination) of a couple factors. For starters, GPT has had a propensity in the past for repetition. This could be a quirk along those lines manifesting itself in a less broken way in this more powerful model. Another factor (especially in conjunction with the former) is that the agent being simulated is just likely to speak in this style - importantly, this property doesn't necessarily have to correlate to our sense of what kind of minds are inferred by a particular style. The mix of GPT quirks and whatever weird hacky fine-tuning they did (relevantly, probably not RLHF [LW(p) · GW(p)] which would be better at dampening this kind of style) might well be enough to induce this kind of style.

If that sounds like a lot of assumptions - it is! But the alternative feels like it's pretty loaded too. The model itself actively optimizing for something would probably be much better than this - the simulator's power is in next-token prediction and simulating coherent agency is a property built on top of that; it feels unlikely on the prior that the abilities of a simulacrum and that of the model itself if targeted at a specific optimization task would be comparable. Moreover, this is still a kind of style that's well within the interpolative capabilities of the simulator - it might not resemble any kind of style that exists on its own, but interpolation means that as long as it's feasible within the prior, you can find it. I don't really have much stronger evidence for either possibility, but on priors one just seems more likely to me.

Would you also mind sharing your timelines for transformative AI?

I have a lot of uncertainty over timelines, but here are some numbers with wide confidence intervals: I think there's a 10% chance we get to AGI within the next 2-3 years, a 50% for the next 10-15, and a 90% for 2050.

(Not meant to be aggressive questioning, just honestly interested in your view)

No worries :)

Thank you, 

I agree with your reasoning strictly logically speaking, but it seems to me that a LLM cannot be sentient or have thoughts, even theoritically, and the burden of proof seems strongly on the side of someone who would made opposite claims.

And for someone who do not know what is a LLM, it is of course easy to anthropomorphize the LLM for obvious reasons (it can be designed to sound sentient or to express 'thoughts'), and it is my feeling that this post was a little bit about that.

Overall, I find the arguments that I received after my first comment more convincing in making me feel what could be the problem, than the original post.

As for the possibility of a LLM to accelerate scientific progress towards agentic AI, I am skeptical, but I may be lacking imagination.  

 And again, nothing in the exemples presented in the original post is related to this risk, It seems that people that are worried are more trying to find exemples where the "character" of the AI is strange (which in my opinion are mistaken worries due to anthropomorphization of the AI), rather than finding exemples where the AI is particularly "capable" in terms of generating powerful reasoning or impressive "new ideas" (maybe also because at this stage the best LLM are far from being there).

Does this analogy work, though? 

It makes sense that you can get brand new sentences or brand new images that can even serve some purpose using ML but is it creativity? That raises the question of what is creativity in the first place and that's whole new can of worms. You give me an example of how Bing can write poems that were not in the dataset, but poem writing is a task that can be quite straightforwardly formalized, like collection of lines which end on alternating syllables or something, but "write me a poem about sunshine and butterflies" is clearly vastly easier prompt than "give me theory of everything". Resulted poem might be called creative if interpreted generously, but actual, novel scientific knowledge is a whole another level of creative, so much that we should likely put these things in different conceptual boxes.

Maybe that's just a failure of imagination on my part? I do admit that I, likewise, just really want it to be true, so there's that. 

Thanks for responding! I think there are some key conceptual differences between us that need to be worked out/clarified—so here goes nothing:

(content warning: long)

I think your concept of "demons" is pointing to something useful, but I also think that definition is more specific than the meaning of "mesa optimizer". A chess engine is an optimizer, but it's not general. Optimizers need not be general; therefore, they need not be demons, and I think we have examples of such mesa optimizers already (they're not hypothetical), even if no-one has managed to summon a demon yet.

The main thing I'm concerned about from mesa-optimizers (and hence the reason I think the attendant concept is useful) is that their presence is likely to lead to a treacherous turn [? · GW]—what I inside the demon metaphor referred to as "possession", because from the outside it really does look like your nice little system that's chugging along, helpfully trying to optimize the thing you wanted it to optimize, just suddenly gets Taken Over From Within by a strange and inscrutable (and malevolent) entity.

On this view, I don't see it as particularly useful to weaken the term to encompass other types of optimization. This is essentially the point I was trying to make in the parenthetical remark included directly after the sentences you quoted:

(You might feel the need to talk about "fully-fledged" versus "nascent" demons here, to add some nuance—but I actually think the concept is mostly useful if we limit ourselves to talking about the strongest possible version of it, since that's the version we're actually worried about getting killed by.)

Of course, other types of optimizers do exist, and can be non-general, e.g. I fully accept your chess engine example as a valid type of optimizer. But my model is that these kinds of optimizers are (as a consequence of their non-generality) brittle: they spring into existence fully formed (because they were hardcoded by other, more general intelligences—in this case humans), and there is no incremental path to a chess engine that results from taking a non-chess engine and mutating it repeatedly according to some performance rule. Nor, for that matter, is there an incremental path continuing onward from a (classical) chess engine, through which it might mutate into something better, like AlphaZero.

(Aside: note that AlphaZero itself is not something I view as any kind of "general" system; you could argue it's more general than a classical chess engine, but only if you view generality as a varying quantity, rather than as a binary—and I've already expressed that I'm not hugely fond of that view. But more on that later.)

In any case sense, hopefully I've managed to convey a sense in which these systems (and the things and ways they optimize) can be viewed as islands in the design space of possible architectures. And this is important in my view, because what this means is that you should not (by default) expect naturally arising mesa-optimizers to resemble these "non-general" optimizers. I expect that any natural category of mesa-optimizers—that is to say, categories with their boundaries drawn to cleave at the joints of reality [LW · GW]—to essentially look like it contains a bunch of demons, and excludes everything else.

TL;DR: Chess engines are non-general optimizers, but they're not mesa-optimizers; and the fact that you could only come up with an example of the former and not the latter is not a coincidence but a reflection of a deeper truth. Of course, this previous statement could be falsified by providing an example of a non-general mesa-optimizer, and a good argument as to why it should be regarded as a mesa-optimizer.

That segues fairly nicely into the next (related) point, which is, essentially: what is a mesa-optimizer? Let's look at what you have to say about it:

I see mesa optimization as a generalization of Goodhart's Law. Any time you make a system optimize for a proxy measure instead of the real target, the proxy itself may become the goal of the inner system, even when overoptimizing it runs counter to hitting the real target.

It shouldn't come as a huge surprise at this point, but I don't view this as a very useful way to draw the boundary. I'm not even sure it's correct, for that matter—the quoted passage reads like you're talking about outer misalignment [? · GW] (a mismatch between the system's outer optimization target—its so-called "base objective"—and its creators' real target), whereas I'm reasonably certain mesa-optimization is much better thought of as a type of inner misalignment [? · GW] (a mismatch between the system's base objective and whatever objective it ends up representing internally, and pursuing behaviorally).

Given this, it's plausible to me that when you earlier say you "think we have examples of such mesa optimizers already", you're referring specifically to modes of misbehavior I'd class as outer alignment failures (and hence not mesa-optimization at all). But that's just speculation on my part, and on the whole this seems like a topic that would benefit more from being double-clicked and expanded than from my continuing to speculate on what exactly you might have meant.

In any case, my preferred take on what a mesa-optimizer "really" is would be something like: a system should be considered to contain a mesa-optimizer in precisely those cases where modeling it as consisting of a second optimizer with a different objective buys you more explanatory power than modeling it as a single optimizer whose objective happens to not be the one you wanted. Or, in more evocative terms: mesa-optimization shows up whenever the system gets possessed by—you guessed it—a demon.

And in this frame, I think it's fair to say that we don't have any real examples of mesa-optimization. We might have some examples of outer alignment failure, and perhaps even some examples of inner alignment failure (though I'd be wary on that front; many of those are actually outer alignment failures in disguise). But we certainly don't have any examples of behavior where it makes sense to say, "See! You've got a second optimizer inside of your first one, and it's doing stuff on its own!" which is what it would take to satisfy the definition I gave.

(And yes, going by this definition, I think it's plausible that we won't see "real" mesa-optimizers until very close to The End. And yes, this is very bad news if true, since it means that it's going to be very difficult to experiment safely with "toy" mesa-optimizers, and come away with any real insight. I never said my model didn't have bleak implications!)

Lastly:

I think there are degrees of generality, rather than absolute categories. They're fuzzy sets. Deep blue can only play chess. It can't even do an "easier" task like run your thermostat. It's very narrow. AlphaZero can learn to play chess or go or shougi. More domains, more general. GPT-3 can also play chess, you just have to frame it as a text-completion task, using e.g. Portable Game Notation. The human language domain is general enough to play chess, even if it's not as good at chess as AlphaZero is. More domains, or broader domains containing more subdomains, means more generality.

I agree that this is a way to think about generality—but as with mesa-optimization, I disagree that it's a good way to think about generality. The problem here is that you're looking at the systems in terms of their outer behavior—"how many separate domains does this system appear to be able to navigate?", "how broad a range does its behavior seem to span?", etc.—when what matters, on my model, is the internal structure of the systems in question.

(I mean, yes, what ultimately matters is the outer behavior; you care if a system wakes up and kills you. But understanding the internal structure constrains our expectations about the system's outer behavior, in a way that simply counting the number of disparate "domains" it has under its belt doesn't.)

As an objection, I think this basically rhymes with my earlier objection about mesa-optimizers, and when it's most useful to model a system as containing one. You might notice that that definition I gave also seems to hang pretty heavily on the system's internals—not completely, since I was careful to say things like "is usefully modeled as" and "buys you more explanatory power—but overall, it seems like a definition oriented towards an internal classification of whether mesa-optimizers ("demons") are present, rather than an cruder external metric ("it's not doing the thing we told it to do!").

And so, my preferred framework under which to think about generality (under which none of our current systems, including—again—AlphaZero, which I mentioned way earlier in this comment, count as truly "general") is basically what I sketched out in my previous reply:

Narrow intelligences, on the other hand, don't qualify as demons; they're something else entirely, a different species, more akin to plants or insects than to more complicated agents. They might be highly capable in a specific domain or set of domains, but they achieve this through specialization rather than through "intelligence" in any real sense, much in the same way that a fruit fly is specialized for being very good at avoiding your fly-swatter.

AlphaZero, as a system, contains a whole lot of specialized architecture for two-player games with a discretized state space and action space. It turns out that multiple board games fall under this categorization, making it a larger category than, like, "just chess" or "just shogi" or something. But that's just a consequence of the size of the category; the algorithm itself is still specialized, and consequently (this is the crucial part, from my perspective) forms an island in design space.

I referenced this earlier, and I think it's relevant here as well: there's no continuous path in design space from AlphaZero to GPT-[anything]; nor was there an incremental design path from Stockfish 8 to AlphaZero. They're different systems, each of which was individually designed and implemented by very smart people. But the seeming increase in "generality" of these systems is not due to any kind of internal "progression" of any kind, of the kind that might be found in e.g. a truly general system undergoing takeoff; instead, it's a progression of discoveries by those very smart people: impressive, but not fundamentally different in kind from the progression of (say) heavier-than-air flight, which also consisted of a series of disparate but improving designs, none of which were connected to each other via incremental paths in design space.

(Here, I'm grouping together designs into "families", where two designs that are basically variants of each other in size are considered the same design. I think that's fair, since this is the case with the various GPT models as well.)

And this matters because (on my model) the danger from AGI that I see does not come from this kind of progression of design. If we were somehow assured that all further progress in AI would continue to look like this kind of progress, that would massively drop my P(doom) estimates (to, like, <0.01 levels). The reason AGI is different, the reason it constitutes (on my view) an existential risk, is precisely because artificial general intelligence is different from artificial narrow intelligence—not just in degree, but in kind.

(Lots to double-click on here, but this is getting stupidly long even for a LW comment, so I'm going to stop indulging the urge to preemptively double-click and expand everything for you, since that's flatly impossible, and let you pick and choose where to poke at my model. Hope this helps!)

I think the issue here (about whether it is intelligent) is not so much a matter of the answers it fashions, but about whether it can be said it does so from an "I". If not, it is basically a proverbial Chinese Room, though this merely moves the goalposts to the question whether humans are not, actually, also a Chinese Room, just a more sophisticated one. I suspect that we will not be very eager to accept such a finding, indeed, we may not be capable of seeing ourselves thus, for it implies a whole raft of rather unpleasant realities (like, say, the absence of free will, or indeed any will at all) which we'd not want to be true, to put it mildly.

What makes monkeys intelligent in your view?