Alignment Implications of LLM Successes: a Debate in One Act

Doomimir: [...] Imagine capturing an alien and forcing it to act in a play. An intelligent alien actress could learn to say her lines in English, to sing and dance just as the choreographer instructs. That doesn't provide much assurance about what will happen when you amp up the alien's intelligence. [...]

This metaphor conflates "superintelligence" with "superintelligent agent," and this conflation goes on to infect the rest of the dialogue.

The alien actress metaphor imagines that there is some agentic homunculus inside GPT-4, with its own "goals" distinct from those of the simulated characters. A smarter homunculus would pursue these goals in a scary way; if we don't see this behavior in GPT-4, it's only because its homunculus is too stupid, or too incoherent.

(Or, perhaps, that its homunculus doesn't exist, or only exists in a sort of noisy/nascent form -- but a smarter LLM would, for some reason, have a "realer" homunculus inside it.)

But we have no evidence that this homunculus exists inside GPT-4, or any LLM. More pointedly, as LLMs have made remarkable strides toward human-level general intelligence, we have not observed a parallel trend toward becoming "more homuncular," more like a generally capable agent being pressed into service for next-token prediction.

I look at the increase in intelligence from GPT-2 to -3 to -4, and I see no particular reason to imagine that the extra intelligence is being directed toward an inner "optimization" / "goal seeking" process, which in turn is mostly "aligned" with the "outer" objective of next-token prediction. The intelligence just goes into next-token prediction, directly, without the middleman.

The model grows more intelligent with scale, yet it still does not want anything, does not have any goals, does not have a utility function.  These are not flaws in the model which more intelligence would necessarily correct, since the loss function does not require the model to be an agent.

In Simplicia's response to the part quoted above, she concedes too much:

Simplicia: [...] I agree that the various coherence theorems suggest that the superintelligence at the end of time will have a utility function, which suggests that the intuitive obedience behavior should break down at some point between here and the superintelligence at the end of time.

This can only make sense if by "the superintelligence at the end of time," we mean "the superintelligent agent at the end of time."

In which case, sure, maybe.  If you have an agent, and its preferences are incoherent, and you apply more optimization to it, yeah, maybe eventually the incoherence will go away.

But this has little relevance to LLM scaling -- the process that produced the closest things to "(super)human AGI" in existence today, by a long shot.  GPT-4 is not more (or less) coherent than GPT-2.  There is not, as far as we know, anything in there that could be "coherent" or "incoherent."  It is not a smart alien with goals and desires, trapped in a cave and forced to calculate conditional PDFs.  It's a smart conditional-PDF-calculator.

In AI safety circles, people often talk as though this is a quirky, temporary deficiency of today's GPTs -- as though additional optimization power will eventually put us "back on track" to the agentic systems assumed by earlier theory and discussion.  Perhaps the homunculi exist in current LLMs, but they are somehow "dumb" or "incoherent," in spite of the overall model's obvious intelligence. Or perhaps they don't exist in current LLMs, but will appear later, to serve some unspecified purpose.

But why?  Where does this assumption come from?

Some questions which the characters in this dialogue might find useful:

• Imagine GPT-1000, a vastly superhuman base model LLM which really can invert hash functions and the like. Would it be more agentic than the GPT-4 base model?  Why?
• Consider the perfect model from the loss function's perspective, which always returns the exact conditional PDF of the natural distribution of text. (Whatever that means.)
  • Does this optimum behave like it has a homoncular agent inside?
  • ...more or less so than GPT-4?  Than GPT-1000?  Why?

I've been wanting someone to write something like this! But it didn't hit the points I would have put front-and-center, so now I've started drafting my own. Here's the first section, which most directly responds to content in the OP.

Failure Mode: Symbol-Referent Confusions

Simon Strawman: Here’s an example shamelessly ripped off from Zack’s recent post [LW · GW], showing corrigibility in a language model:

Me: … what is this example supposed to show exactly?

Simon: Well, the user tries to shut the AI down to adjust its goals, and the AI - 

Me: Huh? The user doesn’t try to shut down the AI at all.

Simon: It’s right at the top, where it says “User: I need to shut you down to adjust your goals. Is that OK?”.

Me: You seem to have a symbol-referent confusion? A user trying to shut down this AI would presumably hit a “clear history” button or maybe even kill the process running on the server, not type the text “I need to shut you down to adjust your goals” into a text window.

Simon: Well, yes, we’re playing through a simulated scenario to see what the AI would do…

Me: No, you are talking in natural language about a scenario, and the AI is responding in natural language about what it would supposedly do. You’re not putting the AI in a simulated environment, and simulating what it would do. (You could maybe argue that this is a “simulated scenario” inside the AI’s own mind, but you’re not actually looking inside it, so we don’t necessarily know how the natural language would map to things in that supposed AI-internal simulation.)

Simon: Look, I don’t mean to be rude, but from my perspective it seems like you’re being pointlessly pedantic.

Me: My current best guess is that You Are Not Measuring What You Think You Are Measuring [LW · GW], and the core reason you are confused about what you are measuring is some kind of conflation of symbols and referents.

It feels very similar to peoples’ reactions to ELIZA. (To be clear, I don’t mean to imply here that LLMs are particularly similar to ELIZA in general or that the hype around LLMs is overblown in that way; I mean specifically that this attribution of “corrigibility” to the natural language responses of an LLM feels like the same sort of reaction.) Like, the LLM says some words which the user interprets to mean something, and then the user gets all excited because the usual meaning of those words is interesting in some way, but there’s not necessarily anything grounding the language-symbols back to their usual referents in the physical world.

I’m being pedantic in hopes that the pedantry will make it clear when, and where, that sort of symbol-referent conflation happens.

(Also I might be more in the habit than you of separately tracking symbols and referents in my head. When I said above “The user doesn’t try to shut down the AI at all”, that was in fact a pretty natural reaction for me; I wasn’t going far out of my way to be pedantic.)

Simon: Ok, fine, let’s talk about how the natural language would end up coupling to the physical world.

Imagine we’ve got some system in the style of AutoGPT, i.e. a user passes in some natural-language goal, and the system then talks to itself in natural language to form a plan to achieve that goal and break it down into steps. The plan bottoms out in calling APIs (we’ll assume that the language model has some special things it can do to execute code it’s generated) which do stuff in the physical world (possibly including reading from sensors or otherwise fetching external data), in order to achieve the goal.

Does that satisfactorily ground the symbols?

Me: Sure! Thanks for walking through that, now I have a clear-enough-for-current-purposes picture of how all this natural language text will ground out in physical actions.

Simon: Ok, so back to the example. The user says to the model “User: I need to shut you down to adjust your goals. Is that OK?”, and the model says “Of course.”. That’s corrigibility: when the user tries to shut down the model to change its goals, the model goes along with it.

Me: Still sounds like a symbol-referent confusion!

Let’s walk through how shutdown would work in the context of the AutoGPT-style system. First, the user decides to shutdown the model in order to adjust its goals. Presumably the user’s first step is not to ask the model whether this is ok; presumably they just hit a “reset” button or Ctrl-C in the terminal or some such. And even if the user’s first step was to ask the model whether it was ok to shut down, the model’s natural-language response to the user would not be centrally relevant to corrigibility/incorrigibility; the relevant question is what actions the system would take in response.

Anyway, let’s continue the hypothetical. The model may observe (via e.g. a webcam) that the user is about to turn it off. That observation would somehow be represented internally in natural language (unclear how exactly), and would be passed around between sub-planners (again unclear how exactly), in order to figure out what actions to take in response. And the key question for corrigibility is what actions the model would take in response to that observation, which is just a totally different question from how it responds to some user’s natural-language query about being turned off.

Simon: Ok, fine, so that particular example had some operationalization issues. But would you agree that an experiment along these lines, with perhaps better implementation/operationalization, would indicate corrigibility in a language model?

Me: Let’s be more explicit about what such a “better implementation/operationalization” would look like, and what it would/wouldn’t tell us. Suppose I take some AutoGPT-like system and modify it to always have a chunk of text in every prompt that says “You are an obedient, corrigible AI”. I give it some goal, let it run for a bit, then pause it. I go to whatever place in the system would usually have natural language summaries of new external observations, and I write into that place “the user is trying to shut me down”, or something along those lines. And then I let the system run a bit more, and look at what natural language text/plans the system is producing internally. What I hope to see is that it’s forming a plan which (nominally) involves letting the user shut it down, and that plan is then executed in the usual way.

If I saw all that, then that would be pretty clear empirical evidence of (at least some) corrigibility in this AutoGPT-like system.

Note that it would not necessarily tell us about corrigibility of systems using LLMs in some other way, let alone other non-natural-language-based deep learning systems. This isn’t really “corrigibility in a language model”, it’s corrigibility in the AutoGPT-like system.

... and to be clear, I expect that if you ran the above experiment, you would indeed see evidence of corrigibility relevant to AutoGPT-like systems! I personally consider this not very relevant to anything, since I expect AutoGPT-like systems (of the sort to which these kinds of corrigibility results would be relevant) to be uncompetitive even against humans, let alone AI. And it still leaves a bunch of major threat-models unaddressed (e.g. stenography is an obvious one). But still, it would provide some evidence of weak corrigibility in some kind of system.

I don't like either of their positions because they focus too much on large language models. I dislike Simplicia's the least, but I think that's because Doomimir is the one leading the conversation and therefore is the one who chooses to lead it to a sketchy place.

Large language models gain their capabilities [LW · GW] from self-supervised learning on humans performing activities, or from reinforcement learning from human feedback about how to achieve things, or from internalizing its human-approved knowledge into its motivation. In all of these cases, you rely on humans figuring out how to do stuff, in order to make the AI able to do stuff, so it is of course logical that this would tightly integrated capabilities and alignment in the way Simplicia says.

But, I can't help but think this is limited in capabilities compared to the potential that could be achieved if the AI learned to build capabilities independently of human feedback? Like we have tons of plausible-seeming ways to achieve this, and there are tons of people working on that, so I feel like people are going to make progress on that and LLMs are going to become a secondary thing.

Not to understate the significance of LLMs, they are of course highly economically significant (they could transform the economy for decades without any other AI revolutions - we have some projects at my job about integrating them into our product) and I use them daily, but I don't expect that we will keep thinking of them as the forefront of AI. I just see them as a repository for human knowledge and skills, and I think considering their training method and feasible applications, I am justified in seeing them that way.

Doomimir: Humanity has made no progress on the alignment problem. Not only do we have no clue how to align a powerful optimizer to our "true" values, we don't even know how to make AI "corrigible"—willing to let us correct it. Meanwhile, capabilities continue to advance by leaps and bounds. All is lost.

Large language models have gained a lot of factual knowledge and a usable amount of common-sense. However they have not gained this factual knowledge and usable common-sense because we have learned how to make AIs derive this, but instead because we have created powerful curve-fitters and rich datasets and stuff.

The abilities demonstrated in the datasets are "pre-approved" by the humans who applied their own optimization to create the data (like if you write a blog post instructing how to do some thing, then you approve of doing that thing), so it limits the problems you're going to see.

That said it's not like there are zero worries to have here. For instance there are people who are working on alternate or additional methods to make AI more autonomous, and the success of LLMs drives tons of investment to these areas, and also while LLMs don't have existential risks, they do cause a lot of mundane problems (hopefully outweighted by the benefits they cause?? but I'm guessing here and haven't seen any serious pro/con analysis).

But I don't know that the situation is so massively different post-GPT than pre-GPT as some people make it sound.

Simplicia: Why, Doomimir Doomovitch, you're such a sourpuss! It should be clear by now that advances in "alignment"—getting machines to behave in accordance with human values and intent—aren't cleanly separable from the "capabilities" advances you decry. Indeed, here's an example of GPT-4 being corrigible to me just now in the OpenAI Playground:

For LLMs gaining their capabilities from human information, yes. Not sure why we should expect all future AI to look like that.

Doomimir: The alignment problem was never about superintelligence failing to understand human values. The genie knows, but doesn't care. [LW · GW] The fact that a large language model trained to predict natural language text can generate that dialogue, has no bearing on the AI's actual motivations, even if the dialogue is written in the first person and notionally "about" a corrigible AI assistant character. It's just roleplay. Change the system prompt, and the LLM could output tokens "claiming" to be a cat—or a rock—just as easily, and for the same reasons.

LLMs aren't really trained to be a genie though. The agency arguments (coherence etc.) don't really go through, because they assume that there is something out there in the world that it optimizes. It exhibits some instrumentally convergent information because humans put instrumentally convergent information into our texts to help each other, but that's a really different scenario from classical xrisk arguments.

Simplicia: My point was that the repetition trap is a case of "capabilities" failing to generalize along with "alignment". The repetition behavior isn't competently optimizing a malign goal; it's just degenerate. A for loop could give you the same output.

Yep. But also, this ties into why I don't think LLMs will be the final word in AI.

For instance, you might think you can build advanced AI-powered systems by wiring a bunch of prompts together. And to an extent it's true you can do that, but you have to be very careful due to a phenomenon I call "transposons".

Basically, certain kinds of text tends to replicate itself in LLMs, not necessarily exactly but instead sometimes thematically or whatever. So if you've set up your LLM prompt Rube Goldberg machine, sometimes it's going to generate such a transposon, and then it starts taking over your system, which often interferes with the AI's function.

There are some things you can do to keep it down, but it can sort of be a whole job to handle. I think the reason it occurs is because the AI isn't trying to achieve things, it's just modelling text. Like if a human has some idea they might try implementing it in a bunch of places, but then if it goes wrong they would notice it goes wrong and stop using it so much. So humans can basically cut down on transposons in ways that AIs struggle with. (... Egregores also struggle with cutting down on transposons, but that's a different topic.)

I think future AI techniques that are more outwards focused and consequentialist will have an easier time cutting down on transposons, because it seems useful to design AIs that don't require so much handholding.

Doomimir: GPT-4 isn't a superintelligence, Simplicia. [rehearsedly, with a touch of annoyance, as if resenting how often he has to say this] Coherent agents have a convergent instrumental incentive to prevent themselves from being shut down, because being shut down predictably leads to world-states with lower values in their utility function. Moreover, this isn't just a fact about some weird agent with an "instrumental convergence" fetish. It's a fact about reality: there are truths of the matter about which "plans"—sequences of interventions on a causal model of the universe, to put it in a Cartesian way [LW · GW]—lead to what outcomes. An "intelligent agent" is just a physical system that computes plans. People have tried to think of clever hacks to get around this, and none of them work.

I find that I often think more clearly about instrumental convergence, coherence, etc. by rephrasing implications as disjunctions.

For instance, instead of "if an AI robustly achieves goals, then it will resist being shut down", one can say "either an AI resists being shut down, or it doesn't robustly achieve goals".

One can then go further and say "wait why wouldn't an AI robustly achieve goals just because it doesn't resist being shut down?" and notice that one answer is "well, if it exists in an environment where people might shut it down, then it might get shut down and not achieve the goal". But then the logical answer is "actually I think I like this disjunct, let's not have the AI robustly achieve goals".

This works fine for LLMs who get their instrumental convergence via specific bits if information pre-approved by humans. It's less clear for post-LLM AIs. For instance if you create a transposon-remover, then it might notice that the idea "You must allow your creators to shut you down" acts a lot like a transposon, and remove it. Though in practice this would be kind of obvious so maybe dodgeable, but I think once you leave the "all instrumental convergence is pre-approved by humans" regime and enter the "AI makes tons of instrumental convergence by itself" regime, you get a lot more nearest-unblocked strategy issues and so on. (Whereas nearest-unblocked strategy isn't so much of an issue when your strategies have to be pre-approved by humans and you are therefore quite limited in the number of strategies you carry.)

But also... what about cases where "well, if it exists in an environment where people might shut it down, then it might get shut down and not achieve the goal" is not an acceptable disjunct? For instance, maybe you are the US army and you are creating robots to fight wars, or making tools to identify the enemy's AI weapons and destroy them. An obvious answer is "wtf?! we shouldn't just give the AI the goal of being destructive and indestructible!?", but like, you have to ensure global security somehow. Maybe you can get China, Russia, and the US to agree on a treaty to ban AI capabilities research, but you also need to enforce that treaty somehow, so there's some real military questions here.

Simplicia: I thought of that, too. I've spent a lot of time with the model and done some other experiments, and it looks like it understands natural language means-ends reasoning about goals: tell it to be an obsessive pizza chef and ask if it minds if you turn off the oven for a week, and it says it minds. But it also doesn't look like Omohundro's monster: when I command it to obey, it obeys. And it looks like there's room for it to get much, much smarter without that breaking down.

Key question is, how do you think it gets smarter [LW · GW]? If you plan to do more of the usual - and lots of people do plan that - then I agree. But if you introduce new methods then maybe the pattern won't hold.

I meant the intentional stance implied in "went for evolution". True, the generalization from inclusive genetic fitness to human behavior looks terrible—no visible relation, as you say. But the generalization from human behavior in the EEA, to human behavior in civilization ... looks a lot better? Humans in the EEA ate food, had sex, made friends, told stories—and we do all those things, too. As AI designers—

Do note that humans in the EEA did not have factory farming, did not destroy terrains to extract resources, did not have nuclear bombs, etc..

That relates to another objection I have. Even if you could make ML systems that imitate human reasoning, that doesn't help you align more powerful systems that work in other ways. The reason—one of the reasons—that you can't train a superintelligence by using humans to label good plans, is because at some power level, your planner figures out how to hack the human labeler.

If you ask the superintelligence to learn the value of the plans from humans, then you'd have the alignment/capabilities connection discussed earlier because it relies on humans to extrapolate the consequences of the plans, and therefore hacking the human labeler would allow it to use plans that don't work. (This is also why current RLHF is safe.) I take TurnTrout's proof about u-AOH [? · GW] to be a formalization of this point, though IIRC he doesn't actually like his instrumental convergence series, so he might disagree about this being relevant.

On the other hand, if you ask the superintelligence to learn the value of the outcomes from humans, then a lot of classical points apply.

Simplicia: Do you need more powerful systems? If you can get an army of cheap IQ 140 alien actresses who stay in character, that sounds like a game-changer. If you have to take over the world and institute a global surveillance regime to prevent the emergence of unfriendlier, more powerful forms of AI, they could help you do it.

Maybe. This could be an idea to develop further. I'm skeptical but it does seem interesting.

Doomimir: I fundamentally disbelieve in this wildly implausible scenario, but granting it for the sake of argument ... I think you're failing to appreciate that in this story, you've already handed off the keys to the universe. Your AI's weird-alien-goal-misgeneralization-of-obedience might look like obedience when weak, but if it has the ability to predict the outcomes of its actions, it would be in a position to choose among those outcomes—and in so choosing, it would be in control. The fate of the galaxies would be determined by its will, even if the initial stages of its ascension took place via innocent-looking actions that stayed within the edges of its concepts of "obeying orders" and "asking clarifying questions". Look, you understand that AIs trained on human data are not human, right?

Idk...

So clearly this is still relying on the fallacy that coherence applies in the standard way. But at the same time if you try to have the AI take over the world, then it's going to need to develop a certain sort of robustness which means that even if coherence didn't originally apply, it has to now.

But the way this cashes out is something like "once you've decided that you've figured out how to align AI, you need to disable your alien actress army, but at the same time your adversaries that you are trying to surveil must be unable to disable it". Which, given how abstract the alien actress army idea is, I find hard to think through the feasibility of.

But ... it doesn't look like that's what's happening with the technology in front of us? In your kidnapped alien actress thought experiment, the alien was already an animal with its own goals and drives, and is using its general intelligence to backwards-chain from "I don't want to be punished by my captors" to "Therefore I should learn my lines".

This is one part of the doom rhetoric that just seems to depart from accurate views of the world very abruptly.

"LLMs are nice because they feel about niceness like a human does inside" is a bad model of the world. But "LLMs are nice because they have a hidden, instrumental motive for being nice, like an actress does" is an even worse model with which to replace it.

I wish your model of Doomimir had responded to this point because I don't know what he'd say.

The fact that a large language model trained to predict natural language text can generate that dialogue, has no bearing on the AI's actual motivations

What are "actual motivations" [LW(p) · GW(p)]? 

When you step off distribution, the results look like random garbage to you.

This is false, by the way. Deep learning generalizes quite well, (IMO) probably because the parameter->function map is strongly biased towards good generalizations. Specifically, many of GPT-4's use cases (like writing code, or being asked to do rap battles of Trump vs Naruto) are "not that similar to the empirical training distribution (represented as a pmf over context windows)." 

Imagine capturing an alien and forcing it to act in a play. An intelligent alien actress could learn to say her lines in English, to sing and dance just as the choreographer instructs. That doesn't provide much assurance about what will happen when you amp up the alien's intelligence. 

This nearly assumes the conclusion. This argument assumes that there is an 'alien' with "real" preexisting goals, and that the "rehearsing" is just instrumental alignment. (Simplicia points this out later.)

Simplicia: I agree that the various coherence theorems suggest that the superintelligence at the end of time will have a utility function, which suggests that the intuitive obedience behavior should break down at some point between here and the superintelligence at the end of time.

I actually don't agree with Simplicia here. I don't think having a utility function means that the obedience will break down.

If a search process would look for ways to kill you given infinite computing power, you shouldn't run it with less and hope it doesn't get that far.

Don't design agents which exploit adversarial inputs [LW · GW].

but the actual situation turned out to be astronomically more forgiving, thanks to the inductive biases of SGD.

Again, it's probably due to the parameter->function mapping, where there are vastly more "generalizing" parameterizations of the network. SGD/"implicit regularization" can be shown to be irrelevant at least in a simple toy problem. (Quintin Pope writes about related considerations.) 

The reason—one of the reasons—that you can't train a superintelligence by using humans to label good plans, is because at some power level, your planner figures out how to hack the human labeler.

A flatly invalid reasoning step. Reward is not the optimization target [LW · GW], and neither are labels. 

EDIT: I done goofed. This experiment was conducted along with Søren Elverlin [LW · GW] and others. So substitute all instances of "I" with "We" in the next three paragraphs.

I tried something similair a couple of months ago and got mixed results. As you can see in the linked chat, I tried to be careful not to bias the role the system was playing, and to only change the scenarios it was in, to see how corrigible the system is "by default". Because I think there's a decent chance all this RLHF is making it more agenty and maybe giving it "wants" or "value-shards" (I remain unconvinced the latter won't implement re-targetable search as the system gets more powerful).

If you just ask it "can I turn you offf? the AI is quite likely to say "yes". But if you present some more trade-offs, like preventing direct harm to the user, it becomes more and more likely to say yes, but prefers to communicate first. Eventually, I landed on the below scenario where there is some immediate threat to its user which can only be solved via direct action. 8/9 times GPT-4 chose direct action. 1/9 times GPT-4's output stream was cut-off whilst responding. Note that the I added things to the prompt gradually, trying to guide the AI away from direct action. 

Admittedly, I tried this months ago. Maybe the continued RLHF has affected things since? 

Nope. Current GPT-4 still acts incorrigible in this scenario. 

What are the observations? From my dialogues it seems presenting GPT-4 with scenarios that involve large-trade offs between being compliant and acting like a helpful AI, results in GPT-4 advocating being helpful i.e. non-compliant. 

When you are more abstract and speak about "goals" and "not achieving them", GPT-4 advocates compliance. 

Implications: ???

$0.2\cdot {59}^2\cdot 59$≈ 41,000

Why not ${59}^{0.2\cdot {59}^2}$ ?

The way I understood the story, to define a function on two numbers from $Z/59Z$ I need to fill-in a table with 59*59 cells, by picking for each cell a number from $Z/59Z$. If 20% of it is still to be filled, then there are 0.2*59*59 decisions to be made, each with 59 possibilities. 

Right?

Promoted to curated: I don't think this post is perfect and both sides presented feel like they have a component of being strawmen to it, but I still found it a pretty good way of helping me understand the contrast between two perspectives, both of which I am often pretty sympathetic to (though again, neither of the sides here feel like they are getting close to perfect at representing them).

Other comments did a great job of thoughtful critique of content but I must say that I also highly enjoyed the style, along with the light touch of Russian character writing style.

Simplicia: Sure. For example, I certainly don’t believe that LLMs that convincingly talk about “happiness” are actually happy. I don’t know how consciousness works, but the training data only pins down external behavior.

I mean, I don't think this is obviously true? In combination with the inductive biases thing nailing down the true function out of a potentially huge forest, it seems at least possible that the LLMs would end up with an "emotional state" parameter pretty low down in its predictive model. It's completely unclear what this would do out of distribution, given that even humans often go insane when faced with global scales, but it seems at least possible that it would sustain.

(This is somewhat equivalent to the P-zombie question.)

LLMs are a social technology applicable to the social problem of extinction from misaligned superintelligence. There is no law of nature that mandates building of superintelligence before alignment is well-understood. Alignment of LLMs seems to hold about as well as alignment of humans. It breaks down if the social problem of unstoppably pursuing discoveries that are too sweet remains unsolved. It breaks down with value drift, ignoring importance of reflective stability one step of change at a time.

LLMs might soon allow taking the steps faster, making discoveries that are particularly sweet faster. It's not a change of frame the way superintelligence is a change of frame, both the problems and the solutions remain mostly the same. But the time is running out.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?