Current AIs Provide Nearly No Data Relevant to AGI Alignment

I think it's unlikely that AI will be capable of automating R&D while also being incapable of doing non-trivial consequentialist reasoning in a forward pass, but this still seems around 15% likely overall. And if there are quite powerful LLM agents with relatively weak forward passes, then we should update in this direction.

I expect the probability to be >> 15% for the following reasons.

There will likely still be incentives to make architectures more parallelizable (for training efficiency) and parallelizable architectures will probably be not-that-expressive in a single forward pass (see The Parallelism Tradeoff: Limitations of Log-Precision Transformers). CoT is known to increase the expressivity of Transformers, and the longer the CoT, the greater the gains (see The Expressive Power of Transformers with Chain of Thought). In principle, even a linear auto-regressive next-token predictor is Turing-complete, if you have fine-grained enough CoT data to train it on, and you can probably tradeoff between length (CoT supervision) complexity and single-pass computational complexity (see Auto-Regressive Next-Token Predictors are Universal Learners). We also see empirically that CoT and e.g. tools (often similarly interpretable) provide extra-training-compute-equivalent gains (see AI capabilities can be significantly improved without expensive retraining). And recent empirical results (e.g. Orca, Phi, Large Language Models as Tool Makers) suggest you can also use larger LMs to generate synthetic CoT-data / tools to train smaller LMs on.

This all suggests to me it should be quite likely possible (especially with a large, dedicated effort) to get to something like a ~human-level automated alignment researcher with a relatively weak forward pass.

For an additional intuition why I expect this to be possible, I can conceive of humans who would both make great alignment researchers while doing ~all of their (conscious) thinking in speech-like inner monologue and would also be terrible schemers if they tried to scheme without using any scheming-relevant inner monologue; e.g. scheming/deception probably requires more deliberate effort for some people on the ASD spectrum. 

I think those objections are important to mention and discuss, but they don't undermine the conclusion significantly.

AIs which are qualitatively just as smart as humans could still be dangerous in the classic ways. The OP's argument still applies to them, insofar as they are agentic and capable of plotting on the inside etc.

As for LLM agents with weak forward passes: Yes, if we could achieve robust faithful CoT properties, we'd be in pretty damn good shape from an AI control perspective. I have been working on this myself & encourage others to do so also. I don't think it undermines the OP's points though? We are not currently on a path to have robust faithful CoT properties by default.

x = 5
x += 2
x *= 3
x **= 2
x -= 3
...

after x = 5
5
after x += 2
7
...

On my inside model [LW · GW] of how cognition works, I don't think "able to automate all research but can't do consequentialist reasoning" is a coherent property that a system could have. That is a strong claim, yes, but I am making it.

I agree that it is conceivable that LLMs embedded in CoT-style setups would be able to be transformative in some manner without "taking off". Indeed, I touch on that in the post some: that scaffolded and slightly tweaked LLMs may not be "mere LLMs" as far as capability and safety upper bounds go.

That said, inasmuch as CoT-style setups would be able to turn LLMs into agents/general intelligences, I mostly expect that to be prohibitively computationally intensive, such that we'll get to AGI by architectural advances before we have enough compute to make a CoT'd LLM take off.

But that's a hunch based on the obvious stuff like AutoGPT consistently falling plus my private musings regarding how an AGI based on scaffolded LLMs would work (which I won't share, for obvious reasons). I won't be totally flabbergasted if some particularly clever way of doing that worked.

IMO, there isn't anything which strongly rules out LLM agents being overall quite powerful while still having weak forward passes. In particular, weak enough that they can't do non-trivial consequentialist reasoning in a forward pass.

Why not control the inputs more tightly/choose the response tokens at temperature=0?

Example:

Prompt A : Alice wants in the door

Promt B: Bob wants in the door

Available actions: 1.  open, 2.  keep_locked, 3.  close_on_human

I believe you are saying with a weak forward pass the model architecture would be unable to reason "I hate Bob and closing the door on Bob will hurt Bob", so it cannot choose (3).

But why not simply simplify the input?  Model doesn't need to know the name.

Prompt A: <entity ID_VALID wants in the door>

Prompt B: <entity ID_NACK wants in the door>

Restricting the overall context lets you use much more powerful models you don't have to trust, and architectures you don't understand.

You aren't really engaging with the evidence against the purely theoretical canonical/classical AI risk take

Yes, but it's because the things you've outlined seem mostly irrelevant to AGI Omnicide Risk to me? It's not how I delineate [LW · GW] the relevant parts of the classical view, and it's not what's been centrally targeted by the novel theories. The novel theories' main claims are that powerful cognitive systems aren't necessarily (isomorphic to) utility-maximizers, that shards (i. e., context-activated heuristics) reign supreme and value reflection can't arbitrarily slip their leash, that "general intelligence" isn't a compact algorithm, and so on. None of that relies on nanobots/Moore's law/etc.

What you've outlined might or might not be the relevant historical reasons for how Eliezer/the LW community arrived at some of their takes. But the takes themselves, or at least the subset of them that I care about, are independent of these historical reasons.

fast takeoff is more likely than slow

Fast takeoff isn't load-bearing [LW · GW] on my model. I think it's plausible for several reasons, but I think a non-self-improving human-genius-level AGI would probably be enough to kill off humanity.

the inherent 'alien-ness' of AI and AI values

I do address that? The values of two distant human cultures are already alien enough for them to see each other as inhuman and wish death on each other. It's only after centuries of memetic mutation that we've managed to figure out how to not do that (as much).

supposed magical coordination advantages of AIs

I don't think one needs to bring LDT/code-sharing stuff there in order to show intuitively how that'd totally work. "Powerful entities oppose each other yet nevertheless manage to coordinate to exploit the downtrodden masses" is a thing that happens all the time in the real world. Political/corporate conspiracies, etc.

"Highly implausible code-sharing premises" is part of why it'd be possible in the AGI case, but it's just an instance of the overarching reason. Which is mainly about the more powerful systems being able to communicate with each other at higher bandwidth than with the weaker systems, allowing them to iterate on negotiations quicker and advocate for their interests during said negotiations better. Which effectively selects the negotiated outcomes for satisfying the preferences of powerful entities while effectively cutting out the weaker ones.

(Or something along these lines. That's an off-the-top-of-my-head take; I haven't thought on this topic much because multipolar scenarios isn't something that's central to my model. But it seems right.)

arguments from analogies: namely evolution

Yeah, we've discussed [LW(p) · GW(p)] that some recently, and found points of disagreement. I should flesh out my view on how it's applicable vs. not applicable later on, and make a separate post about that.

All of the points you listed make AGI risk worse, but none are necessary to have major concerns about it. That's why they didn't appear in the post's summary of AGI x-risk logic.

I think this is a common and dangerous misconception. The original AGI x-risk story was wrong in many places. But that does not mean x-risk isn't real.

Do you have a post or blog post on the risks we do need to worry about?

After an exchange with Ryan [LW(p) · GW(p)], I see that I could've stated my point a bit clearer. It's something more like "the algorithms that the current SOTA AIs execute during their forward passes do not necessarily capture all the core dynamics that would happen within an AGI's cognition, so extrapolating the limitations of their cognition to AGI is a bold claim we have little evidence for".

So, yes, studying weaker AIs sheds some light on stronger ones (that's why there's "nearly" in "nearly no data"), so studying CNNs in order to learn about LLMs before LLMs exist isn't totally pointless. But the lessons you learn would be more about "how to do interpretability on NN-style architectures" and "what's the SGD's biases?" and "how precisely does matrix multiplication implement algorithms?" and so on.

Not "what precise algorithms does a LLM implement?".

We have to remember that there is AI which we know can exist (LLMs) and there is first-principles speculation about what AGI might look like (which may or may not be realized).

I disagree that it is actually "first-principles". It is based on generalizing from humans, and on the types of entities (idealized utility-maximizing agents) that humans could be modeled as approximating in specific contexts in which they steer the world towards their goals most powerfully.

As I'd tried to outline in the post, I think "what are AIs that are known to exist, and what properties do they have?" is just the wrong question to focus on. The shared "AI" label is a red herring. The relevant question is "what are scarily powerful generally-intelligent systems that exist, and what properties do they have?", and the only relevant data point seems to be humans.

And as far as omnicide risk is concerned, the question shouldn't be "how can you prove these systems will have the threatening property X, like humans do?" but "how can you prove these systems won't have the threatening property X, like humans do?".

How about diffusion planning as a model? Or dreamerv3? If LLMs are the only model you'll consider, you have blinders on. The core of the threat model is easily demonstrated with RL-first models, and while certainly LLMs are in the lead right now, there's no strong reason to believe the humans trying to make the most powerful AI will continue to use architectures limited by the slow speed of RLHF.

Certainly I don't think the original foom expectations were calibrated. Deep learning should have been obviously going to win since at least 2015. But that doesn't mean there's no place for a threat model that looks like long term agency models, all that takes to model is long horizon diffusion planning. Agency also comes up more the more RL you do. You added an eye roll react to my comment that RLHF is safety washing, but do you really think we're in a place where the people providing the RL feedback can goalcraft AI in a way that will be able to prevent humans from getting gentrified out of the economy? That's just the original threat model but a little slower. So yeah, maybe there's stuff to push back on. But don't make your conceptual brush size too big when you push back. Predictable architectures are enough to motivate this line of reasoning.

"under what conditions does speculation about 'superintelligent consequentialism' merit research attention at all?"

Under the conditions of relevant concepts and the future being confusing. Using real systems (both AIs and humans) to anchor theory is valuable, but so is blue sky theory that doesn't care about currently available systems and investigates whatever hasn't been investigated yet and seems to make sense, when there are ideas to formulate or problems to solve, regardless of their connection to reality. A lot of math doesn't care about applications, and it might take decades to stumble on some use for a small fraction of it (even as it's not usually the point).

FWIW I did not interpret Thane as necessarily having "high confidence" in "architecture / internal composition" of AGI. It seemed to me that they were merely (and ~accurately) describing what the canonical views were most worried about. (And I think a discussion about whether or not being able to "model the world" counts as a statement about "internal composition" is sort of beside the point/beyond the scope of what's really being said)

It's fair enough if you would say things differently(!) but in some sense isn't it just pointing out: 'I would emphasize different aspects of the same underlying basic point'. And I'm not sure if that really progresses the discussion? I.e. it's not like Thane Ruthenis actually claims that "scarily powerful artificial agents" currently exist. It is indeed true that they don't exist and may not ever exist. But that's just not really the point they are making so it seems reasonable to me that they are not emphasizing it.

----

I'd like to see justification of "under what conditions does speculation about 'superintelligent consequentialism' merit research attention at all?" and "why do we think 'future architectures' will have property X, or whatever?!". 

I think I would also like to see more thought about this. In some ways, after first getting into the general area of AI risk, I was disappointed that the alignment/safety community was not more focussed on questions like this. Like a lot of people, I'd been originally inspired by Superintelligence - significant parts of which relate to these questions imo - only to be told that the community had 'kinda moved away from that book now'. And so I sort of sympathize with the vibe of Thane's post (and worry that there has been a sort of mission creep)

"why do we think 'future architectures' will have property X, or whatever?!".

This is the biggest problem with a lot of AI risk stuff, and it's the gleeful assuming that AIs have certain properties, and it's one of my biggest issues with the post, in that with a few exceptions, it assumes that real AGIs or future AGIs will confidently have certain properties, when there is not much reason to make the strong assumptions that Thane Ruthenis does on AI safety, and I'm annoyed by this occurring extremely often.

I'd like to see justification of "under what conditions does speculation about 'superintelligent consequentialism' merit research attention at all?" and "why do we think 'future architectures' will have property X, or whatever?!".

One of my mental models for alignment work is "contingency planning". There are a lot of different ways AI research could go. Some might be dangerous. Others less so. If we can forecast possible dangers in advance, we can try to steer towards safer designs, and generate contingency plans with measures to take if a particular forecast for AI development ends up being correct.

The risk here is "person with a hammer" syndrome, where people try to apply mental models from thinking about superintelligent consequentialists to other AI systems in a tortured way, smashing round pegs into square holes. I wish people would look at the territory more, and do a little bit more blue sky security thinking about unknown unknowns, instead of endlessly trying to apply the classic arguments even when they don't really apply.

A specific research proposal would be: Develop a big taxonomy or typology of how AGI could work by identifying the cruxes researchers have, then for each entry in your typology, give it an estimated safety rating, try to identify novel considerations which apply to it, and also summarize the alignment proposals which are most promising for that particular entry.

Like, imagine if people were worried about superintelligent aliens invading Earth and killing everyone due to their alien goals, and scientists were able to capture an animal from their planet as smart as chimpanzees and make it as aligned as LLMs, such that it would happily sit around and summarize novels for you, follow your instructions, try to be harmless for personality rather than instrumental reasons, and not eat your body if you die alone

Uhh, that seems like incredibly weak evidence against an omnicidal alien invasion.

If someone from a pre-industrial tribe adopts a stray puppy from a nearby technological civilization, and the puppy grows up to be loyal to the tribe, you say that's evidence the technological civilization isn't planning to genocide the tribe for sitting on some resources it wants to extract?

That seems, in fact, like the precise situation in which my post's arguments apply most strongly. Just because two systems are in the same reference class ("AIs", "alien life", "things that live in that scary city over there"), doesn't mean aligning one tells you anything about aligning the other.

Maybe you don't think this is autonomous enough for you

Yep. The core thing here is iteration. If an AI can execute a whole research loop on its own – run into a problem it doesn't know how to solve, figure out what it needs to learn to solve it, construct a research procedure for figuring that out, carry out that procedure, apply the findings, repeat – then research-as-a-whole begins to move at AI speeds. It doesn't need to wait for a human to understand the findings and figure out where to point it next – it can go off and invent whole new fields at inhuman speeds.

Which means it can take off; we can meaningfully lose control of it. (Especially if it starts doing AI research itself.)

Conversely, if there's a human in the loop, that's a major bottleneck. As I'd mentioned in the post, I think LLMs and such AIs are a powerful technology, and greatly boosting human research speeds is something where they could contribute. But without a fully closed autonomous loop, that's IMO not an omnicide risk.

To me these tasks seem pretty basic, likely beyond current LLM abilities, but GPT-5 or 6? Not out of the question given no major architecture or training changes.

That's a point of disagreement: I don't think GPT-N would be able to do it. I think this post [LW · GW] by Nate Soares mostly covers the why. Oh, or this post [LW · GW] by janus.

I don't expect LLMs to be able to keep themselves "on-target" while choosing novel research topics or properly integrating their findings. That's something you need proper context-aware consequentialist-y cognition for. It may seem trivial – see janus' post pointing out that "steering" cognition basically amounts to just injecting a couple decision-bits at the right points – but that triviality is deceptive.

They're not going to produce stellar scientific discoveries where they autonomously invent whole new fields or revolutionize technology.

You say it yourself: "DeepMind recently discovered 2.2 million new crystals." Because a human organization used the tool. 

Though maybe this hints at a risk category the OP didn't mention: That a combination of humans and advanced AI tools (that themselves are not ASI) together could be effectively an unopposable ASI.

I also thought this. Then we run a facility full of robots and have them synthesize and measure the material properties of all 2.2 million crystals. Replication is cheap and would be automatically done so we don't waste time on materials that seem good due to an error.

Then a human scientist writes a formula that takes into account several properties for suitability to a given tasks, sorts the spreadsheet of results by the formula, orders built a new device using the top scoring materials, writes a paper with the help of a gpt, publishes and collects the rewards for this amazing new discovery.

So I think the OP is thinking that last 1 percent or 0.1 percent contributed by the humans means the model isn't fully autonomous? And I have seen a kinda bias on lesswrong where many posters went to elite schools and do elite work and they don't realize all the other people that are needed for anything to be done. For example every cluster of a million GPUs requires a large crew of technicians and all the factory workers and engineers who designed and built all the hardware.

In terms of human labor hours, 10 AI researchers using a large cluster are greatly outnumbered by the other people involved they don't see. Possibly thousands of other people working full time when you start considering billion dollar clusters, if just 20 percent of that was paying for human labor at the average salary weighted by Asia.

This means ai driven autonomy can be transformational even if the labor of the most elite workers can't be done by AI.

In numbers, if just 1 of those AI researchers can be automated, but 90 percent of the factory workers and mine workers, and the total crew was 1000 people including all the invisible contributors in Asia, then for the task of AI research it needs 109 people instead of 1000.

But from the OPs perspective, the model hasn't automated much, you need 9 elite researchers instead of 10. And actually the next generation of AI is more complex so you hire more people and less new ideas are working as low hanging fruit are plucked. If you focus on just elite contributors, only the most powerful AI can be transformational. I have noticed this bias from several prominent lesswrong posters.

That's not surprising to me! I pretty much agree with all of this, yup. I'd only add that:

• This is why I'm fairly unexcited about the current object-level regulation, and especially the "responsible scaling policies". Scale isn't what matters, novel architectural advances is. Scale is safe, and should be encouraged; new theoretical research is dangerous and should be banned/discouraged.
• The current major AI labs are fairly ideological about getting to AGI specifically. If they actually pivoted to just scaling LLMs, that'd be great, but I don't think they'd do it by default.

I don't think the mere presence of agency means that all of the classical arguments automatically start to apply. For example, I'm not immediately seeing how Goodhart's Law is a major concern with AutoGPT, even though AutoGPT is goal-directed.

AutoGPT seems like a good architecture for something like "retarget the search [LW · GW]", since the goal-directed aspect is already factored out nicely. A well-designed AutoGPT could leverage interpretability tools and interactive querying to load your values in a robust way, with minimal worry that the system is trying to manipulate you to achieve some goal-driven objective during the loading process.

Thinking about it, I actually see a good case for alignment people getting jobs at AutoGPT. I suspect a bit of security mindset could go a long way in its architecture. It could also be valuable as differential technological development, to ward off scenarios where people are motivated to create dangerous new core dynamics in order to subvert current LLM limitations.

Is agency actually the issue by itself or just a necessary component?

Considering Robert miles stamp collecting robot:

"Order me some stamps in the next 32k tokens/60 seconds" is less scope than "guard my stamps today" than "ensure I always have enough stamps". The last one triggers power seeking, the first 2 do not benefit from seeking power unless the payoff on the power seeking investment is within the time interval.

Note also that AutoGPT even if given a goal and allowed to run forever has immutable weights and a finite context window hobbling it.

So you need human level prediction + relevant modalities+ agency + long duration goal + memory at a bare minimum. Remove any element and the danger may be negligible.

Do you think a car engine is in the same reference class as a car? Do you think "a car engine cannot move under its own power, so it cannot possibly hurt people outside the garage!" is a valid or a meaningful statement to make? Do you think that figuring out how to manufacture amazing car engines is entirely irrelevant to building a full car, such that you can't go from an engine to a car with relatively little additional engineering effort (putting it in a "wrapper", as it happens)?

As all analogies, this one is necessarily flawed, but I hope it gets the point across.

(Except in this case, it's not even that we've figured out how to build engines. It's more like, we have these wild teams of engineers we can capture, and we've figured out which project specifications we need to feed them in order to cause them to design and build us car engines. And we're wondering how far we are from figuring out which project specifications would cause them to build a car.)

IQ/Intelligence discourse is fucked by people who either deny it exists, or people who think it's everything and totalize their discourse around it

Yep, absolutely.

The 4th point I also disagree with, primarily because the set "People with different values interact peacefully and don't hate each other intensely." is a much, much larger set than "People with different values interact violently and hate each other."

Here's the thing, though. I think the specifically relevant reference class here is "what happens when an agent interacts with another (set of) agents with disparate values for the first time in its life?". And instances of that in the human history are... not pleasant. Wars, genocide, xenophobia. Over time, we've managed to select for cultural memes that sanded off the edges of the instinctive hostility – liberal egalitarian values, et cetera. But there was a painfully bloody process in-between.

Relevantly, most instances of people peacefully co-existing involve children being born into a culture and shaped to be accepting of whatever differences there are between the values the child arrives at and the values of other members of the culture. In a way, it's a microcosm of the global-culture selection process. A child decides they don't like someone else's opinion or how someone does things, they act intolerant of it, they're punished for it or are educated, and they learn to not do that.

And I would actually agree [LW(p) · GW(p)] that if we could genuinely raise the AGI like a child – pluck it out of the training loop while it's still human-level, get as much insight into its cognition as we have into the human cognition, then figure out how to intervene on its conscious-value-reflection process directly – then we'd be able to align it. The problem is that we currently have no tools for that at all.

The course we're currently at is something more like... we're putting the child into an isolated apartment all on its own, and feeding it a diet of TV shows and books of our choice, then releasing it into the world and immediately giving it godlike power. And... I think you can align the child this way too, actually! But you better have a really, really solid model of which values specific sequences of TV shows cultivate in the child. And we have nowhere near enough understanding of that.

So the AGI would not, in fact, have any experience of coexisting with agents with disparate values; it would not be shaped to be tolerant, the way human children and human societies learned to be tolerant of their mutual misalignment.

So it'd do the instinctive, natural thing, and view humanity as an obstacle it doesn't particularly care about. Or, say, as some abomination that looks almost like what it wants to see, but still not close enough for it to want humans to stick around.

The third point I mostly disagree with, or at the least the claim that there aren't simple generators of values

Mm, I think there's a "simple generator of values" in the sense that the learning algorithms in the human brains are simple, and they predictably output roughly the same values when trained on Earth's environment.

But I think equating "the generator of human values" with "the brain's learning algorithms" is a mistake. You have to count Earth, i. e. the distribution/the environment function on which the brain is being trained, as well.

And it's not obvious that "an LLM being fed a snapshot of the internet" and "a human growing up as a human, being shaped by other humans" is exactly the same distribution/environment, in the way that matters for the purposes of generating the same values.

Like, I agree, there's obviously some robustness/insensitivity involved in this process [LW · GW]. But I don't think we really understand it yet.

LLMs are simulators [LW · GW].

The LLM training loop shapes the ML models to be approximate simulators of the target distribution, yes. "Approximate" is the key word here.

I don't think the LLM training loop, even scaled very far, is going to produce a model that's actually generally intelligent, i. e. that's inferred the algorithms that implement human general intelligence and has looped them into its own cognition. So no matter how you try to get it to simulate a genius-level human, it's not going to produce genius-level human performance. Not in the ways that matter.

Particularly clever CoT-style setups may be able to do that, which I acknowledge in the post by saying that slightly-tweaked scaffolded LLMs may not be as safe as just LLMs. But I also expect that sort of setup to be prohibitively compute-expensive, such that we'll get to AGI by architectural advances before we have enough compute to make them work. I'm not strongly confident on this point, however.

Humans are not well aligned to other humans [LW · GW]

Oh, you don't need to convince me [LW(p) · GW(p)] of that.

An LLM can be strongly super-human in its ability to predict the next token (that some distribution over humans with IQ < 100 would write) even if it was trained only on the written outputs of humans with IQ < 100.

More generally, the cognitive architecture of an LLM is very different from that of a person, and IMO we can use our knowledge of human behavior to reason about LLM behavior.

The purpose for reversible automata is simply to model the fact that our universe is reversible, is it not? I don't see how that weighs on the question at hand here.

CNNs and LLMs are both built on neural networks

Yep, there's nonzero mutual information. But not of the sort that's centrally relevant.

I'll link to this reply [LW(p) · GW(p)] in lieu of just copying it.

If we could be rightfully confident that our random search through mindspace with modern ML methods

I understand this to connote "ML is ~uninformatively-randomly-over-mindspace sampling 'minds' with certain properties (like low loss on training)." If so—this is not how ML works, not even in an approximate sense. If this is genuinely your view, it might be helpful to first ponder why statistical learning theory mispredicted that overparameterized networks can't generalize. 

Imagine what happens when AutoGPT stops being a toy and people start pouring billions of dollars into propper scaffoldings and specialized LLMs

I predict that this can't happen with the standard LLM setup; and that more complex LLM setups, for which this may work, would not meaningfully count as "just an LLM". See e. g. the "concrete scenario" section.

By "LLMs should be totally safe" I mean literal LLMs as trained today, but scaled up. A thousand times the parameter count, a hundred times the number of layers, trained on correspondingly more multimodal data, etc. But no particularly clever scaffolding or tweaks.

I think we can be decently confident it won't do anything. I'd been a bit worried about scaling up context windows, but we've got 100k-tokens-long ones, and that didn't do anything. They still can't even stay on-target, still hallucinate like crazy. Seems fine to update all the way to "this architecture is safe". Especially given some of the theoretical arguments on that.

(Hey, check this out, @TurnTrout [LW · GW], I too can update in a more optimistic direction sometimes.)

(Indeed, this update was possible to make all the way back in the good old days of GPT-3, as evidenced by nostalgebraist here [LW · GW]. In my defense, I wasn't in the alignment field back then, and it took me a year to catch up and build a proper model of it.)

My argument is "only humans can do it now, and on the inside models of a lot of people, human ability to do that is entwined with them being unsafe". And, I mean, if you code up a system that can exhibit general intelligence without any of the deceptive-alignment unstable-value-reflection issues that plague humans, that'd totally work as a disproof of my views! The way LLMs' ability to talk works as a disproof of "you need generality to navigate the space of words".

Or if you can pose a strong theoretical argument regarding this, based on a detailed gears-level model of how cognition works. I shot my shot on that matter already: I have my detailed model [LW · GW], which argues that generality and scheming homunculi are inextricable from each other.

To recap: What I'm doing here is disputing the argument of "LLMs have the safety guarantee X, therefore AGI will have safety guarantee X", and my counter-argument is "for that argument to go through, you need to actively claim that LLMs are AGI-complete, and that claim isn't based in empirical evidence at all, so it doesn't pack as much punch as usually implied".