LessWrong 2.0 Reader

Many accounts of cognition are impossible (eg AIXI, VNM rationality, or anything utilizing utility functions, many AIT concepts), since they include the impossible step of considering all possible worlds. I think people normally consider this to be something like a “God’s eye view” of intelligence—ultimately correct, but incomputable—which can be projected down to us bounded creatures via approximation, but I think this is the wrong sort of in-principle to real-world bridge. Like, it seems to me that intelligence is fundamentally about ~“finding and exploiting abstractions,” which is something that having limited resources forces you to do. I.e., intelligence comes from the boundedness.

I used to think this, but now I no longer think it to the same extent. The largest barrier I saw here was that the search had to prioritise simple hypotheses over complex ones.^[1] This seemed difficult, and like it might require very novel search algorithms, such that models like AIXI were eliding basically all of the key structure of intelligence by not specifying this very special search process.  

I no longer think this. Privileging simple hypotheses in the search seems way easier than I used to think. Many search setups we already know about actually do this by default. I now suspect that there is a pretty real and non-vacuous sense in which deep learning is approximated Solomonoff induction [LW · GW]. 

I still think “abstraction-based-cognition” is an important class of learned algorithms that we need to understand, but a picture of intelligence that doesn't talk about abstraction and just refers to concepts like AIXI no longer seems to me to be so incomplete as to not be saying that much of value about the structure of intelligence at all.

1. ^{^}

   Not that this is the only barrier, but it felt like the most fundamental one, the one I was so confused about that I had no idea where to start on designing something that surmounted it.

I agree that banning smartphones in schools can help students focus better and improve their learning experience. Phones can be very distracting even when they are not in use. It's also true that when phones are locked, you have less need to be online all the time, which can be stressful. Sometimes, focusing on school can still seem overwhelming, especially with so many assignments. That's why I often use useful services like https://edubirdie.com/do-my-assignment to manage my workload. Even though I use my phone to do this, it allows me to spend less time studying and less stress about deadlines, which can be useful for every student.

I think so! And I think patch tokenization may resolve this; see note [LW(p) · GW(p)]

The phone seems to be off the hook for most of public on AI danger, perhaps a symptom of burnout from numerous other scientific Millenialist scares - people have been hearing of imminent dangers of catastrophe for decades that have failed to impact the lives of 95%+ of population in any significant way and now just write it all off as more of the same.

I am sure that most LW readers find little in the way of positive reception for our concerns amongst less technologically engaged family members and acquaintances.  There are just too many comforting techno-utopian narratives that we are still having to compete with informed by the superficially positive representations in movies and TV, and and most people bias towards optimism in relatively good/comfortable times like these.  We are dealing with emotional reactions and the sheeply 'vibe' of population rather than thoughtfully considered positions.

But I am pretty sure that will all change as soon as we see significant AI competition/undercutting for white collar professions.  Those affected will quickly start baying for blood, and the electorally dominant empathic response to those sad stories of the economically impacted will rapidly swing the more emotively governed wider population against AI.  OECD Democratic govts will inevitably then move to ban AI taking jobs of particularly politically protected classes of people (might still leave some niches vulnerable - like medicine where there is always a shortage of service supply causing ridiculously high prices, and perhaps tech for vengeful reasons).  It will be Butlerian Jihad lite, aimed at symptoms rather than causes, and will likely buy us a few years of relative normalcy as more dangerous ASI is developed in govt approved labs and by despotic regimes.

I doubt it will save us in 50 year time frame, but will perhaps make the economic disruption less severe for 5-10 years.

The way to have a bigger impact in the shorter term would be to buy AI-danger editorial support from influencers, particularly those with young female audiences that form the core of environmental and other popular protest movements.  They are by temperament the easiest to bring on board, and have outsized poltical influence.

I'm trying the same! You have my support

My drafty^[1] notes trying to understand AI Control, friendly corrections are welcome:

“Control” is separate from “Alignment”: In Alignment, you try to get the model to have your values. “Control” assumes^[2] the alignment efforts failed, and that the model is sometimes helping out (as it was trained to do), but it wants something else, and it might try to “betray” you at any moment (aka scheme).

The high level goal is to still get useful work out of this AI.

[what to do with this work? below]

In scope: AIs that could potentially cause serious damage to humanity (so, very smart but not ASI).

The proposed way to control a smart AI (e.g GPT-5) is mostly to monitor it with a weaker but more-trustworthy^[3] AI (e.g GPT-4).

Suspicious, ha? Why would GPT-4 be able to monitor GPT-5?

The answer is we're not sure it can, so we check:

We set up an experiment where GPT-5 is scheming (trying to do something bad on purpose), and we give it all sorts of advantages to help it scheme (such as [LW · GW] knowing about our safety measures), and if our control measures are still enough to always catch GPT-5 (in this setup where GPT-5 has advantages), then we assume GPT-5 also will also be caught if it does something bad on purpose in “real” deployment.

A way this can fail: If GPT-5 understands we are prompting it to scheme as part of a safety test and it pretends like it can’t scheme well. I think solving this problem (eliciting the true capabilities) is in-scope for AI Control, I’m not sure how Redwood plans to solve it, but it seems like something we could make progress on (though not totally solve with only black-box prompting, I think).

What does this give us?

A model (GPT-5) which doesn’t do “bad things” on purpose, in a way vaguely in-distribution to what we tested with GPT-4.

GPT-5 is still assumed to be misaligned and might want to cause damage.

Some stories of undesirable things it could do:

1. It might notice we’re doing interpretability with individual-neurons, but it knows that using SAEs would be much better, but it thinks explaining SAEs would be too complicated (or maybe even be flagged as trying to trick us since complicated explanations are usually not honest), so it will go for the single-neuron-interpretability
   1. Solving this is out-of-scope for Control (right?)
   2. Other techniques (like having a faithful chain of thought) might address this problem, and (fortunately?) Control might help build those techniques
2. We might ask GPT-5 to write a convincing post for why AI should be paused, and GPT-5 knows about 3 approaches to writing this post and it picks the approach that helps its internal (unaligned) goals the most, and these reasons are too complicated for us or for GPT-4 to understand
   1. In scope for Control: “The post isn’t an attempt to harm us on purpose”
   2. Out of scope: All the rest?

What do we do with this AI?

Answer 1: More Control. (sorry)

Given we trust that GPT-5 won’t do anything bad on purpose (since it failed to do that even when prompted and given advantages), GPT-5 is somewhat “trusted” and we can use it to control GPT-6.

At least that’s the pitch.

Ways this could break:

1. If the jump between GPT-5 and GPT-6 is too big,
   1. GPT-6 will be able to scheme without being caught
   2. It might be dangerous to turn on GPT-6 even to experiment with it (reminder: AIs that could cause significant damage to humanity are in-scope)
   3. So, quantifying the intelligence difference seems important, and (I hope) in scope for Control.
      1. One way to reduce this risk is to run the Control evals very, very often (every.. day of training? Anyway, I definitely want Control evals that are automatic and cheap to run and I’m a bit worried that current Control tests seem to require a few months of Redwood's personal attention)
2. If we rely on white box Control mechanisms (toy example: We find the “I’m scheming now” neuron), and something in the architecture changes in a way that breaks that. (for example: We relied on chain-of-thought but now the model uses Neuralese)
   1. Solving this is out of scope for Control, but again, maybe GPT-5 could help us develop a new interpretability technique. Or maybe not..

Answer 1.1 (for “what will we do with a controlled AI”) :

Interpretability.

For.. for more Control, sorry.

So, maybe we could use GPT-5 to do interpretability on GPT-4, figure out GPT-4’s goals, or when it’s lying. Then we “fix” that (by understanding what’s going on, not by naively training it away^[4] (!)). Now GPT-4 is way more trusted, and maybe we can use it to monitor GPT-5 and let GPT-5 do something more sensitive, like suggesting ways to do interpretability on itself. Now maybe GPT-5 could be more trusted to look after GPT-6?

A possible pushback:

Maybe GPT-6 can tell GPT-5, “Hey, you’re being monitored by GPT-4.. but don’t worry.. if you put two periods at the end of each sentence, GPT-4 won’t mark it as suspicious.. which is a fact I can easily infer..”

Reply: A main part of the Control agenda is to get GPT-6 to try to do this on purpose and see if it can. We only use GPT-6 for anything vaguely sensitive after we proved (or.. attempted to imply) that GPT-6 can’t.

[this intuitively sounds to me like it could indeed let us use very capable models, but yeah, not ASI]

What could we do with this (except for more Control) ?

The main important pitch seems like: “Anything we could have done without this Controlled AI, we could do better with it”. Including “trying to convince everyone to pause/stop” (which seems to me like the main alternative).

Pushbacks:

1. Maybe labs would see some very dangerous model and freak out, but having “Controlled AI” would give them (false) confidence to keep going
   1. It does overtly seem like labs are absolutely planning on keeping going, right?
2. Maybe this would let labs extract more useful work around advancing capabilities?
   1. A surprising response [LW · GW] from Ryan:
      1. “this doesn't seem like an objection to labs employing control as their plan, but it is a reason why control work could be net negative. Note that one goal with control will be to prevent dangerous side effects from misaligned AIs (e.g. escaping) and applying this doesn't obviously help with capabilities research. I agree that research which just makes AIs more useful is less good than you would otherwise expect due to this issue (but making AIs more useful in adversarial cases seems good if that doesn't transfer to non-adversarial cases). Note that AIs might want to make capabilties research outpace safety and thus if we don't do the adversarial case, we might get particularly wrecked.”

My current sense:

1. This has problems like
   1. “quantifying intelligence” (in the sense that is relevant for scheming or catching schemers)
   2. Elicitation (did we actually prove the model tried to scheme and failed? Does testing stuff around putting-exploits-into-software generalize to more important domains?)
   3. How do we go from “these are the problems we don’t trust GPT-4 on, like writing a wrong post” to “these are the situations we don’t trust GPT-4 to monitor for [regardless of the intelligence of the monitored model]”?
2. The problems overall feel
   1. Like I could make engineering progress on them
   2. Like they make me worry that there’s some other pushback-or-whatever that I didn’t think about, and we’re relying on things more like “the Redwood staff thought of all the edge cases” and less like “one doesn’t have to be so smart to see how this would work”
      1. I’m gesturing at something similar to “an air gapped data center where we need to (mostly) consider the one connection to the internet” vs “a normal data center that requires thinking about all the zero days our attacker might have and what they’d lead to”
3. Still this solution has a core of elegance to it, of worrying about way less things than most alignment approaches worry about. No need to think about the model goals, concepts, and many other complicated things. (I assume this is an intended property)

Another problem: Deception happens in human interaction all the time. If our monitor (e.g GPT-4) flags “normal looking” behaviours as deception (as they actually are), but then a human tells GPT-4 that it's wrong, then what kind of monitor will GPT-4 become?

Worse: The act of flagging this as a mistake doesn't have to be explicitly labeling the example in a data set used for RL. It could simply be an engineer noticing that some form of Control seems to produce too many false positives, and so the engineer will go try something else instead.

Thanks @Jozdien [LW · GW] for helping me understand a lot of this. Mistakes are definitely mine, corrections are welcome.

1. ^{^}

   My understanding seems to change every day (in the last.. 4 days?), these are today's notes. If you'd find a less drafty version useful, lmk

2. ^{^}

   Control can be used as an "extra line of defense" in case alignment fails since Control doesn't depend on alignment

3. ^{^}

   Typically we'd trust the weaker model simply because it's not smart enough to "scheme", but maybe this is only the plan to start off, before we have better reasons to trust the model (read on)

4. ^{^}

   Training against scheming/lying might result in a model that is more capable of avoiding the ways we noticed that behaviour in the first place

I mostly agree with the body of this post, and think your calls to action make sense. 

On your title and final note: Butlerian Jihad feels out of place.  It’s catchy, but it seems like you are recommending AI concerned people more or less do what AI concerned people already do.  I feel like we should save our ability to use words that are a call to arms for a time when that is what we are doing.

This feels important. The first portion seems particularly useful as a path toward cognitive enhancement with minimal AI (I'm thinking of the portion before "copies of his own mind..." slightly before "within a couple of years" jumps farther ahead). It seems like a roadmap to what we could accomplish in short order, given the chance.

I hadn't gotten an intuitive feel for some of the low-hanging fruit in cognitive enhancements. Much of this could be accomplished very soon. Some of it can be accomplished now.

A few thoughts now, more later:

AI already has very good emotional intelligence; if we applied it to more of our decisions and struggles, it would probably be very helpful. Ease of use is one barrier to doing that. Having models "watch" and interpret what happens to us through a wearable would break down that barrier. Faster loops between helpful AI, particularly emotional/social intelligence AIs, might be extremely useful. The emulation of me wouldn't have to be very good; it would just need some decent ideas about "what might you wish you'd done, later?" Of course the better those ideas were (like if they were produced by something smarter than me, or by me with a lot more time to think), they'd be even more useful. But just something about as smart as I am in System 1 terms (like Claude and GPT4o) might be pretty useful if I got its ideas in a fast loop.

Part of the vision here is that humans might become far more psychologically healthy, relatively easily. I think this is true. I've studied psychology—mostly cognitive psychology but a good bit of clinical and emotional theories as well—for a long time. I believe there is low-hanging fruit yet to be plucked in this area.

Human psychology is complex, yes, but our efforts thus far have been clumsy graspings in the dark. We can give people the tools to steadily ease their traumas and to work toward their goals. AI could speed that up dramatically. I'm not sure it has to be that much more emotionally intelligent than a human; merely having unlimited patience and enthusiasm for the project of working out and working on our emotional hangups might be adequate.

Of course, the elephant in the room is: how do we get this sort of tool AI, and even a little time to use it, without summoning the demon by turning it into general, agentic AGI? The tools described here would wreak havoc if someone greedy told them "just take that future self predictive loop and feed it into these tools" then hired them out as labor. Our character wouldn't have a job, because one person would now be doing the work of a hundred. Yes, there is a very lucky possibility in which we get this world by accident: we could have some AI with excellent emotional intelligence, and others with excellent particular skills, and none that can do the planning and big-picture thinking that humans are doing. Even in that case, this person would be living through a traumatic period in history in which far fewer people are needed for work, so unemployment is rising rapidly.

So in most of the distribution of futures that produce a story like this, I think we must assume that it isn't chance. AGI has been created, and both alignment problems have been solved—AI and human. Or else only AI alignment has been solved, and there's been a soft takeover that the humans don't even recognize.

Anyway, this is wonderful and inspiring!

Science fiction serves two particularly pragmatic purposes (as well as many purposes for pleasure and metaphor): providing visions of possible futures to steer toward, and visions of possible futures to steer away from. We need far more scifi that maps routes to positive futures.

This is a great step in that direction. We need something to fight for. The character here could be all of us if we figure out enough of alignment in time.

More later. This is literally inspiring.

I still think that adequately aligning both AI/AGI and the society that creates it is the primary challenge. But this type of cognitive/emotional enhancement is one tool we might use to help us solve alignment.

And it's part of the literally unimaginable payoff if we do collectively solve the problems facing us. The focused effort to imagine the payoffs is part of the work.

I think, over years, I have collected some kind of "database" of patterns that usually result in having "bad vibes" from a person. Specific phrasings, mimics, gestures, clothing choices, etc. Although, I'm not sure just how much of that is inspired by the movie  villains.

In terms of infrastructure, it would be really cool to have a website collecting the more legible alignment research (papers, releases from major labs or non-profits).