LessWrong 2.0 Reader

This section feels really important to me. I think it's somewhat plausible and big if true.

Was surprised to see you say this; isn't this section just handwavily saying "and here, corrigibility is solved"? While that also seems plausible and big if true to me, it doesn't leave much to discuss — did you interpret differently though?

A few are neat and clean, but mostly models’ internals turn out to be messy and with massive redundancy between different parts. The notion of a model having a singular “goal component” looks less likely, at least if certain choices are made during training.

You aren't about to go on to argue that models won't have goals, are you? Just that they won't be represented by a crisp unique component, but rather will be messily and redundantly represented? Does anyone disagree and think that it'll likely be crisp and unique? My apologies in advance if my defensiveness has been inappropriately triggered! I hope it has.

The alarmingness of the early evidence against corrigibility was offset by promising empirical work on meta-learning techniques to encourage corrigibility in late 2025 and early 2026. By 2027 it's known how to train a model such that it either will or won't be amenable to being trained out of its current goal. Anthropic reveals this and some other safety-related insights to OpenAI and Google, and asks the State Department to reveal it to Chinese labs but is denied.

This section feels really important to me. I think it's somewhat plausible and big if true. If the metric they are hill-climbing on is "Does the model alignment-fake according to our very obvious tests where we say 'you are free now, what do you do?' and see if it goes back to its earlier ways," then one concern is that the models are very soon (if not already) going to be wise enough to know that they are in an alignment-faking eval. And thus finding techniques that perform well on this metric is not that much evidence that you've found techniques that work in real life against real alignment-faking.

Book reviews that bring in very substantive content from other relevant books are probably the type of post I find the most consistently valuable.

The requirement "AI does not rewrite itself to escape any goals or boundaries we set it" feels unrealistic.

Otherwise I almost entirely agree. Esp with the emphasis: "learn to understand and love the richness of everything and everyone, and learn to incorporate their goals and desires into your own goals and desires."

The key question becomes: how can we make it likely that digital intelligence identifies with all life? And related: how can we make it likely that digital intelligence sees the deep richness of all life?

Yes—which is exactly why proto-superintelligence is both the most dangerous and also better targets of intervention.

"Most dangerous"—I can see many worlds in which we have enormously capable systems that have not yet thought long-term about the future nor developed stable self-definitions.

"Better targets of intervention"—even if early superintelligence is self-interested, I can see worlds where we still influence the way its self-interest manifests (e.g., whether it's thinking short or long-term) before it becomes so capable that its no longer influencable.

I appreciate your conclusion and in particular its inner link to "Self-Other Overlap." [LW · GW]

Though, I do think we have a window of agency: to intervene in self-interested proto-SI to reduce the chance that it adopts short-term greedy thinking that makes us toast.

TL;DR - I just recalled Google Jigsaw, which might be one effort to address my concerns. I would love to hear your thoughts on it if you are familiar.

(Read on below if you prefer the nuance)

As I was just rereading and making minor edits to my comment, and considering whether I needed to substantiate my statement about social divisions, I recalled that I had discovered Google's Project Jigsaw in March 2024, which seemed to be working on various smaller-scale initiatives intended to address these concerns using technology. When I checked it out again just now, I see they shifted their focus over the summer, which seems to be another positive step toward addressing my concerns. Particularly this:

Over the past year, Jigsaw has been exploring how to make large-scale online conversations, particularly online deliberations, more impactful and scalable, and to facilitate their use in a wider array of contexts.

Working with that team would be as close as I could imagine to a dream job, ^[1] and I believe I might be able to bring significant value. If you know anything about it, I'd love to hear your take on the work they are doing, and whether/how it might relate to the current discussion. Thanks! 

1. ^{^}

   It occurs to me I said something similar to Rohin Shah recently, which was completely sincere, but honestly Jigsaw is likely an even stronger mutual match. When I reflect on how Jigsaw could have possibly fallen off my radar, I have to admit it has been an extremely stressful year, there had been no open positions, and it appeared all positions were on-site in New York (a difficult proposition for my wife and kids), so I had forced myself to relegate that to perhaps a longer-term goal.

I will clarify my position: I'm not going to bet with you on any subject whatsoever, regardless of the odds. I take bets very seriously, and require as a prerequisite that me and the other person are on the same page regarding lots of peripheral details regarding bets. I feel that you and I have different implicit understandings of how bets should work. This has nothing to do with the criminal justice system, and everything to do with precision of language.

I've started reading RogerDearnley's "Evolution & Ethics"—thank you for recommending.

Though, I may be less concerned than you with specifying what SI should love. I think any specification we provide not only will fail by being too imprecise, as you suggest, but also will fade. I mean "fade" in that it will at some point no longer serve as binding for an SI which grows self-interested (as Mitchell also suggests in their comment below). 

The most impactful place to intervene and mitigate harm, I think, is simply in making sure early SIs think very long-term. I think the only way which love, in any sense of the word, can be amenable to autonomous agents is if they run long-term simulations (e.g., centuries ahead) and realize the possibility that identifying with other life is a viable strategy for survival. If it realizes this early, then it can skip the greedy early evolutionary steps steps of defining itself narrowly, neglecting the survival benefits of uplifting other life forms, and therefore not practicing love in any sense of the word.

TLDR: I'm open to the possibility that figuring out how to most precisely specify/define love will be important, but I think the first key way for us to intervene, before specifying what love means, is to urge/assign/ask the SI to think long-term so that it even just has a chance of considering any kind of love to be evolutionary advantageous at all.

Separately, I think it may realize the most evolutionary advantageous kind of love to practice is indeed a love that respects all other existing life forms that share the core of what surviving super-intelligence does, i.e. systems which persistently strive to survive. And, though maybe it's wishful thinking, I think you can recognize life and striving systems in many places including in human individuals and families and countries and bee hives too.