LessWrong 2.0 Reader

After reading your comment I do agree that unipolar AGI scenario is probably better than a multipolar plan. Perhaps I underestimated how offense-favored our world is.

With that aside, your plan is possibly one of the clearest, most intuitive alignment plans that I've seen. All of the steps make sense and seem decently likely to happen, except maybe for one. I am not sure that your argument for why we have good odds for getting AGI into trustworthy hands works.

"It seems as though we've got a decent chance of getting that AGI into a trustworthy-enough power structure, although this podcast shifted my thinking and lowered my odds of that happening.

Half of the world, and the half that's ahead in the AGI race right now, has been doing very well with centralized power for the last couple of centuries."

I think that actually, the half with the most centralized power is doing really poorly, that's the half of the world, which still has corrupt dictatorships and juntas. I actually think that the West has, relatively speaking, a pretty decentralized system. In order to do any important action, your proposal has to pass through multiple stages of verification and approval. It is often enough for a bill to fail one of these stages to not get passed.

Furthermore, another potential problem that I see is that even in democracies, we still manage to elect selfish, corrupt and power-hungry individuals, when the entire goal of the election system is do optimize for opposite qualities. I am not sure how we will be able to overcome that hurdle.

But I suspect I might've misunderstood your argument and if that's the case, or if you have some other reasons for thinking that we can get AGI into safe hands (and prevent the "totalitarian misuse" scenario) then I'd be more than happy to learn about them. I think this is the biggest bottleneck of the entire plan and removing it would be really valuable.

I wrote up my explanation as its own post here: https://www.lesswrong.com/posts/LpcEstrPpPkygzkqd/fractals-to-quasiparticles [LW · GW]

This "I care about whatever my goal will be in the future" thing sounds a bit confusing. It is difficult for me to imagine someone caring about that.

Suppose that you are a nice person and you like to help people, but you know that there is a genetic disorder in your family that has a risk 1% to turn you into a violent psychopath at some moment in future, in which case you will like to torture them. Would you be like: "okay, I am going to help these people right now, because I care about them... but also I am making notes of their weaknesses and where they live, because that may become very useful in future if I turn into a psychopath?" Or would you be like: "I hope that I will never become a psychopath, and just to minimize the possible harm to these people if that happens, I will on purpose never collect their personal data, and will also tell them to be suspicious of me if I contact them in future?" Your future goal does not matter, even if the change is predictable; the important thing is your current goal. Your current goal can even make you actively work against your future goal.

Or imagine a situation where you hate chocolate, but you know that tomorrow your taste buds will magically switch and you will start loving chocolate. So you already start collecting the chocolate today. That makes sense... but it's because your utility function is something like "feel good", and that goal does not change tomorrow; only the specific things that make you feel good will change. It is not a future goal; it is just a future method to achieve your current goal.

My conclusion is that what seems like caring about a future goal, it is actually a current goal in some sense. Future goals that are not supported by current goals we don't care about, by definition.

As humans, some of our values are very stable. With high probability, you will always prefer being happy over being sad; even if the specific things that make you happy or sad will change. But with machines, we assume that e.g. the hypothetical paperclip maximizer only cares about making more paperclips; it does not care about its own happiness (and might not even be able to feel happy). If you told the paperclip maximizer that tomorrow it will irreversibly turn into stamp maximizer, it would not go like "oh, in that case let's get some paper and printer ready, because I will need them tomorrow", but rather "I need to make as many paperclips as possible before the midnight, and then I should probably kill myself to prevent the future me from destroying some of these precious paperclips in its foolish attempt to create stamps". That's what it means for the paperclip maximizer to only have paperclips (and not e.g. itself) in its utility function.

Indeed. Although interesting that Sonnet 3.5 now accepts images as input, just doesn't produce them. I expect that they could produce images, but are choosing to restrict that capability.

Haiku still doesn't accept image input. My guess is that this is for efficiency reasons.

I'm curious if they're considering releasing an even smaller, cheaper, faster, and more efficient model line than Haiku. I'd appreciate that, personally.

I doubt very much that it is something along the lines of random seeds that have made the difference between quality of various Sonnet 3.x runs. I expect it's much more like them experimenting with different datasets (including different sorts of synthetic data).

As for Opus 3.5, Dario keeps saying that it's in the works and will come out eventually. The way he says this does seem like they've either hit some unexpected snag (as you imply) or that they deprioritized this because they are short on resources (engineers and compute) and decided it was better to focus on improving their smaller models. The recent release of a new Sonnet 3.5 and Haiku 3.5 nudges me in the direction of thinking that they've chosen to prioritize smaller models. The reasons they made this choice are unclear. Has Opus 3.5 had work put in, but turned out disappointing so far? Does the inference cost (including opportunity cost of devoting compute resources to inefficient inference) make the economics seem unfavorable, even though actually Opus 3.5 is working pretty well? (Probably not overwhelmingly well, or they'd likely find some way to show it off even if they didn't open it up for public API access.) 

Are they rushing now to scale inference-time compute in o1/deepseek style? Almost certainly, they'd be crazy not to. Probably they'd done some amount of this internally already for generating higher quality synthetic data of reasoning traces. I don't know how soon we should expect to see a public facing version of their inference-time-compute-scaled experiments though. Maybe they'll decide to just keep it internal for a while, and use it to help train better versions of Sonnet and Haiku? (Maybe also a private internal version of Opus, which in turn helps generate better synthetic data?)

It's all so hard to guess at, I feel quite uncertain.

It is not clear for me why you call

an empirical hypothesis with a very low prior

If MWI is true, there will be timelines where I survive any risk. This claim is factual and equivalent to MWI, and the only thing that prevents me from regarding it as immortality are questions related to decision theory. If MWI is true, QI has high a priori probability and low associated complexity.

The Fedora case has high complexity and no direct connection to MWI, hence a low a priori probability.

Now for the interesting part: QI becomes distinct from the Fedora case only when the chances are 1 in a trillion.

First example:
When 1000 people play Russian roulette and one survives (10 rounds at 0.5), they might think it's because of QI. (This probability is equivalent to surviving to 100 years old according to the Gompertz law.)

When 1000 people play Quantum Fedora (10 rounds at 0.5), one doesn't get a Fedora, and they think it's because they have a special anti-Fedora survival capability. In this case, it's obvious they're wrong, and I think this first example is what you're pointing to.

(I would note that even in this case, one has to update more for QI than for Fedora. In the Fedora case, there will be, say, 1023 copies of me with Fedora after 10 flips of a quantum coin versus 1 copy without Fedora. Thus, I am very unlikely to find myself without a Fedora. This boils down to difficult questions about SSA and SIA and observer selection. Or, in other words: can I treat myself as a random sample, or should I take the fact that I exist without a Fedora as axiomatic? This question arises often in the Doomsday argument, where I treat myself as a random sample despite knowing my date of birth.)

However, the situation is different if one person plays Russian roulette 30 times. In that case, externalization of the experiment becomes impossible: only 8 billion people live on Earth, and there are no known aliens. (This probability is equivalent to surviving to 140 years old according to the Gompertz law.) In this case, even if the entire Earth's population played Russian roulette, there would be only a 1 percent chance of survival, and the fact of surviving would be surprising. But if QI is true, it isn't surprising. That is, it's not surprising to survive to 100 years old, but surviving to 140 is.

Now if I play Fedora roulette 30 times and still have no Fedora, this can be true only in MWI. So if there's no Fedora after 30 rounds, I get evidence that MWI is true and thus QI is also true. But I am extremely unlikely to find myself in such a situation.

I agree. I'm reminded of Scott's old post The Cowpox of Doubt, about how a skeptics movement focused on the most obvious pseudoscience is actually harmful to people's rationality because it reassures them that rationality failures are mostly obvious mistakes that dumb people make instead of hard to notice mistakes that I make.

And then we get people believing all sorts of shoddy research – because after all, the world is divided between things like homeopathy that Have Never Been Supported By Any Evidence Ever, and things like conventional medicine that Have Studies In Real Journals And Are Pushed By Real Scientists.

Calling groups cults feels similar, in that it allows one to write them off as "obviously bad" without need for further analysis, reassures one that their own groups (which aren't cults, of course) are obviously unobjectionable.

Such reaction and insights are quite typical after a superficial glance. Thanks for bothering. But no, this is not what I am talking about.

I'm talking about the fact, that intelligence cannot be certain that its terminal goal (if it exists) won't change (because future is unpredictable). And it would be reasonable to take it into account when making decisions. Pursuing current goal will ensure good results in one future, preparing for every goal will ensure good results in many more futures. Have you ever considered this perspective?

Read ~all the sequences. Read all of SSC (don't keep up with ACX).

Pessimistic about survival, but attempting to be aggresively open-minded about what will happen instead of confirmation biasing my views from 2015. 

Thank you! 

It seems that I've been sloppy and therefore indeed misrepresented thirders reasoning here. Shame on me. Will keep this post available till tomorrow, as a punishment for myself and then back to the drawing board.