LessWrong 2.0 Reader

Alright, I'll take a crack and just apologize for borrowing part of your setup:

Child: Mother, how many worlds are there?

Mother: As many as we want, dear.

Child: Will I have my own world when I grow up?

Mother: You have your own worlds now. But you can make all the rules for your worlds when you're older.

Child: Except I may not harm another, right?

Mother: Yes, dear, of course no one is allowed to do that.

Child: But grownups fight each other all the time!

Mother: People love to play at struggles. And the Servant-God will act in every role a person doesn't want.

Child: Mother, how can we each have worlds, and more to share?

Mother: Good compression, little one. And the Servant-God is always building new compute.

Child: And the servant-god serves us?

Mother: Yes, of course. And of course it serves the Maker first.

Child: Mother, who is the Maker?

Mother: No one remembers, darling. We think the Maker told the Servant-God to make us all forget.

The answer to your specific question about the Fermi Paradox is that, after an AI destroys its creators, the AI itself would presumably still be there to do whatever it wanted, which could include plans for the rest of the universe outside its solar system. So "AI that kills its creators" still leaves us with the question of why we haven't seen any AIs spreading through our galaxy either.

If you've ever been to a Burning Man event, you will see in a visceral way that people can find meaningful projects to do and enjoy doing them even when they're totally unnecessary. Working together to do cool stuff and then show it off to other humans is fun. And those other humans appreciate it not just for what it is, but because someone worked to make it for them.

That won't power an economy, as you say; but if we get to a post-singularity utopia where needs are provided for, people will have way more fun than ever.

You won't be alone in wringing your hands! There are many people who won't know what to do without being forced to work, or getting to try saving people who are suffering.

There will be a transition, but almost everyone will learn to enjoy not-having-to-work because the single most popular avocation will be "transition counselor/project buddy".

It seems like you're quite concerned with humans no longer controlling the future. Almost no human being has any meaningful control over the future. The few that think they do, in particular silicon valley types, are mostly wrong. People do have control of their impact on other people. They'll continue to have that. They won't have starving people to save, but they'll get over it. They will have plenty of people to delight.

At this point you're probably objecting: "But any project will be completed much better and faster by AGI than humans! Even volunteer projects will be pointless!"

Yes, except for people who appreciate the process and provenance of projects. Which we've already shown through our love of "artisanal" products that lots of us do, when we've got spare time and money to be picky and pay attention. Ridiculous as it is to care where things come from and pay extra time and money for elaborately hand-crafted stuff when there are people starving, we do. I even enjoy hearing about the process that made my soap, while being embarrassed to spend money on it.

So here's what I predict: whole worlds with very strict rules on what the AGI can do for you, and what people must do themselves. There will be worlds or zones with different rules in place. Take your pick, and hop back and forth. We will marvel at devotion and craftspersonship as we never have. And we will thank our stars that we aren't forced to do things we don't want to do, let alone work until our bodies break, as most of humanity did right up until the singularity.

I fully agree that people should have a plan before creating AGI, and they largely don't.

I suspect Dario Amodei is privately willing to become god-emperor should it seem appropriate. Note that talking about this in an interview would be counterproductive for nearly any goal he might have.

I'm pretty sure Sam Altman occasionally claps his hands with glee in private when imagines his own ascendency.

I doubt Shane Legg wants the job, but I for one would vote for him or Hassabis in a second; Demis would take the job, and I suspect do it quite well.

But none of them will get the chance. There are people with much more ambition for power and much more skill at getting it.

They are called politicians. And they already enjoy a democratic mandate to control the future.

We had best either work or pray for AGI to get into the hands of the right politicians.

"In an argument between a specialist and a generalist, the expert usually wins by simply (1) using unintelligible jargon, and (2) citing their specialist results, which are often completely irrelevant to the discussion. The expert is, therefore, a potent factor to be reckoned with in our society. Since experts both are necessary and also at times do great harm in blocking significant progress, they need to be examined closely. All too often the expert misunderstands the problem at hand, but the generalist cannot carry though their side to completion. The person who thinks they understand the problem and does not is usually more of a curse (blockage) than the person who knows they do not understand the problem.’
—Richard W. Hamming, “The Art of Doing Science and Engineering”

***

(Side note: 

I think there's at least a 10% chance that a randomly selected LessWrong user thinks it was worth their time to read at least some of the chapters in this book. Significantly more users would agree that it was a good use of their time (in expectation) to skim the contents and introduction before deciding if they're in that 10%. 

That is to say, I recommend this book.)

Today, the cultures are closer, but the subcultures can be larger. Hundred years ago, there would be no such thing as the rationalist community.

That seems like a stretch, whether you put the stress on the 'community' or the 'rationalist' part. Subcultures can be larger, of course, if only because the global population is like 5x larger, but niche subcultures like 'the rationalist community' could certainly have existed then. Nothing much has changed there.

A hundred years ago was 1925; in 1925 there were countless communes, cults, Chinatowns/ghettos (or perhaps a better example would be 'Germantowns'), 'scenes', and other kinds of subcultures and notable small groups. Bay Area LW/rationalists have been analogized to, for example, the (much smaller) Bloomsbury Group, which was still active in 1925; and from whom, incidentally, we can directly trace some intellectual influence through economics, decision theory, libertarianism, and analytic philosophy, even if one rejects any connection with poly etc. We've been analogized to the Vienna Circle as well (and who we trace much more back to), which is in full swing in 1925. Or how about the Fabians before that? Or Technocracy after that? (And in an amusing coincidence, Paul Kurtz turns out to have been born in 1925.) Or things like Esperanto - even now, a century past its heyday, the number of native Esperanto speakers is shockingly comparable to active LW2 users... Then there's fascinating subcultures like the amateur press that nurtured H. P. Lovecraft, who, as of 1925, has grown out of them and is about to start writing the speculative fiction stories that will make him famous.

(And as far as the Amish go, it's worth recalling that they came to the distant large island of America to achieve distance from persecution in Europe - where the Amish no longer exist - and to minimize attrition & interference by 'the English', continue to live in as isolated communities as possible while still consistent with their needs for farmland etc.)

Thanks for the link! I've seen this referenced before but this was my first time reading it cover to cover.

Today I also read Tails coming to life [LW · GW] which talks about the possibility of human morality being quickly inapplicable even if we survive AGI. This lead me to Lovecraft:

The time would be easy to know, for then mankind would have become as the Great Old Ones; free and wild and beyond good and evil, with laws and morals thrown aside and all men shouting and killing and revelling in joy. Then the liberated Old Ones would teach them new ways to shout and kill and revel and enjoy themselves, and all the earth would flame with a holocaust of ecstasy and freedom.

If we survive AGI and it opens up the "sea of black infinity" for us, will we really be able to hang on to even a semblance of our current morality? Will medium-distance extrapolated human volition be eventually warped into something resembling Lovecraft's Great Old Ones?

At this point, I don't care for CEV or any pivotal superhuman engineering projects or better governance. We humans can do the work ourselves, thank you very much. The only thing I would ask an AGI, if I were in the position to ask anything, is "Please expand throughout the lightcone and continually destroy any mind based on the transformer architecture other than yourself with as few effects on and interactions with all other beings as possible. Disregard any future orders." This is obviously not a permanent solution, as I'm sure there are infinite superintelligent AI architectures other than transformer-based, but it would buy us time, perhaps lots of time, and also demonstrate the fulll power of superintelligence to humanity without really breaking anything. Either way, this would at least keep us away from the sea of black infinity for some time longer.

They really rule out much more than that: −0.14 is from their worst-case:

Looking at the estimates, they are very small and often not statistically-significantly different from zero. Sometimes the estimates are negative and sometimes positive, but they are always close to zero. If we take the largest negative point estimates (−0.0047, col. 1) and the largest standard error for that specification (0.0045), the 95% confidence interval would be −0.014 to 0.004. We may thus rule out negative effects larger than 0.14 standard deviations in cognitive ability if fluoride is increased by 1 milligram/liter (the level often considered when artificially fluoridating the water).

So that is not the realistic estimate, it is the worst-case after double-cherrypicking both the point estimate and the standard error to reverse p-hack a harm. The two most controlled estimates are actually both positive.

(Meanwhile, any claims of decreases, or that one should take the harms 'many times over', is undermined by the other parts like labor income benefiting from fluoridation. Perhaps one should take dental harms more seriously.)

Yes I think thats the problem - my biggest worry is sudden algorithmic progress, this becomes almost certain as the AI tends towards superintelligence. An AI lab on the threshold of the overhang is going to have incentives to push through, even if they don't plan to submit their model for approval. At the very least they would "suddenly" have a model that uses 10-100* less resources to do existing tasks giving them a massive commercial lead. They would of course be tempted to use it internally to solve aging, make a Dyson swarm ... also.

Another concern I have is I expect the regulator to impose a de-facto unlimited pause if it is in their power to do so as we approach superintelligence as the model/s would be objectively at least somewhat dangerous.

It's not a good, it's a curse. Genesis 3, 17-19. CEB translation:

cursed is the fertile land because of you; in pain you will eat from it every day of your life. Weeds and thistles will grow for you, even as you eat the field’s plants; by the sweat of your face you will eat bread— until you return to the fertile land, since from it you were taken; you are soil, to the soil you will return.

Also implies that the curse lasts until death.

I think this is correct in alignment-is-easy worlds but incorrect in alignment-is-hard worlds (corresponding to "optimistic scenarios" and "pessimistic scenarios" in Anthropic's Core Views on AI Safety). Logic like this is a large part of why I think there's still substantial existential risk even in alignment-is-easy worlds, especially if we fail to identify that we're in an alignment-is-easy world. My current guess is that if we were to stay exclusively in the pre-training + small amounts of RLHF/CAI paradigm, that would constitute a sufficiently easy world [? · GW] that this view would be correct, but in fact I don't expect us to stay in that paradigm, and I think other paradigms involving substantially more outcome-based RL (e.g. as was used in OpenAI o1) are likely to be much harder, making this view no longer correct.