The other side of the tidal wave

I'm middle-aged now, and a pattern I've noticed as I get older is that I keep having to adapt my sense of what is valuable, because desirable things that used to be scarce for me keep becoming abundant. Some of this is just growing up, e.g. when I was a kid my candy consumption was regulated by my parents, but then I had to learn to regulate it myself. I think humans are pretty well-adapted to that sort of value drift over the life course. But then there's the value drift due to rapid technological change, which I think is more disorienting. E.g. I invested a lot of my youth into learning to use software which is now obsolete. It feels like my youthful enthusiasm for learning new software skills, and comparative lack thereof as I get older, was an adaptation to a world where valuable skills learned in childhood could be expected to mostly remain valuable throughout life. It felt like a bit of a rug-pull how much that turned out not to be the case w.r.t. software.

But the rise of generative AI has really accelerated this trend, and I'm starting to feel adrift and rudderless. One of the biggest changes from scarcity to abundance in my life was that of interesting information, enabled by the internet. I adapted to it by re-centering my values around learning skills and creating things. As I contemplate what AI can already do, and extrapolate that into the near future, I can feel my motivation to learn and create flagging.

If, and to the extent that, we get a "good" singularity, I expect that it will have been because the alignment problem turned out to be not that hard, the sort of thing we could muddle through improvisationally. But that sort of singularity seems unlikely to preserve something as delicately balanced as the way that (relatively well-off) humans get a sense of meaning and purpose from the scarcity of desirable things. I would still choose a world that is essentially a grand theme park full of delightful experience machines over the world as it is now, with all its sorrows, and certainly I would choose theme-park world over extinction. But still ... OP beautifully crystalizes the apprehension I feel about even the more optimistic end of the spectrum of possible futures for humanity that are coming into view.

Thank you for saying this. I have tried several times to explain something like it in a post, but I don't think I have the writing skill to convey effectively how deeply distressed I am about these scenarios. It's essential to my ability to enjoy life that I be useful, have political capital, can effect meaningful change throughout the world, can compete in status games with others, can participate in an economy of other people like me, and can have natural and productive relationships with unartificial people. I don't understand at all how I'm supposed to be excited by the "good OpenAI ending" where every facet of human skill and interaction gets slowly commoditized, and that ending seems strictly worse to me in a lot of ways than just dying suddenly in an exploding ball of fire.

Rah to bringing back the short LessWrong post!

I don't currently share this sense of distress.

Insofar as we don't all die and we broadly continue to have agency over the world, I am kind of excited and up for the challenge of the new age. Given no-extinction and no-death and lots of other improvements to wealth and health and power, I'm up for the challenges and pains and difficulties that come with it.

I am further reminded of this quote [LW(p) · GW(p)].

Personally, I've been hearing all my life about the Serious Philosophical Issues posed by life extension, and my attitude has always been that I'm willing to grapple with those issues for as many centuries as it takes.

— Patrick Nielsen Hayden

I feel pretty wary of the alignment community becoming temperamentally pessimistic, in a way that colors our expectations and judgments. I note this post as a fairly central example of that. (May say more about this later, but just wanted to register disagreement.)

i've written before [LW · GW] about how aligned-AI utopia can very much conserve much of what we value now, including doing effort to achieve things that are meaningful to ourselves or other real humans. on top of alleviating all (unconsented) suffering and (unconsented) scarcty and all the other "basics", of course.

and without aligned-AI utopia we pretty much die-for-sure. there aren't really attractors in-between those two.

No offense but I sense status quo bias in this post.

If you replace "AI" with "industrial revolution" I don't think the meaning of the text changes much and I expect most people would rather live today than in the Middle Ages.

One thing that might be concerning is that older generations (us in the future) might not have the ability to adapt to a drastically different world in the same way that some old people today struggle to use the internet.

I personally don't expect to be overly nostalgic in the future because I'm not that impressed by the current state of the world: factory farming, the hedonic treadmill, physical and mental illness, wage slavery, aging, and ignorance are all problems that I hope are solved in the future.

Yeah :/ I've struggled for a long time to see how the world could be good with strong AI, and I've felt pretty alienated in that. Most of the time when I talk to people about it they're like "well the world could just be however you like!" Almost as if, definitionally, I should be happy because in the really strong success cases we'll have the tech to satisfy basically any preference. But that's almost the entire problem, in some way? As you say, figuring things out for ourselves, thinking and learning and taking pride in skills that take effort to acquire... most of what I cherish about these things has to do with grappling with new territory. And if I know that it is not in fact new, if all of it could be easier were I to use the technology right there... it feels as though something is corrupted... The beauty of curiosity, wonder, and discovery feels deeply bound to the unknown, to me. 

I was talking to a friend about this a few months ago and he suggested that because many humans have these preferences, that we ought to be able to make a world where we satisfy them—e.g., something like "the AI does its thing over there and we sit over here having basically normal human lives except that death is a choice and sometimes it helps us figure out hard coordination problems or whatever." And I can almost get behind this, but something still feels off to me. Like how when people get polarized through social media it almost seems like there's no going back? How do we know strange spirals won't happen with an even more advanced technology? It's hard to escape the feeling that a dystopia lurks. Hard to escape the feeling that all the people I know and love might change quickly and radically, that I might change radically, in ways that feel alien to me now. I want to believe that strong AI would be great, and perhaps it would be, perhaps I'm missing something here. But a part of me is terrified.  

Ah, if things go well, it will be an amazing opportunity to find out how much of our minds was ultimately motivated by fear. Suppose that you are effectively immortal and other people can't hurt you, what would you do? Would you still want to learn? Would you bother keeping friends? Or would you maybe just simulate million kinds of experience, and then get bored and decide to die or wirehead yourself?

I think I want to know the answer. If it kills me, so be it... the universe didn't have a better plan for me anyway.

It would probably be nicer to take things slowly. Stop death and pain, and then let people slowly figure out everything. That would keep a lot of life normal. The question is whether we could coordinate on that, because it would be tempting to cheat. If we all voluntarily slow down and try to do "life as normal, but without pain", a little bit of cheating ("hey AI, give me 20 extra IQ points and all university-level knowledge as of 2023, but don't tell anyone; otherwise give me the life as normal but without pain") would keep lot of the benefits of life as normal, but also give one a relative advantage and higher status. It's not even necessary to give me all the knowledge, just make me run 10 times faster when no one is looking, and I will study it myself.

Maybe humanity will split into different bubbles, depending on how fast they want to take it, with AI keeping the boundaries between them, so that the slower ones are protected from interfering of the faster ones.

Probably the difficult choice will be whether to keep our relative disadvantages, especially if they can be fixed just by asking the AI nicely. We could probably agree on "AI, don't tell us the Theory of Everything, we want to figure it out ourselves, and now we have enough time to do so", or at least, to make the rule that anyone who wants to hear the answer from the AI is allowed to, but then is prevented from giving spoilers to others. But it would feel unfair e.g. to require people with low IQ to stay that way. However, the more differences we remove, the less sense remains for the division of labor, which seems like an important part of our relationships. (Not just literally "labor", but even things like "this person typically makes the jokes, because they are better at making jokes; and this person typically says something empathic; and this person typically makes the decision when the others hesitate...".)

I think this might be mapping a regular getting older thing to catastrophe? Part of aging is the signaling landscape that one trained on changing enough that many of the things one once valued don't seem broadly valued any more.

This is more or less why I chose to go into genetics instead of AI: I simply couldn't think of any realistic positive future with AGI in it. All positive scenarios rely on a benevolent dictator, or some kind of stable equilibrium with multiple superintelligent agents whose desires are so alien to mine, and whose actions are so unpredictable that I can't evaluate such an outcome with my current level of intelligence.

I've actually been moving in the opposite direction, thinking that the gameboard might not be flipped over, and actually life will stay mostly the same. Political movements to block superintelligence seem to be gaining steam, and people are taking it seriously.

(Even for more mundane AI, I think it's fairly likely that we'll be soon moving "backwards" on that as well, for various reasons which I'll be writing posts about in the coming week or two if all goes well.)

Also, some social groups will inevitably internally "ban" certain technologies if things get weird. There's too much that people like about the current world, to allow that to be tossed away in favor of such uncertainty.

I am not a fan of the current state of the universe. Mostly the part where people keep dying and hurting all the time. Humans I know, humans I don't know, other animals that might or might not have qualia, possibly aliens in distant places and Everett branches. It's all quite the mood killer for me, to put it mildly. 

So if we pull off not dying and turning Earth into the nucleus of an expanding zero utility stable state, superhuman AI seems great to me.

Did you miss transhumanism? If it's truly important to you, to be useful, alignment would mean that superintelligence will find a way to lift you up and give you a role.

I suppose there might be a period during which we've figured out existential security but the FASI hasn't figured out human augmentation beyond the high priority stuff like curing aging. I wouldn't expect that period to be long.

I can only say there was probably someone in every rolling 100 year period that thought the same about the next 100 years

Aligned AGI means people getting more of what they want. We'll figure out where to get challenge and fulfillment.

People will have more friends, not less, once we have more time for them, and more wisdom about how to get and keep them. And we will still want human support and advice, even if machines can do it better.

People who want to learn skills and knowledge will still do that, and create spaces to compete and show them off. I'm thinking of the many competitions that already happen with rules about not getting outside help.

Most humans have always lived their lives knowing they'll never be the best at anything or accomplish anything significant on a large scale. Many of them still find happiness in small projects and personal connections.

In the long term, if alignment is right, we'll have something as marvelous as hinted in Richard Ngo's The Soul Key [LW · GW].

What do you make of the prospect of neurotech, e.g. Neuralink, Kernel, Openwater, Meta/CTRL-Labs, facilitating some kind of merge of biological human intelligence and artificial intelligence? If AI alignment is solved and AGI is safely deployed, then "friendly" or well-aligned AGI could radically accelerate neurotech. This sounds like it might obviate the sort of obsolescence of human intelligence you seem to be worried about, allowing humans alive in a post-AGI world to become transhuman or post-human cyborg entities that can possibly "compete" with AGI in domains like writing, explanation, friendship, etc. 

I do not see why any of these things will be devalued in a world with superhuman AI.

At most of the things I do, there are many other humans who are vastly better at doing the same thing than me. For some intellectual activities, there are machines who are vastly better than any human. Neither of these stops humans from enjoying improving their own skills and showing them off to other humans.

For instance, I like to play chess. I consider myself a good player, and yet a grandmaster would beat me 90-95 percent of the time. They, in turn, would lose on average 8.5-1.5 in a ten game match against a world-championship level player. And a world champion will lose almost all of their games against Stockfish running on a smartphone. Stockfish running on a smartphone, in turn, will lose most of its games against Stockfish running on a powerful desktop computer or against Leela Chess Zero running on something that has a decent GPU. I think those opponents would probably, in turn, lose almost all of their games against an adversary that has infinite retries, i.e. that can target and exploit weaknesses perfectly. That is how far I am away from playing chess perfectly.

And yet, the emergence of narrow superintelligence in chess has increased and not diminished my enjoyment of the game. It is nice to be able to play normally against a human, and to then be able to find out the truth about the game by interactively looking at candidate moves and lines that could have been played using Leela. It is nice to see a commented world championship game, try to understand the comments, make up one's own mind about them, and then explore using an engine why the alternatives that one comes up with (mostly) don't work.

If we get superintelligence, that same accessibility of tutoring at beyond the level of any human expert will be available in all intellectual fields. I think in the winning scenario, this will make people enjoy a wide range of human activities more, not less.

I kind of understand where that sentiment comes from but I do think it is "wrong". Wrong in the sense that it is neither a necessary position to hold nor a healthy one. There are plenty of things I do today in which I get a lot of satisfaction even though existing machines, or just other people, can do them much better than I can. The satisfaction comes from the challenge to my own ability level rather than some comparison to something outside me -- be it machine, environment or another person.

To me it sounds like you're dividing possible futures into extinction, dystopia, and utopia, and noticing that you can't really envision the latter. In which case, I agree, and I think if any of us could, we'd be a lot closer to solving alignment than we actually are. 

Where my intuition cuts differently is that I think most seemingly-dystopian futures, where humans exist but are disempowered and dissatisfied with our lives and the world, are unstable or at best metastable, and will eventually give way to one of the other two categories. I'm sure stable dystopias are possible, of course, but ending up in one seems like it would require getting almost all of the steps of alignment right, but failing the last step of the grail quest. 

Yes, this means I think most of the non-extinction futures you're considering are really extinction-but-with-a-longer-delay-and-lots-of-suffering futures. But I also think there's a sizable set of utopian-futures-outside-my-capacity-to-concretely-envision such that my P(doom) isn't actually close to 100%.

I predict most humans choose to reside in virtual worlds and possibly have their brain altered to forget that it's not real. 

Short, as near as I can tell, true, and important. This expresses much of my feeling about the world.

You can just create personalized environments to your preferences. Assuming that you have power/money in the post-singularity world.

It really is. My conception of the future is so weighed by the very likely reality of an AI transformed world that I have basically abandoned any plans with a time scale over 5 years. Even my short term plans will likely be shifted significantly by any AI advances over the next few months/years. It really is crazy to think about, but I've gone over every single aspect of AI advances and scaling thousands of times in my head and can think of no reality in the near future not as alien to our current reality as ours is to pre-eukaryotic life.

My taxonomy of possible outcomes is x-risk (risk of extinction), s-risk (risk of suffering), w-risk (risk of a "weirdtopia"), and success. It seems like what you are worried about is a mix of s-risk and w-risk, maybe along lines that no-one has clearly conceptualized yet?

Superhumwn chess AI did not remove people's pleasure from learning/playing chess. I think people are adaptible and can find meaning. Surely, the world will not feel the same but I think there is significant potential for something much better. I wrote about tfhis a little on my blog:

https://martinkunev.wordpress.com/2024/05/04/living-with-ai/

If superhuman AI would prevent you from thinking, learning, or being proud of yourself; that seems to me like the result of some sort of severe psychological issue. I'm sorry that you have that going on, but... maybe get help?

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

I keep wondering if there is an afterlife, and if there is will they be able to break a twenty?

I sort of agree. But there is clearly a potential flip-side: we quite likely get to be post-scarcity on a lot of things (modulo fairness of distribution and absence of tyranny), including: customized art and stories, scientific and mathematical discoveries, medical care, plus any sort of economic goods that depend more on knowledge and technological inputs than material resources. So the video games will be awesome. We might even be somewhat less constrained on material resources, if there are rapid technological improvements in reusable energy, environmental remediation after mining, asteroid mining, or things like that.

This makes an interesting point about scarcity. On one hand, it sucks to be limited in the amount of stuff you have. On the other hand, struggling through adversity or having scarce skills can give people a sense of meaning. We know that people whose skills are automated can suffer a lot from it. 

I think that even once all of humans' cognitive skills can be replaced by AI, we will still be useful to one another. We will still relate to each other on account of our shared biological nature. I think that people will not for the most part have the same emotional relation with silicon-based intelligence as we have with other homo sapiens, because it will not share our emotions and flaws. I think a big part of why we like interacting with other people is because we can see ourselves in them, which we will not be able to do with AI. This is why I support developing AI capabilities, as long as we can keep control. 

Thanks for writing this. I fully agree, by the way.

Anything like an utopia requires a form of stability. But we're going to speed everything up, by a lot. Nothing can change and yet remain the same. And I think it's silly to assume that optimization, or simply just improving things, can co-exist with things remaining as they are. We're necessarily something intermediate, so speeding things up doesn't seem like a good idea.

Furthermore, it seems that slowing down technological advancement is almost impossible, and that keeping people from making optimal choices for themselves is almost impossible (even if they know it's harmful to the whole). And there's two further dangers here, the first is technology completely removing our freedoms and privacy, and the second is technology being used to manipulate human behaviour, erasing our humanity. And this second point is almost guanteed, as human well-being is not possible in a world withour freedom, agency and privacy, these are core needs. So why not effectively turn the population into drones by modifying them? We're already doing this with adderall and SSRIs, and since it makes people suffer less, most people suport it.

That first possibility, technology taking away our agency, freedoms, and privacy, is basically guaranteed as well. It has actually already happened, it will just get much worse. This is because technology makes it easier to cause damages, and while it also makes it easier to defend and prevent damages, there's an inequality between the two growth rates.

All the increases in mental health issues is a direct consequence of the modern life, so it's also extremely naive to assume that more technology and more "progress" is going to fix it. Why eliminate struggle? Even struggle is essential to life and well-being! Are intellectuals really so disconnected from reality that they don't realize this?

In my head, I've sort of just been simplifying to two ways the future could go: human extinction within a relatively short time period after powerful AI is developed or a pretty good utopian world. The non-extinction outcomes are not ones I worry about at the moment, though I'm very curious about how things will play out. I'm very excited about the future conditional on us figuring out how to align AI. 

I'm curious about, for people who think similarly to Katja, what kind of story are you imagining that leads to that? Does the story involve authoritarianism (but I think even then, the world in which the leader of any of the current leading labs has total control and a superintelligent AI that does whatever they want, that future is probably much much more fun and exciting for me than the present - and I like my present life!)? Does it involve us being only presented with pretty meh options for how to build the future because we can't agree on something that wholly satisfies everyone? Does it involve multi-agent scenarios with the AIs or the humans controlling the AIs being bad at bargaining so we end up with meh futures that no one really wants? I find a bunch of stories pretty unlikely after I think about them but maybe I'm missing something important. 

This is also something I'd be excited to have a Dialogue with someone about. Maybe just fleshing out what kind of future you're imagining and how you're imagining we end up in that situation. 

writing, explaining things to people, friendships where you can be of any use to one another, taking pride in skills, thinking, learning, figuring out how to achieve things, making things, easy tracking of what is and isn’t conscious

Used to be, we enjoyed doing those things ourselves through a special uniquely-human flavor of being clever.

Seems like post-AI, we'll get to those things through a special uniquely-human flavor of being un-clever.

It doesn't make sense to judge human accomplishment and effort differently from that of AI -- but humans are great at choosing to do things which don't make sense when it suits us to. Living a generally enjoyable life despite not being particularly good at things compared to the other entities around us is a skill rather anthithetical to this community's values, but it does exist in plenty of cultures of the human population. Call it instrumental irrationality, perhaps.

Those of us who've pinned our self-worth on being better at computers than the computers are, though, are in for a bad time of a magnitude that may warrant replacing that entire part of the worldview to reach happiness.

The Sea of Faith

Was once, too, at the full, and round earth’s shore

Lay like the folds of a bright girdle furled.

But now I only hear

Its melancholy, long, withdrawing roar,

Retreating, to the breath

Of the night-wind, down the vast edges drear

And naked shingles of the world.
 

Ah, love, let us be true

To one another! for the world, which seems

To lie before us like a land of dreams,

So various, so beautiful, so new,

Hath really neither joy, nor love, nor light,

Nor certitude, nor peace, nor help for pain;

And we are here as on a darkling plain

Swept with confused alarms of struggle and flight,

Where ignorant armies clash by night.

Even if we don’t die, it still feels like everything is coming to an end.

Everything? I imagine there will be communities/nations/social groups that completely ban AI and those that are highly dependent on AI. There must be something between those two extremes.

All things arise and all things pass away.