What if a tech company forced you to move to NYC?

Yet I would bet that even that person, if faced instead with a policy that was going to forcibly relocate them to New York City, would be quite indignant

A big difference is that assuming you're talking about futures in which AI hasn't catastrophic outcomes, no one will be forcibly mandated to do anything. 

Another important point is that, sure, people won't need to do work, which means they will be unnecessary to the economy, barring some pretty sharp human enhancement. But this downside, along with all the other downsides, looks extremely small compared to the non-AGI default of dying of aging and having a 1/3 chance of getting dementia, 40% chance of getting cancer, your loved ones dying, etc.

Not sure I love this analogy — moving to NYC doesn't seem like that big of a deal —, but I do think it's pretty messed up to be imposing huge social / technological / societal changes on 8 billion of your peers. I expect most of the people building AGI have not really grasped the ethical magnitude of doing this — I think I sort of have, but also I don't build AGI. 

I'm not as chill as all that, and I absolutely appreciate people worrying about those dimensions.  But I do tend to act in day-to-day behavior (and believe, in the average sense (my probablistic belief range incudes a lot of scenarios, but the average and median are somewhat close together, which is probably a sign of improper heuristics)) as if it'll all be mostly-normal.  I recently turned down a very good job offer in NYC (and happily, later found a better one in Seattle), but I see the analogy, and kind of agree it's a good one, but from the other side - even people who think they'd hate NYC are probably wrong - hedonic adaptation is amazingly strong. I'll try to represent those you're frustrated with.

There will absolutely be changes, many of which will be uncomfortable, and probably regress from my peak-preference.  As long as it's not extinction or effective-extinction (a few humans kept in zoos or the like, but economically unimportant to the actual intelligent agents shaping the future), it'll be ... OK.  Not necessarily great compared to imaginary utopias, but far better than the worst outcomes.   Almost certainly better than any ancient person could have expected.  

I think I understand what you mean.

There are definitely possible futures worse than extinction. And some fairly likely ones that might not be worse than extinction but would still suck big time. Varying from comparable to a forced move to a damnsight worse than moving to anywhere that presently exists. I'm old enough to have already had some disappointments (alongside some positive surprises) about how the "future" has turned out. I could easily see how I could get a lot worse ones.

But what are we meant to do with what you've posted and how you've framed it?

Also, if somebody does have the "non-extinction => good" mindset, I suspect they'll be prone to read your post as saying that change in itself is unacceptable, or at least that any change that every single person doesn't agree to is unacceptable. Which is kind of a useless position since, yeah, there will always be change, and things not changing will also always make some people unhappy.

I've gotta say that, even though I definitely worry about non-extinction dystopias, and think that they are, in the aggregate, more probable than extinction scenarios... your use of the word "meaning" really triggered me. That truly is a word people use really incoherently.

Maybe some more concrete concerns?

From Cognitive Biases Potentially Affecting Judgment of Global Risks:

In addition to standard biases, I have personally observed what look like harmful modes of thinking specific to existential risks. The Spanish flu of 1918 killed 25-50 million people. World War II killed 60 million people. 108 is the order of the largest catastrophes in humanity’s written history. Substantially larger numbers, such as 500 million deaths, and especially qualitatively different scenarios such as the extinction of the entire human species, seem to trigger a different mode of thinking—enter into a “separate magisterium.” People who would never dream of hurting a child hear of an existential risk, and say, “Well, maybe the human species doesn’t really deserve to survive.”

There is a saying in heuristics and biases that people do not evaluate events, but descriptions of events—what is called non-extensional reasoning. The extension of humanity’s extinction includes the death of yourself, of your friends, of your family, of your loved ones, of your city, of your country, of your political fellows. Yet people who would take great offense at a proposal to wipe the country of Britain from the map, to kill every member of the Democratic Party in the U.S., to turn the city of Paris to glass—who would feel still greater horror on hearing the doctor say that their child had cancer— these people will discuss the extinction of humanity with perfect calm. “Extinction of humanity,” as words on paper, appears in fictional novels, or is discussed in philosophy books—it belongs to a different context than the Spanish flu. We evaluate descriptions of events, not extensions of events. The cliché phrase end of the world invokes the magisterium of myth and dream, of prophecy and apocalypse, of novels and movies. The challenge of existential risks to rationality is that, the catastrophes being so huge, people snap into a different mode of thinking. Human deaths are suddenly no longer bad, and detailed predictions suddenly no longer require any expertise, and whether the story is told with a happy ending or a sad ending is a matter of personal taste in stories.

My experience has also been this being very true.

Weird coincidence: I was just thinking about Leopold's bunker concept from his essay. It was a pretty careless paper overall but the imperative to put alignment research in a bunker makes perfect sense; I don't see the surface as a viable place for people to get serious work done (at least, not in densely populated urban areas; somewhere in the desert would count as a "bunker" in this case so long as it's sufficiently distant from passerbys and the sensors and compute in their phones and cars).

Of course, this is unambiguously a necessary evil that a tiny handful of people are going to have to choose to live in a sad uncomfortable place for a while, and only because there's no other option and it's obviously the correct move for everyone everywhere including the people in the bunker.

Until the basics of the situation start somehow getting taught in the classrooms or something, we're going to be stuck with a ludicrously large proportion of people satisfied with the kind of bite-sized convenient takes that got us into this whole unhinged situation in the first place (or have no thoughts at all [LW · GW]).

If a tech company forced me to move to NYC, I would object for a combination of two separate reasons: 1--any change in my life is going to be hard--it may take me away from people I know, I need to learn the geography again, I live in Lothlórien right now and if I move to NYC nobody speaks Quenya, etc. And 2--things that are specific about NYC above and beyond the fact that change is going to be a problem by itself; for instance, I might hate subways, and I might hate subways whether I'm exposed to lots of them or not.

#2 can be a personal problem for me, but I notice that people in NYC aren't, on the average, in general less happy than people who live elsewhere, so it seems like #2 isn't a real issue when averaged over the whole population. #1 can be an issue even averaged over the whole population, of course, but #1 isn't unique to moving to NYC, and applies to a whole bunch of other changes to the point where it's most of the way to being a fully general argument against any change.

I'd expect the same to be true in the case of AI: The "change is a problem" component is negative, but it's no worse than any other sort of change, and the "AI specifically is a problem" component would include some people who are harmed and some people who benefit and overall it's going to be a wash.

Or to put it another way, just because I wouldn't want a tech company to move me to NYC, that doesn't imply that NYC is a worse place to live than where I am now.

Each time you can also apply this argument in reverse: I don't like X about my city, so I'm happy that in the figure, the company will relocate me to NYC. And since NYC is presumed to be overall better, there are more instances of the latter rather than the former.

It seems to me you are taking the argument seriously, but very selectively.

(I think both kinds of thoughts pretty often, and I'm overall happy about the incoming move).

I'd give my right eye in exchange for the chance to live in NYC.

NYC is a great city. Many tech workers I know are trying to move to NYC, especially younger ones. So, not the best example, but I get your point. 

This seems like a rather silly argument. You can apply it to pretty much any global change, any technological progress. The world changes, and will change. You can be salty about it, or you can adapt.