LessWrong 2.0 Reader

True, but Buck's claim is still relevant as a counterargument to my claim about memetic fitness of the scheming story relative to all these other stories.

Yes. That’s really my central claim.

OK, I read you and essentially agree with you.

Two caveats that, which I expect you've already noticed yourself:

1. There are going to be conflicts over human values in the non-AGI, non-ASI world too. Delaying AI may prevent them from getting even worse, but there's still blood flowing over these conflicts without any AI at all. Which is both a limitation of the approach and perhaps a cost in itself.

2. More generally, if you think your values are going to largely win, you have to trade off caution, consideration for other people's values, and things like that, against the cost of that win being delayed.^[1]

I think a lot of people have that. There’s a even meme for that "It ain’t much, but it’s honest work".

All in one, I don’t think either of us has much more evidence that a vague sense of things anyway ? I sure don’t have.

So far as I know, there are no statistics. My only guess is that you're likely talking about a "lot" of people on each side (if you had to reduce it to two sides, which is of course probably oversimplifying beyond the bounds of reason).

[...] "my agency is meaningful if and only if I have to take positive, considered action to ensure my survival, or at least a major chunk of my happiness".

I think that’s the general direction of the thing we’re trying to point, yes ?

I'll take your word for it that it's important to you, and I know that other people have said it's important to them. Being hung up on that seems deeply weird to me for a bunch of reasons that I could name that you might not care to hear about, and probably another bunch of reasons I haven't consciously recognized (at least yet).

If you give me the choice of living the life of a medieval farmer or someone who has nothing in his life but playing chess, I will take the former.

OK, here's one for you. An ASI has taken over the world. It's running some system that more or less matches your view of a "meaningless UBI paradise". It send one of its bodies/avatars/consciousness nodes over to your house, and it says:

"I/we notice that you sincerely think your life is meaningless. Sign here, and I/we will set you up as a medieval farmer. You'll get land in a community of other people who've chosen to be medieval farmers (you'll still be able to lose that land under the rules of the locally prevailing medieval system). You'll have to work hard and get things right (and not be too unlucky), or you'll starve. I/we will protect your medieval enclave from outside incursion, but other than that you'll get no help. Obviously this will have no effect on how I/we run the rest of the world. If you take this deal, you can't revoke it, so the stakes will be real."^[2]

Would you take that?

The core of the offer is that the ASI is willing to refrain from rescuing you from the results of certain failures, if you really want that. Suppose the ASI is willing to edit the details to your taste, so long as it doesn't unduly interfere with the ASI's ability to offer other people different deals (so you don't get to demand "direct human control over the light cone" or the like). Is there any variant that you'd be satisfied with?

Or does having to choose it spoil it? Or is it too specific to that particular part of the elephant?

Does "growing as a person" sounds like a terminal goal to you ?

Yes, actually. One of the very top ones.

Is "real stakes" easier to grasp than Agency/Meaningfulness ? Or have I just moved confusion around ?

It's clear and graspable.

I don't agree with it, but it helps with the definition problem, at least as far as you personally are concerned. At least it resolves enough of the definition problem to move things along, since you say that the "elephant" has other parts. Now I can at least talk about "this trunk you showed me and whatever's attached to it in some way yet to be defined".

Well, the problem is that there is so much concepts, especially when you want to be precise, and so few words.

Maybe it's just an "elephant" thing, but I still get the feeling that a lot of it is a "different people use these words with fundamentally different meanings" thing.

Individually for a particular manifestation of each issue this is true, you can imagine doing a hacky solution to each one. But that assumes there is a list of such particular problems that if you check off all the boxes you win, rather than them being manifestations of broader problems. You do not want to get into a hacking contest if you're not confident your list is complete.

I want to be a little careful here, i'm not saying that this or that thing is "Right" or "Wrong" that's what morality does, I'm trying to describe what "Morality" is. So yes, I suppose a slave would get a lower moral weight than a doctor, shall we say 0.8 of your average society member for the slave and 1.2 for the doctor? This is certainly what we observe in history, where skilled helpful professionals are more valued than the less skilled and not very willing.

A slave's willingness is a lot more important a factor in their utility than that of a shrimp. I would give a shrimp a moral weight of 0.0.

In the American context slavery is also wrapped up with racism, which I think is wrong from both my personal morality and also from my half-baked "recognition of usefulness helps everyone get along and makes for greater prosperity" standard. 

I think that modern wage / economic slavery (doing a job) is much more efficient / effective, in part because the human is recognised and applauded for their usefulness and works much harder because of it.

Calling eligible bachelors in SF: “AI philosopher with a perchant for underwater sci fi and evening bike rides seeks a direct communicator who cares about the world and feels a thrill of human triumph at the sight of a cargo ship.” (@AmandaAskell)

Now there's a temptation.... I am otherwise occupied however.

I watched Fall Guy last month because I noticed it had 5 stars on Zvi's letterboxd. Can confirm it's an amazing movie.

The idea that the labor share of income has been historically stable is often taken as a given, but it’s more of a “stylized fact” than an unchanging law. Measuring non-wage compensation, like healthcare and retirement benefits, is notoriously tricky, which makes it hard to get a clear picture. And if capital—like machines and software—ever becomes a perfect substitute for human labor, that historical stability could completely fall apart. 

On top of that, even though GDP and worker productivity have risen over the decades, many workers, especially those in low and middle-income jobs, haven’t seen their wages keep up. Since the 80s, a bigger share of the productivity gains has gone to capital owners, executives, and highly skilled workers, leaving others behind and deepening income inequality. Declining union membership, globalization, and shifts in labor markets have further eroded workers’ bargaining power, lowering their share of income—even in industries where technology is designed to work alongside people.

Great stuff! I don't have strong fundamentals in math and statistics but I was still able to hobble along and understand the post. It reminds me of what Rissanen said about data/observation - that data is really all we have, and there is no true state of nature. Our job is to squeeze as much alpha out of observation as possible, instead of trying to find a "true" generator function. This post hit the same spot for me :)

I did wonder about maintenance costs, but I figured they wouldn't change the picture too much because I only assume an avg 3 year lifetime for the robot, and figured they wouldn't need a huge amount of maintenance to make it to that point.

Moreover, if there's worthwhile maintenance that extends the lifetime further, then the hardware costs could end up cheaper than my per year estimate. 

I'm also envisioning the costs after a big scale up, and there would be robot repair shops as numerous as car repair, rather than needing to fly in specialists.

That said, I agree it would be interesting to look at how much is spent on car maintenance per year on a car vs. capital costs. (I expect it would be under 10%?)

I'd be happy to put the opening bunch of paragraphs. I was feeling reluctant to cross-post because I often update my articles as I learn more about a topic, and I don't want to keep multiple versions in sync (especially for a lower priority article).