LessWrong 2.0 Reader

Straightforward "strategic ignorance" (avoiding to learn things on purpose to avoid related obligations) seems like an obvious moral failure. The practical problem is that once we start judging people for strategic ignorance, it may motivate them to make their strategy indirect. If you can be blamed for not taking a swimming class that your school provided, it motivates you to choose a school that does not provide swimming classes. Or vote for a government that removes swimming classes from schools, because then it's no longer your fault.

This posits a sort of moral obligation to maximally extend your capacity to help others or take care of yourself in a sustainable way.

Yes. Unfortunately, I have only heard this idea in some form from Eliezer Yudkowsky [? · GW] and Jordan Peterson. It seems to be outside the social Overton window.

I suppose the reason is that, socially, we need a definition of "ethics" such that most people kinda reach it. Otherwise we don't get the peer pressure... and might actually get peer pressure against [LW · GW].

Seems like we have two different things here -- what is the right thing to do, and what is the optimal social norm to promote -- and the relation between them is complicated. It feels it would be nice if these two could be the same thing. Promoting the thing that is the right thing to do, sounds like the right thing to do. But that only works if people already agree, or if there is a cult-like situation that can make them agree (to the degree that they become the enforcers of the norm in private; otherwise you just get two competing moralities). In reality, outside of cults, you don't get an agreement on anything.

Another option is to choose the optimal social norm (the thing that realistically can be approved of by the majority) and pretend that this is the right thing to do. I think that's how it works in practice. The problem is what to do about those parts of "doing the right thing" that don't fit into the "optimal norm that can be socially enforced"? If you openly admit that the social norm is actually not the right thing to do, you undermine the social norm. An alternative is to adopt a (logically inconsistent, if you look too closely) position that something things are "good", but some things are "beyond-good" -- good if you choose to do them, but if you refuse to do them, it doesn't make you bad.

So, using the traditional language, saving the drowning child is an obligation for swimmers; and learning to swim is supererogatory. Until it happens that most of the people in your society learn to swim, and then you can switch and make learning to swim an obligation.

An obvious way to do so is put a hazard sign on "probability" and just not use it, not putting resources into figuring out what "probability" SB should name, isn't it?

It's an obvious thing to do when dealing with simularity clusters poorly defined in natural language. Not so much, when we are talking about a logically pinpointed mathematical concept which we know are crucial for epistemology.

(And now I realize a possible point why you're arguing to keep "probability" term for such scenarios well-defined; so that people in ~anthropic settings can tell you their probability estimates and you, being observer, could update on that information.)

It's not just about anthropic scenarios and not just about me being able to understand other people. It's about general truth preserving mechanism of logical and mathematical reasoning. When people just use different definitions - this is annoying but fine. But when they use different definitions without realizing that these definitions are different and, moreover insist that it's you who is making a mistake - then we have an actual disagreement about math which will provide more confusion along the way. Anthropic scenarious are just the ones where this confusion is noticeable.

As for why I believe probability theory to be useful in life despite the fact that sometimes different tools need to be used

What exactly do you mean by "different tools need to be used"? Can you give me an example?

I like this post but I think redwood has varied some on whether control is for getting alignment work out of AIs vs getting generally good-for-humanity work out of them and pushing for a pause once they reach some usefulness/danger threshold (eg well before super intelligence).

[based on my recollection of Buck seminar in MATS 6]

Bounding the space of controller actions more is the key bit. The (vague) claim is that if you have an argument that an empirically tested automated safety scheme is safe, in sense that you’ll know if the output is correct, you may be able to find a more constrained setup where more of the structure is human-defined and easier to analyze, and that the originally argument may port over to the constrained setup.

I’m not claiming this is always possible, though, just that it’s worth searching for. Currently the situation is that we don’t have well-developed arguments that we can recognize the correctness of automated safety work, so it’s hard to test the “less automation” hypothesis concretely.

I don’t think all automated research is automated safety: certainly you can do automated pure capabilities. But I may have misunderstood that part of the question.

Yes, that was a pretty terrible take. Markets quite clearly do not price externalities well, and never have done. So long as any given investor rates their specific investment as being unlikely to tip the balance into doom, they get the upside of directly financially benefiting from major economic growth due to AI, and essentially the same downside risk as if they didn't invest. Arguments like "short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends" are obviously not even trying to seriously reflect the widespread investment decisions that affect real markets.

I think he's making a bet that, if people could literally get tasty steak without killing cows for the same price (or cheaper), most of them would not pay for factory farmed cow.

alas not easily

Here is the promised comment on what kind of "commitment" I want to make given all the responses. 

I agree with Buck that no one should make very direct commitment about this sort of thing, as there might be blackmail related scary things lurking in the shadows when one does acausal trade. I think we will probably figure out how to handle that, but we shouldn't make any strong promises of specific actions until we figure that out. 

However, the promise I'm intending to keep is that if humanity wins and I'm alive to see it, I will remember how scary it was to live in the world before we knew we'd win, and if there is no currently unknown reason not to do something like the simulation scheme described in the post, and the only cost is not utilizing some small fraction of the conquered Universe, I will advocate for doing it and spend a significant fraction of my resources on it if necessary. I think it would be good for sending future AIs a signal about the seriousness of this, if some other people who are worried about everyone dying from AI, but are not strongly longtermist absolute utilitarians, also left a comment making a similar vague promise that if they ever get to control full star systems, they will contribute to this plan if no currently unknown serious counterargument comes up. Or at least leave an Agreed sing on this sentence.

Other updates from the comments: If I wrote the post now, I would more strongly emphasize a framing that this plan is basically about entering into an acausal insurance contract with other young civilizations who don't yet know what their chances are for aligning AI, neither how big their Universe is compared to the usual Universe-size in the great prior over possible worlds. Then the civilizations who align their AIs and find that they live in a relatively big Universe, bail out everyone else in the insurance contract.

But I think that, while this acausal insurance framing might be more satisfying to the people who are already thinking a lot about acausal trade, in practice the way we implement this "insurance" will likely be very similar to the scheme described in the post. So I maintain that for most people it's better not to think in terms of acausal trade, but just think about the simulation proposal described in the post.

Exactly. You can't make the kid read something, but if he doesn't know the book exists he's not going to read it for sure.

Conversational signposts are just one technique to improve social interactions. For more advanced techniques, I would recommend checking out:

• How to Talk to Anyone: 92 Little Tricks for Big Success in Relationships by Leil Lowndes
• How to Win Friends & Influence People by Dale Carnegie