LessWrong 2.0 Reader

Yes it's obviously an alignment technique, given that it is used as one more or less successfully. @RobertM [LW · GW] Could you perhaps explain your reason for reverting?

The list of once “secret” documents is very cool, thanks for that. (But I skimmed the other parts too)

Thanks, Bill! I appreciate the reframe. I agree teaching and learning are two different activities. However, I think the end goal is that the user can learn whatever they need to learn, in whatever way they can learn it. As such, the learner activity is more central than the teaching activity - Having an ideal learning activity will result in the thing we care about (-> learning). Having the ideal teaching experience may still fall flat if the connection with the learner is somehow not made.

I'm curious what benefits you notice from applying the reframe to focusing on the teaching activity first. Possibly more levers to pull on as it's the only side of the equation we can offer someone from the outside?

After reading all the comments threads, I think there's some framing that hasn't been analyzed adequately:

Why would humans be testing AGIs this way if they have the resources to create simulation that will fool a super intelligence?

Also, the risk of humanity being wiped out seems different and worse while that asi is attempting a takeover - during that time the humans are probably an actual threat.

Finally, leaving humans around would seem to pose a nontrivial risk that they'll eventually spawn a new ASI that could threaten the original.

The Dyson sphere is just a tiny part of the universe so using that as the fractional cost seems wrong. Other considerations in both directions would seem to dominate it.

It's the first, there's a lot of uncertainty. I don't think anyone is lying deliberately, although everyone's beliefs tend to follow what they think will produce good outcomes. This is called motivated reasoning.

I don't think this changes the situation much, except to make it harder to coordinate. Rushing full speed ahead while we don't even know the dangers is pretty dumb. But some people really believe the dangers are small so they're going to rush ahead. There aren't strong arguments or a strong consensus for the danger being extremely high, even though looking at opinions of the most thorough thinkers puts risks in the alarmingly high, 50‰ plus range.

Add to this disagreement the fact that most people are neither longtermist nor utilitarian; they'd like a chance to get rich and live forever even if it risks humanity's future.

I explained why I think tracing back personal history is impractical. 

Your separate method to spot check my model is just a simplified version of the same model.

Not only "good ", but "obedient", "non-deceptive", "minimal impact", "behaviorist" and don't even talk about "mindcrime".

In this sense agent foundations research seems similar to research on normative ethics.

It’s certainly possible for simulations to differ from reality, but they seem less useful the more divergent from reality they are.

Depends on what the simulation is being used for, which you also can't deduce from inside of it.

Maybe the simulation could be for pure entertainment (more like a video game), but you should ascribe a relatively low prior to that IMO.

Why? This statement requires some justification.

I'd expect a decent chunk of high fidelity simulations made by humans to be made for entertainment, maybe even absolute majority, if we take into account how we've been using similar technologies so far.

It does if you simultaneously think your creator will eternally reward you for doing so, and/or eternally punish you for failing to.

Not at all. You still have to evaluate this offer using your own mind and values. You can't sidestep this process by simply assuming that Creator's will by definition is the purpose of your life, and therefore you have no choice but to obey.

• Building a superintelligence under current conditions will turn out fine.
• No one will build a superintelligence under anything like current conditions.
• We must prevent at almost all costs anyone building superintelligence soon.

I don't think this is a valid trilemma: Between fine and worth preventing at "almost all costs" there is a pretty large gap. I think "fine" was intended to mean "we don't all die" or something as bad as that.