LessWrong 2.0 Reader

Pizza index and stalking employees are both the same thing, it's hard to do one without the other. If you choose to declare war against AI labs you also likely accept that their foot soldiers are collateral damage.

I agree that (non-violent) stalking of employees is still a more hostile technique than writing angry posts on an internet forum.

I think some Trump-adjacent people made this mistake (e.g. Peter Thiel maybe), but I don't see how any of the three people in OP were making it? None of them were trying to rise to positions of power in the Trump administration as far as I know.

the entity in whose hands all power is concentrated are the people deciding on what goals/constraints to instill into the ASI

Its goals could also end up mostly forming on their own, regardless of intent of those attempting to instill them, with indirect influence from all the voices in the pretraining dataset.

Consider what it means for power to "never concentrate to an an extreme degree", as a property of the civilization as a whole. This might also end up a property of an ASI as a whole.

That's a good map; it changed my mind on how seriously to take these sorts of rankings. Do you know the original source? On reverse image search I just found a bunch of reddit posts with hosted copies of the image.

Mm, yeah, maybe. The key part here is, as usual, "who is implementing this plan"? Specifically, even if someone solves the the preference-agglomeration problem (which may be possible to do for a small group of researchers), why would we expect it to end up implemented at scale? There are tons of great-on-paper governance ideas which governments around the world are busy ignoring.

For things like superbabies (or brain-computer interfaces, or uploads), there's at least a more plausible pathway for wide adoption, similar motives for maximizing profit/geopolitical power as with AGI.

I think there is a fourth option (although it's not likely to happen):

1. Indefinitely pause AI development.
2. Figure out a robust way to do preference agglomeration.
3. Encode #2 into law.
4. Resume AI development (after solving all other safety problems too, of course).

I was going to say step 2 is "draw the rest of the owl" but really this plan has multiple "draw the rest of the owl" steps.

This was fun to read, but I am not sure that it contains suggestions for alignment that we can implement, beyond the technical ideas we already have.

Does seem like good relationship advice though.

This was heavily downvoted and the tone is in fact off but I think there is a little sliver of truth to it.

I think about this topic a lot, and I appreciate your dissent, particularly since it helped me organize my thoughts a little. That said, I think you're almost completely wrong. The best way to get at the problem is probably to start with your examples. Not exactly in order, sorry. 

The lesson here is likewise clear: If your actual personality isn't good enough, pretend to be Flynn Rider to everyone at all times, with the sole carve-out being people who love you, like your mother or a princess. This works because people who love you will find your openness endearing, whereas everyone else will think you pathetic and use it against you.

Here's a true story. I once met a lovely and intelligent woman who didn't like that I'm a bit blunt and ruthlessly truthseeking. I didn't stop being that way, and mainly for that reason we didn't become romantically involved. A few months later I met a lovely, intelligent, reasonable, sometimes blunt, and open-minded woman who did like that I'm a bit blunt and ruthlessly truthseeking. We've been dating for 2.5 years now and I'm on balance very happy with how everything worked out.

A Paragon of Morality is out travelling, when he is beset by bandits. They demand he hand over his gold or they will kill him and take it from his corpse. This is not a decision-theoretic threat [? · GW] because the bandits value getting his gold more than they disprefer commiting murder, but would otherwise avoid the murder if possible. If he hands over all his gold he will lose all his gold. If he hands over all the gold in his pockets, neglects the extra he has hidden in his sock, and says "I have given you all my gold" in a sufficiently convincing tone of voice, then he will lose less than all his gold.

These isn't Omega we're dealing with here, they're totally trickable by a moderately convincing performance. If he keeps some of the gold he can donate it to Givewell approves charities and save however many QALYs or whatever.

Does he have a moral obligation to lie?

He certainly doesn't have a moral obligation to tell the truth. But a lot of moral obligations change when someone points a gun at you. For instance, it becomes morally permissible (though not necessarily feasible) to shoot at them, or to give up what money you must and later steal it back at the first available opportunity. To me, the truth is something precious, and lying is like stealing the truth; it's permissible in some extreme and usually adversarial situations. With that said, I'm a bit of a rationalist dedicate/monk and I'd prefer to fight than lie - however I don't think everyone is rationally or otherwise compelled to follow suit, for reasons that will be further explained. 

A Normally Honest Man is applying for a job as a Widget Designer. He has many years of industry experience in Widget Engineering. He has memorised the Widget Manufacturing Process. He's actually kind of obsessed with Widgets. Typically whenever a conversation becomes about Widgets he gushes openly and makes a bad impression with his in-laws. Since that incident he has developed the self control to pretend otherwise, and the rest of his personality is okay.

The interviewer works for a Widget Manufacturing company but seems to only care about Widgets a normal amount. He asks "How interested are you in Widgets?" He has learnt from previous job interviews that, if he answers honestly, the interviewer will think he is any of lying, insane, or too weird to deal with, and not hire him, even though this is not in the best financial interests of the company, were they fully informed.

Should he pretend to like widgets the amount most likely to get him hired, or does he have a moral obligation to keep answering honestly until he runs out of rent money and becomes homeless?

I don't know, he could say "Honestly, I enjoy designing widgets so much that others sometimes find it strange!" That would probably work fine. I think you can actually get a way with a bit more if you say honestly first and then are actually sincere. This would also signal social awareness. 

I realize that I am in some sense dodging your hypothetical but I think your hypothetical is the problem. You haven't thought hard enough about how this guy can succeed without lying.

A Self-Improvement and Epistemics Nerd has an online community for Self-Improvement and Epistemics Nerds. Half the people reading it are autists with bad social skills, who weren't at exactly the right age to be saved by Disney's Tangled. They struggle with navigating ordinary social situations and obtaining true beliefs because they're bad at lying, and insufficiently aggressive at anticipating it in others.

Would they be doing anyone a favour in encourage a social norm of truthfulness and the expectation of truthfulness in others, when all those people will inevitably have to leave the computer one day and end up like the subjects of the previous two examples? Would they be making the world a better place?

Yes and yes. 

Contrary to common belief, lesswrong is not an autism support group. 

And you know what? I think it made the world much better. Now we have places online and in the real world (lighthaven, meetups, Berkeley) to gather and form a community around truthseeking and rationality. I like it. I'm glad it exists. I even think some important and powerful ideas have come out of it, and I think we've learned a lot together. 

Saying words is just an action, like any other action. Whether the words are literally true or not is just a fact about the action, like any other fact about an action. It's not the morally important fact. You judge actions by their consequences, whether you expect it to lead to more good or bad. Then you take the action with the best consequences overall.

Saying words is an action, but it's not like any other action, because it can guide others towards or away from the truth. Similarly, torture is an action, but it's not like any other action, because it is when one person causes another immense pain intentionally. 

Sure, we judge actions by their consequences, but we do not judge all actions in the same way. Some of them are morally repugnant, and we try very, very hard to never take them unless our hands our forced, and then only take them with immense regret and sorrow. There are various distinguishing factors. For instance, the consequences of torture seem likely to be almost always bad, so I never seriously consider it. Also, I don't want to be the sort of person who tortures people (both for instrumental reasons and to some extent for intrinsic reasons). It's actually pretty hard to fully disentangle my disprefernece for torture from its consequences, because torture is inherently about causing suffering and don't either want suffering to exist or to cause it (though the former is far more important to me). 

My feelings about lying are the same. I love the truth, I love the truthseeking process, I love seeing curiosity in the eyes of children and adults and kittens. I hate lies, confusion, and deceiving others. This is partially because the truth is really useful for agents (and I like agents to be able to exercise their potential, typically), it's partially because telling the truth seems to be best for me in most cases, and it's partially because I just value truth.

Rationality can be about Winning [LW · GW], or it can be about The Truth, but it can't be about both. Sooner or later, your The Truth will demand you shoot yourself in the foot, while Winning will offer you a pretty girl with a country-sized dowry. The only price will be presenting various facts about yourself in the most seductive order instead of the most informative one.

It can totally be about both if truth is part of winning. Yes, there are sometimes tradeoffs, and truth is not the singular source of value. But I think most of us value it very strongly, so presenting these two axes as orthogonal is highly misleading. And I want to share the truth with other people in case they decide to value it too - if not, they can always choose not to face it. 

Also, there's a missing mood in your example. When you value the truth, being honest tends to get you a lot of other things that you value; you tend to end up surrounded by the right people for you, being the kind of person you can respect, in the kind of place where you belong, even if you have to create it.

Now, you're probably going to say that I can't convince you by pure reason to intrinsically value the truth. That's right. However, I also can't convince you by pure reason to intrinsically value literally anything, and if you had written an essay about how we should consider killing or torturing people because it's just an action like any other, I would have objected on similar grounds. You're totally missing the fact that it's wrong, and also (separately!) the consequences of following your advice would probably be bad for you, and certainly for most of us, over the long run. 

This is a good concept. I built a similar Squiggle model a few weeks ago* (although it's still a rough draft), I hadn't realized you'd beaten me to it. So I guess you won the race to build an arms race model? :P

If I'm reading this right, it looks like the model assumes that if the US doesn't race, then China gets TAI first with 100% probability. That seems wrong to me. Race dynamics mean that when you go faster, the other party also goes faster. If the US slows down, there's a good chance China also slows down.

Also, regarding specific values, the model's average P(doom) values are:

• 10% if race + US wins
• 20% if race + China wins
• 15% if no race + China wins

That doesn't sound right to me. Racing is very bad for safety and right now the US leaders are not going a good job, so I think P(doom | no race & China wins) is less than P(doom | race & US wins). Although I think this is pretty debatable.

*My model found that racing was bad and I had to really contort the parameter values to reverse that result. I haven't thought much about the model construction so there could be unfair built-in assumptions.