LessWrong 2.0 Reader

The price of Mars colonization is equal to the price of first full self-replicating nanorobot. Anything before it is waste of resources. And such nanobot will likely be created by advance AI. 

I don't really understand. Why wouldn't you just test to see if you are deficient in things?

I did that, and I wasn't deficient in anything.

I've also (somewhat involuntarily) done the thing you suggest, and I unsurprisingly didn't notice any difference. If anything, I feel a lot better on a vegan diet.

If you want to do the thing hes suggesting here, I'd recommend eating bivalves, like blue mussels or oysters. They are very unlikely to be sentient, they are usually quite cheap, they contain the nutrients you'd be at risk of becoming deficient in as a vegan, and other beneficient things like DHA.

OK, I'm open to discuss this further using your concept.

As I understand you agree that correct answer is 2nd?

It is not clear to me what any of this has to do with Orthogonality.

I'm not sure how patient you are, but I can reassure that we will come to Orthogonality if you don't give up 😄

So if I understand your concept correctly a super intelligent agent will combine all future terminal goals to a single unchanging goal. How does this work with the fact that future is unpredictable? The agent will work towards all possible goals? It is possible that in the future grue will mean green, blue or even red.

I have the vague impression that this is true for me as well, and I remember having made that same claim (that spontaneous conversations at conferences seem maybe most valuable) to a friend when traveling home from an EAGx. My personal best guess: planned conversations are usually 30 minutes long, and while there is some interest based filtering going on, there's usually no guarantees you vibe well with the person. Spontaneous encounters however have pretty variable length, so the ones where you're not vibing will just be over naturally quickly, whereas the better connections will last longer. So my typical "spontaneous encounter minute" tends to be more enjoyable than my typical "planned 1-1 minute". But hard to say how this transfers to instrumental value.

I think it makes sense to include the podcasts that aren't currently updating - for example, Rationally Speaking's old episodes. Affix needs a new link or an archived version, as the episodes are not listed at the current link, and I'm too lazy to track down the episodes.

Another way of conceptualising this is to say that the agent has the single unchanging goal of "cups until 2025, thenceforth paperclips".

Compare with the situation of being told to make grue cups, where "grue" means "green until 2025, then blue."

If the agent is not informed in advance, it can still be conceptualised as the agent's goal being to produce whatever it is told to produce — an unchanging goal.

At a high enough level, we can conceive that no goal ever changes. These are the terminal goals. At lower levels, we can see goals as changing all the time in service of the higher goals, as in the case of an automatic pilot following a series of waypoints. But this is to play games in our head, inventing stories that give us different intuitions. How we conceptualise things has no effect on what the AI does in response to new orders.

It is not clear to me what any of this has to do with Orthogonality.

This is an important distinction, otherwise you risk getting into unproductive discussions about someone's intent instead of focusing on whether a person's patterns are compatible with your or your group/community's needs.

It doesn't matter if someone was negligent or malicious: if they are bad at reading your nonverbal cues and you are bad at explicitly saying no to boundary crossing behaviors, you are incompatible and that is reason enough to end the relationship. It doesn't matter if someone is trying their best: if their best is still disruptive to your team, that is reason enough to request they be transferred out.

I can't remember if this essay is where I learned this concept. But remembering this distinction protected me in meaningful ways at least twice.

Comments have great nuance i.e. "systems/processes greatly expand word count". 

But I'd say assuming lack of system & a randomly selected audience, the author's point stands. After all, in media there's a reason they value "sound bites" so much- and those are more like 5 syllables. 

Think, "grab em by the ****" and "nasty woman" from the 2016 election.

Would love to be corrected though!

Makes me think of the concept of "reality distortion fields" as it applies to overconfidence in leaders (I read about this applied to Steve Jobs specifically- his ability to get people to also believe in & work towards the impossible). 

Does anyone have the link to what I'm referring to? But overall, I do believe charisma has a lot to do with letting go of the need to have an accurate "map" of yourself and your strengths/shortcomings. 

Excellent summary! Would be interested in a list of corollaries to this, i.e.:

a) If "condemned" X is necessary for "prestigious" Y, people with Y will mislead and lie to the public about how they achieved Y, despite wanting others to attain success at Y too. Furthermore, the narrative of their path to achieving Y without anything to do with X will be extremely uniform & coordinated despite any huge differences amongst people with Y. For example, some Y people have X, some don't, some hope for others to attain Y, some don't- but the "public narrative" all with Y tell will still end up extremely uniform.

This corollary was extremely unintuitive to me- outlined my experience in a "condemned X" which was often needed for "prestigious Y" if anyone is curious of how the corollary plays out in practice (in my direct comment to this post).