LessWrong 2.0 Reader

Correlation between outcomes, not within them. If both players prefer to be in the same box, they are aligned. As we add indifference and opposing choices, they become unalienable. In your example, both people have the exact same ordering of outcome. In a classic PD, there is some mix. Totally unaligned (constant value) example: 0/2 2/0 2/0 0/2

I like this answer, and I'm going to take more time to chew on it.

Yes.

I couldn't come up with a direct example that would block that specific property from propagating but I did have a good example of a different property definetely getting blocked.

If smooth physical time doesn't lead into smooth psychological time why would flowing physical time lead into flowing psychological time?

Good question. I don't have a crisp answer (part of why this is an open question), but I'll try a few responses: 

• To what degree does player 1's actions further the interests of player 2 within this normal form game, and vice versa? 
  • This version requires specific response functions.
• To what degree do the interests of players 1 and 2 coincide within a normal form game?
  • This feels more like correlation of the payout functions, represented as vectors.

Right, the word "feasibly" is referring to the bullet point that starts "Maybe “Reward is connected to the abstract concept of ‘I want to be able to sing well’?”". Here's a little toy example we can run with: teaching an AGI "don't kill all humans". So there are three approaches to reward design that I can think of, and none of them seem to offer a feasible way to do this (at least, not with currently-known techniques):

1. The agent learns by experiencing the reward. This doesn't work for "don't kill all humans" because when the reward happens it's too late.
2. The reward calculator is sophisticated enough to understand what the agent is thinking, and issue rewards proportionate to the probability that the current thoughts and plans will eventually lead to the result-in-question happening. So the AGI thinks "hmm, maybe I'll blow up the sun", and the reward calculator recognizes that merely thinking that thought just now incrementally increased the probability that the AGI will kill all humans, and so it issues a negative reward. This is tricky because the reward calculator needs to have an intelligent understanding of the world, and of the AGI's thoughts. So basically the reward calculator is itself an AGI, and now we need to figure out its rewards. I'm personally quite pessimistic about approaches that involve towers-of-AGIs-supervising-other-AGIs, for reasons in section 3.2 here [LW · GW], although other people would disagree with me on that (partly because they are assuming different AGI development paths and architectures than I am [LW · GW]).
3. Same as above, but instead of a separate reward calculator estimating the probability that a thought or plan will lead to the result-in-question, we allow the AGI itself to do that estimation, by flagging a concept in its world-model called "I will kill all humans", and marking it as "very bad and important" somehow. (The inspiration here is a human who somehow winds up with the strong desire "I want to get out of debt". Having assigned value to that abstract concept, the human can assess for themselves the probabilities that different thoughts will increase or decrease the probability of that thing happening, and sorta issue themselves a reward accordingly.) The tricky part is (A) making sure that the AGI does in fact have that concept in its world-model (I think that's a reasonable assumption, at least after some training), (B) finding that concept in the massive complicated opaque world-model, in order to flag it. So this is the symbol-grounding problem I mentioned in the text. I can imagine solving it if we had really good interpretability techniques (techniques that don't currently exist), or maybe there are other methods, but it's an unsolved problem as of now.

the definition of the normal form game you cited explicitly says that the payoffs are in the form of cardinal or ordinal utilities. Which is distinct from in-game payouts.

No. In that article, the only spot where 'utility' appears is identifying utility with the player's payoffs/payouts. (EDIT: but perhaps I don't get what you mean by 'in-game payouts'?)

that player's set of payoffs (normally the set of real numbers, where the number represents a cardinal or ordinal utility—often cardinal in the normal-form representation)

To reiterate: I'm not talking about VNM-utility, derived by taking a preference ordering-over-lotteries and back out a coherent utility function. I'm talking about the players having payoff functions which cardinally represent the value of different outcomes. We can call the value-units "squiggles", or "utilons", or "payouts"; the OP's question remains.

Also, too, it sounds like you agree that the strategy your counterparty uses can make a normal form game not count as a "stag hunt" or "prisoner's dillema" or "dating game"

No, I don't agree with that.

Thanks Josh, your comments have been informative and I'm glad you made them! A major thing that I think this reveals is that I personally am quite risk-averse — I'm willing to pay a premium for maybe-slightly-better perfusion even though that field is so murky, and for life insurance that won't just stop covering me. A maybe-related personality trait is low confidence, so like even if I believe the arguments for short timelines, I don't have enough confidence in that belief to take on (what I perceive to be) the risk of term insurance just based on that.

Also, I and the three other people who I've helped through this process so far could afford to add an extra  ~$150/month expense, so the significantly higher cost wasn't a major deterrent. If I were financially constrained I do expect I would have made different decisions.

-

I think you're right about most insurance companies being compatible with CI, and based on Oge's signup guide [LW · GW], it seems like most can be made compatible with Alcor as well. Looking at it now, I probably should have been clearer about that, but since it wasn't something I'd looked into in any depth, I didn't feel very comfortable writing about it. If you or someone else wanted to write in more detail about how exactly that works I'd be happy to add it to the sequence.

-

I also want to ask if you have any standby arrangements? I think that's a meaningful difference between signing up with CI vs Alcor, because as I said at some point in the sequence, ischemic time matters way more for preservation quality than what perfusion technique is used. (Like, if I lived in Ann Arbor, I would almost certainly sign up with CI no matter what.) Maybe this is just my intense risk-aversion showing again, but it seems to me that cryonics arrangements without standby arrangements might be nearly useless, and that's something I would worry about with CI.

I was thinking of fields. I am doing a pretty bad math sin (atleast for the context) on not being precise and referring to the right target with "distributes over the other". It is too foggy in my mind, I faintly recall something about being able to project + structures into * via something, but is wasn't as straighforward and trivial than I thought.

While there migth have been at various point paid various amounts of attention on which things should be named and which should not be I doubt that it is completely reconstructible from anonymising everything and building it back again. A thing being named is somewhat sticky and will probably push back to outstay its welcome partly because of terminological inertia. A significant reason why we call things we call them is that other know what we are referring to and that it is customary.

Don't think this study is big enough to be representative? Neither do I. 

I created an economics.stackexchange question for whether that finding generalizes. If you are also interested in the question, go and upvote/star it. 

Some particular aspects of existence we're still a bit confused about.