post by [deleted] · · ? · GW · 0 comments

This is a link post for

0 comments

Comments sorted by top scores.

comment by ike · 2021-09-18T22:57:16.314Z · LW(p) · GW(p)

You're just rejecting one of the premises here, and not coming close to dissolving the strong intuitions / arguments many people have for SIA. If you insist the probability is 50/50 you run into paradoxes anyway (if each agent is offered a 1:3 odds bet, they would reject it if they believe the probability is 50%, but you would want in advance for agents seeing green to take the bet.)

Replies from: Charlie Steiner, rvnnt
comment by Charlie Steiner · 2021-09-23T02:23:10.324Z · LW(p) · GW(p)

You're right, we didn't even get to the part where the proposed game is weird even without the anthropics [LW · GW].

comment by rvnnt · 2021-09-19T06:46:38.342Z · LW(p) · GW(p)

Thanks for the response. I hadn't heard of SIA before. After a bit of searching, I'm guessing you're referring to the Self-Indication Assumption [? · GW].(?)

SIA, intuitions about it:

Looks like there's a lot of stuff to read, under SIA (+ SSA).

My current impression is that SIA is indeed confused (using a confused ontology/Map). But given how little I know of SIA, I'm not super confident in that assessment (maybe I'm just misunderstanding what people mean by SIA).

Maybe if I find the time, I'll read up on SIA, and write a post about why/how I think it's confused. (I'm currently guessing it'd come down to almost the same things I'd write in the long version of this post -- about how people end up with confused intuitions about nonexistent sampling processes inserting nonexistent "I/me" ghosts into some brains but not others.)

If you could share links/pointers to the "strong intuitions / arguments many people have for SIA" you mentioned, I'd be curious to take a look at them.

Bets and paradoxes:

I don't understant what you mean by {running into paradoxes if I insist the probability is 50/50 and each agent is given a 1:3 odds bet}. If we're talking about the bet as described in Eliezer's original post, then the (a priori) expected utility of accepting the bet would be 0.5*(18 - 23) + 0.5(2 - 18*3) = -20, so I would not want to accept that bet, either before or after seeing green, no? I'm guessing you're referring to some different bet. Could you describe in more detail what bet you had in mind, or how a paradox arises?

Replies from: ike
comment by ike · 2021-09-19T13:04:35.349Z · LW(p) · GW(p)

You can start with Bostrom's book on anthropic bias. https://www.anthropic-principle.com/q=book/table_of_contents/

The bet is just each agent is independently offered a 1:3 deal. There's no dependence as in EYs post.

Replies from: rvnnt
comment by rvnnt · 2021-09-20T16:34:39.378Z · LW(p) · GW(p)

It seems to me that, also with the bet you describe, there is no paradox/inconsistency.

To make sure we're talking about the same thing: The bet I'm considering is:

each agent in each room is separately/independently given the option to bet that the coin came up heads. If the agent says "yes, heads", and coin=1, that one agent wins 1 utilon. If the agent says "yes, heads", but coin=0, that one agent loses 3 utilons.

One likely source of confusion that I see here is: If one thinks about {what the agent cares about} in terms of "I", "me", "this agent", or other such concepts which correspond poorly to the Territory (in this kind of dilemma/situation).

To properly deconfuse that, I recommend Tabooing "this", "I", "me", "you", etc. (A trick I found useful for that was to consider variations of the original dilemma, where the agents additionally have numbers tattooed on them; either numbers from 1 to 20, or random UUIDs, or etc.; either visible to the agent or not. Then one can formulate the agent's utility function in terms of "agent with number N tattooed on it", instead of e.g. "instances of me".)

For brevity, below I do use "this" and "I" and etc. Hopefully enough of the idea still comes through to be useful.

If what the agent cares about is something like "utilons gained in total, by computations/agents that are similar to the original agent", then:

  • Before the experiment: The agent would want agents in green rooms to accept the bet, and agents in red rooms to reject the bet.

  • Upon waking up in a green room: The agent has received no information which would allow it to distinguish between coin-flip-outcomes, and its probability for coin=1 is still 50/50. I.e., the agent is in practically the same situation as before the experiment, and so its answer is still the same: accept the bet. (And conversely if in a red room.)

The above seems consistent/paradox-free to me.(?)

If what the agent cares about is something like "utilons gained, by this particular blob of atoms, and the temporal sort-of-continuation of it, as usually understood by e.g. humans", then:

  • Before the experiment: The original description of the dilemma leaves unclear what happens to the original blob of atoms, but here I'll assume that the original blob-of-atoms is destroyed or somehow split 20 ways. In that case, before the experiment, the agent would not care at all how the other copy-agents bet. They're not the same blob-of-atoms, after all. The pre-experiment agent's preferences w.r.t. betting are undefined.

  • Upon waking up in a green room: the agent's probability distribution over coin=1 is still 50/50. And it is a new blob-of-atoms, not the original pre-experiment blob-of-atoms. It doesn't care at all how other blobs-of-atoms fare. It rejects the bet.

To the extent that there's an inconsistency between the pre-experiment agent and in-room agents, I think it's due to them being different agents with different utility functions. So it doesn't seem accurate to say that either agent's individual preferences are inconsistent?

A likely objection I anticipate to the above is something like:

"{The total number of utilons (summed over agents) gained by agents using the above probability-update- and decision-algorithm} is less than {the total number of utilons gained by agents that would update their probability for coin=1 to 90% upon seeing green}."

To which I think Blob-of-atoms #N would respond:

"Yes, but Blob #N does not care that Other-blobs-of-atoms-using-the-same-reasoning-algorithm tend to fare poorly. Blob #N isn't trying to cooperate with near-copies of itself. So what if other blobs-of-atoms don't gain utility? That argument doesn't really weigh against Blob #N's reasoning-algorithm, does it?"

Does this make sense to you? If not, what seems (most) wrong/confused?

Replies from: ike, rvnnt
comment by ike · 2021-09-20T16:52:32.164Z · LW(p) · GW(p)

A couple of things.

If you're ok with time inconsistent probabilities then you can be dutch booked.

I think of identity in terms of expectations. Right before you go to sleep, you have a rational subjective expectation of "waking up" with any number from 1-20 with a 5% probability.

It's not clear how the utility function in your first case says to accept the bet given that you have the probability as 50/50. You can't be maximizing utility, have that probability, and accept the bet - that's just not what maximizes probability under those assumptions.

My version of the bet shouldn't depend on if you care about other agents or not, because the bet doesn't affect other agents.

Replies from: rvnnt
comment by rvnnt · 2021-09-21T07:41:46.218Z · LW(p) · GW(p)

If you're ok with time inconsistent probabilities then you can be dutch booked.

Sure. Has some part of what I've written given the impression that I think time-inconsistent probabilities (or preferences) are OK?

I think of identity in terms of expectations. [...]

I want to give a thumbs-up to the policy of sharing ways-of-thinking-about-stuff. (Albeit that I think I see how that particular way of thinking about this stuff is probably confused. I'm still suggesting Tabooing "I", "me", "you", "[me] waking up in ...", etc.) Thanks.

It's not clear how the utility function in your first case says to accept the bet given that [...]

True, that part of what I wrote glossed over a large bunch of details (which may well be hiding confusion on my part). To try to quickly unpack that a bit:

  • In the given scenario, each agent cares about all similar agents.

  • Pretending to be a Solomonoff inductor, and updating on all available information/observations -- without mapping low-level observations into confused nonsense like "I/me is observing X" -- an agent in a green room ends up with p(coin=1) = 0.5.

  • The agent's model of reality includes a model of {the agent itself, minus the agent's model of itself (to avoid infinite recursion)}.

  • Looking at that model from a bird's-eye-view, the agent searches for an action that would maximize , where is the set of "possible" worlds. (I.e. is the set of worlds that are consistent with what has been observed thus far.) (We're not bothering to weight the summed terms by because here all are equiprobable.)

  • According to the agent's model, all in-room-agents are running the same decision-algorithm, and thus all agents observing the same color output the same decision. This constrains what can contain. In particular, it only contains worlds where if this agent is outputting , then also all other agents (in rooms of the same color) are also outputting .

  • The agent's available actions are "accept bet" and "decline bet". When the agent considers those worlds where it (and thus, all other agents-in-green) outputs "accept bet", it calculates the total utility gained by xeroxed agents to be higher, than in those worlds where it output "decline bet".

  • The agent outputs "accept bet".

If the above is not "maximizing utility", then I'm confused about what (you mean by) "maximizing utility". Did this clarify anything?

My version of the bet shouldn't depend on if you care about other agents or not, because the bet doesn't affect other agents.

It's true that (if the rooms are appropriately sealed off from each other) the blobs-of-atoms in different rooms cannot causally affect each other. But given knowledge that all agents are exact copies of each other, the set of "possible" worlds is constrained to contain only {worlds where all agents (in rooms of the same color) output the same decision}. (I'm thinking very loosely in terms of something like Solomonoff induction here.) Thus it seems to me that {operating/deciding as if agents in other rooms "could" decide something different from each other} is like operating with the wrong set of "possible" worlds; i.e. like doing something wrong relative to Solomonoff induction, and/or having an incorrect model of reality.

Maybe: try Tabooing the word "affect"?

Replies from: ike
comment by ike · 2021-09-21T14:55:07.894Z · LW(p) · GW(p)

I've spent a lot of time and written a handful of posts (including one on the interaction between Solomonoff and SIA) building my ontology. Parts may be mistaken but I don't believe it's "confused". Tabooing core concepts will just make it more tedious to explain, probably with no real benefit.

In particular, the only actual observations anyone has are of the form "I have observed X", and that needs to be the input into Solomonoff. You can't input a bird's eye view because you don't have one.

Anyway, it seems weird that being altruistic affects the agent's decision as to a purely local bet. You end up with the same answer as me on that question, acting "as if" the probability was 90%, but in a convoluted manner.

Maybe you should taboo probability. What does it mean to say that the probability is 50%, if not that you'll accept purely local bets with better odds and not worse odds? The only purpose of probability in my ontology is for predictions for betting purposes (or decision making purposes that map onto that). Maybe it is your notion of probability that is confused.

Replies from: rvnnt
comment by rvnnt · 2021-09-21T16:37:36.969Z · LW(p) · GW(p)

Thanks for the suggestions. Clearly there's still a lot of potentially fruitful disagreement here, some of it possibly mineable for insights; but I'm going to put this stuff on the shelf for now. Anyway, thanks.

comment by rvnnt · 2021-09-21T07:37:41.044Z · LW(p) · GW(p)

Update: Upon considering the situation where

  • each blob-of-atoms cares only about itself (not about similar/copied agents)
  • and the original blob-of-atoms is inserted (e.g. uniformly at random) into one of the rooms (along with 19 copy-agents)

it seems that there is in fact a temporal inconsistency: Before the experiment, original-blob would want all agents (including original-blob) in green rooms to accept the bet, but upon waking up and observing green, original-blob would reject the bet. Will update the post to reflect this.

Replies from: JBlack
comment by JBlack · 2021-09-21T10:57:59.949Z · LW(p) · GW(p)

In general it's not necessary for each blob-of-atoms to care only about itself. It's enough to have any distinction at all in utility of outcomes between itself and other similar blobs-of-atoms. Caring only about itself is just one of the more extreme examples.

comment by JBlack · 2021-09-20T08:37:00.896Z · LW(p) · GW(p)

Probabilities are strongly related to decision theories, and you can't dissolve the question by rejecting them: in these hypothetical scenarios, the agents being discussed still need to make decisions.

So the question remains: what sort of theory leads to making good decisions in such scenarios? Or if you prefer, less wrong ones?

Replies from: rvnnt
comment by rvnnt · 2021-09-20T16:41:15.950Z · LW(p) · GW(p)

I'm not sure what you mean, by me "rejecting decision theories"? Maybe my most recent reply to ike helps clarify what kinds of decisions I think the agents "ought" to take?

Replies from: JBlack, Measure
comment by JBlack · 2021-09-21T10:55:14.664Z · LW(p) · GW(p)

You appeared to be rejecting any meaning for probability of events such as "I am a copy" due to dissolving the distinction between all the copies with indistinguishable observable properties.

The fact remains that the indistinguishable copies still need to make decisions, may have utilities that distinguish the agent making the decision from any others, and their own outcomes may depend upon currently unobservable properties such as "I am a copy". If a decision theory can't assign probabilities to such things, how are they supposed to make decisions?

Declaring that all utilities must be summed or averaged or whatever other symmetric function is insufficient in that it is possible for agents to have a utility function that does not have such a property. The theory fails to cover decisions by such agents.

Replies from: rvnnt
comment by rvnnt · 2021-09-21T16:39:35.127Z · LW(p) · GW(p)

You appeared to be rejecting any meaning for probability of events such as "I am a copy" [...]

If I can translate "I am a copy" to e.g. "an agent that is currently having observations/thoughts x,y,z is a copy" (or to something else that does not depend on (seemingly-to-me ill-defined) things like "me"), then I do think that probabilities can and should be assignable to those kinds of events. I guess my post/writing was even less clear than I thought.

The fact remains that the indistinguishable copies still need to make decisions, may [...]

I think I'm entirely in agreement with that paragraph.

Declaring that all utilities must be summed or averaged or whatever [...]

I don't understand where that paragraph is coming from. (I'm guessing it's coming from my writing being much less clear or much more prone to misinterpretation than I thought. Feel free to not explain where that paragraph came from.)

Replies from: JBlack
comment by JBlack · 2021-09-22T04:52:33.997Z · LW(p) · GW(p)

What do you do when you can't translate "I am a copy" to "an agent with observations X is a copy"? That's the crux of the issue, as I see it. In these problems there are cases where "I" does not just mean "agent with observations X". That's the whole point of them.

Edit: If you want to taboo "I" and "me", you can consider cases where you don't know if other agents are making exactly the same observations (and they probably aren't), but you do know that their observations are the same in all ways relevant to the problem.

In those cases, is probability of such an event meaningful? If not, do you have any replacement theory for making decisions?

Replies from: rvnnt
comment by rvnnt · 2021-09-22T16:14:03.380Z · LW(p) · GW(p)

Ah, the example I gave above was not very good. To clarify:

If I can translate things like "I am a copy" to {propositions defined entirely in terms of non-magical things}, then I think it should be possible to assign probabilities to them.

Like, imagine "possible" worlds are Turing machines, or cellular automata, or some other kind of well defined mathematical object. Then, for any computable function over worlds, I think that

  • it should be possible to assign probabilities to things like , or , or whatever
  • and the above kinds of things are (probably?) the only kinds of things for which probabilities even are "well defined".

(I currently wouldn't be able to give a rigorous definition of what "well defined" means in the above; need to think about that.)

If you can come up with events/propositions that

  • can not (even in principle) be reduced to the form above,
  • but which also would be necessary to assign probabilities to, in order to be able to make decisions,

then I'd be interested to see them!

comment by Measure · 2021-09-21T01:21:34.457Z · LW(p) · GW(p)

The pronoun refers to probabilities, not decision theories. If you say these probabilities are undefined/invalid, then you need to specify what happens when a decision theory tries to run a calculation using those probabilities, and (hopefully) argue why whatever alternative you specify will lead to good outcomes.

Replies from: rvnnt
comment by rvnnt · 2021-09-21T07:47:02.212Z · LW(p) · GW(p)

I'm confused: Where have I said that probabilities are undefined?

I did say that, if the pre-experiment agent-blob-of-atoms cares only about itself, and not the in-room-agents, then its preferences w.r.t. how the in-room-agents bet are undefined. Because its utility function was (by definition) independent of what happens to the in-room-agents. But I don't think I've implied that any probabilities are undefined.

Did this help clarify things?

comment by Charlie Steiner · 2021-09-23T02:21:41.098Z · LW(p) · GW(p)

The issue with this answer is that the paradox works just as well without the anthropic shenanigans!

(See this comment [LW(p) · GW(p)] and this post [LW · GW])

You don't need clones, you could put ordinary people in a room and have them play this game. The paradox comes from a different feature of the game.

On a completely unrelated note, I also disagree about the role of the first-person view in anthropics. Maybe see this post [LW(p) · GW(p)]?