## Posts

## Comments

**benelliott**on Open thread, August 5-11, 2013 · 2013-08-06T21:33:36.668Z · LW · GW

Not quite always

**benelliott**on Gains from trade: Slug versus Galaxy - how much would I give up to control you? · 2013-07-25T22:40:33.185Z · LW · GW

Fair enough

**benelliott**on Gains from trade: Slug versus Galaxy - how much would I give up to control you? · 2013-07-25T13:45:19.938Z · LW · GW

Would it be possible to make those clearer in the post?

I had thought, from the way you phrased it, that the assumption was that for any game, I would be equally likelly to encounter a game with the choices and power levels of the original game reversed. This struck me as plausible, or at least a good point to start from.

What you in fact seem to need, is that I am equally likely to encounter a game with the outcome under this scheme reversed, but the power levels kept the same. This continues to strike me as a very substansive and almost certainly false assertion about the games I am likely to face.

**benelliott**on Gains from trade: Slug versus Galaxy - how much would I give up to control you? · 2013-07-25T03:18:21.228Z · LW · GW

I don't therefore see strong evidence I should reject my informal proof at this point.

I think you and I have very different understandings of the word 'proof'.

**benelliott**on Gains from trade: Slug versus Galaxy - how much would I give up to control you? · 2013-07-24T02:39:52.031Z · LW · GW

In the real world, agent's marginals vary a lot, and the gains from trade are huge, so this isn't likely to come up.

I doubt this claim, particularly the second part.

True, many interactions have gains from trade, but I suspect the weight of these interactions is overstated in most people's minds by the fact that they are the sort of thing that spring to mind when you talk about making deals.

Probably the most common form of interaction I have with people is when we walk past each-other in the street and neither of us hands the other the contents of their wallet. I admit I am using the word 'interaction' quite strangely here, but you have given no reason why this shouldn't count as a game for the purposes of bargaining solutions, we certainly both stand to gain more than the default outcome if we could control the other). My reaction to all but a tiny portion of humanity is to not even think about them, and in a great many cases there is not much to be gained by thinking about them.

I suspect the same is true of marginal preferences, in games with small amounts at stake, preferences should be roughly linear, and where desirable objects are fungible, as they often are, will be very similar accross agents.

In the default, Alice gets nothing. If k is small, she'll likely get a good chunk of the stuff. If k is large, that means that Bob can generate most of the value on his own: Alice isn't contributing much at all, but will still get something if she really cares about it. I don't see this as ultra-unfavourable to Alice!

If k is moderately large, e.g. 1.5 at least, then Alice will probably get less than half of the remaining treasure (i.e. treasure Bob couldn't have acquired on his own) even by her own valuation. Of course the are individual differences, but it seems pretty clear to me that compared to other bargaining solutions, this one is quite strongly biased towards the powerful.

This question isn't precisely answerable without a good prior over games, and any such prior is essentially arbitrary, but I hope I have made it clear that it is at the very least not obvious that there is any degree of symmetry between the powerful and the weak. This renders the x+y > 2h 'proof' in your post bogus, as x and y are normalised differently, so adding them is meaningless.

**benelliott**on Gains from trade: Slug versus Galaxy - how much would I give up to control you? · 2013-07-23T13:49:49.794Z · LW · GW

You're right, I made a false statement because I was in a rush. What I meant to say was that as long as Bob's utility was linear, whatever utility function Alice has there is no way to get all the money.

Are you enforcing that choice? Because it's not a natural one.

It simplifies the scenario, and suggests.

Linear utility is not the most obviously correct utility function: diminishing marginal returns, for instance.

Why is diminishing marginal returns any more obvious that accelerating marginal returns. The former happens to be the human attitude to the thing humans most commonly gamble with (money) but there is no reason to privilege it in general. If Alice and Bob have accelerating returns then in general the money will always be given to Bob, if they have linear returns, it will always be given to Bob, if they have Diminishing returns, it could go either way. This does not seem fair to me.

varying marginal valuations can push the solution in one direction or the other.

This is true, but the default is for them to go to the powerful player.

Look at a moderately more general example, the treasure splitting game. In this version, if Alice and Bob work together, they can get a large treasure haul, consisting of a variety of different desirable objects. We will suppose that if they work separately, Bob is capable of getting a much smaller haul for himself, while Alice can get nothing, mkaing Bob more powerful.

In this game, Alice's value for the whole treasure gets sent to 1, Bob's value for the whole treasure gets sent to a constant more than 1, call it k. For any given object in the treasure, we can work out what proportional of the total value each thinks it is, if Alice's number is at least k times Bob's, then she gets it, otherwise Bob does. This means, if their valuations are identical or even roughly similar, Bob gets everything. There are ways for Alice to get some of it if she values it more, but there are symmetric solutions that favour Bob just as much. The 'central' solution is vastly favourable to Bob.

**benelliott**on Gains from trade: Slug versus Galaxy - how much would I give up to control you? · 2013-07-22T13:02:53.030Z · LW · GW

It does not. See this post ( http://lesswrong.com/lw/i20/even_with_default_points_systems_remain/ ): any player can lie about their utility to force their preferred outcome to be chosen (as long as it's admissible). The weaker player can thus lie to get the maximum possible out of the stronger player. This means that there are weaker players with utility functions that would naturally give them the maximum possible. We can't assume either the weaker player or the stronger one will come out ahead in a trade, without knowing more.

Alice has $1000. Bob has $1100. The only choices available to them are to give some of their money to the other. With linear utility on both sides, the most obvious utility function, Alice gives all her money to Bob. There is no pair of utility functions under which Bob gives all his money to Alice.

**benelliott**on Gains from trade: Slug versus Galaxy - how much would I give up to control you? · 2013-07-22T02:07:28.128Z · LW · GW

If situation A is one where I am more powerful, then I will always face it at high-normalisation, and always face its complement at low normalisation. Since this system generally gives almost everything to the more powerful player, if I make the elementary error of adding the differently normalised utilities I will come up with an overly rosy view of my future prospects.

**benelliott**on Gains from trade: Slug versus Galaxy - how much would I give up to control you? · 2013-07-21T12:46:57.480Z · LW · GW

You x+y > 2h proof is flawed, since my utility may be normalised differently in different scenarios, but this does not mean I will personally weight scenarios where it is normalised to a large number higher than those where it is normalised to a small number. I would give an example if I had more time.

**benelliott**on Harry Potter and the Methods of Rationality discussion thread, part 24, chapter 95 · 2013-07-18T12:11:13.453Z · LW · GW

I didn't interpret the quote as implying that it would actually work, but rather as implying that (the Author thinks) Hanson's 'people don't actually care' arguments are often quite superficial.

**benelliott**on "Stupid" questions thread · 2013-07-14T02:36:11.589Z · LW · GW

consider that "there are no transhumanly intelligent entities in our environment" would likely be a notion that usefully-modelable-as-malevolent transhumanly intelligent entities would promote

Why?

**benelliott**on "Stupid" questions thread · 2013-07-14T02:30:35.884Z · LW · GW

It seems like a mess of tautologies and thought experiments

My own view is that this is precisely correct and exactly why anthropics is interesting, we really should have a good, clear approach to it and the fact we don't suggests there is still work to be done.

**benelliott**on "Stupid" questions thread · 2013-07-14T02:22:54.106Z · LW · GW

I don't know if this is what the poster is thinking of, but one example that came up recently for me is the distinction between risk-aversion and uncertainty-aversion (these may not be the correct terms).

Risk aversion is the what causes me to strongly not want to bet $1000 on a coin flip, even though the expectancy of is zero. I would characterise risk-aversion as an arational preference rather than an irrational bias, primarily becase it arises naturally from having a utility function that is non-linear in wealth ($100 is worth a lot if you're begging on the streets, not so much if you're a billionaire).

However, something like the Allais paradox can be mathematically proven to not arise from any utility function, however non-linear, and therefore is not explainable by risk aversion. Uncertainty aversion is roughly speaking my name for whatever-it-is-that-causes-people-to-choose-irrationally-on-Allais. It seems to work be causing people to strongly prefer certain gains to high probability gains, and much more weakly prefer high-probability gains to low-probability gains.

For the past few weeks I have been in an environment where casual betting for moderate sized amounts ($1-2 on the low end, $100 on the high end) is common, and disentangling risk-aversion from uncertainty aversion in my decision process has been a constant difficulty.

**benelliott**on "Stupid" questions thread · 2013-07-14T01:52:44.783Z · LW · GW

They aren't isomorphic problems, however it is the case that CDT two-boxes and defects while TDT one boxes and co-operates (against some opponents).

**benelliott**on Harry Potter and the Methods of Rationality discussion thread, part 22, chapter 93 · 2013-07-07T02:18:23.670Z · LW · GW

But at some point your character is going to think about something for more than an instant (if they don't then I strongly contest that they are very intelligent). In a best case scenario, it will take you a very long time to write this story, but I think there's some extent to which being more intelligent widens the range of thoughts you can think of ever.

**benelliott**on Harry Potter and the Methods of Rationality discussion thread, part 20, chapter 90 · 2013-07-04T05:57:40.586Z · LW · GW

That's clearly the first level meaning. He's wondering whether there's a second meaning, which is a subtle hint that he has already done exactly that, maybe hoping that Harry will pick up on it and not saying it directly in case Dumbledore or someone else is listening, maybe just a private joke.

**benelliott**on Consequentialism FAQ · 2013-06-18T21:07:26.085Z · LW · GW

I certainly do not define it the second way. Most people care about something other than their own happiness, and some people may care about their own happiness very little, not at all, or negatively, I really don't see why a 'happiness function' would be even slightly interesting to decision theorists.

I think I'd want to define a utility function as "what an agent wants to maximise" but I'm not entirely clear how to unpack the word 'want' in that sentence, I will admit I'm somewhat confused.

However, I'm not particularly concerned about my statements being tautological, they were meant to be, since they are arguing against statements that are tautologically false.

**benelliott**on Consequentialism FAQ · 2013-06-18T15:17:50.653Z · LW · GW

In that case, I would say their true utility function was "follow the deontological rules" or "avoid being smited by divine clippy", and that maximising paperclips is an instrumental subgoal.

In many other cases, I would be happy to say that the person involved was simply not utilitarian, if their actions did not seem to maximise anything at all.

**benelliott**on XKCD - Frequentist vs. Bayesians · 2013-06-16T10:31:58.169Z · LW · GW

(1/36)(1+34p0) is bounded by 1/36, I think a classical statistician would be happy to say that the evidence has a p-value of 1/36 her. Same for any test where H_0 is a composite hypothesis, you just take the supremum.

A bigger problem with your argument is that it is a fully general counter-argument against frequentists ever concluding anything. All data has to be acquired before it can be analysed statistically, all methods of acquiring data have some probability of error (in the real world) and the probability of error is always 'unknowable', at least in the same sense that p0 is in your argument.

You might as well say that a classical statistician would not say the sun had exploded because he would be in a state of total Cartesian doubt about everything.

**benelliott**on Anticipating critical transitions · 2013-06-13T11:20:04.789Z · LW · GW

So, I wrote a similar program to Phil and got similar averages, here's a sample of 5 taken while I write this comment

8.2 6.9 7.7 8.0 7.1

These look pretty similar to the numbers he's getting. Like Phil, I also get occasional results that deviate far from the mean, much more than you'd expect to happen with and approximately normally distributed variable.

I also wrote a program to test your hypothesis about the sequences being too long, running the same number of trials and seeing what the longest string of heads is, the results are

19 22 18 25 23

Do these seem abnormal enough to explain the deviation, or is there a problem with your calculations?

**benelliott**on Terminology suggestion: Say "degrees utility" instead of "utils" to prompt affine thinking · 2013-05-19T14:34:38.226Z · LW · GW

You can double the real numbers representing them, but the results of this won't be preserved under affine transformations. So you can have two people whose utility functions are the same, tell them both "double your utility assigned to X" and get different results.

**benelliott**on Morality should be Moral · 2013-05-18T09:57:59.159Z · LW · GW

A green sky will be green

This is true

A pink invisible unicorn is pink

This is a meaningless sequence of squiggles on my computer screen, not a tautology

A moral system would be moral

I'm unsure what this one means

**benelliott**on The Unselfish Trolley Problem · 2013-05-17T17:59:43.187Z · LW · GW

I'm not sure what 'should' means if it doesn't somehow cash out as preference.

**benelliott**on The Unselfish Trolley Problem · 2013-05-17T17:56:36.818Z · LW · GW

I could not abide someone doing that to me or a loved one, throwing us from relative safety into absolute disaster. So I would not do it to another. It is not my sacrifice to make.

I could not abide myself or a loved one being killed on the track. What makes their lives so much less important.

**benelliott**on Pascal's Muggle: Infinitesimal Priors and Strong Evidence · 2013-05-09T23:16:30.151Z · LW · GW

How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?

First thought, I'm not at all sure that it does. Pascal's mugging may still be a problem. This doesn't seem to contradict what I said about the leverage penalty being the only correct approach, rather than a 'fix' of some kind, in the first case. Worryingly, if you are correct it may also not be a 'fix' in the sense of not actually fixing anything.

I notice I'm currently confused about whether the 'causal nodes' patch is justified by the same argument. I will think about it and hopefully find an answer.

**benelliott**on Pascal's Muggle: Infinitesimal Priors and Strong Evidence · 2013-05-09T21:13:25.884Z · LW · GW

Random thoughts here, not highly confident in their correctness.

Why is the leverage penalty seen as something that needs to be added, isn't it just the obviously correct way to do probability.

Suppose I want to calculate the probability that a race of aliens will descend from the skies and randomly declare me Overlord of Earth some time in the next year. To do this, I naturally go to Delphi to talk to the Oracle of Perfect Priors, and she tells me that the chance of aliens descending from the skies and declaring an Overlord of Earth in the next year is 0.0000007%.

If I then declare this to be my probability of become Overlord of Earth in an alien-backed coup, this is obviously wrong. Clearly I should multiply it by the probability that the aliens pick me, given that the aliens are doing this. There are about 7-billion people on earth, and updating on the existence of Overlord Declaring aliens doesn't have much effect on that estimate, so my probability of being picked is about 1 in 7 billion, meaning my probability of being overlorded is about 0.0000000000000001%. Taking the former estimate rather than the latter is simply wrong.

Pascal's mugging is a similar situation, only this time when we update on the mugger telling the truth, we radically change our estimate of the number of people who were 'in the lottery', all the way up to 3^^^^3. We then multiply 1/3^^^^3 by the probability that we live in a universe where Pascal's muggings occur (which should be very small but not super-exponentially small). This gives you the leverage penalty straight away, no need to think about Tegmark multiverses. We were simply mistaken to not include it in the first place.

**benelliott**on Pascal's Muggle: Infinitesimal Priors and Strong Evidence · 2013-05-07T15:42:26.637Z · LW · GW

I would agree that it is to some extent political. I don't think its very dark artsy though, because it seems to be a case of getting rid of an anti-FAI misunderstanding rather than creating a pro-FAI misunderstanding.

**benelliott**on Pascal's Muggle: Infinitesimal Priors and Strong Evidence · 2013-05-07T09:00:25.005Z · LW · GW

But yeah, "diyer" is too close to "die" to be easily distinguishable. Maybe "rubemond"?

I could see the argument for that, provided we also had saphmonds, emmonds etc... Otherwise you run the risk of claiming a special connection that doesn't exist.

**benelliott**on Pascal's Muggle: Infinitesimal Priors and Strong Evidence · 2013-05-07T00:10:39.241Z · LW · GW

Chemistry would not be improved by providing completely different names to chlorate and perchlorate (e.g. chlorate and sneblobs).

Okay, thats actually a good example. This caused me to re-think my position. After thinking, I'm still not sure that the analogy is actually valid though.

In chemistry, we have a systemic naming scheme. Systematic name schemes are good, because they let us guess word meanings without having to learn them. In a difficult field which most people learn only as adults if at all, this is a very good thing. I'm no chemist, but if I had to guess the words chlorate and perchlorate to cause confusion sometimes, but that this price is overall worth paying for a systemic naming scheme.

For gemstones, we do not currently have a systematic naming scheme. I'm not entirely sure that bringing one in would be good, there aren't all that many common gemstones that we're likely to forget them and frankly if it ain't broke don't fix it, but I'm not sure it would be bad either.

What would not be good would be to simply rename rubies to diyermands without changing anything else. This would not only result in misunderstandings, but generate the false impression that rubies and diamonds have something special in common as distinct from Sapphires and Emeralds (I apologise for my ignorance if this is in fact the case).

But at least in the case of gemstones we do not *already* have a serious problem, I do not know of any major epistemic failures floating around to do with the diamond-ruby distinction.

In the case of Pascal's mugging, we have a complete epistemic disaster, a very specific very useful term have been turned into a useless bloated red-giant word, laden with piles of negative connotations and no actual meaning beyond 'offer of lots of utility that I need an excuse to ignore'.

I know of almost nobody who has serious problems noticing the similarities between these situations, but tons of people seem not to realise there are any differences. The priority with terminology must be to separate the meanings and make it absolutely clear that these are not the same thing and need not be treated in the same way. Giving them similar names is nearly the worst thing that could be done, second only to leaving the situation as it is.

If you were to propose a systematic terminology for decision-theoretric dilemmas, that would be a different matter. I think I would disagree with you, the field is young and we don't have a good enough picture of the space of possible problems, a systemic scheme risks reducing our ability to think beyond it.

But that is not what is being suggested, what is being suggested is creating an ad-hoc confusion generator by making deliberately similar terms for different situations.

This might all be rationalisation, but thats my best guess for why the situations feel different to me.

**benelliott**on Pascal's Muggle: Infinitesimal Priors and Strong Evidence · 2013-05-06T23:48:56.301Z · LW · GW

Do you really think this!? I admit to being extremely surprised to find anyone saying this.

If rubies were called diyermands it seems to me that people wouldn't guess what it was when they heard it, they would simply guess that they had misheard 'diamond', especially since it would almost certainly be a context where that was plausible, most people would probably still have to have the word explained to them.

Furthermore, once we had the definition, we would be endlessly mixing them up, given that they come up in exactly the same context. Words are used many times, but only need to be learned once, so getting the former unambiguous is far more important.

The word 'ruby' exists *primarily* to distinguish them from things like diamonds, you can usually guess that they're not cows from context. Replacing it with diyermand causes it to fail at its main purpose.

EDIT:

To give an example from my own field, in maths we have the terms 'compact' and 'sequentially compact' for types of topological space. The meanings are similar but not the same, you can find spaces satisfying one but not the other, but most 'nice' spaces have both or neither.

If your theory is correct, this situation is good, because it will allow people to form a plausible guess at what 'compact' means if they already know 'sequentially compact' (this is almost always they order a student meets them). Indeed, they do always form a plausible guess, and that guess is 'the two terms mean the same thing'. This guess seems so plausible, they never question it and go off believing the wrong thing. In my case this lasted about 6 months before someone undeluded me, even when I learned the real definition of compactness, I assumed they were provably equivalent.

Had their names been totally different, I would have actually asked what it meant when I first heard it, and would never have had any misunderstandings, and several others I know would have avoided them as well. This seems unambiguously better.

**benelliott**on Pascal's Muggle: Infinitesimal Priors and Strong Evidence · 2013-05-06T14:49:15.552Z · LW · GW

Even if not, they should at least be called something that acknowledges the similarity, like "Pascal-like muggings".

Any similarities are arguments for giving them a maximally *different* name to avoid confusion, not a similar one. Would the English language really be better if rubies were called diyermands?

**benelliott**on Pascal's Muggle: Infinitesimal Priors and Strong Evidence · 2013-05-06T14:37:10.427Z · LW · GW

Why on earth would you expect the downstream utilities to exactly cancel the mugging utility?

**benelliott**on What do professional philosophers believe, and why? · 2013-05-02T22:23:42.182Z · LW · GW

The first is contradictory, you've just told me something, then told me I don't know it, which is obviously false.

**benelliott**on Logic in the language of probability · 2013-04-27T10:38:10.602Z · LW · GW

Sure this is right? After all, the implication is also true in the case of A being false, the conjuntion certainly is not.

He specifically specifies that A is true as well as A => B

Intuitively I suggest there should be an inequality, too, seeing as B|A is not necessarily independent of A.

B|A is not an event, so it makes no sense to talk about whether or not it is independent of A.

To see why this is a valid theorem, break it up into three posibilities, P(A & B) = x, P(A & ~B) = y, P(~A) = 1 - x - y.

Then P(A) = P(A & B) + P(A & ~B) = x + y

For P(B|A), restrict to the space where A is true, this has size x + y of which B takes up x, so P(B|A) = x / (x+y)

Thus P(A)P(B|A) = x = P(A & B)

**benelliott**on Bitcoins are not digital greenbacks · 2013-04-19T23:20:17.181Z · LW · GW

I don't know if this is typical, but I recently a professional trader stated in an email to me that he knew very little about Bitcoin and basically had no idea what to think of it. This may hint that the lack of interest isn't based on certainty that bitcoin will flop, but simply on not knowing how to treat it and sticking to markets where they do have reasonably well-understood ways of making a profit, since exposure to risk is a limited resource.

**benelliott**on Reflection in Probabilistic Logic · 2013-03-26T23:57:07.807Z · LW · GW

I fully agree that is an interesting avenue of discussion, but it doesn't look much like what the paper is offering us.

**benelliott**on Reflection in Probabilistic Logic · 2013-03-26T13:14:50.511Z · LW · GW

Maybe I'm misunderstanding here, but it seems like we have no particular reason to suppose P=NP is independent of ZFC. Unless it is independent, its probability under this scheme must already be 1 or 0, and the only way to find out which is to prove or disprove it.

**benelliott**on Reflection in Probabilistic Logic · 2013-03-25T23:22:24.529Z · LW · GW

In ZF set theory, consider the following three statements.

I) The axiom of choice is false

II) The axiom of choice is true and the continuum hypothesis is false

III) The axiom of choice is true and the continuum hypothesis is true

None of these is provably true or false so they all get assigned probability 0.5 under your scheme. This is a blatant absurdity as they are mutually exclusive so their probabilities cannot possibly sum to more than 1

**benelliott**on You only need faith in two things · 2013-03-13T20:06:39.093Z · LW · GW

So induction gives the right answer 100s of times, and then gets it wrong once. Doesn't seem too bad a ratio.

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-03T20:19:28.527Z · LW · GW

I am indeed suggesting that an agent can assign utility, not merely expected utility, to a lottery.

I am suggesting that this is equivalent to suggesting that two points can be parallel. It may be true for your special definition of point, but its not true for mine, and its not true for the definition the theorems refer to.

Yes, in the real world the lottery is part of the outcome, but that can be factored in with assigning utility to the outcomes, we don't need to change our definition of utility when the existing one works (reading the rest of your post, I now see you already understand this).

It sounds to me like you favour a direct approach. For you, utility is not an as-if: it is a fundamentally real, interval-scale-able quality of our lives.

I cannot see anything I have said to suggest I believe this. Interpreted descriptively, (as a statement about how people actually make decisions) I think it is utter garbage.

Interpreted prescriptively, I think I might believe it. I would at least probably say what while I like the fact that VNM axioms imply EU theory, I think I would consider EU the obviously correct way to do things even if they did not.

In this scheme, the angst I feel while taking a risk is something I can assign a utility to, then shut up and (re-)calculate the expected utilities.

Yes.

Granted, if decision angst is often playing a large part in your decisions, and in particular costing you other benefits, I would strongly suggest you work on finding ways to get around this. Rightly or wrongly, yelling "stop being so irrational!" at my brain has sometimes worked here for me. I am almost certain there are better techniques.

I wonder why you even care to defend the VNM axioms, or what role they play for you.

I defend them because I think they are correct. What more reason should be required?

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-03T14:07:01.932Z · LW · GW

I'm not sure quite what the best response to this is, but I think I wasn't understanding you up to this point. We seem to have a bit of a level mixing problem.

In VNM utility theory, we assign utility to outcomes, defined as a complete description of what happens, and expected utility to lotteries, defined as a probability distribution over outcomes. They are measured in the same units, but they are not the same thing and should not be compared directly.

VNM utility tells you nothing about how to calculate utility and everything about how to calculate expected utility given utility.

By my definitions of risk aversion, type (II) risk aversion is simply a statement about how you assign utility, while type (III) is an error in calculating expected utility.

Type (I), as best I understand it, seems to consist of assigning utility to a lottery. Its not so much an axiom violation as a category error, a bit like (to go back my geometry analogy) asking if two points are parallel to each other. It doesn't violate independence, because its wrong on far too basic a level to even assess whether it violates independence.

Of course, this is made more complicated by f***ing human brains, as usual. The knowledge of having taken a risk affects our brains and may change our satisfaction with the outcome. My response to this is that it can be factored back into the utility calculation, at which point you find that getting one outcome in one lottery is not the same as getting it in another.

I may ask that you go read my conversation with kilobug elsewhere in this thread, as I think it comes down to the exact same response and I don't feel like typing it all again.

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-02T22:20:35.257Z · LW · GW

it remains to show that someone with that preference pattern (and not pattern III) still must have a VNM utility function

Why does it remain to be shown? How does this differ from the claim that any other preference pattern that does not violate a VNM axiom is modelled by expected utility?

Now consider the games involving chance that people enjoy. These either show (subjective probability interpretation of "risk") or provide suggestive evidence toward the possibility (epistemic probability interpretation) that some people just plain like risk.

Interesting. If I had to guess though, the way in which these people like risk depends on the way it is dispensed, and is probably not linear in the amount of risk.

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-02T21:15:43.667Z · LW · GW

So, when people say 'risk aversion', they can mean one of three different things:

I) I have a utility function that penalises world-histories in which I take risks.

II) I have a utility function which offers diminishing returns in some resource, so I am risk averse in that resource

III) I am risk averse in utility

Out of the three (III) is irrational and violates VNM. (II) is not irrational, and is an extremely common preference among humans wrt some things, but not others (money vs lives being the classic one). (I) is not irrational, but is pretty weird, I'm really not sure I have preferences like this, and when other people claim they do I become a bit suspicious that it is actually a case of (II) or (III).

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-02T20:46:38.380Z · LW · GW

I think we have almost reached agreement, just a few more nitpicks I seem to have with your current post.

the independence principle doesn't strictly hold in the real world, like there are no strictly right angle in the real world

Its pedantic, but these two statements aren't analogous. A better analogy would be

"the independence principle doesn't strictly hold in the real world, like the axiom that all right angles are equal doesn't hold in the real world"

"there are no strictly identical outcomes in the real world, like there are no strictly right angle in the real world"

Personally I prefer the second phrasing. The independence principle and the right angle principle do hold in the real world, or at least they would if the objects they talked about ever actually appeared, which they don't.

I'm in general uncomfortable with talk of the empirical status of mathematical statements, maybe this makes me a Platonist or something. I'm much happier with talk of whether idealised mathematical objects exist in the real world, or whether things similar to them do.

What this means is we don't apply VNM when we think independence is relatively true, we apply them when we think the outcomes we are facing are relatively similar to each other, enough that any difference can be assumed away.

But do we have any way to measure the degree of error introduced by this approximation?

This is an interesting problem. As far as I can tell, its a special case of the interesting problem of "how do we know/decide our utility function?".

Do we have ways to recognize the cases where we shouldn't apply the expected utility theory

I've suggested one heuristic that I think is quite good. Any ideas for others?

(Once again, I want to nitpick the language. "Do we have ways to recognize the cases where two outcomes look equal but aren't" is the correct phrasing.

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-02T20:22:35.989Z · LW · GW

To me it's a single, atomic real-world choice you have to make:

To you it may be this, but the fact that this leads to an obvious absurdity suggests that this is not how most proponents think of it, or how its inventors thought of it.

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-02T20:00:47.492Z · LW · GW

Given that people can rationally have preferences that make essential reference to history and to the way events came about, why can't risk be one of those historical factors that matter? What's so "irrational" about that?

Nothing. Whoever said there was?

If your goal is to not be a thief, then expected utility theory recommends that you do not steal.

I suspect most of us do have 'do not steal' preferences on the scale of a few hundred pounds or more.

On the other hand, once you get to, say, a few hundred human lives, or the fate of the entire species, then I stop caring about the journey as much. It still matters, but the amount that it matters is too small to ever have an appreciable effect on the decision. This preference may be unique to me, but if so then I weep for humanity.

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-02T19:52:28.011Z · LW · GW

First, I did study mathematical logic, and please avoid such kind of ad hominem.

Fair enough

That said, if what you're referring to is the whole world state, the outcomes are, in fact, always different. Even if only because there is somewhere in your brain the knowledge that the choice is different.

I thought this would be your reply, but didn't want to address it because the comment was too long already.

Firstly, this is completely correct. (Well, technically we could imagine situations where the outcomes removed your memory of there ever having been a choice, but this isn't usually the case). Its pretty much never possible to make actually useful deductions just from pure logic and the axiom of independence.

This is much the same as any other time you apply a mathematical model to the real world. We assume away some factors, not because we don't think they exist, but because we think they do not have a large effect on the outcome or that the effect they do have does not actually affect our decision in any way.

E.g. Geometry is completely useless, because perfectly straight lines do not exist in the real world. However, in many situations they are incredibly good approximations which let us draw interesting non-trivial conclusions. This doesn't mean Euclidean Geometry is an approximation, the approximation is when I claim the edge of my desk is a straight line.

So, I would say that usually, my memory of the other choice I was offered has quite small effects on my satisfaction with the outcome compared to what I actually get, so in most circumstances I can safely assume that the outcomes are equal (even though they aren't). With that assumption, independence generates some interesting conclusions.

Other times, this assumption breaks down. Your cholera example strikes me as a little silly, but the example in your original post is an excellent illustration of how assuming two outcomes are equal because they look the same as English sentences can be a mistake.

At a guess, a good heuristic seems to be that after you've made your decision, and found out which outcome from the lottery you got, then *usually* the approximation that the existence of other outcomes changes nothing is correct. If there's a long time gap *between* the decision and the lottery then decisions made in that time gap should usually be taken into account.

Of course, independence isn't really that useful for its own sake, but more for the fact that combined with other axioms it gives you expected utility theory.

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-02T18:17:12.050Z · LW · GW

Gwern said pretty much everything I wanted to say to this, but there's an extra distinction I want to make

What you're doing is saying you can't use A, B, and C when there is dependency, but have to create subevents like C1="C when you are you sure you'll have either A or C".

The distinction I made was things like A2="A when you prepare" not A2="A when you are sure of getting A or C". This looks like a nitpick, but is in fact incredibly important. The difference between my A1 and A2, is important, they are fundamentally different outcomes which may have completely different utilities, they have no more in common than B1 and C2. They are events in their own right, there is no 'sub-' to it. Distinguishing between them is not 'mangling', putting them together in the first place was always an error.

you in fact have shown the axioms don't work in the general case

It is easily possible to imagine three tennis players A, B and C, such that A beats B, B beats C and C beats A (perhaps A has a rather odd technique, which C has worked hard at learning to deal with despite being otherwise mediocre). Then we have A > B and B > C but not A > C, I have just shown that the axiom of transitivity is not true in the general case!

Well, no, I haven't.

I've shown that the axiom of transitivity does not hold for tennis players. This may be an interesting fact about tennis, but it has not 'disproven' anything, nobody ever claimed that transitivity applied to tennis players.

What the VNM axioms are meant to refer to, are outcomes, meaning a complete description of what will happen to you. "Trip to Ecuador" is not an outcome, because it does not describe exactly what will happen to you, and in particular leaves open whether or not you will prepare for the trip.

This sort of thing is why I think everyone with the intellectual capacity to do so should study mathematical logic. It really helps you learn to keep things cleanly separated in your mind and avoid mistakes like this.

**benelliott**on The VNM independence axiom ignores the value of information · 2013-03-02T16:45:12.357Z · LW · GW

The problem here is that you've not specified the options in enough detail, for instance you appear to prefer going to Ecaudor with preparation time to going without preparation time, but you haven't stated this anywhere. You haven't given the slightest hint whether you prefer Iceland with preparation time to Ecuador without. VNM is not magic, if you put garbage in you get garbage out.

So to really describe the problem we need six options:

A1 - trip to Ecuador, no advance preparation A2 - trip to Ecuador, advance preparation B1 - laptop B2 - laptop, but you waste time and money preparing for a non-existant trip. C1 - trip to Iceland, no advance preparation C2 - trip to Iceland, advance preparation

Presumably you have preferences A2 > A1, B1 > B2, C2 > C1. You have also stated A > B > C, but its not clear how to interpret this, A2 > B1 > C2 seems the most charitable. You seem to also think C2 > B2, but you haven't said so so maybe I'm wrong.

You have four possible choices, D1 = (A1 or B1), D2 = (A2 or B2), E1 = (A1 or C1) and E2 = (A2 or C2)

The VNM axioms can tell us that E2 > E1, this also seems intuitively right. If we also accept C2 > B2 then they can tell you that E2 > D2. They don't tell us anything about how to judge between D2 and E1, since the decision here depends on the size rather than ordering of your preferences. None of this seems remotely counter-intuitive.

In short, 'value of information' isn't some extra factor that needs to be taken into account on top of decision theory. It can be factored in *within* decision theory by correctly specifying your possible options.

Furthermore, information isn't binary, it doesn't suddenly appear once you have certainty and not before, if you take into account the existence of probabilistic partial information then you should find the exact same results pop out.

The same goes for the Allais paradox: having certitude of receiving a significant amount of money ($24 000) has a value, which is present in choice 1A, but not in all others (1B, 2A, 2B).

Why does it have value? The period where you have certainty in 1A but not in the other 3 probably only lasts a few seconds, and there aren't any other decisions you have to make during it.

**benelliott**on Newcomb's Problem dissolved? · 2013-02-25T20:03:25.596Z · LW · GW

What makes you think you have a reliable way of fooling Omega?

In particular, I am extremely sceptical that simply not making your mind up, and then at the last minute doing something that feels random, would actually correspond to making use of quantum nondeterminism. In particular, if individual neurons are reasonably deterministic, then regardless of quantum physics any human's actions can be predicted pretty perfectly, at least on a 5/10 minute scale.

Alternatively, even if it is possible to be delibrately non-cooperative, the problem can just be changed so that if Omega notices you are deliberately making its judgement hard, then it just doesn't fill the box. The problem in this version seems exactly as hard as Newcomb's.