manfred feed - LessWrong 2.0 Readermanfred’s posts and comments on the Effective Altruism Forumen-usComment by Manfred on Leaving beta: Voting on moving to LessWrong.com
https://lw2.issarice.com/posts/PXMMtQqHSTfNvoWGk/leaving-beta-voting-on-moving-to-lesswrong-com#pc8kYFLamkBbQxtMY
<p>Ditto.</p>
manfredpc8kYFLamkBbQxtMY2018-03-13T02:45:54.518ZComment by Manfred on Teaching rationality in a lyceum
https://lw2.issarice.com/posts/hMNtjFs2JHE9zn5h8/teaching-rationality-in-a-lyceum#oqMBGNWhKFiGivj2E
<p>This site isn't too active - maybe email someone from CFAR directly?</p>
manfredoqMBGNWhKFiGivj2E2017-12-07T17:12:01.137ZComment by Manfred on Letter from Utopia: Talking to Nick Bostrom
https://lw2.issarice.com/posts/btqu7a2SQdJu2Bs5J/letter-from-utopia-talking-to-nick-bostrom#aKW29mNx7j9zMhTuJ
<p>Man, this interviewer sure likes to ask dense questions. Bostrom sort of responded to them, but things would have gone a lot smoother if LARB guy (okay, Andy Fitch) had limited himself to one or two questions at a time. Still, it's kind of shocking the extent to which Andy "got it," given that he doesn't seem to be specially selected - instead he's a regular LARB contributor and professor in an MFA program.</p>
manfredaKW29mNx7j9zMhTuJ2017-11-27T18:53:06.453ZComment by Manfred on Kialo -- an online discussion platform that attempts to support reasonable debates
https://lw2.issarice.com/posts/g3odvaj8opqCF9egv/kialo-an-online-discussion-platform-that-attempts-to-support#siEquqAkFDPePrnob
<p>Hm, the format is interesting. The end product is, ideally, a tree of arguments, with each argument having an attached relevance rating from the audience. I like that they didn't try to use the pro and con arguments to influence the rating of the parent argument, because that would be too reflective of audience composition.</p>
manfredsiEquqAkFDPePrnob2017-11-05T19:46:32.935ZComment by Manfred on Simple refutation of the ‘Bayesian’ philosophy of science
https://lw2.issarice.com/posts/QjxYbo9yotsAH647Z/simple-refutation-of-the-bayesian-philosophy-of-science#gaCpdpduJSmwkrPGf
<blockquote>
<p>Infinity minus one isn't smaller than infinity. That's not useful in that way.</p>
</blockquote>
<p>The thing being added or subtracted is not the mere number of hypotheses, but a measure of the likelihood of those hypotheses. We might suppose an infinitude of mutually exclusive theories of the world, but most of them are <em>extremely</em> unlikely - for any degree of unlikeliness, there are an infinity of theories less likely than that! A randomly-chosen theory is so unlikely to be true, that if you add up the likelihoods of every single theory, they add up to a number less than infinity.</p>
<p>It is for this reason that it is important when we divide our hypotheses between something likely, and everything else. "Everything else" contains infinite possibilities, but only finite likelihood.</p>
manfredgaCpdpduJSmwkrPGf2017-11-02T21:07:28.927ZComment by Manfred on Simple refutation of the ‘Bayesian’ philosophy of science
https://lw2.issarice.com/posts/QjxYbo9yotsAH647Z/simple-refutation-of-the-bayesian-philosophy-of-science#tv7W6PyvuT7YtMeRE
<p>I think this neglects the idea of "physical law," which says that theories can be good when they capture the dynamics and building-blocks of the world simply, even if they are quite ignorant about the complex initial conditions of the world.</p>
manfredtv7W6PyvuT7YtMeRE2017-11-02T20:55:20.679ZComment by Manfred on I Want to Review FDT; Are my Criticisms Legitimate?
https://lw2.issarice.com/posts/BZencW7kagiQyp3AH/i-want-to-review-fdt-are-my-criticisms-legitimate#ukCarXFd2hkYJm59C
<blockquote>
<p>Can't this be modelled as uncertainty over functional equivalence? (or over input-output maps)?</p>
</blockquote>
<p>Hm, that's an interesting point. Is what we care about just the brute input-output map? If we're faced with a black-box predictor, then yes, all that matters is the correlation even if we don't know the method. But I don't think any sort of representation of computations as input-output maps actually helps account for how we should learn about or predict this correlation - we learn and predict the predictor in a way that seems like updating a distribution over computations. Nor does it seem to help in the case of trying to understand to what extend two agents are logically dependent on one another. So I think the computational representation is going to be more fruitful.</p>
manfredukCarXFd2hkYJm59C2017-10-25T21:41:14.556ZComment by Manfred on New program can beat Alpha Go, didn't need input from human games
https://lw2.issarice.com/posts/9s4wbfpd9AsJwapei/new-program-can-beat-alpha-go-didn-t-need-input-from-human#y3sbNjmT24PbcrYcc
<p>Interesting that resnets still seem state of the art. I was expecting them to have been replaced by something more heterogeneous by now. But I might be overrating the usefulness of discrete composition because it's easy to understand.</p>
manfredy3sbNjmT24PbcrYcc2017-10-18T22:59:35.379ZComment by Manfred on Open thread, October 16 - October 22, 2017
https://lw2.issarice.com/posts/z268NgT9AyTsdtHaX/open-thread-october-16-october-22-2017#zx9wPR5L89h7ZNi5s
<p>Plausibly? LW2 seems to be doing okay, which is gonna siphon off posts and comments.</p>
manfredzx9wPR5L89h7ZNi5s2017-10-17T16:45:14.252ZComment by Manfred on Open thread, October 2 - October 8, 2017
https://lw2.issarice.com/posts/XQbCpBkRnYDakqJF8/open-thread-october-2-october-8-2017#AQk5rxM6vQaFzyHta
<p>The dust probably is just dust - scattering of blue light more than red is the same reason the sky is blue and the sun looks red at sunset (Rayleigh scattering / Mie scattering). It comes from scattering off of particles smaller than a few times the wavelength of the light - so if visible light is being scattered less than UV, we know that lots of the particles are of size smaller than ~2 um. This is about the size of a small bacterium, so dust with interesting structure isn't totally out of the question, but still... it's probably just dust.</p>
manfredAQk5rxM6vQaFzyHta2017-10-06T20:04:35.346ZComment by Manfred on Open thread, October 2 - October 8, 2017
https://lw2.issarice.com/posts/XQbCpBkRnYDakqJF8/open-thread-october-2-october-8-2017#CDxzZGz4ras4ER8XH
<p>I think people get too hung up on computers as being mechanistic. People usually think of symbol manipulation in terms of easy-to-imagine language-like models, but then try to generalize their intuitions to computation in general, which can be unimaginably complicated. It's perfectly possible to simulate a human on an ordinary classical computer (to arbitrary precision). Would that simulation of a human be conscious, if they matched the behavior of a flesh and blood human almost perfectly, and could output to you via text channel and output things like "well, I sure feel conscious"?</p>
<p>The reason LWers are so confident that this simulation is conscious is because we think of concepts like "consciousness," to the extent that they exist, as having something to do with the cause of us talking and thinking about consciousness. It's just like how the concept of "apples" exists because apples exist, and when I correctly think I see an apple, it's because there's an apple. Talking about "consciousness" is presumed to be a consequence of our experience with consciousness. And the things we have experience with that we can label "consciousness" are introspective phenomena, physically realized as patterns of neurons firing, that have exact analogies in the simulation. Demanding that one has to be made of flesh to be conscious is not merely chauvinism, it's a misunderstanding of what we have access to when we encounter consciousness.</p>
manfredCDxzZGz4ras4ER8XH2017-10-05T21:58:26.090ZComment by Manfred on Open thread, October 2 - October 8, 2017
https://lw2.issarice.com/posts/XQbCpBkRnYDakqJF8/open-thread-october-2-october-8-2017#PZoSouq3EXNacFbwd
<p><a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3046725">Neat paper about the difficulties of specifying satisfactory values for a strong AI.</a> h/t Kaj Sotala.</p>
<blockquote>
<p>The design of social choice AI faces three sets of decisions: standing, concerning whose ethics views are included; measurement, concerning how their views are identified; and aggregation, concerning how individual views are combined to a single view that will guide AI behavior. [] Each set of decisions poses difficult ethical dilemmas with major consequences for AI behavior, with some decision options yielding pathological or even catastrophic results. </p>
</blockquote>
<p>I think it's slightly lacking in sophistication about aggregation of numerical preferences, and in how revealed preferences indicate that we don't actually have incommensurable or infinitely-strong preferences, but is overall pretty great.</p>
<p>On the subject of the problem, I don't think we should program in values that are ad-hoc on the object level (what values to use - trying to program this by hand is destined for failure), or even the meta level (whose values to use). But I do think it's okay to use an ad-hoc process to try to learn the answers to the meta-level questions. After all, what's the worst that could happen? (irony). Of course, the ability to do this assumes the solution of other, probably more difficult philosophical/AI problems, like how to refer to peoples' values in the first place.</p>
manfredPZoSouq3EXNacFbwd2017-10-04T19:54:06.048ZComment by Manfred on Open thread, October 2 - October 8, 2017
https://lw2.issarice.com/posts/XQbCpBkRnYDakqJF8/open-thread-october-2-october-8-2017#3e5E8gGEYRF8mbdLo
<p>Yeah, whenever you see a modifier like "just" or "merely" in a philosophical argument, that word is probably doing a lot of undeserved work.</p>
manfred3e5E8gGEYRF8mbdLo2017-10-04T18:32:45.213ZComment by Manfred on Open thread, October 2 - October 8, 2017
https://lw2.issarice.com/posts/XQbCpBkRnYDakqJF8/open-thread-october-2-october-8-2017#X4rrmmuFeCnK9xv3Z
<p>I don't, and maybe you've already been contacted, but you could try contacting him on social sites like this one (user paulfchristiano) and Medium, etc. Typical internet stalking skillset.</p>
manfredX4rrmmuFeCnK9xv3Z2017-10-04T18:27:20.199ZComment by Manfred on Open thread, October 2 - October 8, 2017
https://lw2.issarice.com/posts/XQbCpBkRnYDakqJF8/open-thread-october-2-october-8-2017#h7JzKdJtBGBHA6Ffd
<p>Ah, you mean to ask if the brain is special in a way that evades our ability to construct an analogy of the chinese room argument for it? E.g. "our neurons don't indiviually understand English, and my behavior is just the product of a bunch of neurons following the simple laws of chemistry, therefore there is nothing in my body that understands English."</p>
<p>I think such an argument is totally valid imitation. It doesn't necessarily bear on the Chinese room itself, which is a more artificial case, but it certainly applies to AI in general.</p>
manfredh7JzKdJtBGBHA6Ffd2017-10-04T01:09:04.014ZComment by Manfred on Feedback on LW 2.0
https://lw2.issarice.com/posts/vPQ3tfAAkE2Md2ZJH/feedback-on-lw-2-0#7cwrj9aafJbKWi59M
<p>You say impressions, but I'm assuming this is just the "things I want changed" thread :)</p>
<p>Vote button visibility and responsiveness is a big one for me. Ideally, it should require one click, be disabled while it messages the server, and then change color much more clearly.</p>
<p>On mobile, the layout works nicely, but load / render times are too long (how much javascript is necessary to serve text? Apparently, lots) and the text formatting buttons take up far too much space.</p>
<p>First time, non-logged in viewers should probably not see the green messaging blob in the corner, particularly on mobile.</p>
<p>I agree that some kind of demarcation between comments, and between comments and "write a new comment", would be nice. Doesn't have to be 2009 internet boxes, it can be 2017 internet fading horizontal lines or something.</p>
manfred7cwrj9aafJbKWi59M2017-10-01T17:21:44.825ZComment by Manfred on Intuitive explanation of why entropy maximizes in a uniform distribution?
https://lw2.issarice.com/posts/diZLkE7PeRF2rQ6sj/intuitive-explanation-of-why-entropy-maximizes-in-a-uniform#g3DACKnh3K9fAQpsw
<p>Well, it really <em>is</em> defined that way. Before doing math, it's important to understand that entropy is a way of quantifying our ignorance about something, so it makes sense that you're most ignorant when (for discrete options) you can't pick out one option as more probable than another.</p>
<p>Okay, on to using the definition of entropy as the sum over event-space of -P log(P) of all the events. E.g. if you only had one possible event, with probability 1, your entropy would be 1 log(1) = 0. Suppose you had two events with different probabilities. If you changed the probability assignment so their probability gets closer together, entropy goes up. This is because the function -P log(P) is concave downwards between 0 and 1 - this means that the entropy is always higher between two points than you'd get by just averaging those points (or taking any weighted average, represented by a straight line connecting the two points).. So if you want to maximize entropy, you move all the points together as far as they can go.</p>
manfredg3DACKnh3K9fAQpsw2017-09-23T15:21:52.615ZComment by Manfred on LW 2.0 Strategic Overview
https://lw2.issarice.com/posts/rEHLk9nC5TtrNoAKT/lw-2-0-strategic-overview#AWDQBBTEgFEFZTFhS
<p>Moderation is basically the only way, I think. You could try to use fancy pagerank-anchored-by-trusted-users ratings, or make votes costly to the user in some way, but I think moderation is the necessary fallback.</p>
<p>Goodhart's law is real, but people still try to use metrics. Quality may speak for itself, but it can be too costly to listen to the quality of every single thing anyone says.</p>
manfredAWDQBBTEgFEFZTFhS2017-09-15T19:58:46.025ZComment by Manfred on LW 2.0 Strategic Overview
https://lw2.issarice.com/posts/rEHLk9nC5TtrNoAKT/lw-2-0-strategic-overview#PYbJSeR4B7EdjbLYS
<p>The only thing I don't like about the "2017 feel" is that it sometimes feel like you're just adrift in the text, with no landmarks. Sometimes you just want guides to the eye, and landmarks to keep track of how far you've read!</p>
manfredPYbJSeR4B7EdjbLYS2017-09-15T19:49:32.738ZComment by Manfred on LW 2.0 Strategic Overview
https://lw2.issarice.com/posts/rEHLk9nC5TtrNoAKT/lw-2-0-strategic-overview#HQavX5BqZ7MTqLe9e
<p>I also agree that HPMOR might need to go somewhere other than the front page. From a strategic perspective, I somehow want to get the benefits of HPMOR existing (publicity, new people finding the community) without the drawbacks (it being too convenient to judge our ideas by association).</p>
manfredHQavX5BqZ7MTqLe9e2017-09-15T19:47:53.513ZComment by Manfred on LW 2.0 Strategic Overview
https://lw2.issarice.com/posts/rEHLk9nC5TtrNoAKT/lw-2-0-strategic-overview#QbhwKTMLCMnrXC3QD
<p>I think votes have served several useful purposes.</p>
<p>Downvotes have been a very good way of enforcing the low-politics norm.</p>
<p>When there's lots of something, you often want to sort by votes, or some ranking that mixes votes and age. Right now there aren't many comments per thread, but if there were 100 top-level comments, I'd want votes. Similarly, as a new reader, it was very helpful to me to look for old posts that people had rated highly.</p>
manfredQbhwKTMLCMnrXC3QD2017-09-15T17:27:50.648ZComment by Manfred on The Doomsday argument in anthropic decision theory
https://lw2.issarice.com/posts/g94oAbSna8hpGJTSu/the-doomsday-argument-in-anthropic-decision-theory#564G8hacw4ct6Ls9F
<p>And what if the universe is probably different for the two possible copies of you, as in the case of the boltzmann brain? Presumably you have to take some weighted average of the "non-anthropic probabilities" produced by the two different universes.</p>
<p>Re: note. This use of SSA and SIA can also be wrong. If there is a correct method for assigning subjective probabilities to what S.B. will see when she looks at outside, it should not be an additional thing on top of predicting the world, it should be a natural part of the process by which S.B. predicts the world.</p>
<p>EDIT: Okay, getting a better understanding of what you mean now. So you'd probably just say that the weight on the different universes should be exactly this non-anthropic probability, assigned by some universal prior or however one assigns probability to universes. My problem with this is that when assigning probabilities in a principled, subjective way - i.e. trying to figure out what your information about the world really implies, rather than starting by assuming some model of the world, there is not necessarily an easily-identifiable thing that is the non-anthropic probability of a boltzmann brain copy of me existing, and this needs to be cleared up in a way that isn't just about assuming a model of the world. If anthropic reasoning is, as I said above, not some add-on to the process of assigning probabilities, but a part of it, then it makes less sense to think something like "just assign probabilities, but don't do that last anthropic step."</p>
<p>But I suspect this problem actually can be resolved. Maybe by interpreting the non-anthropic number as something like the probability that the universe is a certain way (i.e. assuming some sort of physicalist prior), conditional on there only being at least one copy of me, and then assuming that this resolves all anthropic problems?</p>
manfred564G8hacw4ct6Ls9F2017-09-04T16:40:31.548ZComment by Manfred on The Doomsday argument in anthropic decision theory
https://lw2.issarice.com/posts/g94oAbSna8hpGJTSu/the-doomsday-argument-in-anthropic-decision-theory#MYKxLBemqFFkGSA2n
<p>That's not quite what I was talking about, but I managed to resolve my question to my own satisfaction anyhow. The problem of conditionalization can be worked around fairly easily.</p>
<blockquote>
<p>Suppose that there is 50% ehance of there being a boltzmann brain copy of you</p>
</blockquote>
<p>Actually, the probability that you should assign to there being a copy of you is not defined under your system - otherwise you'd be able to conceive of a solution to the sleeping beauty problem - the entire schtick is that Sleeping Beauty is not merely ignorant about whether another copy of her exists, but that it is supposedly a bad question.</p>
<p>Hm, okay, I think this might cause trouble in a different way that I was originally thinking of. Because all sorts of things are <em>possibilities</em>, and it's not obvious to me how ADT is able to treat reasonable anthropic possibilities different from astronomically-unlikely ones, if it throws out any measure of unlikeliness. You might try to resolve this by putting in some "outside perspective" probabilities, e.g. that an outside observer in our universe would see me as normal most of the time and me as a Boltzmann brain less of the time, but this requires making drastic assumptions about what the "outside observer" is actually outside, observing. If I really was a Boltzmann brain in a thermal universe, an outside observer would think I was more likely to be a Boltzmann brain. So postulating an outside perspective is just an awkward way of sneaking in probabilities gained in a different way.</p>
<p>This seems to leave the option of really treating all apparent possibilities similarly. But then the benefit of good actions in the real world gets drowned out by all the noise from all the unlikely possibilities - after all, for every action, one can construct a possibility where it's both good and bad. If there's no way to break ties between possibilities, no ties get broken.</p>
manfredMYKxLBemqFFkGSA2n2017-09-03T01:06:31.835ZComment by Manfred on Intrinsic properties and Eliezer's metaethics
https://lw2.issarice.com/posts/8ZvKTHH5PbChtZrbp/intrinsic-properties-and-eliezer-s-metaethics#5KgpZZo7fiv9jhRn3
<p>Moral value is not an "intrinsic property" of a mathematical structure - aliens couldn't look at this mathematical structure and tell that it was morally important. And yet, whenever we compute something, there is a corresponding abstract structure. And when we reason about morality, we say that what is right wouldn't change if you gave us brain surgery, so by morality we don't mean "whatever we happen to think," we mean that abstract structure.</p>
<p>Meanwhile, we are actual evolved mammals, and the reason we think what we do about morality is because of evolution, culture, and chance, in that order. I'm not sure what the point is of calling this objective or not, but it definitely has reasons for being how it is. But maybe you can see how this evolved morality can also be talked about as an abstract structure, and therefore both of these paragraphs can be true at the same time.</p>
<p>It seems like you were looking for things with "intrinsic properties" and "objective"-ness that we don't much care about, and maybe this is why the things you were thinking of were incompatible, but the things we're thinking of are compatible.</p>
manfred5KgpZZo7fiv9jhRn32017-09-01T20:34:10.960ZComment by Manfred on The Doomsday argument in anthropic decision theory
https://lw2.issarice.com/posts/g94oAbSna8hpGJTSu/the-doomsday-argument-in-anthropic-decision-theory#j9pWQK9oNWuiCE9Rh
<p>Since we are in the real world, it is a possibility that there is a copy of me, e.g. as a boltzmann brain, or a copy of the simulation I'm in.</p>
<p>Does your refusal to assign probabilities to these situations infect everyday life? Doesn't betting on a coin flip require conditioning on whether I'm a boltzmann brain, or am in a simulation that replaces coins with potatoes if I flip them? You seem to be giving up on probabilities altogether.</p>
manfredj9pWQK9oNWuiCE9Rh2017-09-01T15:58:46.824ZComment by Manfred on Is life worth living?
https://lw2.issarice.com/posts/zehQzjmBoeFYvnqeQ/is-life-worth-living#JccJRhXqA2BsdgvuW
<p>This seems like a question about aesthetics - thia choice won't change my experience, but it will change what kind of universe I live in. I think I'd choose duplication - I put a pretty low value on tiling the universe with conscious experience, but it's larger than zero.</p>
manfredJccJRhXqA2BsdgvuW2017-08-30T16:37:59.816ZComment by Manfred on Intrinsic properties and Eliezer's metaethics
https://lw2.issarice.com/posts/8ZvKTHH5PbChtZrbp/intrinsic-properties-and-eliezer-s-metaethics#pKtctxoGqGRD4nExT
<p>I totally agree. Perceived differences in kind here are largely due to the different methods we use to think about these things.</p>
<p>For the triangle, everybody knows what a triangle is, we don't even need to use conscious thought to recognize them. But for the key, I can't quite keep the entire shape in my memory at once, if I want to know if something is shaped like my front door key, I have to compare it to my existing key, or try it in the lock.</p>
<p>So it naively seems that triangleness is something intrinsic (because I perceive it without need for thought), while front-door-key-shape is something that requires an external reference (because I need external reference). But just because I perceive the world a certain way doesn't mean those are particularly important distinctions. If one wanted to make a computer program recognize triangles and my front door key shape, one could just as well use the same code for both.</p>
manfredpKtctxoGqGRD4nExT2017-08-30T03:14:15.594ZComment by Manfred on Open thread, August 28 - September 3, 2017
https://lw2.issarice.com/posts/3F5zp2SiQbhTnBwWk/open-thread-august-28-september-3-2017#YcrFoLi5T93mCbzpL
<p>What is the analogy of sum that you're thinking about? Ignoring how the little pieces are defined, what would be a cool way to combine them? For example, you can take the product of a series of numbers to get any number, that's pretty cool. And then you can convert a series to a continuous function by taking a limit, <a href="https://en.wikipedia.org/wiki/Product_integral">just like an integral</a>, except rather than the limit going to really small pieces, the limit goes to pieces really close to 1.</p>
<p>You could also raise a base to a series of powers to get any number, then take that to a continuous limit to get an integral-analogue. Or do other operations in series, but I can't think of any really motivating ones right now.</p>
<p>Can you invert these to get derivative-analogues <a href="https://en.wikipedia.org/wiki/List_of_derivatives_and_integrals_in_alternative_calculi">(wiki page)</a>? For the product integral, the value of the corresponding derivative turns out to be the limit of more and more extreme inverse roots, as you bring the ratio of two points close to 1.</p>
<p>Are there any other interesting derivative-analogues? What if you took the inverse of the difference between points, but then took a larger and larger root? Hmm... You'd get something that was 1 almost everywhere for nice functions, except where the function's slope got super-polynomially flat or super-polynomially steep.</p>
manfredYcrFoLi5T93mCbzpL2017-08-29T21:41:37.991ZComment by Manfred on Is there a flaw in the simulation argument?
https://lw2.issarice.com/posts/WuzYpoDoPECqyonQR/is-there-a-flaw-in-the-simulation-argument#jQbNwM6LnCWCTurrX
<p>In problems where multiple different agents in the universe "could be you," (i.e. share information), you really don't have to do anything fancy. Just assign equal probability to all agents in the entire universe who, as far as you know, can match your current state of information.</p>
<p>If there are two copies of Earth, and, hypothetically, only these two copies of me in the universe, I assign 50% probability to being each. This stays the same whether these Earths are at different points in space, or at different times, or sped up or slowed down or played in reverse order due to strange entropic boundary conditions. All that matters is that on each Earth there is a copy of me that has the same information as me from their own internal perspective, and I just count these copies up.</p>
manfredjQbNwM6LnCWCTurrX2017-08-29T17:06:48.015ZComment by Manfred on Request For Collaboration
https://lw2.issarice.com/posts/7tv7SMkRf4zbgueRk/request-for-collaboration#ZYg47nLEjz22Bkr8j
<p>I will be frank. This sounds like a lame deal for anyone who takes you up on the offer. "My physics is shit, but I have a great idea for a new theory of gravity. PM me if you are a professional physicist and want to coauthor a paper." "My writing is shit, but I have a clever idea for a story and would like someone to write it it for me."</p>
<p>First you should do 90% of the work you know about, then maybe you can find a professional to do the last 10% plus the things you didn't know about. Read the relevant philosophy! Go read wikipedia, read the stanford encyclopedia of philosophy, go to your library and check out a book or three. Do a lot of writing! Make arguments, try to find good ways to phrase things, think of counterarguments other people might use, explain how this builds on and extends the stuff you read about. <em>Then</em> maybe, if you put that out in the public and ask for someone to devote their time to making this idea spread, you might get takers.</p>
manfredZYg47nLEjz22Bkr8j2017-08-29T00:06:49.246ZComment by Manfred on Open thread, August 21 - August 27, 2017
https://lw2.issarice.com/posts/QrFHot92Csnjk8Ke9/open-thread-august-21-august-27-2017#CMPKtChgro2o2MCWe
<blockquote>
<p>Across the frozen sea around most of Antartica even in the summertime?</p>
</blockquote>
<p>I'm not sure if you're actually curious, or if you think this is a "gotcha" question.</p>
<p><a href="https://en.wikipedia.org/wiki/Ice_shelf#/media/File:Antarctic_shelf_ice_hg.png">Here's a picture</a>. As the glacier flows outward (<a href="https://nsidc.org/data/docs/measures/nsidc0484_rignot/images/velocitymap.png">here's</a> measured flow rates), it begins floating on the sea and becomes an ice shelf, which then loses mass to the ocean through melting and breaking up into pieces, which then melt. This ice shelf is thick (100m - 1 km scale), because it's a really thick sheet of ice being pushed out into the water by gravity. It then encounters the sea ice, which is ~1-4 meters thick. The sea ice gets pushed out, or piled up, because there are no particular forces holding the sea ice in place.</p>
<p>At this point I'm tapping out of the conversation. Either you're ignorant but curious and there's no point to me typing up things you could look up, or you want to feel superior while remaining ignorant and there's no point to me typing up things you don't care about.</p>
manfredCMPKtChgro2o2MCWe2017-08-22T11:41:49.694ZComment by Manfred on Open thread, August 21 - August 27, 2017
https://lw2.issarice.com/posts/QrFHot92Csnjk8Ke9/open-thread-august-21-august-27-2017#tddYjzshLCGJSafzS
<p>If the glacier is flowing off of the continent into the sea, then sea ice is in an equilibrium between melting at the edges and bottom and being replenished at the middle.</p>
<blockquote>
<p>demands huge melting we don't see</p>
</blockquote>
<p>"See" how? It seems to me that you don't have an involved understanding of the melting of glaciers. If we could measure the mass of the Antarctic glacier straightforwardly, then I'm sure we'd agree on the meaning of changes in that mass. But if we don't see the particular melting process you expect, perhaps you're just expectung the wrong process, and <em>haven't</em> uncovered a conspiracy among all the experts.</p>
<blockquote>
<p>Much smaller numbers, popular now</p>
</blockquote>
<p>In my experience, actually reading the ipcc review has never been popular and still isn't. I'm sure you could still find someone in the press claiming larger sea level rise, if you tried. But why pick the easiest opponent?</p>
manfredtddYjzshLCGJSafzS2017-08-22T07:34:58.519ZComment by Manfred on Open thread, August 21 - August 27, 2017
https://lw2.issarice.com/posts/QrFHot92Csnjk8Ke9/open-thread-august-21-august-27-2017#6XaS3HbErsbqtxxch
<p>Neat!</p>
<p>Glaciers don't have to form icebergs in order to melt. It can just melt where it meets the sea.</p>
<blockquote>
<p>Almost 3 Amazons still missing for the 6 meters sea rise in a century</p>
</blockquote>
<p>You know, now that you mention it, 6 meters sure is a lot. Where did you get that number from? <a href="https://www.ipcc.ch/pdf/assessment-report/ar5/wg1/WG1AR5_Chapter13_FINAL.pdf">See p. 1181 for IPCC projections</a>.</p>
manfred6XaS3HbErsbqtxxch2017-08-21T16:28:22.399ZComment by Manfred on Open thread, August 21 - August 27, 2017
https://lw2.issarice.com/posts/QrFHot92Csnjk8Ke9/open-thread-august-21-august-27-2017#3E3BmcNTgxSdiJSFF
<p>How about glacial flow? Ice doesn't move fast, but it does move. It can postpone melting until it's in contact with seawater. What do you think the ratio of mass moved by rivers vs. glaciers is in Antarctica?</p>
manfred3E3BmcNTgxSdiJSFF2017-08-21T14:41:32.635ZComment by Manfred on Open thread, August 21 - August 27, 2017
https://lw2.issarice.com/posts/QrFHot92Csnjk8Ke9/open-thread-august-21-august-27-2017#BHPC8mrrxw8ZwEvE4
<blockquote>
<p>Where and how some people see three Amazons on Antarctica, is a mystery to me. The amount of ice falling directly into the sea, is quite pathetic, as well.</p>
</blockquote>
<p>The amazon begins distributed across brazil, as the occasional drops of rain. Then it comes together because of the shape and material of the landscape, and flows into streams, which join into rivers, which feed one big river. If global warming is causing antarctica to lose mass, do you expect the same thing to happen in antarctica, with meltwater beginning distributed across the surface, and then collecting into rivers and streams?</p>
manfredBHPC8mrrxw8ZwEvE42017-08-21T14:05:24.396ZComment by Manfred on Multiverse-wide Cooperation via Correlated Decision Making
https://lw2.issarice.com/posts/gBiuKBLof7AyDoG5r/multiverse-wide-cooperation-via-correlated-decision-making#jz6oBb7KoEQfymDin
<p>Why do we care about acausal trading with aliens to promote them acting with "moral reflection, moral pluralism," etc?</p>
manfredjz6oBb7KoEQfymDin2017-08-20T14:25:53.024ZComment by Manfred on We need to think more about Terminal Values
https://lw2.issarice.com/posts/YBxEJHsxc5vvNdP9G/we-need-to-think-more-about-terminal-values#jhkenjrdermNmC3Li
<p>I think writing something like this is a bit like a rite of passage. So, welcome to LW :P</p>
<p>When we talk about someone's values, we're using something like Dan Dennett's <a href="https://ase.tufts.edu/cogstud/dennett/papers/intentionalsystems.pdf">intentional stance</a>. You might also enjoy <a href="http://lesswrong.com/lw/6ha/the_blueminimizing_robot/">this LW post</a> about <em>not</em> applying the intentional stance.</p>
<p>Long story short, there is no "truly true" answer to what people want, and no "true boundary" between person and environment, but there are answers and boundaries that are good enough for what people usually mean.</p>
manfredjhkenjrdermNmC3Li2017-08-17T11:46:08.237ZComment by Manfred on Open thread, August 14 - August 20, 2017
https://lw2.issarice.com/posts/rrBGx5zFPgBjzpLPd/open-thread-august-14-august-20-2017#urwqqDyNhMtqqBnwP
<p>Well, if the acronym "POMDP" didn't make any sense, I think we should start with a simpler example, like a chessboard.</p>
<p>Suppose we want to write a chess-playing AI that gets its input from a camera looking at the chessboard. And for some reason, we give it a button that replaces the video feed with a picture of the board in a winning position.</p>
<p>Inside the program, the AI knows about the rules of chess, and has some heuristics for how it expects the opponent to play. Then it represents the external chessboard with some data array. Finally, it has some rules about how the image in the camera is generated from the true chessboard and whether or not it's pressing the button.</p>
<p>If we just try to get the AI to make the video feed be of a winning position, then it will press the button. But if we try to get the AI to get its internal representation of the data array to be in a winning position, and we update the internal representation to try to track the true chessboard, then it won't press the button. This is actually quite easy to do - for example, if the AI is a jumble of neural networks, and we have a long training phase in which it's rewarded for actually winning games, not just seeing winning board states, then it will learn to take into account the state of the button when looking at the image.</p>
manfredurwqqDyNhMtqqBnwP2017-08-16T07:35:45.062ZComment by Manfred on Open thread, August 14 - August 20, 2017
https://lw2.issarice.com/posts/rrBGx5zFPgBjzpLPd/open-thread-august-14-august-20-2017#XKoZaSkyzB7N7vPfN
<p>To our best current understanding, it has to have a model of the world (e.g. as a POMDP) that contains a count of the number of paperclips, and that it can use to predict what effect its actions will have on the number of paperclips. Then it chooses a strategy that will, according to the model, lead to lots of paperclips. </p>
<p>This won't want to fool itself because, according to basically any model of the world, fooling yourself does not result in more paperclips.</p>
manfredXKoZaSkyzB7N7vPfN2017-08-15T22:16:34.556ZComment by Manfred on Open thread, August 7 - August 13, 2017
https://lw2.issarice.com/posts/DedF6XfDhy8f8WeLe/open-thread-august-7-august-13-2017#fxmk7ntzEfrRPtrBh
<p>I think I don't know the solution, and if so it's impossible for me to guess what he thinks if he's right :)</p>
<p>But maybe he's thinking of something vague like CIRL, or hierarchical self-supervised learning with generation, etc. But I think he's thinking of some kind of recurrent network. So maybe he has some clever idea for unsupervised credit assignment?</p>
manfredfxmk7ntzEfrRPtrBh2017-08-09T17:57:14.805ZComment by Manfred on Open thread, July 31 - August 6, 2017
https://lw2.issarice.com/posts/RjjGFmppPZF3iB5hL/open-thread-july-31-august-6-2017#rMa5b2jhEDTsjNaMz
<p>Cool insight. We'll just pretend constant density of 3M/4r^3.</p>
<p>This kind of integral shows up all the time in E and M, so I'll give it a shot to keep in practice.</p>
<p>You simplify it by using the law of cosines, to turn the vector subtraction 1/|r-r'|^2 into 1/(|r|^2+|r'|^2+2|r||r'|cos(θ)). And this looks like you still have to worry about integrating two things, but actually you can just call r' due north during the integral over r without loss of generality.</p>
<p>So now we need to integrate 1/(r^2+|r'|^2+2r|r'|cos(θ)) r^2 sin(θ) dr dφ dθ. First take your free 2π from φ. Cosine is the derivative of sine, so substitution makes it obvious that the θ integral gives you a log of cosine. So now we integrate 2πr (ln(r^2+|r'|^2+2r|r'|) - ln(r^2+|r'|^2-2r|r'|)) / 2|r'| dr from 0 to R. Which mathematica says is some nasty inverse-tangent-containing thing.</p>
<p>Okay, maybe I don't actually want to do this integral that much :P</p>
manfredrMa5b2jhEDTsjNaMz2017-07-31T22:24:30.906ZComment by Manfred on Sleeping Beauty Problem Can Be Explained by Perspective Disagreement (IV)
https://lw2.issarice.com/posts/Hurzhs3PHNaD52eSD/sleeping-beauty-problem-can-be-explained-by-perspective#LWjMTu3FgwFreNwGC
<p>To spell it out:</p>
<p>Beauty knows limiting frequency (which, when known, is equal to the probability) of the coin flips that she sees right in front of her will be equal to one-half. That is, if you repeat the experiment many times (plus a little noise to determine coin flips), then you get equal numbers of the event "Beauty sees a fair coin flip and it lands Heads" and "Beauty sees a fair coin flip and it lands Tails." Therefore Beauty assigns 50/50 odds to any coin flips she actually gets to see.</p>
<p>You can make an analogous argument from symmetry of information rather than limiting frequency, but it's less accessible and I don't expect people to think of it on their own. Basically, the only reason to assign thirder probabilities is if you're treating states of the world given your information as the basic mutually-exclusive-and-exhaustive building block of probability assignment. And the states look like Mon+Heads, Mon+Tails, and Tues+Tails. If you eliminate one of the possibilities, then the remaining two are symmetrical.</p>
<p>If it seems paradoxical that, upon waking up, she thinks the Monday coin is more likely to have landed tails, just remember that half of the time that coin landed tails, it's Tuesday and she never gets to see the Monday coin being flipped - as soon as she actually expects to see it flipped, that's a new piece of information that causes her to update her probabilities.</p>
manfredLWjMTu3FgwFreNwGC2017-07-31T18:39:29.473ZComment by Manfred on Sleeping Beauty Problem Can Be Explained by Perspective Disagreement (IV)
https://lw2.issarice.com/posts/Hurzhs3PHNaD52eSD/sleeping-beauty-problem-can-be-explained-by-perspective#qFjdvNiF2XG2QvE3Y
<blockquote>
<p>According to SSA beauty should update credence of H to 2/3 after learning it is Monday.</p>
</blockquote>
<p>I always forget what the acronyms are. But the probability of H is 1/2 after learning it's Monday, any any method that says otherwise is wrong, exactly by the argument that you can flip the coin on monday right in front of SB, and if she knows it's Monday and thinks it's not a 50/50 flip, her probability assignment is bad.</p>
manfredqFjdvNiF2XG2QvE3Y2017-07-30T04:09:10.479ZComment by Manfred on Sleeping Beauty Problem Can Be Explained by Perspective Disagreement (IV)
https://lw2.issarice.com/posts/Hurzhs3PHNaD52eSD/sleeping-beauty-problem-can-be-explained-by-perspective#CggxgL2CFxexMBw7z
<blockquote>
<p>He proposes the coin toss could happen after the first awakening. Beauty’s answer ought to remain the same regardless the timing of the toss. A simple calculation tells us his credence of H must be 1/3. As SSA dictates this is also beauty’s answer. Now beauty is predicting a fair coin toss yet to happen would most likely land on T. This supernatural predicting power is a conclusive evidence against SSA.</p>
</blockquote>
<p>So how do you get Beauty's prediction? If at the end of the first day you ask for a prediction on the coin, but you don't ask on the second day, then now Beauty knows that the coin flip is, as you say, yet to happen, and so she goes back to predicting 50/50. She only deviates from 50/50 when she thinks there's some chance that the coin flip has already happened.</p>
<p>Sometimes people absolutely will come to different conclusions. And I think you're part of the way there with the idea of letting people talk to see if they converge. But I think you'll get the right answer even more often if you set up specific thought-experiment processes, and then had the imaginary people in those thought experiments bet against each other, and say the person (or group of people all with identical information) who made money on average (where "average" means over many re-runs of this specific thought experiment) had good probabilities, and the people who lost money had bad probabilities.</p>
<p>I don't think this is what probabilities <em>mean</em>, or that it's the most elegant way to find probabilities, but I think it's a pretty solid and non-confusing way. And there's a quite nice discussion article about it somewhere on this site that I can't find, sadly.</p>
manfredCggxgL2CFxexMBw7z2017-07-28T22:19:23.025ZComment by Manfred on Sleeping Beauty Problem Can Be Explained by Perspective Disagreement (II)
https://lw2.issarice.com/posts/Bfn4sTSyidnDeJm44/sleeping-beauty-problem-can-be-explained-by-perspective#yPYb6h8nJWd6veF4X
<p>Sorry for the slow reply.</p>
<p>The 8 rooms are definitely the unbiased sample (of your rooms with one red room subtracted).</p>
<p>I think you are making two mistakes:</p>
<p>First, I think you're too focused on the nice properties of an unbiased sample. You can take an unbiased sample all you want, but if we know information in addition to the sample, our best estimate might not be the average of the sample! Suppose we have two urns, urn A has 10 red balls and 10 blue balls, while urn B has 5 red balls and 15 blue balls. We choose an urn by rolling a die, such that we have a 5/6 chance of choosing urn A and a 1/6 chance of choosing urn B. Then we take a fair, unbiased sample of 4 balls from whatever urn we chose. Suppose we draw out 1 red ball and 3 blue balls. Since this is an unbiased sample, does the process that you are calling "statistical analysis" have to estimate that we were drawing from urn B?</p>
<p>Second, you are trying too hard to make everything about the rooms. It's like someone was doing the problem with two urns from the previous paragraph, but tried to mathematically arrive at the answer only as a function of the number of red balls drawn, without making any reference to the process that causes them to draw from urn A vs. urn B. And they come up with several different ideas about what the function could be, and they call those functions "the Two-Thirds-B-er method" and "the Four-Tenths-B-er method." When really, both methods are incomplete because they fail to take into account what we know about how we picked the urn to draw from.</p>
<blockquote>
<p>To answer the last part of your statement. If beauty randomly opens 8 doors and found them all red then she has a sample of pure red. By simple statistics she should give R=81 as the estimation. Halfer and thirders would both agree on that. If they do a bayesian analysis R=81 would also be the case with the highest probability. I'm not sure where 75 comes from I'm assuming by summing the multiples of probability and Rs in the bayesian analysis? But that value does not correspond to the estimation in statistics. Imagine you randomly draw 20 beans from a bag and they are all red, using statistics obviously you are not going to estimate the bag contains 90% red bean.</p>
</blockquote>
<p>Think of it like this: if Beauty opens 8 doors and they're all red, and then she goes to open a ninth door, how likely should she think it is to be red? 100%, or something smaller than 100%? For predictions, we use the average of a probability distribution, not just its highest point.</p>
manfredyPYb6h8nJWd6veF4X2017-07-28T19:46:47.861ZComment by Manfred on Type Theory quick question
https://lw2.issarice.com/posts/vStgTQMxcmctTmCXx/type-theory-quick-question#JSiLi9yzfQoxu64ry
<p>The HoTT book is pretty readable, but I'm not in a position to evaluate its actual goodness.</p>
manfredJSiLi9yzfQoxu64ry2017-07-27T00:12:07.177ZComment by Manfred on Steelmanning as an alternative to Rationalist Taboo
https://lw2.issarice.com/posts/scW3MN2GT6zMR92c4/steelmanning-as-an-alternative-to-rationalist-taboo#vC95mAQ2y739vXro3
<p>In your example, I think Bob is doing something unrelated to rationalist Taboo.</p>
<p>In the <a href="https://i5.walmartimages.com/asr/df44febe-1692-468b-b9f4-2cc14d4748c3_1.0a064b061e5b82290140b6362a2b34fa.jpeg?odnHeight=450&odnWidth=450&odnBg=FFFFFF">actual factual game</a> of Taboo, you replace a word with a description that is sufficient to tell your team what the original word is. In rationalist Taboo, you replace a word with a description that is sufficient to convey the ideas you were trying to convey with the original word.</p>
<p>So if Bob tries to taboo "surprise" as "the feeling of observing a low-probability event," and Alice says "A license plate having the number any particular number is low probability - is it surprising?," Bob should think "Oh, the description I replaced 'surprise' with did not convey the same thing as the word 'surprise'. I need to try tabooing it differently."</p>
<p>This works better when you're trying to taboo the usage of a word in a specific context, because the full meaning of a word is very very complicated (though trying to make definitions can still be a fun and profitable game, I agree), but when you look at how you've used it in just one sentence, then you have some hope of pinning down what you mean by it to your satisfaction.</p>
manfredvC95mAQ2y739vXro32017-07-25T19:46:38.336ZComment by Manfred on Is Altruism Selfish?
https://lw2.issarice.com/posts/5LrsvY2NWjRoG4j7g/is-altruism-selfish#CoAfJeBXDWWtovCt9
<p>And yet, people, when giving examples of selfishness, don't just sample the entirety of human behavior. They point out a specific sort of behavior. Or when naming optimization functions, they might call one function "greedy," even though all functions tautologically do what they do. So clearly people have some additional criteria for everyday use of the word not captured by the extremely simple definition in this post.</p>
manfredCoAfJeBXDWWtovCt92017-07-24T18:31:39.028ZComment by Manfred on How long has civilisation been going?
https://lw2.issarice.com/posts/geRMGqLzYzKPzWdt8/how-long-has-civilisation-been-going#3eHB8P2rvpM6qREB2
<p><a href="http://www.gallup.com/poll/163697/approve-marriage-blacks-whites.aspx">First, I checked out the polling data on interracial marriage.</a> Every 10 years the approval rating has gone up by ~15 percentage points. I couldn't find a concise presentation of the age-segregated data from now vs. in the past, but <a href="http://www.gallup.com/poll/28417/most-americans-approve-interracial-marriages.aspx">2007</a> and <a href="http://www.gallup.com/poll/149390/record-high-approve-black-white-marriages.aspx">1991</a> were available, and they look consistent with over 80% of the opinion change being due to old people dying off. This surprised me, I expected to see more evidence of people changing their mind.</p>
<p><a href="http://www.gallup.com/poll/191645/americans-support-gay-marriage-remains-high.aspx">Now look at gay marriage.</a>. It's gained at ~18 points per 10 years. This isn't too different from 15, so maybe this is people dying off too. And indeed it seems to be mostly the case - except the in the last 10 years, where the gains don't follow the right age pattern, indicating that of 18 points of gain, about 40% may actually involve people changing their minds.</p>
manfred3eHB8P2rvpM6qREB22017-07-24T18:01:52.063ZComment by Manfred on Can anyone refute these arguments that we live on the interior of a hollow Earth?
https://lw2.issarice.com/posts/YoH4WQZZftSAoBSYg/can-anyone-refute-these-arguments-that-we-live-on-the#BGiSjKwtcKNriArG7
<p>"Refute" is usually not an objective thing - it's a social thing. You can probably prove to yourself that pi=3 is false, but if you write "pi=3" on a sheet of paper, no argument will make the ink rearrange itself to be correct.</p>
<p>This is one of the problems with a falsificationist idea of scientific progress, where we never prove theories true but make progress by proving them false. If evidence against a theory appears (e.g. the ability to see different stars from different parts of the earth might be thought of as "refuting" the idea of a flat earth), a proponent of that theory never <em>has</em> to give up on it. They can just patch the theory. Maybe light does a special little dance to make all the observations look like we're looking out at a universe, etc. If you try to refute someone, they can just refuse to be refuted and add another patch to their theory.</p>
<p>After doing some reading, I feel like this guy actually does a pretty admirable job of seeing open questions and admitting ignorance. For example, he doesn't know about the coriolis effect, so he calls it "a mysterious thing that happens to objects falling down mineshafts" and wonders whether it could cause an error in the readings of plumb bobs hanging down a mineshaft. Again, I think this is a good thing, though not as good as knowing about the coriolis effect before trying to understand the structure of the cosmos. The trouble seems mostly to be that he's read a lot of books that are full of shit, and believes them.</p>
manfredBGiSjKwtcKNriArG72017-07-21T21:29:18.277ZValue learners & wireheading
https://lw2.issarice.com/posts/GgXpryqWGePMMmYxf/value-learners-and-wireheading
<p><a href="http://www.danieldewey.net/learning-what-to-value.pdf">Dewey 2011</a> lays out the rules for one kind of agent with a mutable value system. The agent has some distribution over utility functions, which it has rules for updating based on its interaction history (where "interaction history" means the agent's observations and actions since its origin). To choose an action, it looks through every possible future interaction history, and picks the action that leads to the highest expected utility, weighted both by the possibility of making that future happen and the utility function distribution that would hold if that future came to pass.</p>
<p><img style="float: right;" src="http://images.lesswrong.com/t3_n5z_0.png?v=65b9c090ba7b9eb530bc95128474d2a7" alt="Drone can bring sandwich either to work or to home" width="348" height="229" />We might motivate this sort of update strategy by considering a sandwich-drone bringing you a sandwich. The drone can either go to your workplace, or go to your home. If we think about this drone as a value-learner, then the "correct utility function" depends on whether you're at work or at home - upon learning your location, the drone should update its utility function so that it wants to go to that place. (Value learning is unnecessarily indirect in this case, but that's because it's a simple example.)</p>
<p>Suppose the drone begins its delivery assigning equal measure to the home-utility-function and to the work-utility-function (i.e. ignorant of your location), and can learn your location for a small cost. If the drone evaluated this idea with its current utility function, it wouldn't see any benefit, even though it would in fact deliver the sandwich properly - because under its current utility function there's no point to going to one place rather than the other. To get sensible behavior, and properly deliver your sandwich, the drone must evaluate actions based on what utility function it will have in the future, after the action happens.</p>
<p>If you're familiar with how wireheading or quantum suicide look in terms of decision theory, this method of deciding based on future utility functions might seem risky. Fortunately, value learning doesn't permit wireheading in the traditional sense, because the updates to the utility function are an abstract process, not a physical one. The agent's probability distribution over utility functions, which is conditional on interaction histories, defines which actions and observations are allowed to change the utility function during the process of predicting expected utility.</p>
<p>Dewey also mentions that so long as the probability distribution over utility functions is well-behaved, you cannot deliberately take action to raise the probability of one of the utility functions being true. But I think this is only useful to safety when we understand and trust the overarching utility function that gets evaluated at the future time horizon. If instead we start at the present, and specify a starting utility function and rules for updating it based on observations, this complex system can evolve in surprising directions, including some wireheading-esque behavior.</p>
<p> </p>
<p>The formalism of Dewey 2011 is, at bottom, extremely simple. I'm going to be a bad pedagogue here: I think this might only make sense if you go look at equations 2 and 3 in the paper, and figure out what all the terms do, and see how similar they are. The cheap summary is that if your utility is a function of the interaction history, trying to change utility functions based on interaction history just gives you back a utility function. If we try to think about what sort of process to use to change an agent's utility function, this formalism provides only one tool: look out to some future time horizon, and define an effective utility function in terms of what utility functions are possible at that future time horizon. This is different from the approximations or local utility functions we would like in practice.</p>
<p>If we take this scheme and try to approximate it, for example by only looking N steps into the future, we run into problems; the agent will want to self-modify so that next timestep it only looks ahead N-1 steps, and then N-2 steps, and so on. Or more generally, many simple approximation schemes are "sticky" - from inside the approximation, an approximation that changes over time looks like undesirable value drift.</p>
<p>Common sense says this sort of self-sabotage should be eliminable. One should be able to really care about the underlying utility function, not just its approximation. However, this problem tends to crop up, for example whenever the part of the future you look at does not depend on which action you are considering; modifying to keep looking at the same part of the future unsurprisingly improve the results you get in that part of the future. If we want to build a paperclip maximizer, it shouldn't be necessary to figure out every single way to self-modify and penalize them appropriately.</p>
<p>We might evade this particular problem using some other method of approximation that does something more like reasoning about actions than reasoning about futures. The reasoning doesn't have to be logically impeccable - we might imagine an agent that identifies a small number of salient consequences of each action, and chooses based on those. But it seems difficult to show how such an agent would have good properties. This is something I'm definitely interested in.</p>
<p> </p>
<p><img style="float: right; border: 5px solid black; margin-left: 10px; margin-right: 10px; margin-top: 5px; margin-bottom: 5px;" src="https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSV5MeBvf8eZBW02PMzo92oybKP4xMaRFsDC4W8LCpy7vTkxIPP" alt="Handwritten 9" width="201" height="212" />One way to try to make things concrete is to pick a local utility function and specify rules for changing it. For example, suppose we wanted an AI to flag all the 9s in the MNIST dataset. We define a single-time-step utility function by a neural network that takes in the image and the decision of whether to flag or not, and returns a number between -1 and 1. This neural network is deterministically trained for each time step on all previous examples, trying to assign 1 to correct flaggings and -1 to mistakes. Remember, this neural net is just a local utility function - we can make a variety of AI designs involving it. The goal of this exercise is to design an AI that seems liable to make good decisions in order to flag lots of 9s.</p>
<p>The simplest example is the greedy agent - it just does whatever has a high score right now. This is pretty straightforward, and doesn't wirehead (unless the scoring function somehow encodes wireheading), but it doesn't actually do any planning - 100% of the smarts have to be in the local evaluation, which is really difficult to make work well. This approach seems unlikely to extend well to messy environments.</p>
<p>Since Go-playing AI is topical right now, I shall digress. Successful Go programs can't get by with only smart evaluations of the current state of the board, they need to look ahead to future states. But they also can't look all the way until the ultimate time horizon, so they only look a moderate way into the future, and evaluate that future state of the board using a complicated method that tries to capture things important to planning. In sufficiently clever and self-aware agents, this approximation would cause self-sabotage to pop up. Even if the Go-playing AI couldn't modify itself to only care about the current way it computes values of actions, it might make suboptimal moves that limit its future options, because its future self will compute values of actions the 'wrong' way.</p>
<p>If we wanted to flag 9s using a Dewian value learner, we might score actions according to how good they will be according to the projected utility function at some future time step. If this is done straightforwardly, there's a wireheading risk - the changes to its utility function are supplied by humans who might be influenced by actions. I find it useful to apply a sort of "magic button" test - if the AI had a magic button that could rewrite human brains, would it pressing that button have positive expected utility for it? If yes, then this design has problems, even though in our current thought experiment it's just flagging pictures.</p>
<p>To eliminate wireheading, the value learner can use a model of the future inputs and outputs and the probability of different value updates given various inputs and outputs, which doesn't model ways that actions could influence the utility updates. This model doesn't have to be right, it just has to exist. On one hand, this seems like a sort of weird doublethink, to judge based on a counterfactual where your actions don't have impacts you could otherwise expect. On the other hand, it also bears some resemblance to how we actually reason about moral information. Regardless, this agent will now not wirehead, and will want to get good results by learning about the world, if only in the very narrow sense of wanting to play unscored rounds that update its value function. If its value function and value updating made better use of unlabeled data, it would also want to learn about the world in the broader sense.</p>
<p> </p>
<p>Overall I am somewhat frustrated, because value learners have these nice properties, but are computationally unrealistic and do not play well with approximation. One can try to get the nice properties elsewhere, such as relying on an action-suggester to not suggest wireheading, but it would be nice to be able to talk about this as an approximation to something fancier.</p>manfredGgXpryqWGePMMmYxf2016-02-03T09:50:25.165ZCommunicating concepts in value learning
https://lw2.issarice.com/posts/iyQ7kTjxXR3SmjpPs/communicating-concepts-in-value-learning
<p><span style="font-size: 8pt;">Epistemic status: Trying to air out some thoughts for feedback, we'll see how successfully. May require some machine learning to make sense, and may require my level of ignorance to seem interesting.</span></p>
<p> </p>
<p>Many current proposals for value learning are garden-variety <a href="https://en.wikipedia.org/wiki/Regression_analysis">regression</a> (or its close cousin, <a href="https://en.wikipedia.org/wiki/Statistical_classification">classification</a>). The agent doing the learning starts out with some model for what human values look like (a utility function over states of the world, or a reward function in a Markov decision process, or an expected utility function over possible actions), and receives training data that tells it the right thing to do in a lot of different situations. And so the agent finds the parameters of the model that minimize some loss function with the data, and Learns Human Values.</p>
<p>All these models of "the right thing to do" I mentioned are called parametric models, because they have some finite template that they update based on the data. Non-parametric models, on the other hand, have to keep a record of the data they've seen - prediction with a non-parametric model often looks like taking some weighted average of nearby known examples (though not always), while a parametric model would (often) fit some curve to the data and predict using that. But we'll get back to this later.</p>
<p>An obvious problem with current proposals is that it's very resource-intensive to communicate a category or concept to the agent. An AI might be able to automatically learn a lot about the world, but if we want to define its preferences, we have to somehow pick out the concept of "good stuff" within the representation of the world learned by the AI. Current proposals for this look like supervised learning, where huge amounts of labeled data are needed to specify "good stuff," and for many proposals I'm concerned that we'll actually end up specifying "stuff that humans can be convinced is good," which is not at all the same. Humans are much better learners than these supervised learning systems - they learn from fewer examples, and have a better grasp of the meaning and structure behind examples. This hints that there are some big improvements to be made in value learning.</p>
<p>This comparison to humans also leads to my vaguer concerns. It seems like the labeled examples are too crucial, and the unlabeled data not crucial enough. We want a value learner to understand concepts based on just a few examples so long as it has unlabeled data to fill in the gaps, and be able to learn more about morality from observation as a core competency, not as a pale shadow of its learning from labeled data. It seems like fine-tuning the model for the labeled data with stochastic gradient descent is missing something important.</p>
<p>To digress slightly, there are additional problems (e.g. <a href="https://intelligence.org/files/Corrigibility.pdf">corrigibility</a>) once you build an agent that has an output channel instead of merely sponging up information, and these problems are harder if we want value learning from observation. If we want a value learning agent that could learn a simplified version of human morality, and then use that to learn the full version, we might need something like the Bayesian guarantee of <a href="http://www.danieldewey.net/learning-what-to-value.pdf">Dewey 2011</a>, or a functional analogue thereof.</p>
<p> </p>
<p>One inspiration for alternative learning schemes might be <a href="https://en.wikipedia.org/wiki/Cluster_analysis">clustering</a>. As a toy example, imagine finding literal <a href="/lw/nl/the_cluster_structure_of_thingspace/">clusters in thing-space</a> by <a href="http://practicalcryptography.com/media/miscellaneous/files/k_mean_send.gif">k-means clustering</a>. If you want to specify a cluster, you can do something like pick a small sample of examples and force them to be in the same cluster, and allow the number of clusters you try to find in the data to vary so that the statistics of the mandatory cluster are not very different from any other's. The huge problem here is that the idea of "thing-space" elides the difficulty of learning a representation of the world (or equivalently, elides how really, really complicated the cluster boundaries are in terms of observations).</p>
<p>Because learning how to understand the world already requires you to be really good at learning things, it's not obvious to me what identifying and using clusters in the data will entail. One might imagine that if we modeled the world using a big pile of autoencoders, this pile would already contain predictors for many concepts we might want to specify, but that if we use examples to try and communicate a concept that was not already learned, the pile might not even contain the features that make our concept easy to specify. Further speculation in this vein is fun, but is likely pointless at my current level of understanding. So even though learning well from unlabeled data is an important desideratum, I'm including this digression on clustering because I think it's interesting, not because I've shed much light.</p>
<p> </p>
<p>Okay, returning to the parametric/non-parametric thing. The problem of being bad at learning from unlabeled data shows up in diverse proposals like inverse reinforcement learning and <a href="https://intelligence.org/files/UnintendedBehaviors.pdf">Hibbard 2012</a>'s two-part example. And in these cases it's not due to the learning algorithm per se, but for the simple reason that at some point the representation of the world is treated as fixed - the value learner is assumed to understand the world, and then proceeds to learn or be told human values in terms of that understanding. If you can no longer update your understanding of the world, naturally this causes problems with learning from observation.</p>
<p>We should instead design agents that are able to keep learning about the world. And this brings us back to the idea of communicating concepts via examples. The most reasonable way to update learned concepts in light of new information seems to be to just store the examples and re-apply them to the new understanding. This would be a non-parametric model of learned concepts.</p>
<p>What concepts to learn and how to use them to make decisions is not at all known to me, but as a placeholder we might consider the task of learning to identify "good actions," given proposed actions and some input about the world (similar to the "Learning from examples" section of Christiano's <a href="https://medium.com/ai-control/model-free-decisions-6e6609f5d99e#.a90hj6a2o">Approval Directed Agents</a>).</p>manfrediyQ7kTjxXR3SmjpPs2015-12-14T03:06:36.193ZMeetup : Urbana-Champaign: Quorum for discourse
https://lw2.issarice.com/posts/SHBhwHBn6gHSFa4PD/meetup-urbana-champaign-quorum-for-discourse
<h2>Discussion article for the meetup : <a href='/meetups/1gx'>Urbana-Champaign: Quorum for discourse</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">06 September 2015 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">Altgeld Hall, W. Green Street, Urbana, IL, 61801</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Another year, another chance to come to a LW meetup. Find us at the scenic north entrance of Altgeld Hall. I'll bring delicious food.
Depending on what people want, discussion may vary in technical level. Math is good, but sometimes fun digressions are fine too.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/1gx'>Urbana-Champaign: Quorum for discourse</a></h2>manfredSHBhwHBn6gHSFa4PD2015-08-24T11:36:21.602ZMoral AI: Options
https://lw2.issarice.com/posts/hHYNwDxLwCYFL9Evj/moral-ai-options
<p>Epistemic status: One part quotes (informative, accurate), one part speculation (not so accurate).</p>
<p>One avenue towards AI safety is the construction of "moral AI" that is good at solving the problem of human preferences and values. Five FLI grants have recently been funded that pursue different lines of research on this problem.</p>
<p>The projects, in alphabetical order:</p>
<ul>
<li><a href="http://futureoflife.org/AI/2015awardees#Conitzer">Vincent Conitzer</a>:</li>
</ul>
<blockquote>
<p><span>Most contemporary AI systems base their decisions solely on consequences, whereas humans also consider other morally relevant factors, including rights (such as privacy), roles (such as in families), past actions (such as promises), motives and intentions, and so on. Our goal is to build these additional morally relevant features into an AI system. We will identify morally relevant features by reviewing theories in moral philosophy, conducting surveys in moral psychology, and using machine learning to locate factors that affect human moral judgments. We will use and extend game theory and social choice theory to determine how to make these features more precise, how to weigh conflicting features against each other, and how to build these features into an AI system. We hope that eventually this work will lead to highly advanced AI systems that are capable of making moral judgments and acting on them.</span></p>
</blockquote>
<p><span>Techniques: Top-down design, game theory, moral philosophy</span></p>
<ul>
<li><a href="http://futureoflife.org/AI/2015awardees#Evans">Owain Evans</a>:</li>
</ul>
<blockquote>
<p>Previous work in economics and AI has developed mathematical models of preferences, along with algorithms for inferring preferences from observed actions. [Citation of inverse reinforcement learning] We would like to use such algorithms to enable AI systems to learn human preferences from observed actions. However, these algorithms typically assume that agents take actions that maximize expected utility given their preferences. This assumption of optimality is false for humans in real-world domains. Optimal sequential planning is intractable in complex environments and humans perform very rough approximations. Humans often don't know the causal structure of their environment (in contrast to MDP models). Humans are also subject to dynamic inconsistencies, as observed in procrastination, addiction and in impulsive behavior. Our project seeks to develop algorithms that learn human preferences from data despite the suboptimality of humans and the behavioral biases that influence human choice. We will test our algorithms on real-world data and compare their inferences to people's own judgments about their preferences. We will also investigate the theoretical question of whether this approach could enable an AI to learn the entirety of human values.</p>
</blockquote>
<p>Techniques: Trying to find something better than inverse reinforcement learning, supervised learning from preference judgments</p>
<ul>
<li><a href="http://futureoflife.org/AI/2015awardees#Rossi">Francesca Rossi</a>:</li>
</ul>
<blockquote>
<p>The future will see autonomous agents acting in the same environment as humans, in areas as diverse as driving, assistive technology, and health care. In this scenario, collective decision making will be the norm. We will study the embedding of safety constraints, moral values, and ethical principles in agents, within the context of hybrid human/agents collective decision making. We will do that by adapting current logic-based modelling and reasoning frameworks, such as soft constraints, CP-nets, and constraint-based scheduling under uncertainty. For ethical principles, we will use constraints specifying the basic ethical ``laws'', plus sophisticated prioritised and possibly context-dependent constraints over possible actions, equipped with a conflict resolution engine. To avoid reckless behavior in the face of uncertainty, we will bound the risk of violating these ethical laws. We will also replace preference aggregation with an appropriately developed constraint/value/ethics/preference fusion, an operation designed to ensure that agents' preferences are consistent with the system's safety constraints, the agents' moral values, and the ethical principles of both individual agents and the collective decision making system. We will also develop approaches to learn ethical principles for artificial intelligent agents, as well as predict possible ethical violations.</p>
</blockquote>
<p>Techniques: Top-down design, obeying ethical principles/laws, learning ethical principles</p>
<ul>
<li><a href="http://futureoflife.org/AI/2015awardees#Russell">Stuart Russell</a>:</li>
</ul>
<blockquote>
<p>The objectives of the proposed research are (1) to create a mathematical framework in which fundamental questions of value alignment can be investigated; (2) to develop and experiment with methods for aligning the values of a machine (whether explicitly or implicitly represented) with those of humans; (3) to understand the relationships among the degree of value alignment, the decision-making capability of the machine, and the potential loss to the human; and (4) to understand in particular the implications of the computational limitations of humans and machines for value alignment. The core of our technical approach will be a cooperative, game-theoretic extension of inverse reinforcement learning, allowing for the different action spaces of humans and machines and the varying motivations of humans; the concepts of rational metareasoning and bounded optimality will inform our investigation of the effects of computational limitations.</p>
</blockquote>
<p>Techniques: Trying to find something better than inverse reinforcement learning (differently this time), creating a mathematical framework, whatever rational metareasoning is</p>
<ul>
<li><a href="http://futureoflife.org/AI/2015awardees#Sotala">Kaj Sotala</a>:</li>
</ul>
<blockquote>
<p>Autonomous AI systems will need to understand human values in order to respect them. This requires having similar concepts as humans do. We will research whether AI systems can be made to learn their concepts in the same way as humans learn theirs. Both human concepts and the representations of deep learning models seem to involve a hierarchical structure, among other similarities. For this reason, we will attempt to apply existing deep learning methodologies for learning what we call moral concepts, concepts through which moral values are defined. In addition, we will investigate the extent to which reinforcement learning affects the development of our concepts and values.</p>
</blockquote>
<p>Techniques: Trying to identify learned moral concepts, unsupervised learning </p>
<p> </p>
<p>The elephant in the room is that making judgments that always respect human preferences is nearly FAI-complete. Application of human ethics is dependent on human preferences in general, which are dependent on a model of the world and how actions impact it. Calling an action ethical also can also depend on the space of possible actions, requiring a good judgment-maker to be capable of search for good actions. Any "moral AI" we build with our current understanding is going to have to be limited and/or unsatisfactory.</p>
<p>Limitations might be things like judging which of two actions is "more correct" rather than finding correct actions, only taking input in terms of one paragraph-worth of words, or only producing good outputs for situations similar to some combination of trained situations.</p>
<p>Two of the proposals are centered on top-down construction of a system for making ethical judgments. Designing a system by hand, it's nigh-impossible to capture the subtleties of human values. Relatedly, it seems weak at generalization to novel situations, unless the specific sort of generalization has been forseen and covered. The good points of a top down approach are that it can capture things that are important, but are only a small part of the description, or are not easily identified by statistical properties. A top-down model of ethics might be used as a fail-safe, sometimes noticing when something undesirable is happening, or as a starting point for a richer learned model of human preferences.</p>
<p>Other proposals are inspired by inverse reinforcement learning. Inverse reinforcement learning seems like the sort of thing we want - it observes actions and infers preferences - but it's very limited. The problem of having to know a very good model of the world in order to be good at human preferences rears its head here. There are also likely unforseen technical problems in ensuring that the thing it learns is actually human preferences (rather than human foibles, or irrelevant patterns) - though this is, in part, why this research should be carried out now.</p>
<p>Some proposals want to take advantage of learning using neural networks, trained on peoples' actions or judgments. This sort of approach is very good at discovering patterns, but not so good at treating patterns as a consequence of underlying structure. Such a learner might be useful as a heuristic, or as a way to fill in a more complicated, specialized architecture. For this approach like the others, it seems important to make the most progress toward learning human values in a way that doesn't require a very good model of the world.</p>manfredhHYNwDxLwCYFL9Evj2015-07-11T21:46:45.327ZLimited agents need approximate induction
https://lw2.issarice.com/posts/8n4CF4u3zxeR2tFQN/limited-agents-need-approximate-induction
<p>[This post borders on some well-trodden ground in information theory and machine learning, so ideas in this post have an above-average chance of having already been stated elsewhere, by professionals, better. EDIT: As it turns out, this is largely the case, under the subjects of the justifications for MML prediction and Kolmogorov-simple PAC-learning.]</p>
<p><strong>I: Introduction</strong></p>
<p>I am fascinated by methods of thinking that work for well-understood reasons - that follow the steps of a mathematically elegant dance. If one has infinite computing power the method of choice is something like <a href="http://www.vetta.org/documents/legg-1996-solomonoff-induction.pdf">Solomonoff induction</a>, which is provably ideal in a certain way at predicting the world. But if you have limited computing power, the choreography is harder to find.</p>
<p>To do Solomonoff induction, you search through all Turing machine hypotheses to find the ones that exactly output your data so far, then use the weighted average of those perfect retrodictors to predict the next time step. So the naivest way to build an ideal limited agent is to merely search through lots of hypotheses (chosen from some simple set) rather than all of them, and only run each Turing machine for time less than some limit. At least it's guaranteed to work in the limit of large computing power, which ain't nothing.</p>
<p>Suppose then that we take this nice elegant algorithm for a general predictor, and we implement it on today's largest supercomputer, and we show it the stock market prices from the last 50 years to try to predict stocks and get very rich. What happens?</p>
<p>Bupkis happens, that's what. Our Solomonoff predictor tries a whole lot of Turing machines and then runs out of time before finding any useful hypotheses that can perfectly replicate 50 years of stock prices. This is because such useful hypotheses are very, very, very rare.</p>
<p>We might then turn to the burgeoning field of logical uncertainty, which has a major goal of handling intractable math problems in an elegant and timely manner. We are logically uncertain about what distribution Solomonoff induction will output, so can we just average over that logical uncertainty to get some expected stock prices?</p>
<p>The trouble with this is that current logical uncertainty methods rely on proofs that certain outputs are impossible or contradictory. For simple questions this can narrow down the answers, but for complicated problems it becomes intractable, replacing the hard problem of evaluating lots of Turing machines with the hard problem of searching through lots and lots of proofs about lots of Turing machines - and so again our predictor runs out of time before becoming useful.</p>
<p>In practice, the methods we've found to work don't look very much like Solomonoff induction. Successful methods don't take the data as-is, but instead throw some of it away: curve fitting and smoothing data, filtering out hard-to-understand signals as noise, and using predictive models that approximate reality imperfectly. The sorts of things that people trying to predict stocks are already doing. These methods are vital to improve computational tractability, but are difficult (to my knowledge) to fit into a framework as general as Solomonoff induction.</p>
<p><strong>II: Rambling</strong></p>
<p>Suppose that our AI builds a lot of models of the world, including lossy models. How should it decide which models are best to use for predicting the world? Ideally we'd like to make a tradeoff between the accuracy of the model, measured in the expected utility of how accurate you expect the model's predictions to be, and the cost of the time and energy used to make the prediction.</p>
<p>Once we know how to tell good models, the last piece would be for our agent to make the explore/exploit tradeoff between searching for better models and using its current best.</p>
<p>There are various techniques to estimate resource usage, but how does one estimate accuracy?</p>
<p>Here was my first thought: If you know how much information you're losing (e.g. by binning data), for discrete distributions this sets the Shannon information of the ideal value (given by Solomonoff prediction) given the predicted value. This uses the relationship between information in bits of data and Shannon information that determines how sharp your probability distribution is allowed to be.</p>
<p>But with no guarantees about the normality (or similar niceness properties) of the ideal value given the prediction, this isn't very helpful. The problem is highlighted by hurricane prediction. If hurricanes behaved nicely as we threw away information, weather models would just be small, high-entropy deviations from reality. Instead, hurricanes can change route greatly even with small differences in initial conditions.</p>
<p>The failure of the above approach can be explained in a very general way: it uses too little information about the model and the data, only the amount of information thrown away. To do better, our agent has to learn a lot from its training data - a subject that workers in AI have already been hard at work on. On the one hand, it's a great sign if we can eventually connect ideal agents to current successful algorithms. On the other, doing so elegantly seems like a hard problem.</p>
<p>To sum up in the blandest possible way: If we want to build successful predictors of the future with limited resources, they should use their experience to learn approximate models of the world.</p>
<p>The real trick, though, is going to be to set this on a solid foundation. What makes a successful method of picking models? As we lack access to the future (yet! Growth mindset!), we can't grade models based on their future predictions unless we descend to solipsism and grade models against models. Thus we're left with grading models based on how well they retrodict the data so far. Sound familiar? The foundation we want seems like an analogue to Solomonoff induction, one that works for known reasons but doesn't require perfection.</p>
<p><strong>III: An Example</strong></p>
<p>Here's a paradigm that might or might not be a step in the right direction, but at least gestures at what I mean.</p>
<p>The first piece of the puzzle is that a model that gets proportion P of training bits wrong can be converted to a Solomonoff-accepted perfectly-precise model just by specifying the bits it gets wrong. Suppose we break the model output (with total length N) into chunks of size L, and prefix each chunk with the locations of the wrong bits in that chunk. Then the extra data required to rectify an approximate model is at most N/L·log(P·L)+N·P·log(L). Then the hypothesis where the model is right about the next bit is simpler than the hypothesis when it's wrong, because when the model is right you don't have to spend ~log(L) bits correcting it.</p>
<p>In this way, Solomonoff induction natively cares about some approximate models' predictions. There are some interesting details here that are outside the focus of this particular post. Does using the optimal chunk length lead to Solomonoff induction reflecting model accuracy correctly? What are some better schemes for rectifying models that handle things like models that output probabilities? The point is just that even if your model is wrong on fraction P of the training data, Solomonoff induction will still promote it as long as it's simpler than N-N/L·log(P·L)-N·P·log(L).</p>
<p>The second piece of the puzzle is that induction can be done over processed functions of observations, like smoothing the data or filtering difficult-to-predict parts (noise) out. If this processing increases the accuracy of models, we can use this to make high-accuracy models of functions the training data, and then use those models to predict the the processed future observations as above.</p>
<p>These two pieces allow an agent to use approximate models, and to throw away some of its information, and still have its predictions work for the same reason as Solomonoff induction. We can use this paradigm to interpret what an algorithm like curve fitting is doing - the fitted curve is a high-accuracy retrodiction of some smoothed function of the data, which therefore does a good job of predicting what that smoothed function will be in the future.</p>
<p>There are some issues here. If a model that you are using is not the simplest, it might have overfitting problems (though perhaps you can fix this just by throwing away more information than naively appears necessary) or systematic bias. More generally, we haven't explored how models get chosen; we've made the problem easier to brute force but we need to understand non-brute force search methods and what their foundations are. It's a useful habit to keep in mind what actually works for humans - as someone put it to me recently, "humans can make models they understand that work for reasons they understand."</p>
<p>Furthermore, this doesn't seem to capture reductionism well. If our agent learns some laws of physics and then is faced with a big complicated situation it needs to use a simplified model to make a prediction about, it should still in some sense "believe in the laws of physics," and not believe that this complicated situation violates the laws physics even if its current best model is independent of physics.</p>
<p><strong>IV: Logical Uncertainty</strong></p>
<p><strong></strong>It may be possible to relate this back to logical uncertainty - where by "this" I mean the general thesis of predicting the future by building models that are allowed to be imperfect, not the specific example in part III. <a href="http://intelligence.org/files/QuestionsLogicalUncertainty.pdf">Soares and Fallenstein</a> use the example of a complex Rube Goldberg machine that deposits a ball into one of several chutes. Given the design of the machine and the laws of physics, suppose that one can in principle predict the output of this machine, but that the problem is much too hard for our computer to do. So rather than having a deterministic method that outputs the right answer, a "logical uncertainty method" in this problem is one that, with a reasonable amount of resources spent, takes in the description of the machine and the laws of physics, and gives a probability distribution over the machine's outputs.</p>
<p>Meanwhile, suppose that we take an approximately inductive predictor and somehow teach it the the laws of physics, then ask it to predict the machine. We'd like it to make predictions via some appropriately simplified folk model of physics. If this model gives a probability distribution over outcomes - like in the simple case of "if you flip this coin in this exact way, it has a 50% shot at landing heads" - doesn't that make it a logical uncertainty method? But note that the probability distribution returned by a single model is not actually the uncertainty introduced by replacing an ideal predictor with a resource-limited predictor. So any measurement of logical uncertainty has to factor in the uncertainty between models, not just the uncertainty within models.</p>
<p>Again, we're back to looking for some prediction method that weights models with some goodness metric more forgiving than just using perfectly-retrodicting Turing machines, and which outputs a probability distribution that includes model uncertainty. But can we apply this to mathematical questions, and not just Rube Goldberg machines? Is there some way to subtract away the machine and leave the math?</p>
<p>Suppose that our approximate predictor was fed math problems and solutions, and built simple, tractable programs to explain its observations. For easy math problems a successful model can just be a Turing machine that finds the right answer. As the math problems get more intractable, successful models will start to become logical uncertainty methods, like how we can't predict a large prime number exactly, but we can predict it's last digit is 1, 3, 7, or 9. Within this realm we have something like low-level reductionism, where even though we can't find a proof of the right answer, we still want to act as if mathematical proofs work and all else is ignorance, and this will help us make successful predictions.</p>
<p>Then we have complicated problems that seem to be beyond this realm, like P=NP. Humans certainly seem to have generated some strong opinions about P=NP without dependence on mathematical proofs narrowing down the options. It seems to such humans that the genuinely right procedure to follow is that, since we've searched long and hard for a fast algorithm for NP-complete problems without success, we should update in the direction that no such algorithm exists. In approximate-Solomonoff-speak, it's that P!=NP is consistent with a simple, tractable explanation for (a recognizable subset of) our observations, while P=NP is only consistent with more complicated tractable explanations. We could absolutely make a predictor that reasons this way - it just sets a few degrees of freedom. But is it the right way to reason?</p>
<p>For one thing, this seems like it's following Gaifman's proposed property of logical uncertainty, that seeing enough examples of something should convince you of it with probability 1 - which <a href="https://intelligence.org/2014/12/16/new-report-computable-probability-distributions-converge/">has been shown</a> to be "too strong" in some sense (it assigns probability 0 to some true statements - though even this could be okay if those statements are infinitely dilute). Does the most straightforward implementation actually have the Gaifman condition, or not? (I'm sorry, ma'am. Your daughter has... the Gaifman condition.)</p>
<p>This inductive view of logical uncertainty lacks the consistent nature of many other approaches - if it works, it does so by changing approaches to suit the problem at hand. This is bad if you want your logical uncertainty methods to be based on a simple prior followed by some kind of updating procedure. But logical uncertainty is supposed to be practical, after all, and at least this is a simple meta-procedure.</p>
<p><strong>V: Questions</strong></p>
<p>Thanks for reading this post. In conclusion, here are some of my questions:</p>
<p>What's the role of Solomonoff induction in approximate induction? Is Solomonoff induction doing all of the work, or is it possible to make useful predictions using tractable hypotheses Solomonoff induction would exclude, or excluding intractable hypotheses Solomonoff induction would have to include?</p>
<p>Somehow we have to pick out models to promote to attention in the first place. What properties make a process for this good or bad? What methods for picking models can be shown to still lead to making useful predictions - and not merely in the limit of lots of computing time?</p>
<p>Are humans doing the right thing by making models they understand that work for reasons they understand? What's up with that reductionism problem anyhow?</p>
<p>Is it possible to formalize the predictor discussed in the context of logical uncertainty? Does it have to fulfill Gaifman's condition if it finds patterns in things like P!=NP?</p>manfred8n4CF4u3zxeR2tFQN2015-04-24T07:42:15.901ZSelfish preferences and self-modification
https://lw2.issarice.com/posts/zgbZNwW7f3C89ZgGK/selfish-preferences-and-self-modification
<p>One question I've had recently is "Are agents acting on selfish preferences doomed to having conflicts with other versions of themselves?" A major motivation of TDT and UDT was the ability to just do the right thing without having to be tied up with precommitments made by your past self - and to trust that your future self would just do the right thing, without you having to tie them up with precommitments. Is this an impossible dream in anthropic problems?</p>
<p> </p>
<p>In my <a href="/lw/lg2/treating_anthropic_selfish_preferences_as_an/">recent post</a>, I talked about preferences where "if you are one of two copies and I give the other copy a candy bar, your selfish desires for eating candy are unfulfilled." If you would buy a candy bar for a dollar but not buy your copy a candy bar, this is exactly a case of strategy ranking depending on indexical information.</p>
<p>This dependence on indexical information is inequivalent with UDT, and thus incompatible with peace and harmony.</p>
<p> </p>
<p>To be thorough, consider an experiment where I am forked into two copies, A and B. Both have a button in front of them, and 10 candies in their account. If A presses the button, it deducts 1 candy from A. But if B presses the button, it removes 1 candy from B and gives 5 candies to A.</p>
<p>Before the experiment begins, I want my descendants to press the button 10 times (assuming candies come in units such that my utility is linear). In fact, after the copies wake up but before they know which is which, they want to press the button!</p>
<p>The model of selfish preferences that is not UDT-compatible looks like this: once A and B know who is who, A wants B to press the button but B doesn't want to do it. And so earlier, I should try and make precommitments to force B to press the button.</p>
<p>But suppose that we simply decided to use a different model. A model of peace and harmony and, like, free love, where I just maximize the average (or total, if we specify an arbitrary zero point) amount of utility that myselves have. And so B just presses the button.</p>
<p>(It's like non-UDT selfish copies can make all Pareto improvements, but not all average improvements)</p>
<p> </p>
<p>Is the peace-and-love model still a selfish preference? It sure seems different from the every-copy-for-themself algorithm. But on the other hand, I'm <a href="/lw/kwd/rationality_quotes_september_2014/bdv2">doing it for myself</a>, in a sense.</p>
<p>And at least this way I don't have to waste time with precomittment. In fact, self-modifying to this form of preferences is such an effective action that conflicting preferences are self-destructive. If I have selfish preferences now but I want my copies to cooperate in the future, I'll try to become an agent who values copies of myself - so long as they date from after the time of my self-modification.</p>
<p> </p>
<p>If you recall, I made an argument in favor of averaging the utility of future causal descendants when calculating expected utility, based on this being the fixed point of selfish preferences under modification when confronted with Jan's tropical paradise. But if selfish preferences are unstable under self-modification in a more intrinsic way, this rather goes out the window.</p>
<p> </p>
<p>Right now I think of selfish values as a somewhat anything-goes space occupied by non-self-modified agents like me and you. But it feels uncertain. On the mutant third hand, what sort of arguments would convince me that the peace-and-love model actually captures my selfish preferences?</p>manfredzgbZNwW7f3C89ZgGK2015-01-14T08:42:51.489ZTreating anthropic selfish preferences as an extension of TDT
https://lw2.issarice.com/posts/gTmWZEu3CcEQ6fLLM/treating-anthropic-selfish-preferences-as-an-extension-of
<p><strong>I</strong></p>
<p>When preferences are selfless, anthropic problems are easily solved by a change of perspective. For example, if we do a Sleeping Beauty experiment for charity, all Sleeping Beauty has to do is follow the strategy that, from the charity's perspective, gets them the most money. This turns out to be an easy problem to solve, because the answer doesn't depend on Sleeping Beauty's subjective perception.</p>
<p>But selfish preferences - like being at a comfortable temperature, eating a candy bar, or going skydiving - are trickier, because they do rely on the agent's subjective experience. This trickiness really shines through when there are actions that can change the number of copies. For recent posts about these sorts of situations, see <a href="/lw/l18/simulation_argument_meets_decision_theory/">Pallas' sim game</a> and <a href="/lw/l3w/baysian_conundrum/">Jan_Ryzmkowski's tropical paradise</a>. I'm going to propose a model that makes answering these sorts of questions almost as easy as playing for charity.</p>
<p>To quote Jan's problem:</p>
<blockquote>
<p style="margin: 0px 0px 1em; font-family: Arial, Helvetica, sans-serif; line-height: 19.5px; text-align: justify;">It's a cold cold winter. Radiators are hardly working, but it's not why you're sitting so anxiously in your chair. The real reason is that tomorrow is your assigned upload, and you just can't wait to leave your corporality behind. "Oh, I'm so sick of having a body, especially now. I'm freezing!" you think to yourself, "I wish I were already uploaded and could just pop myself off to a tropical island."</p>
<p style="margin: 0px 0px 1em; font-family: Arial, Helvetica, sans-serif; line-height: 19.5px; text-align: justify;">And now it strikes you. It's a weird solution, but it feels so appealing. You make a solemn oath (you'd say one in million chance you'd break it), that soon after upload you will simulate this exact scene a thousand times simultaneously and when the clock strikes 11 AM, you're gonna be transposed to a Hawaiian beach, with a fancy drink in your hand.</p>
<p style="margin: 0px 0px 1em; font-family: Arial, Helvetica, sans-serif; line-height: 19.5px; text-align: justify;">It's 10:59 on the clock. What's the probability that you'd be in a tropical paradise in one minute?</p>
</blockquote>
<div>So question one is the probability question: what's your probability that you go to the tropical paradise? And question two is the decision problem: is this actually a good idea?</div>
<div><br /></div>
<div>The probability question is straightforward, and is indeed about a 1000/1001 chance of tropical paradise. If this does not make sense, feel free to ask about it, or go check out these two rambling complementary posts: <a href="/lw/l86/deriving_probabilities_from_causal_diagrams/">Deriving probabilities from causal diagrams</a>, <a href="/lw/lau/more_marbles_and_sleeping_beauty/">More marbles and Sleeping Beauty</a>.</div>
<div><br /></div>
<div>One might then make an argument about the decision question that goes like this: Before I swore this oath, my probability of going to a tropical island was very low. After, it was very high. Since I really like tropical islands, this is a great idea. In a nutshell, I have increased my expected utility by making this oath.</div>
<div><br /></div>
<div>The counterargument is also simple, though: Making copies of myself has no causal effect on me. Swearing this oath does not move my body to a tropical paradise. What really happens is that I just sit there in the cold just the same, but then later I make some simulations where I lie to myself. This is not a higher-utility universe than the one where I don't swear the oath.</div>
<div><br /></div>
<div>Hopefully you can see how this is confusing.</div>
<div><br /></div>
<div><strong>II</strong></div>
<div><br /></div>
<div>So, my proposal, in short form: You are a person. I mean this not in the abstract, non-causal, sense, where if I make a copy of you and then shoot you, "you live on." I mean that the isolated causal agent reading this is a person capable of selfish desires, where if you are one of two copies and I give the other copy a candy bar, your selfish desires for eating candy are unfulfilled<sup>1</sup>. Choose as if you were controlling the output of your decision algorithm, so that you maximize your expected utility, including selfish desires (if you have them), conditioned on the fact that you exist (I'll come back to what this last bit means in part III).</div>
<div><br /></div>
<div>This is at its heart porting TDT to anthropic problems. When there is a decision your original body can make that creates a bunch of copies, and the copies are also faced with this decision, your decision lets you control whether you are the original or a copy. If you don't want to be a copy, as in Pallas' sim game, you have to take that into account. If you do want to be a copy, you take that into account too.</div>
<div><br /></div>
<div>This leads to biting the bullet in Jan's tropical paradise. It is <em>actually a good idea</em> to take an action that, if you're the original body, creates a bunch of high-selfish-expected-utility copies that also undergo the decision you're making right now, because this decision controls whether you're one of those copies.</div>
<div><br /></div>
<div>There is an important caveat: this only holds if you truly would like to be one of those copies. To repeat the short form, this decision algorithm assumes that you are a person trying to increase their own expected utility. These copies can't just be made and disposed of to manipulate your subjective experience - something which is possible, but is a bad idea. The copies have to be people who you would actually like to be, who go on to live long, fulfilling lives. This is not about gaming the system. It's just an extension of TDT to anthropic situations.</div>
<div><br /></div>
<div>Interestingly, this usually gives the same results as playing for charity. Thus there's a sort of locality of money, where you make similar tradeoffs between selfish spending and charitable spending no matter how many copies of you there are.</div>
<div><br /></div>
<div>To deliberately construct an intuitive case, imagine that you are already uploaded, and you're led into a room (a simulated room, of course) where Omega is waiting for you. Omega says hello, and asks you whether you think you're the original or a copy. "Huh? Have I been copied?" you say. Excellent, Omega says. It then presents two boxes to you, box A and box B. Box A always has some okay candy in it for you to eat (eating candy is my go-to selfish reward). If you pick box B and are the original, it is empty, but you will be copied a million times from a snapshot when you entered the room, and offered the same choice - and if you are a copy, box B contains very delicious candy to eat (and then the copies go on to live similar lives to the original). Again there's the odd property that the output of your decision algorithm controls whether you are likely a copy or not. If you would prefer to be a copy, then you should pick box B.</div>
<div><br /></div>
<div>There's a precommitment problem here. Suppose I value my future selves by a sum of their utilities (given some zero point). Then even if being a copy was not so great (like in Pallas' sim game), I'd precommit to making as many copies as possible. But once the game starts, by my definition of selfish preferences I don't care much about whether the other copies get a selfish reward, and so I might try to fight that precommitment to raise my expected utility.</div>
<div><br /></div>
<div>In fact, these precommitment problems crop up whenever I calculate expected value in any other way than by averaging utility among future copies. This is a statement about a small piece of population ethics, and as such, should be highly suspect - the fact that my preferred model of selfish preferences says anything about even this small subset of population ethics makes me significantly less confident that I'm right. Even though the thing it's saying seems sensible.</div>
<div><br /></div>
<div>Footnote <sup>1</sup>: The reader who has been following my posts may note how this identification of who has the preferences via causality makes selfish preferences well-defined no matter <a href="/lw/lcj/how_many_people_am_i/">how many times I define the pattern "I" to map to my brain</a>, which is good because it makes the process well-defined, but also somewhat difficult because it eliminates the last dependence on a lower level where we can think of anthropic probabilities as determined a priori, rather than depending on a definition of self grounded in decision-making as well as experiencing. On the other hand, with that level conflict gone, maybe there's nothing stopping us from thinking of anthropic probabilities on this more contingent level as "obvious" or "a priori."</div>
<div><br /></div>
<div>
<div><strong>III</strong></div>
<div><br /></div>
<div>It's worth bringing up Eliezer's <a href="/lw/19d/the_anthropic_trilemma/">anthropic trilemma</a> (further thought by Katja Grace <a href="http://meteuphoric.wordpress.com/2011/05/19/on-the-anthropic-trilemma/">here</a>). The idea is to subjectively experience winning the lottery by entering a lottery and then replicating yourself a trillion times, wake up to have the experience, and then merge back together. Thus, the argument goes, as long as probability flows along causal channels, by waking up a trillion times I have captured the subjective experience, and will go on to subjectively experience winning the lottery.</div>
</div>
<div><br /></div>
<div>Again we can ask the two questions: What are the probabilities? And is this actually a good idea?</div>
<div><br /></div>
<div>This is the part where I come back to explain that earlier terminology - why is it important that I specified that you condition on your own existence? When you condition on the fact that you exist, you get an anthropic probability. In the story about Omega I told above, your probability that you're the original before you enter the room is 1. But after you enter the room, if your decision algorithm chooses box B, your probability that you're the original should go down to one in a million. This update is possible because you're updating on new information about where you are in the game - you're conditioning on your own existence.</div>
<div><br /></div>
<div>Note that I did not just say "use anthropic probabilities." When calculating expected utility, you condition on your own existence, but you most certainly do not condition on future selves' existence. After all, you might get hit by a meteor and die, so you don't actually know that you'll be around tomorrow, and you shouldn't condition on things you don't know. Thus the player at russian roulette who says "It's okay, I'll subjectively experience winning!" is making a decision by conditioning on information they do not have.</div>
<div><br /></div>
<div>Katja Grace talks about two principles acting in the Anthropic Trilemma: Follow The Crowd, which sends your subjective experience into the branch with more people, and Blatantly Obvious Principle, which says that your subjective experience should follow causal paths. Katja points out that they do not just cause problems when merging, they also conflict when splitting - so Eliezer is being selective in applying these principles, and there's a deeper problem here. If you recall me mentioning my two-fluid model of anthropics, I partially resolved this by tracking two measures, one that obeyed FTC (subjective probability), and one that obeyed BOP (magic reality fluid).</div>
<div><br /></div>
<div>But the model I'm presenting here dissolves those fluids (or would it be 'dilutes'?) - the thing that follows the crowd is who you think you are, and the blatantly obvious thing is your expectation for the future. There's no subjective experience fluid that it's possible to push around without changing the physical state of the universe. There's just people.</div>
<div><br /></div>
<div>To give the probabilities in the Anthropic Trilemma, it is important to track what information you're conditioning on. If I condition on my existence just after I buy my ticket, my probability that I picked the winning numbers is small, no matter what anthropic hijinks might happen if I win, I still expect to see those hijinks happen with low probability<sup>2</sup>. If I condition on the fact that I wake up after possibly being copied, my probability that I picked the winning numbers is large, as is my probability that I will have picked the winning numbers in the future, even if I get copied or merged or what have you. Then I learn the result, and no longer have a single state of information which would give me a probability distribution. Compare this to the second horn of the trilemma; it's easy to get mixed up when giving probabilities if there's more than one set of probabilities to give.</div>
<div><br /></div>
<div>Okay, so that's the probabilities - but is this actually a good idea? Suppose I'm just in it for the money. So I'm standing there considering whether to buy a ticket, and I condition on my own existence, and the chances of winning still look small, and so I don't buy the ticket. That's it. This is especially clear if I donate my winnings to charity - the only winning move is not to play<sub> the lottery</sub>.</div>
<div><br /></div>
<div>Suppose then instead that I have a selfish desire to experience winning the lottery, independent of the money - does copying myself if I win help fulfill this desire? Or to put this another way, in calculating expected utility we weight the selfish utility of the many winning copies less because winning is unlikely, but do we weight it more because there are more of them?</div>
<div><br /></div>
<div>This question is resolved by (possible warning sign) the almost-population-ethics result above, which says that as an attractor of self-modification we should average copies' utilities rather than summing them, and so copying does not increase expected utility. Again, I find this incompletely convincing, but it does seem to be the extension of TDT here. So this procedure does not bite the bullet in the anthropic trilemma. But remember the behavior in Jan's tropical paradise game? It is in fact possible to design a procedure that lets you satisfy your desire to win the lottery - just have the copies created when you win start from a snapshot of yourself before you bought the lottery ticket.</div>
<div><br /></div>
<div>This is a weird bullet to bite. It's like, how come it's a good idea to create copies that go through the decision to create copies, but only a neutral idea to create copies that don't? After all, winning and then creating simulations has the same low chance no matter what. The difference is entirely anthropic - only when the copies also make the decision does the decision control whether you're a copy.</div>
<div><br /></div>
<div>Footnote <sup>2</sup>: One might complain that if you know what you'll expect in the future, you should update to believing that in the present. But if I'm going to be copied tomorrow, I don't expect to be a copy today.</div>
<div><br /></div>
<div><strong>IV</strong></div>
<div><br /></div>
<div>The problem of the Anthropic Trilemma is not actually gone, because if I'm indifferent to merging with my copies, there is some procedure that better fulfills my selfish desire to experience winning the lottery just by shuffling copies of me around: if I win, make a bunch of copies that start from a snapshot in the past, then merge a the copies together.</div>
<div><br /></div>
<div>So let's talk about the merging. This is going to be the section with the unsolved problem.</div>
<div><br /></div>
<div>Here's what Eliezer's post says about merging:</div>
<div><br /></div>
<blockquote>
<div><span style="font-family: Arial, Helvetica, sans-serif; line-height: 19.5px; text-align: justify;">Just as computer programs or brains can split, they ought to be able to merge. If we imagine a version of the Ebborian species that computes digitally, so that the brains remain synchronized so long as they go on getting the same sensory inputs, then we ought to be able to put two brains back together along the thickness, after dividing them. In the case of computer programs, we should be able to perform an operation where we compare each two bits in the program, and if they are the same, copy them, and if they are different, delete the whole program. (This seems to establish an equal causal dependency of the final program on the two original programs that went into it. E.g., if you test the causal dependency via counterfactuals, then disturbing any bit of the two originals, results in the final program being completely different (namely deleted).)</span></div>
</blockquote>
<div><br /></div>
<div>In general, merging copies is some process where many identical copies go in, and only one comes out. If you know they're almost certainly identical, why bother checking them, then? Why not just delete all but one? It's the same pattern, after all.</div>
<div><br /></div>
<div>Well, imagine that we performed a causal intervention on one of these identical copies - gave them candy or something. Now if we deleted all but one, the effect of our intervention is erased with high probability. In short, if you delete all but one, the person who comes out is not actually the causal descendant of the copies who go in - it's just one of the copies.</div>
<div><br /></div>
<div>Just like how "selfish preferences" means that if I give another of your copies candy, that doesn't fulfill your selfish desire for candy, if another of your copies is the one who gets out of the murder-chamber, that doesn't fulfill your selfish desire to not get murdered. This is why Eliezer talks about going through the process of comparing each copy bit by bit and only merging them if they're identical, so that the person who comes out is the causal descendant of all the people who go in.</div>
<div><br /></div>
<div>On the other hand, Eliezer's process is radically different from how things normally go. If I'm one of several copies, and a causal intervention gives me candy, and no merging shenanigans occur, then my causal descendant is me who's had some candy. If I'm one of several copies, and a causal intervention gives me candy, and then we're merged by Eliezer's method, then my causal descendant is <em>utterly annihilated</em>.</div>
<div><br /></div>
<div>If we allow the character of causal arrows to matter, and not merely their existence, then it's possible that merging is not so neutral after all. But this seems like a preference issue independent of the definition of selfish preferences - although I would have said that about how to weight preferences of multiple copies, too, and I would likely have been wrong.</div>
<div><br /></div>
<div>Does the strange behavior permitted by the neutrality of merging serve as a reductio of that neutrality, or of this extension of selfish preferences to anthropic information, or neither? In the immortal words of Socrates, "... I drank what?" </div>
<div><br /></div>
<div><br /></div>
<div><strong>EDIT:</strong></div>
<div><br /></div>
<div><strong>A Problem:</strong></div>
<div><br /></div>
<div>This decision theory has precommitment issues. In the case of Jan's tropical paradise, I want to precommit to creating satisfied copies from a snapshot of my recent self. But once I'm my future self, I don't want to do it because I know I'm not a copy.</div>
<div><br /></div>
<div><strong>Metaproblems:</strong></div>
<div><br /></div>
<div>This decision theory doesn't have very many knobs to turn - it boils down to "choose the decision-algorithm output that causes maximum expected utility for you, conditioning on both the action and the information you possess." This is somewhat good news, because we don't much want free variables in a decision theory. But it's a metaproblem because it means that there's no obvious knob to turn to eliminate the problem above - creativity is required.</div>
<div><br /></div>
<div>One approach that has worked in the past is to figure out what global variable we want to maximize, and just do UDT to this problem. But this doesn't work for this decision theory - as we expected, because it doesn't seem to work for selfish preferences in general. The selves at two different times in the tropical paradise problem just want to act selfishly - so are they allowed to be in conflict?</div>
<div><br /></div>
<div><strong>Solution Brainstorming (if one is needed at all):</strong></div>
<div><br /></div>
<div>One specific argument might run that when you precommit to creating copies, you decrease your amount of indexical information, and that this is just a form of lying to yourself and is therefore bad. I don't think this works at all, but it may be worth keeping in mind.</div>
<div><br /></div>
<div>A more promising line might be to examine the analogy to evidential decision theory. Evidential decision theory fails when there's a difference between conditioning on the action and conditioning on a causal do(Action). What does the analogue look like for anthropic situations?</div>
<div><br /></div>
<div><strong>EDIT 2:</strong></div>
<div><br /></div>
<div>For somewhat of a resolution, see <a href="/r/discussion/lw/lj3/selfish_preferences_and_selfmodification/">Selfish preferences and self-modification</a>.</div>manfredgTmWZEu3CcEQ6fLLM2015-01-01T00:43:56.587ZHow many people am I?
https://lw2.issarice.com/posts/brLgyCqZaMDjGPKsp/how-many-people-am-i
<p>Strongly related: the <a href="/lw/ps/where_physics_meets_experience/">Ebborians</a></p>
<p>Imagine mapping my brain into two interpenetrating networks. For each brain cell, half of it goes to one map and half to the other. For each connection between cells, half of each connection goes to one map and half to the other. We can call these two mapped out halves Manfred One and Manfred Two. Because neurons are classical, as I think, both of these maps change together. They contain the full pattern of my thoughts. (This situation is even more clear in the Ebborians, who can literally split down the middle.)</p>
<p>So how many people am I? Are Manfred One and Manfred Two both people? Of course, once we have two, why stop there - are there thousands of Manfreds in here, with "me" as only one of them? Put like that it sounds a little overwrought - what's really going on here is the question of what physical system corresponds to "I" in english statements like "I wake up." This may matter.</p>
<p>The impact on anthropic probabilities is somewhat straightforward. With everyday definitions of "I wake up," I wake up just once per day no matter how big my head is. But if the "I" in that sentence is some constant-size physical pattern, then "I wake up" is an event that happens more times if my head is bigger. And so using the variable people-number definition, I expect to wake up with a gigantic head.</p>
<p>The impact on decisions is less big. If I'm in this head with a bunch of other Manfreds, we're all on the same page - it's a <a href="/lw/3dy/solve_psykoshs_nonanthropic_problem/">non-anthropic problem</a> of coordinated decision-making. For example, if I were to make any monetary bets about my head size, and then donate profits to charity, no matter what definition I'm using, I should bet as if my head size didn't affect anthropic probabilities. So to some extent the real point of this effect is that it is a way anthropic probabilities can be ill-defined. On the other hand, what about preferences that depend directly on person-numbers like how to value people with different head sizes? Or for vegetarians, should we care more about cows than chickens, because each cow is more animals than a chicken is?</p>
<p> </p>
<p>According to my common sense, it seems like my body has just one person in it. Why does my common sense think that? I think there are two answers, one unhelpful and one helpful.</p>
<p>The first answer is evolution. Having kids is an action that's independent of what physical system we identify with "I," and so my ancestors never found modeling their bodies as being multiple people useful.</p>
<p>The second answer is causality. Manfred One and Manfred Two are causally distinct from two copies of me in separate bodies but the same input/output. If a difference between the two separated copies arose somehow, (reminiscent of <a href="http://www.lehigh.edu/~mhb0/Dennett-WhereAmI.pdf">Dennett's factual account</a>) henceforth the two bodies would do and say different things and have different brain states. But if some difference arises between Manfred One and Manfred Two, it is erased by diffusion.</p>
<p>Which is to say, the map that is Manfred One is statically the same pattern as my whole brain, but it's causally different. So is "I" the pattern, or is "I" the causal system? </p>
<p>In this sort of situation I am happy to stick with common sense, and thus when I say me, I think the causal system is referring to the causal system. But I'm not very sure.</p>
<p> </p>
<p>Going back to the Ebborians, one interesting thing about that post is the conflict between common sense and common sense - it seems like common sense that each Ebborian is equally much one person, but it also seems like common sense that if you looked at an Ebborian dividing, there doesn't seem to be a moment where the amount of subjective experience should change, and so amount of subjective experience should be proportional to thickness. But as it is said, just because there are two opposing ideas doesn't mean one of them is right.</p>
<p>On the questions of subjective experience raised in that post, I think this mostly gets cleared up by precise description an anthropic narrowness. I'm unsure of the relative sizes of this margin and the proof, but the sketch is to replace a mysterious "subjective experience" that spans copies with individual experiences of people who are using a TDT-like theory to choose so that they individually achieve good outcomes given their existence.</p>manfredbrLgyCqZaMDjGPKsp2014-12-15T18:11:39.098ZMeetup : Urbana-Champaign: Finishing Up
https://lw2.issarice.com/posts/EEagRivGG9DYydJey/meetup-urbana-champaign-finishing-up
<h2>Discussion article for the meetup : <a href='/meetups/17z'>Urbana-Champaign: Finishing Up</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">14 December 2014 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St, Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>This is likely the last meetup of the semester. So let's try and look back on the past and collect some reminders of things that might be useful.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/17z'>Urbana-Champaign: Finishing Up</a></h2>manfredEEagRivGG9DYydJey2014-12-13T08:42:33.897ZMeetup : Urbana-Champaign: Seeking advice
https://lw2.issarice.com/posts/BpYHRoaGrNAJfcMd6/meetup-urbana-champaign-seeking-advice
<h2>Discussion article for the meetup : <a href='/meetups/17e'>Urbana-Champaign: Seeking advice</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">30 November 2014 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St, Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Sometimes, one might even say usually, other people know more than me. This is great, because then they can give me advice.</p>
<p>Let's talk about how to evaluate advice resources, and how to ask, and reinforce how useful this can be.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/17e'>Urbana-Champaign: Seeking advice</a></h2>manfredBpYHRoaGrNAJfcMd62014-11-27T00:58:01.285ZMore marbles and Sleeping Beauty
https://lw2.issarice.com/posts/c7fAEStCafGh8TFdf/more-marbles-and-sleeping-beauty
<p><strong>I</strong></p>
<p>Previously I talked about an entirely uncontroversial marble game: I flip a coin, and if Tails I give you a black marble, if Heads I flip another coin to either give you a white or a black marble.</p>
<p>The probabilities of seeing the two marble colors are 3/4 and 1/4, and the probabilities of Heads and Tails are 1/2 each.</p>
<p>The marble game is analogous to how a 'halfer' would think of the Sleeping Beauty problem - the claim that Sleeping Beauty should assign probability 1/2 to Heads relies on the claim that your information for the Sleeping Beauty problem is the same as your information for the marble game - same possible events, same causal information, same mutual exclusivity and exhaustiveness relations.</p>
<p>So what's analogous to the 'thirder' position, after we take into account that we have this causal information? Is it some difference in causal structure, or some non-causal anthropic modification, or something even stranger?</p>
<p>As it turns out, nope, it's the same exact game, just re-labeled.</p>
<p>In the re-labeled marble game you still have two unknown variables (represented by flipping coins), and you still have a 1/2 chance of black and Tails, a 1/4 chance of black and Heads, and a 1/4 chance of white and Heads.</p>
<p>And then to get the thirds, you ask the question "If I get a black marble, what is the probability of the faces of the first coin?" Now you update to P(Heads|black)=1/3 and P(Tails|black)=2/3.</p>
<p><strong>II</strong></p>
<p>Okay, enough analogies. What's going on with these two positions in the Sleeping Beauty problem?</p>
<p><strong>1:</strong><img src="http://images.lesswrong.com/t3_lau_0.png?v=4a2db85b3a25179fb783ff1e562c0c35" alt="" width="275" height="194" /> <strong> 2:</strong><img src="http://images.lesswrong.com/t3_lau_1.png" alt="" width="327" height="248" /></p>
<p>Here are two different diagrams, which are really re-labelings of the same diagram. The first labeling is the problem where P(Heads|Wake) = 1/2. The second labeling is the problem where P(Heads|Wake) = 1/3. The question at hand is really - which of these two math problems corresponds to the word problem / real world situation?</p>
<p>As a refresher, here's the text of the Sleeping Beauty problem that I'll use: Sleeping Beauty goes to sleep in a special room on Sunday, having signed up for an experiment. A coin is flipped - if the coin lands Heads, she will only be woken up on Monday. If the coin lands Tails, she will be woken up on both Monday and Tuesday, but with memories erased in between. Upon waking up, she then assigns some probability to the coin landing Heads, P(Heads|Wake).</p>
<p><strong>Diagram 1: </strong>First a coin is flipped to get Heads or Tails. There are two possible things that could be happening to her, Wake on Monday or Wake on Tuesday. If the coin landed Heads, then she gets Wake on Monday. If the coin landed Tails, then she could either get Wake on Monday or Wake on Tuesday (in the marble game, this was mediated by flipping a second coin, but in this case it's some unspecified process, so I've labeled it [???]). Because all the events already assume she Wakes, P(Heads|Wake) evaluates to P(Heads), which just as in the marble game is 1/2.</p>
<p>This [???] node here is odd, can we identify it as something natural? Well, it's not Monday/Tuesday, like in diagram 2 - there's no option that even corresponds to Heads & Tuesday. I'm leaning towards the opinion that this node is somewhat magical / acausal, just hanging around because of analogy to the marble game. So I think we can take it out. A better causal diagram with the halfer answer, then, might merely be Coin -> (Wake on Monday / Wake on Tuesday), where Monday versus Tuesday is not determined at all by a causal node, merely informed probabilistically to be mutually exclusive and exhaustive.</p>
<p><strong>Diagram 2: </strong>A coin is flipped, Heads or Tails, and also it could be either Monday or Tuesday. Together, these have a causal effect on her waking or not waking - if Heads and Monday, she Wakes, but if Heads and Tuesday, she Doesn't wake. If Tails, she Wakes. Her pre-Waking prior for Heads is 1/2, but upon waking, the event Heads, Tuesday, Don't Wake gets eliminated, and after updating P(Heads|Wake)=1/3.</p>
<p>There's a neat asymmetry here. In diagram 1, when the coin was Heads she got the same outcome no matter the value of [???], and only when the coin was Tails were there really two options. In Diagram 2, when the coin is Heads, two different things happen for different values of the day, while if the coin is Tails the same thing happens no matter the day.</p>
<p> </p>
<p>Do these seem like accurate depictions of what's going on in these two different math problems? If so, I'll probably move on to looking closer at what makes the math problem correspond to the word problem.</p>manfredc7fAEStCafGh8TFdf2014-11-23T02:00:13.584ZMeetup : Urbana-Champaign: Experimentation
https://lw2.issarice.com/posts/8wupMLWqc2GeBm2b5/meetup-urbana-champaign-experimentation
<h2>Discussion article for the meetup : <a href='/meetups/177'>Urbana-Champaign: Experimentation</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">23 November 2014 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St, Urbana IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>What sorts of things would you try out if you could? What meta-level strategies can help turn that list into actually trying things?
I also kind of want to talk about some ideas related to anthropics.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/177'>Urbana-Champaign: Experimentation</a></h2>manfred8wupMLWqc2GeBm2b52014-11-21T09:37:07.484ZDeriving probabilities from causal diagrams
https://lw2.issarice.com/posts/KZE9foCxCu2hBPnah/deriving-probabilities-from-causal-diagrams
<p><strong>What this is:</strong> an attempt to examine how causal knowledge gets turned into probabilistic predictions.</p>
<p>I'm not really a fan of any view of probability that involves black boxes. I want my probabilities (or more practically, the probabilities of toy agents in toy problems I consider) to be derivable from what I know in a nice clear way, following some desideratum of probability theory at every step.</p>
<p>Causal knowledge sometimes looks like a black box, when it comes to assigning probabilities, and I would like to crack open that box and distribute the candy inside to smiling children.</p>
<p><strong>What this is not: </strong>an attempt to get causal diagrams from constraints on probabilities.</p>
<p>That would be silly - see <a href="http://ftp.cs.ucla.edu/pub/stat_ser/r284-reprint.pdf">Pearl's article</a> that was recently up here. Our reasonable desire is the reverse: getting the constraints on probabilities from the causal diagrams.</p>
<p> </p>
<p><strong>The Marble Game</strong></p>
<p>Consider marbles. First, I use some coin-related process to get either Heads or Tails. If Tails, I give you a black marble. If Heads, I use some other process to choose between giving you a black marble or a white marble.</p>
<p>Causality is an important part of the marble game. If I manually interfere with the process that gives Heads or Tails, this can change the probability you should assign of getting a black marble. But if I manually interfere with the process that gives you white or black marbles, this won't change your probability of seeing Heads or Tails.</p>
<p> </p>
<p><strong>What I'd like versus what is</strong></p>
<p>The fundamental principle of putting numbers to beliefs, that always applies, is to not make up information. If I don't know of any functional differences between two events, I shouldn't give them different probabilities. But going even further - if I learn a little information, it should only change my probabilities a little.</p>
<p>The general formulation of this is to make your probability distribution consistent with what you know, in the way that contains the very least information possible (or conversely, the maximum entropy). This is how to not make up information.</p>
<p>I like this procedure; if we write down pieces of knowledge as mathematical constraints, we can find correct distribution by solving a single optimization problem. Very elegant. Which is why it's a shame that this isn't at all what we do for causal problems.</p>
<p>Take the marble game. To get our probabilities, we start with the first causal node, figure out the probability of Heads without thinking about marbles at all (that's easy, it's 1/2), and then move on to the marbles while taking the coin as given (3/4 for black and 1/4 for white).</p>
<p>One cannot do this problem without using causal information. If we neglect the causal diagram, our information is the following: <strong>A</strong>: We know that Heads and Tails are mutually exclusive and exhaustive (MEE), <strong>B</strong>: we know that getting a black marble and getting a white marble are MEE, and <strong>C</strong>: we know that if the coin is Tails, you'll get a black marble.</p>
<p>This leaves three MEE options: Tails and Black (TB), HB, and HW. Maximizing entropy, they all get probability 1/3.</p>
<p>One could alternately think of it like this: if we don't have the causal part of the problem statement (the causal diagram <strong>D</strong>), we don't know whether the coin causes the marble choice, or the marble causes the coin choice - why not pick a marble first, and if it's W we give you an H coin, but if it's B we flip the coin? Heck, why have one cause the other at all? Indeed, you should recover the 1/3 result if you average over all the consistent causal diagrams.</p>
<p>So my question is - what causal constraints is our distribution subject to, and what is it optimizing? Not piece by piece, but all at once?</p>
<p> </p>
<p><strong>Rephrasing the usual process</strong></p>
<p>One method is to just do the same steps as usual, but to think of the rationale in terms of knowledge / constraints and maximum entropy.</p>
<p>We start with the coin, and we say "because the coin's result isn't caused by the marbles, no information pertaining to marbles matters here. Therefore, P(H|ABCD) is just P(H|A) = 1/2" (First application of maximum entropy). Then we move on to the marbles, and applying information B and C, plus maximum entropy a second time, we learn that P(B|ABCD) = 3/4. All that our causal knowledge really meant for our probabilities was the equation P(H|ABCD)=P(H|A).</p>
<p>Alternatively, what if we only wanted to maximize something once, but let causal knowledge change the thing we were maximizing? We can say something like "we want to minimize the amount of information about the state of the coin, since that's the first causal node, and then minimize the amount of information about it's descendant node, the marble." Although this could be represented as one equation using linear multipliers, it's clearly the same process just with different labels.</p>
<p> </p>
<p><strong>Is it even possible to be more elegant?</strong></p>
<p>Both of these approaches are... functional. I like the first one a lot better, because I don't want to even come close to messing with the principle of maximum entropy / minimal information. But I don't like that we never get to apply this principle all at once. Can we break our knowledge down more so that everything happen nice and elegantly?</p>
<p>The way we stated our knowledge above was as P(H|ABCD) = P(H|A). But this is equivalent to the statement that there's a symmetry between the left and right branches coming out of the causal node. We can express this symmetry using the equivalence principle as P(H)=P(T), or as P(HB)+P(HW)=P(TB).</p>
<p>But note that this is just hiding what's going on, because the equivalence principle is just a special case of the maximum entropy principle - we might as well just require that P(H)=1/2 but still say that at the end we're "maximizing entropy subject to this constraint."</p>
<p> </p>
<p><strong>Answer: Probably not</strong></p>
<p>The general algorithm followed above is, for each causal node, to insert the condition that the probabilities of outputs of that node, given the starting information including the causal diagram, are equal to the probabilities given only the starting information related to that node or its parents - information about the descendants does not help determine probabilities of the parents.</p>manfredKZE9foCxCu2hBPnah2014-11-13T00:28:55.636ZMeetup : Urbana-Champaign: TRVTH
https://lw2.issarice.com/posts/odb6gSzqB7cTyPwFD/meetup-urbana-champaign-trvth
<h2>Discussion article for the meetup : <a href='/meetups/16q'>Urbana-Champaign: TRVTH</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">16 November 2014 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St. Urbana IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>How good would knowing the truth be, if we were perfect enough to use it?</p>
<p>How irrational do we have to be before lying to ourselves is a good idea? And how irrational do we have to be before it really is a bad idea after all?</p>
<p>Possibly also featuring our old nemesis, the remote associates test.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/16q'>Urbana-Champaign: TRVTH</a></h2>manfredodb6gSzqB7cTyPwFD2014-11-11T06:07:55.513ZMeetup : Urbana-Champaign: Writing Prompts
https://lw2.issarice.com/posts/QBh5jSvCLpKj7bY6o/meetup-urbana-champaign-writing-prompts
<h2>Discussion article for the meetup : <a href='/meetups/16h'>Urbana-Champaign: Writing Prompts</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">09 November 2014 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St., Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>In honor of RaNoWriMo, let's do a few writing prompts - 15 minutes of just sitting down and writing the first page of something awesome.</p>
<p>Want advice? Check out <a href="http://www.writingexcuses.com/2013/05/19/writing-excuses-8-20-the-short-story-with-mary-robinette-kowal/" rel="nofollow">an episode of writing excuses</a>.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/16h'>Urbana-Champaign: Writing Prompts</a></h2>manfredQBh5jSvCLpKj7bY6o2014-11-04T17:18:44.525ZMeetup : Urbana-Champaign: Fun and Games
https://lw2.issarice.com/posts/5MHwAMsmZyqZv6tYt/meetup-urbana-champaign-fun-and-games
<h2>Discussion article for the meetup : <a href='/meetups/164'>Urbana-Champaign: Fun and Games</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">02 November 2014 03:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St, Urbana IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Come for the fun and games, stay for practicing meditation. Also: halloween-candy-based elocution exercises.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/164'>Urbana-Champaign: Fun and Games</a></h2>manfred5MHwAMsmZyqZv6tYt2014-10-28T20:00:20.917ZMeetup : Urbana-Champaign: Meta-systems and getting things done
https://lw2.issarice.com/posts/9dxfsmXJacQ24FiCH/meetup-urbana-champaign-meta-systems-and-getting-things-done
<h2>Discussion article for the meetup : <a href='/meetups/15u'>Urbana-Champaign: Meta-systems and getting things done</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">26 October 2014 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St., Urbana IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Seed topic:</p>
<p>When I was a little kid doing chores, what worked was for my mom to tell me to do one thing. Then I'd do the thing, and come back and ask her "what next?" And then she'd tell me another thing, and I'd do that, and so on.</p>
<p>A few weeks ago Brienne posted <a href="http://lesswrong.com/lw/kxy/simulate_and_defer_to_more_rational_selves/">a technique that can be used to do this with a mental model of someone smart</a>. You build a model of what an effective person would do, and you ask your model what thing to do next, and the model tells you, and then you do that. This one of many ways of reminding yourself to follow the pattern "find the best strategy, and then do the next step in that strategy."</p>
<p>If you desire homework: try doing this for a few hours at some point during the week.</p>
<p>By contrast, consider the Getting Things Done family of productivity techniques, where you write down what you want to do in advance and then follow your written plan. How well has this worked for you in the past? (As one might predict, results vary).</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/15u'>Urbana-Champaign: Meta-systems and getting things done</a></h2>manfred9dxfsmXJacQ24FiCH2014-10-20T04:16:05.928ZMeetup : Urbana-Champaign: Stoicism, anthropics
https://lw2.issarice.com/posts/rQEKLC4qENQ48F3J4/meetup-urbana-champaign-stoicism-anthropics
<h2>Discussion article for the meetup : <a href='/meetups/15r'>Urbana-Champaign: Stoicism, anthropics</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">19 October 2014 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S Cedar St, Urbana IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>If you want to read up on Stoicism, check out the encyclopedia of philosophy entry.</p>
<p>For anthropics, see a recent discussion <a href="http://lesswrong.com/r/discussion/lw/l3w/baysian_conundrum/">here</a>.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/15r'>Urbana-Champaign: Stoicism, anthropics</a></h2>manfredrQEKLC4qENQ48F3J42014-10-16T13:21:36.037ZMeetup : Urbana-Champaign: Noticing continued, Creativity
https://lw2.issarice.com/posts/6S5u99q379w7njpu3/meetup-urbana-champaign-noticing-continued-creativity
<h2>Discussion article for the meetup : <a href='/meetups/15i'>Urbana-Champaign: Noticing continued, Creativity</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">12 October 2014 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St, Urbana, IL, 61801</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Let's reconvene on noticing. We may have to go a little meta, but ah well.</p>
<p>I'd also like to try some group creativity exercises, for great justice.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/15i'>Urbana-Champaign: Noticing continued, Creativity</a></h2>manfred6S5u99q379w7njpu32014-10-09T16:41:09.908ZMeetup : Urbana-Champaign: Noticing.
https://lw2.issarice.com/posts/7YjwLzreo3wffxEK7/meetup-urbana-champaign-noticing
<h2>Discussion article for the meetup : <a href='/meetups/158'>Urbana-Champaign: Noticing.</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">05 October 2014 05:30:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St., Urbana, IL, 61801</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Let's work on noticing things. Homework before this meetup: work on noticing when you're rationalizing, and do some physical act like standing up, or snapping your fingers, or writing down what you were rationalizing about.</p>
<p>This may be a bit ambitious, but that's what we can work on at the meetup.</p>
<p>Some reading material: <a href="http://agentyduck.blogspot.com/2014/09/what-its-like-to-notice-things.html" rel="nofollow">1</a> <a href="http://agentyduck.blogspot.com/2014/09/noticing-curiosity-and-searching-log.html" rel="nofollow">2</a></p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/158'>Urbana-Champaign: Noticing.</a></h2>manfred7YjwLzreo3wffxEK72014-10-02T20:33:00.781ZMeetup : Urbana-Champaign: The Steep Approach to Crazytown
https://lw2.issarice.com/posts/7iaNZ4rPyAr3eSiLo/meetup-urbana-champaign-the-steep-approach-to-crazytown
<h2>Discussion article for the meetup : <a href='/meetups/14u'>Urbana-Champaign: The Steep Approach to Crazytown</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">28 September 2014 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar, 61801</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>As was suggested last week, we'll be checking out some exercises from the Scientology community. Making fun of them is, of course, mandatory, but the goal is to find some that are interesting enough to try and then trying them. I don't see how this could go wrong.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/14u'>Urbana-Champaign: The Steep Approach to Crazytown</a></h2>manfred7iaNZ4rPyAr3eSiLo2014-09-23T04:00:32.057ZMeetup : Urbana-Champaign: Practical Rationality
https://lw2.issarice.com/posts/jHb8F7nY8kMPFTYX7/meetup-urbana-champaign-practical-rationality
<h2>Discussion article for the meetup : <a href='/meetups/13x'>Urbana-Champaign: Practical Rationality</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">07 September 2014 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St, 61801</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>What techniques do you use for acting effectively given that you are a squishy irrational meatbag? Jack will contribute some ideas which he has smuggled to us from CFAR.</p>
<p>Also, we will continue refining the rules of Wits and Wagers.</p>
<p><a href="https://groups.google.com/forum/#!forum/lesswrong-urbana-champaign">Google group</a>.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/13x'>Urbana-Champaign: Practical Rationality</a></h2>manfredjHb8F7nY8kMPFTYX72014-08-31T22:43:27.697ZMeetup : Urbana-Champaign: Reconstituting
https://lw2.issarice.com/posts/C359ArrQCLafGnCNR/meetup-urbana-champaign-reconstituting
<h2>Discussion article for the meetup : <a href='/meetups/13j'>Urbana-Champaign: Reconstituting</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">31 August 2014 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">206 S. Cedar St, 61801</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>In the puddles and pools formed by the fall rains, meetup groups that seemed dead end their arid hibernation and emerge from the earth.</p>
<p>Come join us for the first meetup of the new school year. Board games and light discussion are likely. A starter topic: increasing exposure to positive black swans.</p>
<p><a href="https://groups.google.com/forum/#!forum/lesswrong-urbana-champaign">Google group</a></p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/13j'>Urbana-Champaign: Reconstituting</a></h2>manfredC359ArrQCLafGnCNR2014-08-23T04:19:55.882ZThought experiments on simplicity in logical probability
https://lw2.issarice.com/posts/FHisTJwPjpcMXEXDx/thought-experiments-on-simplicity-in-logical-probability
<p>A common feature of many proposed logical priors is a preference for simple sentences over complex ones. This is sort of like an extension of Occam's razor into math. Simple things are more likely to be true. So, as it is said, "why not?"</p>
<p> </p>
<p>Well, the analogy has some wrinkles - unlike hypothetical rules for the world, logical sentences do not form a mutually exclusive set. Instead, for every sentence A there is a sentence not-A with pretty much the same complexity, and probability 1-P(A). So you can't make the probability smaller for all complex sentences, because their negations are also complex sentences! If you don't have any information that discriminates between them, A and not-A will both get probability 1/2 no matter how complex they get.</p>
<p>But if our agent knows something that breaks the symmetry between A and not-A, like that A belongs to a mutually exclusive and exhaustive set of sentences with differing complexities, then it can assign higher probabilities to simpler sentences in this set without breaking the rules of probability. Except, perhaps, the rule about not making up information.</p>
<p><strong>The question: is the simpler answer really more likely to be true than the more complicated answer, or is this just a delusion? If so, is it for some ontologically basic reason, or for a contingent and explainable reason?</strong></p>
<p> </p>
<p>There are two complications to draw your attention to. The first is in what we mean by complexity. Although it would be nice to use the Kolmogorov complexity of any sentence, which is the length of the shortest program that prints the sentence, such a thing is uncomputable by the kind of agent we want to build in the real world. The only thing our real-world agent is assured of seeing is the length of the sentence as-is. We can also find something in between Kolmogorov complexity and length by doing a brief search for short programs that print the sentence - this meaning is what is usually meant in this article, and I'll call it "apparent complexity."</p>
<p>The second complication is in what exactly a simplicity prior is supposed to look like. In the case of Solomonoff induction the shape is exponential - more complicated hypotheses are exponentially less likely. But why not a power law? Why not even a Poisson distribution? Does the difficulty of answering this question mean that thinking that simpler sentences are more likely is a delusion after all?</p>
<p> </p>
<p>Thought experiments:</p>
<p>1: Suppose our agent knew from a trusted source that some extremely complicated sum could only be equal to A, or to B, or to C, which are three expressions of differing complexity. What are the probabilities?</p>
<p> </p>
<p>Commentary: This is the most sparse form of the question. Not very helpful regarding the "why," but handy to stake out the "what." Do the probabilities follow a nice exponential curve? A power law? Or, since there are just the three known options, do they get equal consideration?</p>
<p>This is all based off intuition, of course. What does intuition say when various knobs of this situation are tweaked - if the sum is of unknown complexity, or of complexity about that of C? If there are a hundred options, or countably many? Intuitively speaking, does it seem like favoring simpler sentences is an ontologically basic part of your logical prior?</p>
<p> </p>
<p>2: Consider subsequences of the digits of pi. If I give you a pair (n,m), you can tell me the m digits following the nth digit of pi. So if I start a sentence like "the subsequence of digits of pi (10<sup>100</sup>, 10<sup>2</sup>) = ", do you expect to see simpler strings of digits on the right side? Is this a testable prediction about the properties of pi?</p>
<p> </p>
<p>Commentary: We know that there is always a short-ish program to produce the sequences, which is just to compute the relevant digits of pi. This sets a hard upper bound on the possible Kolmogorov complexity of sequences of pi (that grows logarithmically as you increase m and n), and past a certain m this will genuinely start restricting complicated sequences, and thus favoring "all zeros" - or does it?</p>
<p>After all, this is weak tea compared to an exponential simplicity prior, for which the all-zero sequence would be hojillions of times more likely than a messy one. On the other hand, an exponential curve allows sequences with higher Kolmogorov complexity than the computation of the digits of pi.</p>
<p>Does the low-level view outlined in the first paragraph above demonstrate that the exponential prior is bunk? Or can you derive one from the other with appropriate simplifications (keeping in mind Komogorov complexity vs. apparent complexity)? Does pi really contain more long simple strings than expected, and if not what's going on with our prior?</p>
<p> </p>
<p>3: Suppose I am writing an expression that I want to equal some number you know - that is, the sentence "my expression = your number" should be true. If I tell you the complexity of my expression, what can you infer about the likelihood of the above sentence?</p>
<p> </p>
<p>Commentary: If we had access to Kolmogorov complexity of your number, then we could completely rule out answers that were too K-simple to work. With only an approximation, it seems like we can still say that simple answers are <em>less</em> likely up to a point. Then as my expression gets more and more complicated, there are more and more available wrong answers (and, outside of the system a bit, it becomes less and less likely that I know what I'm doing), and so probability goes down.</p>
<p>In the limit that my expression is much more complex than your number, does an elegant exponential distribution emerge from underlying considerations?</p>manfredFHisTJwPjpcMXEXDx2014-08-20T17:25:35.026ZRaven paradox settled to my satisfaction
https://lw2.issarice.com/posts/tAfK3ckM9yrQBLCJL/raven-paradox-settled-to-my-satisfaction
<p>The raven paradox, originated by Carl Gustav Hempel, is an apparent absurdity of inductive reasoning. Consider the hypothesis:</p>
<p><em>H1: All ravens are black.</em></p>
<p>Inductively, one might expect that seeing many black ravens and no non-black ones is evidence for this hypothesis. As you see more black ravens, you may even find it more and more likely.</p>
<p>Logically, a statement is equivalent to its contrapositive (where you negate both things and flip the order). Thus if "if it is a raven, it is black" is true, so is:</p>
<p><em>H1': If it is not black, it is not a raven.</em></p>
<p>Take a moment to double-check this.</p>
<p>Inductively, just like with H1, one would expect that seeing many non-black non-ravens is evidence for this hypothesis. As you see more and more examples, you may even find it more and more likely. Thus a yellow banana is evidence for the hypothesis "all ravens are black."</p>
<p>Since this is silly, there is an apparent problem with induction.</p>
<p> </p>
<p><strong>Resolution</strong></p>
<p>Consider the following two possible states of the world:</p>
<p><img src="http://images.lesswrong.com/t3_kp2_0.png" alt="Either 100 black ravens, or 99 black 1 yellow" width="796" height="100" /></p>
<p>Suppose that these are your two hypotheses, and you observe a yellow banana (drawing from some fixed distribution over things). Q: What does this tell you about one hypothesis versus another? A: It tells you bananas-all about the number of black ravens.</p>
<p>One might contrast this with a hypothesis where there is one less banana, and one more yellow raven, by some sort of spontaneous generation.</p>
<p><img src="http://images.lesswrong.com/t3_kp2_1.png?v=be4cbdb515237d1cec04a4a5f5a97673" alt="" width="375" height="100" /></p>
<p>Observations of both black ravens and yellow bananas cause us to prefer 1 over 3, now!</p>
<p>The moral of the story is that the amount of evidence that an observation provides is not just about whether it whether it is consistent with the "active" hypothesis - it is about the difference in likelihood between when the hypothesis is true versus when it's false.</p>
<p>This is a pretty straightforward moral - it's a widely known pillar of statistical reasoning. But its absence in the raven paradox takes a bit of effort to see. This is because we're using an implicit model of the problem (driven by some combination of outside knowledge and framing effects) where nonblack ravens replace black ravens, but don't replace bananas. The logical statements H1 and H1' are not alone enough to tell how you should update upon seeing new evidence. Or to put it another way, the version of induction that drives the raven paradox is in fact wrong, but probability theory implies a bigger version.</p>
<p> </p>
<p>(Technical note: In the hypotheses above, the exact number of yellow bananas does not have to be the same for observing a yellow banana to provide no evidence - what has to be the same is the measure of yellow bananas in the probability distribution we're drawing from. Talking about "99 ravens" is more understandable, but what differentiates our hypotheses are really the likelihoods of observing different events [there's our moral again]. This becomes particularly important when extending the argument to infinite numbers of ravens - infinities or no infinities, when you make an observation you're still drawing from some distribution.)</p>manfredtAfK3ckM9yrQBLCJL2014-08-06T02:46:19.257ZMaybe we're not doomed
https://lw2.issarice.com/posts/ZoBjMxrHN9MddpZXg/maybe-we-re-not-doomed
<p>This is prompted by Scott's excellent article, <a href="http://slatestarcodex.com/2014/07/30/meditations-on-moloch/">Meditations on Moloch</a>.</p>
<p>I might caricature (grossly unfairly) his post like this:</p>
<p><ol>
<li>Map some central problems for humanity onto the tragedy of the commons.</li>
<li>Game theory says we're doomed.</li>
</ol>
<div>Of course my life is pretty nice right now. But, goes the story, this is just a non-equilibrium starting period. We're inexorably progressing towards a miserable Nash equilibrium, and once we get there we'll be doomed forever. (This forever loses a bit of foreverness if one expects everything to get interrupted by self-improving AI, but let's elide that.)</div>
<div><br /></div>
<div>There are a few ways we might not be doomed. The first and less likely is that people will just decide not to go to their doom, even though it's the Nash equilibrium. To give a totally crazy example, suppose there were two countries playing a game where the first one to launch missiles had a huge advantage. And neither country trusts the other, and there are multiple false alarms - thus pushing the situation to the stable Nash equilibrium of both countries trying to launch first. Except imagine that somehow, through some heroic spasm of insanity, these two countries just decided not to nuke each other. That's the sort of thing it would take.</div>
<div><br /></div>
<div>Of course, people are rarely able to be that insane, so success that way should not be counted on. But on the other hand, if we're doomed forever such events will eventually occur - like a bubble of spontaneous low entropy spawning intelligent life in a steady-state universe.</div>
<div><br /></div>
<div>The second and most already-implemented way is to jump outside the system and change the game to a non-doomed one. If people can't share the commons without defecting, why not portion it up into private property? Or institute government regulations? Or iterate the game to favor tit-for-tat strategies? Each of these changes has costs, but if the wage of the current game is 'doom,' each player has an incentive to change the game.</div>
<div><br /></div>
<div>Scott devotes a sub-argument to why we're still doomed to things be miserable if we solve coordination problems with government:</div>
<div><ol>
<li>Incentives for government employees sometimes don't match the needs of the people.</li>
<li>This has costs, and those costs help explain why some things that suck, suck.</li>
</ol>
<div>I agree with this, but not all governments are equally costly as coordination technologies. Heck, not all governments even <em>are</em> a technology for improving peoples' lives - look at North Korea. My point is that there's no particular reason that costs can't be small, with sufficiently advanced cultural technology.</div>
</div>
<div><br /></div>
<div>More interesting to me than government is the idea of iterating a game to to encourage cooperation. In the normal prisoner's dilemma game, the only Nash equilibrium is defect-defect and so the prisoners are doomed. But if you have to play the prisoner's dilemma game repeatedly, with a variety of other players, the best strategy turns out to be a largely cooperative one. This evasion of doom gives every player an incentive to try and replace one-shot dilemmas with iterated ones. Could Scott's post look like this?</div>
<div><ol>
<li>Map some central problems for humanity onto the iterated prisoner's dilemma.</li>
<li>Evolutionary game theory says we're not doomed.</li>
</ol>
<div>In short, I think this idea of "if you know the Nash equilibrium sucks, everyone will help you change the game" is an important one. Though given human irrationality, game-theoretic predictions (whether of eventual doom or non-doom) should be taken less than literally.</div>
</div>
</p>manfredZoBjMxrHN9MddpZXg2014-08-02T15:22:14.965ZTop-Down and Bottom-Up Logical Probabilities
https://lw2.issarice.com/posts/wYBv4WRuusKNKM2gt/top-down-and-bottom-up-logical-probabilities
<p><strong>I.</strong></p>
<p>I don't know very much model theory, and thus I don't fully understand <a href="http://www.hutter1.net/publ/problogics.pdf">Hutter et al.'s logical prior, detailed here</a>, but nonetheless I can tell you that it uses a <em>very</em> top-down approach. About 60% of what I mean is that the prior is presented as a completed object with few moving parts, which fits the authors' mathematical tastes and proposed abstract properties the function should have. And for another thing, it uses model theory - a dead giveaway.</p>
<p>There are plenty of reasons to take a top-down approach. Yes, Hutter et al.'s function isn't computable, but sometimes the properties you want require uncomputability. And it's easier to come up with something vaguely satisfactory if you don't have to have many moving parts. This can range from "the prior is defined as a thing that fulfills the properties I want" on the lawful good side of the spectrum, to "clearly the right answer is just the exponential of the negative complexity of the statement, <em>duh</em>".</p>
<p>Probably the best reason to use a top-down approach to logical uncertainty is so you can do math to it. When you have some elegant description of global properties, it's a lot easier to prove that your logical probability function has nice properties, or to use it in abstract proofs. Hence why model theory is a dead giveaway.</p>
<p>There's one other advantage to designing a logical prior from the top down, which is that you can insert useful stuff like a complexity penalty without worrying too much. After all, you're basically making it up as you go anyhow, you don't have to worry about where it comes from like you would if you were going form the bottom up.</p>
<p>A bottom-up approach, by contrast, starts with an imagined agent with some state of information and asks what the right probabilities to assign are. Rather than pursuing mathematical elegance, you'll see a lot of comparisons to what humans do when reasoning through similar problems, and demands for computability from the outset.</p>
<p>For me, a big opportunity of the bottom-up approach is to use desiderata that look like principles of reasoning. This leads to more moving parts, but also outlaws some global properties that don't have very compelling reasons behind them.</p>
<p> </p>
<p><strong>II.</strong></p>
<p>Before we get to the similarities, rather than the differences, we'll have to impose the condition of limited computational resources. A common playing field, as it were. It would probably serve just as well to extend bottom-up approaches to uncomputable heights, but I am the author here, and I happen to be biased towards the limited-resources case.</p>
<p>The part of top-down assignment using limited resources will be played by a skeletonized pastiche of <a href="http://intelligence.org/2014/06/23/new-report-non-omniscience-probabilistic-inference-metamathematics/">Paul Christiano's recent report</a>:</p>
<p style="padding-left: 30px;"><strong>i.</strong><span style="white-space: pre;"> </span>No matter what, with limited resources we can only assign probabilities to a limited pool of statements. Accordingly, step one is to use some process to choose the set S<sub>0</sub> of statements (and their negations) to assign probabilities.</p>
<p style="padding-left: 30px;"><strong>ii.</strong><span style="white-space: pre;"> </span>Then we use something a weakened consistency condition (that can be decided between pairs of sentences in polynomial time) to set constraints on the probability function over S<sub>0</sub>. For example, sentences that are identical except for a double-negation have to be given the same probability.</p>
<p style="padding-left: 30px;"><strong>iii.</strong><span style="white-space: pre;"> </span>Christiano constructs a description-length-based "pre-prior" function that is bigger for shorter sentences. There are lots of options for different pre-priors, and I think this is a pretty good one.</p>
<p style="padding-left: 30px;"><strong>iv.</strong><span style="white-space: pre;"> </span>Finally, assign a logical probability function over S<sub>0</sub> that is as similar as possible to the pre-prior while fulfilling the consistency condition. Christiano measures similarity using cross-entropy between the two functions, so that the problem is one of minimizing cross-entropy subject to a finite list of constraints. (Even if the pre-prior decreases exponentially, this doesn't mean that complicated statements will have exponentially low logical probability, because of the condition from step two that P(a statement) + P(its negation) = 1 - in a state of ignorance, everything still gets probability 1/2. The pre-prior only kicks in when there are more options with different description lengths.)</p>
<p>Next, let's look at the totally different world of a bottom-up assignment of logical probabilities, played here by a mildly rephrased version of <a href="/lw/ee2/logical_uncertainty_kind_of_a_proposal_at_least/">my past proposal</a>.</p>
<p style="padding-left: 30px;"><strong>i.</strong><span style="white-space: pre;"> </span>Pick a set of sentences S<sub>1</sub> to try and figure out the logical probabilities of.</p>
<p style="padding-left: 30px;"><strong><span style="white-space: pre;">i</span>i.</strong><span style="white-space: pre;"> </span>Prove the truth or falsity of a bunch of statements in the closure of S<sub>1</sub> under conjugation and negation (i.e. if sentences <em>a </em>and <em>b</em> are in S<sub>1</sub>, <em>a&b</em> is in the closure of S<sub>1</sub>).</p>
<p style="padding-left: 30px;"><strong><span style="white-space: pre;">i</span>ii.</strong><span style="white-space: pre;"> </span>Assign a logical probability function over the closure of S<sub>1</sub> under conjugation with maximum entropy, subject to the constraints proved in part two, plus the constraints that each sentence && its negation has probability 0.</p>
<p>These turn out to be really similar! Look in step three of my bottom-up example - there's a even a sneakily-inserted top-down condition about going through every single statement and checking an aspect of consistency. In the top-down approach, every theorem of a certain sort is proved, while in the bottom-up approach there are allowed to be lots of gaps - but the same sorts of theorems are proved. I've portrayed one as using proofs only about sentences in S<sub>0</sub>, and the other as using proofs in the entire closure of S<sub>1</sub> under conjunction, but those are just points on an available continuum (for more discussion, see Christiano's section on positive semidefinite methods).</p>
<p>The biggest difference is this "pre-prior" thing. On the one hand, it's essential for giving us guarantees about inductive learning. On the other hand, what piece of information do we have that tells us that longer sentences really are less likely? I have unresolved reservations, despite the practical advantages.</p>
<p> </p>
<p><strong>III.</strong></p>
<p>A minor confession - my choice of Christiano's report was not coincidental at all. The causal structure went like this:</p>
<p>Last week - Notice dramatic similarities in what gets proved and how it gets used between my bottom-up proposal and Christiano's top-down proposal.</p>
<p>Now - Write post talking about generalities of top-down and bottom-up approaches to logical probability, and then find as a startling conclusion the thing that motivated me to write the post in the first place.</p>
<p>The teeensy bit of selection bias here means that though these similarities are cool, it's hard to draw general conclusions.</p>
<p>So let's look at one more proposal, this one due to <a href="http://ict.usc.edu/pubs/Logical%20Prior%20Probability.pdf">Abram Demski</a>, modified by to use limited resources.</p>
<p style="padding-left: 30px;"><strong>i.</strong> Pick a set of sentences S<sub>2</sub> to care about.</p>
<p style="padding-left: 30px;"><strong>ii.</strong> Construct a function on sentences in S<sub>2</sub> that is big for short sentences and small for long sentences.</p>
<p style="padding-left: 30px;"><strong>iii.</strong> Start with the set of sentences that are axioms - we'll shortly add new sentences to the set.</p>
<p style="padding-left: 30px;"><strong>iv.</strong> Draw a sentence from S<sub>2</sub> with probability proportional to the function from step two.</p>
<p style="padding-left: 30px;"><strong>v.</strong> Do a short consistency check (can use a weakened consistency condition, or just limited time) between this sentence and the sentences already in the set. If it's passed, add the sentence to the set.</p>
<p style="padding-left: 30px;"><strong>vi.</strong> Keep doing steps four and five until you've either added or ruled out all the sentences in S<sub>2</sub>.</p>
<p style="padding-left: 30px;"><strong>vii.</strong> The logical probability of a sentence is defined as the probability that it ends up in our set after going through this process. We can find this probability using Monte Carlo by just running the process a bunch of times and counting up what portion of the time each sentences is in the set by the end.</p>
<p>Okay, so this one looks pretty different. But let's look for the similarities. The exact same kinds of things get proved again - weakened or scattershot consistency checks between different sentences. If all you have in S<sub>2</sub> are three mutually exclusive and exhaustive sentences, the one that's picked first wins - meaning that the probability function over what sentence gets picked first is acting like our pre-prior.</p>
<p>So even though the method is completely different, what's really going on is that sentences are being given measure that looks like the pre-prior, subject to the constraints of weakened consistency (via rejection sampling) and normalization (keep repeating until all statements are checked).</p>
<p>In conclusion: not everything is like everything else, but some things are like some other things.</p>manfredwYBv4WRuusKNKM2gt2014-07-22T08:53:15.473ZMore and Less than Solomonoff Induction
https://lw2.issarice.com/posts/od2hkE2CS9tYZXHkx/more-and-less-than-solomonoff-induction
<p>I've been thinking about how to put induction with limited resources on a firmer foundation. This may just be retracing the steps of others, but that's okay with me. Mostly I just want to talk about these thoughts.</p>
<p>After a few points of introduction.</p>
<p><strong>What's Solomonoff induction?</strong></p>
<p>Suppose we're given some starting data, and asked to predict the future. Solomonoff induction predicts the future by combining the predictions of all programs that (1) output the starting data up until now, and (2) aren't the continuation of another such program. The predictions are combined according to a weighting that decreases exponentially as the length of the program increases.</p>
<p><strong>Why is it a good idea?</strong></p>
<p>The simplest answer is that we have a frequentist guarantee that if the "true program" generating our input has some length N (that is, if the observable universe is a big but finite-sized computer), then our predictions will only be wrong a limited number of times, and after that we'll predict the correctly every time.</p>
<p>A more bayesian answer would start with the information that our observations can be generated by some finite-sized program, and then derive that something like Solomonoff induction has to represent our true prior over generating programs - as the length gets bigger, our probability is required to go to zero at infinity, and an exponential is the maximum-entropy such curve. This is not a complete answer, but it at least makes the missing pieces more apparent.</p>
<p><strong>Why won't it work with limited resources</strong></p>
<p>The trouble with using Solomonoff induction in real life is that to pick out which programs output our data so far, we need to run every program - and if the program doesn't ever halt, we need to use a halting oracle to stop it or else we'll take infinite time.</p>
<p><strong>Limited resources require us to only pick from a class of programs that is guaranteed to not run over the limit.</strong></p>
<p>If we have limited time and no halting oracle, we can't check every program. Instead, we are only allowed to check programs drawn from a class of programs that we can check the output of in limited time. The simplest example would be to just check programs but not report the result if we go over some time limit, in which case the class we're picking from is "programs that don't go over the time limit."</p>
<p>This is an application of a general principle - when we impose resource constraints on a real agent, we want to be able to cash them out as properties of an abstract description of what the agent is doing. In this case, we can cash out limited time for our real agent as an inability for our abstract description to have any contributions from long programs.</p>
<p>This isn't really a Bayesian restriction - we don't actually know that the true program is within our search space. This also weakens our frequentist guarantee.</p>
<p><strong>The problem of overfitting - real models allow for incorrect retrodictions.</strong></p>
<p>If we just take Solomonoff induction and put restrictions on it, our predictions will still only come from hypotheses that exactly reproduce our starting data. This is a problem.</p>
<p>For example, if we have a ton of data, <em>like we do</em>, then finding even one program that exactly replicates our data is too hard, and our induction is useless.</p>
<p>We as humans have two main ways of dealing with this. The first is that we ignore most of our data and only try to predict the important parts. The second is that we don't require our models to perfectly retrodict our observations so far (retrodiction- it's like prediction, for the past!).</p>
<p>In fact. having imperfect retrodictions is such a good idea that we can have a problem called "overfitting." Isn't that kinda odd? That it's better to use a simple model that is actually 100% known to be wrong, than to use a very complicated model that has gotten everything right so far.</p>
<p>This behavior makes more sense if we imagine that our data contains contributions both from the thing we're trying to model, and from stuff we want to ignore, in a way that's hard to separate.</p>
<p><strong>What makes real models better than chance? The example of coarse-graining.<img style="float: right;" src="http://images.lesswrong.com/t3_k65_0.png" alt="" width="294" height="401" /></strong></p>
<p>Is it even possible to know ahead of time that an imperfect model will make good predictions? It turns out the answer is yes.</p>
<p>The very simplest example is of just predicting the next bit based on which bit has been more common so far. If every pattern were equally likely this wouldn't work, but if we require the probability of a pattern to decrease with its complexity, we're more likely to see repetitions.</p>
<p>A more general example: an imperfect model is always better than chance if it is a likely <em>coarse-graining</em> of the true program. Coarse-graining is when you can use a simple program to predict a complicated one by only worrying about some of the properties of the complicated program. In the picture at right, a simple cellular automaton can predict whether a more complicated one will have the same or different densities of white and black in a region.</p>
<p>When a coarse-graining is exact, the coarse-grained properties follow exact rules all the time. Like in the picture at right, where the coarse-grained pattern "same different same" always evolves into "different different different," even though there are multiple states of the more complicated program that count as "different" and "same."</p>
<p>When a coarse-graining is inexact, only most of the states of the long program follow the coarse-grained rules of the short program. but it turns out that, given some ignorance about the exact rules or situation, this is also sufficient to predict the future better than chance.</p>
<p>Of course, when doing induction, we don't actually know the true program. Instead what happens is that we just find some simple program that fits our data reasonably well (according to some measurement of fit), and we go "well, what happened before is more likely to happen again, so this rule will help us predict the future."</p>
<p>Presumably we combine predictions according to some weighting that include both the length of the program and its goodness of fit.</p>
<p><strong>Machine learning - probably approximately correct</strong></p>
<p>Since this is a machine learning problem, there are already solutions to similar problems, one of which is probably approximately correct learning. The basic idea is that if you have some uniformly-sampled training data, and a hypothesis space you can completely search, then you can give some probabilistic guarantees about how good the hypothesis is that best fits the training data. A "hypothesis," here, is a classification of members of data-space into different categories.</p>
<p>The more training data you have, the closer (where "close" can be measured as a chi-squared-like error) the best-fitting hypothesis is to the actually-best hypothesis. If your hypothesis space doesn't contain the true hypothesis, then that's okay - you can still guarantee that the best-fitting hypothesis gets close to the best hypothesis in your hypothesis space. The probability that a far-away hypothesis would masquerade as a hypothesis within the small "successful" region gets smaller and smaller as the training data increases.</p>
<p>There is a straightforward extension to cases where your data contains contributions from both "things we want to model" and "things we want to ignore." Rather than finding the hypothesis that fits the training data best, we want to find the hypothesis for the "stuff we want to model" that has the highest probability of producing our observed data, after we've added some noise, drawn from some "noise distribution" that encapsulates the basic information about the stuff we want to ignore.</p>
<p>There are certainly some issues comparing this to Solomonoff induction, like the fact that our training data is randomly sampled rather than a time series, but I do like this paradigm a bit better than looking for a guarantee that we'll find the one true answer in finite time.</p>
<p><strong>Bayes' theorem</strong></p>
<p>If there's an easy way to combine machine learning with Solomonoff induction, it's Bayes' theorem. The machine learning was focused on driving down P(training data | chance), and picking a hypothesis with a high P(training data | hypothesis, noise model). We might want to Use Bayes' rule to say something like P(hypothesis | training data, noise model) = P(hypothesis | complexity prior) * P(training data | hypothesis, noise model) / P(training data | noise model).</p>
<p> </p>
<p>Anyhow - what are the obvious things I've missed? :)</p>manfredod2hkE2CS9tYZXHkx2014-05-21T04:55:39.054ZMeetup : Urbana-Champaign: Recreation
https://lw2.issarice.com/posts/funNavdrwn52Taffz/meetup-urbana-champaign-recreation
<h2>Discussion article for the meetup : <a href='/meetups/100'>Urbana-Champaign: Recreation</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">11 May 2014 01:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">40.1112,-88.2274</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Since the weather is nice, let's meet outside, on the benches south of the engineering quad. I'll bring kites - coincidentally, the engineering quad is a great place to fly kites.</p>
<p>We may talk about mechanism design (<a href="http://lesswrong.com/lw/k5r/mechanism_design_constructing_algorithms_for/">reading</a>).</p>
<p>Bonus topic: what do debates like <a href="http://lesswrong.com/r/discussion/lw/k7b/link_sean_carroll_against_afterlife/">this</a> recent one tell us about what's going on in peoples' heads?</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/100'>Urbana-Champaign: Recreation</a></h2>manfredfunNavdrwn52Taffz2014-05-08T13:52:35.938ZMeetup : Urbana-Champaign: Planning and Re-planning
https://lw2.issarice.com/posts/NAz6mEWvxk7kGTZ5j/meetup-urbana-champaign-planning-and-re-planning
<h2>Discussion article for the meetup : <a href='/meetups/ze'>Urbana-Champaign: Planning and Re-planning</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">20 April 2014 12:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">412 W. Elm St, Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>When things get complicated enough, you have to plan them in advance or they fail. You need blueprints and logistics before you can build a skyscraper. On a personal level, good plans improve our chances of success at anything we can make a plan for.</p>
<p>One trouble with plans is that once you've made them they're sticky. What kind of life to lead, what to study, when to marry - we inherit plans about these things.from the past and we don't always rethink them when appropriate.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/ze'>Urbana-Champaign: Planning and Re-planning</a></h2>manfredNAz6mEWvxk7kGTZ5j2014-04-17T05:56:36.094ZMeetup : Urbana-Champaign: Rationality and Cooking
https://lw2.issarice.com/posts/ksJ2Sci5ncRj8eZ6m/meetup-urbana-champaign-rationality-and-cooking
<h2>Discussion article for the meetup : <a href='/meetups/yq'>Urbana-Champaign: Rationality and Cooking</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">06 April 2014 12:01:01PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">412 W. Elm St, Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>There are a lot of rationality skills applicable to cooking. On the one hand, this is nice for cooking. On the other hand, it's an interesting example of how little we use explicit rationality by default, and what we do instead.</p>
<p>Also, I may cook some food for A/B testing.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/yq'>Urbana-Champaign: Rationality and Cooking</a></h2>manfredksJ2Sci5ncRj8eZ6m2014-04-03T16:18:18.737ZMeetup : Urbana-Champaign: Lose Arguments to Win Fabulous Prizes
https://lw2.issarice.com/posts/yzbRufYALPJimqSR3/meetup-urbana-champaign-lose-arguments-to-win-fabulous
<h2>Discussion article for the meetup : <a href='/meetups/y5'>Urbana-Champaign: Lose Arguments to Win Fabulous Prizes</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">23 March 2014 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">412 W. Elm St, Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>The only time it's ever possible to learn is when you change your mind. This is even true if you're in disagreement with someone else - being right can mean losing an argument.</p>
<p>Learning how to lose those arguments, and thus be more right, is a really useful skill. Let's talk about it.</p>
<p>Also see: <a href="http://wiki.lesswrong.com/wiki/How_To_Actually_Change_Your_Mind">an entire sequence</a>.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/y5'>Urbana-Champaign: Lose Arguments to Win Fabulous Prizes</a></h2>manfredyzbRufYALPJimqSR32014-03-19T09:46:54.699ZSolutions and Open Problems
https://lw2.issarice.com/posts/qiokMkCdFjXa9PTR3/solutions-and-open-problems
<p><strong>Followup To:</strong> <a href="/lw/jjl/approaching_logical_probability/">Approaching Logical Probability</a></p>
<p>Last time, we required our robot to only assign logical probability of 0 or 1 to statements where it's checked the proof. This flowed from our desire to have a robot that comes to conclusions in limited time. It's also important that this abstract definition has to take into account the pool of statements that our actual robot actually checks. However, this restriction doesn't give us a consistent way to assign numbers to unproven statements - to be consistent we have to put limits on our application of the <a href="/lw/jfx/foundations_of_probability/">usual rules of probability</a>.</p>
<p><a id="more"></a></p>
<div>
<div><strong>Total Ignorance</strong></div>
<div><br /></div>
<div>The simplest solution is to assign logical probabilities to proven statements normally, but totally refuse to apply our information to unproven statements. The principle of maximum entropy means that every unproven statement then gets logical probability 0.5.</div>
<div><br /></div>
<div>There is a correspondence between being inconsistent and ignoring information. We could just as well have said that when we move from proven statements to unproven statements, we refuse to apply the product rule, and that would have assigned every unproven statement logical probability 0.5 too. Either way, there's something "non-classical" going on at the boundary between proven statement and unproven statements. If you are certain that 100+98=198, but not that 99+99=198, some unfortunate accident has befallen the rule that if (A+1)+B=C, A+(B+1)=C.</div>
<div><br /></div>
<div>Saying 0.5 for everything is rarely suggested in practice, because it has some glaringly bad properties: if we ask about the last digit of the zillionth prime number, we don't want to say that the answer being 1 has logical probability 0.5, we want our robot to use facts about digits and about primes to say that 1, 3, 7, and 9 have logical probability 0.25 each.</div>
<div><br /></div>
<div><strong>Using Our Pool of Proof-Steps</strong></div>
<div><br /></div>
<div>The most obvious process that solves this problem is to assign logical probabilities by ignoring most of the starting information, but using as information all proven statements containing no variables (that is, can't regenerate a set of axioms that will require us to take infinite time). So if we prove that our prime number can't simultaneously end with 1 and 3, we'll never assign a combined probability to them greater than 1.</div>
<div><br /></div>
<div>This is the most typical family of suggestions (of the ones that won't require infinite resources). See <a href="/lw/eaa/a_model_of_udt_with_a_concrete_prior_over_logical/">Benja</a>, for example. Another phrasing of this solution is that it's like we start with the inconsistent prior of "1/2 to everything," and then update this according to checked proof steps, until we run out of time. The updating can also be described as if there's a bunch of different tables (models) that assign a true or false value to every statement, and when we learn that the prime can't end with 1 and 3 simultaneously, we rule out the models that say both of those are true and redistribute the probability evenly. To get answers in finite time, we can't actually compute any of these fancy descriptions, but we can compute the updates of just the statement we want.</div>
<div><br /></div>
<div>Even though the probabilities assigned by this approach are more sensible, violations of normal probability still occur at the boundary between proven and unproven statements. We're giving our robot more information to work with, but still not as much as a robot with infinite computing power could use.</div>
<div><br /></div>
<div>A sneaky issue here is that since using checked steps as information takes time, that's less time available to find the solution. This is a general rule - as tricks for assigning logical probabilities to unproven statements get better, they take up more time, so you only want to use them if you don't expect that time to be important. But calculating that tradeoff also takes time! Someone has probably solved this problem in other contexts, but I do not know the solution.</div>
<div><br /></div>
<div><strong>Logical Pattern-Matching</strong></div>
<div><br /></div>
<div>There is another interesting property we might want, which could be called <a href="/lw/igq/a_basis_for_patternmatching_in_logical_uncertainty/">logical pattern-matching</a>. Suppose that our robot is trying to predict a very complicated machine. Further suppose that our robot knows the complete description of the machine, but it is too complicated for our robot to predict the output, or even to find any useful proofs about the machine's behavior.</div>
<div><br /></div>
<div>At time step 1, our robot observes that the machine outputs "1." At time step 2, our robot observes that the machine outputs "2." We might now want our robot to "see the pattern," and guess that the machine is about to output 3.</div>
<div><br /></div>
<div>Our solutions so far don't do this - our robot would need to prove a statement like "if its last output was 2, its next output is logically forbidden from being 5." If our machine is too complicated to prove statements like that about, our previous solutions won't even think of the previous outputs as information.</div>
<div><br /></div>
<div>One way to make our robot care about complexity is to restrict the length of hypotheses to less than the length of the description of the machine. This is like giving the robot the information that the answer comes from a machine, that the description length of this machine is less than some number.</div>
<div><br /></div>
<div>A big problem with this is time. If our robot has to average over the outcome of all these different hypotheses, this takes longer than just using the description itself to find the answer. In a sense, directly using knowledge about the description of the machine is too much for our robot to handle. When we just used the checked proof-steps, that was okay, but as you give the robot more information you also burden it by making it spend more time interpreting that information.</div>
<div><br /></div>
<div>And yet, we want our robot to be able to do logical pattern matching quickly if it actually goes out and observes a complicated machine that prints "1, 2...". But this is another problem I don't know how to solve - we could just say "monte carlo" and wave our hands, but handwaving is frowned upon here, and for good reason.</div>
<div><br /></div>
<div><strong>Further Open Problems</strong></div>
<div><br /></div>
<div>In this post I've already mentioned two open problems: the tradeoff of searching for an exact solution versus having a good approximation, and the correct way to do logical pattern-matching. There are more unsolved problems that also deserve mention.</div>
<div><br /></div>
<div><span style="background-color: #f9f9f9; color: #333333; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px;">• </span>Handling infinities. Current proposals have some pretty bad properties if there are an infinite number of possible answers. For example, if you prove that answers 1-1000 all have small logical probability, but don't prove anything about answer 1001, the robot might decide that since you didn't prove anything about it, it has probability 0.5, and is thus a really good idea. An example direction to go might be to restrict our robot to taking actions it's actually proved things about - but we can also come up with perverse situations where that's bad. Is there a better way?</div>
<div><br /></div>
<div><span style="background-color: #f9f9f9; color: #333333; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px;">• </span>Integrating this approach into a <a href="/lw/jjl/approaching_logical_probability/">larger framework of decision-making</a>. This builds off of making tradeoffs with your computing time and handling infinities. Basically, we want our robot to make decisions in limited time, not just output logical probabilities in limited time, and making decisions requires considering your possible actions and the utility of outcomes, which are allowed to be really complicated and require approximation. And then, we need to somehow direct computational resources into different tasks to make the best decision.</div>
<div><br /></div>
<div><span style="background-color: #f9f9f9; color: #333333; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px;">• I</span>ntegrating this approach with second-order arithmetic. If you look at <a href="http://intelligence.org/wp-content/uploads/2013/03/Christiano-et-al-Naturalistic-reflection-early-draft.pdf">MIRI's paper</a> that uses a probability distribution over logical statements, their approach is quite different - for one, they don't allow for any limits on the robot's resources. And for another, there are all sorts of properties that are important when considering second-order arithmetic that we haven't needed yet. For example, what happens when we ask for P(P(this statement)<0.3)?</div>
</div>
<div><br /></div>
<div><br /></div>
<div>Thank you for reading the Logical Uncertainty sequence. I hope that things which were not obvious now seem obvious. If you want your logical probability distribution to have certain nice properties, it is a good idea to only slightly depart from the original desiderata of probability, and build up from there. Jumping straight to an answer is not necessary, and is probably a bad idea anyhow.</div>
<div><br /></div>
<div><img src="http://oikosjournal.files.wordpress.com/2011/04/thenmiracleoccurs.jpg?w=275&h=300" alt="" width="275" height="299" /><br /></div>
<p style="text-align:right">End of the sequence <em>Logical Uncertainty</em></p>
<p style="text-align:right">Previous Post: <a href="/lw/jjl/approaching_logical_probability/">Approaching Logical Probability</a></p>manfredqiokMkCdFjXa9PTR32014-03-15T06:53:36.039ZMeetup : Urbana-Champaign: Discussion
https://lw2.issarice.com/posts/rGtvaNniygrSbsqKw/meetup-urbana-champaign-discussion
<h2>Discussion article for the meetup : <a href='/meetups/xk'>Urbana-Champaign: Discussion</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">09 March 2014 01:00:02PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">412 W. Elm St., Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Starting topic: should we expect there to be easy ways to improve your life (life hacks)? Two articles that bear on this by <a href="http://www.gwern.net/Drug%20heuristics" rel="nofollow">Gwern</a> and <a href="http://slatestarcodex.com/2014/03/03/do-life-hacks-ever-reach-fixation/" rel="nofollow">Scott</a>.</p>
<p>Also, I'm going to finish up my sequence of posts on logical uncertainty (<a href="http://lesswrong.com/lw/jfx/foundations_of_probability/">starts here</a>) by then, and would be pleased to answer questions.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/xk'>Urbana-Champaign: Discussion</a></h2>manfredrGtvaNniygrSbsqKw2014-03-05T09:14:28.458ZApproaching Logical Probability
https://lw2.issarice.com/posts/GcWjDFsAit7CmzYka/approaching-logical-probability
<p><strong>Followup To:</strong> <a style="text-align: right;" href="/lw/jjk/logic_as_probability/">Logic as Probability</a></p>
<p>If we design a robot that acts as if it's uncertain about mathematical statements, that <a href="/lw/jjk/logic_as_probability/">violates</a> <a href="/lw/jfx/foundations_of_probability/">some desiderata for probability</a>. But realistic robots cannot prove all theorems; they have to be uncertain about hard math problems.</p>
<p>In the name of practicality, we want a foundation for decision-making that captures what it means to make a good decision, even with limited resources. "Good" means that even though our real-world robot can't make decisions well enough to satisfy Savage's theorem, we want to approximate that ideal, not throw it out. Although I don't have the one best answer to give you, in this post we'll take some steps forward.</p>
<p><a id="more"></a></p>
<div>The objects we call probabilities are specified by desiderata that tell us how they behave. Any uncertainty about math problems violates those desiderata, but we still want to be able to assign logical probabilities that behave a lot like probabilities. The basic principles - not making up or ignoring information, not throwing away money or being inconsistent - should be deviated from as little as possible even when computing power is scarce. We want to develop a foundation for logical probabilities by starting from the rules governing ordinary probability, and then minimally restricting the application of those rules.</div>
<div><br /></div>
<div>As we do this, it's important to keep track of what changes we make and why. Sometimes people just define logical probabilities, without worrying about desiderata. This is fine, when it works, and is often patchable if it doesn't have the right properties. But if you use it for something important and and get a surprise failure, it's really bad. My hope here is to construct logical probabilities that have the good properties, while keeping <a href="http://www.catb.org/jargon/html/H/handwave.html">handwaving</a> and mysterious assumptions to a minimum.</div>
<div><br /></div>
<div>The perils of handwaving are more dire than they appear, and they are at their most dire in the hardest and most confusing reaches of physics. After better approaches fail, many theorists resort to just making up approximations and then trying to justify them. Doing this is known colloquially as a "1/ego expansion." Simply put, it doesn't work; there are too many vital little details. It's why even condensed matter theorists tell you not to trust condensed matter theorists about high temperature superconductivity.</div>
<div><br /></div>
<div><br /></div>
<div>We must abandon regular probabilities because our robot has limited time, but other parts of the decision-making process can also go over the time limit. If the robot's resources are limited, expected utility maximization breaks down at many points: there might be too many strategies to search through, too many outcomes to foresee, there might be probabilities that are too hard to find, and the utility of the outcomes might be too complicated.</div>
<div><br /></div>
<div>The logical probabilities considered in this sequence will help approximate hard math problems, but they don't seem to help much when there are too many outcomes to consider, or if you want to make the best use of limited computational resources. They are only a part of the full solution.</div>
<div><br /></div>
<div><br /></div>
<div>Time for a desideratum: we want our robot to only assign a logical probability of 1 or 0 to a statement after it's actually checked the proof of that statement.</div>
<div><br /></div>
<div>We can think of this as limiting what statements our robot is allowed to be certain about - only statements with short proofs can be found by our agent. However, this desideratum is not just about proof length, because a real robot won't check every checkable proof - it will spend time generating proofs, maybe trying to prove some specific statement, and will end up only checking some subset of short proofs.</div>
<div><br /></div>
<div>Logical probabilities, unlike probabilities, are not determined just by the starting information. If our real robot only verifies some small collection of proofs, the robot's logical probabilities depend heavily upon what proof-steps were checked. One proof-step is just one <a href="/lw/f43/proofs_implications_and_models/">truth-preserving</a> step by our robot, like one <a href="/r/lesswrong/lw/jjk/logic_as_probability/">application of modus ponens</a> - it's a little proof one step long. The import is that they're the atomic unit of proofs, and once all the steps of a proof are checked, the proof is checked.</div>
<div><br /></div>
<div>If we condition on which proof-steps get checked, does that determine the logical probabilities?</div>
<div><br /></div>
<div>For any statement our robot is going to prove or disprove, we can use the checked proof steps to find whether it's logical probability 1 or 0. This gives the same answer as a real robot that checks steps according to some process and then returns 1 or 0 if it manages to prove or disprove the statement we give it. We just have to take the steps that the real robot ends up checking, and say that those are the proved steps for our abstract mathematical definition.</div>
<div><br /></div>
<div>There's a problem, though. We haven't changed the old axioms, so they're still only satisfied if we get the right answer for everything. Meanwhile our new desideratum says we can't get the right answer for everything - we've made our axioms internally contradictory. In order to talk about the logical probabilities of unproven statements, we'll need to weaken the original axioms so that they no longer require certainty about everything. We'll explore ways to do this next time. Then we can assign numbers to statements in the usual way, by using our weakened axioms to find constraints, then maximizing entropy subject to those constraints.</div>
<div><br /></div>
<div><br /></div>
<p style="text-align:right">Part of the sequence <em>Logical Uncertainty</em></p>
<p style="text-align:right">Previous Post: <a href="/lw/jjk/logic_as_probability/">Logic as Probability</a></p>
<p style="text-align:right">Next post: <a href="/lw/joy/solutions_and_open_problems/">Solutions and Open Problems</a></p>manfredGcWjDFsAit7CmzYka2014-02-27T07:44:38.172ZMeetup : Urbana-Champaign: Bridging laws
https://lw2.issarice.com/posts/ejMWnaJW9q6ioGZBJ/meetup-urbana-champaign-bridging-laws
<h2>Discussion article for the meetup : <a href="/meetups/x1">Urbana-Champaign: Bridging laws</a></h2>
<div class="meetup-meta">
<p><strong>WHEN:</strong> <span class="date">23 February 2014 02:00:00PM (-0600)</span></p>
<p><strong>WHERE:</strong> <span class="address">412 W. Elm St, Urbana, IL</span></p>
</div>
<!-- .meta -->
<div class="content">
<div class="md">
<p><a href="/user/RobbBB/submitted/">RobBB</a> has made some recent posts about how an agent's representation of the world can interact with its perceptions and plans. That seems like a good topic, so let's discuss it.</p>
<p>Bonus question: If an agent represents its utility as a function over states of the world including itself, can it still wirehead?</p>
</div>
</div>
<!-- .content -->
<h2>Discussion article for the meetup : <a href="/meetups/x1">Urbana-Champaign: Bridging laws</a></h2>manfredejMWnaJW9q6ioGZBJ2014-02-20T03:23:31.605ZLogic as Probability
https://lw2.issarice.com/posts/WKkDD6u79pzmvyQ6a/logic-as-probability
<p><strong>Followup To:</strong> <a href="/lw/jfl/putting_in_the_numbers/">Putting in the Numbers</a></p>
<p>Before talking about logical uncertainty, our final topic is the relationship between probabilistic logic and classical logic. A robot running on probabilistic logic stores probabilities of events, e.g. that the grass is wet outside, P(wet), and then if they collect new evidence they update that probability to P(wet|evidence). Classical logic robots, on the other hand, deduce the truth of statements from axioms and observations. Maybe our robot starts out not being able to deduce whether the grass is wet, but then they observe that it is raining, and so they use an axiom about rain causing wetness to deduce that "the grass is wet" is true.</p>
<p>Classical logic relies on complete certainty in its axioms and observations, and makes completely certain deductions. This is unrealistic when applied to rain, but we're going to apply this to (<a href="/lw/g1y/godels_completeness_and_incompleteness_theorems/">first order</a>, for starters) math later, which a better fit for classical logic.</p>
<p>The general pattern of the deduction "It's raining, and when it rains the grass is wet, therefore the grass is wet" was modus ponens: if 'U implies R' is true, and U is true, then R must be true. There is also modus tollens: if 'U implies R' is true, and R is false, then U has to be false too. Third, there is the law of non-contradiction: "It's simultaneously raining and not-raining outside" is always false.</p>
<p>We can imagine a robot that does classical logic as if it were writing in a notebook. Axioms are entered in the notebook at the start. Then our robot starts writing down statements that can be deduced by modus ponens or modus tollens. Eventually, the notebook is filled with statements deducible from the axioms. Modus tollens and modus ponens can be thought of as consistency conditions that apply to the contents of the notebook.</p>
<p><a id="more"></a>Doing math is one important application of our classical-logic robot. The robot can read from its notebook "If variable A is a number, A=A+0" and "SS0 is a number," and then write down "SS0=SS0+0."</p>
<p>Note that this requires the robot to interpret variable A differently than symbol SS0. This is one of many upgrades we can make to the basic robot so that it can interpret math more easily. We also want to program in special responses to symbols like 'and', so that if A and B are in the notebook our robot will write 'A and B', and if 'A and B' is in the notebook it will add in A and B. In this light, modus ponens is just the robot having a programmed response to the 'implies' symbol.</p>
<p>Certainty about our axioms is what lets us use classical logic, but you can represent complete certainty in probabilistic logic too, by the probabilities 1 and 0. These two methods of reasoning shouldn't contradict each other - if a classical logic robot can deduce that it's raining out, a probabilistic logic robot with the same information should assign P(rain)=1.</p>
<p>If it's raining out, then my grass is wet. In the language of probabilities, this is P(wet|rain)=1. If I look outside and see rain, P(rain)=1, and then the product rule says that P(wet and rain) = P(rain)·P(wet|rain), and that's equal to 1, so my grass must be wet too. Hey, that's modus ponens!</p>
<p>The rules of probability can also behave like modus tollens (if P(B)=0, and P(B|A)=1, P(A)=0) and the law of the excluded middle (P(A|not-A)=0). Thus, when we're completely certain, probabilistic logic and classical logic give the same answers.</p>
<p>There's a very short way to prove this, which is that one of <a href="/lw/jfx/foundations_of_probability/">Cox's desiderata</a> for how probabilities must behave was "when you're completely certain, your plausibilities should satisfy the rules of classical logic."</p>
<p>In <a href="/lw/jfx/foundations_of_probability/">Foundations of Probability</a>, I alluded to the idea that we should be able to apply probabilities to math. Dutch book arguments work because our robot must act as if it had probabilities in order to avoid losing money. Savage's theorem applies because the results of our robot's actions might depend on mathematical results. Cox's theorem applies because beliefs about math behave like other beliefs.</p>
<p>This is completely correct. Math follows the rules of probability, and thus can be described with probabilities, because classical logic is the same as probabilistic logic when you're certain.</p>
<p>We can even use this correspondence to figure out what numbers the probabilities take on:</p>
<p>1 for every statement that follows from the axioms, 0 for their negations.</p>
<p> </p>
<p>This raises an issue: what about betting on the last digit of the 3^^^3'th prime? We dragged probability into this mess because it was supposed to help our robot stop trying to prove the answer and just bet as if P(last digit is 1)=1/4. But it turns out that there is one true probability distribution over mathematical statements, given the axioms. The right distribution is obtained by straightforward application of the product rule - never mind that it takes 4^^^3 steps - and if you deviate from the right distribution that means you violate the product rule at some point.</p>
<p>This is why logical uncertainty is different. Even though our robot doesn't have enough resources to find the right answer, using logical uncertainty violates <a href="/lw/jfx/foundations_of_probability/">Savage's theorem and Cox's theorem</a>. If we want our robot to act as if it has some "logical probability," it's going to need a stranger sort of foundation.</p>
<p> </p>
<p style="text-align:right">Part of the sequence <em>Logical Uncertainty</em></p>
<p style="text-align:right">Previous Post: <a href="/lw/jfl/putting_in_the_numbers/">Putting in the Numbers</a></p>
<p style="text-align:right">Next post: <a href="/lw/jjl/approaching_logical_probability/">Approaching Logical Uncertainty</a></p>manfredWKkDD6u79pzmvyQ6a2014-02-08T06:39:36.824ZMeetup : Urbana-Champaign: Logical uncertainty
https://lw2.issarice.com/posts/fFZBmH9Ry2dCoW8te/meetup-urbana-champaign-logical-uncertainty
<h2>Discussion article for the meetup : <a href='/meetups/wj'>Urbana-Champaign: Logical uncertainty</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">09 February 2014 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">412 W. Elm St, Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>I'm writing a sequence on logical uncertainty, so come ask me anything about it. Digressions encouraged. There may also be cookies.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/wj'>Urbana-Champaign: Logical uncertainty</a></h2>manfredfFZBmH9Ry2dCoW8te2014-02-05T04:12:05.662ZPutting in the Numbers
https://lw2.issarice.com/posts/bnFP4yWmsFxaKjg3E/putting-in-the-numbers
<p><strong>Followup To:</strong> <a href="/lw/jfx/foundations_of_probability/">Foundations of Probability</a></p>
<p>In the previous post, we reviewed reasons why having probabilities is a good idea. These foundations defined probabilities as numbers following certain rules, like the product rule and the rule that mutually exclusive probabilities sum to 1 at most. These probabilities have to hang together as a coherent whole. But just because probabilities hang together a certain way, doesn't actually tell us what numbers to assign.</p>
<p>I can say a coin flip has P(heads)=0.5, or I can say it has P(heads)=0.999; both are perfectly valid probabilities, as long as P(tails) is consistent. This post will be about how to actually get to the numbers.</p>
<p><a id="more"></a></p>
<p>If the probabilities aren't fully determined by our desiderata, what do we need to determine the probabilities? More desiderata!</p>
<p>Our final desideratum is motivated by the perspective that our probability is based on some state of information. This is acknowledged explicitly in Cox's scheme, but is also just a physical necessity for any robot we build. Thus we add our new desideratum: Assign probabilities that are consistent with the information you have, but don't make up any extra information. It turns out this is enough to let us put numbers to the probabilities.</p>
<p>In its simplest form, this desideratum is a symmetry principle. If you have the exact same information about two events, you should assign them the same probability - giving them different probabilities would be making up extra information. So if your background information is "Flip a coin, the mutually exclusive and exhaustive probabilities are heads and tails," there is a symmetry between the labels "heads" and "tails," which given our new desideratum lets us assign each P=0.5.</p>
<p>Sometimes, though, we need to pull out the information theory. Using the fact that it doesn't produce information to split the probabilities up differently, we can specify something called "information entropy" (For more thoroughness, see chapter 11 of <a href="http://www-biba.inrialpes.fr/Jaynes/prob.html">Jaynes</a>). The entropy of a probability distribution is a function that measures how uncertain you are. If I flip a coin and don't know about the outcome, I have one bit of entropy. If I flip two coins, I have two bits of entropy. In this way, the entropy is like the amount of information you're "missing" about the coin flips.<img style="margin: 10px;" src="http://images.lesswrong.com/t3_jfl_0.png?v=db23affc813431eb24d7e9748fc7fa8f" alt="Entropy of weighted coin" width="318" height="260" align="right" /></p>
<p>The mathematical expression for information entropy is that it's the sum of each probability multiplied by its log. Entropy = -Sum( P(x)<span style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;"><strong>·</strong></span>Log(P(x)) ), where the events x are mutually exclusive. Assigning probabilities is all about maximizing the entropy while obeying the constraints of our prior information.</p>
<p>Suppose we roll a 4-sided die. Our starting information consists of our knowledge that there are sides numbered 1 to 4 (events 1, 2, 3, and 4 are exhaustive), and the die will land on just one of these sides (they're mutually exclusive). This let's us write our information entropy as -P(1)<strong style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;">·</strong>Log(P(1)) - P(2)<strong style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;">·</strong>Log(P(2)) - P(3)<strong style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;">·</strong>Log(P(3)) - P(4)<strong style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;">·</strong>Log(P(4)).</p>
<p>Finding the probabilities is a maximization problem, subject to the constraints of our prior information. For the simple 4-sided die, our information just says that the probabilities have to add to 1. Simply knowing the fact that the entropy is concave down tells us that to maximize entropy we should split it up as evenly as possible - each side has a 1/4 chance of showing.</p>
<p>That was pretty commonsensical. To showcase the power of maximizing information entropy, we can add an extra constraint.</p>
<p>If we have additional knowledge that the average roll of our die is 3, then we want to maximize -P(1)<strong style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;">·</strong>Log(P(1)) - P(2)<strong style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;">·</strong>Log(P(2)) - P(3)<strong style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;">·</strong>Log(P(3)) - P(4)<strong style="color: #444444; font-family: arial, sans-serif; line-height: 14.654545783996582px;">·</strong>Log(P(4)), given that the sum is 1 and the average is 3. We can either plug in the constraints and set partial derivatives to zero, or we can use a maximization technique like Lagrange multipliers.</p>
<p>When we do this (again, more details in Jaynes ch. 11), it turns out the the probability distribution is shaped like an exponential curve. Which was unintuitive to me - my intuition likes straight lines. But it makes sense if you think about the partial derivative of the information entropy: 1+Log(P) = [some Lagrange multiplier constraints]. The steepness of the exponential controls how shifted the average roll is.</p>
<p> </p>
<p>The need for this extra desideratum has not always been obvious. People are able to intuitively figure out that a fair coin lands heads with probability 0.5. Seeing that their intuition is so useful, some people include that intuition as a fundamental part of their method of probability. The counter to this is to focus on constructing a robot, which only has those intuitions we can specify unambiguously.</p>
<p>Another alternative to assigning probabilities based on maximum entropy is to pick a standard prior and use that. Sometimes this works wonderfully - it would be silly to rederive the binomial distribution every time you run into a coin-flipping problem. But sometimes people will use a well-known prior even if it doesn't match the information they have, just because their procedure is "use a well-known prior." The only way to be safe from that mistake and from interminable disputes over "which prior is right" is to remember that a prior is only correct insofar as it captures some state of information.</p>
<p>Next post, we will finally get to the problem of logical uncertainty, which will shake our foundations a bit. But I really like the principle of not making up information - even a robot that can't do hard math problems can aspire to not make up information.</p>
<p> </p>
<p style="text-align:right">Part of the sequence <em>Logical Uncertainty</em></p>
<p style="text-align:right">Previous Post: <a href="/lw/jfx/foundations_of_probability/">Foundations of Probability</a></p>
<p style="text-align:right">Next post: <a href="/lw/jjk/logic_as_probability/">Logic as Probability</a></p>manfredbnFP4yWmsFxaKjg3E2014-01-30T06:41:42.112ZFoundations of Probability
https://lw2.issarice.com/posts/EQ33emneF3Fh62Nn2/foundations-of-probability
<h3><strong style="font-size: small;">Beginning of:</strong><span style="font-size: small;"> </span><span style="font-size: small; font-weight: normal;">Logical Uncertainty sequence</span></h3>
<p>Suppose that we are designing a robot. In order for this robot to reason about the outside world, it will need to use probabilities.</p>
<p>Our robot can then use its knowledge to acquire cookies, which we have programmed it to value. For example, we might wager a cookie with the robot on the motion of a certain stock price.</p>
<p>In the coming sequence, I'd like to add a new capability to our robot. It has to do with how the robot handles very hard math problems. If we ask "what's the last digit of the <a href="http://en.wikipedia.org/wiki/Knuth's_up-arrow_notation">3^^^3</a>'th prime number?", our robot should at some point <em>give up</em>, before the sun explodes and the point becomes moot.</p>
<p>If there are math problems our robot can't solve, what should it do if we offer it a bet about the last digit of the 3^^^3'th prime? It's going to have to approximate - robots need to make lots of approximations, even for simple tasks like finding the strategy that maximizes cookies.</p>
<p>Intuitively, it seems like if we can't find the real answer, the last digit is equally likely to be 1, 3, 7 or 9; our robot should take bets as if it assigned those digits equal probability. But to assign some probability to the wrong answer is logically equivalent to assigning probability to 0=1. When we learn more, it will become clear that this is a problem - we aren't ready to upgrade our robot yet.</p>
<p>Let's begin with a review of the foundations of probability.</p>
<div>
<p><a id="more"></a></p>
</div>
<p>What I call foundations of probability are arguments for why our robot should ever want to use probabilities. I will cover four of them, ranging from the worldly ("make bets in the following way or you lose money") to the ethereal ("here's a really elegant set of axioms"). To use the word "probability" to describe the subject of such disparate arguments can seem odd, but keep in mind the naive definition of probability as that number that's 1/6 for a fair die rolling 6 and 30% for clear weather tomorrow.</p>
<p><strong>Dutch Books</strong></p>
<p>The concretest of concrete foundations is the Dutch book arguments. A Dutch book is a collection of bets that is certain to lose you money. If you violate the rules of probability, you'll agree to these certain-loss bets (or not take a certain-win bet).</p>
<p>For example, if you think that each side of the coin has a 55% chance of showing up, then you'll pay $1 for a bet that pays out $0.98 if the coin lands heads and $0.98 if the coin lands tails. If taking bets where you're guaranteed to lose is bad, then you're not allowed to have probabilities for mutually exclusive things that sum to more than 1.</p>
<p>Similar arguments hold for other properties of probability. If your probabilities for exhaustive events add up to less than 1, you'll pass up free money, which is bad. If you disobey the sum rule or the product rule, you'll agree to a guaranteed loss, which is bad, etcetera. Thus, say the Dutch book arguments, our probabilities have to behave the way they do because we don't want to take guaranteed losses or pass up free money.</p>
<p>There are many assumptions underlying this whole scenario. Our agent in these arguments already tries to decide using probability-like numbers, all we show is that the numbers have to follow the same rules as probabilities. Why can't our agent follow a totally different method of decision making, like picking randomly or alphabetization?</p>
<p>One can show that e.g. picking randomly will sometimes throw away money. But there is a deeper principle here: an agent that wants to avoid throwing away money or passing up free money has to act <em>as if</em> it had numbers that followed probability-rules, and that's a good enough reason for our agent to have probabilities.</p>
<p>Still, some people dislike Dutch book arguments because they focus on an extreme scenario where a malicious bookie is trying to exploit our agent. To avoid this, we'll need a more abstract foundation.</p>
<p>You can learn more about Dutch book arguments <a href="http://plato.stanford.edu/entries/epistemology-bayesian/supplement2.html">here</a> and <a href="http://m-phi.blogspot.com/2013/09/the-mathematics-of-dutch-book-arguments.html">here</a>.</p>
<p><strong>Savage's Foundation</strong></p>
<p>Leonard Savage formulated a basis for decision-making that is sort of a grown-up version of Dutch book arguments. From seven desiderata, none of which mention probability, he derived that an agent that wants to act consistently will act as if it had probabilistic beliefs.</p>
<p>What are the desiderata about, if not probability? They define an agent that has preferences, and is able to take actions, which are defined as things that lead to outcomes, and can lead to different outcomes depending on external possibilities in event-space. They require that the agent's actions be consistent in commonsensical ways. These requirements are sufficient to show that assigning probabilities to the external events is the best way to do things.</p>
<p>Savage's theorem provides one set of conditions for when we should use probabilities. But it doesn't help us choose which probabilities to assign - anything consistent works. The idea that probabilities are degrees of belief, and that they are derived from some starting information, is left to our next foundation.</p>
<p>You can learn more about Savage's foundation <a href="http://www.econ2.jhu.edu/people/Karni/savageseu.pdf">here</a>.</p>
<p><strong>Cox's Theorem</strong></p>
<p>Cox's theorem is a break from justifying probabilities with gambling. Rather than starting from an agent that wants to achieve good outcomes, and showing that having probabilities is a good idea, Richard Cox started with desired properties of a "degree of plausibility," and showed that probabilities are what a good belief-number should be.</p>
<p>One special facet of Cox's desiderata is that they refer to plausibility of an event, given your information - what will eventually become P(event | information).</p>
<p>There are six or so desiderata, but I think there are three interesting ones: When you're completely certain, your plausibilities should satisfy the rules of classical logic. Every rational plausibility has at least one event with that plausibility. P(A and B|X) can be found as a function of P(A|X) and P(B|A and X).</p>
<p>These desiderata are a motley assortment. The desideratum that there's an infinite variety of events is the most strange, but it is satisfied if our universe contains a continuous random process or if we can flip a coin as many times as we want. If the desiderata obtain, Cox's theorem shows that we can give pretty much any belief a probability. The perspective of Cox's theorem is useful because it lets us keep talking straightforwardly about probabilities even if betting or decision-making has become nontrivial.</p>
<p>You can learn more about Cox's theorem in the first two chapters of Jaynes <a href="http://www-biba.inrialpes.fr/Jaynes/prob.html">here</a> (in fact, the next few posts are parallel to the first two chapters of Jaynes), and also <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13.4276&rep=rep1&type=pdf">here</a>. Jaynes includes an additional desideratum in this foundation, which we will cover in the next post.</p>
<p><strong>Kolmogorov Axioms</strong></p>
<p>At the far extreme of abstraction, we have the Kolmogorov axioms for probability. Here they are:</p>
<p>P(E) is a non-negative real number, E is an event that belongs to event-space F.</p>
<p>P(some event occurs)=1.</p>
<p>Any countable sequence of disjoint events (E1, E2...) satisfies P(E1 or E2 or...) = sum of all the P(E).</p>
<p>Though it was not their intended purpose, these can be seen as a Cox-style list of desiderata for degrees of plausibility. Their main virtue is that they're simple and handy to mathematicians who like set theory.</p>
<p>You can learn more about Kolmogorov's axioms <a href="http://en.wikipedia.org/wiki/Probability_axioms">here</a>.</p>
<p> </p>
<p>Look back at our robot trying to bet on the 3^^^3'th prime number. Our robot has preferences, so it can be Dutch booked. Its reward depends on the math problem and we want it to act consistently, so Savage's theorem applies. Cox's theorem applies if we allow our robot to make combined bets on math and dice. It even seems like the Kolmogorov axioms should hold. Resting upon these foundations, our robot should assign numbers to mathematical statements, and they should behave like probabilities.</p>
<p>But we can't get specific about that, because we have a problem - we don't know how to actually find the numbers yet. Our foundations tell us that the probabilities of the two sides of a coin will add to 1, but they don't care whether P(heads) is 0.5 or 0.99999. If Dutch book arguments can't tell us that a coin lands heads half the time, what can? Tune in next time to find out.</p>
<p> </p>
<p style="text-align:right">First post in the sequence <em>Logical Uncertainty</em></p>
<p style="text-align:right">Next post: <a href="/lw/jfl/putting_in_the_numbers/">Putting in the Numbers</a></p>manfredEQ33emneF3Fh62Nn22014-01-26T19:29:42.378ZMeetup : Urbana-Champaign: Probability Discussion
https://lw2.issarice.com/posts/7uwK9wjcmDtA2MnWi/meetup-urbana-champaign-probability-discussion
<h2>Discussion article for the meetup : <a href='/meetups/w7'>Urbana-Champaign: Probability Discussion</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">26 January 2014 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">412 W. Elm St, Urbana, IL</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Let's have a meetup! I'm writing an article on the foundations of probability - why having probabilities is a good idea for agents. So that would be a good thing to talk about. And then, other things!</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/w7'>Urbana-Champaign: Probability Discussion</a></h2>manfred7uwK9wjcmDtA2MnWi2014-01-25T03:16:09.989ZDouble-thick transistors and other subjective phenomena
https://lw2.issarice.com/posts/bJyYdyiESgfDsevw9/double-thick-transistors-and-other-subjective-phenomena
<p>If I'm running on a silicon computer, do I have twice as much subjective experience if my computer is twice as thick?</p>
<p>Why is this even a good question?</p>
<p>Consider a computer that was printed on a flat sheet. If we stick two of these computers (one a mirror image) together face to face, we get a thicker computer. And then if we peel them apart again, we get two thin computers! Suppose that we simulate a person using these computers. It makes sense that a person running on two thin computers has twice as much "experience" as a person running on just one (for example, in the Sleeping Beauty problem, the correct betting strategy is to bet as if the probability of making the bet in a given world is proportional to the number of thin computers). So if we take two people-computers and stick them together into one thicker person-computer, the thicker person contains twice as much "experience" as a thinner one - each of their halves has as much experience as a thin person, so they have twice as much experience.</p>
<p>Do I disagree? Well, I think it depends somewhat on how you cash out "experience." Consider the Sleeping Beauty problem with these computers - in the classic version, our person is asked to give their probability that they're in the possibility where there's one thin computer, or the world where there are two thin computers. The correct betting strategy is to bet as if you think the probability that there are two computers is 2/3 - weighting each computer equally.</p>
<p>Now, consider altering the experiment so that either there's one thin computer, or one double computer. We have two possibilities - either the correct betting probability is 1/2 and the computers seem to have equal "experience", or we bite the bullet and say that the correct betting probability is 2/3 for a double computer, 10/11 for a 10x thicker computer, 1000/1001 for a 1000x thicker computer, etc.</p>
<p>The bullet-biting scenario is equivalent to saying that the selfish desires of the twice-thick computer are twice as important. If one computer is one person, a double computer is then two people in a box.</p>
<p>But of course, if you have a box with two people in it, you can knock on the side and go "hey, how many of you people are in there? I'm putting in an order for chinese food, how many entrees should I get?" Instead, the double-thick computer is running exactly the same program as the thin computer, and will order exactly the same number of entrees. In particular, a double-thick computer will make evaluations of selfish vs. altruistic priorities exactly the same as a thin computer.</p>
<p>There is one exception to the previous paragraph - what if the computer is programmed to care about its own thickness, and measure it with external instruments since introspection won't do, and weight its desires more when it's thicker? This is certainly possible, but by putting the caring straight into the utility function, it removes any possibility that the caring is some mysterious "experience." It's just a term in the utility function - it doesn't have to be there, in fact by default it's not. Or, heck, your robot might as easily care more about things when the tides are high, that doesn't mean that high tides grant "experience."</p>
<p>The original Sleeping Beauty problem, now *that's* mysterious "experience." Ordinary computers enter, weighting the possibility by the number of computers leaves. So something happens when you merge the two computers into a double computer, to destroy that experience rather than conserving it.</p>
<p>What do I claim explains this? The simple fact that you only offer the double computer one bet, not two. Sure, the exact same signals go to the exact same wires in each case. Except for the prior information that says that the experimenter can only make 1 bet, not 2. In this sense, "experience" just comes from the ways in which our computer can interact with the world.</p>
<p>So since the a double-thick computer is not more selfish than a thin one (neglecting the tides), and will not expect to be a thick computer more often in the Sleeping Beauty problem, I'd say it doesn't have more "experience" than a thin computer.</p>
<p>EDIT: I use betting behavior as a proxy for probability here because it's easy to see which answer is correct. However, using betting behavior as a probability is not always valid - e.g. in the absent-minded driver problem. In the sleeping beauty case it only works because the payout structure is very simple. A safer way would be to derive the probabilities from the information available to the agents, which has been done elsewhere, but is harder to follow.</p>manfredbJyYdyiESgfDsevw92014-01-12T19:32:41.358ZMeetup : Urbana-Champaign fun and games
https://lw2.issarice.com/posts/xDDDoijKm3z62pFaY/meetup-urbana-champaign-fun-and-games-1
<h2>Discussion article for the meetup : <a href='/meetups/ug'>Urbana-Champaign fun and games</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">07 December 2013 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">412 W. Elm St., Urbana, IL, 61801</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Come drop by my house for board games, food, and word-talky-stuff.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/ug'>Urbana-Champaign fun and games</a></h2>manfredxDDDoijKm3z62pFaY2013-12-06T01:46:51.570ZMeetup : Urbana-Champaign fun and games
https://lw2.issarice.com/posts/HZbygXYT2jwheJ2kT/meetup-urbana-champaign-fun-and-games-0
<h2>Discussion article for the meetup : <a href='/meetups/ts'>Urbana-Champaign fun and games</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">24 November 2013 02:00:00PM (-0600)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">412 W. Elm St, Urbana, Illinois</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>We'll be meeting at a house to copy those Washington DC people and play games. If you have never played Wits and Wagers, you should definitely come.</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/ts'>Urbana-Champaign fun and games</a></h2>manfredHZbygXYT2jwheJ2kT2013-11-20T04:47:47.021ZGame theory and expected opponents
https://lw2.issarice.com/posts/tsY7HBrRaPvN2Sdm9/game-theory-and-expected-opponents
<p>Thanks to V_V and Emile for some great discussion. Since writing up a post seems to reliably spark interesting comments, that's what I'll do!</p>
<p> </p>
<p><strong>Summary</strong></p>
<p>If I wanted to write down a decision theory that gets the correct answer to game-theoretic problems (like playing the middle Nash equilibrium in a blind <a href="/r/discussion/lw/ixl/kidnapping_and_the_game_of_chicken/">chicken-like game</a>), it would have to, in a sense, implement all of game theory. This is hard because human-generated solutions to games use a lot of assumptions about what the other players will do, and putting those assumptions into our algorithm is a confusing problem. In order to tell what's really going on, we need to make that information more explicit. Once we do that, maybe we can get a UDT-like algorithm to make good moves in tricky games.</p>
<p> </p>
<p><strong>Newcomb's Problem</strong></p>
<p>For an example of a game with unusually good information about our opponent, how about Newcomb's problem. Is it really a game, you ask? Sure, I say!</p>
<p><img style="float: right;" src="http://images.lesswrong.com/t3_j1t_0.png" alt="" width="381" height="275" />In the payoff matrix to the right, you play red and Omega plays blue. The numbers for Omega just indicate that Omega only wants to put in the million dollars if you will 1-box. If this was a normal game-theory situation, you wouldn't easily know what to do - your best move depends on Omega's move. This is where typical game theory procedure would be to say "well, that's silly, let's specify some extra nice properties the choice of both players should have so that we get a unique solution."</p>
<p>But the route taken in Newcomb's problem is different - we pick out a unique solution by increasing how much information the players have about each other. Omega knows what you will play, and you know that Omega knows what you will play. Now all we need to figure out what to do is some information like "If Omega has an available strategy that will definitely get it the highest possible payoff, it will take it." The best strategy, of course, is to one-box so that Omega puts in the million dollars.</p>
<p> </p>
<p><strong>Newcomb's Game vs. an Ignorant opponent</strong></p>
<p>Consider another possible opponent in this game - one who has no information about what your move will be. Whereas Omega always knows your move, an ignorant opponent has no idea what you will play - they have no basis to think you're more likely to 1-box than 2-box, or vice versa.</p>
<p>Interestingly, for this particular payoff matrix this makes you ignorant too - you have no basis to think the ignorant opponent would rather put the money in than not, or vice versa. So you assign a 50% chance to each (probability being quantified ignorance) and find that two-boxing has the highest rewards. This didn't even require the sophistication of taking into account your own action, like the game against Omega did, since an ignorant opponent can't respond to your action.</p>
<p> </p>
<p><strong>Human opponents</strong></p>
<p>Ok, so we've looked at a super-knowledgeable opponent, and a super-ignorant opponent, what does a more typical game theory situation look like? Well, it's when our opponent is more like us - someone trying to pick the strategy that gets them the best reward, with similar information to what we have. In typical games between humans, both know that the other is a human player - and they know that it's known, etc.</p>
<p>In terms of what we know, we know that our opponent is drawn from some distribution of opponents that are about as good as we are at games, and that they have the same information about us that we have about them.</p>
<p>What information do we mean we have when we say our opponent is "good at games"? I don't know. I can lay out some possibilities, but this is the crux of the post. I'll frame our possible knowledge in terms of past games, like how one could say of Newcomb's problem "you observe a thousand games, and Omega always predicts right."</p>
<p>Possibility 1: We know our opponent has played a lot of games against completely unknown opponents in the past, and has a good record, where "good" means "as good or better than the average opponent."</p>
<p>Possibility 2: We know our opponent played some games against a closed group of players who played each other, and that group collectively had a good record.</p>
<p>Possibility 3: We know our opponent is a neural net that's been trained in some standard way to be good at playing a variety of games, or some sort of hacked-together implementation of game theory, or a UDT agent if that's a good idea. (Seems more complicated than necessary, but on the other hand opponents are totally allowed to be complicated)</p>
<p> </p>
<p>Suppose we know information set #2. I think it's the most straightforward. All we have to do to turn this information into a distribution over opponents is to figure out what mixtures of players get above-average group results, then average those together. Once we know who our opponent is on average, we just follow the strategy that on average gets the best average payoff.</p>
<p>Does the strategy picked this way look like what game theory would say? Not quite - it assumes that the opponent has a medium chance of being stupid. And in some games, like the prisoner's dilemma, the best-payoff groups are actually the ones you can exploit the most. So on closer examination, someone in a successful group isn't the game-theory opponent we're looking for.</p>manfredtsY7HBrRaPvN2Sdm92013-11-14T23:26:19.693ZKidnapping and the game of Chicken
https://lw2.issarice.com/posts/GQy2BSQG9Dd6vPhs8/kidnapping-and-the-game-of-chicken
<p><br /><img style="float: right; margin: 10px;" src="http://images.lesswrong.com/t3_ixl_0.png" alt="(0,0) (1,2) \n (2,1) (0,0)" width="239" height="188" />Observe the payoff matrix at right (the unit of reward? Cookies.). Each player wants to play 'A', but only so long as the two players play different moves.</p>
<p>Suppose that Red got to move first. There are some games where moving first is terrible - take Rock Paper Scissors for example. But in this game, moving first is great, because you get to narrow down your opponent's options! If Red goes first, Red picks 'A', and then Blue has to pick 'B' to get a cookie.</p>
<p>This is basically kidnapping. Red has taken all three cookies hostage, and nobody gets any cookies unless Blue agrees to Red's demands for two cookies. Whoever gets to move first plays the kidnapper, and the other player has to decide whether to accede to their ransom demand in exchange for a cookie.</p>
<p> </p>
<p>What if neither player gets to move before the other, but instead they have their moves revealed at the same time?</p>
<p><em>Pre-Move Chat:</em></p>
<p><em>Red: "I'm going to pick A, you'd better pick B."</em></p>
<p><em>Blue: "I don't care what you pick, I'm picking A. You can pick A too if you really want to get 0 cookies."</em></p>
<p><em>Red: "Okay I'm really seriously going to pick A. Please pick B."</em></p>
<p><em>Blue: "Nah, don't think so. I'll just pick A. You should just pick B."</em></p>
<p>And so on. They are now playing a game of Chicken. Whoever swerves first is worse off, but if neither of them give in, they crash into each other and die and get no cookies.</p>
<p> </p>
<p>So, The Question: is it better to play A, or to play B?</p>
<div>This is definitely a trick question, but it can't be <em>too</em> trickish because at some point Red and Blue will have to figure out what to do. So why is it a trick question?</div>
<div><br /></div>
<div>Because this is a two-player game, and whether it's good to play A or not depends on what your opponent will do.</div>
<div><br /></div>
<div><br /></div>
<div>A thought experiment: suppose we threw a party where you could only get dessert (cookies!) by playing this game. At the start, people are unfamiliar with the game, but they recognize that A has higher payoffs than B, so they all pick A all the time. But alas! When both people pick A, neither get anything, so no cookies are distributed. We decide that everyone can play as much as they want until we run out of cookies.</div>
<div><br /></div>
<div>Quite soon, one kind soul decides that they will play B, even though it has a lower payoff. A new round of games is begun, and each person gets a turn to play against our kind altruist. Soon, each other person has won their game, and they have 2 cookies each, while our selfless altruist has just one cookie per match they played. So, er, 11 or so cookies?</div>
<div><br /></div>
<div>Many of the other party-goers are enlightened by this example. They, too, want to be selfless and altruistic so that they can acquire 11 cookies / win at kidnapping. But a funny thing happens as each additional person plays B - the people playing A win two more cookies per round (one round is everyone getting to play everyone else once), and the people playing B win<em> </em>one <em>fewer</em> cookie, since nobody gets cookies when both play B either. Eventually, there are eight people playing A and four people playing B, and all of them nom 8 cookies per round.</div>
<div><br /></div>
<div><br /></div>
<div>It's inevitable that the people playing B eventually get the same number of cookies as the people playing A - if there was a cookie imbalance, then people would switch to the better strategy until cookies were balanced again. Playing A has a higher payoff, but all that really means is that there are eight people playing A and only 4 playing B. It's like B has an <em>ecological niche</em>, and that niche is only of a certain size.</div>
<div><br /></div>
<div>What does the party case say about what Red and Blue should do when playing a one-shot game? The ratios of players turn into probabilities: if you're less than 67% sure the other person will play A, you should play A. If you're more than 67% sure, you should play B. This plan only works for situations similar to drawing an opponent out of a pool of deterministic players, though.</div>
<div><br /></div>
<div><br /></div>
<div>Stage two of the problem: what if we allow players access to each others' source code?</div>
<div><br /></div>
<div>While you can still have A-players and B-players, you can now have a third strategy, which is to play B against A-players and play A against B-players. This strategy will have a niche size in between playing A and playing B.</div>
<div><br /></div>
<div>What's really great about reading source code, though, is that running into a copy of yourself no longer means duplicate moves and no cookies. The best "A-players" and "B-players" now choose moves against their copies by flipping coins, so that half the time they get at least one cookie. Flipping a coin against a copy of yourself averages 3/4 of a cookie, which is almost good enough to put B-players out of business. In fact, if we'd chosen our payoff matrix to have a bigger reward for playing A, we actually <em>could</em> put B-players out of business. Fun question: is it possible to decrease the total number of cookies won by increasing the reward for playing A?</div>
<div><br /></div>
<div>An interesting issue is how this modification changes the advice for the one-shot game. Our advice against simpler opponents was basically the "predictor" strategy, but that strategy is now in equilibrium with the other two! Good advice now is more like a meta-strategy. If the opponent is likely to be an A-player or a B-player, be a predictor, if the opponent is likely to be a predictor, be an A-player. Now that we've been this cycle before, it should be clearer that this "advice" is really a new strategy that will be introduced when we take the game one meta-level up. The effect on the game is really to introduce gradations of players, where some play A more often and some play B more often, but the populations can be balanced such that each player gets the same average reward.</div>
<div><br /></div>
<div><br /></div>
<div>An interesting facet of a competition between predictors is what we might call "stupidity envy" (See <a href="/lw/5rq/example_decision_theory_problem_agent_simulates/">ASP</a>). If we use the straightforward algorithm for our predictors (simulate what the opponent will do, then choose the best strategy), then a dumb predictor cannot predict the move of a smarter predictor. This is because the smarter predictor is predicting the dumb one, and you can't predict yourself in less time than you take to run. So the dumber predictor has to use some kind of default move. If its default move is A, then the smarter predictor has no good choice but to take B, and the dumber predictor wins.</div>
<div><br /></div>
<div>It's like the dumber predictor has gotten to move first. Being dumb / moving first isn't always good - imagine having to move first in rock paper scissors - but in games where moving first is better, and even a dumb predictor can see why, it's better to be the dumber predictor.</div>
<div><br /></div>
<div>On our other axis of smartness, though, the "meta-level," more meta usually produces better head-to-head results - yet the humble A-player gets the best results of all. It's only the fact that A-players do poorly against other A-players that allows a diverse ecology on the B-playing side of the spectrum.</div>manfredGQy2BSQG9Dd6vPhs82013-11-03T06:29:09.725ZMeetup : Urbana-Champaign: The löbstacle for picking good actions.
https://lw2.issarice.com/posts/CHes3ee7R78CdrFJh/meetup-urbana-champaign-the-loebstacle-for-picking-good
<h2>Discussion article for the meetup : <a href='/meetups/rb'>Urbana-Champaign: The löbstacle for picking good actions.</a></h2>
<div class="meetup-meta">
<p>
<strong>WHEN:</strong> 
<span class="date">22 September 2013 02:00:00PM (-0500)</span><br>
</p>
<p>
<strong>WHERE:</strong> 
<span class="address">40.111240, -88.227370</span>
</p>
</div><!-- .meta -->
<div id="" class="content">
<div class="md"><p>Let's have a slightly more technical meetup!
When: 2:00 PM this Sunday.
Where: The benches overlooking UIUC's Bardeen Quad, Urbana-Champaign, IL.
What: I'll give a primer on why even the best agent can't prove that they're always right (see e.g. <a href="http://lesswrong.com/lw/t8/you_provably_cant_trust_yourself/">You Provably Can't Trust Yourself</a>), and why this poses a problem for some common models of decision-making (discussed at length <a href="http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/">here</a>). And then we can talk about what other models we could use!</p></div>
</div><!-- .content -->
<h2>Discussion article for the meetup : <a href='/meetups/rb'>Urbana-Champaign: The löbstacle for picking good actions.</a></h2>manfredCHes3ee7R78CdrFJh2013-09-21T12:46:50.675ZA basis for pattern-matching in logical uncertainty
https://lw2.issarice.com/posts/trjysoG4g7X79mkmM/a-basis-for-pattern-matching-in-logical-uncertainty
<p>Previous logical uncertainty robot designs (e.g. <a href="/lw/iax/some_miri_workshop_stuff/">here</a>, <a href="/lw/eaa/a_model_of_udt_with_a_concrete_prior_over_logical/">here</a>, <a href="/lw/eaa/a_model_of_udt_with_a_concrete_prior_over_logical/">here</a>) have relied on proving theorems that relate to the statement (e.g. "the trillionth digit of pi is 5") under inspection (where a useful theorem would be e.g. "there are 10 possible mutually exclusive digits [1,2,3,4,5,6,7,8,9,0] the trillionth digit of pi could be."). This is nice. But it doesn't do pattern-matching - if you tell a theorem-proving robot that a statement is true for the first thousand integers, it doesn't even suspect that the statement might be true for the 1001st too. The statements are just independent black boxes.</p>
<p>In order to go further, our robot needs to peek inside the black box. But how, and why should it see patterns?</p>
<p> </p>
<p>We tell our robot these facts: "3 is 'odd'. 5 is 'odd'. 7 is 'odd'. 11 is 'odd'."</p>
<p>The robot asks "Could you just tell me what this concept 'odd' means explicitly? I don't know English very well."</p>
<p>"No," we say. "You'll have to guess."</p>
<p> </p>
<p>For robots that don't make up information out of thin air, guesses about 'odd'-ness will obey the maximum-entropy principle, which roughly means "give everything equal probability until you learn more."</p>
<p> </p>
<p>"Well, robot, what do you think is the probability that 9 is 'odd', given what we've told you?"</p>
<p>"50 percent. Beep boop."</p>
<p>"But all those other things were odd, weren't they?"</p>
<p>"Those other things were not the number 9. Imagine going number by number and labeling each number as 'odd' or 'not odd'. There are equal numbers of possible labelings with 9 odd and 9 not odd, regardless of what those other numbers are. Since by the principle of maximum entropy I start out with all labelings equally likely, 9 has a 50% chance of being 'odd'."</p>
<p>"But isn't it just sensible to think that the next number we give you will be 'odd' if all the others were? What of the appeal of simplicity?"</p>
<p>"What's so great about simplicity? I'm trying not to make up information out of thin air here."</p>
<p> </p>
<p>What do we tell our robot to make it guess that 9 is probably odd? Well, we want it to think that patterns of odd-ness that are simpler are more likely. So how about we let the robot peek at our definition of 'odd', just long enough to see that for every integer we can test if it's odd, and how complicated the test is.</p>
<p>Even if our robot has no clue what the constituent symbols <em>were</em> (or, more to the point, not enough processing power to fully explore the ramifications in a more difficult and realistic scenario), knowing the mere number of characters defining 'odd' puts an upper bound on the Kolmogorov complexity of how the numbers are labeled. Heck, even just learning that we can write down the oddness-test is huge, since there are an infinite number of infinite-complexity rules.</p>
<p>Once the robot knows that the complexity of 'odd' is below a certain smallish value, low-complexity hypotheses like "all numbers are 'odd'" and "odd numbers are 'odd'" start to outweigh<sup>1</sup> bigger hypotheses like "all numbers except {long list including 9} are 'odd'." These simple hypotheses contain just the sorts of patterns we sometimes want the robot to see, like 'odd'-ness.</p>
<p> </p>
<p>We tell our robot these facts: "3 is odd. 5 is odd. 7 is odd. 11 is odd. A number is 'odd' if a 14-character predicate is true."</p>
<p>The robot asks "Could you just tell me what this concept 'odd' means explicitly?"</p>
<p>"No," we say. "You'll have to guess. What do you think is the probability that 9 is 'odd', given what we've told you?"</p>
<p>"65.1 percent."</p>
<p>"Hah, got you! When we said 'odd' we secretly meant prime!"</p>
<p> </p>
<p>I suspect that this kind of peeking handles most of the cases where we want something like pattern-matching (specifically, minimum-message-length prediction of patterns) in logical uncertainty. The obvious un-handled case - the choice of axioms, or <a href="/lw/g7n/secondorder_logic_the_controversy/">logical systems</a>, or that sort of thing, seems more like model uncertainty than logical uncertainty - that is, the question is not really "is second-order logic true," it's "does second-order logic correspond to the world I want to model," which is beyond the scope of this Discussion article.</p>
<p> </p>
<p> </p>
<p><span style="font-size: xx-small;"><sup>1 </sup>How do you turn the number of symbols we wrote it down with into a distribution over possible rules? It's easiest to just maximum-entropy all valid rules with the correct number of symbols. Since many rules can be compressed, the set of rules with some number of symbols is smeared over lower possible Kolmogorov complexities, I think with an exponential preference towards lower complexity. But on the other hand, humans don't hand out maximum-entropy samples of definitions - we'll need probabilistic logic to do probabilistic logic. (Yo dawg, I heard you liked recursion, so we put this joke in itself so you can [this joke] while you [this joke].)</span> </p>manfredtrjysoG4g7X79mkmM2013-08-29T08:53:55.914ZRationalist fiction brainstorming funtimes
https://lw2.issarice.com/posts/bBcStSetHwXv4ovmj/rationalist-fiction-brainstorming-funtimes
<p>The title should make things clear enough, so let's start with my description of the target, rationalist fiction: fiction that tries to teach the audience rationalist cognitive skills by having characters model those skills for the reader.</p>
<p>So for example, <a href="http://luminous.elcenia.com/chapters/ch1.shtml"><em>Luminosity</em></a> is to a large extent about the questions "What do I want?, What do I have?, and How can I best use the latter to get the former?" Oh, and using empiricism on magic.</p>
<p>Another example is <a href="http://hpmor.com/"><em>Harry Potter and the Methods of Rationality</em></a>, which goes more in-depth about the laundry list of human biases. In fact, many of the more iconic moments (measured by what I remember and what other people like to copy) are about biases to avoid, rather than about modeling good behavior.</p>
<p> </p>
<p>This thread is about ideas, from general to specific, for rationalist fiction. I'll give some obvious examples.</p>
<p>General idea: having a rational character encountering magic or amazing technology is a great chance to showcase the power of empiricism. (Has anyone gotten on this one yet? :3 )</p>
<p>Story idea: Okay, so we take the Dresden Files universe, and our rational protagonist is some smart kid who just started a summer job as an assistant radio technician or something. It turns out he's got one in a hundred magical talent, enough to cut off his budding career, he manages to find the magic community, figures out just enough, embarks on heroic quest to run a magitech radio station. (Okay, this last bit isn't obvious - for one, more character development would probably have him wanting something else. For another, the obvious thing is to take over the world if Luminosity and HPMOR are anything to go by.)</p>
<p>Specific idea: A character could model the skill of testing stuff by testing stuff. When characters are performing a big search, have someone actually stop to think about false positives, or more generally "how could things be going wrong, and how can I prevent that?", and have it actually be a false positive once.</p>
<p> </p>
<p>But really, there's an<em> explosion</em> of possibilities out there to explore, and I feel like we have "Rationalist meets magic. Rationalist does science to magic. Rationalist kicks butt with magic" fairly well-covered. We have <a href="/lw/fc3/checklist_of_rationality_habits/">all these different biases categorized, with corresponding right ways to do things</a>, and there are plenty of good behaviors we can try to teach an audience without the empiricism-fodder and high stakes that is a fantasy setting. Or even if you do a Dresden Files fic, you could ignore the empricism stuff and just, like, pick a habit from Anna's checklist and write a short story :D. Here's an idea I quite fancy, I'll save everything else for comments:</p>
<p>General idea: Giving people the benefit of the doubt and managing to lose arguments when they need to be lost is the closest thing to a rationalist superpower I have. Can I work that into a story somehow?</p>
<p>Story ideas: A <a href="http://www.e-reading-lib.org/bookreader.php/1007512/Herriots_-_Every_living_thing.html">James Herriot</a> sort of thing, where the protagonist has their daily life (Maybe veterinarian, or materials scientist, or line cook, or model rocket hobbyist), and relatably goes about it, occasionally giving people the benefit of the doubt and losing arguments, and sometimes using other rationalist skills, and usually ending up on the right side of things in the end. At this point it might be too subtle to actually teach the audience, one solution to this would be a designated person in-story to periodically notice how awesome the protagonist is.</p>manfredbBcStSetHwXv4ovmj2013-03-09T13:53:52.270ZLogical uncertainty, kind of. A proposal, at least.
https://lw2.issarice.com/posts/K2YZPnASN88HTWhAN/logical-uncertainty-kind-of-a-proposal-at-least
<p>If you want context and are fast at seeing the implications of math, see <a href="/lw/eaa/a_model_of_udt_with_a_concrete_prior_over_logical/">Benja's post</a>. This post is much lighter on the math, though it may take more background reading and more laborious interpolation, since it's, well, lighter on the math.</p>
<p> </p>
<p>Imagine I introduced my pet robot to a game. The robot has 10 seconds to pick a digit, and if the trillionth prime number ends with that digit, the robot gets a cookie (it likes peanut butter cookies the best). 10 seconds is not enough time for my robot to calculate the answer deductively. And yet, guessing an answer is superior to running out of time quietly. What sort of general logic should my robot follow in under 10 seconds to figure out that it should be indifferent between answering 1, 3, 7 or 9? Does it even make sense to be indifferent between the real answer and an impossible answer, even if you don't know which is which?</p>
<p> </p>
<p>As you might expect from context, the proposed solution will involve assigning every true or false math statement a probabability-esque degree of plausibility, with numbers other than 0 or 1 indicating logical uncertainty. Why is this a good idea?</p>
<p> </p>
<p>To explain logical uncertainty, let's first take a step back and reframe logical <em>certainty</em> in terms of rules for reasoning that apply to both deductive logic and probabilistic logic. An important resource here is E.T. Jaynes' <a href="http://bayes.wustl.edu/etj/prob/book.pdf">Probability Theory</a> (pdf) - the most relevant part being page 31 of the book. The key idea is that each of the probability axioms applies just fine no matter what kind of Boolean statement you want to find the probability of. Which is to say probability already applies to arithmetic - the laws of probability are also laws of arithmetic, just in the limit that probabilities go to 1 or 0. Our robot starts with a collection of definitions labeled with probability 1 (like "0 is a number" or "S(0)+0=S(0)" [if this S(0) stuff needs context, see <a href="http://en.wikipedia.org/wiki/Peano_axioms#Addition">wikipedia</a>]), and then applies deductive rules according to the universal rules of probability. We translate "A implies B" into the language of probabilities as P(AB|C) = P(A|C), and then apply the always-true product rule P(B|AC)=P(AB|C) / P(A|C). If P(A|C)=1, that is, A|C is deductively true, and A implies B, then P(B|AC)=P(B|C)=1. The machinery that underlies deduction is in fact the same machinery that underlies probabilistic reasoning. And we're just going to exploit that a little.</p>
<p>An alternate axiomatization due to Savage (hat tip to articles by <a href="/lw/5te/a_summary_of_savages_foundations_for_probability/">Sniffoy</a> and <a href="/lw/9e4/the_savage_theorem_and_the_ellsberg_paradox/">fool</a>) is based just on actions - it doesn't seem necessary for every agent to store numerical plausibilities, but every agent has to act, and if our agent is to act as if it had consistent preferences when presented with bets, it must act <em>as if</em> it calculated probabilities. Just like the conditions of Cox's theorem as used by E.T. Jaynes, the conditions of Savage's theorem apply to bets on arithmetic just fine. So our robot always behaves as if it assigns some probabilities over the last digit of the trillionth prime number - it's just that when our robot's allowed to run long enough, all but one of those probabilities is 0.</p>
<p> </p>
<p>So how do we take the basic laws of belief-manipulation, like the product rule or the sum rule, and apply them to cases where we run out of time and can't deduce all the things? If we still want to take actions, we still want to assign probabilities, but we can't use deduction more than a set number of times...</p>
<p>Okay fine I'll just say it. The proposal outlined here is to treat a computationally limited agent's "correct beliefs" as the correct beliefs of a computationally <em>unlimited</em> agent with a limited definition of what deduction can do. So this weakened-deduction agent has a limitation, in that starting from axioms it can only prove some small pool of theorems, but it's unlimited in that it can take the pool of proven theorems, and then assign probabilities to <em>all</em> the unproven true or false statements. After we flesh out this agent, we can find a computationally limited algorithm that finds correct (i.e. equal to the ones from a sentence ago) probabilities for <em>specific</em> statements, rather than all of them. And finally, we have to take this and make a decision procedure - our robot. After all, it's no good for our robot to assign probabilities if it proceeds to get stuck because it tries to compare the utilities of the world if the end of the trillionth prime number were 1 versus 7 and doesn't even know what it <em>means</em> to calculate the utility of the impossible. We have to make a bit of a modification to the whole decision procedure, we can't just throw in probabilities and expect utility to keep up.</p>
<p> </p>
<p>So, formally, what's going on when we limit deduction? Well, remember the process of deduction outlined earlier?</p>
<blockquote>
<p>We translate "A implies B" into the language of probabilities as P(AB|C) = P(A|C), and then apply the always-true product rule P(B|AC)=P(AB|C) / P(A|C). If P(A|C)=1, that is, A|C is deductively true, and A implies B, then P(B|AC)=P(B|C)=1</p>
</blockquote>
<p>There is a chain here, and if we want to limit deduction to some small pool of provable theorems, we need one of the links to be broken outside that pool. As implied, I don't want to mess with the product rule, or else we violate a desideratum of belief. Instead, we'll mess with implication itself - we translate "A implies B" into "P(AB|C)=P(A|C) only if we've spent less than 10 seconds doing deduction." Or "P(AB|C)=P(A|C) only if it's been less than 10<sup>6</sup> steps from the basic axioms." These limitations are ugly and nonlocal because they represent the intrusion of nonlocal limitations on our agent into a system that previously ran forever.</p>
<p>Note that the weakening of implication does not necessarily determine the shape of our pool of deduced theorems. A weakened-deduction agent could spiral outward from shortest to longest theorems, or it could search more cleverly to advance on some specific theorems before time runs out.</p>
<p>If a weakened-deduction agent just had the product rule and this new way of translating the axioms into probabilities, it would accumulate some pool of known probabilities - it could work out from the probability-1 axioms to show that some short statements had probability 1 and some other short statements had probability 0. It could also prove some more abstract things like P(AB)=0 without proving anything else about A or B, as long as it followed the right search pattern. But it can't assign probabilities outside of deduction - it doesn't have the rules. So it just ends up with a pool of deduced stuff in the middle of a blank plain of "undefined."</p>
<p> </p>
<p>Okay, back to referring to E.T. Jaynes (specifically, the bottom of page 32). When deriving the laws of probability from Cox's desiderata, the axioms fall into different groups - there are the "laws of thought" parts, and the "interface" parts. The laws of thought are things like Bayes' theorem, or the product rule. They tell you how probabilities have to fit with <em>other</em> probabilities. But they don't give you probabilities <em>ex nihilo</em>, you have to start with probability-1 axioms or known probabilities and build out from them. The parts that tell you how to get <em>new</em> probabilities are the interface parts, ideas like "if you have equivalent information about two things, they should have the same probability."</p>
<p>So what does our limited-deduction agent do once it reaches its limits of deduction? Well, to put it simply, it uses deduction as much as it can, and then it uses the principle of maximum entropy for the probability of everything else. Maximum entropy corresponds to minimum information, so it satisfies a desideratum like "don't make stuff up."</p>
<p> </p>
<p>The agent is assigning probabilities to true or false logical statements, statements like S(0)+S(0)=S(S(0)). If it had an unrestricted translation of "A implies B," it could prove this statement quickly. But suppose it can't. Then this statement is really just a string of symbols. The agent no longer "understands" the symbols, which is to say it can only use facts about the probability of these symbols that were previously proved and are within the pool of theorems - it's only a <em>part </em>of an algorithm, and doesn't have the resources to prove everything, so we have to design the agent to assign probabilities based just on what it proved deductively.</p>
<p>So the design of our unlimited-computation, limited-deduction agent is that it does all the deduction it can according to some search algorithm and within some limit, and this can be specified to take any amount of time. Then, to fill up the infinity of un-deduced probabilities, the agent just assigns the maximum-entropy probability distribution consistent with what's proven. For clever search strategies that figure out things like P(AB)=0 without figuring out P(A), doing this assignment requires interpretation of AND, OR, and NOT operations - that is, we still need a Boolean algebra for statements. But our robot no longer proves new statements about probabilities of these symbol strings, in the sense that P(S(0)+0=S(0))=P(S(0)+S(0)=S(S(0))) is a new statement. An example of a non-new statement would be P(S(0)+0=S(0)) AND S(0)+S(0)=S(S(0))) = P(S(0)+0=S(0)) * P(S(0)+S(0)=S(S(0)) | S(0)+0=S(0)) - that's just the product rule, it hasn't actually changed any of the <em>equations</em>.</p>
<p> </p>
<p>End of part 1 exercise: Can deducing an additional theorem lead to our agent assigning less probability to the right answer under certain situations? (Reading part 2 may help)</p>
<p> </p>
<p>Okay, now on to doing this with actual bounded resources. And back to the trillionth prime number! You almost forgot about that, didn't you. The plan is to break up the strict deduction -> max entropy procedure, and do it in such a way that our robot can get better results (higher probability to the correct answer) the longer it runs, up to proving the actual correct answer. It starts with no theorems, and figures out the max entropy probability distribution for the end of the trillionth prime number. Said distribution happens to be one-half to everything, e.g. p(1)=1/2 and p(2)=1/2 and p(3)=1/2. The robot doesn't know yet that the different answers are mutually exclusive and exhaustive, much less what's wrong with the answer of 2. But the important thing is, assigning the same number to everything of interest is <em>fast</em>. Later, as it proves relevant theorems, the robot updates the probability distribution, and when it runs out of resources it stops.</p>
<p>Side note: there's also another way of imagining how the robot stores probabilities, used in Benja's post, which is to construct a really big mutually exclusive and exhaustive basis (called "disjunctive normal form"). Instead of storing P(A) and P(B), which are not necessarily mutually exclusive or exhaustive, we store P(AB), P(A¬B) (the hook thingy means "NOT"), P(¬AB), and P(¬A¬B), which are mutually exclusive and exhaustive. These things would then each have probability 1/4, or 1/2<sup>N</sup>, where N is the number of statements you're assigning probabilities to. This is a pain when N goes to infinity, but can be useful when N is approximately the number of possible last digits of a number.</p>
<p>Back on track: suppose the first thing the robot proves about the last digit of the trillionth prime number is that answers of 1, 2, 3, 4, 5, 6, 7, 8, 9, and 0 are exhaustive. What does that do to the probabilities? In disjunctive normal form, the change is clear - exhaustiveness means that P(¬1¬2¬3¬4¬5¬6¬7¬8¬9¬0)=0, there's no leftover space. Previously there were 2<sup>10</sup>=1024 of these disjunctive possibilities, now there are 1023, and the remaining ones stay equivalent in terms of what's been proven about them (nothing), so the probability of each went from 1/1024 to 1/1023. Two things to note: figuring this out took a small amount of work and is totally doable for the robot, but we don't want to do this work every time we use modus tollens, so we need to have some way to tell whether our new theorem matters to the trillionth prime number.</p>
<p>For example, image we were interested in the statement A. The example is to learn that A, B, and C are mutually exclusive and exhaustive, step by step. First, we could prove that A, B, C are exhaustive - P(¬A¬B¬C)=0. Does this change P(A)? Yes, it changes from 4/8 (N is 3, so 2<sup>3</sup>=8) to 4/7. Then we learn that P(AB)=0, i.e. A and B are mutually exclusive. This leaves us only A¬BC, ¬ABC, A¬B¬C, ¬AB¬C, and ¬A¬BC. P(A) is now 2/5. Now we learn that A and C are mutually exclusive, so the possibilities are ¬ABC, A¬B¬C, ¬AB¬C, and ¬A¬BC. P(A)=1/4. Each of the steps until now have had the statement A right there inside the parentheses - but for the last step, we show that B and C are mutually exclusive, P(BC)=0, and now we just have P(A)=P(B)=P(C)=1/3. We just took a step that didn't mention A, but it changed the probability of A. This is because we'd previously disrupted the balance between ABC and ¬ABC. To tell when to update P(A) we not only need to listen for A to be mentioned, we have to track what A has been entangled with, and what's been entangled with that, and so on in a web of deduced relationships.</p>
<p> </p>
<p>The good news is that that's <em>it</em>. The plausibility assigned to any statement A by this finite-computation method is the same plausibility that our computationally-unlimited deductively-limited agent would have assigned to it, given the same pool of deduced theorems. The difference is just that the limited-deduction agent did this for every possible statement, which as mentioned doesn't make as much sense in disjunctive normal form.</p>
<p>So IF we accept that having limited resources is like having a limited ability to do implication, THEN we know how our robot should assign probabilities to a few statements of interest. It should start with the good old "everything gets probability 1/2," which should allow it to win some cookies even if it only has a few milliseconds, and then it should start proving theorems, updating its probabilities when it proves something that should impact those probabilities.</p>
<p> </p>
<p>Now onto the last part. The robot's utility function wasn't really designed for U(last digit of trillionth prime number is 1), so what should it do? Well, what <em>does</em> our robot like? It likes having a cookie over not having a cookie. C is for cookie, and that's good enough for it. So we want to transform a utility over cookies into a an expected utility that will let us order possible actions.</p>
<p>We have to make the exact same transformation in the case of ordinary probabilities, so let's examine that. If I flip a coin and get a cookie if I call it correctly, I don't have a terminal U(heads) or U(tails), I just have U(cookie). My expected utility of different guesses comes from not knowing which guess leads to the cookie.</p>
<p>Similarly, the expected utility of different guesses when betting on the trillionth prime number comes from not knowing which guess leads to the cookie. It is possible to care about the properties of math, or to care about whether coins land heads or tails, but that just means we have to drag in causality - your guess doesn't affect how math works, or flip coins over.</p>
<p> </p>
<p>So the standard procedure for our robot looks like this:</p>
<p>Start with some utility function U over the world, specifically cookies.</p>
<p>Now, face a problem. This problem will have some outcomes (possible numbers of cookies), some options (that is, strategies to follow, like choosing one of 10 possible digits), and any amount of information about how options correspond to outcomes (like "iff the trillionth prime ends with this digit, you get the cookie").</p>
<p>Now our robot calculates the limited-resources probability of getting different outcomes given different strategies, and from that calculates an expected utility for each strategy.</p>
<p>Our robot then follows one of the strategies with maximum expected utility.</p>
<p> </p>
<p>Bonus exercises: Does this procedure already handle probabilistic maps from the options to the outcomes, like in the case of the flipped coin? How about if flipping a coin isn't already converted into a probability, but is left as an underdetermined problem a la "a coin (heads XOR tails) is flipped, choose one."</p>manfredK2YZPnASN88HTWhAN2013-01-13T09:26:15.670Z