Comment by lanrian on Recommendation Features on LessWrong · 2019-06-15T13:07:29.540Z · score: 5 (3 votes) · LW · GW

Re addictiveness: a potential fix could be to add an option to only refresh the recommended archive posts once per day (or some other time period of your choice).

Comment by lanrian on LessWrong FAQ · 2019-06-14T21:52:40.034Z · score: 8 (5 votes) · LW · GW

Thanks a lot for this Ruby! After skimming, the only thing I can think of adding would be a link to the moderation log, along with a short explanation of what it records. Partly because it's good that people can look at it, and partly because it's nice to inform people that their deletions and bans are publicly visible.

Comment by lanrian on Get Rich Real Slowly · 2019-06-10T18:39:20.948Z · score: 5 (3 votes) · LW · GW

LW footnotes can be created like EA forum footnotes.

Comment by lanrian on Ramifications of limited positive value, unlimited negative value? · 2019-06-10T10:15:27.536Z · score: 1 (1 votes) · LW · GW

If the Universe is infinite, every positive experience is already instantiated once. This view could then imply that you should only focus on preventing suffering. That depends somewhat on exactly what you mean with "I" and "we", though, and if you think that the boundary between our lightcone and the rest of the Universe has any moral significance.

Comment by lanrian on Ramifications of limited positive value, unlimited negative value? · 2019-06-10T10:04:08.891Z · score: 3 (2 votes) · LW · GW

What do you think about the argument that the Universe might well be infinite, and if so, your view means that nothing we do matters, since every brainstate is already instantiated somewhere? (Taken from Bostrom's paper on the subject.)

Comment by lanrian on Drowning children are rare · 2019-06-06T17:47:18.607Z · score: 1 (1 votes) · LW · GW

I don't think anyone has claimed that "there's a large funding gap at cost-per-life-saved numbers close to the current GiveWell estimates", if "large" means $50B. GiveWell seem to think that their present top charities' funding gaps are in the tens of millions.

Comment by lanrian on The Inner Alignment Problem · 2019-06-05T22:01:32.359Z · score: 2 (2 votes) · LW · GW

Main point:

I agree that inner alignment is a really hard problem, and that for a non-huge amount of training data, there is likely to be a proxy goal that's simpler than the real goal. Description length still seems importantly different from e.g. computation time. If we keep optimising for the simplest learned algorithm, and gradually increase our training data towards all of the data we care about, I expect us to eventually reach a mesa-optimiser optimising for the base objective. (You seem to agree with this, in the last section?) However, if we keep optimising for the fastest learned algorithm, and gradually increase our training data towards all of the data we care about, we won't ever get a robustly aligned system (until we've shown it every single datapoint that we'll ever care about). We'll probably just get a look-up table which acts randomly on new input.

This difference makes me think that simplicity could be a useful tool to make a robustly aligned mesa optimiser. Maybe you disagree because you think that the necessary amounts of data is so ludicrously big that we'll never reach them, even by using adversarial training or other such tricks?

I'd be more willing to drop simplicity if we had good, generic methods to directly optimise for "pure similarity to the base objective", but I don't know how to do this without doing hard-coded optimisation or internals-based selection. Maybe you think the task is impossible without some version of the latter?

Comment by lanrian on The Inner Alignment Problem · 2019-06-05T21:39:50.308Z · score: 2 (2 votes) · LW · GW

Minor point:

as you mention, food, pain, mating, etc. are pretty simple to humans, because they get to refer to sensory data, but very complex from the perspective of evolution, which doesn't.

I chose status and cheating precisely because they don't directly refer to simple sensory data. You need complex models of your social environment in order to even have a concept of status, and I actually think it's pretty impressive that we have enough of such models hardcoded into us to have preferences over them.

Since the original text mentions food and pain as "directly related to our input data", I thought status hierarchies was noticeably different from them, in this way. Do tell me if you were trying to point at some other distinction (or if you don't think status requires complex models).

Comment by lanrian on The Inner Alignment Problem · 2019-06-05T17:39:50.699Z · score: 4 (3 votes) · LW · GW
Since there are more pseudo-aligned mesa-objectives than robustly aligned mesa-objectives, pseudo-alignment provides more degrees of freedom for choosing a particularly simple mesa-objective. Thus, we expect that in most cases there will be several pseudo-aligned mesa-optimizers that are less complex than any robustly aligned mesa-optimizer.

This isn't obvious to me. If the environment is fairly varied, you will probably need different proxies for the base objective in different situations. As you say, representing all these proxies directly will save on computation time, but I would expect it to have a longer description length, since each proxie needs to be specified independently (together with information on how to make tradeoffs between them). The opposite case, where a complex base objective correlates with the same proxie in a wide range of environments, seems rarer.

Using humans as an analogy, we were specified with proxy goals, and our values are extremely complicated. You mention the sensory experience of food and pain as relatively simple goals, but we also have far more complex ones, like the wish to be relatively high in a status hierarchy, the wish to not have a mate cheat on us, etc. You're right that an innate model of genetic fitness also would have been quite complicated, though.

(Rohin mentions that most of these things follow a pattern where one extreme encourages heuristics and one extreme encourages robust mesa-optimizers, while you get pseudo-aligned mesa-optimizers in the middle. At present, simplicity breaks this pattern, since you claim that pseudo-aligned mesa-optimizers are simpler than both heuristics and robustly aligned mesa-optimizers. What I'm saying is that I think that the general pattern might hold here, as well: short description lengths might make it easier to achieve robust alignment.)

Edit: To some extent, it seems like you already agree with this, since Adversarial training points out that a sufficiently wide range of environments will have a robustly aligned agent as it's simplest mesa-optimizer. Do you assume that there isn't enough training data to identify , in Compression of the mesa-optimizer? It might be good to clarify the difference between those two sections.

Comment by lanrian on Chapter 7: Reciprocation · 2019-06-04T10:06:17.927Z · score: 3 (2 votes) · LW · GW

This version,,, and the pdf all have "honor" (without a u) at two places in chapter 7. I don't think there is a more updated version.

Comment by lanrian on Can movement from Conflict to Mistake theorist be facilitated effectively? · 2019-06-03T20:16:28.074Z · score: 2 (2 votes) · LW · GW

Link to SSC's explanation of the concept.

I'd say most positions are in between complete conflict theory and complete mistake theory (though they're not necessarily 'transitional', if people tend to stay there once they've reached them). It all depends on how much of political disagreements you think is fueled by different interests and how much is fueled by different beliefs. I also think that the best position lies there, somewhere in between. It is in fact correct that a fair amount of political conflict happens due to different interests, so a complete mistake theorist would frequently fail to predict why politics works the way it does.

(Of course, even if you agree with this, you may think that most people should become more mistake theorist, on the margin.)

Comment by lanrian on Chapter 7: Reciprocation · 2019-06-03T19:57:09.398Z · score: 3 (2 votes) · LW · GW

In the first chapter, it's noted "The story has been corrected to British English up to Ch. 17, and further Britpicking is currently in progress (see the /HPMOR subreddit).". Given your points, it seems like it's not even thouroughly britpicked up 'til 17. I expect Eliezer to have written that note quite some time ago, so I'm not too hopeful about this still going on at the subreddit, either.

Comment by lanrian on Feedback Requested! Draft of a New About/Welcome Page for LessWrong · 2019-06-01T08:45:19.007Z · score: 22 (8 votes) · LW · GW

If this is something that everyone reads, it might be nice to provide links to more technical details of the site. I imagine that someone reading this who then engages with LW might wonder:

  • What makes a curated post a curated post? (this might fit into the site guide on personal vs frontpage posts)
  • Why do comments/posts have more karma than votes?
    • What's the mapping between users' karma and voting power?
  • How does editing work? Some things are not immediately obvious, like:
    • How do I use latex?
    • How do I use footnotes?
    • How do I create images?
  • How does moderation work? Who can moderate their own posts?

This kind of knowledge isn't gathered in one place right now, and is typically difficult to google.

Comment by lanrian on Egoism In Disguise · 2019-05-31T18:34:51.304Z · score: 5 (6 votes) · LW · GW

I'm sceptical that pushing egoism over utilitarianism will make people less prone to punish others.

I don't know any system of utilitarianism that places terminal value on punishing others, and (although there probably exists a few,) I don't know of anyone who identifies as a utilitarian who places terminal value on punishing others. In fact, I'd guess that the average person identifying as a utilitarian is less likely to punish others (when there is no instrumental value to be had) than the average person identifying as an egoist. After all, the egoist has no reason to tame their barbaric impulses: if they want to punish someone, then it's correct to punish that person.

I agree that your version of egoism is similar to most rationalists' versions of utilitarianism (although there are definitely moral realist utilitarians out there). Insofar as we have time to explain our beliefs properly, the name we use for them (hopefully) doesn't matter much, so we can call it either egoism or utilitarianism. When we don't have time to explain our beliefs properly, though, the name does matter, because the listener will use their own interpretation of it. Since I think that the average interpretation of utilitarianism is less likely to lead to punishment than the average interpretation of egoism, this doesn't seem like a good reason to push for egoism.

Maybe pushing for moral anti-realism would be a better bet?

Comment by lanrian on Drowning children are rare · 2019-05-29T21:54:50.015Z · score: 10 (4 votes) · LW · GW

I still have no idea of how the total amount of dying people is relevant, but my best reading of your argument is:

  • If givewells cost effectiveness estimates were correct, foundations would spend their money on them.
  • Since the foundations have money that they aren't spending on them, the estimates must be incorrect.

According to this post, OpenPhil intends to spend rougly 10% of their money on "straightforward charity" (rather than their other cause areas). That would be about $1B (though I can't find the exact numbers right now), which is a lot, but hardly unlimited. Their worries about displacing other donors, coupled with the possibility of learning about better opportunities in the future, seems sufficient to justify partial funding to me.

That leaves the Gates Foundation (at least among the foundations that you mentioned, of course there's a lot more). I don't have a good model of when really big foundations does and doesn't grant money, but I think Carl Shulman makes some interesting points in this old thread.

Comment by lanrian on Drowning children are rare · 2019-05-29T11:24:15.809Z · score: 39 (10 votes) · LW · GW

In general, I'd very much like a permanent neat-things-to-know-about-LW post or page, which receives edits when there's a significant update (do tell me if there's already something like this). For example, I remember trying to find information about the mapping between karma and voting power a few months ago, and it was very difficult. I think I eventually found an announcement post that had the answer, but I can't know for sure, since there might have been a change since that announcement was made. More recently, I saw that there were footnotes in the sequences, and failed to find any reference whatsoever on how to create footnotes. I didn't learn how to do this until a month or so later, when the footnotes came to the EA forum and aaron wrote a post about it.

Comment by lanrian on Drowning children are rare · 2019-05-28T18:17:36.860Z · score: 19 (7 votes) · LW · GW

I'm confused about the argument you're trying to make here (I also disagree with some things, but I want to understand the post properly before engaging with that). The main claims seem to be

There are simply not enough excess deaths for these claims to be plausible.

and, after telling us how many preventable deaths there could be,

Either charities like the Gates Foundation and Good Ventures are hoarding money at the price of millions of preventable deaths, or the low cost-per-life-saved numbers are wildly exaggerated.

But I don't understand how these claims interconnect. If there were more people dying from preventable diseases, how would that dissolve the dilemma that the second claim poses?

Also, you say that $125 billion is well within the reach of the GF, but their website says that their present endowment is only $50.7 billion. Is this a mistake, or do you mean something else with "within reach"?

Comment by lanrian on Habryka's Shortform Feed · 2019-05-27T10:02:22.819Z · score: 2 (3 votes) · LW · GW

Any reason why you mention timeless decision theory (TDT) specifically? My impression was that functional decision theory (as well as UDT, since they're basically the same thing) is regarded as a strict improvement over TDT.

Comment by lanrian on Feature Request: Self-imposed Time Restrictions · 2019-05-16T22:51:03.864Z · score: 2 (2 votes) · LW · GW

Leechblock is excellent. I presently use it to block facebook (except for events and permalinks to specific posts) all the time except for 10min between 10pm and midnight; I have a list of webcomics that I can only view on saturdays; there is a web-based game that I can play once every saturday (whereafter the expired time prevents me from playing a second game), etc.

Comment by lanrian on Moral Weight Doesn't Accrue Linearly · 2019-04-23T23:24:23.987Z · score: 7 (4 votes) · LW · GW
Yes, these are among the reasons why moral value is not linearly additive. I agree.

I think the SSC post should only be construed as arguing about the value of individual animals' experiences, and that it intentionally ignores these other sources of values. I agree with the SSC post that it's useful to consider the value of individual animals' experiences (what I would call their 'moral weight') independently of the aesthetic value and the option value of the species that they belong to. Insofar as you agree that individual animals' experiences add up linearly, you don't disagree with the post. Insofar as you think that individual animals' experiences add up sub-linearly, I think you shouldn't use species' extinction as an example, since the aesthetic value and the option value are confounding factors.

Really? You consider it to be equivalently bad for there to be a plague that kills 100,000 humans in a world with a population of 100,000 than in a world with a population of 7,000,000,000?

I consider it equally bad for the individual, dying humans, which is what I meant when I said that I reject scope insensitivity. However, the former plague will presumably eliminate the potential for humanity having a long future, and that will be the most relevant consideration in the scenario. (This will probably make the former scenario far worse, but you could add other details to the scenario that reversed that conclusion.)

Comment by lanrian on Moral Weight Doesn't Accrue Linearly · 2019-04-23T22:59:05.573Z · score: 6 (4 votes) · LW · GW

When people consider it worse for a species to go from 1000 to 0 members, I think it's mostly due to aesthetic value (people value the existence of a species, independent of the individuals), and because of option value (we might eventually find a good reason to bring back the animals, or find the information in their genome important, and then it's important that a few remain). However, none of these have much to do with the value of individual animals' experiences, which usually is what I think about when people talk about animals' "moral weight". People would probably also find it tragic for plants to go extinct (and do find languages going extinct tragic), despite these having no neurons at all. I think the distinction becomes more clear if we consider experiences instead of existence: to me, it's very counterintuitive to think that an elephant's suffering matter less if there are more elephants elsewhere in the world.

To be fair, scope insensitivity is a known bias (though you might dispute it being a bias, in these cases), so even if you account for aesthetic value and option value, you could probably get sublinear additivity out of most people's revealed preference. On reflection, I personally reject this for animals, though, for the same reasons that I reject it for humans.

Comment by lanrian on [HPMOR] "the Headmaster set fire to a chicken!" · 2019-04-03T19:31:42.226Z · score: 1 (1 votes) · LW · GW
I'll have to go back and re-read - was it clear that the chicken that burned wasn't actually Fawkes? I took that scene as Harry's interpretation of "normal" phoenix renewal.

Even after encountering Fawkes, Harry keeps insisting that the first encounter was with a chicken. A lot of chapters later, Flitwick suggests that it was probably a transfigured chicken.

In fact, I burn chicken often, then eat it (granted, I have someone else kill it and dissect it first, but that's not an important moral distinction IMO).

I think most people see an important moral distinction between killing a chicken painlessly and setting fire to it. Although the vast majority of meat isn't produced painlessly, a lot of people believe that their meat is. This implies that they might not be so casual about setting fire to a chicken, themselves.

Comment by lanrian on [HPMOR] "the Headmaster set fire to a chicken!" · 2019-04-03T19:27:51.577Z · score: 11 (4 votes) · LW · GW

I think Eliezer believes that chickens aren't sentient, and at the time of writing HPMOR, he probably thought this was the most common position among people in general (which was later contradicted by a poll he ran, see ). If Dumbledore believed that chickens weren't sentient, he might not think there's anything wrong with setting fire to one.

For lots of discussion about Eliezer's and others' philosophy of mind, see

Comment by lanrian on New versions of posts in "Map and Territory" and "How To Actually Change Your Mind" are up (also, new revision system) · 2019-02-26T20:59:18.059Z · score: 13 (5 votes) · LW · GW

Neat! Two questions:

Is it possible to create a link that always lead to the newest version of an article, or will all links lead to the version that existed when the link was created?

Will all edits be visible in the history, or can the author make minor edits without triggering a new version?

Comment by lanrian on Quantifying anthropic effects on the Fermi paradox · 2019-02-17T18:00:17.602Z · score: 1 (1 votes) · LW · GW

Hm, I still can't find a way to interpret this that doesn't reduce it to prior probability.

Density corresponds to how common life is (?), which is proportional to . Then the "size" of an area with a certain density corresonds to the prior probability of a certain ? Thus, "the total number of people in low density areas is greater than the total number of people in high density areas, because the size of the low density area is so much greater" corresponds to ", because the prior probability (denoted by p()) of is so much greater".

Comment by lanrian on Quantifying anthropic effects on the Fermi paradox · 2019-02-16T13:05:36.455Z · score: 2 (2 votes) · LW · GW
I expected to find here a link on the Grace SIA Doomsday argument. She uses the same logic as you, but then turns to the estimation of the probability that Great filter is ahead. It looks like you ignore possible high extinction rate implied by SIA (?). Also, Universal DA by Vilenkin could be mentioned.

Yup, I talk about this in the section Prior probability distribution. SIA definitely predicts doomsday (or anything that prevents space colonisation), so this post only applies to the fraction of possible Earths where the probability of doom isn't that high. Despite being small, that fraction is interesting to a total consequentialist, since it's the one where we have the best chance at affecting a large part of the Universe (assuming that our ability to reduce x-risk gets proportionally smaller as the probability of spreading to space goes below 0.01 % or so).

Another question, which is interesting for me, is how all this affects the possibility of SETI-attack - sending malicious messages with the speed of light on the intergalactic distance.

There was a bunch of discussion in the comments of this post about whether SETI would even be necessary to find any ETI that wanted to be seen, given that the ETI would have a lot of resources available to look obvious. At least Paul concluded that it was pretty unlikely that we could have missed any civilisation that wanted to be seen. I think that analysis still stands.

Including the possibility of SETI-attacks in my analysis would mean that no early civiliation could ever develop in an advanced civilisation's light cone, but the borders between advanced civilisations would still be calculated with the civilisations' actual expansion speed (with the additional complication that advanced civilisations could 'jump' to any early civilisation that appears in their light cone). If we assume that the time left until we become invulnerable to SETI-attacks is negligible (a dangerous assumption?), I think that's roughly equivalent to the scenario under Visibility of civilisations in Appendix C, from Earth's perspective.

The third idea I had related to this is the possibility that "bad fine tuning" of the universe will overweight the expected gain of the civilisation density from SIA. For example, if a universe will be perfectly fine-tuned, every star will have a planet with life; however, it requires almost unbelievable fidelity of its parameters tuning. The more probable is the set of the universes there fine tuning is not so good, and the habitable planets are very rare.

If I understand you correctly, this is an argument that our prior probability of should be heavily weighted towards life being very unlikely? That could change the conclusion if the prior probability of was inversely proportional to , or even more extremely tilted towards lower numbers. I don't see any particular reason why we would be that confident that life is unlikely, though, especially since the relevant probability mass in my analysis already puts beneath . Having a prior that puts times more probability mass on than is very extreme, given the uncertainty about this area.

Quantifying anthropic effects on the Fermi paradox

2019-02-15T10:51:04.298Z · score: 21 (12 votes)
Comment by lanrian on Probability space has 2 metrics · 2019-02-10T22:01:17.637Z · score: 3 (2 votes) · LW · GW

I think this should have b instead of p:

Comment by lanrian on Thoughts on Ben Garfinkel's "How sure are we about this AI stuff?" · 2019-02-06T22:20:38.314Z · score: 8 (5 votes) · LW · GW

What does OTTMH mean?

Comment by lanrian on Why is this utilitarian calculus wrong? Or is it? · 2019-02-01T18:34:56.316Z · score: 1 (1 votes) · LW · GW

Wait, are you claiming that humans have moral intuitions because it maximizes global utility? Surely moral intuitions have been produced by evolution. Why would evolution select for agents with behaviour that maximize global utility?

Comment by lanrian on The 3 Books Technique for Learning a New Skilll · 2019-01-25T22:15:49.119Z · score: 6 (5 votes) · LW · GW

3blue1brown has a series on the essence of linear algebra as well. It's pretty great, and and could do well as the Why.

I also like Linear Algebra Done Right a lot, but it doesn't fit neatly into this framework. It's a bit too rigorous to be Why, not practical enough to be How, and it's approach differs enough from other books to make it difficult to look things up in.

Comment by lanrian on Life can be better than you think · 2019-01-21T08:48:09.519Z · score: 7 (5 votes) · LW · GW

The last person that I remember writing something along the lines of You don’t have to experience negative emotion, on LessWrong didn't turn out well.

I'm not sure what this means.

Either you're saying that the LW community disapproved of this person. In this case, see the frontpage guidelines: try to present arguments for your own view instead of stating opinions of others.

Or you're saying that this person hacked around with they're own emotions until they lost meaning. In this case, I am very curious about what they did!

(Or you're saying something completely different.)

Comment by lanrian on Why not tool AI? · 2019-01-20T08:58:30.980Z · score: 3 (2 votes) · LW · GW

Eric Drexler's report on comprehensive AI services also contains relevant readings. Here is Rohin's summary of it.

Comment by lanrian on Does anti-malaria charity destroy the local anti-malaria industry? · 2019-01-06T00:49:12.541Z · score: 26 (9 votes) · LW · GW

While not comprehensively covered, GiveWell mentions this in a few places. The second point here links to a report with this section discussing whether people are willing to pay for nets, as well as a link to this old blog post which briefly makes the argument that people won't buy their own nets, since previous hand-outs (from other charities) have resulted in a lack of local producers and an expectation of free nets. They also mention that nets have some positive externalities, and mostly benefits children, who aren't the ones paying, which gives some reason to subsidize them.

Comment by lanrian on You can be wrong about what you like, and you often are · 2018-12-21T21:20:24.144Z · score: 3 (2 votes) · LW · GW

Blind spots and biases can be harmful to your goals without being harmful to your reproductive fitness. Being wrong about which future situations will make you (permanently) happier is an excellent example of such a blind spot.

Comment by lanrian on 2018 AI Alignment Literature Review and Charity Comparison · 2018-12-19T16:49:48.265Z · score: 6 (4 votes) · LW · GW

Shah et al.'s Value Learning Sequence is a short sequence of blog posts outlining the specification problem.

The link goes to the Embedded Agency sequence, not the value learning sequence (

Comment by lanrian on An Extensive Categorisation of Infinite Paradoxes · 2018-12-16T16:29:25.976Z · score: 3 (2 votes) · LW · GW
"Indeed Pascal's Mugging type issues are already present with the more standard infinities."
Right, infinity of any kind (surreal or otherwise) doesn't belong in decision theory.

But Pascal's Mugging type issues are present with large finite numbers, as well. Do you bite the bullet in the finite case, or do you think that unbounded utility functions don't belong in decision theory, either?

Comment by lanrian on An Extensive Categorisation of Infinite Paradoxes · 2018-12-14T09:18:03.742Z · score: 1 (1 votes) · LW · GW

Great post!

Satan's Apple: Satan has cut a delicious apple into infinitely many pieces. Eve can take as many pieces as she likes, but if she takes infinitely many pieces she will be kicked out of paradise and this will outweigh the apple. For any finite number i, it seems like she should take that apple piece, but then she will end up taking infinitely many pieces.

Proposed solution for finite Eves (also a solution to Trumped, for finite Trumps who can't count to surreal numbers):

After having eaten n pieces, Eve's decision isn't between eating n pieces and eating n+1 pieces, it's between eating n pieces and whatever will happen if she eats the n+1st piece. If Eve knows that the future Eve will be following the strategy "always eat the next apple piece", then it's a bad decision to eat the n+1st piece (since it will lead to getting kicked out of paradise).

So what strategy should Eve follow? Consider the problem of programming a strategy that an Eve-bot will follow. In this case, the best strategy is the strategy that will lead to the largest amount of finite pieces being eaten. What this strategy is depends on the hardware, but if the hardware is finite, then there exists such a strategy (perhaps count the number of pieces and stop when you reach N, for the largest N you can store and compare with). Generalising to (finite) humans, the best strategy is the strategy that results in the largest amount of finite pieces eaten, among all strategies that a human can precommit to.

Of course, if we allow infinite hardware, then the problem is back again. But that's at least not a problem that I'll ever encounter, since I'm running on finite hardware.

Comment by lanrian on The Bat and Ball Problem Revisited · 2018-12-13T09:00:00.323Z · score: 1 (1 votes) · LW · GW
However, for the other two I 'just see' the correct answer. Is this common for other people, or do you have a different split?

I think I figured out and verified the answer to all 3 questions in 5-10 seconds each, when I first heard them (though I was exposed to them in the context of "Take the cognitive reflection test which people fail because the obvious answer is wrong", which always felt like cheating to me).

If I recall correctly, the third question was easier than the second question, which was easier than bat & ball: I think I generated the correct answer as a suggestion for 2 and 3 pretty much immediately (alongside the supposedly obvious answers), and I just had to check them. I can't quite remember my strategy for bat & ball, but I think I generated the $0.1 ball, $1 bat answer, saw that the difference was $0.9 instead of $1, adjusted to $0.05, $1.05, and found that that one was correct.

Comment by lanrian on The Bat and Ball Problem Revisited · 2018-12-13T08:45:26.648Z · score: 8 (6 votes) · LW · GW
I suspect that this is less true the other two problems - ratios and exponential growth are topics that a mathematical or scientific education is more likely to build intuition for.

This seems to be contradicted by:

the bat and ball question is the most difficult on average – only 32% of all participants get it right, compared with 40% for the widgets and 48% for the lilypads. It also has the biggest jump in success rate when comparing university students with non-students.
Comment by lanrian on Worth keeping · 2018-12-07T12:18:30.360Z · score: 4 (4 votes) · LW · GW
If there are better replacements in general, then you will be inclined to replace things more readily.
The social analog is that in a community where friends are more replaceable—for instance, because everyone is extremely well selected to be similar on important axes—it should be harder to be close to anyone, or to feel safe and accepted

I can come up with a countervailing effect here, as well. Revealing problems is a risk: you might get help and be in a more trusting friendship, or you might be dumped. If there are lots of good replacements around, then getting dumped matters less, since you can find someone else. This predicts that people in communities that gather similar people might expose their problems more often, despite being replaced a higher fraction of the time.

Another difference between cars and friends is that you're going to get equally good use out of your car regardless of how you feel about it, but you're friendship is going to be different if you can credibly signal that you won't replace it (taking the selfish-rational-individual model to the extreme, you probably want to signal that you'd replace it if the friend started treating you worse, but that you wouldn't leave it just because your friend revealed problems). In a close community, that signal might get worse if you repeatedly replace friends, which predicts that you'd be less likely to replace friends in closer communities.

No empirical evidence of any of this.

Comment by lanrian on Double-Dipping in Dunning--Kruger · 2018-12-02T22:49:30.878Z · score: 24 (5 votes) · LW · GW
Participants scoring in the bottom quartile on our humor test (...) overestimated their percentile ranking
A less well-known finding of Dunning--Kruger is that the best performers will systematically underestimate how good they are, by about 15 percentile points.

Isn't this exactly what you'd expect if people were good bayesians receiving scarce evidence? Everyone starts out with assuming that they're in the middle, and as they find something easy or hard, they gradually update away from their prior. If they don't have good information about how good other people are, they won't update too much.

If you then look at the extremes, the very best and the very worst people, of course you're going to see that they should extremify their beliefs. But if everyone followed that advice, you'd ruin the accuracy of the people more towards the middle, since they haven't received enough evidence to distinguish themselves from the extremes.

(Similarly, I've heard that people often overestimate their ability on easy tasks and underestimate their ability on difficult tasks, which is exactly what you'd expect if they had good epistemics but limited evidence. If task performance is a function of task difficulty and talent for a task, and the only things you can observe is your performance, then believing that you're good at tasks you do well at and bad at tasks you fail at is the correct thing to do. As a consequence, saying that people overestimate their driving ability doesn't tell you that much about the quality of their epistemics, in isolation, because they might be following a strategy that optimises performance across all tasks.)

The finding that people at the bottom overestimate their position with 46 percentile points is somewhat more extreme than this naïve model would suggest. As you say, however, it's easily explained when you take into account that your ability to judge your performance on a task is correlated with your performance on that task. Thus, the people at the bottom are just receiving noise, so on average they stick with their prior and judge that they're about average.

Of course, just because some of the evidence is consistent with people having good epistemics doesn't mean that they actually do have good epistemics. I haven't read the original paper, but it seems like people at the bottom actually thinks that they're a bit above average, which seems like a genuine failure, and I wouldn't be surprised if there are more examples of such failures which we can learn to correct. The impostor syndrome also seems like a case where people predictably fail in fixable ways (since they'd do better by estimating that they're of average ability, in their group, rather than even trying to update on evidence).

But I do think that people often are too quick to draw conclusions from looking at a specific subset of people estimating their performance on a specific task, without taking into account how well their strategy would do if they were better or worse, or were doing a different task. This post fixes some of those problems, by reminding us that everyone lowering the estimate of their performance would hurt the people at the top, but I'm not sure if it correctly takes into account how the people in the middle of the distribution would be affected.

(The counter-argument might be that people who know about Dunning-Kruger is likely to be at the top of any distribution they find themselves in, but this seems false to me. I'd expect a lot of people to know about Dunning-Kruger (though I may be in a bubble) and there are lots of tasks where ability doesn't correlate a lot with knowing about Dunning-Kruger. Perhaps humor is an example of this.)

Comment by lanrian on The Fermi Paradox: What did Sandberg, Drexler and Ord Really Dissolve? · 2018-07-11T21:51:03.643Z · score: 1 (1 votes) · LW · GW

Sure, there are lots of ways to break calculations. That's true for any theory that's trying to calculate expected value, though, so I can't see how that's particularly relevant for anthropics, unless we have reason to believe that any of these situations should warrant some special action. Using anthropic decision theory you're not even updating your probabilities based on number of copies, so it really is only calculating expected value.

Comment by lanrian on The Fermi Paradox: What did Sandberg, Drexler and Ord Really Dissolve? · 2018-07-10T21:42:52.256Z · score: 1 (1 votes) · LW · GW

When you repeat this experiment a bunch of times, I think an SSA advocate can choose their reference class to include all iterations of the experiment. This will result in them assigning similar credences as SIA, since a randomly chosen awakening from all iterations of the experiment is likely to be one of the new copies. So the update towards SIA won't be that strong.

This way of choosing the reference class lets SSA avoid a lot of unintuitive results. But it's kind of a symmetric way of avoiding unintuitive results, in that it might work even if the theory is false.

(Which I think it is.)

Comment by lanrian on The Fermi Paradox: What did Sandberg, Drexler and Ord Really Dissolve? · 2018-07-10T21:25:34.283Z · score: 3 (2 votes) · LW · GW

I'm not sure. The simplest way that more copies of me could exist is that the universe is larger, which doesn't imply any crazy actions, except possible to bet that the universe is large/infinite. That isn't a huge bullet to bite. From there you could probably get even more weight if you thought that copies of you were more densely distributed, or something like that, but I'm not sure what actions that would imply.

Speculation: The hypothesis that future civilisations spend all their resources simulating copies of you get a large update. However, if you contrast it with the hypothesis that they simulate all possible humans, and your prior probability that they would simulate you is proportional to the number of possible humans (by some principle of indifference), the update is proportional to the prior and is thus overwhelmed by the fact that it seems more interesting to simulate all humans than to simulate one of them over and over again.

Do you have any ideas of weird hypothesis that imply some specific actions?

Comment by lanrian on The Fermi Paradox: What did Sandberg, Drexler and Ord Really Dissolve? · 2018-07-09T23:47:32.726Z · score: 3 (2 votes) · LW · GW

Sure, SIA assigns very high probability to us being in a simulation. That conclusions isn't necessarily absurd, though I think anthropic decision theory ( with aggregative ethics is a better way to think about it, and yields similar conclusions. Brian Tomasik has an excellent article about the implications

Comment by lanrian on The Fermi Paradox: What did Sandberg, Drexler and Ord Really Dissolve? · 2018-07-09T23:41:03.186Z · score: 9 (2 votes) · LW · GW

SSA and SIA aren't exactly untestable. They both make predictions, and can be evaluated according to them, e.g. SIA predicts larger universes. It could be said to predict an infinite universe with probability 1, insofar as it at all works with infinities.

The anthropic bits in their paper looks like SSA, rather than SIA.

Comment by lanrian on Paradoxes in all anthropic probabilities · 2018-06-22T20:25:41.992Z · score: 2 (1 votes) · LW · GW

My preferred way of doing anthropics while keeping probabilities around is to update your probabilities according to the chance that at least one of the decision making agents that your decision is logically linked to exists, and then prioritise the worlds where there are more of those agents by acknowledging that you're making the decision for all of them. This yields the same (correct) conclusions as SIA when you're only making decisions for yourself, and FNC when you're making decisions for all of your identical copies, but it avoids the paradoxes brought up in this article and it allows you to take into account that you're making decisions for all of your similar copies, which you want to have for newcombs problem like situations.

However, I think it's possible to construct even more contorted scenarios where conservation of expected evidence is violated for this as well. If there are 2 copies of you, a coin is flipped, and:

  • If it's heads the copies are presented with two different choices.
  • If it's tails the copies are presented with the same choice.

then you know that you will update towards heads when you're presented with a choice after a minute, since heads make it twice as likely that anyone would be presented with that specific choice. I don't know if there's any way around this. Maybe if you update your probabilities according to the chance that someone following your decision theory is around, rather than someone making your exact choice, or something like that?

Comment by lanrian on Sleeping Beauty Resolved? · 2018-06-22T15:35:16.851Z · score: 2 (1 votes) · LW · GW

Actually, I realise that you can get around this. If you use a decision theory that assumes that you are deciding for all identical copies of you, but that you can't affect the choices of copies that has diverged from you in any way, math says you will always bet correctly.

Comment by lanrian on Paradoxes in all anthropic probabilities · 2018-06-22T14:01:54.400Z · score: 2 (1 votes) · LW · GW
Yes, it's weird when you are motivated to force your future copy to do things

If you couple these probability theories with the right decision theories, this should never come up. FNC yields the correct answer if you use a decision theory that lets you decide for all your identical copies (but not the ones who has had different experiences), and SIA yields the correct answer if you assume that you can't affect the choices of the rest of your copies.

Comment by lanrian on Prisoners' Dilemma with Costs to Modeling · 2018-06-14T18:58:55.609Z · score: 6 (4 votes) · LW · GW
We will use a prisoner's dilemma where mutual cooperation produces utility 2, mutual defiction (sic) produces utility 0, and exploitation produces utility 3 for the exploiter and 0 for the exploited. Each player will also pay a penalty of ε times its depth.

Am I reading this correctly if I think that (cooperate, defect) would produce (0, 3) and (defect, defect) would produce (0, 0)? Is that an error? Because in other parts of the text it looks like (defect, defect) should be (1, 1). Also, (cooperate, defect) should be a nash equilibrium if my interpretation is correct.