What happens when your beliefs fully propagate

I find this comment vague and abstract, do you have examples in mind?

I mostly-agree, except that question 1 shouldn't say:

"In a least convenient world, would you utterly forgo all interest in return for making some small difference to global utility".

It should say: "… is there any extent to which impact on strangers' well-being would influence your choices? For example, if you were faced with a choice between reading a chapter of a kind-of-interesting book with no external impact, or doing chores for an hour and thereby saving a child's life, would you sometimes choose the latter?"

If the answer to that latter question is yes -- if expected impact on others' well-being can potentially sway your actions at some margin -- then it is worth looking into the empirical details, and seeing what bundles of global well-being and personal well-being can actually be bought, and how attractive those bundles are.

Why are you even thinking about this question?

Because even if we don't have the strength or desire to willingly renounce all selfishness, we recognize that better versions of ourselves would do so, and that perhaps there's a good way to make some lifestyle changes that look like personal sacrifices but are actually net positive (and even more so when we nurture our sense of altruism)?

Isn't this statement also a clever argument for why you're not going to do it anyway, at least to an extent?

For what it's worth, I have no trouble answering "yes" to 1, because for me it doesn't have the altruistic connotations it probably has for other people. My utility function is very selfish and I'm okay with that.

Maybe the personal interests are the real utility, but we don't want to admit it -- because for our survival as members of social species it is better to pretend that our utility is aligned with utility of others, although it is probably just weakly positively correlated. In more complex society the correlation is probably even weaker, because the choicespace is larger.

Or maybe just the utility choice mechanism is broken in this society, because it evolved in an ancient environment. It probably uses some mechanism like "if you are rewarded for doing something, or if you see that someone is rewarded for something, then develop a strong desire... and keep the desire even in times when you are not rewarded (because it may require long-term effort)". In ancient environment you would see rewards mostly for useful things. Today there are too many exciting things -- I don't say they are mostly bad, just that there is too much of them, so people's utility functions are spread too much, and sometimes there are not enough people with desire to do some critical tasks.

How to become really interested in something?

I would suggest spending time with people interested in X, because this would give one's brain signal "X is socially rewarded", which would motivate them to do X. Any other good ideas?

Why do you say it's not a little quirky interest? I'm asking this as I've been fixated on various game-making stuff for close to 20 years now, but now feel like I'm mostly going on because it's something I was really interested at 14 and subsequently tinkered with enough that it now seems like the thing I can do most interesting things with, but I suspect it's not what I'd choose to have built a compulsive interest in if I could make the choice today.

Nowadays I am getting alienated from the overall gaming culture that's still mostly optimized to primarily appeal to teenagers, and I often have trouble coming up with a justification why most games should be considered anything other than shiny escapist distractions and how the enterprise of game development aspires to anything other than being a pageant for coming up with the shiniest distraction. So I would go for both quirky, gaming has a bunch of weird insider culture things going for it, and little, most gaming and gamedev has little effect in the big picture of fixing things that make life bad for people (though distractions can be valuable too), and might have a negative effect if clever people who could make a contribution elsewhere get fixated into gamedev instead.

It does translate to a constantly growing programming skill for me, so at least there's that good reason to keep up at it. But that's more a side effect than a primary value of the interest.

Just being curious, what are you doing now, and what were your interests before?

Thanks, your statements align very well with my anticipations. I don't expect this to be easy, and I don't know exactly what I should be doing, but I know I really want to figure it out.

tl;dr: If you can spot non-sequiturs in your writing, and you put a lot of weight on the conclusion it's pointing at, it's a really good idea to take the time to fill in all the sequiturs.

Writing an argument in detail is a good way to improve the likelihood that your argument isn't somewhere flawed. Consider:

• Writing allows reduction. By pinning the argument to paper, you can separate each logical step, and make sure that each step makes sense in isolation.
• Writing gives the argument stability. For example, the argument won't secretly change when you think about it while you're in a different mood. This can help to prevent you from implicitly proving different points of your argument from contradictory claims.
• Writing makes your argument vastly easier to share. Like in open source software, enough eyeballs makes all bugs trivial.

Further, notice that we probably underestimate the value of improving our arguments, and are overconfident in apparently-solid logical arguments. If an argument contains 20 inferences in sequence, and you're wrong about such inferences 5% without noticing the misstep, then you have about a 64% chance of being wrong somewhere in the argument. If you can reduce your chance of a misstep in logic to 1% per inference, then you only have an 18% chance of being wrong, somewhere. Improving the reliability of the steps in your arguments, then, has a high value-of-information -- even when 1% and 5% both feel like similar amounts of uncertainty. Conjunction fallacy. It is probable, then, that we underestimate the value of information attained by subjecting ourselves to processes that improve our arguments.

If being wrong about an argument is highly costly -- if you would stand to lose much by believing incorrectly -- then it is well worth writing these sorts of arguments formally, and ensuring that you're getting them right.

All that said... I suspect I know exactly what you're talking about. I haven't performed a similar, convulsive update myself, but I can practically feel the pressure for it in my own head, growing. I fight that update longer than parts of me think I should, because I'm afraid of strong mental attractors. If you can write the sound, solid argument publicly, I will be happy to double-check your steps.

Yes. Even if this one is right, you're still running on corrupt hardware and need to know when to consciously lower your enthusiasm.

I am guessing that a part of what EY had in mind was that large organizations tend to lose its purposes and work towards self-preservation as much as towards the original objective. $10M/year translates into less than 100 full-time jobs, which is probably a good rule of thumb for an organization becoming too big to keep its collective eyes on the ball.

What chores did you have in mind? Seems like you wouldn't even need 100 million dollars to hire the world's best mathematicians (if you have any hope of doing so) and give all involved a comfortable lifestyle. Do you mean, billions total by the time we get FAI? Because you started out speaking of donating such-and-such "a year".

I have to admit that no particular examples come to mind, but usually in the comments threads on topics such as optimal giving, and occasional posts arguing agains the probability of the singularity. I certainly have seen some, but can't remember where exactly, so any search you do will probably be as effective as my own. To present you with a few possible arguments (which I believe to varying degrees of certainty)

-A lot of the arguments for becoming commited to donating to FAI are based on "even if theres a low probability of it happening, the expected gains are incredibly huge". I'm wary of this argument because I think it can be applied anywhere. For instance, even now, and certainly 40 years ago, one could make a credible argument that theres a not insignificant chance of a nuclear war eradicating human life from the planet. So we should contribute all our money to organisations devoted to stopping nuclear war. -This leads directly to another argument- how effective do we expect the SI to be? Is friendly AI possible? Are SI going to be the ones to find it? If SI create friendliness, will it be implemented? If I had devoted all my money to the CND, I would not have had a significant impact on the proliferation of nuclear weaponry. -A lot of the claims based on a singularity assume that intelligence can solve all problems. But there may be hard limits to the universe. If the speed of light is the limit, then we are trapped with finite resources, and maybe there is no way for us to use them much more efficiently than we can now. Maybe cold fusion isn't possible, maybe nanotechnology can't get much more sophisticated? -Futurism is often inaccurate. The jokes about "wheres my hover car" are relevant- the progress over the last 200 years has rocketed in some spheres but slowed in others. For instance, current medical advances have been slowing recently. They might jump forwards again, but maybe not. Predicting which bits of science will advance in a certain time scale are unlikely. -Intelligence might have a hard limit, or an exponential decay. It could be argued that we might be able to wire up millions of humanlike intelligence in a computer array, but that might hit physical limits

Like you seemed to be looking toward in your post, I'm focusing on making as much money as possible right now. For me, the best path in that direction is probably in the field of tech startups.

This has the added benefit of probably being the right choice even if I decide for some reason that SIAI isn't the most important cause. Pretty much any set of preferences seems to me like it will be best served by making a lot of money (even something like "live a simple and humble life and focus only on your own happiness" is probably easier if you can just retire early and then have the freedom to do whatever you want).

Edited to add: It's worth noting that making as much money as possible via a tech startup is also good for me personally because I actually do have a fair amount of motivation and interest in that area (if I never heard of FAI, tech entrepreneurship would still be appealing to me, although I probably wouldn't dive into it as much as I did once I decided I wanted to work toward FAI). So I don't necessarily think the best thing for everyone who wants to contribute to FAI is to try to make as much money as possible (and certainly not necessarily via a tech startup), but for me personally it seems like the best path.

I've had pretty much the exact same experience. Some random thoughts (despite the surface similarities, I'm describing myself not you here so usual caveats apply).

For me the trigger was deciding that I wanted to be a utilitarian. At that point the Intelligence Explosion hypothesis (and also GiveWell's message that some charities are orders of magnitude more effective than others) went from being niggling concerns to urgently important concerns. Unlike you, I do have a visceral negative feeling associated with the end of the world. Or rather I did - now it just feels like the new normal (but I'd still put a lot of effort into reducing its probability very slightly).

It's entirely nonobvious to me that FAI is the marginally most important thing for humanity to be working on right now. That depends factors such as whether the approach can be made safe and viable, how different x-risks compare with each other, and some icky stuff to do with social dynamics and information hazards. In other words, it's a question that I have no hope of correctly answering as an individual. But this doesn't suggest that inaction is the best strategy, but rather that I should be supporting smarter people who are trying to answer that kind of question.

The most important thing for humanity to be doing right now is probably very different from the most important thing for me to be doing. It depends on where my comparative advantage is. Current top priority for me is actually just sorting my own life out.

Other commenters here have made the analogy with religious conversion and/or brainwashing. I think that the analogy is valid in as far as something similar may have been happening in my brain; on the other hand, I don't really know what to do with this information. It doesn't mean I'm wrong. I'm still on the lookout for any really good skeptical arguments that address the main issues.

I've really been suffering due to a lack of a community of like-minded people. (At least face-to-face - there are people on the Internet obviously, but I have trouble getting much from that either because they're too busy or because I'm just not good at staying in touch that way). Obviously if I surround myself only with people with the exact same worldview it increases the risk of epistemic closure; on the other hand, having no-one around who shares my goals is very frustrating. Incidentally I've generally got on well with people from Giving What We Can when I have the opportunity to meet them.

I've been suffering depression-like symptoms - they are very likely connected to this whole thing somehow, but I don't know the exact reason. Hypothesis 1 is the lack-of-community thing. Hypothesis 2 is that like you I felt I could do anything, and then discovered that actually no I couldn't.

The "conversion" experience felt very much like "beliefs fully propagating" but I'm not sure that's entirely accurate - I think it was just a few (important) beliefs that propagated. Like you, I felt like my belief map was wiped clean, but actually most of my old beliefs still seem to be valid and useful. It seems to be mainly just information relevant to the safety of far people that I'd kept compartmentalized.

Do not take the frequentist approach!

This looks good. Thanks!

I see. I wasn't asserting that you are going to do work you hate, however. I was mainly looking at the value of having a seemingly unachievable and incredibly broad goal as one's primary motivation.

I'm sure you have a much more nuanced view of how and why you are undertaking this life change, and I don't want to discourage you. Seeing as how the general consensus is that FAI is the most important thing to be doing, I think it would take a lot of effort to discourage you. I just can't help but think that there should be a primary technical interest in the problems presented by FAI motivating these kinds of decisions. If it was me, I would be confused as to what exactly I would be working on, which would be very discouraging.