[Resolved] Is the SIA doomsday argument wrong? 2014-12-13T06:01:31.965Z · score: 6 (10 votes)
[Link] Physics-based anthropics? 2014-11-14T07:02:03.307Z · score: 5 (8 votes)
Multiverse-Wide Preference Utilitarianism 2014-01-30T18:08:55.878Z · score: 14 (25 votes)
International cooperation vs. AI arms race 2013-12-05T01:09:33.431Z · score: 20 (24 votes)


Comment by brian_tomasik on Electrons don’t think (or suffer) · 2019-01-03T00:09:36.011Z · score: 7 (4 votes) · LW · GW

Electrons have physical properties that vary all the time: position, velocity, distance to the nearest proton, etc (ignoring Heisenberg uncertainty complications). But yeah, these variables rely on the electron being embedded in an environment.

Comment by brian_tomasik on Preliminary thoughts on moral weight · 2018-08-18T15:18:30.967Z · score: 1 (1 votes) · LW · GW

The naive form of the argument is the same between the classic and moral-uncertainty two-envelopes problems, but yes, while there is a resolution to the classic version based on taking expected values of absolute rather than relative measurements, there's no similar resolution for the moral-uncertainty version, where there are no unique absolute measurements.

Comment by brian_tomasik on Preliminary thoughts on moral weight · 2018-08-15T12:16:42.542Z · score: 10 (4 votes) · LW · GW

I think the moral-uncertainty version of the problem is fatal unless you make further assumptions about how to resolve it, such as by fixing some arbitrary intertheoretic-comparison weights (which seems to be what you're suggesting) or using the parliamentary model.

Comment by brian_tomasik on Is life worth living? · 2017-10-03T03:51:17.640Z · score: 1 (1 votes) · LW · GW

Currently I don't care much about strongly positive events, so at this point I'd say no. In the throes of such a positive event I might change my mind. :)

Comment by brian_tomasik on Is life worth living? · 2017-10-03T03:49:47.457Z · score: 1 (1 votes) · LW · GW

Yes, because I don't see any significant selfish upside to life, only possible downside in cases of torture/etc. Life is often fun, but I don't strongly care about experiencing it.

Comment by brian_tomasik on Is life worth living? · 2017-10-03T03:46:47.176Z · score: 0 (0 votes) · LW · GW

Yeah, but it would be very bad relative to my altruistic goals if I died any time soon. The thought experiment in the OP ignores altruistic considerations.

Comment by brian_tomasik on Naturalized induction – a challenge for evidential and causal decision theory · 2017-09-23T05:11:16.963Z · score: 1 (1 votes) · LW · GW

However, if you believe that the agent in world 2 is not an instantiation of you, then naturalized induction concludes that world 2 isn't actual and so pressing the button is safe.

By "isn't actual" do you just mean that the agent isn't in world 2? World 2 might still exist, though?

Comment by brian_tomasik on Is life worth living? · 2017-09-03T06:35:05.468Z · score: 1 (1 votes) · LW · GW

I assume the thought experiment ignores instrumental considerations like altruistic impact.

For re-living my actual life, I wouldn't care that much either way, because most of my experiences haven't been extremely good or extremely bad. However, if there was randomness, such that I had some probability of, e.g., being tortured by a serial killer, then I would certainly choose not to repeat life.

Comment by brian_tomasik on S-risks: Why they are the worst existential risks, and how to prevent them · 2017-06-20T22:06:55.802Z · score: 3 (3 votes) · LW · GW

Is it still a facepalm given the rest of the sentence? "So, s-risks are roughly as severe as factory farming, but with an even larger scope." The word "severe" is being used in a technical sense (discussed a few paragraphs earlier) to mean something like "per individual badness" without considering scope.

Comment by brian_tomasik on S-risks: Why they are the worst existential risks, and how to prevent them · 2017-06-20T22:04:12.248Z · score: 1 (1 votes) · LW · GW

Thanks for the feedback! The first sentence below the title slide says: "I’ll talk about risks of severe suffering in the far future, or s-risks." Was this an insufficient definition for you? Would you recommend a different definition?

Comment by brian_tomasik on False thermodynamic miracles · 2015-08-14T08:28:49.577Z · score: 1 (1 votes) · LW · GW

I guess you mean that the AGI would care about worlds where the explosives won't detonate even if the AGI does nothing to stop the person from pressing the detonation button. If the AGI only cared about worlds where the bomb didn't detonate for any reason, it would try hard to stop the button from being pushed.

But to make the AGI care about only worlds where the bomb doesn't go off even if it does nothing to avert the explosion, we have to define what it means for the AGI to "try to avert the explosion" vs. just doing ordinary actions. That gets pretty tricky pretty quickly.

Anyway, you've convinced me that these scenarios are at least interesting. I just want to point out that they may not be as straightforward as they seem once it comes time to implement them.

Comment by brian_tomasik on False thermodynamic miracles · 2015-08-12T23:07:33.628Z · score: 2 (2 votes) · LW · GW

Fair enough. I just meant that this setup requires building an AGI with a particular utility function that behaves as expected and building extra machinery around it, which could be more complicated than just building an AGI with the utility function you wanted. On the other hand, maybe it's easier to build an AGI that only cares about worlds where one particular bitstring shows up than to build a friendly AGI in general.

Comment by brian_tomasik on False thermodynamic miracles · 2015-08-12T00:43:50.770Z · score: 0 (0 votes) · LW · GW

I'm nervous about designing elaborate mechanisms to trick an AGI, since if we can't even correctly implement an ordinary friendly AGI without bugs and mistakes, it seems even less likely we'd implement the weird/clever AGI setups without bugs and mistakes. I would tend to focus on just getting the AGI to behave properly from the start, without need for clever tricks, though I suppose that limited exploration into more fanciful scenarios might yield insight.

Comment by brian_tomasik on Satisficers want to become maximisers · 2015-08-11T22:25:57.345Z · score: 0 (1 votes) · LW · GW

As I understand it, your satisficing agent has essentially the utility function min(E[paperclips], 9). This means it would be fine with a 10^-100 chance of producing 10^101 paperclips. But isn't it more intuitive to think of a satisficer as optimizing the utility function E[min(paperclips, 9)]? In this case, the satisficer would reject the 10^-100 gamble described above, in favor of just producing 9 paperclips (whereas a maximizer would still take the gamble and hence would be a poor replacement for the satisficer).

A satisficer might not want to take over the world, since doing that would arouse opposition and possibly lead to its defeat. Instead, the satisficer might prefer to request very modest demands that are more likely to be satisfied (whether by humans or by an ascending uncontrolled AI who wants to mollify possible opponents).

Comment by brian_tomasik on Two-boxing, smoking and chewing gum in Medical Newcomb problems · 2015-07-01T21:52:11.985Z · score: 0 (0 votes) · LW · GW

If there were a perfect correlation between choosing to one-box and having the one-box gene (i.e., everyone who one-boxes has the one-box gene, and everyone who two-boxes has the two-box gene, in all possible circumstances), then it's obvious that you should one-box, since that implies you must win more. This would be similar to the original Newcomb problem, where Omega also perfectly predicts your choice. Unfortunately, if you really will follow the dictates of your genes under all possible circumstances, then telling someone what she should do is useless, since she will do what her genes dictate.

The more interesting and difficult case is when the correlation between gene and choice isn't perfect.

Comment by brian_tomasik on Two-boxing, smoking and chewing gum in Medical Newcomb problems · 2015-07-01T21:51:57.203Z · score: 0 (0 votes) · LW · GW

(moved comment)

Comment by brian_tomasik on Two-boxing, smoking and chewing gum in Medical Newcomb problems · 2015-06-29T22:15:10.147Z · score: 1 (1 votes) · LW · GW

I assume that the one-boxing gene makes a person generically more likely to favor the one-boxing solution to Newcomb. But what about when people learn about the setup of this particular problem? Does the correlation between having the one-boxing gene and inclining toward one-boxing still hold? Are people who one-box only because of EDT (even though they would have two-boxed before considering decision theory) still more likely to have the one-boxing gene? If so, then I'd be more inclined to force myself to one-box. If not, then I'd say that the apparent correlation between choosing one-boxing and winning breaks down when the one-boxing is forced. (Note: I haven't thought a lot about this and am still fairly confused on this topic.)

I'm reminded of the problem of reference-class forecasting and trying to determine which reference class (all one-boxers? or only grudging one-boxers who decided to one-box because of EDT?) to apply for making probability judgments. In the limit where the reference class consists of molecule-for-molecule copies of yourself, you should obviously do what made the most of them win.

Comment by brian_tomasik on Taking Occam Seriously · 2015-06-14T02:26:34.997Z · score: 6 (2 votes) · LW · GW

Paul's site has been offline since 2013. Hopefully it will come back, but in the meanwhile, here are links to most of his pieces on Internet Archive.

Comment by brian_tomasik on Seeking Estimates for P(Hell) · 2015-03-23T20:46:17.261Z · score: 0 (0 votes) · LW · GW

Good point. Also, in most multiverse theories, the worst possible experience necessarily exists somewhere.

Comment by brian_tomasik on Seeking Estimates for P(Hell) · 2015-03-22T22:18:26.310Z · score: 3 (3 votes) · LW · GW

From a practical perspective, accepting the papercut is the obvious choice because it's good to be nice to other value systems.

Even if I'm only considering my own values, I give some intrinsic weight to what other people care about. ("NU" is just an approximation of my intrinsic values.) So I'd still accept the papercut.

I also don't really care about mild suffering -- mostly just torture-level suffering. If it were 7 billion really happy people plus 1 person tortured, that would be a much harder dilemma.

In practice, the ratio of expected heaven to expected hell in the future is much smaller than 7 billion to 1, so even if someone is just a "negative-leaning utilitarian" who cares orders of magnitude more about suffering than happiness, s/he'll tend to act like a pure NU on any actual policy question.

Comment by brian_tomasik on Seeking Estimates for P(Hell) · 2015-03-22T00:03:33.217Z · score: 9 (9 votes) · LW · GW

Short answer:

Donate to MIRI, or split between MIRI and GiveWell charities if you want some fuzzies for short-term helping.

Long answer:

I'm a negative utilitarian (NU) and have been thinking since 2007 about the sign of MIRI for NUs. (Here's some relevant discussion.) I give ~70% chance that MIRI's impact is net good by NU lights and ~30% that it's net bad, but given MIRI's high impact, the expected value of MIRI is still very positive.

As far as your question: I'd put the probability of uncontrolled AI creating hells higher than 1 in 10,000 and the probability that MIRI as a whole prevents that from happening higher than 1 in 10,000,000. Say such hells used 10^-15 of the AI's total computing resources. Assuming computing power to create ~10^30 humans for ~10^10 years, MIRI would prevent in expectation ~10^18 hell-years. Assuming MIRI's total budget ever is $1 billion (too high), that's ~10^9 hell-years prevented per dollar. Now apply rigorous discounts to account for priors against astronomical impacts and various other far-future-dampening effects. MIRI still seems very promising at the end of the calculation.

Comment by brian_tomasik on [Link] Physics-based anthropics? · 2015-03-17T01:47:57.251Z · score: 0 (0 votes) · LW · GW

Nice point. :)

That said, your example suggests a different difficulty: People who happen to be special numbers n get higher weight for apparently no reason. Maybe one way to address this fact is to note that what number n someone has is relative to (1) how the list is enumerated and (2) what universal Turing machine is being used for KC in the first place, and maybe averaging over these arbitrary details would blur the specialness of, say, the 1-billionth observer according to any particular coding scheme. Still, I doubt the KCs of different people would be exactly equal even after such adjustments.

Comment by brian_tomasik on Can we decrease the risk of worse-than-death outcomes following brain preservation? · 2015-02-21T23:29:15.813Z · score: 3 (3 votes) · LW · GW

Ah, got it. Yeah, that would help, though there would remain many cases where bad futures come too quickly (e.g., if an AGI takes a treacherous turn all of a sudden).

Comment by brian_tomasik on Can we decrease the risk of worse-than-death outcomes following brain preservation? · 2015-02-21T23:16:57.764Z · score: 7 (7 votes) · LW · GW

A "do not resuscitate" kind of request would probably help with some futures that are mildly bad in virtue of some disconnect between your old self and the future (e.g., extreme future shock). But in those cases, you could always just kill yourself.

In the worst futures, presumably those resuscitating you wouldn't care about your wishes. These are the scenarios where a terrible future existence could continue for a very long time without the option of suicide.

Comment by brian_tomasik on Inverse relationship between belief in foom and years worked in commercial software · 2015-01-15T03:12:51.414Z · score: 1 (1 votes) · LW · GW

This is awesome! Thank you. :) I'd be glad to copy it into my piece if I have your permission. For now I've just linked to it.

Comment by brian_tomasik on Inverse relationship between belief in foom and years worked in commercial software · 2015-01-09T02:30:00.696Z · score: 1 (1 votes) · LW · GW

Cool. Another interesting question would be how the views of a single person change over time. This would help tease out whether it's a generational trend or a generic trend with getting older.

In my own case, I only switched to finding a soft takeoff pretty likely within the last year. The change happened as I read more sources outside LessWrong that made some compelling points. (Note that I still agree that work on AI risks may have somewhat more impact in hard-takeoff scenarios, so that hard takeoffs deserve more than their probability's fraction of attention.)

Comment by brian_tomasik on Inverse relationship between belief in foom and years worked in commercial software · 2015-01-08T03:19:12.983Z · score: 0 (0 votes) · LW · GW

Good question. :) I don't want to look up exact ages for everyone, but I would guess that this graph would look more like a teepee, since Yudkowsky, Musk, Bostrom, etc. would be shifted to the right somewhat but are still younger than the long-time software veterans.

Comment by brian_tomasik on Inverse relationship between belief in foom and years worked in commercial software · 2015-01-07T21:06:17.432Z · score: 0 (0 votes) · LW · GW

Good points. However, keep in mind that humans can also use software to do boring jobs that require less-than-human intelligence. If we were near human-level AI, there may by then be narrow-AI programs that help with the items you describe.

Comment by brian_tomasik on Inverse relationship between belief in foom and years worked in commercial software · 2015-01-07T20:51:52.236Z · score: 3 (3 votes) · LW · GW

Thanks for the comment. There is some "multiple hypothesis testing" effect at play in the sense that I constructed the graph because of a hunch that I'd see a correlation of this type, based on a few salient examples that I knew about. I wouldn't have made a graph of some other comparison where I didn't expect much insight.

However, when it came to adding people, I did so purely based on whether I could clearly identify their views on the hard/soft question and years worked in industry. I'm happy to add anyone else to the graph if I can figure out the requisite data points. For instance, I wanted to add Vinge but couldn't clearly tell what x-axis value to use for him. For Kurzweil, I didn't really know what y-axis value to use.

Comment by brian_tomasik on Inverse relationship between belief in foom and years worked in commercial software · 2015-01-07T20:44:44.710Z · score: 1 (1 votes) · LW · GW

This is a good point, and I added it to the penultimate paragraph of the "Caveats" section of the piece.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-15T05:03:27.217Z · score: 2 (2 votes) · LW · GW

Thanks for the correction! I changed "endorsed" to "discussed" in the OP. What I meant to convey was that these authors endorsed the logic of the argument given the premises (ignoring sim scenarios), rather than that they agreed with the argument all things considered.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T19:16:29.456Z · score: 0 (0 votes) · LW · GW

I don't think question pits SSA against SIA; rather, it concerns what SIA itself implies. But I think my argument was wrong, and I've edited the top-level post to explain why.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T19:08:26.866Z · score: 0 (0 votes) · LW · GW

Not sure of the relevance of eternal inflation. However, I think I've realized where my argument went astray and have updated the post accordingly. Let me know if we still disagree.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T19:07:03.545Z · score: 0 (0 votes) · LW · GW

Thanks! What you explain in your second paragraph was what I was missing. The distinction isn't between hypotheses where there's one copy of me versus several (those don't work) but rather between hypotheses where there's one copy of me versus none, and an early filter falsely predicts lots of "none"s.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T08:31:45.787Z · score: 2 (2 votes) · LW · GW

Yes, but the Fermi paradox and Great Filter operate within a given branch of the MWI multiverse.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T08:30:56.153Z · score: 0 (0 votes) · LW · GW

Thanks! I think this is basically a restatement of Katja's argument. The problem seems to be that comparing number of brains like ours isn't the right question. The question is how many minds are exactly ours, and this number has to be the same (ignoring simulations) between (B) and (C): namely, there is one civilization exactly like ours in either case.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T07:46:45.752Z · score: 0 (0 votes) · LW · GW

Sorry, I'm not seeing it. Could you spell out how?

I agree that allowing simulation arguments changes the ball game. For instance, sim args favor universes with lots of simulated copies of you. This requires that at least one alien civilization develops AI within a given local region of the universe, which in turn requires that the filters can't be too strong. But this is different from Katja's argument.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T07:26:58.592Z · score: 0 (2 votes) · LW · GW

That's what I originally thought, but the problem is that the probabilities of each life-form having your experiences are not independent. Once we know that one (non-simulated) life-form has your experiences in our region of the universe, this precludes other life-forms having those exact experiences, because the other life forms exist somewhere else, on different-looking planets, and so can't observe exactly what you do.

Given our set of experiences, we filter down the set of possible hypotheses to those that are consistent with our experiences. Of the (non-simulation) hypotheses that remain, they all contain only one copy of us in our local region of the universe.

Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T07:21:26.724Z · score: 1 (1 votes) · LW · GW

Sorry, that sentence was confusing. :/ It wasn't really meant to say anything at all. The "filter" that we're focusing on is a statistical property of planets in general, and it's this property of planets in general that we're trying to evaluate. What happened on Earth has no bearing on that question.

That sentence was also confusing because it made it sound like a filter would happen on Earth, which is not necessarily the case. I edited to say "We know the filter on Earth (if any)", adding the "if any" part.

Comment by brian_tomasik on You have a set amount of "weirdness points". Spend them wisely. · 2014-12-13T07:17:15.264Z · score: 4 (4 votes) · LW · GW

Thanks, Peter. :) I agree about appearing normal when the issue is trivial. I'm not convince about minimizing weirdness on important topics. Some counter-considerations:

  • People like Nick Bostrom seem to acquire prestige by taking on many controversial ideas at once. If Bostrom's only schtick were anthropic bias, he probably wouldn't have reached FP's top 100 thinkers.
  • Focusing on only one controversial issue may make you appear single-minded, like "Oh, that guy only cares about X and can't see that Y and Z are also important topics."
  • If you advocate many things, people can choose the one they agree with most or find easiest to do.
Comment by brian_tomasik on [Resolved] Is the SIA doomsday argument wrong? · 2014-12-13T07:02:20.601Z · score: 1 (1 votes) · LW · GW

Thanks, Wes_W. :)

When using SIA (which is actually an abbreviation of SSA+SIA), there are no reference classes. SIA favors hypotheses in proportion to how many copies of your subjective experiences they contain. Shulman and Bostrom explain why on p. 9 of this paper, in the paragraph beginning with "In the SSA+SIA combination".

We know the filter on Earth (if any) can't be early or middle because we're here, though we don't know what the filter looks like on planets in general. If the filter is late, there are many more boxes at our general stage. But SIA doesn't care how many are at our general stage; it only cares how many are indistinguishable from us (including having the label "Earth" on the box). So no update.

Comment by brian_tomasik on [Link] Physics-based anthropics? · 2014-11-22T11:28:01.449Z · score: 0 (0 votes) · LW · GW

One of the most striking things about anthropics is that (seemingly) whatever approach is taken, there are very weird conclusions.

Yes. :) The first paragraph here identifies at least one problem with every anthropic theory I'm aware of.

Comment by brian_tomasik on [Link] Physics-based anthropics? · 2014-11-15T06:50:05.015Z · score: 1 (1 votes) · LW · GW

I think this essentially leads to SIA. Since you're adding utilities over different copies of you, it follows that you care more about universes in which there are more copies of you.

Of course, it's slightly different from SIA because SIA wants more copies of anyone, whether you or not. If the proportion of individuals who are you remains constant, then SIA is equivalent.

Elsewhere in my essay, I discuss a prudential argument (which I didn't invent) for assuming there are lots of copies of you. Not sure if that's the same as Armstrong's proposal.

PSA is essentially favoring more copies of you per unit of spacetime / physics / computation / etc. -- as long as we understand "copy of you" to mean "instance of perceiving all the data you perceive right now" rather than just a copy of your body/brain but in a different environment.

Comment by brian_tomasik on [Link] Physics-based anthropics? · 2014-11-15T06:20:41.938Z · score: 2 (2 votes) · LW · GW

Yes, that's right. Note that SIA also favors sim hypotheses, but it does so less strongly because it doesn't care whether the sims are of Earth-like humans or of weirder creatures.

Here's a note I wrote to myself yesterday:

Like SIA, my PSA anthropics favors the sim arg in a stronger way than normal anthropics.

The sim arg works regardless of one's anthropic theory because it requires only a principle of indifference over indistinguishable experiences. But it's a trilemma, so it might be that humans go extinct or post-humans don't run early-seeming sims.

Given the existence of aliens and other universes, the ordinary sim arg pushes more strongly for us being a sim because even if humans go extinct or don't run sims, whichever civilization out there runs lots of sims should have lots of sims of minds like ours, so we should be in their sims.

PSA doesn't even need aliens. It directly penalizes hypotheses that predict fewer copies of us in a given region of spacetime. Say we're deciding between

H1: no sims of us


H2: 1 billion sims of us.

H1 would have a billion-fold bigger probability penalty than H2. Even if H2 started out being millions of times less probable than H1, it would end up being hundreds of times more probable.

Also note that even if we're not in a sim, then PSA, like SIA, yields Katja's doomsday argument based on the Great Filter.

Either way it looks very unlikely there will be a far future, ignoring model uncertainty and unknown unknowns.

Comment by brian_tomasik on [Link] Physics-based anthropics? · 2014-11-14T08:13:32.156Z · score: 0 (0 votes) · LW · GW

Cool -- thanks! Yeah, my proposal is just about how to conceptualize the sample space, but it would be trivial to replace

count(stuff I observe) / count(all stuff)


measure(stuff I observe) / measure(all stuff)

for some non-constant measure function.

Comment by brian_tomasik on A personal history of involvement with effective altruism · 2014-07-03T04:53:15.742Z · score: 1 (1 votes) · LW · GW

This is where his fellow scientists call him a "crackpot" because he can't replicate any of his experimental findings. ;)

More seriously, the sim could modify his observations to make him observe the right things. For instance, change the photons entering his eyes to be in line with what they should be, change the historical records a la 1984, etc. Or let him add an epicycle to his theory to account for the otherwise unexplainable results.

In practice, I doubt atomic-level effects are ever going to produce clearly observable changes outside of physics labs, so 99.99999% of the time the simulators wouldn't have to worry about this as long as they simulated macroscopic objects to enough detail.

Comment by brian_tomasik on A personal history of involvement with effective altruism · 2014-07-02T11:07:05.712Z · score: 1 (1 votes) · LW · GW

What is and isn't simulated to a high degree of detail can be determined dynamically. If people decide they want to investigate a hill, some system watching the sim can notice that and send a signal that the sim needs to make the hill observations correspond with quantum/etc. physics. This shouldn't be hard to do. For instance, if the theory predicts observation X +/- Y, you can generate some random numbers centered around X with std. dev. Y. Or you can make them somewhat different if the theory is wrong and to account for model uncertainty.

If the scientists would do lots of experiments that are connected in complex ways such that consistency requires them to come out with certain complex relationships, you'd need to get somewhat more fancy with faking the measurements. Worst case, you can actually do a brute-force sim of that part of physics for the brief period required. And yeah, as you say, you can always revert to a previous state if you screw up and the scientists find something amiss, though you probably wouldn't want to do that too often.

There's no guarantee (if we are in a sim) that the laws of physics are the same in our universe as they are in baseline; we may, in fact, have laws of physics specifically designed to be easier to compute.


Comment by brian_tomasik on A personal history of involvement with effective altruism · 2014-06-30T11:09:50.309Z · score: 2 (2 votes) · LW · GW

Hi CCC :)

there's no reason why someone in the future couldn't run a simulation up to (say) 1800, save that, and then run several simulations from that date forwards, each with little tweaks

Yep, exactly. That's how you can get more than 7/107 of the people in 2014.

That is, if we're being simulated, then every individual atom, every electron and proton, is being simulated.

Probably not, though. In Bostrom's simulation-argument paper, he notes that you only need the environment to be accurate enough that observers think the sim is atomically precise. For instance, when they perform quantum experiments, you make those experiments come out right, but that doesn't mean you actually have to simulate quantum mechanics everywhere. Because superficial sims would be vastly cheaper, we should expect vastly more of them, so we'd probably be in one of them.

Many present-day computer simulations capture high-level features of a system without delving into all the gory details. Probably most sims could suffice to have intermediate levels of detail for physics and even minds. (E.g., maybe you don't need to simulate every neuron, just their higher-level aggregate behaviors, except when neuroscientists look at individual neurons.)

Of course, the fact that our sim (if we are a sim) is running at all implies that the baseline must have the computing power to run us; in comparison with which, everything that we could possibly do with computing power is so trivial

This is captured by the N term in my rough calculations above. If the basement has gobs of computing power, that means N is really big. But N cancels out from the final action-relevant ie/f expression.

Comment by brian_tomasik on Motivators: Altruistic Actions for Non-Altruistic Reasons · 2014-06-28T05:01:00.739Z · score: 5 (5 votes) · LW · GW

This is a really excellent post!

I agree the revulsion against ulterior motives for altruism is somewhat detrimental but also somewhat rational. Using ulterior motives seems often like a good idea, but genuine caring can be good to cultivate too because it may be more robust against your pursuits changing when the next big thing comes along.

Two examples come to mind not doing things because of insufficient recognition:

  • Wikipedia contributions: People sometimes write blog posts or essays almost exclusively summarizing factual content. Such summaries could be added to Wikipedia and presumably would have had much bigger impact there, but one reason people don't contribute to Wikipedia is lack of authorship. I've tried to get around this barrier by compiling a list of my Wikipedia contributions and sharing the more important ones on Facebook.
  • Google Grants: I think signing up charities for Google Grants can be a high-impact activity, but I can only do so much of it because it's not the most interesting project, and it doesn't currently have high status in the EA community.

In other areas of life, I've also seen lots of examples where people are reluctant to help others because they won't get enough credit for helping. One way to help address this is to give acknowledgements to others (like you did at the end of your post!). Another thing that sometimes helps me is remembering that "we're all in this together," and picturing my sense of "ownership" as extending over the achievements of the whole group rather than just myself. ("There's no 'i' in 'team'".)

Not wanting to do tedious manual chores can be somewhat sensible if it's not your comparative advantage. It would be better to earn money and hire someone else to do them, unless you'd be doing them in your leisure time or you don't otherwise have high earning/research/outreach potential.

Comment by brian_tomasik on A personal history of involvement with effective altruism · 2014-06-28T04:37:36.163Z · score: 2 (1 votes) · LW · GW

Thanks, CCC. :)

Simulating humans near the singularity may be more interesting than simulating hunter-gatherers, so it may be that the fraction of sims around now is more than 7/107.

One reason not to expect the sims to go into the far future is that any far future with high altruistic import will have high numbers of computations, which would be expensive to simulate. It's cheaper to simulate a few billion humans who have only modest computing power. For the same reason, it's not clear that we'd have lots of sims within sims within sims, because those would get really expensive -- unless computing power is so trivially cheap in the basement that it doesn't matter.

That said, you're right there could be at least a reasonable future ahead of us in a sim, but I'm doubtful many sims run the whole length of galactic history -- again, unless the basement is drowning in computing power that it doesn't know what to do with.

Interesting point about coming up with a really good idea. But one would tend to think that the superintelligent AIs in the basement would be much better at that. Why would they bother creating dumb little humans who go on to create their own superintelligences in the sim when they could just use superintelligences in the basement? If the simulators are interested in cognitive/evolutionary diversity, maybe that could be a reason.