Should we discount extraordinary implications?

It's usually much harder to find a specific flaw in an argument than it is to see that there is probably something wrong with the conclusion. For example, I probably won't be able to spot the specific flaw in most proposed designs for a perpetual motion machine, but I can still conclude that it won't work as advertised!

Just for fun: Classic Mathematical Fallacies - can you spot the step that's wrong?

The probability of us being wiped out by badly done AI is at least at 20%. I agree. The assumption of risks from AI is by itself reasonable. But I am skeptical of making complex predictions that are based on that assumption. I am skpetical of calculating the expected utility of mitigating risks from AI according to the utility associated with its logical implications.

Take your following comment:

I’ll readily concede that my exact species extinction numbers were made up. But does it really matter? Two hundred million years from now, the children’s children’s children of humanity in their galaxy-civilizations, are unlikely to look back and say, “You know, in retrospect, it really would have been worth not colonizing the Herculus supercluster if only we could have saved 80% of species instead of 20%”. I don’t think they’ll spend much time fretting about it at all, really. It is really incredibly hard to make the consequentialist utilitarian case here, as opposed to the warm-fuzzies case.

I don't disagree that friendly AI research is currently a better option for charitable giving than charities concerned with environmental problems. Yet I have a hard time to accept that discounting the extinction of most species on the basis of the expected utility of colonizing the Herculus supercluster is sensible.

If you want to convince people like Holden Karnofsky and John Baez then you have to show that risks from AI are more likely than they believe and that contributing to SI can make a difference. If you just argue in terms of logical implications then they will continue to frame SI in terms of Pascal's mugging.

...you ought to be able to exhibit a particular premise or reasoning step that you disagree with.

I can't. I can only voice my discomfort. And according to your posts on the Lifespan Dilemma and Pascal's mugging you share that discomfort, yet you are also unable to pinpoint a certain step that you disagree with.

Michael Vassar views this as a fundamental divide that separates sanitykind from Muggles; with Tyler Cowen, for example, rejecting cryonics but not feeling obligated to reject any particular premise of Hanson's.

If there is an argument that relies on many premises I can reject the conclusion, i.e., assign it a low probability while accepting, i.e., assigning high probability to, each individual premise.

Perhaps we should call ourselves the Modusponenstsukai.

One man's modus ponens is another man's modus tollens.

Using quantum suicide as a reason to ignore logical implications is blatantly fallacious.

It's just an example. I'll now add to the post what I thought was already implicitly clear, if quantum suicide would make sense then I wouldn't do it. You know very well that mentioning the example that I really mean to talk about is forbidden around here.

I actually have a different idea on a possible approach to this Problem. I tried to get a discussion about it started in the decision theory mailing group, but it sort of died, and I'm not sure it's safe to post in public.

You choose some numbers to represent "max goodness" and "max badness"; really good and bad things approach these bounds asymptotically; and when you meet Pascal's mugger, you take "max badness", multiply it by an extremely tiny value, and get a very tiny value.

Even if you had solid evidence that your utility function was bounded, there would still be a small probability that your utility function was unbounded or bounded at a much higher level than you presumed. Pascal's mugger simply has to increase his threat to compensate for your confidence in low-level utility bounding.

If I develop an algorithm A1 for solving certain problems that is more reliable than my own intuition, the fact that A1 is not perfectly reliable is a great reason to try and develop a superior algorithm A2.

Exactly. What I am arguing is 1) that we should be risk averse about the actions suggested by algorithm A1, if they exceed a certain scope, and 2) that we should devote resources to 2.1) verifying the correctness of A1 by empirical evidence and or 2.2) trying to improve A1 or develop a superior algorithm A2. What we shouldn't do is to simply accept the actions that are recommended by A1 and follow through on them.

The trouble is that you can split a big event into P / epsilon chance-of-epsilon events, or average P / epsilon chance-of-epsilon events into a big event. In order to avoid inconsistency, you have to actually say what information you think should be treated as an average of nearby information.

A is absurd

No, not absurd. I was worried that we'll never get to the point where we actually "enjoy life" as human beings.

Don't trust reasoning. (Or, at least, don't trust the reasoning of people who disagree with me.)

No, that's not what I wanted to argue. I wrote in the post that we should continue to use our best methods. We should try to solve friendly AI. I said that we should be careful and discount some of the implied utility.

Take for example the use of Bayes' theorem. I am not saying that we shouldn't use it, that would be crazy. What I am saying is that we should be careful in the use of such methods.

If for example you use probability theory to update on informal arguments or anecdotal evidence you are still using your intuition to assign weight to to evidence. Using math and numeric probability estimates might make you unjustifiably confident of your results because you mistakenly believe that you don't rely on your intuition.

I am not saying that we shouldn't use math to refine our intuition, what I am saying is that we can still be wrong by many orders of magnitutes as long as we are using our heuristics in an informal setting rather than evaluating data supplied by experimentation.

Take this example. Julia Galef wrote:

But then I thought about Bayes’ rule and realized I was wrong — even a convincing-sounding “yes” gives you some new information. In this case, H = “He thinks I’m pretty” and E = “He gave a convincing-sounding ‘yes’ to my question.” And I think it’s safe to assume that it’s easier to sound convincing if you believe what you’re saying than if you don’t, which means that P(E | H) > P(E | not-H). So a proper Bayesian reasoner encountering E should increase her credence in H.

But by how much should a proper Bayesian reasoner increase her credence in H? Bayes' rule only tells us by how much given the input. But the variables are often filled in by our intuition.

unless you are very careful, the worlds where you escape death are not likely to be pleasant

I think some people don't understand that if quantum suicide works it wont prevent the gun from being fired: that doesn't kill you; it wont prevent the bullet from hitting your head: that doesn't kill you; it wont prevent the bullet from inflicting massive brain damage as long as certain information survives.

It wont prevent the rest of your body from dying, for that doesn't immediately kill you. At this point there are no further worlds of significant quantum measure. Who knows what happens subjectively?

You shouldn't commit quantum suicide because it decreases your measure...

Yes, I understand that quantum suicide would be stupid. I always use such examples to avoid mentioning the forbidding topic. To be clear, I started posting here regularly mainly due to the forbidding topic. I was simply shocked but also fascinated by how seriously it was taken. I concluded that those people either are in possession of vast amounts of evidence that I lack or that they are too confident of the methods they use to arrive at correct beliefs.

For example, building a machine that is sceptical of Pascal's wager doesn't seem harder than building a machine that is sceptical of other verbal offers unsupported by evidence.

The verbal offer isn't actually relevant to the problem, it's just there to dramatize the situation.

I don't see what's wrong with the idea that "extraordinary claims require extraordinary evidence".

Please formulate that maxim precisely enough to program into an AI in a way that solves the problem. Because the best way we currently have of formulating it, i.e., Bayseanism with quasi-Solomonoff priors doesn't solve it.

Verbal offers are evidence. Sometimes even compelling evidence. For example, I don't currently believe my friend Sam is roasting a turkey -- in fact, my prior probability of that is < 1%. But if Sam calls me up and says "Wanna come over for dinner? I'm roasting a turkey" my posterior probability becomes > 90%.

Designing a system well-calibrated enough that its probability estimates cause it to make optimal choices across a narrow band of likelihoods is a simpler problem than designing a system that works across a much wider band.

I don't see what's wrong with the idea that "extraordinary claims require extraordinary evidence".

Me neither, but quite a few people on lesswrong don't seem to share that opinion or are in possession of vast amounts of evidence that I lack. For example, some people seem to consider "interference from an alternative Everett branch in which a singularity went badly" or "unfriendly AI that might achieve complete control over our branch by means of acausal trade". Fascinating topics for sure, but in my opinion ridiculously far detached from reality to be taken at all seriously. Those ideas are merely logical implications of theories that we deem to reasonable. Another theory that is by itself reasonable is then used to argue that logical implications do not have to pay rent in future anticipations. And in the end, due to a combination of reasonable theories, one ends up with completely absurd ideas. I don't see how this could have happened if one would follow the rule that "extraordinary claims require extraordinary evidence".

Are you trying to give specific examples of cases in which doing the rational thing could be the wrong thing to do? Surely not, that would be oxymoronic - if you already know that the 'rational thing' is a mistake then it isn't the rational thing.

Rational acts can be wrong - if the agent doesn't already know that. This happens regularly in Bayesian hell.

What are you trying to do here?

I am trying to hint at the possibility that our methods might be mathematically justified but that they might lead to unexpected side-effects when applied by computationally bounded agents under extreme circumstances, as some of our thought experiments indicate.

...if you already know that the 'rational thing' is a mistake then it isn't the rational thing.

Our methods are the best we have and they work perfectly well on most problems we encounter. I am saying that we should discount some of the associated utility implications if we encounter edge cases. Ignoring the the implications would be irrational but taking them at face value wouldn't be wise either.

Sorry, what's wrong with Lob's theorem?

I have only skimmed over a very few comments so this might be very wrong:

As far as I can tell, if an agent assumes that its methods are consistent then it might update on its past decisions as evidence on how to act correctly on future occasions. But while that line of reasoning misses the obvious fact that computationally bounded agents are fallible, given what we know an agent has to work under the assumption that its decision procedures are consistent. This leads to contradictions.

An agent build according to our current understanding of decision theory can't be expected to self-modify to a correct decision theory (don't ask me why humans can do this).

Anyway, my point in mentioning those problems (because they were listed as problems here or elsewhere) was to show that if we were to turn ourselves into the agents we desire to build, according our current laws of thought, then in some cases we would be worse off than we are now and in other cases like Pascal's mugging we couldn't be sure if we were going to make correct decisions (we are not sure if an agent with an unbounded finite utility function over outcomes is consistent).

The model uncertainty involved is still profound and we shouldn't factor out human intuition at this point. Also, perceived absurdity does constitute evidence.

However, in the "donating to SIAI" case, I don't think we're actually talking about particularly large or small probabilities, or fallacious arguments. Implications can be labeled "extraordinary" for being socially unusual. This sort of extraordinary doesn't seem like it should be discounted.

This behavior isn't actually "socially unusual", in fact there are many social institutions that this resembles at least from an outside view, they're commonly called "cults". What this means is that humans seem to have a bias in favor of donating to "their cult" and believing they're acting rationally while doing so. As such you should consider whether you're belief that it's rational to donate to SIAI is affected by the same bias.

...because it seems reasonable to believe that Bayesianism can explain much of the success of science and thus can claim a lot of science's successes for itself.

True, but science was invented because people are really bad at judging evidence. I am troubled by the prospect of people using the core principles of science, e.g. Bayesianism, and applying them loosely and informally to vaguely understood conjectures and follow through on the implied actions.

Prediction, experimentation, peer review and the demand of empirical evidence is what makes science strong. If you think that you can use your rationality in a combination with Bayesianism and run with it then you confound your puny human brain with that of the hypothetical superintelligence that you dreamed up.

What I am arguing for is to be more conservative when it comes to the theoretically superior heuristics being discussed within this community.

Oops, I managed to misread that for some reason.