Posts
Comments
Psy-Kosh: I was using the example of pure baby eater values and conscious babies to illustrate the post Nick Tarleton linked to rather than apply it to this one.
Michael: if it's "inevitable" that they will encounter aliens then it's inevitable that each fragment will in turn encounter aliens, unless they do some ongoing pre-emptive fragmentation, no? But even then, if exponential growth is the norm among even some alien species (which one would expect) the universe should eventually become saturated with civilizations. In the long run, the only escape is opening every possible line from a chosen star and blowing up all the stars at the other ends of the lines.
Hmm. I guess that's an argument in favour of cooperating with the superhappies. Though I wonder if they would still want to adopt babyeater values if the babyeaters were cut off, and if the ship would be capable of doing that against babyeater resistance.
Nick, note that he treats the pebblesorters in parallel with the humans. The pebblesorters' values lead them to seek primeness and Eliezer optimistically supposes that human values lead humans to seek an analogous rightness.
What Eliezer is trying to say in that post, I think, is that he would not consider it right to eat babies even conditional on humanity being changed by the babyeaters to have their values.
But the choice to seek rightness instead of rightness' depends on humans having values that lead to rightness instead of rightness'.
It's evidence of my values which are evidence of typical human values. Also, I invite other people to really think if they are so different.
Eliezer tries to derive his morality from human values, rather than simply assuming that it is an objective morality, or asserting it as an arbitrary personal choice. It can therefore be undermined in principle by evidence of actual human values.
Also, I think I would prefer blowing up the nova instead. The babyeater's children's suffering is unfortunate no doubt but hey, I spend money on ice cream instead of saving starving children in Africa. The superhappies' degrading of their own, more important, civilization is another consideration.
(you may correctly protest about the ineffectiveness of aid - but would you really avoid ice cream to spend on aid, if it were effective and somehow they weren't saved already?)
If blowing up Huygens could be effective, why did it even occur to you to blow up Earth before you thought of this?
Sure it's a story, but one with an implicit idea of human terminal values and such.
I'm actually inclined to agree with Faré that they should count the desire to avoid a few relatively minor modifications over the eternal holocaust and suffering of baby-eater children.
I originally thought Eliezer was a utilitarian, but changed my mind due to his morality series.
(Though I still thought he was defending something that was fairly similar to utilitarianism. But he wasn't taking additivity as a given but attempting to derive it from human terminal values themselves - so if human terminal values don't say that we should apply equal additivity to baby-eater children, and I think they don't, then Eliezer's morality, I would have thought, would not apply additivity to them.)
This story however seems to show suspiciously utilitarian-like characteristics in his moral thinking. Or maybe he just has a different idea of human terminal values.
...with babyeater values.
Actually, I'm not sure if that's what I thought about their intentions towards the babyeaters, but I at least didn't originally expect them to still intend to modify themselves and humanity.
John Maxwell:
No, they are simply implementing the original plan by force.
When I originally read part 5, I jumped to the same conclusion you did, based presumably on my prior expectations of what a reasonable being would do. But then I read nyu2's comment which assumed the opposite and went back to look at what the text actually said, and it seemed to support that interpretation.
It seems we are at a disadvantage relative to Eliezer in thinking of alternative endings, since he has a background notion of what things are possible and what aren't, and we have to guess from the story.
Things like:
How quickly can you go from star to star?
Does the greater advancement of the superhappies translate into higher travel speed, or is this constrained by physics?
Can information be sent from star to star without couriering it with a ship, and arrive in a reasonable time?
How long will the lines connected to the novaing star remain open?
Can information be left in the system in a way that it would likely be found by a human ship coming later?
Is it likely that there are multiple stars that connect the nova to one, two or all three alderson networks?
And also about behaviour:
Will the superhappies have the system they use to connect with the nova under guard?
How long will it be before the babyeaters send in another ship? the humans, if no information is received?
How soon will the superhappies send in their ships to begin modifying the babyeaters?
Here's another option with different ways to implement it depending on the situation (possibly already mentioned by others, if so, sorry):
Cut off the superhappy connection, leaving or sending info for other humans to discover, so they deal with the babyeaters at their leisure.
Go back to give info to humans at Huygens, then cut off the superhappy connection.
Go back to get reinforcements, then quickly destroy the babyeater civilization (suicidally if necessary) and the novaing star (immediately after the fleet goes from it to the babyeater star(s), if necessary).
In all cases, I assume the superhappies will be able to guess what happened in retrospect. If not, send them an explicit message if possible.
... but relative to simply cooperating, it seems a clear win. Unless the superhappies have thought of it and planned a response.
Of course, the corollary for the real world would seem to be: those people who think that most people would not converge if "extrapolated" by Eliezer's CEV ought to exterminate other people who they disagree with on moral questions before the AI is strong enough to stop them, if Eliezer has not programmed the AI to do something to punish that sort of thing.
Hmm. That doesn't seem so intuitively nice. I wonder if it's just a quantitative difference between the scenarios (eg quantity of moral divergence), or a qualitative one (eg. the babykillers are bad enough to justifiably be killed in the first place).
If the humans know how to find the babyeaters' star,
and if the babyeater civilization can be destroyed by blowing up one star,
then I would like to suggest that they kill off the babyeaters.
Not for the sake of the babyeaters (I consider the proposed modifications to them better than annihilation from humanity's perspective)
but to prevent the super-happies from making even watered down modifications adding baby-eater values -
not so much to humans, since this can also be (at least temporarily) prevented by destroying Huygens -
but to themselves, as they are going to be the dominant life form in the universe over time, being the fastest growing and advancing species.
Of course, relative to destroying Huygens the price to pay in terms of modifications to human values is high, so I would not make this decision lightly.
James Andrix: I don't claim that the aliens would prefer modification over death, only that it is more consonant with my conception of human values to modify them than exterminate them, notwithstanding that the aliens may prefer the latter.
Akon claims this is a "true" prisoner's dilemma situation, and then tries to add more values to one side of the scale. If he adds enough values to make cooperation higher value than defecting, then he was wrong to say it was a true prisoner's dilemma. But the story has made it clear that the aliens appear to be not smart enough to accurately anticipate human behaviour (or vice versa for that matter), so this is not a situation where it is rational to cooperate in a true prisoner's dilemma. If it really is a true prisoner's dilemma, they should just defect.
Of course, there may be a more humane approach than extermination or requiring them to live under human law: forcible modification to remove the desire to eat babies, and reduce the amount of reproduction. It might be a little tricky to do this without completely messing up the aliens' psychology.
Also, it seems a little unlikely that a third ship would arrive given that the arrival of even one alien ship was considered so surprising in the first installment.
Strong enough to disrupt personal identity, if taken in one shot? That's a difficult question to answer, especially since I don't know what experiment to perform to test any hypotheses. On one hand, billions of neurons in my visual cortex undergo massive changes of activation every time my eyes squeeze shut when I sneeze - the raw number of flipped bits is not the key thing in personal identity. But we are already talking about serious changes of information, on the order of going to sleep, dreaming, forgetting your dreams, and waking up the next morning as though it were the next moment.
It sounds as if you believe in a soul (or equivalent) that is "different" for some set of possible changes and "the same" for other possible changes. I would suggest that that whether an entity at time n+1 is the same person as you at time n is not an objective fact of the universe. Humans have evolved so that we consider the mind that wakes up in the body of the mind that went to sleep to be the same person, but this intuitive sense is not an intuitive understanding of an objective reality; one could modify oneself to consider sleep to disrupt identity, and this would not be a "wrong" belief but just a different one.
I think most people are most comfortable retaining their evolution-given intuitions where they are strong, but where they are weak I think it is a mistake to try to overgeneralize them; instead one should try to shape them consciously. If you want to try being female for a while, why spoil your fun with hang ups about identity? Just decide that it's still you.
Eliezer, whenever you start thinking about people who are completely causally unconnected with us as morally relevant, alarm bells should go off.
What's worse though, is that if your opinion on this is driven by a desire to justify not agreeing with the "repugnant conclusion", it may signify problems with your morality that could annihilate humanity if you give your morality to an AI. The repugnant conclusion requires valuing the bringing into existence of hypothetical people with total utility x by as much as reducing the utility of existing people by x, or annihilating people with utility x. Give that morality to a fast takeoff AI and they'll quickly replace all humans with entities with greater capacity for utility. If the AI is programmed to believe the problem with the "repugnant conclusion" is what you claim, the AI will instead create randomized (for high uniqueness) minds with high capacity for utility, still annihilating humans.
I think what he means by "calibrated" is something like it not being possible for someone else to systematically improve the probabilities you give for the possible answers to a question just from knowing what values you've assigned (and your biases), without looking at what the question is.
I suppose the improvement would indeed be measured in terms of relative entropy of the "correct" guess with respect to the guess given.
Responding to Gaffa (I kind of intended to respond right after the comment, bot got sidetracked):
When approaching a scientific or mathematical problem, I often find myself trying hard to avoid having to calculate and reason, and instead try to reach for an "intuitive" understanding in the back of my mind, but that understanding, if I can even find it, is rarely sufficient when dealing with actual problems.
I would advise you to embrace calculation and reason, but just make sure you think about what you are doing and why. Use the tools, but try to get an intuitive understanding of why and how they work, both in general and each time you apply them. It is true that formualaic rules can serve as a crutch to avoid the need for understanding, but if you throw away calculation and reason, you are not likely to make much progress.
Finally, be realistic in your expectations: for a complicated problem, you should not expect to be able to get an intuitive understanding of the solution as a single step, but you can aim for a chain of individually intuitive steps and, if the chain is sufficiently short, an overall intuitive understanding of how the steps relate to one another.
It might make an awesome movie, but if it were expected behaviour, it would defeat the point of the injunction. In fact if rationalists were expected to look for workarounds of any kind it would defeat the point of the injunction. So the injunction would have to be, not merely to be silent, but not to attempt to use the knowledge divulged to thwart the one making the confession in any way except by non-coercive persuasion.
Or alternatively, not to ever act in a way such that if the person making the confession had expected it they would have avoided making the confession.
To the extent that a commitment to ethics is externally verifiable, it would encourage other people to cooperate, just as a tendency to anger (a visible commitment to retribution) is a disincentive to doing harm.
Also, even if it is not verifiable, a person who at least announces their intention to hold to an ethical standard has raised the impact their failure to do so will have on their reputation, and thus the announcement itself should have some impact on the expectation that they will behave ethically.
Just for the sake of devil's advocacy:
4) You want to attribute good things to your ethics, and thus find a way to interpret events that enables you to do so.
Miguel: it doesn't seem to be a reference to something, but just a word for some experience an alien might have had that is incomprehensible to us humans, analogous to humour for the alien.
Psy-Kosh, my argument that Boltzmann brains go poof is a theoretical argument, not an anthropic one. Also, if we want to maximize our correct beliefs in the long run, we should commit to ignore the possibility that we are a brain with beliefs not causally affected by the decision to make that commitment (such as a brain that randomly pops into existence and goes poof). This also is not an anthropic argument.
With regard to longer-lived brains, if you expect there to be enough of them that even the ones with your experience are more common than minds in a real civilization with your experience, then you really should rationally expect to be one (although as a practical matter since there's nothing much a Boltzmann brain can reasonably expect to do one might as well ignore it*). If you expect there to be more long lived Boltzmann brains than civilization-based minds in general, but not enough for ones with your experience to outnumber civilization-based minds with your experience, then your experience tips the balance in favour of believing you are not a Boltzmann brain after all.
I think your confusion is the result of you not being consistent about whether you accept self-indication, or maybe you being inconsistent about whether you think of the possible space with Boltzmann brains and no civilizations as being additional to or a substitute for space with civilizations. Here's what different choices of those assumptions imply:
(I assume throughout that that the probability of Boltzmann brains per volume in any space is always lower than the probability of minds in civilizations where they are allowed by physics)*
Assumptions -> conclusion
self-indication, additional -> our experience is not evidence** for or against the existence of the additional space (or evidence for its existence if we consider the possibility that we may be unusually order-observing entities in that space)
self-indication, substitute -> our experience is evidence against the existence of the substitute space
instead of self-indication, assume the probability of being a given observer is inversely proportional to number of observers in possible universe containing that observer (this is the most popular alternative to self-indication) -> our experience is evidence against the existence of the additional or substitute space
*unless the Boltzmann brain, at further exponentially reduced probability, also obtained effective means of manipulating its environment...
** basically, define "allowed" to mean (density of minds with our experience in civ) >> (density of Boltzmann brains with our experience), and not allowed to mean the opposite (<<). One would expect the probability of a space with comparable densities to be low enough not to have a significant quantitative or qualitative affect on the conclusions.
*It seems rather unlikely that a space with our current apparent physical laws allows more long-lived B-brains than civilization-based brains. I am too tired to want to think about and write out what would follow if this is not true.
**I am using "evidence" here to mean shifts of probability relative to the outside view prior (conditional on the existence of any observers at all), which means that any experience is evidence for a larger universe (other things being equal) given self-indication, etc.
Nick, do you use the normal definition of a Boltzmann brain?
It's supposed to be a mind which comes into existence by sheer random chance. Additional complexity - such as would be required for some support structure (e.g. an actual brain), or additional thinking without a support structure - comes with an exponential probability penalty. As such, a Boltzmann brain would normally be very short lived.
In principle, though, there could be so much space uninhabitable for regular civilizations that even long-lived Boltzmann brains which coincidentally have experiences similar to minds in civilizations outnumber minds in civilizations.
It's not clear whether you are worrying about whether you already are a Boltzmann brain, or if you think you are not one but think that if a Boltzmann brain took on your personality it would be 'you'. If the former, I can only suggest that nothing you do as a Boltzmann brain is likely to have much effect on what happens to you, or on anything else. If the latter, I think you should upgrade your notion of personal identity. While the notion that personality is the essence of identity is a step above the notion that physical continuity is the essence of identity, by granting the notion that there is an essence of identity at all it reifies the concept in a way it doesn't deserve, a sort of pseudosoul for people who don't think they believe in souls.
Ultimately what you choose to think of as your 'self' is up to you, but personally I find it a bit pointless to be concerned about things that have no causal connection with me whatsoever as if they were me, no matter how closely they may coincidentally happen to resemble me.
Let's suppose, purely for the sake of argument of course, that the scientists are superrational.
The first scientist chose the most probable theory given the 10 experiments. If the predictions are 100% certain then it will still be the most probable after 10 more successful experiments. So, since the second scientist chose a different theory, there is uncertainty and the other theory assigned an even higher probability to these outcomes.
In reality people are bad at assessing priors (hindsight bias), leading to overfitting. But these scientists are assumed to have assessed the priors correctly, and given this assumption you should believe the second explanation.
Of course, given more realistic scientists, overfitting may be likely.
It may be that most minds with your thoughts do in fact disappear after an instant. Of course if that is the case there will be vastly more with chaotic or jumbled thoughts. But the fact that we observe order is no evidence against the existence of additional minds observing chaos, unless you don't accept self-indication.
So, your experience of order is not good evidence for your belief that more of you are non-Boltzmann than Boltzmann. But as I said, in the long term your expected accuracy will rise if you commit to not believing you are a Boltzmann brain, even if you believe that you most likely are one now.
A somewhat analogous situation may arise in AGI - AI makers can rule out certain things (e.g. the AI is simulated in a way that the simulated makers are non-conscious) that the AI cannot. Thus by having the AI rule such things out a priori, the makers can improve the AI's beliefs in ways that the AI itself, however superintelligent, rationally could not.
Nick and Psy-Kosh: here's a thought on Boltzmann brains.
Let's suppose the universe has vast spaces uninhabited by anything except Boltzmann brains which briefly form and then disappear, and that any given state of mind has vastly more instantiations in the Boltzmann-brain only spaces than in regular civilizations such as ours.
Does it then follow that one should believe one is a Boltzmann brain? In the short run perhaps, but in the long run you'd be more accurate if you simply committed to not believing it. After all, if you are a Boltzmann brain, that commitment will cease to be relevant soon enough as you disintegrate, but if you are not, the commitment will guide you well for a potentially long time.
And by elementary I mean the 8 different ways W, F, and the comet hit/non hit can turn out.
Err... I actually did the math a silly way, by writing out a table of elementary outcomes... not that that's silly itself, but it's silly to get input from the table to apply to Bayes' theorem instead of just reading off the answer. Not that it's incorrect of course.
Richard, obviously if F does not imply S due to other dangers, then one must use method 2:
P(W|F,S) = P(F|W,S)P(W|S)/P(F|S)
Let's do the math.
A comet is going to annihilate us with a probability of (1-x) (outside view) if the LHC would not destroy the Earth, but if the LHC would destroy the Earth, the probability is (1-y) (I put this change in so that it would actually have an effect on the final probability)
The LHC has an outside-view probability of failure of z, whether or not W is true
The universe has a prior probabilty w of being such that the LHC if it does not fail will annihilate us.
Then:
P(F|W,S) = 1
P(F|S) = (ywz+x(1-w)z)/(ywz+x(1-w)z+x(1-w)(1-z))
P(W|S) = (ywz)/(ywz+x(1-w)+x(1-w)(1-z))
so, P(W|F,S) = ywz/(ywz+x(1-w)z) = yw(yw+x(1-w))
I leave it as an exercise to the reader to show that there is no change in P(W|F,S) if the chance of the comet hitting depends on whether or not the LHC fails (only the relative probability of outcomes given failure matters).
Really though Richard, you should not have assumed in the first place that I was not capable of doing the math. In the future, don't expect me to bother with a demonstration.
Allan: you're right, I should have thought that through more carefully. It doesn't make your interpretation correct though...
I have really already spent much more time here today than I should have...
You have another inconsistency as well. As you should have noticed in the "How many" thread, the assumptions that lead you to believe that failures of the LHC are evidence that it would destroy Earth are the same ones that lead you to believe that annihilational threats are irrelevant (after all, if P(W|S) = P(W), then Bayes' rule leads to P(S|W) = P(S)).
Thus, given that you believe that failures are evidence of the LHC being dangerous, you shouldn't care. Unless you've changed to a new set of incorrect assumptions, of course.
I might add, for the benefit of others, that self-sampling forbids playing favourites among which observers to believe that you are in a single universe (beyond what is actually justified by the evidence available), and self-indication forbids the same across possible universes.
Nominull: It's a bad habit of some people to say that reality depends on, or is relative to observers in some way. But even though observers are not a special part of reality, we are observers and the data about the universe that we have is the experience of observers, not an outside view of the universe. So long as each universe has no more than one observer with your experience, you can take your experience as objective evidence that you live in a universe with one such observer instead of zero (and with this evidence to work with, you don't need to talk about observers). But it's difficult to avoid talking about observers when a universe might have multiple observers with the same subjective experience.
Why do you reject self-indication? As far as I can recall the only argument Bostrom gave against it was that he found it unintuitive that universes with many observers should be more likely, with absolutely no justification as to why one would expect that intuition to reflect reality. That's a very poor argument considering the severe problems you get without it.
I suppose you might be worried about universes with many unmangled worlds being made more likely, but I don't see what makes that bullet so hard to bite either.
Whoops, I didn't notice that you did specifically claim that P(W|S)=P(W).
Do you arrive at this incorrect claim via Bostrom's approach, or another one?
Not particularly. I use 4 but with P(W|S) = P(W) which renders it valid. (We're not talking about two side-by-side universes, but about prior probabilities on physical law plus a presumption of survival.)
You mean you use method 2. Except you don't, or you would come to the same conclusion that I do. Are you claiming that P(W|S)= P(W)? Ok, I suspect you may be applying Nick Bostrom's version of observer selection: hold the probability of each possible version of the universe fixed independent of the number of observers, then divide that probability equally amongst the observers. Well, that approach is BS whenever the number of observers differs between possible universes, since if you imagine aliens in the universe but causally separate, the probabilities would depend on their existence.
Also, does it really make sense to you, intuitively, that you should get a different result given two actually existing universes compared to two possible universes?
This could only reflect uncertainty that anthropic reasoning was valid. If you were certain anthropic reasoning were valid (I'm sure not!) then you would make no such update. In practice, after surviving a few hundred rounds of quantum suicide, would further survivals really seem to call for alternative explanations?
As I pointed out earlier, if there was even a tiny chance of the machine being broken in such a way as to appear to be working, that probability would dominate sooner or later.
One last thing: if you really believe that annihilational events are irrelevant, please do not produce any GAIs until you come to your senses.
Eliezer, I used "=>" (intending logical implication), not ">=".
I would suggest you read my post above on this second page, and see if that changes your mind.
Also, in a previous post in this thread I argued that one should be surprised by externally improbable survival, at least in the sense that it should make one increase the probability assigned to alternative explanations of the world that do not make survival so unlikely.
Sorry Richard, well of course they aren't necessarily independent. I wasn't quite sure what you were criticising. But I pointed out already that, for example, a new physical law might in principle both cause the LHC to fail and cause it to destroy the world if it did not fail. But I pointed out that this was not what people were arguing, and assuming that such a relation is not the case then the failure of the LHC provides no information about the chance that a success would destroy the world. (And a small relation would lead to a small amount of information, etc.)
While I'm happy to have had the confidence of Richard, I thought my last comment could use a little improvement.
What we want to know is P(W|F,S)
As I pointed out F=> S so P(W|F,S) = P(W|F)
We can legitimately calculate P(W|F,S) in at least two ways:
1. P(W|F,S) = P(W|F) = P(F|W)P(W)/P(F) <- the easy way
2. P(W|F,S) = P(F|W,S)P(W|s)/P(F|S) <- harder, but still works
there are also ways you can get it wrong, such as:
3. P(W|F,S) != P(F|W,S)P(W)/P(F) <- what I said other people were doing last post
4. P(W|F,S) != P(F|W,S)P(W)/P(F|S) <- what other people are probably actually doing
In my first comment in this thread, I said it was a simple application of Bayes' rule (method 1) but then said that Eliezer's failure was not to apply the anthropic principle enough (ie I told him to update from method 4 to method 2). Sorry if anyone was confused by that or by subsequent posts where I did not make that clear.
Allan: your intuition is wrong here too. Notice that if Zeus were to have independently created a zillion people in a green room, it would change your estimate of the probability, despite being completely unrelated.
Eliezer: F => S -!-> P(X|F) = P(X|F,S)
All right, give me an example.
And yeah, anthropic reasoning is all about conditioning on survival, but you have to do it consistently. Conditioning on survival in some terms but not others = fail.
Richard: your first criticism has too low an effect on the probability to be significant. I was of course aware that humanity could be wiped out in other ways but incorrectly assumed that commenters here would be smart enough to understand that it was a justifiable simplification. The second is wrong: the probabilities without conditioning on S are "God's eye view" probabilities, and really are independent of selection effects.
I'm going to try another explanation that I hope isn't too redundant with Benja's.
Consider the events
W = The LHC would destroy Earth F = the LHC fails to operate S = we survive (= F OR not W)
We want to know P(W|F) or P(W|F,S), so let's apply Bayes.
First thing to note is that since F => S, we have P(W|F) = P(W|F,S), so we can just work out P(W|F)
Bayes:
P(W|F) = P(F|W)P(W)/P(F)
Note that none of these probabilities are conditional on survival. So unless in the absence of any selection effects the probability of failure still depends on whether the LHC would destroy Earth, P(F|W) = P(F), and thus P(W|F) = P(W).
(I suppose one could argue that a failure could be caused by a new law of physics that would also lead the LHC to destroy the Earth, but that isn't what is being argued here - at least so I think; my apologies to anyone who is arguing that)
In effect what Eliezer and many commenters are doing is substituting P(F|W,S) for P(F|W). These probabilities are not the same and so this substitution is illegitimate.
Benja, I also think of it that way intuitively. I would like to add though that it doesn't really matter whether you have branches or just a single nondeterministic world - Bayes' theorem applies the same either way.
Robinson, I could try to nitpick all the things wrong with your post, but it's probably better to try to guess at what is leading your intuition (and the intuition of others) astray.
Here's what I think you think:
- Either the laws of physics are such that the LHC would destroy the world, or not.
- Given our survival, it is guaranteed that the LHC failed if the universe is such that it would destroy the world, whereas if the universe is not like that, failure of the LHC is not any more likely than one would expect normally.
- Thus, failure of the LHC is evidence for the laws of physics being such that the LHC would destroy the world.
This line of argument fails because when you condition on survival, you need to take into account the different probabilities of survival given the different possibilities for the laws of the universe. As an analogy, imagine a quantum suicide apparatus. The apparatus has a 1/2 chance of killing you each time you run it and you run it 1000 times. But, while the apparatus is very reliable, it has a one in a googol chance of being broken in such a way that every time it will be guaranteed not to kill you, but appear to have operated successfully and by chance not killed you. Then, if you survive running it 1000 times, the chance of it being broken in that way is over a googol squared times more likely than the chance of it having operated successfully.
Here's what that means for improving intuition: one should feel surprised at surviving a quantum suicide experiment, instead of thinking "well, of course I would experience survival".
Finally a note about the anthropic principle: it is simply the application of normal probability theory to situtations where there are observer selection effects, not a special separate rule.
Allan, I am of course aware of that (actually, it would probably take time, but even if the annihilation were instantaneous the argument would not be affected).
There are 4 possibilities:
- The LHC would destroy Earth, but it fails to operate
- The LHC destroys Earth
- The LHC would not destroy Earth, but it fails anyway
- The LHC works and does not destroy Earth
The fact that conditional on survival possibility 2 must not have happened has no effect on the relative probabilities of possibility 1 and possibility 3.
To clarify, I mean failures should not lead to a change of probability away from the prior probability; of course they do result in a different probability estimate than if the LHC succeeded and we survived.
Actually, failures of the LHC should never have any effect at all on our estimate of the probability that if it did not fail it would destroy Earth.
This is because the ex ante probability of failure of the LHC is independent of whether or not if it turned on it would destroy Earth. A simple application of Bayes' rule.
Now, the reason you come to a wrong conclusion is not because you wrongly applied the anthropic principle, but because you failed to apply it (or applied it selectively). You realized that the probability of failure given survival is higher under the hypothesis that the LHC would destroy the Earth if it did not fail, but you didn't take into account the fact that the probability of survival is itself lower under that hypothesis (i.e. the anthropic principle).
As previously mentioned, there are tricky aspects to this. You can't say: "You see those humans over there? Whatever desire is represented in their brains, is therefore right." This, from a moral perspective, is wrong - wanting something doesn't make it right - and the conjugate failure of the AI is that it will reprogram your brains to want things that are easily obtained in great quantity. If the humans are PA, then we want the AI to be PA+1, not Self-PA... metaphorically speaking.
Before reading this post, if I had been programming a friendly AI I would have attempted to solve this issue by programming the AI it to take into account only minds existing at the moment it makes its decisions. (the AI still cares about the future, but only to the extent that currently existing minds, if extrapolated, would care about the future). This technique has the flaw that it would be likely to fail in the event that time travel is easy (the AI invents it before it reprograms itself to eliminate the bug). But I think this would be easier to get right, and what is the chance of time travel being easy compared to the chance of getting the "right" solution wrong?
It's not clear to me where you are going with it.
To argue that a proof is being made concluding ?C using the assumption ?(◻C -> C) given the theory PA, to which proof we can apply the deduction theorem to get (PA |- "?(◻C -> C) -> ?C") (i.e. my interpretation of Löb's Theorem)
We use 10 steps, 9 of which are proofs inside of PA
But the proof uses an additional assumption which is the antecedent of an implication, and comes to a conclusion which is the consequent of the implication. To get the implication, we must use the deduction theorem or something like it, right?
the fact that if PA |- X then PA |- "PA |- X"
Is this fact a theorem of first order logic without any additional assumptions, or is it merely a theorem of PA? I admit I don't know, as I'm not very familiar with first order logic, but it intuitively seems to me that if first order logic were powerful enough on its own to express concepts like "PA proves X" it would probably be powerful enough to express arithmetic, in which case the qualification in Gödel's theorem that it only applies to theories that express arithmetic would be superfluous.
Hmm. I was thinking that Löb's Theorem was a theorem in PA, in which case the step going from
PA + ?(?C -> C) |- ?(?L -> C)
to
PA + ?(?C -> C) |- ?(?(?L -> C))
seems legitimate given
PA |- (?X -> ?(?X))
which we ought to be able to use since PA is part of the theory before the |- symbol.
If we don't have PA on the left, can we use all the "ingredients" without adding additional assumptions?
In any case, if we do not use the deduction theorem to derive the implication in Löb's Theorem, what do we use?
We don't have PA + X proving anything for any X.
It seems to me that we do have (PA + "?(◻C -> C)" |- "?C")
from which the deduction theorem gives: (PA |- "?(◻C -> C) -> ?C") which is Löb's Theorem itself.
The hypothesis was that PA proves that "if PA proves C, then C" This enabled it to be proved that "PA proves C" So I think what we actually get applying the deduction theorem is ?((◻C)->C)->?C
I don't think you're talking about my sort of view* when you say "morality-as-preference", but:
Why do people seem to mean different things by "I want the pie" and "It is right that I should get the pie"? Why are the two propositions argued in different ways?
A commitment to drive a hard bargain makes it more costly for other people to try to get you to agree to something else. Obviously an even division is a Schelling point as well (which makes a commitment to it more credible than a commitment to an arbitrary division).
When and why do people change their terminal values? Do the concepts of "moral error" and "moral progress" have referents? Why would anyone want to change what they want?
I think humans tend not to have very clean divisions between instrumental and terminal values. Although there is no absolute moral progress or error, some moralities may be better or worse than others by almost any moral standard a human would be likely to use. Through moral hypocrisy, humans can signal loyalty to group values while disobeying them. Since humans don't self modify easily, a genuine desire to want to change may be a cost-effective way to improve the effectiveness of this strategy.
Why and how does anyone ever "do something they know they shouldn't", or "want something they know is wrong"? Does the notion of morality-as-preference really add up to moral normality?
See above on signaling and hypocrisy.
*moral nihilist with instrumental view of morality as tool for coordinating behaviour.
"From a utilitarian perspective", where does the desire to do things better than can be done with the continued existence of humans come from? If it comes from humans, should not the desire to continue to exist also be given weight?
Also, if AI researchers anchor their expectations for AI on the characteristics of the average human then we could be in big trouble.