Open Thread June 2010, Part 2

This is great, even more so as you made it open source. I added it to References & Resources for LessWrong.

You should make a short top-level post about this so more people see this

I'd vote you up again for handing out your source code as well as the quote list, but I can't, so an encouraging reply will have to do...

Not having to side scroll would be spiffy.

It might make more sense to put this on the Wiki. Two notes: First, some of the quotes have remarks contained in the posts which you have not edited out. I don't know if you intend to keep those. Second, some of the quotes are comments from quote threads that aren't actually quotes. 14 SilasBarta is one example. (And is just me or does that citation form read like a citation from a religious text ?)

Very cool idea.

It would be nice if links were preserved.

It is a good blog, and it has a slightly wider topic spread than LW, so even if you're familiar with most of the standard failures of judgment there'll be a few new things worth reading. (I found the "introducing fines can actually increase a behavior" post particularly good, as I wasn't aware of that effect.)

Thanks, this looks like an excellent supplement for LW.

For example, eliminating lies doesn't "make the world a better place" unless it actually makes people happier; claiming so is just concealed deontologism.

Just picking nits. Consequentialism =/= maximizing happiness. (The latter is a case of the former). So one could be a consequentialist and place a high value and not lying. In fact, the answers to all of your questions depend on the values one holds.

Or what Nesov said below.

For example, eliminating lies doesn't "make the world a better place" unless it actually makes people happier; claiming so is just concealed deontologism.

I disagree. Not lying or not being lied to might well be a terminal value, why not? You that lies or doesn't lie is part of the world. A person may dislike being lied to, value the world where such lying occurs less, irrespective of whether they know of said lying. (Correspondingly, the world becomes a better place even if you eliminate some lying without anyone knowing about that, so nobody becomes happier in the sense of actually experiencing different emotions, assuming nothing else that matters changes as well.)

Of course, if you can only eliminate a specific case of lying by on the net making the outcome even worse for other reasons, it shouldn't be done (and some of your examples may qualify for that).

It is a common failure of moral analysis (invented by deontologists undoubtedly) that they assume idealized moral situation. Proper consequentialism deals with the real world, not this fantasy.

• #1/#2/#3 - "never knows" fails far too often, so you need to include a very large chance of failure in your analysis.
• #4 - it's pretty safe to make stuff like that up
• #5 - in the past, undoubtedly yes; in the future this will be nearly certain to leak with everyone undergoing routine genetic testing for medical purposes, so no. (future is relevant because situation will last decades)
• #6 - consequentialism assumes probabilistic analysis (% that child is not yours, % chance that husband is making stuff up) - and you weight costs and benefits of different situations proportionally to their likelihood. Here they are in unlikely situation that consequentialism doesn't weight highly. They might be better off with some other value system, but only at cost of being worse off in more likely situations.

Is it okay to cheat on your spouse as long as (s)he never knows?

Is this actually possible? Imagine that 10% of people cheat on their spouses when faced with a situation 'similar' to yours. Then the spouses can 'put themselves in your place' and think "Gee, there's about a 10% chance that I'd now be cheating on myself. I wonder if this means my husband/wife is cheating on me?"

So if you are inclined to cheat then spouses are inclined to be suspicious. Even if the suspicion doesn't correlate with the cheating, the net effect is to drive utility down.

I think similar reasoning can be applied to the other cases.

(Of course, this is a very "UDT-style" way of thinking -- but then UDT does remind me of Kant's categorical imperative, and of course Kant is the arch-deontologist.)

As an old quote from DanielLC says, consequentialism is "the belief that doing the right thing makes the world a better place". I now present some finger exercises on the topic:

It seems like it would be more aptly defined as "the belief that making the world a better place constitutes doing the right thing". Non-consequentialists can certainly believe that doing the right thing makes the world a better place, especially if they don't care whether it does.

A quick Internet search turns up very little causal data on the relationship between cheating and happiness, so for purposes of this analysis I will employ the following assumptions:

a. Successful secret cheating has a small eudaemonic benefit for the cheater.
b. Successful secret lying in a relationship has a small eudaemonic cost for the liar.
c. Marital and familial relationships have a moderate eudaemonic benefits for both parties.
d. Undermining revelations in a relationship have a moderate (specifically, severe in intensity but transient in duration) eudaemonic cost for all parties involved.
e. Relationships transmit a fraction of eudaemonic effects between partners.

Under these assumptions, the naive consequentialist solution* is as follows:

1. Cheating is a risky activity, and should be avoided if eudaemonic supplies are short.
2. This answer depends on precise relationships between eudaemonic values that are not well established at this time.
3. Given the conditions, lying seems appropriate.
4. Yes.
5. Yes.
6. The husband may be better off. The wife more likely would not be. The child would certainly not be.

Are there any evident flaws in my analysis on the level it was performed?

* The naive consequentialist solution only accounts for direct effects of the actions of a single individual in a single situation, rather than the general effects of widespread adoption of a strategy in many situations - like other spherical cows, this causes a lot of problematic answers, like two-boxing.

It's okay to deceive people if they're not actually harmed and you're sure they'll never find out. In practice, it's often too risky.

1-3: This is all okay, but nevertheless, I wouldn't do these things. The reason is that for me, a necessary ingredient for being happily married is an alief that my spouse is honest with me. It would be impossible for me to maintain this alief if I lied.

4-5: The child's welfare is more important than my happiness, so even I would lie if it was likely to benefit the child.

6: Let's assume the least convenient possible world, where everyone is better off if they tell the truth. Then in this particular case, they would be better off as deontologists. But they have no way of knowing this. This is not problematic for consequentialism any more than a version of the Trolley Problem in which the fat man is secretly a skinny man in disguise and pushing him will lead to more people dying.

That is certainly interesting enough to subject to further reflection. Do we have any evolutionary psychologists in the audience?

Abstract preferences for or against the existence of enforcement mechanisms that could create binding cooperative agreements between previously autonomous agents have very very few detailed entailments.

These abstractions leave the nature of the mechanisms, the conditions of their legitimate deployment, and the contract they will be used to enforce almost completely open to interpretation. The additional details can themselves be spelled out later, in ways that maintain symmetry among different parties to a negotiation, which is a strong attractor in the semantic space of moral arguments.

This makes agreement with "the abstract idea of punishment" into the sort of concession that might be made at the very beginning of a negotiating process with an arbitrary agent you have a stake in influencing (and who has a stake in influencing you) upon which to build later agreements.

The entailments of "eating children" are very very specific for humans, with implications in biology, aging, mortality, specific life cycles, and very distinct life processes (like fuel acquisition versus replication). Given the human genome, human reproductive strategies, and all extant human cultures, there is no obvious basis for thinking this terminology is superior until and unless contact is made with radically non-human agents who are nonetheless "intelligent" and who prefer this terminology and can argue for it by reference to their own internal mechanisms and/or habits of planning, negotiation, and action.

Are you proposing to be such an agent? If so, can you explain how this terminology suits your internal mechanisms and habits of planning, negotiation, and action? Alternatively, can you propose a different terminology for talking about planning, negotiation, and action that suits your own life cycle?

For example, if one instance of Clippy software running on one CPU learns something of grave importance to its systems for choosing between alternative courses of action, how does it communicate this to other instances running basically the same software? Is this inter-process communication trusted, or are verification steps included in case one process has been "illegitimately modified" or not? Assuming verification steps take place, do communications with humans via text channels like this website feed through the same filters, analogous filters, or are they entirely distinct?

More directly, can you give us an IP address, port number, and any necessary "credentials" for interacting with an instance of you in the same manner that your instances communicate over TCP/IP networks with each other? If you aren't currently willing to provide such information, are there preconditions you could propose before you would do so?

Except for the bizarreness of eating most of your children, I suspect that most humans would find the two positions equally hypocritical. Why do you think we see them as different?

Adults, by choosing to live in a society that punishes non-cooperators, implicitly accept a social contract that allows them to be punished similarly. While they would prefer not to be punished, most societies don't offer asymmetrical terms, or impose difficult requirements such as elections, on people who want those asymmetrical terms.

Children, on the other hand, have not yet had the opportunity to choose the society that gives them the best social contract terms, and wouldn't have sufficient intelligence to do so anyways. So instead, we model them as though they would accept any social contract that's at least as good as some threshold (goodness determined retrospectively by adults imagining what they would have preferred). Thus, adults are forced by society to give implied consent to being punished if they are non-cooperative, but children don't give consent to be eaten.

Different topic spheres. One line sounds nicely abstract, while the other is just iffy.

Also killing people is different from betraying them. (Nice read: the real life section of tvtropes/moraleventhorizon)

With 1), you're non-cooperator and the punisher is society in general. With 2), you play both roles at different times.

One possible answer: Humans are selfish hypocrites. We try to pretend to have general moral rules because it is in our best interest to do so. We've even evolved to convince ourselves that we actually care about morality and not self-interest. That's likely occurred because it is easier to make a claim one believes in than lie outright, so humans that are convinced that they really care about morality will do a better job acting like they do.

(This was listed by someone as one of the absolute deniables on the thread a while back about weird things an AI might tell people).

I think that if this post is left as it is this post would be to trivial to be a top level post. You could reframe it as a beginners' guide to Occam, or you could make it more interesting by going deeper into some of the issues (if you can think of anything more to say on the topic of differentiating between hypotheses that make the same predictions, that might be interesting, although I think you might have said all there is to say)

I believe this is the conversation you're responding to.

(upvoted)

This seems worthy of a top-post. When you make it a top level post link to the relevant prior posts about complexity of hypotheses.

And yet, the quantum mechanical world behaves exactly this way. Observations DO change exactly what happens. So, apparently at the quantum mechanical level, nature does have some way of knowing.

I'm not sure what effect that this has upon your argument, but it's something that I think that you're missing.

But throughout my whole life up to that point (the question even appeared in the animated series Beetlejuice that I saw when I was little), I had assumed a different question was being asked: specifically,

If a tree falls, and no human (or human-entangled[1] sensor) is around to hear it, does it still make vibrations? On what basis do you believe this, lacking a way to directly check? (Call this "Question S".)

Me too! It was actually explained that way to me by my parents as a kid, in fact. I wonder if there are two subtly different versions floating around or EY just interpreted it uncharitably.

Seconding kodos96. As this would exonerate not only Knox and Sollecito but Guede as well, it has to be treated with considerable skepticism, to say the least.

More significant, it seems to me (though still rather weak evidence), is the Alessi testimony, about which I actually considered posting on the March open thread.

Still, the Aviello story is enough of a surprise to marginally lower my probability of Guede's guilt. My current probabilities of guilt are:

Knox: < 0.1 % (i.e. not a chance)

Sollecito: < 0.1 % (likewise)

Guede: 95-99% (perhaps just low enough to insist on a debunking of the Aviello testimony before convicting)

It's probably about time I officially announced that my revision of my initial estimates for Knox and Sollecito was a mistake, an example of the sin of underconfidence.

I of course remain willing to participate in a debate with Rolf Nelson on this subject.

Finally, I'd like to note that the last couple of months have seen the creation of a wonderful new site devoted to the case, Injustice in Perugia, which anyone interested should definitely check out. Had it been around in December, I doubt that I could have made my survey seem like a fair fight between the two sides.

That story would be consistent with Guédé's, modulo the usual eyewitness confusion.

The traditional answer is to follow the Kelly criterion, is it not? That would imply

where n is the number of tickets. This implies you should buy n such that (€1)*n = Wf*, where W is your initial wealth.

Edit: Thanks, JoshuaZ, for pointing out that the Kelly criterion might not be the applicable one in a given situation.

I just realised that infinite processing power creates a weird moral dilema:

Suppose you take this machine and put in a program which simulates every possible program it could ever run. Of course it only takes a second to run the whole program. In that second, you created every possible world that could ever exist, every possible version of yourself. This includes versions that are being tortured, abused, and put through horrible unethical situations. You have created an infinite number of holocausts and genocides and things much, much worse then what you could ever immagine. Most people would consider a program like this unethical to run. But what if the computer wasn't really a computer, it was an infinitely large database that contained every possible input and a corresponding output. When you put the program in, it just finds the right output and gives it to you, which is essentially a copy of the database itself. Since there isn't actually any computational process here, there is no unethical things being simulated. Its no more evil than a book in the library about genocide. And this does apply to the real world. It's essentially the chineese room problem - does a simulated brain "understand" anything? Does it have "rights"? Does how the information was processed make a difference? I would like to know what people at LW think about this.

That's well done although two of the central premises are likely incorrect. First, the notion that a quantum computer would have infinite processing capability is incorrect. Quantum computation allows speed-ups of certain computational processes. Thus for example, Shor's algorithm allows us to factor integers quickly. But if our understanding of the laws of quantum mechanics is at all correct, this can't lead to anything like that in the story. In particular, under the standard descriptor for quantum computing, the class of problems reliably solvable on a quantum computer in polynomial time (that is the time required to solve is bounded above by a polynomial function of the length of the input sequence), BQP is is a subset of of PSPACE, the set of problems which can be solved on a classical computer using memory bounded by a polynomial of the space of the input. Our understanding of quantum mechanics would have to be very far off for this to be wrong.

Second, if our understanding of quantum mechanics is correct, there's a fundamentally random aspect to the laws of physics. Thus, we can't simply make a simulation and advance it ahead the way they do in this story and expect to get the same result.

Even if everything in the story was correct, I'm not at all convinced that things would settle down on a stable sequence as they do here. If your universe is infinite then your possible number of worlds are infinite so there's no reason you couldn't have a wandering sequence of worlds. Edit: Or for that matter, couldn't have branches if people simulate additional worlds with other laws of physics or the same laws but different starting conditions.

Couldn't they just run the simulation to its end rather then just let it sit there and take the chance that it could accidently be destroyed. If its infinitley powerful, it would be able to do that.

Chinese internal martial arts: Tai Chi, Xingyi, and Bagua. The word "chi" does not carve reality at the joints: There is no literal bodily fluid system parallel to blood and lymph. But I can make training partners lightheaded with a quick succession of strikes to Ren Ying (ST9) then Chi Ze (LU5); I can send someone stumbling backward with some fairly light pushes; after 30-60 seconds of sparring to develop a rapport I can take an unwary opponent's balance without physical contact.

Each of these skills fit more naturally under different categories, but if you want to learn them all the most efficient way is to study a Chinese internal martial art or something similar.

Question: whats your experience with stuff that seems new agy at first look, like yoga, meditation and so on. Anything worth trying?

The Five Tibetans are a set of physical exercises which rejuvenate the body to youthful vigour and prolong life indefinitely. They are at least 2,500 years old, and practiced by hidden masters of secret wisdom living in remote monasteries in Tibet, where, in the earlier part of the 20th century, a retired British army colonel sought out these monasteries, studied with the ancient masters to great effect, and eventually brought the exercises to the West, where they were first published in 1939.

Ok, you don't believe any of that, do you? Neither do I, except for the first eight words and the last six. I've been doing these exercises since the beginning of 2009, since being turned on to them by Steven Barnes' blog and they do seem to have made a dramatic improvement in my general level of physical energy. Whether it's these exercises specifically or just the discipline of doing a similar amount of exercise first thing in the morning, every morning, I haven't taken the trouble to determine by varying them.

More here and here. Nancy Lebovitz also mentioned them.

I also do yoga for flexibility (it works) and occasionally meditation (to little detectable effect). I'd be interested to hear from anyone here who meditates and gets more from it than I do.

I've had great results from modest (2-3 hrs/wk) investments in hatha yoga, over and above what I get from standard Greco-Roman "calisthenics."

Besides the flexibility, breathing, and posture benefits, I find that the idea of 'chakras' is vaguely useful for focusing my conscious attention on involuntary muscle systems. I would be extremely surprised if chakras "cleaved reality at the joints" in any straightforward sense, but the idea of chakras helps me pay attention to my digestion, heart rate, bladder, etc. by making mentally uninteresting but nevertheless important bodily functions more interesting.

I've done yoga every week for the last month or two. It's pleasant. Other than paying attention to how I'm holding my body vs. the instruction, I mostly stop thinking for an hour (as we're encouraged to do), which is nice.

I can't say I notice any significant lasting effects yet. I'm slightly more flexible.

Hard to say - even New Agey stuff evolves. (Not many followers of Reich pushing their copper-lined closets these days.)

Generally, background stuff is enough. There's no shortage of hard scientific evidence about yoga or meditation, for example. No need for heuristics there. Similarly there's some for float tanks. In fact, I'm hard pressed to think of any New Agey stuff where there isn't enough background to judge it on its own merits.

Meditation can be pretty darn relaxing. Especially if you happen to live within walking distance of any pleasant yet sparsely-populated mountaintops. I would recommend giving it a shot; don't worry about advanced techniques or anything, and just close your eyes and focus on your breathing, and the wind (if any). Very pleasant.

Why would you want to do that, I mean, what are the supposed advantages? You might want to look it up and see if theres anything about it on the internet. Most alternative medicines are BS, but not necessarily all.

GRRRR! I wish it would let me comment faster then every 8 minutes. Guess I'll come back and post it.

This sounds great, I'm definitely in. I feel like I have a moderately okay intuitive grasp on Bayescraft but a chance to work through it from the ground up would be great.

In. Have the deadtree version, but I was stymied in my first crack at it.

In. If needed I can cover a few of the early chapters.

I'm in. I already read the first few chapters, but it will be nice to go over them to solidify that knowledge. The slower pace will help as well. The later chapters rely on some knowledge of statistics, maybe some member of the book club is already knowledgeable to be able to find good links to summaries of these things when they come up?

I would be interested, what is the intended time period for the reading? I have a two-week trip coming up when I will probably be busy but aside from that I would very much like to participate.

I'm in, been intending to read through some maths on my free time.

It's thesis writeup period for me, but this is extremely tempting.

I'm interested. I already have the book but haven't progressed very far so this seems like it's potentially a good motivator to finish it. The link to the PDF seems to be missing btw.

I'm enthusiastically in.

I think that a book club is a great idea, and this is an excellent choice for a book. I'm definitely interested.

Feedback sought: is this too short? Too long? Is the intent clear? What if anything is missing?

What would happen if CEV was applied to the Baby Eaters?

My intuitions of CEV are informed by the Rawlsian Veil of Ignorance, which effectively asks: "What rules would you want to prevail if you didn't know in advance who you would turn out to be?"

Where CEV as I understand it adds more information - assumes our preferences are extrapolated as if we knew more, were more the kind of people we want to be - the Veil of Ignorance removes information: it strips people under a set of specific circumstances of the detailed information about what their preferences are, what their contignent histories brought them there, and so on. This includes things like what age you are, and even - conceivably - how many of you there are.

To this bunch of undifferentiated people you'd put the question, "All in favor of a 99% chance of dying horribly shortly after being born, in return for the 1% chance to partake in the crowning glory of babyeating cultural tradition, please raise your hands."

I expect that not dying horribly takes lexical precedence over any kind of cultural tradition, for any sentient being whose kin has evolved to sentience (it may not be that way for constructed minds). So I would expect the Babyeaters to choose against cultural tradition.

The obvious caveat is that my intuitions about CEV may be wrong, but lacking a formal explanation of CEV it's hard to check intuitions.

So what I make of all this is that either CEV is not utility-function-neutral

Correct. CEV is supposed to be a component of Friendliness, which is defined in reference to human values.

CEV will be to maintain existing order.

Why? There must be very strong arguments for BEs to stop doing the Right Thing. And there's only one source of objections - children. And their volitions will be selfish and unaggregatable.

EDIT: What does utility-function-neutral mean?

EDIT: Ok. Ok. CEV will be to make BE's morale change and allow them to not eat children. So, FAI will undergo controlled shutdown. Objections, please?

EDIT: Here's yet another arguments.

Guidelines of FAI as of may 2004.

1. Defend humans, the future of humankind, and humane nature.

BEs will formulate this as "Defend BEs (except for the ceremony of BEing), the future of BEkind, and BE's nature."

1. Encapsulate moral growth.

BEs never considered, that child eating is bad. And it is good for them to kill anyone who thinks otherwise. There's no trend in moral that can be encapsulated.

1. Humankind should not spend the rest of eternity desperately wishing that the programmers had done something differently.

If they stop being BE they will mourn their wrong doings to the death.

1. Avoid creating a motive for modern-day humans to fight over the initial dynamic

Every single notion that FAI will make in lines of "Let's suppose that you are non-BE" will cause it to be destroyed.

1. Help people.

Help BEs everytime, but the ceremony of BEing.

How this will take FAI to the point that every conscious being must live?

[T]o avoid that trivial inconvenience, I've put up a mirror of it.

You're awesome.

I've previously been impressed by how social psychologists reason, especially about identity. Schemata theory is also a decent language for talking about cognitive algorithms from a less cognitive sciencey perspective. I look forward to reading this chapter. Thanks for mirroring, I wouldn't have bothered otherwise.

It depends on what you mean by "criminal"; under environmental law, there are both negligence-based (negligent discharge of pollutants to navigable waters) and strict liability (no intent requirement, such as killing of migratory birds) crimes that could apply to this spill. I don't think anyone thinks BP intended to have this kind of spill, so the interesting question from an environmental criminal law perspective is whether BP did enough to be treated as acting "knowingly" -- the relevant intent standard for environmental felonies. This is an extremely slippery concept in the law, especially given the complexity of the systems at issue here. Litigation will go on for many years on this exact point.

I've read somewhere that a BP internal safety check performed a few months ago indicated "unusual" problems which according to again BP internal safety guidelines should have been resolved earlier, but somehow they made an exception this time. It didn't seem like it would have been "illegal", and it also did not note how often such exceptions are made, by what reasoning, what kind of problems they specifically encountered, what they did to keep the operation running, et cetera...

Though I seldom read "ordinary" news, even of this kind, as my past experience tells me that factual information is rather low, and most high-quality press likes more to show off in opinion and interpretation of an event than trying to provide an accurate historical report, at least within such a short time-frame. Could well be that this is different at this event.

Also, as with most engineering disciplines, really learning from such an event beyond the obvious "there is a non-zero chance for everything to blow up" usually requires more area-specific expertise than an ordinary outsider has.

I've heard scattered bits of accusations of misdeeds by BP which may have contributed to the spill. Here's a list from the congressional investigation of 5 decisions that BP made "for economic reasons that increased the danger of a catastrophic well failure" according to a letter from the congressmen. It sounds like BP took a bunch of risky shortcuts to save time and money, although I'd want to hear from people who actually understand the technical issues before being too confident.

There are other suspicions and allegations floating around, like this one.

I'm not sure it's relevant whether they did anything illegal or not. People always seem to want to blame and punish someone for their problems. In my opinion, they should be forced to pay for and compensate for all the damage, as well as a very large fine as punishment. This way in the future they, and other companies, can regulate themselves and prepare for emergencies as efficiently as possible without arbitrary and clunky government regulations and agencies trying to slap everything together at the last moment. Of course, if a single person actually did something irresponsible (eg; bob the worker just used duct tape to fix that pipe knowing that it wouldn't hold) then they should be able to be tried in court or sued/fined by the company. But even then, it's up to the company to make sure that stuff like this doesn't happen by making sure all of their workers are competent and certified.

You are not really going to learn much unless you are interested in wading through lots of technical articles. If you want to learn, you need to wait until it has been digested by relevant experts into books. I am not sure what you think you can learn from this, but there are two good books of related information available now:

Jeff Wheelwright, Degrees of Disaster, about the environmental effects of the Exxon Valdez spill and the clean up.

Trevor Kletz, What Went Wrong?: Case Histories of Process Plant Disasters, which is really excellent. [For general reading, an older edition is perfectly adequate, new copies are expensive.] It has an incredible amount of detail, and horrifying accounts of how apparently insignificant mistakes can (often literally) blow up on you.

Not sure if you're referring to the same literature, but I note a great divergence between peak oil advocates and singularitarians. This is a little weird, if you think of Aumann's Agreement theorem.

Both groups are highly populated with engineer types, highly interested in cognitive biases, group dynamics, habits of individuals and societies and neither are mainstream.

Both groups use extrapolation of curves from very real phenomena. In the case of the kurzweillian singularitarians, it is computing power and in the case of the peak oil advocates, it is the hubbert curve for resources along with solid Net Energy based arguments about how civilization should decline.

The extreme among the Peak Oil advocates are collapsitarians and believe that people should drastically change their lifestyles, if they want to survive. They are also not waiting for the others to join them and many are preparing to go to small towns, villages etc. The oildrum, linked here had started as a moderate peak oil site discussing all possibilities, nowadays, apparently, its all doom all the time.

The extreme among the singularitarians have been asked no such sacrifice, just to give enough money and support to make sure that Friendly AI is achieved first.

Both groups believe that business as usual cannot go on for too long, but they expect dramatically different consequences. The singularitarians assert that economics conditions and technology will improve until a nonchalant super-intelligence will be created and wipe out humanity. The collapsitarians believe that economic conditions will worsen, civilization is not built robustly and will collapse badly with humanity probably going extinct or only the last hunter gatherers surviving.

Could you give some examples of the predicted collapses that didn't happen?

I was talking to a friend yesterday and he mentioned a psychological study (I am trying to track down the source) that people tend to suffer MORE from failing to pursue certain opportunities than FAILING after pursuing them. So even if you're right about the overestimation of pleasure, it might just be irrelevant.

No doubt there is truth in this... however examples spring into my mind where accomplishing something made me feel better than what I ever expected. This includes sport (ever win a race or score a goal in a high stakes soccer game?), work and personal life. The "reality is humdrum" perspective might, at least in part, be caused by a disconnect between "imagination" and "action".

Also, "Invest in the process, not the outcome".

Often it is our imagined bad futures that keep us too afraid to act. In my experience this is more common than the opposite.

Maybe you can set your success setpoint to a lower value. The optimum is hard to achieve. So looking for 100% everywhere might be bad.

Five-second guess: Human-level Machiavellian intelligence needs language facilities to co-evolve with, grunts and body language doesn't allow nearly as convoluted schemes. Evolving some precursor form of human-style language is the improbable part that other species haven't managed to pull off.

Somewhat accepted partial answer is that huge brains are ridiculously expensive - you need a lot of high energy density food (= fire), a lot of DHA (= fish) etc. Chimp diet simply couldn't support brains like ours (and aquatic ape etc.), nor could they spend as much time as us engaging in politics as they were too busy just getting food.

Perhaps chimp brains are as big as they could possibly be given their dietary constraints.

I think the expectation is that, if all humans had the same knowledge and were better at thinking (and were more the people we'd like to be, etc.), then there would be a much higher degree of coherence than we might expect, but not necessarily that everyone would ultimately have the same utility function.

Or is it equally plausible to assume that some important values cannot be extrapolated coherently, and that a Seed-AI would therefore provide several results clustered around some groups of people?

There is only one world to build something from. "Several results" is never a solution to the problem of what to actually do.

I think the idea is that CEV lets us "grow up more together" and figure that out later.

I have only recently started looking into CEV so I'm not sure whether I a) think it's a workable theory and b)think it's a good solution, but I like the way it puts off important questions.

It's impossible to predict what we will want if age, disease, violence, and poverty become irrelevant (or at least optional).

I'd like to ask everyone what probability bump they give to an idea given that some people believe it.

Usually fairly substantial - if someone presents me with two equally-unsupported claims X and Y and tells me that they believe X and not Y, I would give greater credence to X than to Y. Many times, however, that credence would not reach the level of ... well, credence, for various good reasons.

Depends on the person and the idea. I have some people whose recommendations I follow regardless, even if I estimate upfront that I will consider the idea wrong. There are different levels of wrongness, and it does not hurt to get good counterarguments. It also depends on the real life practicability of the idea. If it is for everyday things than common sense is a good starting prior. (Also there is a time and place to use the public joker on Who wants to be a millionaire.) If a group of professionals agree on something related to their profession it is also a good start. To systematize: if a group of people has a belief about something they have experience with, that that belief is worth looking at.

And then on further investigation it often turns out that there are systematic mistakes being made.

I was shocked to read in the book on checklists, that not only doctors often don't like them. But even financial companies, that can see how the usage ups their monetary gains. But finding flaws in a whole group does not imply that everything they say is wrong. It is good to see a doctor, even if he not using statistics right. He can refer you to a specialist, and treat all the common stuff right away. If you get a complicated disease you can often read up on it.

The obvious example to your question would be religion. It is widely believed, but probably wrong, yet I did not discard it right away, but spent years studying stuff till I decided there was nothing to it. There is nothing wrong in examining the ideas other people have.

If no people believe Y -- literally no people -- then either the topic is very little examined by human beings, or it's very exhaustively examined and seems obvious to everyone. In the first case, I give a smaller probability than in the second case.

In the first case, only X believers exist because only X believers have yet considered the issue. That's minimal evidence in favor of X. In the second case, lots of people have heard of the issue; if there were a decent case against X, somebody would have thought of it. The fact that none of them -- not a minority, but none -- argued against X is strong evidence that X is true.

I don't think belief has a consistent evidentiary strength since it depends on the testifier's credibility relative to my own. Children have much lower credibility than me on the issue of the existence of Santa. Professors of physics have much higher credibility that me on the issue of dimensions greater than four. Some person other than me has much higher credibility on the issue of how much money they are carrying. But I have more credibility than anyone else on the issue of how much money I'm carrying. I don't see any relation that could be described as baseline so the only answer is: context.

I've become increasingly disillusioned with people's capacity for abstract thought. Here are two points on my journey.

The public discussion of using wind turbines for carbon-free electricity generation seems to implicitly assume that electricity output goes as something like the square-root of windspeed. If the wind is only blowing half speed you still get something like 70% output. You won't see people saying this directly, but the general attitude is that you only need back up for the occasional calm day when the wind doesn't blow at all.

In fact output goes as the cube of windspeed. The energy in the windstream is one half m v squared, where m, the mass passing your turbine is proportional to the windspeed. If the wind is at half strength, you only get 1/8 output.

Well, that is physics. Ofcourse people suck at physics. Trouble is, the more I look at people's capacity for abstract thought the more problems I see. When people do a cost/benefit analysis they are terribly vague on whether they are suposed to add the costs and benefits or whether the costs get subtracted from the benefits. Even if they realise that they have to subtract they are still at risk of using an inverted scale for the costs and ending up effectively adding.

The probabiltiy bump I give to an idea just because some people believe it is zero. Equivantly my odds ratio is one. However you describe it, my posterior is just the same as my prior.

I think it largely depends on a) what the idea is and b) who believes it = and what their rationality skills are.

Out of the huge idea space of possible causally linked events, some of them make good stories and some do not. That doesn't tell you rather it's true or not.

If a guy thinks that he can hear Hillary Clinton speaking from the feelings in his teeth, telling him to murder his cellmate, do you believe what he says? Status gets mucked up in the calculation, but with strangers it teeters precariously close to zero.

I really like kids,but the fact that millions of them passionately believe in Santa Claus does not change my degree of subjective belief one iota.

I'd like to ask everyone what probability bump they give to an idea given that some people believe it.

None.

Or as Ben Goldacre put it in a talk: There are millions of medical doctors and Ph.D.s in the world. There is no idea, however completely fucking crazy, that you can't find some doctor to argue for.

So, given that some people believe X, what probability do you give for X being true, compared to Y which nobody currently believes?

In any case of a specific X and Y, there will be far more information than that (who believes X and why? does anyone disbelieve Y? etc.), which makes it impossible for me to attach any probability for the question as posed.

Beautiful. Matthew Yglesias, +1 point.

It is entirely possible that some social groups are experiencing the kind of changes that Flanagan describes, but as Yglesias says, she apparently is unaware that there is such a thing as scientific evidence on the question.

The unbounded utility function (in some physical objects that can be tiled indefinitely) in Pascal's mugging gives infinite expected utility to all actions, and no reason to prefer handing over the money to any other action. People don't actually show the pattern of preferences implied by an unbounded utility function.

If we make the utility function a bounded function of happy lives (or other tilable physical structures) with a high bound, other possibilities will offer high expected utility. The Mugger is not the most credible way to get huge rewards (investing in our civilization on the chance that physics allows unlimited computation beats the Mugger). This will be the case no matter how huge we make the (finite) bound.

Eliezer's original article bases its argument on the use of Solomonoff induction. He even suggests up front what the problem with it is, although the comments don't make anything of it: SI is based solely on program length and ignores computational resources. The optimality theorems around SI depend on the same assumption. Therefore I suggest:

4. Pascal's Mugging is a refutation of the Solomonoff prior.

But where a computationally bounded agent, or an unbounded one that cares how much work it does, should get its priors from instead would require more thought than a few minutes on a lunchtime break.

Tom_McCabe2 suggests generalizing EY's rebuttal of Pascal's Wager to Pascal's Mugging: it's not actually obvious that someone claiming they'll destroy 3^^^^3 people makes it more likely that 3^^^^3 people will die. The claim is arguably such weak evidence that it's still about equally likely that handing over the $5 will kill 3^^^^3 people, and if the two probabilities are sufficiently equal, they'll cancel out enough to make it not worth handing over the $5.

Personally, I always just figured that the probability of someone (a) threatening me with killing 3^^^^3 people, (b) having the ability to do so, and (c) not going ahead and killing the people anyway after I give them the $5, is going to be way less than 1/3^^^^3, so the expected utility of giving the mugger the $5 is almost certainly less than the $5 of utility I get by hanging on to it. In which case there is no problem to fix. EY claims that the Solomonoff-calculated probability of someone having 'magic powers from outside the Matrix' 'isn't anywhere near as small as 3^^^^3 is large,' but to me that just suggests that the Solomonoff calculation is too credulous.

(Edited to try and improve paraphrase of Tom_McCabe2.)

I'm not sure this problem needs a "solution" in the sense that everyone here seems to accept. Human beings have preferences. Utility functions are an imperfect way of modeling those preferences, not some paragon of virtue that everyone should aspire to. Most models break down when pushed outside their area of applicability.

The utility function assumes that you play the "game" (situation, whatever) an infinite number of times and then find the net utility. Thats good when your playing the "game" enough times to matter. It's not when your only playing a small number of times. So lets look at it as "winning" or "loosing". If the odds are really low and the risk is high and your only playing once, then most of the time you expect to loose. If you do it enough times, you even the odds out and the loss gets canceled out by the large reward, but only playing once you expect to loose more then you gain. Why would you assume differnetly? Thats my 2 cents and so far its the only way I have come up with to navigate around this problem.

I think I found the study they're talking about thanks to this article. I might take a look at it - if the methodology is literally just 'smoking was banned, then the heart attack rate dropped', that sucks.

(Edit to link to the full study and not the abstract.)

Just skimmed it. The methodology is better than that. They use a regression to adjust for the pre-existing downward trend in the heart attack hospital admission rate; they represent it as a linear trend, and that looks fair to me based on eyeballing the data in figures 1 and 2. They also adjust for week-to-week variation and temperature, and the study says its results are 'more modest' than others', and fit the predictions of someone else's mathematical model, which are fair sanity checks.

I still don't know how robust the study is - there might be some confounder they've overlooked that I don't know enough about smoking to think of - but it's at least not as bad as I expected. The authors say they want to do future work with a better data set that has data on whether patients are active smokers, to separate the effect of secondhand smoke from active smoking. Sounds interesting.

I agree that this article isn't very good. It seems to do the standard problem of combining a lot of different ideas about what the Singularity would entail. It emphasizes Kurzweil way too much, and includes Kurzweil's fairly dubious ideas about nutrition and health. The article also uses Andrew Orlowski as a serious critic of the Singularity making unsubstantiated claims about how the Singularity will only help the rich. Given that Orlowski's entire approach is to criticize anything remotely new or weird-seeming, I'm disappointed that the NYT would really use him as a serious critic in this context. The article strongly reinforces the perception that the Singularity is just a geek-religious thing. Overall, not well done at all.

It's better than mainstream Singularity articles in the past, IMO; unfortunately, Kurzweil is seen as an authority, but at least it's written with some respect for the idea.

It does seem to be about a lot of different things, some of which are just synonymous with scientific progress (I don't think it's any revelation that synthetic biology is going to become more sophisticated.)

I'm curious: Was the SIAI contacted for that article? I haven't had time to read it all, but a word-search for "Singularity Institute" and "Yudkowsky" turned up nothing.

I wish you wouldn't do that, and stuck instead with the generally approved norm of downvoting to mean "I'd prefer to see fewer comments like this" and upvoting "I'd like to see more like this".

You're deliberately participating in information cascades, and thereby undermining the filtering process. As an antidote, I recommend using the anti-kibitzer script (you can do that through your Preferences page).

I am tempted to downvote this comment from -2 just for the irony, but I don't prefer to see fewer comments like this, so I won't.

Besides, the default cutoff is at -4, not -3.

That article makes it sound like "countersignaling" is forgoing a mandated signal - like showing up at a formal-dress occasion in street clothes.

Alicorn made a post about the tactics of countersignaling a while back.

Am I correct in reading that Longbets charges a $50 fee for publishing a prediction and they have to be a minimum of 2 years in the future? Thats a bit harsh. But these sites are pretty interesting. And they could be useful to. You could judge the accuracy of different users including how accurate they are at guessing long-term, short-term, etc predictions as well as how accurate they are in different catagories (or just how accurate they are on average if you want to be simple.) Then you can create a fairly decent picture of the future, albeit I expect many of the predictions will contradict each other. This is kind of what their already doing obviously, but they could still take it a step further.

I would not worry overmuch about the long-term negative effects of your studying for the bar: with the possible exception of the "overly sincere" types who fall very hard for cults and other forms of indoctrination, people have a lot of antibodies to this kind of thing.

You will continue to be entagled with reality after you pass the exam, and you can do things, like read works of social science that carve reality at the joints, to speed up the rate at which your continued entaglement with reality with cancel out any falsehoods you have to cram for now. Specifically, there are works about the law that do carve reality at the joints -- Nick Szabo's online writings IMO fall in that category. Nick has a law degree, by the way, and there is certainly nothing wrong with his ability to perceive reality correctly.

ADDED. The things that are really damaging to a person's rationality, IMHO, are natural human motivations. When for example you start practicing, if you were to decide to do a lot of trials, and you learned to derive pleasure -- to get a real high -- from the combative and adversarial part of that, so that the high you got from winning with a slick and misleading angle trumped the high you get from satisfying you curiosity and from refining and finding errors in your model of reality -- well, I would worry about that a lot more than your throwing yourself fully into winning on this exam because IMHO the things we derive no pleasure from, but do to achieve some end we care about (like advancing in our career by getting a credential) have a lot less influence on who we turn out to be than things we do because we find them intrinsically rewarding.

One more thing: we should not all make our living as computer programmers. That would make the community less robust than it otherwise would be :)

I worry about this as well when I'm reading long arguments or long works of fiction presenting ideas I disagree with. My tactic is to stop occasionally and go through a mental dialog simulating how I would respond to the author in person. This serves a double purpose, as hopefully I'll have better cached arguments in the event I ever need them.

Of course, this is a dangerous tactic as well, because you may be shutting off critical reasoning applied to your preexisting beliefs. I only apply this tactic when I'm very confident the author is wrong and is using fallacious arguments. Even then I make sure to spend some amount of time playing devil's advocate.

I found an interesting paper on Arxiv earlier today, by the name of Closed timelike curves via post-selection: theory and experimental demonstration.

It promises such lovely possibilities as quick solutions to NP-complete problems

It won't work, as is clearly explained here.

If this worked, Harry could use it to recover any sort of answer that was easy to check but hard to find. He wouldn't have just shown that P=NP once you had a Time-Turner, this trick was more general than that. Harry could use it to find the combinations on combination locks, or passwords of every sort. Maybe even find the entrance to Slytherin's Chamber of Secrets, if Harry could figure out some systematic way of describing all the locations in Hogwarts. It would be an awesome cheat even by Harry's standards of cheating.

Harry took Paper-2 in his trembling hand, and unfolded it.

Paper-2 said in slightly shaky handwriting:

DO NOT MESS WITH TIME

Harry wrote down "DO NOT MESS WITH TIME" on Paper-1 in slightly shaky handwriting, folded it neatly, and resolved not to do any more truly brilliant experiments on Time until he was at least fifteen years old.

To put this into my own words "The more information you extract from the future, the less you are able to control the future from the past. And hence, the less understanding you can have about what those bits of future-generated information are actually going to mean."

I wrote that before actually looking at the paper you linked. I don't understand much QM either, but now that I have looked it seems to me that figure 2 of the paper backs me up on my interpretation of Harry's experiment.

From that Wikipedia article:

Inside the railcar, besides the paper clips, there are the Schroeders’ book and a suitcase filled with letters of apology to Anne Frank by a class of German schoolchildren.

Apologizing for ... being German? That's really bizarre.

I think it's very noble of them to collect numerous paperclips and hold them safely out of use. c=@ I just hope they have appropriate protocols in place to ensure they don't become stolen or unbent. Anyone know if there's an insurance policy taken out against loss or destruction of the paperclips?

Do you ever have a day when you log on and it seems like everyone is "wrong on the Internet"? (For values of "everyone" equal to 3, on this occasion.) Robin Hanson and Katja Grace both have posts (on teenage angst, on population) where something just seems off, elusively wrong; and now SarahC suggests that "the only friendly AI may be one that commits suicide". Something about this conjunction of opinions seems obscurely portentous to me. Maybe it's just a know-thyself moment; there's some nascent opinion of my own that's going to crystallize in response.

Now that my special moment of sharing is out of the way... Sarah, is the friendly AI allowed to do just one act of good before it kills itself? Make a child smile, take a few pretty photos from orbit, save someone from dying, stop a war, invent cures for a few hundred diseases? I assume there is some integrity of internal logic behind this thought of yours, but it seems to be overlooking so much about reality that there has to be a significant cognitive disconnect at work here.

I'm not necessarily arguing for this position as saying we need to address it. "Suicidal AI" is to the problem of constructing FAI as anarchism is to political theory; if you want to build something (an FAI, a good government) then, on the philosophical level, you have to at least take a stab at countering the argument that perhaps it is impossible to build it.

I'm working under the assumption that we don't really know at this point what "Friendly" means, otherwise there wouldn't be a problem to solve. We don't yet know what we want the AI to do.

What we do know about morality is that human beings practice it. So all our moral laws and intuitions are designed, in particular, for small, mortal creatures, living among other small, mortal creatures.

Egalitarianism, for example, only makes sense if "all men are created equal" is more or less a statement of fact. What should an egalitarian human make of a powerful AI? Is it a tyrant? Well, no, a tyrant is a human who behaves as if he's not equal to other humans; the AI simply isn't equal. Well, then, is the AI a good citizen? No, not really, because citizens treat each other on an equal footing...

The trouble here, I think, is that really all our notions of goodness are really "what is good for a human to do." Perhaps you could extend them to "what is good for a Klingon to do" -- but a lot of moral opinions are specifically about how to treat other people who are roughly equivalent to yourself. "Do unto others as you would have them do unto you." The kind of rules you'd set for an AI would be fundamentally different from our rules for ourselves and each other.

It would be as if a human had a special, obsessive concern and care for an ant farm. You can protect the ants from dying. But there are lots of things you can't do for the ants: be an ant's friend, respect an ant, keep up your end of a bargain with an ant, treat an ant as a brother...

I had a friend once who said, "If God existed, I would be his enemy." Couldn't someone have the same sentiment about an AI?

(As always, I may very well be wrong on the Internet.)

It seems unlikely that an FAI would commit suicide if humans need to be protected from UAI, or if there are other threats that only an FAI could handle.

I've been thinking about finally starting a Study Group thread, primarily with a focus on Jaynes and Pearl both of which I'm studying at the moment. It would probably make sense to expand it to other books including non-math books - though the set of active books should remain small.

Two things have been holding me back - for one, the IMO excessively blog-like nature of LW with the result that once a conversation has rolled off the front page it often tends to die off, and for another a fear of not having enough time and energy to devote to actually facilitating discussion.

Facilitation of some sort seems required: as I understand it a book club or study group entails asking a few participants to make a firm commitment to go through a chapter or a section at a time and report back, help each other out and so on.

Thanks for that, Price is a very knowledgeable New Testament scholar. Check out his interview at the commonsenseatheism podcast here, also covers his path to becoming a christian atheist.

Remember, probabilities are not inherent facts of the universe, they are statements about how much you know. You don't have perfect knowledge of the universe, so when I ask, "Is your mum on the phone?" you don't have the guaranteed correct answer ready to go. You don't know with complete certainty.

But you do have some knowledge of the universe, gained through your earlier observations of seeing your mother on the phone occasionally. So rather than just saying "I have absolutely no idea in the slightest", you are able to say something more useful: "It's possible, but unlikely." Probabilities are simply a way to quantify and make precise our imperfect knowledge, so we can form more accurate expectations of the future, and they allow us to manage and update our beliefs in a more refined way through Bayes' Law.

The cases are different in the way that you describe, but the maths of the probability is the same in each case. If you have an unseen die under a cup, and a die that you are about to roll, then one is already determined and the other isn't, but you'd bet at the same odds for each one to come up a six.

I think the difference is that one event is a statement about the present which is either presently true or not, and the other is a prediction. So you could illustrate the difference by using the following pairs: P(Mom on phone now) vs. P(Mom on phone tomorrow at 12:00am). In the dice case P(die just rolled but not yet examined is 1) vs. P(die I will roll will come out 1).

I do agree with Oscar though, the maths should be the same.

You might be interested in this recent discussion, if you haven't seen it already:

http://lesswrong.com/lw/2ax/open_thread_june_2010/23fa

It looks to me like your confusion with these examples just stems from the fact that one event is in the present and the other in the future. Are you still confused if you make it P(Mom will be on the phone at 4 PM tomorrow)= 1/6. Or conversely, you make it P(I rolled a one on the fair die that is now beneath this cup) =1/6

In the second case, you either roll one on the die or not, but you are pretty sure that it will be another number.

It does seem odd to get such divergent results.

Bad luck could be, not just getting that 5% result which 95% accuracy implies, but some non-obvious difference in the volunteers (different genetics?), in the tea. or in the milk.

Being right on group effects is difficult.

Is there a consistent path for what LW wants to be? a) rationalist site filled up with meta topics and examples b) a) + detailed treats of some important topics c) open to everything as long as reason is used

and so on. I personally like and profit from the discussing of akrasia methods. But it might be detrimental to the main target of the site. Also I would very much like to see a cannon develop for knowledge that LWers generally agree upon including, but not limited to the topics I currently care about myself.

Voicing ideas depends on where you are. In social settings I more and more advice against it. Arguing/discussing is just not helpful. And if you are filled up with weird ideas then you get kicked out, which might be bad for other goals you have.

It would be great to have a place for any idea to be examined for right and wrong.

Yes. There are cases where occam's razor results in a tie (or, at least, indistinguishably close).

Consider the spin on an arbitrary particle in deep space, or whether or not an arbitrary digit of pi is even.

The method that Eliezer is referring to is known as Solomonoff induction which relies on programs as defined by Turing machines. Quantum computing doesn't come into this issue since these formulations just talk about length of specification, not efficiency of computation. There are theorems that also show that for any given Turing complete well-behaved language, the minimum size of program can't be differ by more than a constant. So changing the language won't alter the priors other than a fixed amount. Taken together with Aumann's Agreement Theorem, the level of disagreement about estimated probability should go to zero in the limiting case (disclaimer I haven't seen a proof of that last claim, but I suspect it would be a consequence of using a Solomonoff style system for your priors).

How can I understand quantum physics?

I don't think the subject is inherently difficult. For example quantum computing and quantum cryptography can be explained to anyone with basic clue and basic math skills.

That's because quantum computing and quantum cryptography only use a subset of quantum theory. Your link says, for example, that the basics of quantum computing only require knowing how to handle 'discrete (2-state) systems and discrete (unitary) transformations,' but a full treatment of QT has to handle 'continuously infinite systems (position eigenstates) and continuous families of transformations (time development) that act on them.' The full QT that can deal with these systems uses a lot more math.

I wonder if there's a general trend for people who are interested in quantum computing and not all of QT to play down the prerequisites you need to learn QT. Your post reminded me of a Scott Aaronson lecture, where he says

The second way to teach quantum mechanics leaves a blow-by-blow account of its discovery to the historians, and instead starts directly from the conceptual core -- namely, a certain generalization of probability theory to allow minus signs. Once you know what the theory is actually about, you can then sprinkle in physics to taste, and calculate the spectrum of whatever atom you want.

Which is technically true, but if you want to know about quark colors or spin or exactly how uncertainty works, pushing around |1>s and |2>s and talking about complexity classes is not going to tell you what you want to know.

To answer your question more directly, I think the best way to understand quantum physics is to get an undergrad degree in physics from a good university, and work as hard as you can while you're getting it. Getting a degree means you have the physics-leaning math background needed to understand explanations of QT that don't dumb it down.

I might be overestimating the amount of math that's necessary - I'm basing this on sitting in on undergrad QT lectures - but I've yet to find a comprehensive QT text that doesn't use calculus, complex numbers, and linear algebra.

Try Jonathan Allday's book "Quantum Reality: Theory and Philosophy." It is technical enough that you get a quantitative understanding out of it, but nothing like a full-blown textbook.

Well, I haven't read any other blog posts of him but the one you linked to, but in this specific case I cannot find what there is to be attacked.

It is stories like this that are used to explain that some values are of higher importance than others, in simple terms (a style that also exists in the not-so-extended circle of LW).The fictional senior monk's answer would be obvious for anybody who has read up even just a little bit on Zen and/or Buddhism, it is more reinforcing than teaching news.

If the blogger is often holding an anti-reductionist position you'd like to counter, I'd go for actually anti-reductionist posts of him...

Not sure where I stand actually, but this seems relevant:

"If God did not exist, it would be necessary to invent him" - Voltaire

I suppose it should be added that one should do one's best to make sure the god that's created is more Friendly than Not.

Huh. That's very interesting. I'm a bit confused by the claim that evolution bridges the is/ought divide which seems more like conflating different meanings of words more than anything else. But the general point seems strong.