## Posts

## Comments

**larry_d-anna**on The Pascal's Wager Fallacy Fallacy · 2009-03-18T02:50:43.000Z · score: 1 (1 votes) · LW · GW

"first-order logic cannot, in general, distinguish finite models from infinite models."

Specifically, if a fist order theory had arbitrarily large finite models, then it has an infinite one.

**larry_d-anna**on War and/or Peace (2/8) · 2009-01-31T20:33:56.000Z · score: 0 (0 votes) · LW · GW

Did Ira Howard actually say that? In which story?

**larry_d-anna**on Lawful Creativity · 2008-11-09T08:10:16.000Z · score: 0 (0 votes) · LW · GW

I'm getting Deja Vu again. Are you recycling bits of older posts or other things you've written?

**larry_d-anna**on Economic Definition of Intelligence? · 2008-10-29T20:27:52.000Z · score: 1 (1 votes) · LW · GW

Eliezer: have you given any thought to the problem of choosing a measure on the solution space? If you're going to count bits of optimization, you need some way of choosing a measure. In the real world solutions are not discrete and we cannot simply count them.

**larry_d-anna**on Expected Creative Surprises · 2008-10-25T21:22:31.000Z · score: 0 (0 votes) · LW · GW

I swear to god I've read these Kasparov posts before...

**larry_d-anna**on Belief in Intelligence · 2008-10-25T21:17:57.000Z · score: -2 (1 votes) · LW · GW

I feel like I've read this exact post before. Deja Vu?

**larry_d-anna**on Ethics Notes · 2008-10-22T04:56:26.000Z · score: 0 (0 votes) · LW · GW

Moral questions are terminal. Ethical questions are instrumental.

I would argue that ethics are values that are instrumental, but treated as if they were terminal for almost all real object-level decisions. Ethics are a human cognitive shortcut. We need ethics because we can't really compute the expected cost of a black swan bet. An AI without our limitations might not need ethics. It might be able to keep all it's instrumental values in it's head *as* instrumental, without getting confused like we would.

**larry_d-anna**on My Bayesian Enlightenment · 2008-10-05T21:31:08.000Z · score: 0 (0 votes) · LW · GW

"But it was PT:TLOS that did the trick. Here was probability theory, laid out not as a clever tool, but as The Rules, inviolable on pain of paradox"

I am unaware of a statement of Cox's theorem where the full *technical* statement of the theorem comes even close to this informal characterization. I'm not saying it doesn't exist, but PT:TLOS certainly doesn't do it.

I found the first two chapters of PT:TLOS to be absolutely, wretchedly awful. It's full of technical mistakes, crazy mischaracterizations of other people's opinions, hidden assumptions and skipped steps (that he tries to justify with handwaving nonsense), and even a discussion of Godel's theorems that mixes meta levels and completly misses the point.

**larry_d-anna**on The Sheer Folly of Callow Youth · 2008-09-19T06:06:40.000Z · score: 2 (2 votes) · LW · GW

Eliezer, I think you have dissolved one of the most persistent and venerable mysteries: "How is it that even the smartest people can make such stupid mistakes".

Being smart just isn't *good* *enough*.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-23T05:59:00.000Z · score: 0 (0 votes) · LW · GW

J Thomas *Larry, you have not proven that 6 would be a prime number if PA proved 6 was a prime number, because PA does not prove that 6 is a prime number.*

No I'm afraid not. You clearly do not understand the ordinary meaning of implications in mathematics. "if a then b" is equivalent (in boolean logic) to ((not a) or b). They mean the exact same thing.

*The claim that phi must be true because if it's true then it's true*

I said no such thing. If you think I did then you do not know what the symbols I used mean.

*It's simply and obviously bogus, and I don't understand why there was any difficulty about seeing it.*

No offense, but you have utterly no idea what you are talking about.

*Similarly, if PA proved that 6 was prime, it wouldn't be PA*

PA is an explicit finite list of axioms, plus one axiom schema. What PA proves or doesn't prove has absolutely nothing to do with it's definition.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-22T17:56:00.000Z · score: 0 (0 votes) · LW · GW

J Thomas: "How is that useful?"

I'm just answering your question

"Can you prove that if PA proved 6 was a prime number then 6 would be a prime number? How would you do it?"

I'm not stating that proving implications with false antecedent is particularly useful, just that it is valid. Also aside from 6 being prime it is true that for any sentence phi, ZF |- "if PA |- phi then phi", but that ZF cannot even say, yet alone prove that "forall phi. if PA |- phi then phi". But it can prove "forall phi. if PA |- phi then N |= phi".

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-22T13:36:00.000Z · score: 0 (0 votes) · LW · GW

"But Larry, PA does not actually say that 6 is prime, and 6 is not prime."

Well of course 6 isn't prime. But if PA said it was, then it would be. There's nothing invalid about proving that A -> B if you know ~A. It's just not very useful. But for a somewhat less vacuous example, let RH be the riemann hypothesis. Then if PA |- RH then RH is true and if PA |- ~RH then RH is false. At least one of these implications has a false hypothesis, but they are both perfectly valid.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-22T02:49:07.000Z · score: 1 (1 votes) · LW · GW

J Thomas:

*Once more through the mill. If PA proves that 6 is a prime number, then 6 is really a prime number. Can you prove that if PA proved 6 was a prime number then 6 would be a prime number? How would you do it?*

If PA |- "forall x y . x * y = 6 => |x|=1 || |y|=1"
then N |= "forall x y . x * y = 6 => |x|=1 || |y|=1"
(N = the natural numbers equiped with + and *)
so for all x and y in N, N |= ",x * ,y = 6 => |,x|=1 || |,y|=1"
(where ,x means a constant symbol for x)
if x*y = 6 then N |= ",x * ,y = 6" so therefore
N |= "|,x|=1 || |,y|=1"
thus either N |= "|,x| = 1" or N |= "|,y| = 1"
thus either |x|=1 or |y|=1
therefore we have that if x*y = 6 then either |x| = 1 or |y| = 1
therefore 6 is prime
therefore if PA |- "6 is prime" then 6 is actually prime

Of course it is also a meta-theorem that for any sentence phi in the language of PA that

ZF |- "PA |- phi => phi_omega"

where phi_omega is phi relativeized to the finite ordinals.

**larry_d-anna**on No License To Be Human · 2008-08-21T20:27:48.000Z · score: 1 (1 votes) · LW · GW

Caledonian:
*That's relativism, right there - the idea that rightness is not only socially determined, but individually socially determined. *

What!? That's just not what I said at all.

**larry_d-anna**on No License To Be Human · 2008-08-21T18:40:26.000Z · score: 3 (3 votes) · LW · GW

Richard: It seems to me that asking how is it that the word 'right' came to refer to rightness is like asking why 'green' means green, instead of meaning zebra.

The fact is that there is some concept that we've been calling "right", and though we don't exactly know what we mean by it, we're pretty certain it means something, and in some cases we know it when we see it.

It strikes me as unfair to accuse Eliezer of having his own private meaning of "right" that isn't in accordance with the common one, because hasn't endorsed a criterion or decision procedure for 'right', he hasn't tried to define it, he hasn't made clearly-wrong claims about it like "murder is right", he really hasn't said much of anything about the object-level practical meaning of 'right'. He has mostly just discussed certain meta-level features of the concept, such as the fact that isn't all-possible-minds-universal, and the idea that one who explicitly thinks "If i think X is right, then X is right" can think that anything is right.

**larry_d-anna**on No License To Be Human · 2008-08-21T18:07:18.000Z · score: 0 (2 votes) · LW · GW

Roko:

*
Eliezer decides to start using the symbol "c" to denote the real number 3*10^8.
*

No, he has continuously refused to spell out an explicit description of morality, because it admits no concise description. When Eliezer writes a list of values ending with "etcetera" he's saying (in your analogy) "c is 3*10^8, up to one significant digit".

Richard:

*but others have pointed out that this reference fact is fixed by means of a seemingly 'relative' procedure*

I think you are mixing meta-levels here. The seemingly relative procedure is used to describe morality in blog posts, not to chose what morality is in the first place.

**larry_d-anna**on No License To Be Human · 2008-08-21T15:43:27.000Z · score: 1 (1 votes) · LW · GW

Roko: It certainly is possible to opine that 22 is prime. Watch this:

22 is prime!

See, I did it. If you claim murder is right, then you aren't talking about something other than *right*, you are just making false statements about *right*.

**larry_d-anna**on No License To Be Human · 2008-08-21T15:32:09.000Z · score: 2 (4 votes) · LW · GW

@Roko

Also, Echoing Jadagul: as most people use the words, you're a moral relativist

Honestly I do not understand how you can continue calling Eliezer a relativist when he has persistently claimed that what is right *doesn't depend on who's asking and doesn't depend on what anyone thinks is right*.

Is anyone who does not believes in universally compelling arguments a relativist?

Is anyone who does not believe that morality is ontologically primitive a relativist?

Is anyone who does not believe that morality admits a concise description a relativist?

**larry_d-anna**on No License To Be Human · 2008-08-21T01:50:59.000Z · score: 0 (0 votes) · LW · GW

*
With these distinctions in mind, I hope my intent will be clearer, when I say that although I am human and have a human-ish moral framework, I do not think that the fact of acting in a human-ish way licenses anything.
*

hah. I was wondering what this Lob stuff had to do with morality.

good job on an excellent post.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-20T15:32:53.000Z · score: 0 (0 votes) · LW · GW

simon:

To argue that a proof is being made concluding ?C using the assumption ?(◻C -> C) given the theory PA, to which proof we can apply the deduction theorem to get (PA |- "?(◻C -> C) -> ?C") (i.e. my interpretation of Löb's Theorem)

OK so the question marks are boxes right? In that case then yes, PA |- "?(?C -> C) -> ?C". This is OK. The contradiction comes if PA |- "(?C->C)->C". But *morally* this doesn't have anything to do with the deduction theorem. PA proves Lob because everything in the proof of Lob is expressible inside of PA.

Like I said before, the deduction theorem is really just a technical lemma. If I'm doing ordinary mathematics (not logic), and I assume X, and prove Y, and then say "ok well now I've proved X -> Y", then I have not used the deduction theorem, because I'm writing a proof, not explicitly reasoning *about* proofs. The deduction theorem lies a meta level up, where we have a explicit, specific, technical definition of what constitutes a proof, and we are trying to prove theorems from that definition.

But the proof uses an additional assumption which is the antecedent of an implication, and comes to a conclusion which is the consequent of the implication. To get the implication, we must use the deduction theorem or something like it, right?

Nope, we are using an ordinary principle of mathematical reasoning. The deduction theorem says that if you have a proof that uses this principle and is otherwise first-order, you can convert it into a pure first order proof.

Is this fact a theorem of first order logic without any additional assumptions, or is it merely a theorem of PA? I admit I don't know, as I'm not very familiar with first order logic, but it intuitively seems to me that if first order logic were powerful enough on its own to express concepts like "PA proves X" it would probably be powerful enough to express arithmetic, in which case the qualification in Gödel's theorem that it only applies to theories that express arithmetic would be superfluous.

First order logic without any additional assumptions can't even express concepts like like PA. So, yea; that's why Gödel's theorem has that qualification, because there are plenty first order theories that are simple enough that they can't express integers.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-20T01:05:20.000Z · score: 0 (0 votes) · LW · GW

simon:

I was thinking that Löb's Theorem was a theorem in PA

It is a theorem about PA, although everything the statement and the proof of it can be expressed in PA if you like, in which case it is a theorem inside of PA about PA. There's no contradiction there, as long as you have everything quoted properly.

in which case the step going from PA + ?(?C -> C) |- ?(?L -> C) to PA + ?(?C -> C) |- ?(?(?L -> C)) seems legitimate given PA |- (?X -> ?(?X))

That's a valid deduction, but I don't think it's a step that anyone has written down in this thread before. It's not clear to me where you are going with it.

In any case, if we do not use the deduction theorem to derive the implication in Löb's Theorem, what do we use?

We use 10 steps, 9 of which are proofs inside of PA, and one of which is the fact that if PA |- X then PA |- "PA |- X".

**larry_d-anna**on You Provably Can't Trust Yourself · 2008-08-19T21:42:25.000Z · score: 0 (0 votes) · LW · GW

A puzzle: How can one rigorously construct Self-PA as recursively axiomatized first order theory in the language of PA?

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-19T14:57:13.000Z · score: 0 (0 votes) · LW · GW

simon: Let me explain some of the terminology here, because that may be where the confusion lies.

A *scentence* is a finite string symbols that satisfies a certain set of syntactic constraints.

A *theory* is a (possibly infinite) set of sentences. PA is a theory.

A *proof* from a theory T is a finite set of sentences, each of which is either an element of T, or follows from the ones before according to a fixed set of rules.

The notation PA + X, where X is a sentences is just the union of PA and {X}.

The notation PA |- Y means that a proof from PA that ends in Y exists.

Now I have left out some technical details, like what exactly are the syntactic constraints on sentences, and what is the fixed set of rules for proofs, but we have enough to say what the deduction theorem means. It says

PA + X |- Y => PA |- "X -> Y"

or in english: if there is a proof from the theory PA + X to the scentence Y, then there is a proof from PA alone that X->Y.

So, you see, the deduction theorem is really just a technical lemma. It meerly shows that (in one particular way) our technical definition of a first order proof behaves the way it ought to.

Now on to Lob's theorem, which says that if PA |- "◻C -> C" then PA |- "C". Now in general if you want to prove PA |- A implies that PA |- B, one way to do it is to write a first order proof inside of PA that starts with A and ends with B. But that is *not* what is going on here. Instead we start with a proof of "◻C->C" and do something *totally different than a first order proof inside of PA* in order to come up with a proof that PA |- C.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-19T05:21:55.000Z · score: 2 (2 votes) · LW · GW

Psy-Kosh: There are two points of view of where the flaw is.

My point view is that the flaw is here:

"Löb's Theorem shows that, whenever we have ((◻C)->C), we can prove C. Applying the Deduction Theorem to Löb's Theorem gives us, for all C: ((◻C)->C)->C"

Because, in fact Lob's Theorem is: (PA |- "◻C -> C") => (PA |- "C") and the Deduction theorem says (PA + X |- Y) => (PA |- "X->Y"). We don't have PA + X proving anything for any X. The deduction theorem just doesn't apply. The flaw is that the informal prose just does not accurately reflect the actual math.

Eliezer's point of view (correct me if I'm wrong) is that in the cartoon, we have 10 steps all of the form "PA proves ...." They each follow logically from the ones that came before. So they look like a proof inside of PA. And if they were a proof inside of PA then the deduction theorem would apply, and his contradiction would go through. So the flaw is that while all of the steps are justified, one of them is only justified from *outside* of PA. And that one is step 8.

Both of these views are correct.

Brian Jaress:

I think if you used Lob's Theorem correctly, you'd have something like: if PA |- []C -> C then PA |- C [Lob] PA |- ([]C -> C) -> C [Deduction]

This is incorrect because the if-then is outside of PA. The deduction theorem does not apply.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-19T01:31:08.000Z · score: 0 (0 votes) · LW · GW

"I just tested and anecdotally confirmed a hypothesis made with very little data: I suspected that neither Douglas Knight nor Larry D'Anna, the two who pinpointed 8 as the critical step, would be among the objectors to my metaethics. (Either of them can torpedo this nascent theory by stating otherwise.)"

I like your metaethics just fine.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-18T17:09:54.000Z · score: 1 (1 votes) · LW · GW

Psy-Kosh: No that isn't the problem. If there is a proof that 1=2, then 1=2. If there isn't, then 1=2. Either way 1=2. The problem is the mixing of meta-levels / inappropriate use of the deduction theorem / tacit assumption that ◻a -> ◻b is the same as "a -> b".

Sebastian Hagen: no ◻X is PA |- "X". My best guess as to what Eliezer meant by "interpret the smiley face as saying.." is that he meant to interpret the cartoonproof as a proof from the assumption "(◻C -> C)" to the conclusion "C", rather than a proof from "◻(◻C -> C)" to "◻C".

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-18T15:35:02.000Z · score: 6 (6 votes) · LW · GW

Eliezer: "Larry, interpret the smiley face as saying:

PA + (◻C -> C) |-"

OK ignoring the fact that this is exactly what it *doesn't* say, I suppose we could notice that every step is surrounded at the top by a happy-face-says, so, if we may be inclined to think we can take those off and get a proof inside PA, starting with "◻C -> C" as a hypothesis. Lets see what happens.

- ◻L <-> ◻(◻L -> C)
- ◻C -> C

ok these are our hypothesis.

- ◻(◻L->C) -> (◻◻L -> ◻C)

Modus Ponens works in PA, fine

- ◻L -> (◻◻L -> ◻C)

ordinary MP

- ◻L -> ◻◻L

if PA can prove it, then PA can prove it can prove it

- ◻L -> ◻C

ordinary MP

- ◻L -> C

ordinary MP

- ◻(◻L -> C)

ARGH WHAT ARE YOU DOING. THERE IS NO RULE OF FIRST ORDER LOGIC THAT ALLOWS YOU TO DO THIS. TO SAY "if i can prove X then i can prove i can prove X" STEPS OUTSIDE OF FIRST ORDER LOGIC YOU LOSE.

- ◻L

ordinary MP

10 C

ordinary MP

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-18T06:47:28.000Z · score: 2 (2 votes) · LW · GW

Eliezer: "Why doesn't the given proof of Lob's Theorem, the steps 1-10, show that PA + "◻C -> C" |- C?"

That's just not what it *says*. The hypothesis in step 2 isn't "◻C -> C" it's "◻(◻C -> C)". I suppose if the Hypothesis were "◻C -> C" we could try to find where it breaks. Step 7 is the only step that depends on 2 so it has to be there. Translating the amusing cartoons, we have

- ◻(◻C -> C)
- ◻(◻L -> ◻C)
- ◻(◻L -> C)

Lets say that instead we have

2'. ◻C -> C

Well what are we supposed to do with it? We're stuck. Just because ◻C -> C doesn't mean that PA can prove it.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-18T06:10:01.000Z · score: 3 (3 votes) · LW · GW

Robert: "You can only say (((◻C)->C)->(◻C))"

No that's not right. The theorem says that if PA proves "◻C -> C" then PA proves C. so that's ◻(◻C -> C) -> ◻C.

The flaw is that the deduction theorem does not cross meta levels. Eliezer says "Löb's Theorem shows that, whenever we have ((◻C)->C), we can prove C." and goes on to claim that (◻C->C)->C. But he's intentionally failed to use quotes and mixed up the meta levels here. Lob's Theorem does not give us a proof in first order logic from ((◻C)->C) to C. It gives us a proof that if there is a proof of ((◻C)->C) then there is a proof of C. Which is an entirely diffirent thing altogether.

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-18T05:04:45.000Z · score: 2 (2 votes) · LW · GW

Eliezer: "Any supposed analysis that does not tell you to mention a particular step, is inexact at best, and more probably mistaken."

I'm pretty sure my answer was completely correct. Care to point out the inexactness/mistakes?

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-18T02:05:16.000Z · score: 0 (0 votes) · LW · GW

oops that should be "what the deduction theorem really says"

**larry_d-anna**on The Cartoon Guide to Löb's Theorem · 2008-08-18T01:54:12.000Z · score: 8 (8 votes) · LW · GW

The flaw is the instant you used the deduction theorem. You used it to go from

PA |- "◻C -> C" -> PA |- "C"

to

PA |- "(◻C->C) -> C"

What the induction theorem really says is

T + X |- Y implies T |- "X -> Y"

so what you really would have needed to apply the deduction theorm would have been

PA + "◻C -> C" |- C

do I win?

**larry_d-anna**on The Bedrock of Morality: Arbitrary? · 2008-08-15T15:21:30.000Z · score: 0 (0 votes) · LW · GW

Roko: What the heck does morality have to do with category theory at all?

**larry_d-anna**on "Arbitrary" · 2008-08-12T19:43:16.000Z · score: 4 (4 votes) · LW · GW

A related sense of "arbitrary", which is common in math and CS, is "could be anything, and will probably be chosen specifically to annoy you".

wikipedia on nets:

net or Moore-Smith sequence is a generalization of a sequence, intended to unify the various notions of limit and generalize them to arbitrary topological spaces.

**larry_d-anna**on Moral Error and Moral Disagreement · 2008-08-12T19:13:00.000Z · score: 0 (0 votes) · LW · GW

Virge: Why is it "not the point"? In this discussion we are talking about differences in moral computation as implemented within individual humans. That the blind idiot's global optimization strategy defines homosexuality as a defect is of no relevance.

well because we're trying to characterize the sort of psychological diversity that can exist within our species. And this psychological unity argument is saying "we're all the same, except for a mix of one-step changes". This means that any complex adaptation in any human is in almost all humans. The exceptions being: sexual dimorphism, and the fact that certain individuals are "defective", in the sense that one of their complex adaptations is broken. So if you're arguing against this position and saying: look at homosexuals, they are different but not broken, then you aren't talking about the same kind of "broken". I'm not arguing that we should make base any moral judgment on evolution's silly ideas of what's broken. I'm just arguing that homosexuals don't serve as a counterexample to the idea of psychological unity.

Virge: I wasn't claiming a complex adaptation. I was claiming "other computations that could exhibit a superficial unity, but with a broad spread."

I think you're right here. But hopefully the spread is not so much that we could not come to agree if we "knew more, thought faster, were more the people we wished we were, and had grown up farther together."

An example that gives me hope that this could be true is Vengeance. We seem to be born with a terminal value saying that it is good to see our enemies suffer. In terms of explicit moral argument, if not in deeds, we have mostly come to agree that it is wrong to take vengeance; even though some of us came from cultures that once reveled in it; even though some of us may be more naturally inclined towards it.

When you do Bayesian updates, the specific values of your initial priors become less important as you gather more evidence. The evidence piles up exponentially, and you need exponentially bad priors to keep the wrong answer in the face of it. Perhaps our moral debates are similar. Perhaps the "broad spread" is not so great that we cannot come to agree, if we consider enough evidence, if we hear enough arguments. Psychological unity does not prove that this is so, but without psychological unity, there would be no reason to hope it is possible.

**larry_d-anna**on Moral Error and Moral Disagreement · 2008-08-12T16:06:00.000Z · score: 0 (0 votes) · LW · GW

Roko: I think Eliezer has explicitly stated that he is a realist.

**larry_d-anna**on Moral Error and Moral Disagreement · 2008-08-12T14:45:26.000Z · score: 1 (1 votes) · LW · GW

Virge: The argument for psychological unity is that, as a sexually reproducing species, it is almost impossible for one gene to rise in relative frequency if the genes it depends on are not already nearly universal. So the all the diversity within any species at any given time consists of only one-step changes; no complex adaptations. The one exception of course is that males can have complex adaptations that females lack, and vice versa.

So, with respect to your specific examples:

Homosexuals: sexual preference certainly is a complex adaptation, but obviously one that differs between males and females. Homosexuals just got the wrong sexual preference for their equipment. And it doesn't do any good to say that they aren't defective. They aren't defective from a human, moral point of view, but that's not the point. From evolutions view, there's hardly anything more defective, except perhaps a fox that voluntarily restrains it's own breeding.

Stanford Prison Experiment, Psychopaths: I'm not sure if I see where the complex adaptation is here. Some people have more empathy, some less. Even if the difference is supposed to be genetic, there seem to be a lot of these flexible parameters in our genome. Empathy-level could be like skin-color, height, hairiness, etc. We all have the machinery to compute empathy (we all have the same complex adaptation), but it's used more often, or carries more influence in some people and less in others. Those that totally lack empathy are like albinos. They have the genes that are supposed to code for empathy, but they're broken.

Of course you are right that empirical data on this question is needed. But absent that, we have what looks like a strong theoretical argument for psycological unity.

**larry_d-anna**on Moral Error and Moral Disagreement · 2008-08-11T17:04:30.000Z · score: 0 (0 votes) · LW · GW

Will Pearson: Why not just treat them as pure functions in the State monad?

**larry_d-anna**on Inseparably Right; or, Joy in the Merely Good · 2008-08-10T03:43:17.000Z · score: 1 (1 votes) · LW · GW

Richard: You were correct. That is indeed the strongest pro-life argument I've ever read. And although it is quite wrong, the error is one of moral reasoning and not merely factual.

**larry_d-anna**on Inseparably Right; or, Joy in the Merely Good · 2008-08-10T01:16:34.000Z · score: 1 (5 votes) · LW · GW

Richard: Abortion isn't a moral debate. The only reason people disagree about it is because some of them don't understand what souls are made of, and some of them do. Abortion is a factual debate about the nature of souls. If you know the facts, the moral conclusions are indisputable and obvious.

**larry_d-anna**on Inseparably Right; or, Joy in the Merely Good · 2008-08-10T01:13:01.000Z · score: 1 (3 votes) · LW · GW

Roko: "And, of course, this lack of objectivity leads to problems, because different people will have their own notions of goodness."

Don't forget the psychological unity of mankind. Whatever is in our DNA that makes us care about morality at all is a complex adaptation, so it must be pretty much the same in all of us. That doesn't mean everyone will agree about what is right in particular cases, because they have considered different moral arguments (or in some cases, confused mores with morals), but that-which-responds-to-moral-arguments is the same.

**larry_d-anna**on Morality as Fixed Computation · 2008-08-08T15:51:06.000Z · score: 1 (1 votes) · LW · GW

@Tom McCabe: "Beware shutting yourself into a self-justifying memetic loop. If you had been born in 1800, and just recently moved here via time travel, would you have refused to listen to all of our modern anti-slavery arguments, on the grounds that no moral argument by negro-lovers could be taken seriously?"

Generally I think this is a valid point. One shouldn't lightly accuse a fellow human of being irredeemably morally broken, simply because they disagree with you on any particular conclusion. But in this particular case, I'm willing to take that step. If I know anything at all about morality, then I know murder is wrong.

@Alan Crossman, Roko:
No, I do not think that the moral theory that Eliezer is arguing for is relativism. I am willing to say a paperclip maximizer is an abomination. It is a thing that **should** not be. Wouldn't a relativist say that passing moral judgments on a thing as alien as that isn't meaningful? Don't we lack a common moral context by which to judge (according to the relativist)?

Let me attempt a summary of Eliezer's theory:

Morality is real, but it is something that arose here, on this planet, among this species. It is nearly universal among humans and that is good enough. We shouldn't expect it to be universal among all intelligent beings. Also it is not possible to concisely write down a definition for "should", any more than it is possible to write a concise general AI program.

**larry_d-anna**on Morality as Fixed Computation · 2008-08-08T03:34:09.000Z · score: 2 (4 votes) · LW · GW

Tom McCabe: speaking as someone who morally disapproves of murder, I'd like to see the AI reprogram everyone back, or cryosuspend them all indefinitely, or upload them into a sub-matrix where they can think they're happily murdering each other without all the actual murder. Of course your hypothetical murder-lovers would call this imoral, but I'm not about to start taking the moral arguments of murder-lovers seriously. You just have to come to grips with the fact that the thing we call Morality isn't anything special from a global, physical perspective. It isn't written in the stars, it doesn't follow from pure logic, it isn't simple or easy to describe. It's a big messy, complicated aspect of *our* specific nature as a species.

Coming to grips with this fact doesn't mean you have to turn into a moral relativist, or claim that morality is made of nothing but arbitrary individual preference. Those conclusions just don't follow.

**larry_d-anna**on Contaminated by Optimism · 2008-08-06T02:41:20.000Z · score: 1 (1 votes) · LW · GW

steven: your "not 100% sure" is a perfect example of the problem eliezer is trying to explain. "not 100% sure that X is false" is not a valid excuse to waste thought on X if the prior improbability of X is as incredibly tiny as it is for thoughts like "paperclip maximizers will find their own paperclip-related reasons not to murder everyone".

**larry_d-anna**on Anthropomorphic Optimism · 2008-08-05T00:20:31.000Z · score: 2 (2 votes) · LW · GW

Roko: What would it even mean for an objective value to be implicit in the structure of the universe? I'm having a hard time imagining any physical situation where that would even make sense. And even if it did, it would still be *you* that decides to follow that value. Surely if you discovered an objective value implicit in the structure of the universe that commanded you to torture kittens, you would ignore it.

**larry_d-anna**on Anthropomorphic Optimism · 2008-08-04T21:28:18.000Z · score: 4 (4 votes) · LW · GW

james andrix: we have to worry about what other Optimizers want, not just if they "think correctly". Evolution still manages to routinely defeat us without being able to think at all.

**larry_d-anna**on The Meaning of Right · 2008-07-29T15:10:12.000Z · score: 2 (2 votes) · LW · GW

Caledonian: He isn't using "too-big" in the way you are interpreting it.

The point is not: Mindspace has a size X, X > Y, and any set of minds of size > Y cannot admit universal arguments.

The point is: For any putative universal argument you can cook up, I can cook up a mind design that isn't convinced by it.

The reason that we say it is too big is because there are subsets of Mindspace that do admit universally compelling arguments, such as (we hope) neurologically intact humans.

**larry_d-anna**on The Meaning of Right · 2008-07-29T15:01:45.000Z · score: 0 (0 votes) · LW · GW

Roko: You think you can convince a paperclip maximizer to value human life? Or do you think paperclip maximizers are impossible?

**larry_d-anna**on Math is Subjunctively Objective · 2008-07-28T01:19:26.000Z · score: 1 (1 votes) · LW · GW

"But my dear sir, if the fact of 2 + 3 = 5 exists somewhere outside your brain... then where is it?"

For some reason most mathematicians don't seem to feel this sort of ontological angst about what math really means or what it means for a mathematical statement to be true. I can't seem to articulate a single reason why this is, but let me say a few things that tend to wash away the angst.

it doesn't matter "where it is", it is a proven consequence of our axioms.

it is in every structure in the universe capable of representing integers and performing arithmetic on them.

there are many ways you can define the real numbers, but they're all isomorphic. When making statements like "2 + 3 = 5" we don't need to worry about which version of the reals we're talking about; it's true for all of them.

there's a hierarchy of types of mathematical questions. At the bottom are recursive ones: questions we could answer with a big enough computer and enough time. Then there are R.E. questions: questions that if-the-answer-is-yes, we can confirm with a big enough computer and enough time (also, co-R.E., for if-the-answer-is-no). R.E. + co-R.E. is exactly the questions you can write in first-order logic (with the variables taking on integer values) with symbols for all recursive functions and only one quantifier. More quantifiers move you further up the hierarchy. Past that there are questions like the continuum hypothesis that aren't even

*about*numbers, and don't seem to be constrained by anything physical. So even if you feel quite uneasy about what some mathematics means, remember that the stuff low on the hierarchy can be on solid ground even if the higher stuff isn't.

**larry_d-anna**on The Conscious Sorites Paradox · 2008-04-28T15:35:25.000Z · score: 0 (0 votes) · LW · GW

stephen: If we had a full understanding of fundamental physics then the only other a priori assumption we should need to derive the Born rule should be this: We aren't special. Our circumstances are typical. In other words: it is possible that at a fundamental physical level there is no Born rule and no reason one should expect a Born rule. But just by some fantastic coincidence, *our* little branch has followed the born rule all this time. In fact, we should expect it to stop following the Born rule immediately, for the same reason someone who's just won the lottery doesn't expect to win again next time. It's not physically impossible for us to be this lucky, but it's not physically impossible for an egg to unscramble itself either.

Fundamental physics + eggs don't unscramble + anthropic principle should give you the Born rule. If it doesn't then physicists aren't done yet.